ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
Authors: Haoran You
Yipin Guo
Yichao Fu
Wei Zhou
Huihong Shi
Xiaofan Zhang
Souvik Kundu
Amir Yazdanbakhsh
Yingyan Celine Lin
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models (LLMs); Efficient LLMs; Multiplication-less networks; Hardware acceleration
Primary area: deep_learning_architectures
TL;DR: We propose accelerating pretrained LLMs through a "post-training" shift and add reparameterization, towards efficient multiplication-less LLMs, dubbed ShiftAddLLM.

Abstract: Large language models (LLMs) have shown impressive performance on language tasks but face challenges when deployed on resource-constrained devices due to their extensive parameters and reliance on dense multiplications, resulting in high memory demands and latency bottlenecks. Shift-and-add reparameterization offers a promising solution by replacing costly multiplications with hardware-friendly primitives in both the attention and multi-layer perceptron (MLP) layers of an LLM. However, current reparameterization techniques require training from scratch or full parameter fine-tuning to restore accuracy, which is resource-intensive for LLMs. To address this, we propose accelerating pretrained LLMs through post-training shift-and-add reparameterization, creating efficient multiplication-free models, dubbed ShiftAddLLM. Specifically, we quantize each weight matrix into binary matrices paired with group-wise scaling factors. The associated multiplications are reparameterized into (1) shifts between activations and scaling factors and (2) queries and adds according to the binary matrices. To reduce accuracy loss, we present a multi-objective optimization method to minimize both weight and output activation reparameterization errors. Additionally, based on varying sensitivity across layers to reparameterization, we develop an automated bit allocation strategy to further reduce memory usage and latency. Experiments on five LLM families and eight tasks consistently validate the effectiveness of ShiftAddLLM, achieving average perplexity reductions of 5.6 and 22.7 points at comparable or lower latency compared to the most competitive quantized LLMs at 3- and 2-bit precision, respectively, and more than 80% memory and energy reductions over the original LLMs. Codes and models are available at https://github.com/GATECH-EIC/ShiftAddLLM.

ShiftAddLLM: Accelerating Pretrained LLMs via Post- Tratniraining Multiplication-Les Reparameterization
Haoran You', Yipin Guoi, Yichao Fu', Wei Zhou', Huihong Shit, Xiaofan Zhang' Souvik Kundu Amir Yazdanbakhshi Yingyan (Celine) Lin
Georgia Institute of Technology Intel Labs *Google Google DeepMind
[haoran. celine.lin I @ gatech.edu, eic-lab @ groups groups.gatech.ed. souvikk intel.com, *1 [xiaooanz. ayazdan| @ google.com
Abstract
Large language models (LLMs) have shown impressive performance on language tasks but face challenges when deployed on resource-constraile devices due to their extensive parameters and reliance dense multiplications, resulting in high memory demands and latency bottlenecks. Shift-and-ade reparameterization offers promising solution by replacing costly multiplications with hardware-friende primitives in both the attention and multi-layer perceptron (MLP) layers of an LLM. However, current reparameterization techniques require training from scratch full parameter fine-tuning to restore accuracy, which is resource-intensive for LLMs. To address this, we propose accelerating pretrained LLMs through post-training shift-and-add reparameterization, creating efficient mulippicattinn-fre models, dubbed ShiMtAddLLM. Specifically, we quantize each weight matrix into binary matrices paired with group-wise scaling factors. The associated multiplications are reparameterized into (1) shifts between activations and scaling factors and (2) queries and adds according to the binary matrices. To reduce accuracy loss, we present a multi-objective optimization method to minimize both weight and output activation reparameterization errors. Additionally, based on varying sensitivity across layers to reparameterization, we develop an automated bit allocation strategy to further reduce memory usage and latency. Experiments on five LLM families and eight tasks consistently validate the effectiveness of ShiftAddLLM, achieving average perplexity reductions of 5.6 and 22.7 points at comparable or lower latency compared to the most competitive quantized LLMs at 3- and 2-bit precision, respectively, and more than 80% memory and energy reductions over the original LLMs. Codes and models are available at https : /tt://gttttt ///ggtth.b..... COmm/GATECH-ELC ShiftAddLLM.
Introduction
Pretrained LLMs have demonstrated state-of-the-arr performance in language understanding and generation tasks [46, 47, 59, 3, 74, 57, 58, 2]. However, deploying these LLMs incurs significant hardware demands, including high latency, memory, and energy consumption, especially on edge or cloud GPU devices. The primary bottlenecks are their immense parameter sizes and the associated multiplication operations. For instance, GPT-3, with 175 billion parameters, requires 350GB of memory in FP16 format [38] and performs 10 floating-point operations (FLOPs) for a single forward pass [19]. Previous efforts to improve LLM efficiency have focused on pruning [40, 55, 20, 24, 44], quantization [63, 38, 18, 48], and attention optimization [12, 71, 67]. However, these methods still rely on costly multiplication operations in both the attention and MLP layers.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Hardware cost under 45mm CMOS [27, 69, 23, 50, 5].
Multiplication
Add
Shift
LUTs
OPs
FPS2
FP16
INT32
INT8
FP32
FP16
INTS2
INT8
INT32
IN716
INT8
(8-bit Query)
Energy (pJ)
3.7
0.9
3.1
1.1
0.4
0.1
0.03
0.13
0.057
0.024
0.37 (8 OPs)
Area (am²)
7700
1640
3495
282
4184
1360
137
157
73
34
787 OPs)
Note that I LUT corresponds to 8 operations, as each bit in queries is from weight element.
We identify a promising yet unexplored opportunity for improving LLM efficiency: reparameterizing their extensive multiplications with more cost-effective hardware substitutes, such as bitwise shifts and adds. Inspired by practices in computer architecture and digital signal processing, replacing multiplications with bitwise shifts and adds [66, 22] can offer up to 3.1/o.1 = 31x energy and 1195/137 R 26x area reductions (see Tab. 1). This hardware-inspired approach can lead to efficient and fast implementations, previous techniques that require training from scratch or extensive fine-tuning, we propose a new method to integrate the shift-and-add concept into LLMs through post-training optimization.
shown by previous research on Shift [69, 70, 72]. Unlike
To design multiplication-less LLMs, we need to address three key challenges: First, how can we effectively reparameterize pretrained LLMs with shifts and adds in post-training manner? Previous reparameterization techniques [69, 72] can result in nontrivial quantization errors, requiring fine-tuning or retraining to avoid accuracy drops. We aim to develop ready- -to-use post-training reparameterization method for LLMs. Second, how can we mitigate the accuracy drop from shift- and-add reparameterizationn? Approximating original multiplications with lower-bit shifts and adds typically reduces model accuracy. Most studies resort to fine-tuning or increasing model sizes, complicating L.LM deployment. We hypothesize that optimizing both weight and activation erross can minimize overall reparameterization error, aligning with recent activation- -wware weight quantization methods in LLMs. Third, how can we handle varying sensitivities to reparameterization across different layers and blocks in LLMs? An automated strategy to determine the optimal number of bits for reparameterized weights in each layer is needed. More vulnerable layers should have higher- bit representations, while less sensitive layers can use lower-bii bit representations. This ensures no bottlenecked layers due to aggressive reparameterization and maximizes redundancy exploitation. To the best of our knowledge, this is the first attempt to address these three challenges for multiplication- less LLMS through post-training reparameterization. Our contributions are summarized as follows:
We propose accelerating pretrained LLMs via post-training bitwise shift-and-add reparameter- ization, resulting in efficient multiplication-les LLMs, dubbed ShiftAddLLM. All weights are quantized into binary matrices paired with group-wise scaling factors; the associated multiplications are reparameterized into shift-and-ade operations.
To mitigate accuracy loss, we present multi-objece objectiv optimization method aligning and optimizing both weight and output activation objectives. minimizing overall reparameterization error, and achieving lower perplexity and better task accuracy.
We introduce mixed and automated bit allocation strategy that determines the optimal number of bits for reparameterized weights per layer, based on their vulnerability to compression. Susceptible layers receive higher-bic bit representations, while less sensitive ones get lower-bit representations.
Our extensive results across five LLMSs and eight tasks consistently show the superior accuracy and efficiency trade-offss achieved by A.ft with average perplexity reductions of 5.6 and 22.7 at comparable or even lower latency compared the most competitive quantized LLMs at three and two bits, respectively, and more than 80% memory and energy reductions over the original LLMs.
2 Related Works
LLM Quantization. Significant efforts have been made to quantize LLMs, including quantization-
aware training (QAT) [39, 52] and post-training quantization (PTQ) [18, 38, 63, 15]. QAT requires calibrated data and significant retraining resources, whereas PTQ is more dominant due to it lower
computational and time overhead. There are two prevalent PTQ strategies for LLMs: (I) uniform quantization of both weights and activations [63, 15, 68]. often limited to 8 bits (W8A8) as lower bit representations can significantly reduce accuracy; and (2) lower bit weight-only quantization [18, 48, 14, 28, 6], which quantizes LLM weights to lower bits while keeping activations in FP16 format.

---

This approach alleviates memory bottlenecks associated with the vast parameters of LLMs. For instance, GPTQ [18] uses gradient-based weight quantization and develops INT3/4 kernels to reduce data movements, and LUT-GEMM [48] eliminates the dequantization and uses custom LUTT-based CUDA kernels to reduce memory and computation costs. In contrast, SMiftAddI.LM is the first to employ the shift-and-add idea for reparameterizing pre-trained LLMs. This reparameterization reduces bit usage for weights and replaces costly multiplications with hardware-friendl] primitives, further reducing energy, latency, and memory.
Multiplication-les Models. The efficient model community has focused on reducing or replacing
multiplications. CNNs, binary networks [10, 32] binarize weights and activations, while shift-based
networks use spatial shifts [62] or bitwise shifts [16] to substitute for multiplications. AdderNet [7, 65, 61] replaces multiplications with additions, albeit with a small accuracy drop. ShiftAddNet [69] reparameterizes CNNs with cascaded shift and add layers. These techniques have been adapted to
Transformers. BiLLM [28] introduces binary LLMs, while [54] and [60] extend the addition or shift concepts to the attention mechanisms, respectively. ShiftAddViT [72] reparameterizes pretrained Vision Transformers (ViTs) with shifts and adds. Contemporary work MatMul free LM [76] leverages additive operators and Hadamard products for multiplication--fee free language model training, relying on FPGAs for speedups. Compared to closely related works like ShiftAddNet [69] and MatMul-free LM [76]. which requires training from seratch, and ShiftAddViT [72], which demands extensive parameter fine-tuning, ShiftAddLLLM applies the shift-and-add concept to pre-trained L.Mss without additional training or fine-tuning. We also use a multi-objective optimization and automated bit allocation strategy to further improve accuracy or reduce GPU latency, energy, and memory usage.
3 Preliminaries
Binary-coding Quantization (BCQ). BCQ [64] quantizes each weight tensor an L-layer LLM w E Rmxxn into q bits using a linear combination of binary matrices {b,}== and corresponding scal-
Algorithm Alternating Multi-bit BCQ [64]
ing factors {ar) where b, E {-1,1)mxn The weights are then approximated by W. = Ei= a, b, result of minimizing the quantization error, i.e., arg min b. l/ - Ei=1 b,l; to obtain the opti-
I: Input: Full-precision weight w E R",
bit-width g. alternating cycles T
2: Output: a; b, E {-1,1 Function MULTI-BIT BCQ v,,,,,T)
4: {a;, b, -- + GREEDY(w)
mal oz, bj. If is 1, then the problem collapses to binary quantization, which has an analytical solution:
for 4- 1 to T do
b" sign(w) [egnww)cr a* = w b" /n. For multi-bit quan-
6:
{a;]] + LS w)
[b, [= + BS (a), aa.......pp. Kgg w)
tization, we resort to greedy and alternating meth- ods [64, 30, 33]. shown in Alg. 1. Initially, we use the greedy method [21] to initialize a;, b,, where the i-th bit quantization is performed by minimizing the residual r from the (i - 1) bit:
end for 9: end Function
8:
min |r-11 a.b:ll? where r Σ aj 1<isg. a.,b.
(1)
We then obtain the initialized oi, by sequentially as b, sign(r:) and a; r. b,/n (Line 4). Next, we perform alternating optimization to further minimize the quantization error. Specifically, {o: can be iteratively refined using ordinary least squares (I.S) [21] as [oi,..., 0g)
0g]= - T B) BB w)
where B [b....., by] E {-1, 1 (Line 6). The binary codes (b, == can then be iteratively
recalibrated using a binary search (BS) given the refined {a, (Line 7) [64].
Such BCQ can support both uniform and non-uniform quantization formats by adjusting the scaling factors and biases accordingly [48]. Our ShiftAddLLM is built on top of BCQ but further replaces all associated multiplications with lower-cost hardware substitutes (e.g., shifts, adds, and LUT queries). We optimize not only the weight quantization eтror but also the output activation error, thereby achieving lower quantization bits along with savings in energy, memory, and computational costs.
Shift and Add Primitives. Direct hardware implementation of multiplications often inefficient. Using shift and add operations as "shortcuts" provides more efficient alternative. Shifts, which are
3

---

(b) Previous Weight only Quantization
(c)
Scaling Input Act.
FP16 Shift Using MultL
FP16 Shift Using UINT16 Add
(SAbitt)
X (FP16)
Factor
3.14
22
De-Quant
FP16 Shift
Sign
Exponent
Mantissa
MatMul. (FP16)
B-bit Key
Shifted
Act.
(d) Construct LUTs and Query&Add
(FP16)
Construct
LUTs
Xx
X1
Construct LUTA
LUT Key Nalue
LUT Key Vatue
LUT nus
Valua
Add
(b) Ours ShintAdLLM
Binary
Xg
Lg
Weight
X3d
W (<4bit) (FP16)
D=O+
Query &Add
Quary
ShiRAddLLM
C
Shifted
(FP16)
Output Act.
Binary Weight by (1100 1++1)
Output Act. O
Figure I: Illustration of our proposed post-training reparameterization for ShiftAddLLMM.
equivalent multiplying by powers of two, offer non-uniform quantization solution and can result in significant savings. For example, we tested matrix multiplication from one MLP layer of OPT-66B between weight W E I2 and activation A R x9116 using FP16 MACs and our 3-bit ShiftAddLLM. Energy consumption was 80.36J vs. 9.77J, achieving 87.8% savings with our method. Both primitives have inspired many innovations in efficient model innovations [7, 16, 69, 72].
4 The Proposed ShiftAddLLM Framework
Overview. We introduce our ShiftAddLLM as follows: First, we describe the reparameterization of pretrained LLMs through post-training shift-and-ade approach Sec. 4.1. Second, enhance accuracy, we introduce a multi-objective optimization method that accounts for both weight quantiza- tion error and output activation error, detailed Sec. 4.2. Third, to improve efficiency, we explore mixed and automated bit allocation strategy, illustrated in Sec. 4.3.
4.1 ShiftAddLLM: Post-training Reparameterization of LLMs with Shift and Add Primitives
Post-fraining Reparameterization of LLMs. To avoid the need for fine-tuning after reparameteriza- tion, our method closely mimics the original multiplications used in LLMs. Previous methods, such as weight-only quantization techniques [18), employ gradient-based activation-aware uniform quan- tization to fit the pretrained weight distribution better, thereby achieving lower quantization errors. However, these methods often lack direct hardware support and require on-the-fly dequantization to FP16 for multiplication with activations, as depicted in Fig. (a). In contrast, our ShiftAddLLM uses the BCQ format, supporting non-uniform quantization with customized CUDA kernels [48, 29], bypassing the need for dequantization, as illustrated in Fig. 1 (b). In particular, our method employs the Alg. quantize pretrained weights into binary matrices (b, [b,} and scaling factors {as )2
Note that during the alternating optimization cycles, we further quantize all scaling factors to powers of two (PoT) [37], as described by the equation:
og POT (rk-1) POT Σ ag), where POT(a) sign(a) 2P. 15kSK. (2)
This additive PoT method adopts greedy strategy to enhance the representational capacity of PoT, using K scaling factors, where the k-th PoT minimizes the residual PoT effectively quantizes the scaling factor into sign(a): 2P, where sign(a: indicates sign flips, P = round(log, (abua())) (abs) and 2P denotes a bitwise shift to the left (P > 0) or right (P < <).
the (I- PoT. Each
After the above reparameterization, we can then replace the associated multiplication between weights and activations into two steps: (1) Bitwise shifts between activations and scaling factors. Note that the activation is still in the FP16 format, and the multiplication between floating-point number and a positive or negative PoT integer can efficiently implemented by an integer addition instruction on existing hardware following SenseShift [36], as also illustrated in Fig. (c); (2) Queries and

---

(a) Weight Objective
(c) Ours Multi-OOjective Optimization
min|
Wall
min|| wx
min| ||w
wall"
min wx wax]
(
bi
biz
bis
ba
oq
On
X
Ora baz X2
Foi
ba
bag
X2
02
og ban
ba
b12
Dan
or bag Xg
02
bai
Dui
Dast
Dan
Dm
basz
Bann
or Dua X2
Weight Wa
Input Oulput
Weight Wa
Input.Act...
Output Act
(b) Activation Objective
min| wx
wox||?
min/ l/wl
wall"
min |WX-
W
min||w-
byg b1s
biu bon
[o:
[2
0g
los dn
te1::
(ba baa
ban
by bgg_
b1s,
[=1:
[b1
bya
ban
by byg
bre
Dun
Dud"
bun
2=
bas
bgg
bzu
Dud
baa
bas
Output Act
Welght Wq
t=n:
(bn
byz
bmn
Output Act.
Weight
Wa
[b2
b2a
bau
Figure 2: Illustration of our proposed multi multi-objective optimization framework.
adds intermediate shifted activations with the binary matrices. To implement this efficiently and reduce redundant additions or accumulations, as shown Fig. 1 (d), we pre-compute 256 (= 2a) possible values for every eight elements in the shifted activations to construct LUTs. Here every eight grouped binary weights form an 8-bit key. Suppose the shifted activation is an re-dimensional vector. In that case, will get n/s LUTs, where the grouped binary weights used as keys, and the precomputed partial sums are stored as values. This allows us to handle the multiplication between the binary matrix b, and the shifted activations as queries to the LUTss. We then add all the partial sums to obtain the final output activations in FP16 format. Such LUTss are well supported by existing GPU kernels [48, 29]. The reparameterization post-training manner, replacing costly multiplications with efficient hardware operations.
be applied to all weights in pretrained LLMs in
Takcaway. ShiftAddLLM presents a novel multiplication-les approach that leverages non-uniform quantization via BCQ and additive PoT. This methodology enhances the representation capacity for outlier weights and activations of large magnitude compared to uniform quantization. Moreover, additive PoT effectively resolves the issue limited quantization resolution for non-outlier weights and activations. Overall, it allows the quantization levels to better align with the data distribution.
4.2 ShiftAddLLM: Multi-objective Optimization
Motivating Analysis on Previous LLM Quantization Objectives. We examine previous weight- only quantization methods to understand the causes large quantization error and accuracy drop. These methods typically use either weight or activation objective to minimize quantization error. Specifically, the "weight objective" (see Fig. 2 (a)) aims to minimize the weight quantization error, i.., I| Wgll and adopts scaling factors for each row of quantized weights. However, this does not optimize output activation eиror, each weight element is multiplied by a unique input activation before summing to produce the output. Varying input activations, especially outliers [63, 38), rescale the weight quantization еттоr differently. causing significant divergence in the output activation. For example, LUT-GEMM [48] adopts this weight objective. On the other hand, the "activation objective" (see Fig. 2 (b)) minimizes the output activation error, i.e., ||W W, ||. by quantizing one column of weights at a time and continuously updating the remaining unquantized weights to compensate for the quantization error incurred by quantizing single weight column. However, the fixed scaling factors may not adequately accommodate the weights adjusted afterward. OPTQ [18] employs this activation objective.
Our Multi-Objective Optimization. To further mitigate accuracy drop after reparameterizztion (see Sec. 4.1). we introduce multi-objective optimization framework that combines weight and activation objectives using column- wise scaling factors. This framework effectively reduces quantization error for both weights and activations, thereby improving the accuracy of LL.
As shown in Fig. 2 (c), using column-wise scaling factors overcomes the limitations of the previous weight objective [48] climinating the impact of varying input activations quantized weights.
5

---

Each scaling factor corresponds to constant activation value. Additionally, scaling factors for subsequent columns are updated gradually after compensating for the corresponding column's weights, ensuring better fit than the previous activation objective [18].
Accuracy vs. Latency Trade- offs. The column-wise scaling factor design significantly boosts accuracy after reparameterization. However, it does not fully lever- age BCQ [48, 29I, which process eight elements per row of weights in parallel as LUT keys, resulting in lateney overhead for models with 230B parameters. For exam- ple, testing on the OPT-30B [74] model and WikiText-2 dataset [41] showed (16.3 9.6) = 6.7 perplexity reduction but with (44.1 (4..1-33.2//M.1 R 24.7% latency overhead, as shown Fig. (b).
(a) Block-wise Scaling Factors
(b) Design Comparisons
bigs
Q1
bis
bigo
b19
21
16
bis
bia bgo
OPT-30B
Row-wise
16.3 [ 33
PPL ()) Lat.
03
b
bis
CK
b
b
b
(4 biu
Column-wise
9.6 44.1
Block-wise
9.9 33.5 (
bm
bis
bma
bmis
bmn
Figure 3: (a) the block-wise scaling factors and (b) the compari- son among different designs on OPT-30B [74].
To address this, we propose a block-wise scaling factor design that groups 8 columns and V8 of the original rows to share scaling factor, ensuring compatibility with the BCQ kernel and achieving latency reductions, as shown in Fig. 3 (a). We refer to ShiftAddLMM with column- scaling factors as "Ours (Acc.)" for high accuracy optimization, and with block-wise scaling factors "Ours (Lat.)" for optimized accuracy-latency trade-off.
Takeaway. Our multi-objective optimization approach integrates both weight and activation ob- jectives, reducing weight quantization error in an activation-aware manner and output activation error reduction in weight-aware manner. This synergy. achieved through simple column-wise or block-wise design, significantly boosts the accuracy of weight-only quantization. This aligns with the principles of previous activation-awan weight quantization methods [38].
4.3 ShiftAddLLM: Mixed and Automated Bit Allocation
Sensitivity Analysis. We analyze the sen- sitivity of different layers and blocks in LLMs to shift-and-add reparameterization. As shown in Fig. 4, later blocks incur more quantization or reparameterization errors. Within each block, Query/Key (Q/K) layers are generally more sensitive reparameterization than other linear lay- ers. This diverse sensitivity motivates us explore mixed bit allocations for L.LM reparameterization and develop strategies automatically determine the optimal bit allocations given the average bit budgets.
Quant. per Param.
10-1
bits bits
0.5
bits
bits
0.4
bits
10-1
0.2
10-2
1
17
V
20 FC1
Blocks
Linear Layers Block
Figure 4: Sensitivity analysis on OPT-1.3B [74].
Criteria and Automated Bit Allocation.
To develop the bit alloca-
tion scheme, we propose criteria to estimate the importance linear weights and formulate the bit allocation problem. For weight W, from the i-th layer of an L.M,, the criterion G is defined as follows:
Kendall 0.905
integer programming
Rank w.rt. Reparam. Error
200
150
C ||[||| r STD(IS)? where
(3)
100
IS W:=:://iag(ehhmee (w./daa((cholessx(xxxx))]
where the importance score (IS) inspired by Optimal Brain Com- pression [25, 17, 18] and is correlated to the increase in the quadratic reconstruction error |W - W x after reparameterizing the weights, i.e., IS t, error increases J. The F-norm of indicates the overall importance of W, while the standard deviation (STD) high- lights the reparameterization difficulty for outliers. Considering both factors, achieve more effective evaluation metric proportional
100
150
200
Rank Criteria
Figure 5: Rank comparisons.

---

Table Perplexity comparisons of the OPT models on WikiText-2. Note that we set the group size of all methods as the length of rows following the setting of OPTQ [18] for fair comparison.
OPT (PPL.)
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
FPI6
27.65
22.00
14.62
12.47
10.86
10.13
9.56
9.34
OPTQ [18]
53.85
33.79
20.97
16.88
14,86
11.61
10.27
14.16
LUT-GEMM [48]
60.00
42.32
49.10
17.55
17.44
12.50
139.90
100.33
AWQ [38]
3
54.75
35416.00
24.60
39.01
16.47
16.53
31.01
5622.00
Ours (Acc.)
31.29
24.24
21.53
13.68
11.18
10.39
9.63
9,43
OPTQ [18]
2
2467.50
10433,30
4737.05
6294.68
442.63
126.09
71.70
20.91
LUT-GEMM [48]
4844.32
2042.90
3851.50
616.30
17455.52
4963.27
7727.27
6246.00
AWQ [38]
3514.61
18313.24
9472.81
22857.70
8168.30
5014.92
7780.96
103843.84
QulP [6]
2
92.84
146.15
27.90
30.02
16.30
12.34
11.48
10.92
Ours (Acc.)
51.15
40.24
29.03
20.78
13.78
12.17
10.67
10.33
to the actual reparameterization error. As shown in Fig. 5, the rankings derived from our defined criteria and the actual reparameterization error are highly correlated, with Kendall T of 0.905. To refine the criteria by incorporating the bit-width, we use least squares polynomial fits to estimate each bit's corresponding reparameterization error as Ci
Given the criteria, we formulate the automated bit allocation as integer programming problem:
Σ
Σ Bi. Cr
E
3r, 1, ΣΣ B.b.
arg min BLb
s.t.
where L is the number of layers in the target LLM. b E (2,3.4) denotes the available bit widths, and 3e,6 E {0, 1} is the one-hot indicator for the i-th layer to determine the assigned bits, c.g., {0, 1, means 3 bits. The objective is to minimize the summed criteria C of all layers under the given average bit budget B per layer. The final Bi, represents the assigned bits for the i-th layer the target LLM.
Takeaway. Using mixed bits instead of static ones can improve the accuracy-efin -ffircccffficinncy tradeoffs by adapting the varying sensitivities across layers, e.g., Q/K linear layers exhibit higher sensitivity to reparameterization; Our adopted criteria provide a quick estimation of the reparameterization error.
5
Experiments
5.1 Experiment Settings
Models. We consider five representative SOTA LLM families, including OPT [74], LLaMA-1/243 [58, 21. Gemma [42], Mistral [31], and Bloom [49]. Tasks and Datasets. We evaluate all five LLMs on the commonly adopted language modeling task using the WikiText-2 [41] dataset for perplexity measurement. Additionally, we extend the evaluation of the two largest models, OPT-66B and LLaMA-2-70B, eight downstream tasks for zero-shot accuracy evaluation. These tasks include ARC (Challenge/Fasy) [4], BoolQ [9]. Copa [1], PIQA [56], RTE [11], StoryCloze [43], and MMLU [26]. Baselines. We consider four SOTA LLM quantization methods: OPTQ [18], LUT- GEMM [48], QulP [6], and AWQ [38]. Evaluation Metrics. We evaluate SdiftAddLLM and the baselines using both accuracy and efficiency metrics. For accuracy, we evaluate perplexity on the WikiText-2 dataset and zero-shot accuracy on eight downstream tasks. For efficiency, we measure the latency on a single A100-80GB GPU (PCIe) [45] and estimate the energy costs using an Eyeriss like hardware accelerator [8, 75], which calculates not only computational but also data movement energy (within 18% of the differences with Eyeriss's chip measurement results as claimed).
5.2 ShiftAddLLM over SOTA LLM Quantization Baselines
Results on OPT Models. To evaluate the effectiveness of our ShiftAddLLM, we compare against four SOTA LLM quantization baselines: OPTQ [18], LUT-GEMM [48], QulP [6], and AWQ [38]. Using the OPT model family [74] and the WikiText-2 dataset [41], we assess perplexity. GPU latency, and energy costs. As shown in Tab. 2, Ours (Acc.) consistently outperforms all baselines, achieving average perplexity reduction of
compared to OPTQ, LUT-GEMM.

---

OPTQ
LUT-GEMM
(Lat.)
(a) OPT (3-bit)
(b) OPT (2-bit)
(c) LLaMA-2 (3-bit)
(d) LLaMA-3 (2-bit)
(e) Gemma (3-bit)
6.5
30
Perplexity
10a
25
10a
5.0
10
15
10'
10
10
10a
15
20
25
10
20
30
4.0
50
25
50
75
50
75
18
20
Latency (ms)
Latency (ms)
Latency (ms)
Latancy (ms)
Latency (ms)
Figure 6: Accuracy-latency tradeoff comparisons on the OPT, LLaMA-2/3, and Gemma models.
and AWQ, respectively, at bits. 2 bits, where most baselines fail with significantly high perplexity, method maintains low perplexity, and achieves average 22.74 perplexity reduction over the most competitive QulP. Also, as shown in Fig. 6 (a & b), Ours (Lat.) consistently achieves better accuracy-latenes tradeoffs, with a perplexity reduction of or 6.5%~~60. 1% latency reductions and 26.0%~<4..79 energy savings at similar or even lower perplexity. Complete quantitative data on accuracy, latency, and energy is provided in Appendix A.
at comparable latency
Results on LLaMA Models. We further evaluate ShiftAddLLM on LLaMA models [57, 58, 2] due to their superior performance among open-source LLMs. As shown in Tab. 3, Ours (Acc.) consis- tently outperforms all baselines, achieving average perplexity reduction of 1.82/1.47/0.29 and 80.87/4606.988 988.996666777774 com- pared to OPTQ, LUT-GEMM, and AWQ at and bits, respectively. Evaluating Ours (Lat.) with both accuracy and latency metrics as shown in Fig. 6 (c & d), Ours (Lat.) demonstrates better accuracy-latency tradeoffs. It achieves 1. ~ 111.199997.6 perplexity reduction at comparable latency or 19.9%~65.00% latency reductions and 28.4%~~89.99 energy savings at similar or even lower perplexity. Complete quantitative data accuracy, latency, and energy provided in Appendix B.
Table 3: Perplexity comparisons of the LLaMA models on WikiText-2. The group size is set 128 following [48, 38].
LLaMA-I
LLaMA-2
LLaMA-3
LLaMA (PPL.
Bits
7B
7B
13B
70B
8B
70B
FP16
16
5,68
5.47
4.88
3.32
6.14
2.86
OPTQ [18]
8.81
6.43
3.88
13.69
4.91
LUT-GEMM [48]
3
7.18
7.02
5.89
4.01
11.10
5.92
AWQ [38]
3
6.35
6.24
5.32
3.74
8.15
4.69
Ours (Acc.)
6.04
5,89
5.16
3.64
7.20
4.35
OPTQ [18]
2
68.60
19.92
12.75
6.82
398.0
26.65
LUT-GEMM [48]
303.00
2242.0
2791.0
136.4
19096
3121
AWQ [38]
2.6e5
2.2e5
1.2e5
7.2c4
1.7c6
1.7c6
Ours (Acc.)
7.98
Results on Gemma/Mistral/Bloom Models. We also evaluate Shif- IAddLLM on Gemma [42], Mis- tral [31], and Bloom [49] models, which are among the most popular
Table 4: Results on Gemma/ViistallBlooom models.
PPL (.)
Bits
Gemma-2B Mistral-7B Bloom-3B Bloom-7B
FP16
16
13.88
5.25
13.48
11.37
open-source LLMs and Mixture-of- Expert (MoE) models. in Tab. 4, Ours (Acc.) achieves per- plexity reductions of 11.12/29.4 for Gemma-2B. 1.67/16.76 76 for Mistral- 7B, and 3.30/6.93 and 1.76/5.58 for BLOOM-3B/7B, respectively, compared to OPTQ and LUT- GEMM. As shown in Fig. 6 (e), Ours (Lat.) shows better accuracy-latency tradeoffs, e.g., achieving 9.56 perplexity reduction and 11% latency reductions over the OTPQ baseline on Gemma models. These results five LLM families consistently validate the effectiveness of our ShiftAddLLM.
OPTQ
3
26.08
7.27
17.40
13.47
shown
LUT-GEMM
44.36
22.36
21.03
17.29
Ours (Acc.)
3
14,96
5.60
14.10
11.71
Zero-shot Downstream Tasks. We extend our evaluation to zero-shot downstream datasets for more comprehensive assessment. As shown in Tab. 5, Ours (Acc.) consistently improves performance over previous SOTA baselines, achieving an average accuracy gain of 13.37/13. 19 and 2.55/2.39 over OPTQ and LUT-GEMI basclines at 3 bits when evaluated on OPT-66B and LLaMA-2-700

---

Table 5: Accuracy comparisons on seven downstream tasks for OPT-66B and LLaMA-2-70BB
Models
Methods
Bits
ARC_ C
ARC_E E
Copa
BoolQ
PIQA
Storycloze
RTE
MMLU
Mean
Floating Point
16
37.20
71.25
69.82
78.67
77.47
60.65
25.89=0.37
63.37
OPT-66B
OPTQ [18]
3
24.66
48.86
70
52.05
64.47
67.09
53.07
23.98-40.36
50.52
LUT-GEMM [48)
3
24.15
51.85
81
53.52
61.97
60.60
48.74
23.73:10.36
50.70
Ours (Acc.)
3
35.24
70.88
87
72.45
77.64
77.15
63.18
27.56-00.38
63.89
Flouting Point
16
49.57
76.14
82.57
80.79
78.61
68.23
65.24:10.3
72.89
LLaMA-2-70B
OPTQ [18]
45.82
76.34
77.34
67.51
60.14:+0.36
72.33
LUT-GEMM [48]
3
47.70
76.42
80.31
80.20
77.78
68.59
Ours (Acc.)
3
48.38
77.06
93
84.25
80.47
78.49
75.09
62.33- 00.38
74.88
Table 6: Perplexity and latency results of our mixed bit allocation.
PPL (L)
Latency (ms)
Methods
Bits
125M
350M
1.3B
2.7B
6.78
13B
125M
350M
1.3B
2,7H
6.7B
13B
Ours (Lat.)
2
712.55
445.78
40.28
50.95
18.56
14.76
6.3
12.4
12.3
16.9
16.9
20.9
Ours (Mixed)
2.2
435.84
279.19
27.37
31.97
17.99
13.79
6.3
12.6
12.5
16.8
16.7
21.0
respectively. These experiments demonstrate that our method not only reduces perplexity but also improves downstream task accuracy.
GPU Memory Savings. Our LM also reduces GPU memory usage. For OPT-66B, our method saves 81% and 87% memory costs over FP16 at 3 (23GB vs. 122GB) and 2 bits (16GB vs. 122GB), respectively. For LLaMA-2-70B, it saves 80% and 87% memory costs at 3 (25GB vs. 128GB) and 2 bits (17GB vs. 128GB), respectively.
Results of Mixed Bit Allocation. We evaluate our mixed bit allocation strategy (see Sec. 4.3) and compare Ours (Mixed) with Ours (Lat.). As shown in Tab. 6, Ours (Mixed) further reduces the perplexity by an average of 79.45 for OPT model families under comparable or even less latency. We provide more results in Appendix F to validate the effectiveness of our mixed bit allocation strategy.
5.3 Ablation Studies of ShiftAddLLM
Visualization of Mixed Bit Allocation. We visualize the bit allocations after applying our automated bit allocation strategy with an aver- age bit budget of 2.2 (Fig. 7). The allocation pattern correlates with the sensitivity to reparam- eterization identified in Sec. 4.3 and shown in Fig. 4. For instance, later blocks, which expe- rience more quantization or reparameterization errors, receive more bits. The K linear layers and the first MLP (FC1) in each block are also allocated higher bits. This visualization con- firms that our strategy effectively adjusts bits according to reparameterization errors.
Average Bits
2.4
188
1.6
1111151719222
Blocks
Linear Layers
Figure 7: Visualizing the average bit allocation.
Performance and Energy Breakdown. To examine the contribution of each proposed technique, we con-
Table 7: Performance breakdown analysis.
ducted ablation studies OPT-6. 7B-/3B models. As
shown in Tab. 7, the vanilla ShiftAddLLM (Sec. 4.1) suffers from a significant perplexity increase with 2- bit reparameterization. Our mujti-obeecctve optimiza- tion (Sec. 4.2) reduces perplexity by an average of 3. 3,9e4, and the mixed bit allocation strategy (Sec. 4.3) further reduces perplexity by 0.77, maintaining com- parable latency. These experiments validate the ef-
PPI.
Latency (ms)
OPT w/ Sec.
Bits
6.7B
13B
6.7B
13B
4.1
6.4e4
1.5e4
16.5
20.1
4.1&4.2
18.56
14.76
16.9
20.9
4.184.2&
4.3
2.2
17.99
13.79
16.7
21.0
fectiveness of each component in ShiftAddI.LA an Eyeriss accelerator illustrates the energy breakdown of the original LLMs and ShiftAddLLMs.
addition, profiling the two largest models on

---

Energy Breakdown OPT-66B
Energy Breakkdow LLaMa-2-708
PP Shift
LUTx
Ours
83.1% Saving
Ours
87.4% Saving
FP Molt
Others
Figure 8: Energy breakdown for OPT-66B and LLaMA- models using an Eyeriss accelerator.
As shown in Fig. S reduces energy consumption by 87.2% for OPT-66B and 86.0% for LLaMM-2-77BB, with shift-and-add leading to 89.7% and 89.9% energy reduction compared to original multiplications.
5.4 Discussion Limitation
We demonstrated the accuracy and efficiency of post-training shift-and-add reparameterization of LLMs using multi-objective optimization and automated bit allocation, addressing the challenge of efficient LLM serving. However, achieving GPU speedup relied on BCQ kernels and the compatible Ours (Lat.) with a block-wise scaling factor design. While Ours (Acc.) with column-wiss design delivers high accuracy, we lack the fast CUDA kernel required for similar speedups.
5.5 Discussion on Technique Applicability Beyond LLMs
We acknowledge that the idea of shift-and-add reparameterization is general and can be extended to other smaller models like CNNs [69] or ViTs [72]. Meanwhile, this work's implementation specifically dedicated to large-scale LLMs: It is the first instance of applying the shift-and-ade technique at the scale of LLMs with billions of parameters. While many ideas perform well with models having millions of parameters, they often fail to scale effectively. Unlike previous methods that require additional training and do not yield good results for large-scale LLMs, our approach is uniquely tailored for LLMs. We incorporate "post-training" reparameterization and carefully designed scaling factor patterns, enabling multi-objective optimization for LLMs and ensuring superior performance compared prior quantization methods.
6
Conclusion
We propose accelerating pretrained LLMs through post-training shift-and-add reparameterization, creating efficient multiplication free models. Our method reparameterizes weight matrices into binary matrices with group-wise scaling factors, transforming multiplications into shifts and adds. To mitigate accuracy loss, introduce multi-objective optimization strategy that minimizes weight and activation reparameterization errors. Additionally, we develop an automated bit allocation strategy based on layer sensitivity further improve the accuracy-efficiency tradeoff. Extensive results across various LLM families and tasks validate the effectiveness ShiftAddl L.M. This work opens a new perspective on designing efficient LLM serving systems through post-training optimization.
Acknowledgments and Disclosure of Funding
This work is supported by the National Science Foundation (NSF) RTML program (Award number: 1937592) and the CoCoSys, of the seven centers in JUMP 2.0, Semiconductor Research Corporation (SRC) program sponsored by DARPA. We extend our gratitude towards Mitchelle Rasquinha, and Robert Hundt for reviewing the paper and providing insightful feedback. We also thank the extended team at Google DeepMind who enabled and supported this research direction.
References
[1] Ardavan Afshar, loakeim Perros, Evangelos E Papalexakis, et al. COPA: Constrained PARAFAC2 for sparse & large datasets. In CIKM, 2018.
Meta AI. LLaMA
wtwpaa/....oomettaaalaaaaa aalaammmmmmmmaa 11ama3, 2024.
[3] Rohan Anil. Sebastian Borgeaud, et al. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:2312. 11805 2023.
10

---

[4] Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, et al. A Systematic Classification of Knowledge. Reasoning, and Context within the ARC Dataset. arXiv preprint arXiv:1806.00358, 2018.
[5] Diogo Brito, Taimur G Rabuske, Jorge R Fernandes, et al. Quaternary logic lookup table in standard CMOS. TVLSI, 2014.
[6] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, al. QulP: 2-Bit Quantization of Large Language Models With Guarantees. NeurIPS, 2024.
[7] Hanting Chen, Yunhe Wang. Chunjing Xu, et al. AdderNet: Do We Really Need Multiplications in Deep learming? In CVPR, 2020.
[8] Yu- Hsin Chen, Tushar Krishna, Joel S Emer, et al. Eyeriss: An Energy-efficcce Reconfigurable Accelerator for Deep Convolutional Neural Networks, JSSCC, 2016.
[9] Christopher Clark, Kenton Lee. Ming- Ming-Wei Chang, al. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv prepriat 1..Kv:1995.100444 2019.
[10] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, al. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or-1. arXiv preprint 1rXiv:160..02830.
2016.
[11] Ido Dagan, Dan Roth, Fabio Zanzotto, et al. Recognizing Textual Entailment: Models and Applications. Springer Nature, 2022.
[12] Tri Dao, Dan Fu, Stefano Ermon, et al. FlashAttention: Fast and Memory-Fificient Exact Attention with 10-Awareness. NeurlPS, 2022.
[13] Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovteharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow precision inferencing cloud scale with microsoft floating point. Advances in neural information processing systems, 33:10271-1022 2020.
[14] Tim Dettmers and Luke Zettlemoyer. The Case for 4-bit Precision: k-bit Inference Scaling Laws. In ICML, 2023.
[15] Tim Dettmers. Mike Lewis, Younes Belkada, et al. GPT3.int8() 8-bit Matrix Multiplication for Trans- formers at Scale. NeurIPS, 2022.
[16] Mostafa Elhoushi, Zihao Chen, Farhan Shafiq, al. DeepShift: Towards Multiplication-Les, Neural Networks. In CVPR, 2021.
[17] Elias Frantar and Dan Alistarh. Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning. NeurIPS, 2022.
[18] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, et al. OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In ICLR, 2022.
[19] Amir Gholami, Zhewei Yao, Schoon Kim, ct al. Al and Memory Wall. IEEE Micro Journal, 2024.
[20] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, al. The Unreasonable Ineffectiveness of the Deeper Layers. arxiv preprint arXiv:2403.18887 2024.
[21] Yiwen Guo, Anbang Yao, Hao Zhao, al. Network Sketching: Exploiting Binary Structure in Deep CNNs. In CVPR, 2017.
[22] Bah-Hwee Gwee, Joseph Chang, Yiqiong Shi, al. Low-Voltage Micropower Asynchronous Multiplier With Shift-Add Multiplication Approach. IEBE Transactions on Circuits and Systems I: Regular Papers, 2008.
[23] Song Han, Xingyu Liu, Huizi Mao, et al. EIE: Efficient Inference Engine on Compressed Deep Neural Network. ACM SIGARCH Computer Architecture News, 2016.
[24] Simla Burcu Harma, Ayan Chakraborty, Elizaveta Kostenok, Danila Mishin, Dongho Ha, Babak Falsafi, Martin Jaggi, Ming Liu, Yunho Oh, Suvinay Subramanian, and Amir Yazdanbakhsh. Effective Interplay between Sparsity and Quantization: From Theory to Practice. arXiv preprint arXiv:2405. 2400:203555 2024.
[25] Babak Hassibi, David G Stork. and Gregory J Wolff. Optimal Brain Surgeon and General Network Pruning. In IEEE international conference neural networks, 1993.
11

---

[26] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
[27] Mark Horowitz. 1.1 Computing's Energy Problem (and what we can about it). ISSCC, 2014.
[28] Wei Huang, Yangdong Liu, Haotong Qin, al. BiLLM: Pushing the Limit of Post- -as-Training Quantization for LLMs. arXiv preprint arXiv:2402.04291, 2024.
[29] Yongkweon Jcon, Baeseong Park, Sc Jung Kwon, et al. BiQGEMM: Matrix Multiplication with Lookup Table For Seiryy-Coding-aaan Quantized DNNs. In SC, 2020.
[30] Yongkweon Jeon, Chungman Lee, Eulrang Cho, al. Mr. Post-Training Non-Unifoma Quantization based on Minimizing the Reconstruction Error. In CVPR, 2022.
[31] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, ct. Mistral 7B, arXiv proprinntaxXivv2211000222 2023.
[32] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides. Local Binary Convolutional Neural Networks. In CVPR, 2017.
[33] Se Jung Kwon, Dongsoo Lee, Yongkweon Jeon, al. Post-Training Weighted Quantization of Neural Networks Language Models. https://openrreiiew.nee http:///wwweeeeeeerettootteeaaaa
2021.
[34] Jung Hyun Lec, Jconghoon Kim, Se Jung Kwon, and Dongsoo Lec. Flexround: Learablec rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning, pages 18913-18939. PMLR, 2023.
[35] Jung Hyun Lee. Jeonghoon Kim, June Yong Yang. Se Jung Kwon, Eunho Yang, Kang Min Yoo, and Dongsoo Lee. Lrq: Optimizing post-training quantization large language models by learning low-rank seight-scaling matrices. arXiv preprint arXiv:2407.11534 2024.
[36] Xinlin Li, Bang Liu, Rui Heng Yang, et al. DenseShift: Towards Accurate and Efficient Low-Bit Power-of- Two Quantization. In ICCV, 2023.
[37] Yuhang Li, Xin Dong, and Wei Wang. Additive Powers-of Two Quantization: An Efficient Non-uniform Discretization for Neural Networks. arXiv preprint
13144, 2019.
[38] Ji Lin, Jiaming Tang, Haotian Tang, et AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv preprint arXiv:2306.00975, 2023.
[39] Zechun Liu, Barlas Oguz, Changsheng Zhao, et al. LLM-QAT: Data-free Quantization Aware Training for Large Language Models. arXiv preprint arXiv:2305. 2023.
[40] Xinyin Ma. Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. NeurlPS, 2023.
[41] Stephen Merity, Caiming Xiong, James Bradbury, et al. Pointer Sentinel Mixture Models. ICLR, 2017.
[42] Thomas Mesnard, Cassidy Hardin, et al. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.0822 082955, 2024.
[43] Nasrin Mostafazadeh, Michael Roth, Annie Louis, al. L.DSee 2017 Shared Task: The Story Cloze Test. In Proceedings of the 2nd Workshop Linking Models of Lexical. Sentential and Discourse-Ievel Semantics, 2017.
[44] Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, and Maryam Mehri Dahnavi. SLoPe: Double- Pruned Sparse Plus Lazy Low-rank Adapter Pretraining of LLMs.. arXiv preprint arXiv:2405.1622 2024.
[45] NVIDIA Corporation. www. nvidia. com/ http////tttttt//eee////////////e aatdiaaaa0o--dattaee us- wwwiiiiiii----- 1058100- r4- wew.wwwbb.b... pdf, 2020. Datashcet.
NVIDIA
A100 Tensor
Core Center/aa00/ a100/pdf/
GPU.
https://
[46] OpenAI. CharGPT: Language Model for Dialogue Generation. https: httpp:://www.opnae/ com/ ccm/chatgpt// 2023. Website.
[47] OpenAL. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023.
12

---

[48] Gunho Park. Baeseong Park, Minsub Kim, et al. LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models. arXiv preprint arXiv:220.005557, 2022.
[49] Teven Le Scao, Angela Fan, Christopher Akiki, et al. BLOOM: PoB--Parameter Open-Aceess Multilin- gual Language Model. arXiv preprint arXiv:221 1.05100, 2022.
[50] Olivier Sentieys. Approximate Computing for DNN. In CSW 2021-HiPEAC Computing Systems Week, 2021.
[51] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models, arXiv preprint wwxxv22308.131333 2023.
[52] Xuan Shen, Zhenglun Kong, Changdi Yang. al. EdgeQAT: Entropy and Distribution Guided Quantization- Aware Training for the Accelcration of Lightweight LLMs on the Edge. arXiv preprint arXiv:2402. 10787, 2024.
[53] Huihong Shi, Haoran You, Yang Zhao, Zhongfeng Wang. and Yingyan Lin. Nasa: Neural architecture search and acceleration for hardware inspired hybrid networks. Proceedings of the 41st IEEE/ACM International Conference on Computer
Design, pages 1 1-9, 2022.
[54] Han Shu, Jiahao Wang, Hanting Chen, et al. Adder Attention for Vision Transformer. NeurIPS, 2021.
[55] Mingjie Sun, Zhuang Liu, Anna Bair, al. A Simple and Effective Pruning Approach for Large Language Models. arXiv preprint arXiv:2306. 1695, 2023.
[56] Sandeep Tata and Jignesh M Patel. PiQA: An Algebra for Querying Protein Data Sets. In SSDBM, 2003.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 2023.
[58] Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open Foundation and Fine- Chat Models. arXiv preprint arXiv:2307.0022000 2023.
[59] Ethan Waisberg. Joshua Ong. Mouayad Masalkhi, et al. Google's Al chatbot "Bard": A Side-by-Side Side Comparison with ChatGPT and its Utilization in Ophthalmology. Eye, 2023.
[60] Guangting Wang, Yucheng Zhao, Chuanxin Tang, et al. When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism. In AAAI, 2022.
[61] Yunhe Wang, Mingqiang Huang, Kai Han, et al. AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence. arviv preprint arXiv:2101.10015, 2021.
[62] Bichen Wu, Alvin Wan, Xiangyu Yue, et al. Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions. In CVPR, 2018.
[63] Guangxuan Xiao, Ji Lin, Mickael Seznee, et al. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In ICML, 2023.
[64] Chen Xu, Jianqiang Yao, Zhouchen Lin, al. Altemating Multi-bit Quantization for Recurrent Neural Networks. arXiv preprint arXiv: 180.00010,. 2018
[65] Yixing Xu, Chang Xu, Xinghao Chen, al. Kernel Based Progressive Distillation for Adder Neural Networks. In Neur 2020.
[66] Ping Xue and Bede Liu. Adaptive Equalizer Based a LwweerrO--Twoo-uunieeel-LLM Algorithm. IEEE transactions on acoustics, speech, and signal processing, 1986.
[67] Songlin Yang. Bailin Wang, Yikang Shen, et al. Gated Linear Attention Transformers with Hardware- Effficient Training. arXiv preprint arXiv:2312.06665, 2023.
[68] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, et al. ZeroQuant: Efficient and Affordable Post- Training Quantization for Large-Scale Transformers. NeurIPS, 2022.
[69] Haoran You, Xiaohan Chen, Yongan Zhang, et al. ShiftAddNet: A Hardware-Inspired Deep Network. NeurIPS, 2020.
[70] Haoran You, Baopu Li, Shi Huihong, et al. ShiftAddNAS: Hardware- Inspired Search for More Accurate and Efficient Neural Networks. In ICLR, 2022.
13

---

[71] Haoran You, Yichao Fu, Zheng Wang, et al. When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Models. In ICML, 2024.
[72] Haoran You, Huihong Shi, Yipin Guo, et al. ShiftAddViT: Mixture of multiplication primitives towards efficient vision transformer. NeurIPS, 2024.
[73] Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee. Yan Yan, et al. Llm inference unveiled: Survey and roofline model insights. arXiv preprint 7Xiv:2402.16366 2024.
[74] Susan Zhang, Stephen Roller, Naman Goyal, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01065 2022.
[75] Yang Zhao, Chaojian Li, Yue Wang, al. DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures. ICASSP. 2020.
[76] Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yigiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian. Scalable matmul-free language modeling. arXiv preprint arXiv:2406.02522 2024.
14

---

A Complete Accuracy & Latency & Energy Data for OPT Models
We supply the complete quantitative accuracy, latency, and energy data measured the OPT model family in Tab. 8, 9, and 10, respectively.
Table 8: Perplexity comparisons of the OPT models WikiText-2. Note that set the group size of all methods as the number of columns following the setting OPTQ [18] for a fair comparison.
OPT (PPL.))
Bits
125M
350M
1.3B
2.7B
13B
30B
FP16
27.65
22.00
14.62
12.47
10.86
10.13
9.56
9.34
OPTQ [18]
53.85
33.79
20.97
16.88
14.86
11.61
10.27
14.16
LUT-GEMM [48]
60.00
42.32
49.10
17.55
17.44
12.50
139.90
100.33
AWQ [38]
54.75
35416.00
24.60
39.01
16.53
31.01
5622.00
Ours (Lat.)
3
56,96
28.72
19.69
15.28
11.80
10.70
9.89
9.62
OPTQ [18]
2467.50
10433,30 30
4737.05
6294.68
442.63
126.09
71.70
20.91
LUT-GEMM [48]
2
4844.32
2042.90
3851.50
616.30
17455.52
4963.27
7727.27
6246.00
AWQ [38]
3514.61
18313.24
9472.81
22857.70
8168.30
5014.92
7780.96
103843.84
(Lat.)
712.55
445.78
40.28
50.95
18.56
14.76
12.55
12.20
Ours (Mixed)
2.2
435.84
279.19
27.37
31.97
17.99
13.79
11.62
11.17
Table 9: A100 GPU latency comparisons the OPT model family.
OPT Latency (ms)
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
FP16
7.8
15.1
16.7
20.9
22.2
29.5
51.7
OOM
OPTQ [18]
8.3
15.9
15.0
21.5
21.1
26.4
30.1
51.5
LUT-GEMM [48]
6.3
11.7
12.6
15.5
17.0
19.5
23.7
39.5
AWQ [38]
6.2
12.1
12.3
16.3
20.0
24.5
40.9
Ours (Lat.)
6.4
13.3
12.6
16.6
16.9
20.8
30.7
54.1
OPTQ [18]
8.2
16.1
15.9
19.7
19.9
24.7
31.5
50.4
LUT-GEMM [48]
6.2
11.8
11.8
15.7
15.7
19.8
23.6
33.2
AWQ [38]
6.2
12.1
12.3
16.3
16.3
20.0
24.5
40.9
Ours (Lat.)
6.3
12.4
12.3
16.9
16.9
20.9
25.4
42.9
Ours (Mixed)
2.2
6.3
12.6
12.5
16.8
16.7
21.0
27.1
45.7
Note that we use AWQ's open-sourced INT4 kernel for measuring its latency.
Table 10: Energy comparisons on the OPT model family.
OPT Energy (J)
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
FP16
16
29.26
83.72
310.33
625.80
1573.41
3036.99
7088.39
15539.87
OPTQ [18]
3
13.77
28.63
90.12
167.37
399.28
745.37
1695.45
3658.17
LUT-GEMM [48]
3
12.83
25.19
75.73
137.08
321.54
592.31
1332.95
2858.87
Ours (Lat.)
3
11.68
21.13
59.59
103.53
235.45
424.58
938.98
1990.17
OPTQ [18]
2
12.58
24.42
73.27
132.30
309.50
570.06
1283.04
2749.32
LUT-GEMM [48]
2
12.48
23.96
70.90
127.07
295.77
542.28
1215.36
2599.77
Ours (Lat.)
2
11.41
20.17
55.80
95.67
215.27
385.31
846.54
1786.62
Ours (Mixed)
2.2
11.45
20.33
56.43
96.98
218.64
391.86
861.95
1820.55
15

---

B Complete Accuracy & Latency & Energy Data for LLaMA Models
We supply the complete quantitative accuracy, latency, and energy data measured on the LLaMA model family in Tab. 11, 12, and 13, respectively.
Table 11: Perplexity comparisons of the LLaMA models on WikiText-2.
LLaMA-2
LLaMA-3
LLaMA (PPL .)
Bits
7B
13B
70B
8B
70B
FP16
16
5.47
4.88
3.32
6.14
2.86
OPTQ [18]
3
6.43
5.48
3,88
13.69
4.91
LUT-GEMM [48]
3
7.02
5.89
4.01
11.10
5.92
AWQ [38]
6.24
5.32
3.74
8.15
4.69
Ours (Lat.)
3
6.04
5.33
3.72
7.71
4.66
OPTQ [18]
2
19.92
12.75
6,82
398.0
26.65
LUT-GEMM [48]
2
2242.0
2791.0
136.4
19096
3121
AWQ [38]
2
2.22c5
1.22c5
7.24c4
1.71c6
1.72c6
Ours (L.t.)
2
9.58
12.57
5.71
34.4
12.4
Note that the group size is set to 128 following [48, 38].
Table 12: A100 GPU latency comparisons of the LLaMA models.
LLaMA-2
LLaMA-3
LLaMA Latency (ms)
Bits
7B
13B
70B
8B
70B
FP16
16
32.6
43.1
0OM
38.8
0OM
OPTQ [18]
31.1
81.9
36.2
LUT-GEMM [48]
27.4
34.7
72.6
31.7
77.5
AWQ [38]
25.4
31.8
68.0
28.5
67.7
Ours (Lat.)
26.7
33.8
70.9
31.4
72.9
OPTQ [18]
2
34.2
38.8
82.5
36.8
91.2
LUT-GEMM [48]
2
27.5
33.3
71.0
31.7
77.2
AWQ [38]
25.4
31.8
68.0
28.5
67.7
Ours (Lat.)
2
27.7
33,9
72.1
31.9
78.3
Ours (Mixed)
2.2
27.2
34.3
75.1
30.1
76.4
Table 13: Energy comparisons of the LLaMA models.
I.L.MA-2
I.LaMA-3
LLaMA Energy (J)
Bits
7B
13B
70B
8B
70B
FP16
16
1563.44 44
3040.26
18482.5
1776.05
16445.98
OPTQ [18]
3
383.40
728.98
4297.33
504.07
3972.72
LUT-GEMM [48]
305.06
574.71
3349.01
419.64
3139.34
Ours (Lat.)
3
218.59
405.53
2309.87
326.47
2225.71
OPTQ [18]
2
293.15
552.20
3212.56
406.81
3018.87
LUT-GEMM [48]
2
279.20
524.16
3037.94
391.74
2865.8 81
Ours (Lat.)
2
198.33
365.85
2065.90
304.59
2011.15
Ours (Mixed)
2.2
201.69
372.40
2099.53
306.69
2066.64 64
16

---

C
Ablation Studies on Multi-Objective Optimization
We conduct ablation studies on different optimization objectives. As shown in Tab. 14, our multi- objective optimization demonstrates superior performance in both column- wise and block-wise scaling factor formats. It achieves average perplexity reductions of 123.25, 2.22, and 403. 18 compared to the weight-only objective, activation-only objective, and the vanilla combination of both weight and activation objectives, respectively. These experiments validate the effectiveness of multi-objective optimization approach.
Table 14: Ablation studies various optimization objectives.
OPT PPL
13B
30B
66B
Wei. Obj.
13.8
222.6
163.2
Act. Obj.
11.7
10.5
14.3
Wei. Act.
45.0
16.3
1178.1
Ours (Col.-wise)
10.4
9.6
9.4
Ours (Bik.-wise)
10.8
9,9
9.6
D Impact of Batch Sizes on Throughput
To investigate the impact of batch sizes on the achievable throughput, we have further tested the throughput our CUDA kernels and end-to-end models with increased batch sizes, as demonstrated in Fig. 9, Our Shif still outperforms all three baselines at batch size of 8 in terms of accuracy-effiiinccy trade-offs, achieving on average x2.55 x/1.39 throughput improvements compared to OPTQ, AWQ, and LUT-GEMM at similar or much better accuracy.
OPTQ LUT-GEMM
AWQ
(Lat.)
Throughput (tokens(s)
(a) OPT (3-bit)
(b) OPT (2-bit)
500
800
400
600
300
400
100
10a
10
10"
Perplexity
Perplexity
(c) Throughput of 9216X36864 MatMul (2-bit)
(d) Throughput OPT-66B (2-bit)
(e) Throughput of Llama-2-708 (2-bit)
Throughput (OPs/s)
OPTQ
Throughput (tokens/s)
Throughput (tokens/s)
(2bit)
8000
205
325
30
30
6000
4000
20
20
2000
10
10
Batch Size
Batch Size
Batch Size
Figure 9: (a-b):
tradeoff comparisons among ShiftAddLLM, OPTQ, LUT-
GEMM, and AWQ at batch size of 8. (c) Kernel throughput evaluation under batch sizes of 1, 2, and 8. (d) LLaMA-2-70B end-to-end model throughput evaluation under batch sizes of 1, 2, 4, and 8. (e) OPT-66B end-to-end model throughput evaluation under batch sizes of 2, 4, and 8.
Previously, we assumed batch size of one for mobile applications where only one user is using the L.M.. This assumption also stems from the sequential nature of LLMSs during generation, i.e., generating one token at time based on all previously generated contexts. The assumption of a batch size of also used in previous literature, such as AWQ, OPTQ, and LUT-GEMM,, to measure the latency or throughput for LLM serving.
17

---

E Benchmark with More Recent Baselines
We fürther compare our ShiftAddLLM with recent LLM quantization baselines FlexRound [34] and OmniQuant [51] on OPT and LLaMA models, As shown in Tabs. 15 & 16, our ShiftAddLLM
consistently shows better eccuracy-effcienne trade-offs, achieving an average of 0. (4-bit) 0.39 (3-bit) and 0.30 (4-bit) / 0.52 (3-bit) perplexity reduction, compared to FlexRound and OmniQuant, respectively. Note that the baseline results are directly obtained from the original paper and follow-up work LRQ [35]. In addition, we tested OmniQuant at 2 bits ourselves and found it fails for OPT models, whereas ours performs well for OPT models and also achieves an average 1.96 perplexity reduction than OmniQuant on LLaMA at 2 bits.
Table 15: Perplexity comparisons between Shif and OmniQuant using OPT models and LLaMA models on WikiText-2. The group size is set as the length of rows for OPT models and 128 for LLaMA models following baselines.
OPT
ILLMA-2
Method
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
66B
7B
13B
70B
OmniQuant [51]
4
29.45
23.19
15.04
12.76
11.03
10.30
9.65
5.58
4.95
Ours (Acc.)
4
28.72
21.59
14.98
12.65
10.95
10.20
9.63
5.58
4.96
OmniQuant [51]
3
35.66
28.2
16.68
13.8
11.65
10.87
10.00
9.83
6.03
5.28
3.78
Ours (Acc.)
3
31.29
24.24
21.53
13.68
11.18
10.39
9.63
9.43
5.89
5.16
3.64
OmniQuant [51]
2
311.39
186.9
484.51
1.1c6
9.6c5
3.6e4
9.3e3
5.2e3
11.06
8.26
6.55
Ours (Acc.)
2
51.15
40.24
29.03
20.78
13.78
12.17
10.67
10.33
8.51
6.77
4.72
Table 16: Perplexity comparisons between ShiftAddLLM and FlexRound. FlexRound set as the length of rows following the paper.
The group size of
LLaMA-2
Method
Bits
7B
13B
70B
FlexRound [34]
5.83 5.01
Ours (Acc.)
5.58 4.96
FlexRound [34]
6.34 5.59 3.92
Ours (Acc.)
5.89 5.16 3.64
F More Results for Mixed Bit Allocation
To validate the effectiveness and applicability of our automated bit allocation across different LLM models, we evaluated and compared Ours (Mixed) with Ours (Lat.). The results are shown in Tab. 17. Ours (Mixed) further reduces perplexity by an average of 96.86, 3.23, and 2.63 for OPT, LLaMA, and Gemma models, respectively, under comparable even less latency. This set of experiments further validates the applicability of our automated bit allocation strategy to different L.MMs.
Table 17: Perplexity and correlation results of our mixed bit allocation.
OPT
LLaMA
Gemma
Methods
Bits
125M
1.3B
13B
2-7B
2-13B
3-8B
2B
Correlation (r)
0.910
0.905
0.915
0.931
0.929
0.897
Ours (Lat.)
712.55
40.28
14.76
9.58
12.57
34.40
16.52 (3 bits)
Ours (Mixed)
2.2
435.84
27.37
13.79
8.97
8.16
29.72
13.89
In addition, we want to clarify that, for each model, we search for the optimal bit allocation with negligible overhead (e.g., 1% 10% of the reparameterization time). For example, it takes 0.5 seconds for searching versus 72 seconds for reparameterizing OPT-125M with single bit configuration, and minute for searching versus 13 minutes for reparameterizing OPT-13R with single bit configuration. This is achieved by leveraging the proposed proxy criteria (as shown Sec. 4.3), instead of searching according to the reparameterization errors, which is time-consuming and requires running models at
18

---

each bit. Using the proxy criteria, the bit allocation candidate rankings are highly correlated with the rankings obtained using actual reparameterizatioo errors. with a Kendall 7 of 0.910/0. 10500990 for OPT-125M/1.3B/1333 and 1991//0.2990.897 for LLLMA-7B//13B88 respectively.
G 4-Bit Results and Explanation for Using Lower Bit Widths
We further provide the 4-bit results in Tab. 18. These results show that ShiftAddLLM consistently outperforms the baselines at 4 bits, achieving average perplexity reductions of 0.90/1 1.90/1.3//1.00 ..32/1.00 and 0.44/0.22/0.02 compared OPTQILUT-GEMM//AWG using OPT models and LLaMA models, respectively.
Table 18: Perplexity comparisons of the OPT models and LLaMA models with 4-bit quantization on WikiText-2. We set the group size the length of rows for OPT models and 128 for LLaMA models following baselines for fair comparisons.
OPT
LLaMA
Method
Bits
125M
350M
1.3B
2.7B
6.7B
13B
30B
1-7B
2-7B
2-13B
3-88
OPTQ [18]
31.12
24.24
15.47
12.87
11.39
10.31
9.63
6.22
5.69
4.98
7.63
LUT-GEMM [48]
4
31.93
24.09
16.15
13.34
12.09
10.40
9.99
5.94
5.78
5.06
6.85
AWQ [38]
31.66
7.4c3 (outlier)
15.22
13.19
11.23
5.78
5.60
4.97
Ours (Acc.)
28.72
21.59
14.98
12.65
10.95
10.20
9.63
5.76
5.58
4.96
6.46
We previously considered lower-bit quantization because we aim to push the accuracy-efficieng boundary to lower bits with minimal accuracy compromise. This is meaningful for large-scale LLMs, where even
bits, they remain memory-bound. As analyzed using the Roofline model shown in Figure 5 of [73], for Nvidia A6000 GPUs, the turning point from memory-bound to compute- bound is 200 arithmetic intensity (OPs/bytes). For LLaMA-7B models, all the operators in the decode/generation phase have around or less than arithmetic intensity, as shown in Table I of [73]. Even at 4 bits, the arithmetic intensity is approximately
32 8 (same ops but 4/32 fewer memory accesses), which is far less than the turning point of 200 and thus remains memory-bound let alone larger models like LLaMA-70B or beyond. Reducing from bits to bits can help increase the arithmetic intensity and thus the theoretically maximum performance by 2x, from 6144G OPS to 12288G OPS. If memory not a bottleneck for much smaller cases or prefill stages, higher bits can be used for better accuracy. Our goal is to offer an additional option and trade-off for large, memory-bound cases, without forcing the exclusive use of 2 bits.
H Comparison with MSFP
MSFP [13] is an important prior work that employs a shared exponent across a group of elements and shifts the mantissa accordingly, mimicking multiplication by powers of two. In contrast, clarify that our approach differs from MSFP in two key aspects:
1. Nature of Approach: MSFP uses shared exponents but relies various shifted mantissa to represent the weights; without this, all weights would collapse to the same value. In contrast, we do not use shared exponents for scaling factors and eliminate the need for mantissa. In particular, each scaling factor is represented as a distinct power-of-two integer (equivalent the exponents in floating- hoating-point numbers, completely removing the mantissa bits). In this way, the multiplication between a floating-poin activation and power-of integer scaling factor can be simplified to adding the corresponding integer to the exponent bit of the floating-point activation, as described in Fig. (c). In addition, rather than sharing the exponents, the entire scaling factor in ShiftAddLLM is shared across groups of binary weights in column/block wise manner, as illustrated in Fig. 3 (a) and detailed in Sec. 4.2. carefully designed optimize both weight quantization and output activation errors without conflicts. Hence, there are clear differences between the MSFP datatype and our quantization scheme. In fact, our method is orthogonal MSFP and can be combined with by representing input activations in MSFP for more aggressive performance improvements.
2. Determining Shared Exponents or Scaling Factors: The method for determining shared exponents in MSFP or shared scaling factors in our quantization scheme is different. MSFP
19

---

selects the maximum exponent to share across the bounding- box size, i.e., the number of elements sharing one exponent [13], which is simpler implementation yet might not be as adaptive. In contrast, in our ShiftAddIL.N L.M the reparameterized binary weights and scaling factors result from multi-objective optimization. This optimization adaptively designs scaling factor patterns to avoid conflicts between optimizing weight enrors and optimizing output activation errors.
Finally, in terms of the performance outcomes, MSFP bits (1-bit sign and 3-bit mantissa) already suffers from large quantization errors, as evidenced by the significant KL divergence shown in Fig. 3 of [13]. In contrast, our ShiftAddLLM at 3 or 4 bits can still achieve comparable accuracy to FP baselines. To directly compare ShiftAddLLa with MSFP, we conducted additional experiments to compare (1) quantization errors and (2) KI. divergence using both methods against their floating- point counterparts. We randomly selected ten weight matrices from OPT-350M, quantizing or reparameterizing them using both methods. The results, as summarized in Tab. 19, indicate that ShiftAddLLM consistently outperforms MSFP, achieving lower KL divergence by 0.0065, 0.0271, and 0.0952, and reducing quantization errors by 1707.3, 3251.1, 1, and 5862.0 at 4-bit, 3-bit, and 2-bit quantization, respectively.
Table 19: Comparison between MSFP and ShiftAddI.LM with varying bits on KI. Divergence and Quantization Error.
Methods
Bits
Avg. KL Divergence Avg. Quant. Error
MSFP (hounding-hox size 128)
0.0117
4129.1
ShiftAddLLM (group size 128)
0.0052
2421.8
MSFP (bounding-boo box size 128)
3
0,0434
7859.9
ShiftAdd (group size 128)
3
0,0163
4608.8
MSFP (hounding-how box size 128)
0.1485
14355.7
Shi (group size 128)
0.0533
8493.7
I Additional Clarifications on Eyeriss
As emphasized in Sec. 5, our primary focus is on GPU acceleration, specifically through the develop- ment of dedicated CUDA kernel support. worth noting that, we intentionally did not delve into specific ASIC designs in the main manuscript, which were referenced only to demonstrate potential energy savings.
To clarify the Eyeriss in estimating the energy costs, Eyeriss [8] is a well-known energy-efficient reconfigurable accelerator architecture designed for deep convolutional neural networks (CNNs). optimizes both dataflow and memory access to reduce energy consumption during neural network processing. In our work, we adapt the Eyeriss architecture by modifying its MAC (Multiply- Accumulate) array, a key component responsible for performing heavy arithmetic computations in CNNs. Instead of using traditional MAC units across the array, we replace selected units with shift, add, and lookup table (LUT) operations, aligning with our proposed ShiftAddLLA approach. This modification significantly reduces both the area and power requirements, with savings ranging from 26% to 89% in different configurations. We refer readers to Fig. 4 of NASA [53], which visually demonstrates the design principles of the overall architecture, and illustrates how replacing traditional MAC units with shift and add operations leads to significant reductions in both area and energy consumption. By adapting these principles, we enhance Eyeriss better align with the computational needs of both LLMs and ShiftAddL..e while maintaining power and area efficiency.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Sec. 5.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the commu- nity. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification:
This paper does not have theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We clarify the experiment settings and the datasets used in Sec. 5. All LLMs employed in our study are open-source models, aligning with the spirit of reproducibility. In addition, we will release the code and checkpoints upon acceptance further ensure that others can reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e-g., in the case of large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should de- scribe the architecture clearly and fully.
(c) If the contribution a new model (e.g., large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibil- models, it may be that access to the model is
ity. In the case of closed-soune limited in some way (c.g., to registered users), but should be possible for other researchers to have some path reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: All models and datasets used in this work open-source. We will release the code and checkpoints upon acceptance to ensure the reproducibility of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / wttpccccpppppccccccccccccccccccc MuuiidesCooeSSubee ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experiment settings, including models, datasets, tasks, evaluation metrics, and baselines, as described in Sec. 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the high computational cost of running all LLM experiments multiple times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, con- fidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the used GPUs, memory, and latency in Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: atttps://eeuriie /tt.rieu....pe/e
Answer: [Yes]
Justification:
Our research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (c.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader societal impact of our research in Sec. 5.4.
Guidelines:
The answer NA means that there is societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, is legitimate point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
If there are negative societal impacts, the authors could also discuss possible mit- igation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not introduce any new data or models; all experiments are conducted on existing data or models. Therefore, this paper does not pose any associated risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Yes, we properly cite all used models, datasets, and related assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. wwt.cooo//aat/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection. curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "QlVCWwFRWV", "reviewer": "Reviewer_3fe7", "summary": "The presented work replaces multiplications by shift and add operations as a post-training processing step of LLM neural network models. The proposed quantization improves a lot over SOTA methods using improved trade-offs and better control of the quantization error. Despite bit-level operations, the resulting models seem to still execute well and fast on GPUs.", "questions": "Figure 1 could use some improved explanations how the Shifted Activations are turned into LUTs. It is also unclear where the \"8-bit key\" in the ShiftAddLLM block comes from and how it is being used. By intuition I would say that the binary weights select from the LUTs. Why is then \"another\" 8-bit key necessary? LUT output is FP16 if not mistaken, please state the format as well to add clarity.\n\nLine 282: \"... using an Eyeriss-like hardware accelerator...\" Did you really use that in this work or in fact the cited \"DNN-chip predictor\" [67]? Please add clarity, especially if energy consumption was calculated and not measured.", "limitations": "The authors already list a limitation: despite the fact that GPUs can still execute the resulting quantized model well and fast, different quantization schemes require customized CUDA kernels. Reference implementations using FP16 math run without any additional effort and usually require little customization to speed them up (e.g. enable using tensor cores in ML graphs).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Unlike SOTA approaches, the proposed method is a post-processing step applied to a model and does not require any data-intensive steps (e.g. (re-)training or fine-tuning).", "weaknesses": "Clarity of the presentation could be improved.\n\nThe technique is not specific to LLMs, the presentation even lacks reference to any specific NN architecture. Therefore it is a bit unfortunate that even the title already names the technique ShiftAddLLM. As the major contribution suggests replacing multiplications in context of matrices by shift and add operations, the field of applications may extend very well beyond the restricted focus of LLMs even to outside machine learning. At least a discussion on this would be great, but I even suggest to find a better name for the technique and clearly distinguish between the generality of the approach and the specific field of application in the discussed context.\n\nLines 41-42 mentions \"... up to 31x energy and 26x area reductions (see Tab. 1).\" Please refer to the exact number pairs in the Table to help the reader.\n\nLines 83-84: please add here mention of the correct baseline for these perplexity improvements as the experimental results in fact are perplexity degrations from the FP16 baseline.\n\nLines 141-146: the presentation of energy (and area) savings are a bit misleading. In fact, a naive implementation of 32-bit multiplication from shifts and adds as of Table 1 consumes 2x the energy of the 32-bit multiplication listed there. In addition you don't list the LUT operations. This section could use some improvement to sum over all the operations in Figure 1 and compare it to the equivalent operations that it replaces instead of just repeating some maximum savings of Table 1.\n\nLines 218-219: The numbers in the text are not mentioned in the referenced table of Figure 3. Please correct and explain in more detail.\n\nLine 241, but also Figure 4 & 8: Please briefly introduce abbreviations of the layer identifiers, e.g. Q/K, V, Out., FC1, FC2.\n\nTable 3: to me moving from 3 bits to 2 bits hurts substantially wrt perplexity, being at best marginally faster. If relevant at all, only memory savings are noticable and enable to use 2 bits for the first time (over SOTA baselines) with some slight compromises on perplexity. A discussion of this would be useful.\n\nTable 5 lacks numbers of an FP16 baseline.\n\nFigure 8: although the patterns support readability in other Figures (4 & 7), they don't help here and I personally find them rather disturbing. I suggest to either use different patterns for different colors here as well or just colors with different luminance as the order of colors is the same in both diagrams.\n\n\nThe two \"Insights.\" sections (lines 181-185 and 230-234) do not add information and I suggest to remove them to gain space to improve clarity and add discussions elsewhere where needed.\n\n\nThe order and placement of tables and figures does not support the flow of reading the publication. A few suggestions:\n\nFigure 1 is better placed in front of Section 4.1 (referenced there first).\n\nFigure 2 is better placed in front of Section 4.2 (referenced there first).\n\nTable 2 is better placed in front of Section 5.2 (referenced there first).\n\nTables 5 and 4 should switch numbers.\n\nTables 7 and 6 should switch numbers.\n\nSection 5.4: the section title \"Limitation and Broader Societal Impact Discussion\" hints at more than limitations, but the section only discusses limitations. Please correct the title.\n\n\nTypos:\n\nLine 348: \"... ShiftAdLLM.\" -> \"... ShiftAddLLM.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive comments and constructive suggestions. Below are our detailed responses to your questions.\n\n**W1: The clarity of the presentation could be improved.**\n\n**(1) Technique applicability beyond LLMs: discussion needed.** You are correct that the idea is general and can be extended to other smaller models like CNNs [1] or ViTs [2]. Meanwhile, this work’s implementation is specifically dedicated to large-scale LLMs: It is the first instance of applying the shift-and-add technique at the scale of LLMs with billions of parameters. While many ideas perform well with models having millions of parameters, they often fail to scale effectively. Unlike previous methods that require additional training and do not yield good results for large-scale LLMs, our approach is uniquely tailored for LLMs. We incorporate \"post-training\" reparameterization and carefully designed scaling factor patterns, enabling multi-objective optimization for LLMs and ensuring superior performance compared to prior quantization methods. We will add this discussion to the final revision.\n\n[1] ShiftAddNet: A Hardware-Inspired Deep Network, NeurIPS'20\n\n[2] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient ViTs, NeurIPS'23\n\n**(2) Lines 41-42: Refer to the exact numbers.** Thanks for pointing this out! The number is derived by comparing adds to multiplications in terms of the INT32 format. The energy savings are 3.1 / 0.1 = 31x, and the area savings are 3495 / 137 ≈ 26x. We will clarify this in the tables.\n\n**(3) Lines 83-84: Perplexity reduction or improvements?** We apologize for the misuse of the word \"improvements.\" We are actually referring to perplexity reductions, as lower perplexity denotes better results. We will correct this.\n\n**(4) Lines 141-146: The presentation of energy savings could be improved.** We greatly value this suggestion and consider a summed energy comparison between equivalent computations. We tested matrix multiplication from one MLP layer of OPT-66B between weight $W \\in \\mathbb{R}^{9216 \\times 36884}$ and activation $A \\in \\mathbb{R}^{1 \\times 9216}$ using: (1) FP16 MACs, (2) OPTQ with 3 bits weights and (3) our ShiftAddLLM with 3 bits weights. The resulting energy consumption are 80.36J, 18.48J, and 9.77J, respectively. Our method achieves energy savings of 87.8% compared to FP16 and 47.1% compared to OPTQ.\n\n**(5) Lines 218-219: Refer to numbers in the table.** The number is derived by comparing the first two rows of Figure 3 (b). The perplexity is reduced by 16.3 - 9.6 = 6.7, and the latency overhead is (44.1 - 33.2) / 44.1 ≈ 24.7%. We will make this clear.\n\n**(6) Line 241: Briefly introduce abbreviations.** Sure, in self-attention, Q/K/V refers to linear layers for queries, keys, and values, respectively. Out, on the other hand, refers to the output linear layer. In MLPs, FC1 and FC2 refer to the two adopted linear layers. We will clarify this.\n\n**(7) Table 3: Discussion on why reducing from 3 bits to 2 bits.** Great point! We will add the following discussion: We aim to push the accuracy-efficiency boundary to lower bits with minimal accuracy compromise. This is meaningful for large-scale LLMs, where even at 3 bits, they remain memory-bound. As analyzed using the Roofline model shown in Figure 5 of [3], for Nvidia A6000 GPUs, the turning point from memory-bound to compute-bound is 200 arithmetic intensity (OPs/bytes). For LLaMA-7B models, all the operators in the decode/generation phase have around or less than 1 arithmetic intensity, as shown in Table 1 of [3]. Even at 3 bits, the arithmetic intensity is approximately 1 / 3 * 32 ≈ 10 (same ops but 3/32 fewer memory accesses), which is far less than the turning point of 200 and thus remains memory-bound, let alone larger models like LLaMA-70B or beyond. Reducing from 3 bits to 2 bits can help increase the arithmetic intensity and thus the theoretically maximum performance by 1.5x, from 8192G OPS to 12288G OPS. If memory is not a bottleneck for much smaller cases or prefill stages, higher bits can be used for better accuracy. Our goal is to offer an additional option and trade-off for large, memory-bound cases, without forcing the exclusive use of 2 bits. We will add this discussion to the final revision.\n\n[3] LLM Inference Unveiled: Survey and Roofline Model Insights, arXiv'24 \n\n**(8) Table 5 lacks an FP16 baseline.** We provide the FP16 baseline results in Table 4 of the attached PDF in our global response. Our ShiftAddLLM at 3 bits achieves comparable or even better accuracy than the FP16 baseline, for example, 72.45 vs. 69.82 for BoolQ and on average 0.52% accuracy gain across eight tasks using the OPT-66B model.\n\n**(9) Figure 8: Remove the patterns.** We will remove the patterns and use the recommended colors.\n\n**(10) Remove “Insights” sections.** Will remove.\n\n**(11) Change the order of tables/figures.** Will change orders accordingly.\n\n**(12) Correct Section 5.4 title.** Will correct the title.\n\n**(13) Typos.** Will correct it.\n\n---\n**Q1: Figure 1 could benefit from improved explanations. Intuitively, the binary weights select from the LUTs? LUT output is FP16?**\n\nYour intuition is correct. We use binary weights to serve as the key for the LUTs. The \"8-bit key\" refers to grouped binary weights, with eight binary weights forming an INT8 key. To construct the LUTs, we precompute 256 (2^8) possible values for every eight elements in the shifted activations. Suppose the shifted activation is an n-dimensional vector. In that case, we will get n/8 LUTs, where the grouped binary weights are used as keys, and the precomputed partial sums are stored as values. Yes, the LUT output is in FP16 format. We will add these details to the final revision for clarity.\n\n**Q2: Clarify whether using Eyeriss or DNN-chip Predictor.**\n\nWe used the cited DNN-chip predictor to simulate and calculate the energy (within 18% of the differences with Eyeriss’s chip measurement results as claimed). We will clarify this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jmEHIPdsEZ", "reviewer": "Reviewer_7Vce", "summary": "In this paper, the authors propose the ShiftAddLLM method to simplify complex matrix multiplications using simple shift and add operations. To enable efficient computation, they suggest assigning scaling factors in a column-wise and block-wise manner, where these scaling factors follow the form of powers of 2, replacing scalar multiplications with bit shifts. This scaling factor assignment provides an efficient trade-off between latency and accuracy. The authors conduct experiments using several well-known large language models (LLMs) across various tasks.", "questions": "Please refer to weakness", "limitations": "- The authors do not discuss various batch sizes, which are highly relevant to the limitations of the proposed scheme.\n- Only very low-bit quantization schemes are considered, and the authors do not explain why this selection was made.\n- It is difficult to estimate latency improvement from the manuscript. Do the authors plan to release open-source code?", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed methods consider quantization techniques not only for accuracy but also for efficient computation, addressing the growing demand for LLM services with an increasing number of parameters.\n2. The bit allocation method, though simple, proves effective and is demonstrated through various models.\n3. The authors show that it is possible to constrain scaling factors to follow a specific form, enabling shift operations instead of multiplications, with reasonable accuracy degradation.\n4. The experiments encompass various design explorations.", "weaknesses": "1. It appears that the authors do not discuss the impact of batch size on performance and focus solely on latency. What would be the limitations when the batch size increases in terms of throughput? While latency is an important factor, the overall service cost is dominated by throughput, which can be improved by increasing the batch size and by reducing memory footprint through quantization. The experimental results should include a description of batch sizes. If only a batch size of 1 is considered in the manuscript, the authors should explain why this consideration is practical.\n\n2. The authors claim that replacing multiplications with shift operations is important. While this might benefit ASIC design for AI inference, GPUs already have numerous multiplication units. The authors demonstrate these benefits using an Eyeriss-like hardware simulation. If this paper focuses on new ASIC design, it should be rewritten to discuss the necessary ASIC design comprehensively. Discussing shift operations alone is insufficient for a thorough discussion on new ASIC design.\n\n3. The authors do not include recently published state-of-the-art quantization methods. For example, FlexRound and OmniQuant are more advanced schemes that should be considered. LUT-GEMM does not suggest new quantization formats but addresses efficient computation methods for previously existing quantization schemes.\n\n4. This reviewer is skeptical about using perplexity (PPL) as the main metric for experiments. Measuring scores for MMLU and conducting A/B tests (using AlpacaEval with GPT-4) would better represent the quality of the proposed scheme.\n\n5. For 3-bit and 2-bit experiments, even though the proposed method might be superior to previous ones, the authors do not present results for a 4-bit setup. Compared to full-precision, score degradation is noticeable with 3-bit and 2-bit methods. This reviewer cannot understand why the authors selected such extremely low-bit methods only.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and suggestions in reviewing our work. Below are our detailed responses to your concerns.\n\n**W1 & L1: Discuss the impact of batch sizes on throughput. If only a batch size of 1 is considered, explain why this is practical?**\n\nFollowing your suggestion, we have further tested the throughput of our CUDA kernels and end-to-end models with increased batch sizes, as demonstrated in Figure 1 of the attached PDF in our global response. Our ShiftAddLLM still outperforms all three baselines at a batch size of 8  in terms of accuracy-efficiency trade-offs, achieving on average 3.37x/2.55x/1.39x throughput improvements compared to OPTQ, AWQ, and LUT-GEMM at similar or much better accuracy.\n\nPreviously, we assumed a batch size of one for mobile applications where only one user is using the LLM. This assumption also stems from the sequential nature of LLMs during generation, i.e., generating one token at a time based on all previously generated contexts. The assumption of a batch size of 1 is also used in previous literature, such as AWQ, OPTQ, and LUT-GEMM, to measure the latency or throughput for LLM serving. We will clarify this assumption.\n\n**W2: The authors show the benefits via an Eyeriss-like H.W. simulation, but GPUs already have numerous multiplication units. Rewritten to address the new ASIC design if that is focus?**\n\nWe humbly clarify and emphasize that we reported real-measured GPU latency (see Lines 281-282 and Figure 6 of the submitted manuscript) instead of simulated results to demonstrate the up to 65% latency savings, which benefit from our proposed shift-and-add reparameterization and dedicated CUDA kernel optimization for GPUs.\n\nRegarding energy savings, since we cannot directly measure energy on GPUs, we used existing simulators to estimate the energy savings. However, our focus remains on showing the practical benefits of current GPUs. We are not aiming to propose a new ASIC design in this work, as that is not our main focus.\n\n**W3: Benchmark with FlexRound and OmniQuant?**\n\nAs suggested, we further compare our ShiftAddLLM with both FlexRound and OmniQuant on OPT and LLaMA models. As shown in Tables 2 & 3 of the attached PDF in our global response, our ShiftAddLLM consistently shows better accuracy-efficiency trade-offs, achieving average 0.15 (4-bit) / 0.39 (3-bit) and 0.30 (4-bit) / 0.52 (3-bit) perplexity reduction, as compared to FlexRound and OmniQuant, respectively. Note that the baseline results are directly obtained from the original paper and follow-up work LRQ [1]. In addition, we tested OmniQuant at 2 bits ourselves and found it fails for OPT models, whereas ours performs well for OPT models and also achieves an average 1.96 perplexity reduction than OmniQuant on LLaMA at 2 bits.\n\n[1] LRQ: Optimizing PTQ for LLMs by Learning Low-Rank Weight-Scaling Matrices, arXiv'24\n\n**W4: Skeptical about using perplexity as the main metric. Show scores for MMLU and conduct A/B tests?**\n\nWe acknowledge that perplexity is not the gold metric and therefore also provided the accuracy of seven downstream tasks in Table 5 of the submitted manuscript.\n\nFurthermore, as suggested, we extend the evaluation to include the MMLU and A/B test using AlpacaEval with GPT-4. The results are shown in Table 4 and Figure 2 of the attached PDF in our global response. Our ShiftAddLLM consistently achieves 3.58% accuracy improvements over OPTQ and 3.83% over LUT-GEMM for MMLU when using the OPT-60B model. For A/B tests, we used the GPT-4 score to evaluate the quantized models against the FP16 counterpart. Our ShiftAddLLM achieves 8.6%/20.7% higher winning rates than OPTQ (29.3%) and LUT-GEMM (17.2%) when using the LLaMA-2-7B model. We provide an example of the generation comparison in Figure 2 of the attached PDF.\n\n**W5 & L2: Do not present results for a 4-bit setup. Why select such extremely low-bit methods like 3/2 bits only?**\n\nAs requested, we have provided the 4-bit results in Table 1 of the attached PDF in our global response. These results show that ShiftAddLLM consistently outperforms the baselines at 4 bits, achieving average perplexity reductions of 0.90/1.32/1.00 and 0.44/0.22/0.02 as compared to OPTQ / LUT-GEMM / AWQ, using OPT models and LLaMA models, respectively. In addition, we have also included comparisons with FlexRound and OmniQuant at 4 bits (see response to W3).\n\nWe consider lower-bit quantization because we aim to push the accuracy-efficiency boundary to lower bits with minimal accuracy compromise. This is meaningful for large-scale LLMs, where even at 3 bits, they remain memory-bound. As analyzed using the Roofline model shown in Figure 5 of [2], for Nvidia A6000 GPUs, the turning point from memory-bound to compute-bound is 200 arithmetic intensity (OPs/bytes). For LLaMA-7B models, all the operators in the decode/generation phase have around or less than 1 arithmetic intensity, as shown in Table 1 of [2]. Even at 4 bits, the arithmetic intensity is approximately 1 / 3 * 32 = 8 (same ops but 4/32 fewer memory accesses), which is far less than the turning point of 200 and thus remains memory-bound, let alone larger models like LLaMA-70B or beyond. Reducing from 4 bits to 2 bits can help increase the arithmetic intensity and thus the theoretically maximum performance by 2x, from 6144G OPS to 12288G OPS. If memory is not a bottleneck for much smaller cases or prefill stages, higher bits can be used for better accuracy. Our goal is to offer an additional option and trade-off for large, memory-bound cases, without forcing the exclusive use of 2 bits.\n\n[2] LLM Inference Unveiled: Survey and Roofline Model Insights, arXiv'24 \n\n**L3: Difficult to estimate latency improvement. Plan to release the code?**\n\nYes, we do plan to open-source the code to ensure reproducibility, as we also promised in the abstract.\n\nWe have reported the real-measured GPU latency. Our ShiftAddLLM achieves 6.5% ~ 65.0% latency reductions for OPT/LLaMA models, at similar or even lower perplexity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WqdG9rZHJ5", "reviewer": "Reviewer_4dkb", "summary": "In this paper, “ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization,” the authors propose a novel post-training reparameterization method that can perform multiplication-free operations in LLMs (Large Language Models). Through a modified APoT (Adaptive Power of Two) quantization and a new post-training method, the authors achieve higher accuracy than the other quantization methods on LLM models while reducing quantization errors. This quantization method also offers a quick quantization process without the need for a retraining procedure, which is typically time-consuming and effort-intensive when applied to LLM models. Once parameters are quantized, multiplication can be done using a simple adder with a LUT-based query multiplicator, which stores precalculated multiplication results.\n\nThis work makes a significant contribution to the field of LLM optimization by introducing a novel method that enhances efficiency and reduces computational demands without sacrificing accuracy. Its strengths lie in its innovative approach to error reduction and automated bit allocation, which together offer a compelling solution for deploying LLMs on resource-constrained edge devices. However, the method's reliance on specialized hardware support and the need for further testing in diverse scenarios are notable limitations. Despite these weaknesses, the paper's advancements in quantization strategies and performance improvements make it a valuable addition to current research efforts in model compression and an efficient AI deployment.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "By replacing multiplication operations in both the attention and multi-layer perceptron (MLP) layers, the logic area and power can be reduced. Unlike traditional quantization methods that only reduce either activation errors or weight errors, this work tries to reduce both by applying awareness of the other parameter in the error reduction effort (Multi-Objective Optimization method). To my knowledge, this is the first work that attempts to reduce errors for both weight and activation. Finally, this method is supported by an automated bit allocation strategy, which further simplifies the application of this new method to LLMs.\t\nThere are many APoT quantization variants [2, 3, 4] since it first appeared in 2019 [1]. This quantization method is already established or too old to be considered as new. However, I think this paper holds some meaningful adaptations compared to those similar works. First, the performance of the PTQ (Post Training Quantization) method is indeed better than other new quantization strategies. APTQ [5] also tried to implement Optimal Brain Quantization (OBQ) and uses Hessian trace as a sensitivity metric to reduce quantization error in an attention-aware manner. Compared to another SoTA work, APTQ, this work shows better performance within the same quantization settings (3 bits). In Table 1 of [5], the LLaMa1-7B for the WikiText-2 dataset perplexity test is given, and this work shows a better perplexity score of 6.04, which is 0.72 lower than the APTQ. Unfortunately, other results with larger LLM models (> 13B) are not given in APTQ, so this is the only comparison that could be made.\n\n\n[1] Li, Yuhang, Xin Dong, and Wei Wang. \"Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks.\" International Conference on Learning Representations, 2020. (published in arxiv first at 2019).\n\n[2] Geng, Xinkuang, et al. \"Compact Powers-of-Two: An Efficient Non-Uniform Quantization for Deep Neural Networks.\" 2024 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2024.\n\n[3] Przewlocka-Rus, Dominika, et al. \"Power-of-two quantization for low bitwidth and hardware compliant neural networks.\" arXiv preprint arXiv:2203.05025 (2022).\n\n[4] Oh, Sangyun, et al. \"Automated log-scale quantization for low-cost deep neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[5] Guan, Ziyi, et al. \"APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models.\" arXiv preprint arXiv:2402.14866 (2024).", "weaknesses": "The multi-objective optimization method is quite novel and impressive, but I think it needs to be tested with more cases. As I mentioned, this is the first work that introduces this method. One drawback is that the quality of quantization is not greatly improved, and we don’t know how much performance improvement will be there if the column-wise scaler is used, since GPU kernels for supporting this do not currently exist. Though, it offers another layer of trade-off between latency and accuracy by letting the user choose between column-wise scaler use or block-wise scaler use.\n\tAnother point I would like to note is that automated bit allocation may not be fully effective, since the different sensitivity among LLM’s layers and the use of mixed precision degrade the advantage of this work. The use of LUT query-based multiplier and mixed precision for different sensitivity is a good idea, but it is for one LLM model and may not be applicable to different LLM models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful review and constructive suggestions. Below are our detailed responses to your concerns.\n\n**W1: The multi-objective optimization method is quite novel and impressive, but it needs to be tested with more cases.**\n\nThank you for acknowledging the novelty of our multi-objective optimization method! Following your suggestions, we’ve conducted additional tests and comparisons on the OPT and LLaMA model families. \n\n***First,*** we conducted additional ablation studies using both OPT and LLaMA models, supplementing the results in Table 8 of the submitted manuscript. The new results are presented in Table 5 of the attached PDF in our global response. Our multi-objective optimization demonstrates superior performance, achieving average perplexity reductions of 20.85/8.56/112.26 on OPT models and 1.00/1.21/2.25 on LLaMA models compared to weight-only objective, activation-only objective, or their vanilla combinations, respectively.\n\n***Second,*** we tested the average layer-wise quantization errors for a direct comparison. The results are shown in the table below. Our multi-objective optimization achieves significantly lower quantization errors for both weights and output activations. Specifically, we observed average per-parameter weight quantization error reductions of 0.4 and 0.1, and total output activation error reductions of 18.9 and 53.1 compared to OPTQ and LUT-GEMM, respectively.\n\n| Model | Methods | Wei. Error (Per Param.) | Output Act. Error (Total) |\n|---|---|:---:|:---:|\n| OPT-350M | Wei. Obj. | 0.08 | 31.15 |\n| OPT-350M | Act. Obj. | 0.32 | 23.14 |\n| OPT-350M | **Ours Multi-Obj.** | **0.02** | **8.83** |\n| OPT-2.7B | Wei. Obj. | 0.12 | 155.38 |\n| OPT-2.7B | Act. Obj. | 0.55 | 68.07 |\n| OPT-2.7B | **Ours Multi-Obj.** | **0.01** | **27.15** |\n| LLaMA-2-7B | Wei. Obj. | 0.17 | 12.22 |\n| LLaMA-2-7B | Act. Obj. | 0.37 | 5.24 |\n| LLaMA-2-7B | **Ours Multi-Obj.** | **0.02** | **3.50** |\n\nThese tests demonstrate that our multi-objective optimization achieves lower quantization errors and better model accuracy compared to previous methods focused solely on weight-only or activation-only objectives.\n\n---\n**W2: The use of mixed precision for layers with different sensitivities is a good idea, but it is designed for one LLM model. How effective is the automated bit allocation? And how applicable is this approach to different LLM models?**\n\nThank you for your constructive questions! We agree that the effectiveness and applicability of our automated bit allocation need to be validated across different LLM models.\n\nTo address your questions, we evaluated our mixed bit allocation strategy and compared Ours (Mixed) with Ours (Lat.). The results are shown in Table 6 of the attached PDF in our global response. Ours (Mixed) further reduces perplexity by an average of 96.86, 3.23, and 2.63 for OPT, LLaMA, and Gemma models, respectively, under comparable or even less latency. This set of experiments further validates the applicability of our automated bit allocation strategy to different LLMs.\n\nIn addition, we want to clarify that, for each model, we search for the optimal bit allocation with negligible overhead (e.g., 1%~10% of the reparameterization time). For example, it takes 0.5 seconds for searching versus 72 seconds for reparameterizing OPT-125M with a single bit configuration, and 1 minute for searching versus 13 minutes for reparameterizing OPT-13B with a single bit configuration. This is achieved by leveraging the proposed proxy criteria (as shown in Eq. 3 of the submitted manuscript), instead of searching according to the reparameterization errors, which is time-consuming and requires running models at each bit. Using the proxy criteria, the bit allocation candidate rankings are highly correlated with the rankings obtained using actual reparameterization errors, with a Kendall $\\tau$ of 0.910/0.905/0.915 for OPT-125M/1.3B/13B and 0.931/0.929/0.897 for LLaMA-7B/13B/8B, respectively.\n\nIn summary, our proposed automated bit allocation strategy is effective and applicable to different LLMs with negligible overhead using the proposed proxy criteria.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Dear ACs and Reviewers,**

First, we deeply appreciate the time and effort you have devoted to providing reviews for our paper, particularly given the substantial scale of a conference like NeurIPS. Your efforts are truly valued.

We are immensely grateful for the positive feedback our paper has received. The accolades, highlighting its innovative approach, significant contributions to the field of LLM optimization, well-executed and fast GPU implementation, absence of data-intensive steps like retraining or finetuning, reduced logic area and power, and a simple yet effective method, along with extensive and thorough experiments, are all deeply gratifying. It is particularly encouraging that these aspects have garnered such appreciation from the reviewers.

In addition to the aforementioned commendations, we have also received requests for additional experiments and further clarifications from reviewers. In response, we have conducted the requested experiments and provided detailed clarifications to the questions raised, as summarized below.

---
**To summarize, the following experiments have been supplied:**

* **(1)** Lack the numbers for an FP16 baseline:
  * We have responded to Reviewer 3fe7’s W1-(8) and included the results in Table 4 of the attached PDF.
* **(2)** The impact of batch sizes on throughput:
  * We have responded to Reviewer 7Vce’s W1 & L1 and included the results in Figure 1 of the attached PDF.
* **(3)** Comparison with FlexRound and OmniQuant:
  * We have responded to Reviewer 7Vce’s W3 and included the results in Tables 2 and 3 of the attached PDF.
* **(4)** Show scores for MMLU and conduct A/B tests using AplacaEval with GPT-4:
  * We have responded to Reviewer 7Vce’s W4 and included the results in Table 4 and Figure 2 of the attached PDF.
* **(5)** Lack of results for a 4-bit setup:
  * We have responded to Reviewer 7Vce’s W5 and included the results in Table 1 of the attached PDF.
* **(6)** More test cases for the multi-objective optimization:
  * We have responded to Reviewer 4dkb’s W1 and included the results in Table 5 of the attached PDF.
* **(7)** More evaluation of the effectiveness and applicability of the automated bit allocation:
  * We have responded to Reviewer 4dkb’s W2 and included the results in Table 6 of the attached PDF.

---
**To summarize, the following questions have been clarified:**

* **(1)** The technique's applicability beyond LLMs:
  * We clarify this in our response to Reviewer 3fe7’s W1-(1).
* **(2)** Presentation details and clarity improvements:
  * We clarify those in our response to Reviewer 3fe7’s W1-(2-6), (9-13).
* **(3)** Why reduce from 4 bits to 3 bits or even 2 bits:
  * We clarify this in our response to Reviewer 3fe7’s W1-(7) and Reviewer 7Vce’s W5 and L2.
* **(4)** Figure 1 could benefit from improved explanations:
  * We clarify this in our response to Reviewer 3fe7’s Q1.
* **(5)** Whether using Eyeriss or DNN-chip Predictor:
  * We clarify this in our response to Reviewer 3fe7’s Q2.
* **(6)** Why consider the batch size of one:
  * We clarify this in our response to Reviewer 7Vce’s W1 and L1.
* **(7)** New ASIC design and simulation instead of GPU acceleration:
  * We clarify that our results are real-measured GPU speedups in our response to Reviewer 7Vce’s W2.
* **(8)** Plan to release the code: 
  * We promised this in the abstract of our submitted manuscript and here also clarify this in our response to Reviewer 7Vce’s L3.

---
Regarding Reviewer 7Vce’s questions about batch sizes, hardware simulation or real-measured GPU latency, downstream task performance, and comparison with FlexRound and OmniQuant, we've provided clarifications and results within the length limitations. We are open to providing further details in case any points still need to be clarified. As committed in the abstract, we will release both the codebase and pre-trained models, enabling others to replicate our results effectively.

We would greatly appreciate it if you could review our rebuttal responses. We hope that the new experiments and clarifications address your concerns. We are always willing to engage in further discussion, so please let us know if our responses do not fully resolve your concerns, and we will be happy to provide additional clarifications. Thank you!

**Best regards,**

**Paper 12025 Authors**

Meta-review: The paper introduces a novel method that replaces complex matrix multiplications with shift-and-add operations as a post-training optimization step in neural network models. This approach improves computational efficiency, particularly in terms of energy and area reductions, without the need for re-training, unlike state-of-the-art (SOTA) methods. Despite utilizing bit-level operations, the optimized models maintain strong performance on GPUs. Extensive experiments validate the effectiveness of the method. After a thorough discussion, all reviewers recommend accepting the paper. The AC agrees and suggests acceptance, with the authors expected to incorporate all revisions in the final version.
Decision: Accept (poster)