SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
Authors: Rasoul Shafipour
David Harrison
Maxwell Horton
JEFFREY MARKER
Houman Bedayat
Sachin Mehta
Mohammad Rastegari
Mahyar Najibi
Saman Naderiparizi
Venue: ICLR 2025 Poster (2025)
Keywords: Model Compression
Large Language Models
Post-Training Quantization
Primary area: foundation or frontier models, including LLMs
TL;DR: SeedLM is a novel post-training compression technique that uses pseudo-random generators to compress model weights, achieving significant memory savings and inference speed improvements with minimal accuracy loss in large language models.

Abstract: Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of a pseudo-random generator to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama3 70B, which is particularly challenging, show zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-art methods, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases, approaches a 4x speed-up over an FP16 Llama 2/3 baseline.

Published as a conference paper at ICLR 2025
SEEDLM: COMPRESSING LLM WEIGHTS INTO SEEDS OF
GENERATORS
Rasoul Shafipour', David Harrison', Maxwell Horton', Jeff Marker", Houman Bedayat',
Sachin Mehta Mohammad Rastegari", Mahyar Najibi', Saman Naderiparizi
AApple University of Washington Meta Al
ABSTRACT
Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due their high run- time cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of pseudo-random generator to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into Linear Feedback Shift Register (LFSR) during inference to efficiently gen- erate a random matrix. This matrix is then linearly combined with compressed coefficients reconstruct the weight block. M. reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory- bound tasks by trading compute for fewer memory accesses. Unlike state-of-the ohe art methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama3 70B, which is particularly challenging. show zero-shot accuracy retention at 4- and 3-bit compression to be on par with or better than state-of-the-ar methods, while maintaining performance comparable to FP16 baselines. Additionally, FPGA- tests demonstrate that 4-bit SeedLM, as model size increases, approaches a 4x speed-up over an FP16 Llama 2/3 baseline.
INTRODUCTION
Large Language Models (LI.Ms) have demonstrated impressive performance across numerous benchmarks (Achiam et al.] 2023 Touvron et al.] (003).. However, the practical deployment of these models often encounters limitations due to substantial memory transfer requirements. This issue is especially pronounced during autoregressive generation, which is primarily memory-bound and takes the majority of the inference time (Lee et al. 2024). In contrast, operations like 8-bit inte- ger multiplication performed at 45mm 0.9V are demonstrated to be over 200x more energy-efficte than reading the same 8 bits from DRAM (Horowitz) 2014). In this paper, we explore the fol- lowing question: Can we trade increased compute for reasonable reduction in memory accesses? A positive answer here not only transforms energy-intensive memory access operations into more energy-efficieee compute operations but also alleviates the memory bandwidth limitations that pose a significant bottleneck during LLM inference.
Post-training weight compression is effective method to reduce the size of pretrained LLMs, mak- ing them suitable for on-device execution or reducing power consumption through fewer memory reads. Current state-of-the techniques for compressing weights typically require calibration data and involve meticulously adjusting the weights to ensure that the learned knowledge is retained.
We introduce MeedI.M, simple yet effective compression technique which can compress weights to 3-4 bits with minimal accuracy loss. SeedLM is an innovative method for compressing the weights of LLMs by projecting weight blocks into pseudo-random projection basis sets. By finding the optimal seeds to generate these pseudo-random projections per weight block, SeedLM ensures low compression error and consequently maintains the accuracy of the original model. Our approach
Corresponding author: fishafipourr@app com. All work conducted at Apple.

---

Published as a conference paper at ICLR 2025
Mcthod
ScddLM
AWQ
OuniQQann
100
Retained Accuracy (%) Relained Accuracy
98.3
95.0
92.2
99.7
98.5
99.2
92.00
97..9
80
78.0
60
41.3
43.9
42:4
29.5
92.0
96.1
99.6
93.4
70.2
72.0
80.L
26.7
Bit--
60
40
40.3
13.9
$1.8
3D0
ARC-Easy
ARC-Challnge
HeliaSwag
WinoGrande
BoolQ
Mese
Task
Figure I: Retained zero-shot accuracy across variety of tasks and compression methods, compared the standard Llama 70B model. The top row shows data for 4-bit compression, while the bottom row shows data for 3-bit compression. We compare the performance of SeedLM, AWQ, and Om- niQuant across the ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, and BoolQ tasks. While being completely data-free, SeedLM outperforms state-of-the-art weight quantization methods that rely on a calibration dataset.
only requires storing the seed and few projection coefficients instead of all the weight values to reconstruct high dimensional weight blocks. As result, SeedL. significantly reduces the memory footprint required for operating large-scale models during inference. To generate pseudo-random matrix blocks given a seed, we leverage Linear Feedback Shift Register (LFSR) hardware blocks that widely used in applications such cryptography, communication, error detection, etc (Gaitonde & Ramabadran 1988 Zeng et al. 2013 Xiang et al. (016). LFSRs can be efficiently implemented in the silicon with minimal energy and area footprint.
Figure shows the Retained Accuracy (%), which is the ratio of the compressed model's accuracy the full-precision FP16 model's accuracy, the Llama 3 70B model. As shown, SeedL.M retains approximately 97.9% .%% of zero-shot accuracy across various tasks in a data-free setting, using 4 bits per weight element (see Section 44 for more details). It also consistently outperforms state-of-the- art 3-bit and 4-bit compression techniques that rely on calibration data. To the best of our knowledge, this is the first time nearly identical accuracy has been achieved with 4-bit compression on LLMs without data, using deterministic offline algorithm.
The summary our contributions follows:
For the first time, we demonstrate how to leverage LFSR hardware blocks to trade increased compute with lower memory accesses.
We show the first instance of achieving nearly identical accuracy with 4-bit quantization without data, using a deterministic offline algorithm.
We demonstrate an effective solution to find the optimized seed LFSR modules while maximizing compression ratio.
We prototype SeedLM on an FPGA (Field-Pros SreeddProgrrammbbe Gate Array) and demonstrate its efficacy in reducing the inference latency with custom hardware.
RELATED WORK
Significant research has been conducted in model compression for LLMs, critical approach for reducing both the memory footprint and computational demands these models. this section, highlight some of the most relevant techniques from prior work.
2

---

Published as a conference paper at ICLR 2025
Compression With Random Basis: Recent works have demonstrated that neural networks can be decomposed into random number generator seeds and weight coefficients. In PRANC (Noora- linejad et al. 2022), full networks are compressed by orders of magnitude to improve storage and transmission efficiency. LoRA (Hu et al. 2021) compresses the weights by injecting trainable rank decomposition matrices into each layer of the network. NOLA (Koohpayeganí et al., 2023) builds upon LoRA by compressing the low-rank matrices through a linear combination of random basis vectors, further reducing memory and computational overhead.
Our work (SeedLM) is conceptually similar in that we compress networks using random basis. These other models rely on much larger basis ranks applied globally rather than per-block. As result, these methods need far more operations per parameter to preserve accuracy and are not computationally feasible at inference for L.MMs.
Data-Free Post- Training Compression: A few previous works have explored data-free post- training compression (Nagel et al., 2019: Horton et al.) 2020; Nunez et al.) [202)). Such works capable of producing a compressed model after training without the need for calibration data. They usually apply quantization or pruning techniques to obtain a smaller model. Similarly, SeedLM does not require any data for model compression. This is in contrast to most recent works on LLM compression, which require calibration data.
There are more computationally expensive methods for data-frec compression that involve gener- ating data from a teacher model and performing distillation (Lopes et al. 2017 Gou et al.) 2020).
These techniques applied to LLMs as demonstrated in Liu et al. (2023
Post-Training Compression with Calibration Data: An early example of post-training quanti- zation with calibration data is found in Nagel al. (2020). where activation statistics are used to decide whether to round quantized values up or down. The cost of post-training quantization is a small fraction of the total training cost.
Recently, calibration data has been leveraged for post-training compression in LLMs. AWQ (Lin al. 2024) rescales salient weights before compression using activation statistics. In QulP# (Tenn et al. 2024 and GPTQ (Frantar et al. (2022), Hessian analysis of calibration data helps make round- ing decisions during quantization. SpQR (Dettmers al., 2023) retains outliers during quantization to preserve accuracy. OmniQuant (Shao et al. 2023) employs weight clipping and other transfor- mations maintain accuracy. Additive Quantization (Egiazarian et al., (024) learns a codebook for performing additive quantization. In our study, we used AWQ and OmniQuani our main baselines because they avoid costly training while delivering strong results.
Training-Award Compression: Compressing the model during the training process has the dis- advantage of fixing the compression method and parameters beforehand, but usually offers better accuracy. An overview of quantization-awwal training is provided in Jacob et al. (2017): Nagel et al. (2022) : Xi et al. (2023). while pruning techniques are discussed in/Aiizdeeh- Vahid et al. ); Sun et al. (2023); Kusupati et al. )2 LeCun al. (1989). In Rouhani al. (2023), stable training and post-training quantization are demonstrated using prrdware-friendly 4-8 bit weights, activations, and gradients with minimal accuracy loss by utilizing micro-scaled data formats.
In this work, we focus on post-training weight compression with SeedLMM, and in the following sections, outline our methodology and present the results.
METHODOLOGY
In this section, we introduce SeedLM, our method for compressing the weights of LLMs by using seeds from a pseudo-random generator. Initially, each weight matrix is segmented into blocks of C contiguous elements. Representing each block as a vector w E R we approximate it as a linear combination of columns from matrix U CCP. This matrix U is constructed using pseudo- random generator given a seed specifically selected to generate a subspace that most effectively reconstructs w linearly.
3

---

Published as a conference paper at ICLR 2025
Figure illustrates this setup. Our primary goal is to find the optimal seed, and coefficient vector, E R., that minimize the reconstruction error between the original and the approxi- mated weights. For this recon- struction, only the seed and the coefficients are stored. In the fol- lowing subsection, we first out- line mechanism to efficiently generate U using K-bit seed in our Linear Feedback Shift Regis- ter (LFSR) framework. We will then discuss the methodologies used to determine and t.
Seed $
X
U(+)
W
Coefficients
Random Matrix
Weight Block
Figure 2: Compression of weights using pscudo-random gen- erated matrices.
3.1 LINEAR FEEDBACK SHIFT REGISTER (LFSR)
A Linear Feedback Shift Register (LFSR) is simple yet effective type of shift register, ideal for generating pseudo-random binary sequences. The primary advantages of LFSRs in hardware include cost-effectiveness and minimal resource consumption due to their straightforwat with basic flip-flops and XOR gates. This simplicity facilitates rapid and efficient sequence genera- tion, which is integral our compression technique.
implementation
LFSR operation can be characterized by its length (which determines the number of bits in its shift register) and its feedback polynomial. To generate next pseudo-random number in the sequence, each bit in the register is first shifted to the next position. Then, the new bit entering the register is calculated feedback polynomial. typically implemented by XOR operations. Mathematically, the new bit 2n+1 generated by the LFSR can be expressed
linear combination of certain bits of the current state specified by the
K Σ
In+
Inti-K4I mod 2,
where K 2 and co....., OKK are the binary coefficients that define the feedback polynomial, with each a, determining whether the bit æ, is selected not.
The state transition in the LFSR can described as follows: the current state is represented by the bits Fn, www...n...... - then after the shift, the state will be 2y Fn Tu.............................. .......... -n K+2- This transition reflects the shift of every bit to the right by one position, with the new bit an+1 entering at the leftmost position. Given its finite state nature, an LFSR will eventually enter repeating cycle, suggesting an asymptotic uniform distribution over this cycle. An LFSR can cycle through at most - states, excluding the all-zero state.
A key goal when designing an LFSR is to guarantee maximal-length sequence. This ensures that the LFSR will produce the longest possible sequence non-repeating states before repeating. Intuitively, this means the LFSR will cycle through every possible state (except the all-zero state), maximizing the number of distinct pseudo-random values generated. To achieve this maximal-lengte property, the feedback polynomial must be primitive over the Galois field GF(2). In simple terms, primitive polynomial ensures that LFSR explores all 2 states without prematurely entering repeating cycle.
For our experiments, this means a fixed set coefficients (a, : 0 S K-13 that hard- wired, ensuring maximal length if and only if it avoids the all-zero state. in which it stays in zero. summary of the algorithmic implementation can be found in Appendix AA.3] Refer to Section A.I for the indexed coefficients used for each K where a, equals one; all other coefficients For a comprehensive understanding of LFSRs and their properties, see (Bhattacharjee Das 2022).
zero.
To optimize the efficiency of generating random matrices through LFSR, for fixed length K and a set of coefficients {ay). we cache all the 2 -1 states that sequentially follow each other each state uniquely determined by its preceding state along with K and {a This cache allows

---

Published as a conference paper at ICLR 2025
us to extract arbitrarily sized random matrix from the sequence given random seed We can cycle through these states to generate matrices of any desired size. For an illustration, refer to Section A.2 With K = 16 and maximal length LFSR, all states will occupy approximately (2 1) x 2 Bytes R 130KB of memory, which is negligible. This setup ensures a highly efficient and scalable mechanism for generating the necessary random matrices for our compression technique. K is a hyper-parameter of our method which we will elaborate on in Section 3.4
3.2 WEIGHT COMPRESSION USING PSEUDO-RANDOM GENERATORS
Building on the foundations laid by the LFSR mechanisms, our methodology seeks to represent a block of data derived from a K-bit LFSR. During inference, and t. which require fewer bits when considering all the bits in block compared to the original weights, are retrieved from memory. This reduces the memory footprint and accelerates memory-bount generation tasks. The weights are reconstructed on-the-fly fly using and t, and these reconstructed weights activations and, ultimately, the outputs. As the output of the LFSR is in the range of [1, 2 - which unsigned integers, normalize them ensure they fall within the range of [-1, 1]. enhancing their expressiveness. More specifically, to construct U(s). we first generate a matrix (ss) using an LFSR seeded by illustrated in Figure 4 This matrix initially contains integers and undergoes the following normalization to ensure that its components lie between [-1,
RC using the decomposition U(s)t. Here, U(s) E gCxP is random matrix
then used to compute intermediate
U(s) 2k-1 (V(s) -200)22-1 2k-11)
where 1 E RCP represents matrix of all ones, and K is the LFSR bit length. To determine the optimal seed and coefficients, we solve the following optimization problem:
minimize 5,t
||w U())t subject to: 8 (1, 2 - 1}: and te E
(1)
where ||-|| denotes the Euclidean norm, and Q represents the of valid quantized values. The objective is to identify an optimal seed s" and coefficients t" that most effectively approximate w. This problem is NP-hard due to the discrete nature of the seed selection and the quantization of coefficients since it involves searching through a combinatorially large set of possibilities. Next, we describe the design choices and hcuristics used to solve the problem.
The quantization scheme for the vector t plays critical role in balancing reconstruction accuracy with bit efficiency, adhering to our bit budget constraints. We represent each element of t as a 4- bit 2's complement integer, paired with shared 4-bit exponent. This configuration allows to capture a broad dynamic range of values within the interval [-8 x 2 8 7 x 27 The shared 4-bit exponent extends the dynamic range, enabling representation across several orders of magnitude. We have specifically chosen for the exponent to be power of two, which is hardware-friendly and can be efficiently implemented using simple shift operations in digital circuits; see (Darvish Rouhani let al. 2020).. By combining the 4-bit integer with the shared exponent, each quantized element is expressed as t = qi x 2", where qi is the 4-bit 2's complement integer and is the shared exponent. The shared exponent e is selected as:
C = max logg (Ita))
where |4:| is the absolute value of each element of t. After determining each t, is scaled by dividing it by 2", and then quantized 4-bit 2's complement integer to derive q..
3.3 APPROXIMATION APPROACH
Looking back the optimization problem in Eq. [ while the unconstrained case admits closed- form solution given by U(s) w, where U(s)² denotes the Moore-Penros, pseudo-inverse U(s). our discrete constraints convert it to an NP-hard optimization problem. Hence, to solve Eq. [ we employ an approximate heuristic approach that involves the following steps:
1. Generate N 2 random matrices (U(s1), U(s2).... U(sN)}, each of size C x P, with an LFSR of length K.
5

---

Published as a conference paper at ICLR 2025
2. For each matrix U(sy), project the vector onto the subspace spanned by U(sy):
= U(s)) Lw. w.
3. Quantize E, to obtain the vector i, using 4-bit integers and a 4-bit shared exponent ej-
4. Compute the reconstruction error for each pair (U(s)), (88)), t2) as follows:
y= w|w U st,
5. Select the pair (s*, that minimizes the reconstruction error:
(s" t²* = arg min Cj-
(2)
4j.t
Our heuristic algorithm leverages randomness to explore multiple subspaces and selects the one that best approximates w under the given constraints, In summary, we apply the above heuristics across all weight blocks in parallel to find the seeds and coefficients that minimize the reconstruction error. To enhance computational efficiency, we precompute and cache the pseudo-inverse matrices for all seeds and perform steps 2-5 in parallel across all blocks. The complete algorithm is provided in Appendix A.5
3.4 DESIGN SPACE EXPLORATION
The minimum reconstruction error obtained from Eq. depends on the block size C, the latent dimension P. and the LFSR length K. Here, we explore how we select the optimal configuration for M-bit compression. First, let's examine the total number of bits required to store a SeedLM block of C' elements, which consists of the following:
bits to index the selected seed 8* to generate matrix U(s*) among 2* candidates.
4 bits to store the shared exponent e.
4P bits to store the quantized vector t" (P elements each requiring 4 bits).
So, the effective bit per element is function of hyper- parameers K, C, and P. In particular, for an M-bit compression, we have the bit budget per element as
K 4 4P
M
(3)
C
To determine the optimal configuration for given budget per element, M, we evaluate the reconstruction accuracy of method in the search space. Specifically, explore how a standard normal Gaussian vector w can approximated using any combination of valid hyperparameters given the bit budget M. While assuming a Gaussian distribution may have its limitations, it has proven effective within our design space and aligns well with real-world benchmarks. Our objective find appropriate values for block size C, latent dimension and LFSR length K, such that the reconstruction error is minimized when the optimal seed is selected. More specifically, let PCP. and P, denote the solutions obtained from Eq. 2 For an M-bit compression, we solve the following optimization problem:
E mial) = min
E[[lw - U.C.UU
C.P.K
(4)
subject to: MC=K+44
and C,K,PEZ*,
where 2+ represents the set of positive integers. Since the optimization problem equation not analytically tractable, numerically solved it by conducting grid search over C. P. and K constrained to positive integers. Understanding the trade-off's among C, P, and K is important for optimizing the approximation within given bit budget. Each of these parameters influences the overall performance and contributes to minimizing the reconstruction error.
One critical trade-off is between the LFSR length K and the latent dimension P. Increasing K reduces the expected minimum error E[emin] by providing more opportunities to find a better pro- jection. However, this comes at cost: K increases, the number of required bits grows, reducing

---

Published as a conference paper at ICLR 2025
the available bit budget for P. The objective is to find optimal K that effectively lowers [[cmin] without overly constraining P. as that could lead to a significant increase in the overall error E <min Similarly, increasing P helps capture more of the energy of the vector w, thus reducing E[émin) However, this also requires more bits, which may limit the value of K. The key is to strike a balance where enough energy is captured without overly sacrificing the exploration of better projections through K. Finally, increasing C' expands the total bit budget, allowing for larger values of both P and K. However, this also increases the potential for higher error, as expanding the space may dilute the precision of projections.
Overall, C, P, and K are interdependent, and optimizing these parameters requires a careful numer- ical exploration of different combinations. Based on this analysis, we selected the configurations shown in Table for =33 and M 4, which are used the experiments reported next.
Table I: Selected configurations of C, P. and K for M 3 and M- = 4.
bits per element M
Block size C
Latent Dimension P
LFSR Seed Bits K
3
12
16
4
8
16
4 EXPERIMENTS
In this section, we apply the heuristics across all weight blocks of pretrained LLMs in parallel to find the seeds and coefficients, described in Eq. [ that minimize the reconstruction error. Using the configurations from Table [ we evaluate our compression methods in terms of accuracy and performance. Our experiments focus on Llama 2 and Llama 3 models (Touvron al. (202)), and unlike other methods, SeedLM does not require fine-tuning or calibration data while still achieving competitive results. We assess model quality using perplexity and accuracy, followed by perfor- mance analysis through FPGA- baseed matrix multiplication with low-level LFSR generation. This highlights the cost and performance benefits of SeedLM, especially in hardware-constrain envi- ronments.
4.1 ACCURACY RESULTS
To evaluate the quality of SeedLM, measure perplexity the WikiText-2 dataset (Merity al., 2016) and assess accuracy across various zero-shot tasks using LM Evaluation Harness (Gao et al. 2021 We compare our method against established compression techniques such as AWQ (Lin) et al. 2024) and OmniQuant (Shao et al., 2023), using the official GitHub repositories for each baseline as of September 2024. Additionally, we compare with GPTQ (Frantar al.. 2022) 4- bits, summarized in Appendix A.6] A key strength of SeedLM is that it can operate entirely data-free, in contrast to other methods that require calibration data to achieve comparable results. For the baseline methods, we use the default calibration sets from their official repositories. Our experiments involve Llama models (7B, 13B, 70B) and Llama 3 models (8B, 70B), tested with use 4-bit integers with
3-bit and 4-bit representations. In the case of AWQ and OmniQuant, channel-we wiieel-wieeeae scaling to avoid significantly increasing the bits per clement beyond the allocated 3 or 4 bits (since group size of 128 in these methods adds roughly 0.25 extra bits per parameter). For OmniQuant, we ensure a fair comparison with SeedLM and AWQ by not performing fine-tuning on the quantized models. However, unlike SeedLM, all of them still require per-layer calibration, which relies on calibration data and activations, whereas SeedLM achieves its results without any data dependency.
For all compression methods, use LM Evaluation Harness v0.4.3 and the following task versions: are- challenge=1.00 .0, arc-easy=1.0,, hellaswag=1.0, winogrande=1... boolg=2.0.
7

---

Published as a conference paper at ICLR 2025
To evaluate language model performance, we measure per- plexity on WikiText-2 using 166 test windows of 2048 tokens each. The results, shown in Table 2 illustrate a trade-off between compression level and model quality. In some cases, aggressively compressed large models even underperform smaller models with mild compres- sion. SeedLM consistently outperforms state-of-thes art compression particularly at higher compres- sion levels. Notably, Seedl M achieves these results without the need for any calibration data.
Table 2: WikiText-2 perplexity results for 3- and 4-bit represen-
tations of Llama 2 and Llama models with sequence length of 2048. The notation x-yB refers to the Llama x model with yB parameters (e.g., 2-7B means Llama with billion parameters). Perplexity values above 100 are shown as inf. The best perplexity values are highlighted.
Method
Bits
2-7B
2-13B
2-70B
3-8B
3-70B
Baseline
16
5.5
4.9
3.3
6.1
2.9
SeedLM
5.7
5.1
3.5
7.0
3.8
OmniQuant
4
6.1
5.2
3.7
inf
inf
AWQ
5.8
5.1
3.5
7.1
4.7
techniques,
SeedLM OmniQuant
6.6
5.8
4.0
10.1
5.7
inf
10.7
7.5
inf
inf
AWQ
3
15.6
6.5
4.4
11.8
11.6
Next, we show zero-shot accuracy across various tasks. As seen in Table B SeedLM performs on par with better than of-te-of-the-art methods at the same bit rates. This underscores SeedLM's ability to maintain competitive accuracy without relying much more sensitive to compression than Llama 2, likely due its more advanced architecture and significantly larger training dataset. Llama 3 was trained on 15 trillion tokens, around seven times more than Llama 2's trillion tokens, enabling it to capture more detailed language patterns and nuances. This increased complexity and sensitivity to nuance make it less compressible without noticeable drop in performance. However, as shown in Figure I SeedLM still significantly outper- forms other ofte--of-the-art methods on Llama in terms of Retained Accuracy (%), i.c. ratio of the compressed model's ss accuracy to the full-precision FP16 model's 's accuracy. This demonstrates that SeedLM is not only effective for such complex model but is also robust across various tasks by maintaining high accuracy.
calibration data. Note that Llama 3 is
4.2 PERFORMANCE ANALYSTS
In this section, we shift focus from accuracy performance analysis hardware. Specifically, we explore how SeedLM can be efficiently implemented on an FPGA. FPGAs are ideal for this task because they allow for highly parallelized computations and be reconfigured to handle specific workloads, making them well-suited for running compressed models with lower bit rates that are well supported by conventional GPUs. With FPGA, the LFSR generation be supported at low-level in hardware rather than relying on relatively expensive software implementations.
To evaluate SeedLM on an FPGA, we benchmark matrix-vector multiplication ation in most large language models - both with and without SeedLM's 4-bit parametrization. Figure 3 shows the RTL design block diagram, with the target device being an AMD Virtex7 FPGA XC7V585T- (202T).
core oper-
The implementation utilizes 128 DSP48 slice multipliers in parallel, calculating 128 elements of the activation vector simultaneously. The DDR response interface has maximum bandwidth of 64 bytes per 200 MHz clock cycle, and the data path designed to compute at the maximum DDR response throughput. When SeedLM compression is bypassed, the LFSR weight decompression is also skipped.
In the reference design, FP16 weights are read directly from DRAM, bypassing decompression. Due to the DDR interface's 's 64-byte bus width, a maximum of 32 weight values can be read per cycle, limiting the utilization to only 32 of the 128 MACs. With SeedLM's 4-bit compression, however, 128 weight values can be read per cycle, resulting in a theoretical 4x performance improvement in memory access.
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison across different models and zero-shot tasks for 4-bit and 3-bit
configurations.
Model
Method
Bits
ARC-Easy
ARC-Challenge
HellaSwag
WinoGrande
BoolQ
Mean
Baseline
74.58
46.33
75.98
69.06
77.74
68.74
SocdLM
73.23
44.54
74.45
68.43
77.19
67.57
AWQ
70.58
43.94
74.96
68.75
78.29
67.30
Llama 7B
OmniQuant
70.71
43.52
74.20
68.27
73.64
66.07
SeedLM
69.87
41.21
70.72
66.30
74.28
64.48
AWQ
53.37
33.62
56.66
61.09
57.58
52.46
OmniQuant
35.69
25.77
35.48
52.88
42.48
38.46
Baseline
16
77.44
48.98
79.38
72.22
80.55
71.71
SeedLM
4
76.98
49.83
78.54
72.77
79.20
71.46
AWQ
77.44
49.32
78.57
71.90
78.47
71.14
Llama 13B
OmniQuant
76.18
47.95
78.10
72.14
81.77
71.23
SeedTM
72.85
45.39
74.50
71135
78.81
68.58
AWQ
70.58
45.14
72.72
64.96
72.45
65.17
OmniQuant
55.85
34.47
59.54
53.04
63.39
53.26
Baseline
16
80.98
57.25
83.81
77.98
83.70
76.74
SocdLM
87.14
56.40
82.97
76.72
82.26
75.90
AWQ
80.98
56.66
83.24
77.19
83.27
76.27
Llama 2 70B
OmniQuant
79.59
55.97
82.67
76.80
83.43
75.69
SeedLM
79.00
53.64
80.31
76.80
79.02
73.83
AWQ
80.26
55.80
80.50
73.01
80.00
73.91
OmniQuant
63.59
39.51
68.24
62.04
65.23
59.72
Baseline
16
76.81
52.73
76.97
72.93
81.87
72.26
SeedLM
76.52
49.74
76.61
72.93
80.76
71.31
AWQ
74.49
51.54
78.03
73.09
80.40
71.51
Llama 8B
OmniQuant
73.95
47.78
73.42
69.69
71.99
67.37
SeedTM
67.21
41.55
68.34
69.22
67.61
62.79
AWQ
64.90
40.19
68.40
65.04
74.62
62.63
OmniQuant
30.26
22.53
28.96
49.33
48.47
35.91
Baseline
85.23
64.33
84.07
77.66
86.27
79.51
SecdLM
83.80
$9.30
83.84
77.74
85.60
78.06
AWQ
80.98
57.94
82.84
60.54
79.39
72.34
Llama
OmniQuant
25.13
26.54
26.36
51.38
37.83
33.45
70B
SeedCM
78.45
52.22
80.77
71.33
84,39
74.68
AWQ
65.87
45.14
70.76
55.88
69.08
61.35
OmniQuant
25.21
26.15
49.64
37.83
32.95
Activation SRAM
DDR
Xilixx MIG Controller
Multiply Accumulate
FixedPoint To FP16
128
DDR
LSFR Weight Decompress
Figure Block diagram of the RTL design.
We the reference implementation without compression (FP16 for all data) baseline assess both performance and resource utilization on the FPGA. Table g provides a detailed comparison of the FPGA resource usage, including LUTss, FFs, BlockRAM, and DSP counts.

---

Published as a conference paper at ICLR 2025
LUT FPGA Fabric Lookup Table, used to create combinational logic.
FF FPGA Fabric Register (Flip-Flop).
BRAM FPGA Block RAM (SRAM), each BlockRAM is 36Kb.
DSPs FPGA DSP48 resources with 18-bit by 18-bit multiplier and accumulator.
In the reference design, only MACs are used due to the input bandwidth limitation of 64 bytes per cycle. The SeedL.M design, utilizing 128 MACs per cycle, results in approximately 4x increase MAC Block resources, aligning with the expected performance improvement.
Table 4: FPGA resource utilization comparison
Total
MAC Block
LSFR Decompress
PreMAC Fix2Float
Reference
SeedLM
Reference
SeedLM
Reference
SeedLM
Reference
SeedLM
LUTs
20800
120105
9902
42174
67034
0
12292
FFs
10164
73666
3118
12594
45407
0
13448
BRAMs
10.5
154.5
144
0
0
DSPs
32
128
32
128
0
0 0
Table reveals that SeedLM increases the total LUT count to 67K and register count to 45.4K, with the fixed-point to FP16 conversion accounting for 12.3K LUTs and 13.4K registers. The design includes 128 fully pipelined fixed-to-FP16 converters to sustain the DDR response data rate. We can note that performing all the compute in fixed-point math could climinate the need for fixed- point to FP16 conversion, but this optimization is outside the scope of this paper. Table shows the number of cycles required to complete matrix operations of various sizes, measured from the first DDR read to the final write to the activation SRAM. The SeedLM design achieves 4x throughput compared to the reference design. As matrix size increases, the startup costs become less significant, leading the speedup approach the theoretical 4x gain. In summary, SeedLM achieves near iso- accuracy at 4-bit compared to FP16. while offering close to 4x speedup for memory- cecnory-bound tasks such generation in LLMs with billions of parameters and beyond.
Table 5: Performance comparison for different matrix sizes
512 x 512
1024
1024
2048 2048
Reference
8593
34201
136559
ScedLM
2341
8723
34331
Speed Up
3.67
3.92
3.98
5 CONCLUDING REMARKS
In this paper, we presented MeedL.M, a post-training compression method that uses pseudo-random generators to efficiently encode and compress model weights. SeedLM offers a data-free approach, avoiding the need for calibration data while retaining competitive accuracy, achieving up to around 98% zero-shot accuracy 3- and 4-bit quantization levels. We demonstrated the method's per- formance on both Llama 2 and Llama 3 models, showing that it performs comparably to existing state-of-the-as -frtthe-rrt techniques. Furthermore, our FPGA implementation highlights SeedLM's potential for improved computational efficiency in hardware-consstranee environments.
SeedL.M primarily focused on 3-bit and 4-bit quantization, applying the same design choices across all blocks for these levels. However, we believe that pushing below the 3-bit level might require dif- ferent configurations tailored to each block to manage compression noise effectively or leveraging data for additional fine-tuning enhance results. We leave these directions for future work. Further- more, as hardware continues to evolve, SeedLM's applicability could expand, with its LFSR-based design offering low-area, low-energy, nightthroughput benefits. This work highlights the practicality of such techniques, aiming to inspire hardware innovations that support broader device compatibil- ity, including GPUs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We are grateful Dmitry Belenko, Karen Khatamifard, Qingqing Cao, Seyed Mohsen Moosavi Dezfooli, Arsalan Farooq, Moin Nabi, and Devi Krishna for their invaluable feedback and support.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.08777 2023.
Keivan Alizadeh- Vahid,, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. Llm in flash: Efficient large language model inference with limited memory. arXiv preprint (txXxx:33/2.1/// 2023.
Kamalika Bhattacharjee and Sukanta Das. A search for good pseudo-random number generators: Survey and empirical studies. Computer Science Review, 45:100471,,022
Bita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu, Jeremy Fowers, Kalin Ovtcharov, Anna Vinogradsky, Sarah Massengill, Lita Yang, Ray Bittner, et al. Pushing the limits of narrow pre- cision inferencing at cloud scale with microsoft floating point. Advances in neural information 2020.
processing systems,
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint 0rXiv:2336603077. 2023.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06115 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq:
Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. /7323, 2022.
Sunil S Gaitonde and Tenkasi V Ramabadran. A tutorial on ere computations. IEEE Micro, 8(4): 62-75, 1988.
Leo Gao. Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. O. Sept, 10:8-9, 2021.
Jaanping Gou, B. Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789 1819, 2020.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). In 2014 IEEE international solid-state circuits conference digest of technical papers' (ISSCC), pp. 10-14. IEEE, 2014.
Maxwell Horton, Yanzi Jin, Ali Farhadi, and Mohammad Rastegari. Layer-wise data-free enn com- pression. 2022 26th International Conference on Pattern Recognition (ICPR), pp. 2019-2026, 2020.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient amteger-rithmeeic-onnl inference. 2018 IEEE/VV Conference on Computer Vision and
Pattern Recognition, pp. 2704-2713, 2713, 2017.
Soroush Abbasi Koohpayegani, K. L. Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash. Nola: Compressing lora using linear combination of random basis. In International Conference Learning Representations, 2023.

---

Published as a conference paper at ICLR 2025
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateck Jain, Sham M. Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, 2020.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. Neural Information Processing Systems, 1989.
Hyungdeok Lee, Gubyun Kim, Dayeon Yun, Ilkon Kim, Yongkee Kwon, and Euicheol Lim. Cost- effective Ilm accelerator using processing in memory technology. In 2024 JEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), pp. 1-2. IEEE, 2024.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Zechun Liu, Barlas Oğuz, Changsheng Zhao, Emnie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free free quantization aware training for large language models. arXiv preprint arXiv:2305. /7888, 2023.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: arXiv:1609.07843 2016.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1325-1334 2019.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. arXiv preprint arXiv:2004. 10568, 2020.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming oscillations in quantization-aware training. In International Conference on Machine Learning, pp. 16318-16330. PMLR, 2022.
Parsa Nooralinejad, Ali Abbasi, Soheil Kolouri, and Hamed Pirsiavash. Pranc: Pseudo random networks for compacting deep models. arXiv preprint arXiv:2206.0844 2022.
Elvis Nunez, Maxwell Horton, Anurag Ranjan, Ali, and Mohammad Rastegari. Les: Learning compressible subspaces for efficient, adaptive, real-time network compression at inference time. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3807-3816, 2023.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310. 10537, 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint 0rXiv:2308.13137 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11699 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402. 04396. 2024.
12

---

Published as a conference paper at ICLR 2025
XC7V585T-3. Virtex-7 T and XT FPGAs Data Sheet. Sn-US/ds183 Virtex D Sheet 2021.
https://docs.c and ctt/v//../v//////
Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers.
Advances in Neural Information Processing Systems, 36:49 (6649946--4168,, 2023.
Dong Xiang. Xiaoqing Wen, and Laung-Terng Wang. Low-power scan-based built-in self-test based on weighted pseudorandom test pattern generation and reseeding. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 25(3):942-9)) 22322995355 2016.
Guang Zeng, Xiaodai Dong, and Jens Bomneaan. Reconfigurable feedback shift register based stream cipher for wireless sensor networks. IEEE Wireless Communications Letters, 2(5):559- 562, 2013.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 COEFFICIENTS USED IN LFSRS
The following table lists the indexed coefficients used for each K, where oj equals one, with all other coefficients being zero. These specific coefficients correspond to the Linear Feedback Shift Registers (LFSRs) for each register length K, with the coefficients indexed starting from 0, repre- senting the tap positions in the shift register. These coefficients are hard-wired into the hardware configuration of the LFSRs used in our experiments. These coefficients define the feedback polyno- mial for each LFSR, ensuring maximal-length cycles.
Table 6: Indexed j coefficients used LFSRs for each register length K, where aj = and all other coefficients zero.
Length of LFSR (k) Indices (j)
2
(0, 1)
3
(0, 1)
4
(0, 1)
5
(0, 2)
6
(0, 1)
7
(0, 1)
8
(0, 3,44)
9
(0,4)
10
(0,3)
11
(0,2)
12
(0, 1,2,8)
13
(0, 1,2,5)
14
(0, 1,2,12
15
10,
16
1,311))
17
(0,3)
18
(0,
19
(0, 1,2,5)
20
(0,3)
21
(0,2)
22
(0, 1)
23
(0,
24
(0, 1,2,7)
A.2 LFSR STATE SEQUENCE ILLUSTRATION
2 5
2
V(4)
1
2
3
Figure 4: Illustration of the state sequence for K = 3 LFSR with all possible states with the feedback polynomial defined Table The matrix V(4) is generated from these states. The seed state is highlighted with thick circle.
14

---

Published as a conference paper at ICLR 2025
A.3 LFSR SEQUENCE GENERATION
This section explains how pseudo-random sequences are generated for our weight compression method using LFSRs which efficiently produce pseudo-random binary sequences, enabling on-the- fly generation. A key function generates a sequence of states by taking inputs such as a seed, the number of bits (K) in the LFSR. the sequence length, and the positions of the taps. These taps, which determine which bits are XORed to produce the feedback bit, are chosen to ensure maximal length and are summarized in Table
Process Overview. Starting with an initial state (seed), the LFSR computes the next state iteratively by:
Shifting all bits in the current state to the right by one position. XORing the bits at the specified tap positions to calculate the feedback bit, which is shifted into the leftmost position. This process repeated for the desired sequence length, with the generated states stored in an array for downstream use.
The pscudocode for this process is provided in Algorithm |
Algorithm LFSR Sequence Generation
Require:
K: Number of bits the LFSR
seed: Initial state of the LFSR length: Length of the sequence to generate taps: List of tap positions affecting feedback (Extracted from Table 6 for SeedLM}
1: Set current_state + seed.
2: for 0 length
do 3: Initialize 6y + 0. {bş is the new feedback bit}
4: for each lap in taps do
5: Update by + by @ staae >> tap)&1).
6: end for
7: Update current-state +- (current_state > 1) (bx < (k 1)).
8: Assign results[i] + current.state.
9: end for
10: return results
15

---

Published as a conference paper at ICLR 2025
A.4 RECONSTRUCTION PROCESS IN SEEDLM
The reconstruction process of weight matrix in SeedLM transforms compressed data back into its approximate original form using combination of pseudo-random sequence generation and compressed coefficients. Given of seeds {5: jaum_biocks and coefficients [t: i=l E RR, Algorithm i performed:
jaumbblocks where
Algorithm 2 Reconstruction Process in SeedLM
Require:
K: Number of bits in the LFSR
{8, (=1
Luum_blocks, Set of seeds
[t; [=1
1num_blacks, Set of coefficients, where t; R
C: Block size
P: Latent dimension
1: Initialize length + 2
2: Generate all LFSR states from length, store as states.
3: Initialize blocks + []
4: for to num_blocks do
5: Compute starl 8; s,eeegghh {% denotes the remainder operation]
6: Extract a slice of stales starting at slart and containing C.P elements. the slice exceeds length, cycle through states.
7: Rearrange the slice into a matrix U(s;) E RCXP, following a row-wise order.
8: Compute block U(s:) t; (matrix-vector multiplication).
9: Append block to blocks.
end for
11: Rearrange blocks to reconstruct the final weight matrix output.
return output
A.5 SEEDLM ALGORITHM
Algorithm summarizes SeedLM's method, which op- erates in parallel across all weight blocks to identify the seeds and coefficients used reconstruct weights To en- hance computational efficiency, precompute and cache the pseudo-inverse matrices for all seeds and execute steps 2-5 concurrently across all blocks. Additionally, the inner loop be parallelized. For the selected values of C, P. and K used in Table [ the pseudo-inverses re- quire at most 6.3MB of memory, which is negligible.
Algorithm Seed and Coefficient Selection for a Weight Block
Require:
E RC (U
st null, f <- null
2: min_norm
3: for to 2 do
4: t (U()) [w). where q(-) quantizes its arguments to
the set Q
5:
norm ||w - U()) t|l
6:
if norm < min_norm then
7:
min norm
8:
+ t-t
9:
10: end if
end for
12: return 8*, and t*
16

---

Published as a conference paper at ICLR 2025
A.6 ABLATION STUDY ON 4-BIT COMPRESSION
Table 7: WikiText-2 perplexity results for 4-bit representations of Llama and Llama models with a sequence length of 2048. The notation x-yB refers to the Llama X model with yB parameters (e.g., 2-7B means Llama with 7 billion parameters). Perplexity values above 100 are shown inf. The best perplexity values are highlighted.
We extend Table 2 include the well-known GPTQ from (Frantar et al., 2022) for 4-bit weight compression. We com- pare the perplexity of various 4-bit weight compression tech- niques on WikiText-2 using 166 test windows, each con- taining 2048 tokens. As shown in the results, SeedLM outper- forms other methods while re- quiring calibration data fine-tuning.
Method
Bits
2-7B
2-13B
2-70B
3-8B
3-70B
Baseline
16
5.5
4.9
3.3
6.1
2.9
SeedLM
4
5.7
5.1
3.5
7.0
3.8
OmniQuant
6.1
3.7
inf
inf
AWQ
4
5.8
5.1
3.5
7.1
4.7
GPTQ
8.3
5.7
3.9
8.2
6.7
17


Reviews:
[{"review_id": "LxwVSFoxXf", "reviewer": "Reviewer_i822", "summary": "This is an interesting method of quantization, using pseudo-random generator to point to almost evenly distributed codebook items and fast adjustments.\nThe paper has high quality presentation with necessary formulas and diagrams\nThe paper has shortcomings in comparisons with the other methods, specifically it lacks finetuning upside analysis.\n\nThe paper may qualify for acceptance if the weaknesses are reasonably addressed in the review process.", "questions": "1) Please compare your method with SOTA methods which use finetuning after quantization.\n2) What is your method performance when quantizing into 1.5-2.5 bits per weight? This area is where most quantization research progress is happening, your method may be competitive there too.\n3) please address the OOM situation in Quip# L2-70 benchmarks (Table2). What is different in your setup vs the original one from https://arxiv.org/pdf/2307.13304? What have you tried to avoid OOM? \n4) Add performance timing benchmarks vs other methods, this solidifies your claim of speed advantage.\n5) How would speedup of the method change after post-quantization finetuning?\n6) Unlike some other submission, this one has no implementation code. I wonder if a private code repository could be provided for review purposes?\n7) Please provide comparison to AQ, SPQR, as these are strong and valid SOTA benchmarks", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Novel method to store and retrieve codes with pseudo-random number generator.\nHigh quality presentation with necessary formulas and diagrams\nAttention to implementation details in Performance Analysis section.", "weaknesses": "1) Comparison with other quantization methods is incomplete. Most striking shortcoming is lack of comparison with finetuned model which is what most current SOTA models use.\n\n2) The paper dismisses comparison with strong methods like AQ, SPQR in desire to \"avoid costly training\". Yet these are quite good benchmarks to compare with, theu=y have reported figures, and to larg share of practitioners the extra training time (hours actually) could be acceptable.\n\n3) some of the results in table 2 are not consistent with those published in the respective papers.\nfor instance, AWQ claims 3.41 perplexity  for L2-70, (https://arxiv.org/pdf/2306.00978, table 4) while you indicate 3.5 (Table 2)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iuPhJqHchE", "reviewer": "Reviewer_WtPj", "summary": "The paper introduces SeedLM, a novel post-training compression technique leveraging seeds of pseudo-random generators, specifically using Linear Feedback Shift Registers (LFSRs), to encode and compress weights in large language models (LLMs). SeedLM achieves memory efficiency by encoding model weights into compact seeds, reconstructing weights at runtime, and minimizing memory access during inference. Experimental results show its performance in zero-shot tasks, achieving competitive accuracy retention at 3- and 4-bit quantization levels, especially with the LLaMA 3 model, and demonstrating improved latency and throughput in FPGA implementations.", "questions": "1- **Comparison with GPTQ** : Why was GPTQ omitted from the comparisons? Can you clarify its potential impact on results, given that it’s a standard benchmark in LLM compression?\n\n2- **Compression Limitations for LLaMA 3**: Many methods struggle to compress LLaMA 3 effectively. Do the authors have insights on why this model, in particular, is challenging to compress?\n\n3- **Full Precision Storage Impact**: Have the authors tested SeedLM with full precision storage during inference? If so, could this be the cause of out-of-memory issues observed with some baseline methods?\n\n4- **GPU Implementation Challenges**: Could the authors comment on the challenges of implementing SeedLM on GPUs, specifically the support for LFSR in recent GPUs and whether additional kernel launches could reduce performance for memory-bound applications on NVIDIA hardware?\n\n5- **Parameter Optimization Overhead**: What is the computational cost of finding optimal seeds and coefficients? Could the process be streamlined for faster deployment?\n\n6- **Sensitivity in Compressing LLaMA 3**: Many quantization techniques, appear to struggle with LLaMA 3, indicating potential limitations in compressing this model type. What features does this family of models have that makes it difficult for other methods to quantize, and why does SeedLM perform well in this case?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- **Innovative Use of Arbitrary Data Formats**: The adoption of arbitrary data formats with shared exponents is a commendable design choice, enhancing the flexibility of SeedLM's quantization approach.\n\n2- **Efficient FPGA Implementation**: Proposing an efficient FPGA implementation demonstrates the hardware viability of SeedLM and highlights potential real-world deployment in resource-constrained environments.\n\n3- **Data-Free Compression**: SeedLM operates without calibration data, which differentiates it from many other compression methods that rely on data for fine-tuning and accuracy adjustments.\n\n4- **Memory Efficiency**: By using seeds for reconstructing weights, SeedLM minimizes memory footprint, crucial for memory-bound applications like LLM inference, and shows considerable speedup, particularly on FPGAs.", "weaknesses": "1- **Absence of GPTQ Comparison**: The paper does not provide a comparison with GPTQ, a commonly used quantization baseline, which is a notable omission given GPTQ's relevance to LLM compression.\n\n2- **Inference Efficiency Assumptions**: While the paper mentions using the latest repositories, many of these codebases likely store compressed weights in full precision during inference, leading to potential memory inefficiencies.\n\n3- **GPU Implementation Challenges**: Although FPGA implementation is shown, the challenges of porting SeedLM to GPUs are unaddressed. Issues like increased kernel launches for memory-bound tasks and limited support for LFSRs in recent GPU hardware could impact performance.\n\n4- **Optimization Overhead for Parameter Selection**: The process of determining optimal parameters for each weight block, such as the seeds, coefficients, and latent dimensions, may introduce significant overhead.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEudpy1aW5", "reviewer": "Reviewer_GgWo", "summary": "The paper proposes SeedLM, a novel method for compressing large language model (LLM) weights by encoding them into seeds of pseudo-random generators.  SeedLM is a data-free, post-training compression method that achieves competitive accuracy at 3/4-bit compression context, as demonstrated on models like Llama 2 and Llama 3. The technique utilizes Linear Feedback Shift Registers (LFSRs) to generate pseudo-random matrices that reconstruct weight blocks during inference. Hardware implementation on an FPGA further supports the potential of SeedLM.", "questions": "Please refer to the weaknesses items.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Weight compression using pseudo-random generator seeds is a novel-sounding technique. It enables significant compression while maintaining high accuracy.\n2. Unlike many state-of-the-art compression methods, the proposed method does not require calibration data, reducing the need for correction data acquisition and potentially further reducing the quantization offset problem caused by the calibration data distribution.\n3. The authors validated the computational characteristics and efficiency of their proposed algorithm using FPGA, and the FPGA implementation verifies the SeedLM in some ideal hardware-constrained environments.", "weaknesses": "1. The author compared the AWQ, Omniquant, and QuIP# methods. However, Omniquant and QuIP# were primarily designed for ultra-low bit-width quantization compression, such as 2-bit, but the author only compared the performance of 3/4-bit and did not show the quantization results of 2-bit. In the field of LLM quantization, SOTA methods specifically designed for 4/3-bit, such as GPTQ[1], were not included in the comparison. This makes the results unconvincing.\n\n2. The author mentions in Section 4.1, lines 356-358, that to ensure a fair comparison with QuIP# and Omniquant, no fine-tuning was performed on them. This is a fair comparison for QuIP#, which combines codebook and fine-tuning of pre-trained parameters to improve performance. However, Omniquant does not fine-tune any pre-trained parameters, instead using block-wise gradient propagation to update the quantizer parameters, including the scaling factor and zero factor. By not using this technique in the comparison, the author is essentially not using the Omniquant method, but rather a basic statistical quantization. And, AWQ also uses calibration data to pre-compute the scaling parameters. This comparison is unfair and may cause confusion for readers. Additionally, the data in Table 2 is different from what is reported in the AWQ paper, and the author should provide an explanation for this discrepancy.\n\n3. AWQ can quickly determine the scaling of weights and perform quantization through calibration, and the compression time for a 7B model is only a few minutes. However, the LFSR technique proposed in the paper involves matrix decomposition and optimization approximation, and the efficiency of this compression process for extremely large-scale LLMs is lacking in discussion and comparison.\n\n4. The overall writing of the article is not clear in some details. In Eq (1), I can infer that the compression matrices for LFSR are **U** and **t**, but the details of how the input activation **X** is computed with the LFSR-compressed weights during the actual inference decoding stage are not discussed in detail. The article should provide more information on how the LFSR-compressed weights are used in the inference process and how they differ from other quantization methods. This would help to clarify the advantages and characteristics of the LFSR-based method.\n\n\n\n\n\n[1] GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)