ESPACE: Dimensionality Reduction of Activations for Model Compression
Authors: Charbel Sakr
Brucek Khailany
Venue: NeurIPS 2024 poster (2024)
Keywords: Activation tensor decomposition
model compression
matrix factorization
Primary area: optimization_for_deep_networks
TL;DR: We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations which enables retraining LLMs with no loss in expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix associativity.

Abstract: We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. Unlike prior works on weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of principal components. The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity. Theoretical results on the construction of projection matrices with optimal computational accuracy are provided. Experimentally, we find ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation, as low as a 0.18 perplexity increase on GPT3-22B. At lower compression rates of 20% to 40%, ESPACE drives GPT3 models to outperforming their baseline, by up to a 0.38 decrease in perplexity for GPT3-8B. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware. Comparison with related works on compressing Llama2-7B via matrix factorization shows that ESPACE is a first step in advancing the state-of-the-art in tensor decomposition compression of LLMs.

ESPACE: Dimensionality Reduction of Activations for Model Compression
Charbel Sakr NVIDIA Research csakronvidia.com com
Brucek Khailany NVIDIA Research
bzhallanyEnvidia.com
Abstract
We propose ESPACE!, an LLM compression technique based on dimensionality re- duction of activations. Unlike prior works weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of principal components. The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity. Theoretical results the construction of pro- jection matrices with optimal computational accuracy are provided. Experimentally, we find ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation, as low as a 0.18 perplexity increase on GPT3-22B. At lower compression rates of 20% 40%, ESPACE drives GPT3 models to outperforming their baseline, by up to a 0.38 decrease in perplexity for GPT3-8B. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware. Comparison with related works on compressing Llama2-7B via matrix factorization shows that ESPACE is first step in advancing the state-of ofite-of-the-arr in tensor decomposition compression of LLMs.
Introduction
Capabilities of large language models (LLMs) have recently soared in natural language understanding and generative power. It appreciated that there exists correlation between model size and achievable accuracy. Indeed, as LLMs consume trillions of tokens during their training, a large parameter volume is required capture intricate linguistic features [1]. This leads trade-off in LLMs: larger parameter counts improve accuracy but come with increased serving cost.
However, is also appreciated that the computational requirements of inference may be lower than those of training [2). To that end, numerous studies have investigated compression of LLMs to reduce inference cost. The most popular I.LM compression techniques are quantization [3] and pruning [4]. A less explored, but powerful technique is tensor decomposition, and in our work, we propose a novel, activation-centee
way decompose LLM tensors. Our proposal a static set of components optimizing fidelity. The projection reduces activation dimensionality and leads to weight compression at inference as a byproduct of matrix multiplication associativity.
project activations onto
1.1 Related work and motivation for activation-centric tensor decomposition
Recent research has proposed many quantization and pruning techniques for compressing L.Mss.
Examples of advances in LLM quantization include SmoothQuant [3], AWQ [5], and GPTQ [6]: while
notable LLM pruning works include SparseGPT [4], LLM--runer [7]. and ReL.U-based masking [8]. These methods are conceptually orthogonal to our proposal for activation projection which can be
We use the french pronunciation "espace", which means "space".
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

implemented in low precision or sparse formats. Nevertheless, compression fundamentally introduces noise, and an open problem study the impact of combining different methods, e.g. quantization and matrix factorization. This is beyond the scope of our paper, but good direction for future work.
Our work is also orthogonal non-compressive L.LM serving acceleration such continuous batching [9] or speculative decoding [10]. and attention optimizations such as PttenAttention [11], RadixAAtention [12], and FlashAttention [13]. Our study on matrix multiplication layers involving weights and activations, and hence is mutually exclusive to works improving cross multiplications of activation tensors in attention. In fact, all our experiments use FlashAttention.
Finally, we turn to tensor decomposition, also known as factorization. Thus far, compression for LLM inference using factorization has been focused weight decomposition. KnGPT [14] uses the Kronecker transform to pack a large matrix into two smaller ones. TSVD [15] performs iterative singular value decomposition (SVD) weight matrices to produce high rank ternary components. TensorGPT [16] and HEAT [17] compress weight matrices into a cascade product of small matrices using the tensor-train algorithm. SVD-LoRa [18] uses truncated SVD on weights and finetunes the model using LoRa [19]. The LoRa adapters are then merged to the main branch using bounds on the rank of sum of low rank matrices. ASVD [20] performs a truncated SVD on the weights after re-scaling them by a diagonal matrix and their inverse encapsulating activation statisties. This work realizes the importance of activation awareness but still uses weight- compression. SliceGPT [21] extracts principal components in normalization layers to guide the deletion of rows and columns in weight matrices. The compression achieved using a factoriza- tion made implicit via computational invariance. The statistical method employed by sliceGPT shares similarities with one of our results, but our problem formulation and solution are different.
cedvatiionawwarnnsss
LLM size vs accuracy
Factorization can also streamline LLM training and finetuning. For instance, LoRa [19] finetunes pre- trained models using residual low rank adapters which are then absorbed into the main branch. Similarly, GaLore [22] applies low rank approxima- tion to gradients in pack-propagation. These works do not modify inference parameter and operation count, and are hence orthogonal to ours. Our method could be applied in tandem with LoRa or Gal.ore, but this is be- yond the scope of this paper.
11
Wikitext-103 perplexity
GPT3-133
(lower is better)
Baseline GPT3
GP
ESPACE GPT3
228
Baseline- LiamaZ
ESPACE Llama2
Nemotron4-158
Baseline Nematron4
Uama2-78
ESPACE- Nemotron4
Uama2-138
10a
10
Model size (# of weights)
Figure 1: Perplexity? versus model size for GPT3 and Llama2 models and comparison to compressed models using ESPACE.
Since factorization increases the number of LLM tensors, achieving high compression rates requires the intermediate dimensions to be much smaller than that of original dot product. This breakage in computation usually necessitates retraining or finetuning stage to be healed. Unfortunately, this healing process is impeded because factorized LLMs have fewer learnable parameters which decreases expressivity [17, 22].
To our knowledge, no prior art has explored activation decomposition. Indeed, applying factor- ization solvers (e.g., SVD) dynamically incurs large inference runtime overheads. Yet, activation decomposition has several desired features which we examine in Section 2 and motivate via the following insights: (a) weights stay uncompressed during retraining, preventing the aforementioned loss of expressivity; (b) large activation tensors contain inherent redundancies making them prime candidates for compression; and (c) since most LLM computation comprises multiplications of weights and activations, decomposing the latter can lead to compressing the former at inference.
1.2 Contributions
We propose Eigen Static Principal Activation Component Estimation (ESPACE), an LLM compres- sion technique based on activation dimensionality reduction. Our contributions are as follows:
We project activation tensors onto a static and pre-calibrated orthonormal matrix. The projection lowers activation dimensionality but keeps weight matrices intact and fully available for training.

---

At inference, leveraging matrix multiplication associativity, model compression is achieved through pre-computation of the product of weight and projection matrices.
We theoretically derive optimal constructions for activation dimensionality reduction. Specifically, the projection matrix is calibrated in a manner to minimize activation decomposition mean squared error and forward propagated noise metrics. The solution is based on an eigenvalue decomposition of activation auto-correlation and yields multiple candidate projections for each activaton tensor.
We empirically study compression of models in the GPT3. Llama2, and Nemotron4 families evaluated on the Wikitext- 103 dataset for perplexity and the LM evaluation harness for downstream task accuracy. The amelioration in size versus perplexity² trade-offs is summarized in Figure
We show that ESPACE can compress LLMs by ~50% at the cost of a small accuracy loss, as low as 0.18 increase perplexity on GPT3-22B.
At lower compression rates, we find encouraging empirical evidence that ESPACE filters out noise and improves accuracy; e.g., ~00% compressed GPT3-8B lowers its baseline perplexity by 0.38. As an additional benefit of ESPACE, tangible latency reduction of 35%-to-45% obtained in matrix multiplication layers. This speed-up translates to up to ~ 40% faster prefill inference latency metricized by the time to first token and measured on existing hardware.
By comparison to existing works on tensor decomposition, we determine that ESPACE is a first step in pushing the frontier of compression rate versus accuracy retention (see Figure 4).
2
Dimensionality Reduction & Projections
In this section, we introduce notation for matrix multiplication, review weight decomposition, and introduce our proposed mechanism of dimensionality reduction via activation projections.
2.1
Matrix Multiplication and Weight Decomposition
We consider general matrix multiplications (GEMMs) described in Figure 2(a) of the form
Y = W
where W is weight matrix of size K x N and X is an input activation tensor of size K x M so that the output activation tensor Y is of size N x M. Typically, K and N are defined by network topology and layer instance, they are commonly referred to as embedding or hidden size. In contrast, M stacks multiple dimensions in an activation tensors to obtain a 2D matrix view. Generally, these are the sequence and batch dimensions.
Transformer- LLMs have four GEMM layers per block: query-key-value (QKV), projection (Proj). fully connected I (FC1), and fully connected 2 (FC2) layers. Our study is concerned with these layers, while cross activation multiplication and embedding layers are untouched. For notational simplicity, in this paper, we do not include layer indices in our equations.
The matrix W in (1) stores layer parameters and dictates the model's inference accuracy. To improve convergence of these parameters, an optimizer state is stored alongside weights during training and tracks historical values of gradients and updates [23, 24]. On the other hand, the activation tensor X depends on the input stimulus to the network, and is therefore generated on the fly.
Thus, at inference, weights are fixed but activations are dynamic. As consequence, prior work on tensor decomposition has focused on compressing frozen weight matrices. One way of doing so is breaking W into a low-rank approximation using some form of truncated SVD [20, 18], which is described in Figure 2(b). Specifically, (1) approximated as:
Y RJ UVX
(2)
where U and V are matrices of size N x and Lx K, respectively, with L being the factorization rank. For the decomposition procedure to be useful, two conditions need to be met: (a) I, << min( K, N) for compression, and (b) the approximation WT zs UV should be accurate. However, achieving both conditions simultancously may be challenging because a very low rank factorization usually leads to significant accuracy drop [22].
Perplexxity depends on tokenizer that comparisons across LLM families (GPT3 vs Llama2) are useful.
3

---

Weight Decomposition
General Multiplication (GEMM))
(+-E- truncated SVD)
Y UVX
parameters: expressivity
trainatie parameteen: Esprrssiiity X
W
Weight Matrix
Activation Tensor X
Output Tensor
Activation Tersor
Output Tensor
LxK
N X
K
N X.M.
K Input dimension, N- output dimension, M batch sequence
K input dimension, Ni output dimension, M batch sequence
L intermediate dimension
(a)
(b)
ESPACE: Activation Dimensionality Reduction Using Static Projections
Y
WT
(PTW) (PTX)
During Training:
+ trainable parameters: expressivity
During Inference: model size:
inference cost
Projection
Mabrix W!
X
X
Matrit
Stattc
x
Activation Tensor
Temior
(P"W>
Activation Temoo
Output Temar
L
K
Static
Pre-computed
K.
Kx
NxL
input
output dlimension,
sequence, intermediate dimension
(c)
Figure 2: Decompositions in GEMMs: (a) baseline multiplication of weight matrix and activation tensor, (b) truncated SVD the weight matrix, and (c) proposed approach of inserting static matrix to project activations. With ESPACE, weights are available for training, while inference compression is achieved per-computation (P"W).
As with other compression techniques, e.g., quantization and pruning, retraining of the compressed model may be employed to recover accuracy. However, the decomposition in (2) introduces two training-related hurdles: (a) the effective number of trainable parameters has decreased significantly which reduces model expressivity, and (b) the breakage of spatial weight structure prevents the retraining procedure from loading the original optimizer state. Retraining model without its optimizer state is known to introduce significant difficulty in convergence [17].
Activation Decomposition via Static Projection
Since weight decomposition poses the above hurdles, we motivate the need for activation-centric solution. some measure, activation compression may be more achievable due the large stack dimension M comprising batches and sequences. Statistically, the Central Limit Theorem claims that stacking data is likely to exhibit redundancies [25]. In the case of LLMs, such redundancies are further pronounced due to the likelihood of repeated tokens and information in natural language.
Therefore, activations should be prime candidates for tensor decomposition. Nevertheless, prior arts have not explored activation decomposition due to one fundamental limitation: unlike weights, activations are generated on the fly; meaning that tensors must be compressed during inference, potentially incurring large runtime penalties.
We propose to apply static dimensionality reduction the activation tensor X in (1). Concretely, our proposal is to project X onto a pre-computed static orthonormal matrix of size x L, where crucially L << K. Reconstructing X requires re-expansion using the transpose of the projection matrix, i.c., X se PP"X. X. While p P = II we note that PP" & I since L << K. Thus, the proposed activation transformation is noisy, and in Section 3, we derive optimal conditions on the calibration of P to minimize the effects of this noise.
Our proposal, described in Figure 2(c) is to approximate the GEMM in (1) using the following:
Y= W v W'PP"x W (PP"x) = (p"w)" (p"x)
where we used associativity of matrix multiplication to highlight key aspects of approach.
During training/finetuning: we view our GEMM as WT (PP"X) where has been replaced by its approximation. We emphasize that P is static and does not get updated during training.

---

Meanwhile, W is fully available for adaptation to the activation approximation. The availability of all learnable weights elides losing model expressivity. The structure of W is also unchanged and can be mapped to the baseline's optimizer state. Thus, the proposed approach does not suffer from the same limitations as weight decomposition techniques. We do note that introducing P induces small storage overhead at train time. However, when I, << K, and the order of computation is properly compiled, the number of operations per iteration is lower than baseline training; and, though not central to our contribution, we did observe up to 15% reduction in training iteration time for 50% compressed models.
During inference: we view our GEMM as (P"W)* (P"X) where the required matrices are P of size K x L and (P"W) of size N x L, which is pre-computed before deployment. Thus, per-layer parameter count required for inference has decreased from KN to L(K + N). which, provided L <<<<> (K, .) presents an opportunity for significant model compression. For instance, if N K, i.e., W is square, and L K/4, then our method yields 50% compression at inference time. This is one of the compression rates we target in Section
We emphasize that P is not shared across GEMM layers; rather, each GEMM layer decomposed according (3) has its own pre-calibrated matrix P. Furthermore, by virtue of (3) not introducing dependencies across mini-batches, our method is fully compatible with data parallelism.
Eigen Static Principal Activation Component Estimation
Our proposed activation decomposition induces an approximation error as X * PP"X. In this section, we first introduce an ergodic estimation of activation auto-correlation. This important statistic is then used for theoretical constructions of P with guarantees on computational accuracy. Multiple results are presented and then combined in our compression studies in Section 4.
3.1 Activation auto-correlation estimation
Let x be an arbitrary K-dimensional vector in X; we define the activation auto-correlation matrix of size K x K as Cx E [xx"] where expectation is taken over activation vectors. This matrix symmetric positive semi-definite having a real eigenvalue decomposition (EVD) Cx VDVT where V is an orthonormal matrix whose columns are eigenvectors, and D is a diagonal matrix containing the corresponding non-negative eigenvalues, assumed to be sorted in decreasing order. The eigenvector corresponding to the jth largest eigenvalue is called 7th principal eigenvector.
This autocorrelation matrix be empirically estimated using instance of the activation tensor:
X = [x
xx"" [xixi +... + xM + Cx XxT/M /MM
(4)
However, evaluating (4) and its EVD dynamically introduces a prohibitive computational overhead. Thus, we estimate Cx in pre-deployment calibration process. Specifically, during calibration, we sample and forward pass B random input batches, and for each, calculate C x x /M, where superscript denotes batch index. Then, we average our estimate of the auto-correlation matrix as Cx Ei / and use its eigenvalue decomposition for further optimizations.
This ergodic approach of estimating activation statistics part of calibration process has been employed to great effect in other compression works on quantization [26, 27] and pruning [28].
3.2 Activation decomposition with minimum mean squared error
Let us write X -PP7X;
X; ; for a vector x E X, its counterpart in & E X given by:
x = (p. x)p;
(5)
where {p are the orthonormal column vectors of i.e. (p., p.) = E1.
We define the mean squared error (MSE) of the decomposition as
[[|x x| []
(6)
with the -orm used throughout this paper. Our first result constructs P minimizing this MSE.
5

---

Theorem 1. For an activation tensor X whose auto-correlation matrix has an eigenvalue decompo- sition given by Cx VDVV", the projection matrix P minimizing the mean squared error in (6) given by P = [vil [vr.] where V; is the ith principal eigenvector in V.
Proof. See Appendix A.I. The result is readily obtained by substituting x in (5) into (6) and minimizing the MSE which involves quadratic forms involving the positíve semi-definite Cx. E
Theorem 1 shares similarities with the Principal Component Analysis (PCA) algorithm [29]. PCA extracts low dimensional features having maximum correlation with input data. Unlike PCA, we omit input normalization to elide its computational cost. Still, term the columns of P in Theorem I as Principal Activation Components. Since those are obtained using an EVD on a static estimation of Cx, call our method Eigen Static Principal Activation Component Estimation (ESPACE).
The MSE in Theorem 1 is strong indicator of the quality of an approximation technique, e.g., often employed in quantization studies [26, 27]. However, empirical data may contain large outliers which can dominate the optimization process, say a few high-magnitude vectors in (6) masking the contribution of small data on the solution. An alternate metric to the MSE can be employed to prevent such artifacts in averaging: the normalized MSE (NMSE) defined as:
E [|X-x|
x1 [[|M|]]]
The solution of Theorem I can be slightly modified to minimize the NMSE in (7).
Corollary 2. For an activation tensor X. let Öx E ((//I)))) (5/11xI)] 71
be its input input-normalized
auto-correlation matrix having an eigenvalue decomposition given by Cx = VDVT, the projection matrix P minimizing the normalized mean squared ervor in (7) is given by |vll... [v]] where V; is the th principal eigenvector in V.
Proof. The proof in Appendix A.2 uses equivalence of NMSE and MSE with Lz-normalized vectors.
Corollary 2 applies to the decomposition in (3) with no activation normalization required compute time. Rather, normalization is done during calibration, where Cx is estimated instead of Cx.
Both solutions in Theorem 1 and Corollary 2 are options to be employed in ESPACE, where either may be more suitable on a layer-wise basis. Next we present further options for ESPACE based on the optimization of alternate metrics to the MSE and NMSE
3.3 Activation decomposition with optimized forward propagated accuracy metrics
While local fidelity metrics, such as the MSE and NMSE above, are good indicators of the quality of an approximation technique, has been shown that better insights on a neural network's accuracy may be derived via the study of forward propagated noise [30, 31, 32]. In this section, we study the effects of the decomposition in (3) on the output of the GEMM, and the output loss of the model.
At a given layer, let us write an arbitrary scalar in the GEMM output tensor in (1) as E Y. Note that y = (w, x) for some weight vector w W and activation vector x E X. We also let ý be the associated output when the GEMM is approximated by (3), which is given by ú = (w, x) with x given by (5). We define the GEMM Output-referred MSE (GO-MSE) as E [vv "
Similarly, given an input to the network, we write the output loss function (the vocab cross-entropy) as C. When one arbitrary activation tensor is transformed as per (3), a mismatch in computation is introduced and propagated all the way to the output. We let L be the resulting new value of the loss function. We define the Network Loss-referred MSE (NL-MSS)) as I..
(C 2)3
closed form solution for in (3) minimizing the GO-MSE and NL-MSE elusive to us. Therefore, derive upper bounds these metrics which we use as a proxy for optimization.
Proposition 3. For a GEMM in (1) and its decomposition in (3), the GO-MSE is upper bounded by:
E (v 5) < 21E
|||]
||x|
2E [(w x) (w, (w]
(8)

---

and the NL-MSE is upper bounded by
E - 2TF [
||x]]]]
21 (V x: x) (Vx; 8)]
(9)
where first order Taylor approximation on the loss function is assumed and its gradient with respect to vector denoted as Xx
Proof. The proof Appendix A.3 first shows ||x]| < ||x|| and then uses the Cauchy Schwarz inequality to establish both bounds.
Next, we provide closed form solutions for P in (3) minimizing the bounds in Proposition 3.
Theorem 4. For GEMM in (1) and its decomposition in (3), the projection matrix minimizing the bounds in Proposition is given by P == [vil.... ... |vr.] where v, the it principal eigenvector V obtained via eigenvalue decomposition on matrix C = VDVT defined as:
C [xxxxww" ww xxx" and C IE V + V
(10)
minimize the upper bounds on GO-MSE in (8) and NL-MSE in (9), respectively.
Proof. The proof is included in Appendix A.4, where also include modifications required in calibration. Specifically, Cx is reused and left/right multiplied by ww /N to yield C in (10) minimizing the bound on GO-MSE. An additional backward pass is needed to properly scale activation vectors and their gradients when calibrating C in (10) minimizing the bound on NL-MSE.
Theorem 4 augments Theorem and Corollary 2 with two options for the design of P. Much like Corollary 2, we supplement our new solutions with nooormaalization to include
= IE' [(xxx" ww" +ww" xxx" ////| [x]]]] and Č = LE' C 1/1I./ 12
as alternate choices for the calibrated matrices C in (10). Unlike Corollary 2, Iız-normalization in these matrices does not correspond notable optimization. Nevertheless, these options retained in the spirit of suppressing the influence of large data in calibration.
Thus, overall we have six choices for P. Since each can be obtained part of fast and pre- deployment calibration phase, we may simply seleet the best one for each layer. our experiments of Section 4, the best candidate is determined via a per-layer validation over all six choices. A sensitivity study on the impact of each of the six candidates is provided in Appendix B.4.
4 Model Compression Studies
In this section, we report on experimental studies investigating LLM compression using ESPACE.
4.1 Experimental setup
We employ three sets of open source LLMs: GPT3 [33], Llama2 [34], and Nemotron4 [35]. Specifi- cally, we experiment on GPT3-(1.3B, 8B, 22B), Llama2-(7B. 13B), and Nemotron4-1557 Accuracy evaluated two ways: perplexity measured on the Wikitext- 103 dataset [36] and zero-shot down- stream task accuracy of: BoolQ (BQ) [37], Hellaswag (HS) [38], PIQA (PQ) [39], RACE (RA) [40], and WinoGrande (WG) [41].
The Wikitext-103 dataset is split into train, validation, and test sets. We use 512 random sequences from the training set for calibrating projection matrices required by ESPACE. We use the validation set for layer- layer-wise sensitivity studies. The test set is used to report perplexity results in this section.
Our implementation uses NVIDIA's Megatron LM [33] and downstream task evaluation invokes Eleuther Al's LM evaluation harness [42]. For the latter, we report raw accuracy scores, and their average; we do not post process results apply normalization to the scores.
When ESPACE is applied, we retrain the models adapt to the approximation error activation projection discussed in Section 2. Retraining simply extends the models' pre-training sessions and
7

---

uses the 330B-toker MTNLG dataset [43], which was used to train GPT3 models. All implementation details are included Appendix B to help reproducibility of our results.
We metricize model size reduction via inference compression rate. Specifically, for layers decomposed per (3), we count the number of entries in P and (PTW), for other layers, count those in WT. We also report the latency of executing all network GEMMs in (1) or (3), which we measure using a NVIDIA A 110 GPU and simple, un-optimized implementation (see Appendix B.4). We also report prefill inference latency, metricized via the Time to First Token (TTFT), and measured using the Megatron- implementation. In our measurements, we use a batch size of I and sequence length of 2048 and 4096 for GPT3 and Llama2/Nemotron4 models, respectively. The reported reductions in total GEMM lateney and TTFT constitute evidence that compression improves inference throughput. It is beyond the scope of this paper to evaluate the impact of ESPACE on end-to-end token throughput and latency on I.IM inference serving systems, since this requires a complex set of optimizations including but not limited to optimization of back-to-back GEMMs into fused kernels, KV caching, continuous batching, as well as thorough performance studies with varying input and output sequence lengths. Thus, we leave an evaluation of token generation throughput and energy savings and improvements to future work.
4.2 Validation perplexity studies
Our experiments start with a calibration phase where we prepare the static projection matrix for each layer. The dimension L in P is chosen as the lowest power of two such that layer compression at least 50%. The power-of-tve compression rate depends on the dimensions of the original layer (K and N). Exact details of these values for all layers and models are included in Appendix B.2.
restriction ensures best tensor core utilization, and the resulting
We perform a sensitivity study on the Wikitext-103 validation perplexity when ESPACE is applied out-of-the-box (no retraining) one layer at time. For each layer, we identify which of our six candidates projection matrices in Section 3 yields lowest validation perplexity. Layers are then sorted according to their impact on perplexity from least to most destructive. We then evaluate the validation perplexity when ESPACE is progressively applied to out-of- sff-the-bxx to all layers according to this ranking. Fine-grained details of this exploration are included in Appendix for all models.
This exploration yields an interesting finding: as we progres- sively apply ESPACE to more layers, the perplexity marginally increases until an inflection point after which accuracy degrada- tion accelerates. This inflection occurs at 20% 40% compres- sion depending on the model. Figure 3 depicts this phenomenon for GPT3-22B, and the same data for other models be found in Appendix C.2.
GPT3-22B Compression
perplexity (lower is better)
Baseline Retrained
Wikitext- 103 validation
Inflection
We find that out-of-the the-box application of ESPACE works bet- ter for larger models; GPT3-22B, the largest model we exper- imented on, exhibits an inflection in perplexity at 40% com- pression, which is the highest in our results. This is consistent
@40% compression
Up to lossless compression DOTB
Healing
with many earlier works on general compression of neural net- works [44, 45, 46, 47]. Interestingly, a 20% out-of-the-box compressed GPT3-22B is iso-accurate to its uncompressed counterpart (see Figure 3); without retraining, its test perplexity of 6.61 which is within 1% of the 6.55 baseline.
Compressed model size (36)
Figure 3: Validation perplexity for GPT3-22B when ESPACE is progres- sively applied to its GEMM layers. The order of layer selection is based on a layer-wise sensitivity analysis.
After the above validation study is performed, we select two configurations for layers to be compressed using ESPACE: (a) layers corresponding to the inflection point, i.e., 20% to 40% compression, and (b) as many layers needed to achieve a compression of ~50%. For both configurations, retrain the compressed models and further evaluate their achievable accuracy.
4.3 Compression of GPT3 models
Once compression targets and layer configurations are set, we retrain GPT3 models on the MTNLG dataset. Although we use all of the 330B available tokens, we do observe the training loss quickly
8

---

Table 1: GEMM latency, time to first token, Wikitext-103 perplexity (WK-103 PPL), and downstream task accuracy of GPT3, Llama2, and Nemotron4 models compressed with ESPACE".
Method (Compression)
of Weights
Total GFMM
TIFT impl. in Megatron-LM
WK-103
Downstream Task Accuracy
Latency
4
HS
PQ
RA
WG
GPT3-133
Baseline
1.21 10
24.2ms
39.8ms
9.94
64.3
43,5
74.2
37.6
58.1
55.5
ESPACE (20%)
9.71 10
20.6ms (-15%)
Jmi (-9%)
9.53
60.6
45.1
73.0
36.4
62.9
55.6
ESPACE (47%)
6.42 10"
15.9ms (-34%)
31.7ms (-20%)
11.07
39.9
71.6
34.5
58.7
GPT3-88
Baseline
8.05 10"
1.36m
186ms
7,38
69.0
54.2
78.1
41.4
67.8
62.1
ESPACE (21%)
6.33 10"
110ms (-19%
155ms (-16%)
7.00
70.3
55.3
78.9
40.7
69.3
62.9
ESPACE (50%)
4.08 x 10
76.8ms ( (-44%)
122ms (-35%)
7.65
66.5
52.3
77.6
38.9
66.9
60.4
GPT3-228
Raseline
2.17 x 10
354ms
457ms
6.55
76.4
57.2
79.3
40.7
70.5
64.8
ESPACE (40%)
1,30 x 10
229ms (-35%)
313ms (-32%)
6.29
76.6
57.3
79.5
40.2
70.2
64.8
ESPACE (55%)
9.74 10"
181ms (-49%)
261 (33%))
6.73
72.2
53.8
79.3
40.1
69.7
6,3.4
Llama2-78
Baseline
6.48 10"
210ms
368ms
5,06
79.2
57.1
78.1
44.0
69.5
65.6
Retrained (0%)
6.48 10
210ms
368ms
5,06
78.2
$7.9
78.0
43.7
70.6
65.7
ESPACE (21%)
5.11 10"
169ms (-19%)
322ms (-12%)
5.07
77.1
57.7
78.7
42.7
69.2
65.0
ESPACE (50%)
3.24 x 10"
113ms (-46%)
266ms (-28%)
5.67
72.2
52.0
76.5
38
63.5
60.4
Llunn2:
Bascline
1.27 x 1040
406ms
643ms
79.5
46.8
71.9
ESPACE (20%)
1.01 x 10
336ms (-17%)
562ms (-13%)
4.59
78.3
60.5
79.5
43.0
72.8
66.8
ESPACE (50%)
6.34 10
259ms (-36%)
447ms (-31%)
5.13
75.7
56.2
78.0
41.5
69.7
64.1
Nemotron4-1S
Baseline
1.25 x 10
414ms
7s
6.06
78.3
62.1
81.1
47.0
75.2
ESPACE (25%)
9.54 10"
324ms (.22%)
(.12))
80.0
46.4
72.8
ESPACE (50%)
10"
223ms (-46%)
(-26%)
77.8
42.4
69.9
converging. We leave training hyperparameters unchanged except for one: we disable dropout. Our rationale is that activation projection is one form of deterministic and structured dropout such that additional regularization may not be needed. Results on GPT3 models are included in Table 1.
We find that ESPACE can compress GPT3 models by ~50% at the cost of small accuracy degrada- tion. In the case of GPT3-22B, the perplexity increase is of only 0,18; in general, the gap decreases for larger overall model size. By and large, similar trends are observed for downstream task accuracies and we note that most scores of 50% compressed models fall within 5% of the baseline.
For lower compression ratios (inflection points at 20% to 40%), ESPACE converges to accuracy better than that of the baseline. The best improvement occurs for GPT3-8B, where ESPACE produces 6B model with 0.38 lower perplexity than its 8B baseline. The improvements terms of perplexity and downstream task accuracy. While GPT3 models may be over- wver-parameterized we posit that ESPACE acts regularizer moderate compression rates. Specifically, we believe that projection onto principal activation components filters out unnecessary information coming from small eigenvalue components.
observed both in
For all models, we observe an encouraging translation of compression to GEMM latency reduction by up to 49% which leads to noticeable speed-up in TTFT by up to 43%..
4.4 Compression of Llama2 models and comparison to related works
For Llama2, we only retrain using 200B MTNLG tokens because we observed quick convergence for GPT3. Llama2 models were trained on an undisclosed dataset of 2T tokens [34]. Therefore, with 200B tokens, the healing phase of ESPACE constitutes no more than 10% of the original pre-training session. Since Llama2 pre-training details are not openly available, we re-used all hyperparameters from GPT3, which is likely to be sub-optimal. In spite of the two handicaps of dataset disparity and hyperparameter sub-optimality, we obtained promising results as reported Table
For Llama2-78 we first retrained the uncompressed bascline. The purpose of this experiment is twofold: (a) ensure that our hyperparameters at least do not corrupt the model, and (b) verify that the
BBlldace indicates best result per task. Italics indicates ESPACE results within 5% of the baseline

---

healing process is not just an artifact of processing more tokens. Both hypotheses appeared to be valid: the retrained baseline has nearly identical accuracy compared to the original model.
Generally, we find that the trends of ESPACE compression for Llama2 are similar to GPT3, albeit slightly less successful. Though the results are still promising, we attribute the slight shortcomings in accuracy to the handicaps above. We find that 50% ESPACE compression on Llama2 leads to ~0.6 perplexity increase and similar degradation in terms of downstream task accuracy. Notably, compressing the Llama2-13B model to to a 6.3B model yields comparable accuracy to the Llama2-7B baseline which itself is 6.5B model.
In addition, for 20% compression, we find that ESPACE matches the accuracy of the baseline for Llama2 models. While not impressive the improvements observed with GPT3, ESPACE is able to produce 5B and 10B models matching the 7B and 13B baselines, which does push the pareto frontier of accuracy versus model size in the right direction shown in Figure 1.
The Llama2-7B model has been used related works on tensor decomposition mentioned in Section 1.1; specifi- cally, ASVD [20], SVD-LoRa [18], and SliceGPT [21]. Both ASVD and sliceGPT have reported perplexity on Wikitext, but SVD-LoRa performed specifcc finetun- ing variety of datasets and averaged perplexities. Therefore, in Figure 4, we compare our results to these works using perplexity increase raw perplexity, for maximum inclusivity.
Llama2-78 perplexity and compression
Perplexity incresse
ESPACE Joun)
ASVD
3.5
SVD-Loro
SicetPt
ESPACE is stop the right direction
baseline, rather than
225
.20
Compressed model size (%)
SVD-LoRa performed an SVD decomposition on the weights such that the intermediate dimension is half dot- product which leads to compression. On the other hand, ASVD and sliceGPT can only achieve modest compres- sion ratios of up 25% with some loss in accuracy. Recall that these works apply factorization on weights which is the fundamental difference to ESPACE. As seen in Figure 4, ESPACE is step in the right direction towards improving the state-of-the-att in tensor decomposition of I.LMs.
Figure 4: Comparison to related works com- pressing Llama2-7B using matrix factoriza- tion techniques.
4.5 Compression of Nemotron4-15B
Finally, we used ESPACE to compress Nemotron4- into 9.54 and 6.25 billion parameters, as reported in Table 1. Retraining consumed 275B tokens which corresponds to ~ 3% of this model's original training session. Once more, compression with ESPACE leads to minimal degradation the moderate regime (25%) and yields small accuracy drop in the aggressive regime (50%).
Consistently with our findings for the above models, ESPACE reduces GEMM execution time by up to 46%. This, in turn, improves the TTFT by up to 26%. An interesting observation is that, for Llama2 and Nemotron4 models, the TTFT improvement slightly less pronounced than for GPT3 models. This is simply due to the fact that the latter uses a sequence length of 2048, whereas the former two use 4096. larger sequence length means more time is spent in attention cross-cctivttion products which amortizes the speed-up up in the GEMM layers.
Conclusion
We have presented ESPACE, novel compression technique realizing tensor decomposition of LLMs in an activation-centrie manner. A set of theoretical results were derived to guide the construction of activation projection which is done statically. Experimentally, we have shown promising results where ESPACE is able to ~50% compress modern LLMs the cost of small accuracy degradation. Compared to related works, ESPACE is first step in pushing the frontier of model size versus accuracy trade-offs. Future work includes combining ESPACE with alternate compression techniques such as quantization and pruning, and evaluating decomposition of activation tensors in attention. As potential extension to our algorithm, the use of matrix sketching and random projections may pave the way for better overall compressibility.
10

---

References
[1] E. Xue, Fu, W. Zhou, Z. Zheng, and Y. You, "To repeat or not to repeat: Insights from scaling IIm under token-crisis," Advances in Neural Information Processing Systems, vol. 37, 2023.
[2] Y. Liu, H. He, T. Han, X. Zhang, M. Liu, J. Tian, Y. Zhang, J. Wang. X. Gao, T. Zhong, al., "Understanding Ilms: A comprehensive overview from training to inference," arXiv preprint arXiv:2401.0200
2024.
[3] G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, "Smoothquant: Accurate and efficient post-training quantization for large language models," in International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
[4] E. Frantar and D. Alistarh, "Sparsegpt: Massive language models can be accurately pruned in one-shot," " International Conference on Muchine Leurning, pp. 10323-10337, PMLR, 2023.
[5] Lin, Tang, H. Tang, S. Yang, X. Dang, and S. Han, "Awq: Activation aware weight quantization for Ilm compression and acceleration," arXiv preprint wrX.v:2336600977. 2023.
[6] E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, "Gptq: Accurate post-training quantization for generative pre-trained transformers," arXiv preprint arXiv:22100 10. 17323, 2022.
[7] X. Ma, G. Fang, and X. Wang. -Jlm-runnr: On the structural pruning of large language models," Advances in neural information processing systems, vol. 36, pp. 21 21702-21720, 2023.
[8] I. Mirzadeh, K. Alizadeh, S. Mehta, C. C. Del Mundo, O. Tuzel, G. Samei, M. Rastegari, and M. Farajtabar, "Relu strikes back: Exploiting activation sparsity in large language models," arXiv preprint arXiv:2310.04564 2023.
[9] G.-I. Yu, S. Jeong, G.-W. Kim, Kim, and B.-G. Chun, "Orca: A distributed serving system for (Transformer--asee Based] generative models," in 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
[10] S. Kim. K. Mangalam, S. Moon, J. Malik, M. W. Mahoney, A. Gholami, and K. Keutzer, "Speculative decoding with big little decoder," Advances in Neural Information Processing Systems, vol. 36, 2024.
[11] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. Yu, J. Gonzalez, H. Zhang, and Stoica, "Efficient memory management for large language model serving with pagedattention, in Proceedings of the 29th Symposium on Operating Systems Principles, pp. 611-6161666 1-1626, 2023.
[12] L. Zheng, L. Yin, Z. Xie, J. Huang, C. Sun, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez, et al., "Efficiently programming large language models using sglang," arXiv preprint arXiv:2312.07104. 2023.
[13] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficiae exact attention with io-awareness," Advances in Neural Information Processing Systems, vol. 35, pp. 16344-16359, 2022.
[14] A. Edalati, M. Tahaei, A. Rashid, V. P. Nia, J. Clark, and M. Rezagholizadeh, "Kronecker decomposition for gpt compression," arXiv preprint 10.0v:2110008/1... 2021.
[15] B. Chen, H. Chen, J. He, Sun, and S. Jui, "Ternary singular value decomposition as better parameterized form in linear mapping," arXiv preprint arXiv:2308.076/4 07644, 2023.
[16] M. Xu, Y. L. Xu, and D. P. Mandic, "Tensorgpt: Efficient compression of the embedding layer in Ilms based on the tensor-train decomposition." " arXiv preprint arXiv:2307.00526 2023.
[17] Gu, B. Keller, J. Kossaifi, A. Anandkumar, B. Khailany, and D. Z. Pan, "Heat: Hardware- efficient automatic tensor decomposition for transformer compression." arXiv preprint
arXiv:2211.1674 2022.
[18] H. Badri and A. Shaji, "Low-rank pruning of llama2." https: htttps://mobuusm.ggr thw... io/ 11t-rarkk1lama22 2024.
[19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models." arXiv preprint 21Xx:2200600685, 2021.
[20] z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun, "Asvd: Activation singular value decomposition for compressing large language models," arXiv preprint arXiv:2312.0558 2023.
11

---

[21] S. Ashkboos, M. L. Croci, M. G. d. Nascimento, T. Hoefler, and J. Hensman, "Slicegpt: Com- press large language models by deleting rows and columns," arXiv preprint arXiv:2401.15024, 2024.
[22] J. Zhao, Z. Zhang, B. Chen, Z. Wang. A. Anandkumar, and Y. Tian, "Galore: Memory-efficient Ilm training by gradient low-rank projection," arXiv preprint arXi::2403.0350) 2024.
[23] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint urXiv:1412.6988 2014.
[24] G. Hinton, N. Srivastava, and K. Swersky, "Neural networks for machine learning lecture 6a overview of mini-batch gradient descent," Cited on, vol. 14, no. 8, p. 2, 2012.
[25] O. Johnson, Information theory and the central limit theorem. World Scientific, 2004.
[26] C. Sakr, S. Dai, R. Venkatesan, B. Zimmer, W. Dally, and B. Khailany, "Optimal clipping and magnitude-aware differentiation for improved quantization-awar training," in International Conference on Machine Learning, pp. 19123-19138, PMLR, 2022.
[27] H. Wu, P. Judd, X. Zhang, M. Isaev, and P. Micikevicius, "Integer quantization for deep learning inference: Principles and empirical evaluation," " arXiv preprint MXXiv:004.009600 2020.
[28] Dai, H. Gene, R. Venkatesan, and B. Khailany, "Efficient transformer inference with statically structured sparse attention," in 2023 60th ACM/IEEE Design Automation Conference ( (DAC),
pp. 1 1-6, IEEE, 2023.
[29] H. Abdi and L. J. Williams, "Principal component analysis," Wiley interdisciplinary reviews: computational statistics, vol.
pp. 433-459, 2010.
[30] C. Sakr, Kim, and Shanbhag. "Analytical guarantees on numerical precision of deep neural networks," in International Conference on Machine Learning, pp. 3007-3016, PMLR, 2017.
[31] C. Sakr and N. Shanbhag, "An analytical method to determine minimum per-layer precision of deep neural networks." in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1090-1094 1094, IEEE, 2018
[32] C. Sakr and N. Shanbhag, "Per-tensor fixed- fixed-point quantization of the back-propagee propagation algo- rithm," in 7th International Conference on Learning Representations, ICLR 2019, 2019.
[33] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, "Megatron-Im: Training multi billii-iliion parameter language models using model parallelism." 2020.
[34] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi. Y. Babaci, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al., "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 2023.
[35] J. Parmar, S. Prabhumoye, J. Jennings, M. Patwary, S. Subramanian, D. Su, C. Zhu, D. Narayanan, A. Jhunjhunwala, A. Dattagupta, et al., "Nemotron-4 15b technical report," arXiv preprint arXiv:2402. /68119 2024.
[36] S. Merity, C. Xiong, Bradbury, and R. Socher, "Pointer sentinel mixture models," 2016.
[37] C. Clark, K. Lee, M.-W. Chang. T. Kwiatkowski. M. Collins, and K. Toutanova, "Boolq: Exploring the surprising difficulty of natural yes/no questions," in NAACL, 2019.
[38] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, "Hellaswag: Can machine really finish your sentence?," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[39] Y. Bisk, R. Zellers, L. Bras, Gao, and Y. Choi, "Piqa: Reasoning about physical com- monsense in natural language," in Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
[40] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, "RACE: Large-scale ReAding comprehension dataset from examinations," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, (Copenhagen, Denmark), pp. 785-794, Association for Computational Linguisties, Sept. 2017.
[41] S. Keisuke, I. B. Ronan, B. Chandra, and C. Yejin, "Winogrande: An adversarial winograd schema challenge at scale," 2019.
12

---

[42] L. Gao, J. Tow. B. Abbasi. S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac"h, h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa. Phang, L. Reynolds. H. Schoelkopf, A. Skowron, I.. Sutawika, E. Tang, A. Thite, B. Wang. K. Wang, and A. Zou, "A framework for few-shot language model evaluation," 12 2023.
[43] S. Smith, M. Patwary, B. Norick, P. LeGireseey S. Rajbhandari, J. Casper, 7. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al., "Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model," arXiv preprint arXiv:2201.11990 2022.
[44] E. Park and S. Yoo, "Profit: A novel training method for sub-4-bit mobilenet models," in Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 430-446, Springer, 2020.
[45] M. Nagel, M. Fournarakis, Y. Bondarenko, and T. Blankevoort, "Overcoming oscillations in quantizaiion-wwar training," in International Conference on Machine Learning, pp. 16318- 16330, PMLR, 2022.
[46] A. Kuzmin, M. Nagel, M. Van Baalen, A. Behboodi, and T. Blankevoort, "Pruning vs quan- tization: Which is better?," Advances Neural Information Processing Systems, vol. 36, 2024.
[47] L. Dery, S. Kolawole, J.-F. Kagey, V. Smith, G. Neubig, and A. Talwalkar, "Everybody prune now: Structured pruning of Ilms with only forward passes," arXiv preprint arXiv:2402.05406, 2024.
[48] R. A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012.
[49] S. Ma, H. Wang, L. Ma, L. Wang, W. Wang, S. Huang, L. Dong, R. Wang, Xue, and F. Wei, "The era of 1-bit llms: All large language models are in 1.58 bits," arXiv preprint
140.:22002.1764 2024.
[50] M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, Alben, H. Anand, Banerjee.
I. Bayraktaroglu, et al., "Chipnemo: Domain-adapted 1lms for chip design," arXiv preprint
1xXiv:2311.00177 2023.
[51] X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen, "Shortgpt: Layers in large language models are more redundant than you expect," arXiv preprint arXiv:2403.03853, 2024.
13

---

Supplementary Material
A Proofs of theoretical results
In this first appendix, provide proofs for the various theoretical results in Section 3.
A.1 Proof of Theorem
For a pair of vectors x E and x E X, and using (5), we have the squared error:
1.
xx x ||x||
|
2x
| +
|||1
(pi x) p
||x
x|
+ ||x1
|||| Σ
(p: xx²
1=1
Furthermore, note that
|
xTx
Σ
(p: (x:"
E (pi"x)) pi
and since {p:) are orthonromal, cross product terms vanish and we have: ||*| Σt= (pr²x) x)2 which we plug back into the expression for the squared error:
Xxx - x||
||x|
(pi"'x) xx""))= Σ (p. x) = =|xx|2
Σ
(p: x))
x))
1=1
1m1
Furthermore, note that:
(p: (pi (piTx) = (pi (x" p:) = =p;" (xx")
where used the commutativity of dot product and associativity of matrix multiplication. Thus the squared error is given by:
L Ept (xxx") pi
|x - x||²= ||x||²
)
Finally, we take expectation both sides and obtain a formula for the MSE:
L P. (xx") pi E ||x|
E x - x
|
E
Σp.
Σ
p: E
xxx
pi
where used linearity of expectation and the fact that {p:k= not random. In this formula for the MSE, E [||x||]] does not depend on {p. ) and therefore. minimizing the MSE is equivalent to maximizing the following expression involving the auto-correlation matrix:
Σ
pi TE xx
Σ P TCppi
pi
where each term in the summation is quadratic form the positive semi-definite auto-correlation matrix Cx. Since { are orthonormal, this an equivalent form the Rayleigh quotient [48] and the solution assign {p as the principal eigenvectors of Cx. This concludes the proof of Theorem 1.
14

---

A.2 Proof of Corollary 2
The result can be readily obtained as a consequence of the following:
x
x
x
Σ
Pr.
x)P:
x
E
EE
Ex
12
X
x
|
X x x E
x
x
E
X 1
=
P
P:
|
Therefore, the setup is identical to that of Theorem and we may apply the same solution as Appendix A.1 above. The only difference is that activation vectors are normomlle which is why Cx (which is also positive semi-definite) is used in lieu of Cx in Corollary 2.
A.3 Proof of Proposition 3
prelimnary result needed is to show that for any activation vector, we have ||x|| < ||x|2. We first x)P: note that while x = Σ x)): per (5), we also have that x = Σ x),,, where { extend the set of orthonormal vectors {pi} to be complete, i.e., equivalent to no truncation of columns of the full rank matrix V when constructing P, regardless of the metric being optimized.
Using orthonormality of projection vectors, similar to the proof of Theorem I in Appendix A.1 above, we obtain | E/=
"x"" and ||I||
Di=l
K
(pi"x)² Therefore:
K
|x| 2 |
Σ
(pi (pi"x) 2 2 0
| |x|| 2
where we used the fact that a sum of non-negative quantities is non-negative.
Then for scalar y Y and its counterpart ÿ Y, we have:
(y 0) = (w* x w x) = (w" x) 2 + (w"*x) 2 - 2 2w"xx="T
< w || ||x| + ||w|2 ||||| -2 (w" xw""x) x)
w/12 ||x|| + ||w| 2 ||x|| - 2 (wT xw
= 2|| -x 2w""xw"xx
where the first upper bound uses the Cauchy-Sethe Schwarz inequality while the second uses ||x|| < ||x| which we proved above. Taking expectations on both sides of the inequality yields the upper bound on GO-MSE in (8).
Next, when first order Taylor approximation on the loss function is assumed, we have the following relation between the unperturbed loss value L and its counterpart E when an activation vector x is projected per (5):
i = + ( (V V
x) E
V
V
(V
(V
once more, we use the Cauchy-Schweae inequality and the fact that
||
2 ||x | establish:
(Vx))² 2 < 1 2 ||x|2 & (VT*) < |x
| V ||
|||
which we plug into the difference network losses above to obtain:
(2- (2-c)² s 22 |||| - 2 (VTVV<>
Taking expectations on both sides yields the upper bound on NL-MSE (9). This completes the proof of Proposition 3.
15

---

A.4
Proof of Theorem 4
In order to minimize the upper bound on the GO-MSE in (8), note that suffices to maximize
quantity 21 [w.x). (w,x)] since 21E [|//|| ||x]|²] | following:
does not depend on {p: We have the
2(w,x) (w, 22 xw" Σ
(Dfx) pi Σ
w xw" p.p. x
Σ
xw P.p. + w xw" p.P.
xw
But since the dot product is commutative, i.e., a b for any two vectors a and b, we may rearrange each of the two identical terms inside the summation as follows:
w x "PPP x = D. xxxxx p. x"xw w'xw"p.p/' = P. w
Therefore, we obtain:
2(w,x) (w, Σ
(p) xx2 ww Pt + www xx² p.)
i=1
I.
Σ pt (xx" xxx"'wee ww" + ww ww""xx") pi
Taking expectations, we find that the quantity that needs to be maximized in order to minimize the bound on GO-MSE in (8) is:
P E xx ww" + www xx" p
Similar to the proof of Theorem in Appendix A.1, this yet again sum of quadratic form over the orthonormal set of vectors [p: [D: and the solution is therefore to assign these vectors as the L principal vectors of = xxx" ww" + wwwxxx xx"" as per (10).
Note that the derivation above decomposed the dot products to obtain quadratic form on the matrix xx wwT + ww x because its symmetry is required for real eigenvalue decomposition. Also note that if positive definiteness is not achieved, observe that this solution requires no overhead on the calibration process. Indeed, assuming weights Cx Cx Cw Cx where Cw E. [ww?] is the weight auto-correlation matrix, which can simply be calibrated as Cw ww /N. Thus we only require left and right scaling of the calibrated auto-correlation matrix.
sort absolute values of eigenvalues. Finally,
and activations are independent, we note that EE xxx wwf + ww xx="
Similarly, to minimize the upper bound on NL-MSE in (9), it suffices to maximize (VV, x) (Vx; Using the exact same derivation the above, replacing by Vx, obtain that the quantity to be maximized is
Σ EX V xxx" p
which is done by assigning (pi)/2 as the L principal vectors of C - as per (10). Calibrating this matrix does require an extra step. where we perform a backward pass to estimate activation gradients. For ease of implementation, on the per-sequence independence of activation vectors and their gradients. This greatly reduces the memory requirements the calibration process. And for each sample sequence in the calibration set, we compute (() = (0)) ( ((0) + Ga0 x90 x())) ) /M where ( is the gradient tensor (whose vectors are instantiations of Vx). As always, we end the calibration phase by averaging across samples: C Σ c This completes the proof of Theorem 4.
+ V
our results, we make an approximation
16

---

B Implementation details
In this appendix, we discuss all details behind our implementation in Section These details include per-model specific application of ESPACE well as retraining recipes. We strive provide excessive details such that independent reproducibility of our results is seamless. We also encourage readers to reach out to us (after blind reviewing) for any questions implementations.
B.1 Software implementation
As was mentioned in Section 4, our implementation is built on top Megatron- [33] which itself is based on the Pytorch framework. We use Pytorch for all extra introductions needed by ESPACE except for eigenvalue decomposition. Our experiments were carried out a cluster of A100 GPUs and use BF16 precision.
Specifically, during the calibration process, matrices; this simply done by averaging repeated instantiations of XX" as described in Section 3.
Pytorch to track the required auto-correlation
Once the calibration of cuto-correlation matrices is over, Pytorch framework to RAPIDS framework. We then use the CUPY library in RAPIDS to perform fast few milliseconds per nuto-correlation matrix) eigenvalue decomposition GPUs. After truncating eigenvectors, we send back the projection matrix to Pytorch using DLPack.
DLPack to transfer them from the
Once the projection matrix P is calibrated and inference/rraining is to be done using ESPACE, we simply insert a projection operation within the Megatron implementation to perform the operations in (3) as appropriate. The projection matrices are inserted as Pytorch buffers, rather than parameters, since they do not get updated during training.
B.2 ESPACE configurations
In Section 4, we mentioned that ESPACE was applied at each layer such that the number of compo- nents L satisfies two constraints: (a) power of for best tensor core utilization, and (b) yield compression of at least 50% at that layer. The exact values of L for each model and layer type are included in Table Note that the only exception corresponds to QKV layers in Llama2-133 -3BB and Nemotron4-15B. where we use a value of L 2048 which corresponds to compression of ~ 45% instead of 550% at least. This is only because this amount of compression already significant that we didn't it feel the need to push for J. = 1024, which would have lead to compression of >> 70%.
B.3 Retraining hyperparameters
By and large,
the exact same recipe that was used pretrain the open source GPT3 models [33]. As mentioned in Section 4, the only modification to byperparameters disabling dropout and weight decay, and identical hyperparameters
used for both Llama2 families. The only arbitrary choices we had to make was on the selection of learning rate schedule and global batch size. We cosine decay for all runs, and remaining choicesa as follows:
of experiments on GPT3 and
For GPT3-L.3B, the initial learning rate is set to 1.0 x 10-4, the final learning rate is set to 1.0 x 10 and the global batch size is set to 512.
For GPT3-8B, the initial learning rate is set to 5.0 x 10-5, the final learning rate is set to 5.0 10-6, and the global batch size is set to 512.
For GPT3-22B, the initial learning rate is
5.0 10-5 the final learning rate is set to 5.0 x 10-6, and the global batch size is set to 1024.
For 1lama2-7B and 1.3ma2-13BB training is done in two stages (each of 100B tokens). In the first stage, the initial learning rate set to 5.0 x 10-1, and the final learning rate is set to 5.0 10-5 In the second stage, the initial learning rate is set 5.0 10-5 and the final learning rate is set to 5.0 x 10-6, For both stages, the global batch size is set to 256.
For Nemotron4- 15B, the initial learning rate is set to 1.0 x 10-5, 55 the final learning rate is set to 0, and the global batch size is set to 512.
We did not perform hyperparameter tuning, the above was purely arbitrary, but based on the following intuition:
17

---

Table 2: Number components retained by ESPACE for each layer. For models implementing Tensor Parallelism (TP), we apply ESPACE per rank.
Atto QKV layors
Atn Proj layers
FCI (H-to-4H) layers
FC2 (4H-to-H) Layers
GPT3 133 (TP=1)
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
K 2018 N 6141
K 20:18 N 2048
K 2048 N 8192
K 8192 N 2018
ESPACE: I, 512
ESPACE: I. 512
ESPACE: I. 512
ESPACE: I. 512
Compressin:
Compression: GPT3-8B (TP=4)
Compression:
Compression:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
K- 1096 N 12288
K 4096 N 4096
K 4096 N 16384 K 16381
1096
GEMM per rank:
GEMM per rank:
GEMM per rank:
GEMM per rank:
K = 4096 N 3072
K 1024 N = 4096
K 4096 N = 4096
K = 1096 N = 1096
ESPACE: I. 1024
ESPACE: I. 256
ESPACE: L 1024
ESPACE: I. 1024
Compression: 66%
Compression: 69%
Compression: 69%
Compression: 50%
GPT3-228 (TP=8)
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
K 6144 N 18432
K 6144 N 6144
K 6144 N 24576
K 24376 N 6144
GEMM per rank:
GEMM per rank:
GEMM per
GEMM per
6144 N 2304
K 768
6144
6144 N 3072
K 3072 N 6144
ESPACE: L 20-18
ESPACE: L = 256
ESPACE: L = 2018
ESPACE: L= = 1021
Compression: 56%
Compression: 63%
Compression: 58%
Compression: 50%
Llama2 (TP=4)
GEMM Dimension:
GEMM Dimension:
GEMM Dimensãon:
GEMM Dimension:
K = 4096 N 12288
K = 4096 N= = 4096
K = 4096 N = 22016
K = 11008 N 4096
GEMM per rank:
GEMM per rank:
GEMM per rank:
GEMM per rank:
1096 3072
K 1021 N 4096
1096
5501
2752
4096
ESPACE: L 1024
ESPACE: L 256
ESPACE: L 1024
ESPACE: L 512
Compression: 66%
Compression: 69%
Compression: 69%
Compression: 69%
IJama2 13 (TP=8)
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
K 5120 N = 15360
K = 5120 N = 5120
K 5120 N = 27648
K 13824
51
GEMM per rank:
GEMM per rank:
GEMM rank:
GEMM per rank:
K 5120 N 1920
K 640 5120
K 5120 N 3456
K 1728 N 5120
ESPACE: L 2048
ESPACE:
ESPACE: L 2048
ESPACE: 1024
Compression 47%
Compression 55%
Compression: 53%
Compresioon 60%
Nemotron4 (TP=8)
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
GEMM Dimension:
6144 8192
K = 6144 N = 6144
K 6144 N = 24576
K = 24576
G144
GEMM per rank:
GEMM per rank:
GEMM per rank:
GEMM per rank:
K- 6144 N 1024
K 768 N = 6144
K 6144 N 3072
K 3072 N = 6144
ESPACE: I. 2048
ESPACE: I. 256
ESPACE: L 2048
ESPACE: I. 1024
Compression: 42%
Compression: 67%
Compression:
Compression:
For GPT3 models, we use a smaller learning rate for larger models, and start with a learning rate 10x smaller than it's pre- pre-training value. We use identical batch sizes pre-training
For Llama2 models, as pre-training hyperparameters are undisclosed, we use our best guess of what could work well. The two stage training approach is inspired by recent work on 1it8-bit LLMs [49], while the choice of a batch size of 256 is inspired by ChipNemo [50].
B.4 GEMM latency measurements
Here we describe the methodology employed measure GEMM latency as reported in Table 1. We assume a batch size of 1, such that the M dimension of tensor X equals the sequence length (2048 and 4096 for GPT3 and Llama2 models, respectively). We also assume single-GPU GPU implementation throughout, such that any tensor parallelism is first folded into single GEMM per-layer. Similar to our accuracy experiments, we use 6F16 precision for all latency measurements.
For each GEMM layer implementing either (1) or (3), we measure latency individually. Table 1, report aggregated measurements depending the model configuration. Specifically, we measure
18

---

NMSE
GO-MSE W/ Lynorrm ization
NL-MSE
with Ly-normalizttion
Attn QKV
Attn Proj
FC1 (H to 4H)
FC2 (4H to H)
104
512 comp. 66%
512 comp. 550
L 512
512 comp. 69%
Validation perplexity (lower betten)
comp. 699
163
154
200
Layer index (a)
Attn QKV
Attn Proj
FC1 (H to 4H)
(4H H)
L 1024
Validation perplexity
(per comp, 69%
[per rank)
(per comp. 50%
(lowerf is betterr
comp. 6666
Layer index (b)
Attn QKV
Attn Proj
FC1 (H to 4H)
FC2 (4H to H)
2048 [per rank)
= 256
1024
Validation perplexity lexity
(per rank)
(per comp. 58%
(per rank)
(lower better)
56%
50%
60
60
65
Layer index (c)
8
Figure 5: Sensitivity studies on the choice of projection construction for (a) GPT3-1.3B, (b) GPT3-8B, (c) GPT3-22B, For cach layer, we apply ESPACE out-of-the-bbe box using the six various candidates for the projection matrix P constructed in Section 3. The black line corresponds the baseline perplexity.
latency of computing QKV, Proj, FCI, and FC2 GEMMs with dimensions listed Table 2, and then add all results together for each transformer block of the corresponding model.
We measure individual GEMM latencies in Pytorch. Specifically, for each configuration, we sample 1000 set of matrices of appropriate dimension and compute the appropriate GEMM. We synchronize before and after the computation occurs, and record times after synchronization. The elapsed times are averaged and then aggregated. Since the measurements were taken with native PyTorch code, we note that the implementation un-optimized. Further improvements could be possible future work from removing Py Torch overheads, implementing custom fused kernels, other optimizations.
19

---

NMSE
GO-MSE w/ Ly-orrm ization
NL-MSE
with Ly-normalizttion
Attn QKV
Attn Proj
FC1 (H to 4H)
FC2 (4H to H)
55
1024
256 (per rank) 65%
1024 [per rank] comp. = 70%
1. - 512
Validation perplexity (lower betten)
(per rank)
(per rank)
66%
69%
Layer index (a)
Attn QKV
Attn Proj
FC1 (H to 4H)
(4H H)
L 255 (per rank) comp.
2048
L 1024 rank) comp.
Validation perplexity
(per rank)
(lower is betteer)
comp. 4.77%
Layer index (b)
Figure 6: Sensitivity studies on the choice of projection construction for (a) Llama2-7B. (b) Llama2- 13B. For each layer, we apply ESPACE out-of-the oo-thh-boo box using six various candidates the projection matrix constructed in Section 3. The black line corresponds to the baseline perplexity.
Attn QKV
Attn Proj
FC1 (H-to-4H)
FC2 (4H-to-H)
GPT3-1.3B
GPT3-8B
GPT3-22B
15
Llama2-78
Llama2-138
Validation perplexit plexity (lower is better)
Layer index
Figure 7: Validation perplexity when ESPACE applied out-of-the-box one layer at using the best calibrated projection matrix P as identified by the sensitivity study Figures and Figures The black line corresponds to the baseline perplexity and the dashed lines correspond to 1% increments over it.
C Additional experimental results
In this appendix, we include additional experimental results that were not included in the main paper. These results are not essential to the description our work nor its conclusion, and the main paper integrally contains all essential information related to our contribution. The additional results listed in this appendix are for the benefit of readers interested in going further and learning about fine-grained details behind the main results of Section 4.
20

---

Baseine
-.. ESPACE: Out Sor-of-he--or
ESPACE: Natramned
GPT3-1.38
GPT3-8B
Llama2-78
Llama2-138
Validation perplexity
Inflection
(lower is better)
Inflection 200
@20%
Inflection 22%%
Hualingg
Healing
Healingss
Healing
04
30
Compressed model size (%6)
Figure 8: Progressive out-of-the-box application of ESPACE on GL33 8B) and Llama2- 13B). The plot GPT3-22B was provided the main text in Figure progressive application of ESPACE is based on the ranking of layers from least to most destructive based on validation perplexity sensistivity in Figure 7.
C.1 Sensitivity studies on the construction of projection matrix
In Section 3, we presented theoretical results leading to six choices for the construction of projection matrix P. These constructions were done in a manner to optimize one of the following fidelity metrics: MSE, NMSE, GO-MSE, GO-MSE with L2 Lz-normalization, NL-MSE, and NL-MSE with L2 nn-normalization. Here we show the impact of various choices of P constructions on the validation perplexity, at a per-layer granularity. Specifically, we apply the ESPACE projection out-of-the-box one layer at a time, for each of the six candidates, and evaluate the resulting validation perplexity. In this way, we are able to determine which of the six candidate choices of P works best at each layer.
The results of this sensitivity analysis are included in Figures 5 and 6 for GPT3 and Llama2 models, respectively. It is shown that the best choice of projection matrix P depends on layer instance, and there no clear pattern to find out which solution works best a priori. This result justifies the need to optimize several proxy metric for accuracy, not just the MSE of activation approximation. Particularly, most solutions do appear to be related GO-MSE and NL-MSE, as well as thei normmallee variants. Therefore, these results provide supporting evidence on the importance of the results in Proposition 3 and Theorem A.4. This also validates the choice of using bounds on GO-MSE and NL-MSE for optimization since closed form solution for the unbounded metrics are elusive.
C.2 Progressive application ESPACE to the layers of network
Once the best projection matrix P is identified for each layer, we plot the corresponding validation perplexity for out-of-the-box application of ESPACE at the corresponding layer using the correspond- ing choice of P. These results are shown in Figure 7, where several observations are made. First, larger models have more resilience to out-of-the made Section 4. Second, it appears that FC1 layers are the most sensitive ones, followed by FC2, and OKV//roj layers generally robust to the application of ESPACE. Finally, we observe that in some instances, some layers close to the input and output (i.e., on either ends of the model) appear to be most sensitive the application of ESPACE. This behavior observed in other works on compression, such as shortGPT [51].
application of ESPACE; this observation was
As mentioned in Section 4, layers then sorted according to their impact perplexity from least to most destructive. ESPACE is then progressively applied to out-of-the-bre to all layers according to this ranking. In Figure 3 in the main text, we had shown the results corresponding to this applciation for GPT3-22B. In Figure 8, we show similar results for the other four networks we experimented on, ie., EPT3-((13B. 8B) and Llama2- (7B, 13B), ). Similar to the findings GPT3-22B. we do observe an inflection point after which accuracy degradation accelerates. This inflection occurs around 20% for GPT3-(1.3. (1.3B, 8B) } and Llama2- 13B). With retraining, the healing process recovers accuracy for all models, as detailed in Section 4.
We note the following:
21

---

For GPT3-1.3B. we exclude some layers from the application of ESPACE. These are the layers for which out-of-the-box application of ESPACE leads a validation perplexity increase of more than 2% compared to the baseline. These layers can be found in Figure 7 and correspond to several FC1 and FC2 layers close to either ends of the model. For this reason, GPT3-1.3B is compressed to 47% instead of 50% in Section 4 and Table 1.
For GPT3-22B, we apply ESPACE to all layers since validation perplexity increase is very small in Figure 7. For this reason, the overall compression for GPT3-22B is slightly over 50%; it is 55% in Section and Table 1.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately report the paper's contribution in the abstract and introduction; this is why we chose to have a bulleted list in standalone subsection in Section 1.2. For clarity, we listed all contributions the paper, in the order in which they appear: our proposal, followed by theoretical results, and summaries of experimental resulis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In our discussion on related works in Section 1.1, we have listed possible combinations of applying ESPACE alongside other methods (e.g., quantization or parameter efficient tuning), and have mentioned that this was not the scope of our paper. Furthermore, in the experimental Section 4, we have discussed the limitations of only metricizing compres- sion via model size and weight times activation GEMM latency reductions. We argued that more nuanced study needed on the inference cost implications Transformers comprise other GEMMs, and the regime (context pre-fill versus auto-regre regressveee phase) needs to be taken into account. We have mentioned that a detailed study on inference cost with ESPACE is part of future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
23

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide detailed proofs in Appendix A for all results in the theoretical Section 3. Furthermore, following each Theorem, Proposition, and Corollary in Section we also provide a teaser to the full proof in the main text. for the benefit of interested readers, where we try provide the main intuition before referring to full proofs Appendix A. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All details, including fine tneeggrined configurations, software toolkit employed, and training recipes including hyperaparameters are included Appendix B. These are also briefly mentioned in the main paper in Section 4, with references Appendix B for the readers interested in more details needed for full reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b)
If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: As was mentioned in the previous question, we have provided all implementa- tion details required to reproduce our results, including hyperparameters and description of software implementation in Appendix B. We note that our implementation is based on the open-source Megatron-LM [33] and that additions made this software toolkit needed to reproduce our results are included in Appendix B, where we also include an invitation to the reader to reach out to us (after blind reviewing over) with any questions regarding reproducibility. As such, we believe the description of the work in the paper is sufficient for reproducibility; yet, we are happy to consider open sourcing our code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips ec/ public/get wwtee//guid///eeeeee (CodeSSbmissionne oolicyyy) for more details.
While encourage the release of code and data, understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttgggidddd//duuuuiiiiiieeee
/ //nips.
www.ssssiiciiyi.iii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, and this was already mentioned in our responses to the two questions above. We provide all fine- fmninnd details of our implementation Appendix B, which is referred to in the main text.
25

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While we do not report error bars, we do evaluate our method on vari- ery of downstream tasks, for the specific purpose of making conclusions with statistical significance; rather than relying on one number here and there. We also note that this approuch of evaluating on several tasks is also widely udopted the community for works compression of LLMs..
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These are reported in Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
26

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips..cm httpp//pp...pppppooocccccceceeea
Question:
Answer: [Yes]
Justification: Yes, we are only working method to compress LLMs using tensor decom- position of activations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
As per the guideline below, work falls under the umbrella optimization to the implementation of LLMs. Similar to the example provided in the guideline, we believe that there is need point out societal implications making LLMs run faster.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There no such risks associated with our work, and as such the paper does not describe safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have done our best to provide credit to any prior work upon which we have built ours. This is mostly the case for Megatron-LM [33], which used for our experiments, and cited extensively throughout Section and Appendix B.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
28

---

Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "4yX5b99zs4", "reviewer": "Reviewer_yYTB", "summary": "This paper introduces ESPACE, a method that reduces activation in models via tensor decomposition, thereby aiding in the reduction of model size and GEMM latency.", "questions": "Please see the weaknesses.", "limitations": "The authors address the limitations.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "- The paper is easy to follow;\n- The method is simple but efficient.", "weaknesses": "- Wikitext-103 is a specific type of language dataset that focuses on knowledge-based content. However, models like GPT-3 and Llama2 are capable of handling a broader range of tasks, including math and question-answering. Therefore, it is necessary to evaluate perplexity (PPL) on a more diverse set of data.\n- ESPACE requires ongoing training. It is important to understand the duration required for ESPACE's continual training process.\n- In Table 1 and Table 2, since ESPACE is continually trained with more data than the original model, a direct comparison may not be entirely fair. It would be more convincing to compare the original model after it has been trained on the same amount of data as ESPACE.\n- For generative models such as GPT-3 and Llama2 discussed in this paper, there is a need for more comprehensive performance validation, including metrics like mmlu and math benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback provided. We provide detailed answers to the reviewer below:\n\n* Response to weaknesses #1 and #4 on Wikitext being an insufficient benchmark for empirical results and the need for a more comprehensive set of validation metrics.\n\n* * The reviewer is correct in that Wikitext is a language dataset focusing on knowledge-based content. As such, evaluating Wikitext perplexity is not the only metric to report on for proper evaluation of LLMs. In our submission, we had included evaluations of the models on the diverse LM evaluation harness suite of tasks (BoolQ, Hellaswag, Piqa, Race, Winogrande – see Tables 1 and 2 in Section 4). Of these question-answering tasks, BoolQ and Race are knowledge-based, while Hellaswag, Piqa, and Winogrande comprise common-sense and logical reasoning. Our results (in the submission) for these tasks showed that the accuracy-preserving strength of ESPACE is not limited to Wikitext perplexity, since the scores obtained on these tasks were very close to those of the baseline. In case the reviewer missed the results on downstream task accuracy in the original review, we kindly ask to check again. \n\n* * We’ve further expanded these results by adding MMLU evaluation as per the reviewer’s recommendation. The following table (also included as Table A in the common rebuttal attachment) summarizes these results on the MMLU benchmark (which itself comprises a broad range of tasks including mathematical and logical reasoning in addition to human knowledge). Here we use the same models, compression settings, and checkpoints as those discussed in our submission. MMLU inference is done using the zero-shot approach:\n\n\n|Model (Compression)    |MMLU score|\n|-|-|\n|Baseline GPT3-1.3B     |     25.5%|\n|ESPACE GPT3-1.3B (20%) |     25.3%|\n|ESPACE GPT3-1.3B (47%) |     25.1%|\n|-|-|\n|Baseline GPT3-8B       |     26.3%|\n|ESPACE GPT3-8B (21%)   |     31.5%|\n|ESPACE GPT3-8B (50%)   |     27.4%|\n|-|-|\n|Baseline GPT3-22B      |     36.3%|\n|ESPACE GPT3-22B (40%)  |     42.2%|\n|ESPACE GPT3-22B (55%)  |     39.8%|\n|-|-|\n|Baseline Llama2-7B     |     42.2%|\n|ESPACE Llama2-7B (21%) |     39.6%|\n|ESPACE Llama2-7B (50%) |     32.7%|\n|-|-|\n|Baseline Llama2-13B    |     52.9%|\n|ESPACE Llama2-13B (20%)|     49.4%|\n|ESPACE Llama2-13B (50%)|     39.4%|\n|-|-|\n\n* * As we can see, the results are generally consistent with our earlier findings from our experiments on Wikitext and the set of downstream tasks from the LM evaluation harness that were included in the original submission. We note that MMLU is a complex benchmark, such that the GPT3-1.3B model is too weak to produce meaningful results beyond a random guess. For larger GPT3-models, we once again find that moderate compression using ESPACE leads to improvements over the baseline, specifically for GPT3-8B with 21% compression and GPT3-22B with 40% compression. The improvements in MMLU scores are significant, and even higher than for the benchmarks covered in our submission. Indeed, ESPACE leads to a ~5% absolute increase in MMLU score, which corresponds to a 16%-to-19% relative improvement. On the other hand, 50% compression with ESPACE leads to an accuracy comparable to the baseline, which is consistent with our findings in our original submission. Our results for Llama2 models are not as strong, and we attribute this to the handicaps of our training sessions that were discussed in Section 4.4. Specifically, our experimental setup did not include the pre-training dataset and hyperparameter selection used by Meta to produce accurate Llama2 models. As such, while ESPACE does retain some MMLU accuracy, it does not perform as strongly for Llama2 as it does for GPT3. We also note that the MMLU benchmark is known to be challenging for base (unaligned) models (which we use throughout our studies). Overall, we are glad to include these additional results in our paper as we believe, in agreement with the reviewer, that they improve our work. Thank you for the suggestion!\n\n\n* Response to weakness #2 on the duration of the continual training process.\n\n* * In the original submission, we discussed training duration in Section 4.3 (330B tokens for GPT3) and section 4.4 (200B tokens for Llama2). The reviewer may have missed this discussion in the original review. We ask the reviewer to check again. \n\n* Response to weakness #3 on a direct comparison between compressed model and baseline not being fair due to the consumption of additional data in the continual training process.\n\n* * The reviewer makes an excellent point! Continued training consumes more tokens overall compared to the pre-trained baseline, therefore it is important to understand the impact of the continuous training session and dissociate it from the adaptation to compression. We would like to point out that this is exactly why we chose to retrain the Llama2-7b baseline using the same tokens and training session as those used for the ESPACE models. Indeed, see Table 2 (second row), where we included results for a retrained Llama2-7B model and the motivation for it at line 343 which is specifically to address the concern raised by the reviewer. This retrained baseline now matches the amount of training data as ESPACE, while its accuracy closely matches the original baseline. Therefore, this result allows us to compare ESPACE models to the baselines, either original or retrained ones (since they have matching accuracy).  We kindly ask the reviewer to check this again. \n\nIn conclusion, we first acknowledge that the reviewer has made very good points. At the same time, the concerns raised by the reviewer were in fact already addressed in our original submission. In our revision, we will highlight the above points to be more explicit. Nevertheless, we hope the reviewer will consider increasing their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Te8zijrpqM", "reviewer": "Reviewer_maTu", "summary": "The paper introduces a novel technique for compressing large language models (LLMs) by reducing the dimensionality of activation tensors. The ESPACE method differs from traditional weight-centric compression approaches by focusing on activation tensors instead. ESPACE projects these activations onto a pre-calibrated set of principal components, preserving expressivity during retraining while enabling weight compression at inference through associative matrix multiplication.", "questions": "1. How does ESPACE perform on different types of inference workloads, particularly those requiring higher precision or longer inference times? Additional details on these scenarios would help assess the method’s practical applicability in diverse real-world settings.\n\n2. Could you elaborate on the decision-making process for selecting specific compression settings and projection matrices? A detailed discussion on how to balance compression rate and model performance in various contexts would be highly valuable for practitioners.\n\n3. Have you conducted any tests to evaluate ESPACE’s robustness under unusual conditions or edge cases? Sharing any findings or future plans for such analyses would provide a clearer understanding of the method’s limitations and potential areas for improvement.", "limitations": "1. Include evaluations or discuss plans for testing ESPACE on various model architectures to better understand its generalizability.\n\n2. Discuss potential challenges and solutions for deploying ESPACE in real-world systems, focusing on real-time performance and integration with existing workflows.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach to compressing large language models by focusing on the dimensionality reduction of activation tensors rather than the weights themselves. This method, termed ESPACE (Eigen Static Principal Activation Component Estimation), represents a creative combination of principles from tensor decomposition and matrix multiplication associativity. \n\n2. This paper is highly technical and presents a robust foundation for constructing optimal projection matrices that minimize the mean squared error and forward-propagated noise metrics. \n\n3. The paper is well-written and organized logically. The authors effectively communicate complex ideas, making the novel approach accessible to readers.", "weaknesses": "1. More direct comparisons with other state-of-the-art activation-based and tensor decomposition methods could provide a clearer benchmark of ESPACE's effectiveness.\n\n2. The paper could benefit from a more thorough discussion of the trade-offs in selecting compression settings and projection matrices for different application scenarios.\n\n3. Including a robust error analysis would help identify potential weaknesses or failure modes of ESPACE, enhancing understanding of its limitations.\n\n4. The study focuses primarily on GPT3 and Llama models, potentially limiting the generalizability of the results to other architectures.\n\n5. The paper does not extensively explore ESPACE's performance across various inference tasks that differ in length or complexity, which could provide a deeper understanding of its practical implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback provided, and for recommending acceptance of our paper. We also appreciate the positive comments provided with respect to the novelty, technicality, and clarity of our work. Here we provide answers to the concerns and questions raised:\n* Response to Weakness #1 on a lack of comparison to SOTA tensor decomposition methods .\n* * We agree with the reviewer that a comparison with other SOTA methods is required to establish a benchmark of ESPACE’s effectiveness. In our submission, we did include this comparison in Figure 4 and section 4.4 (Page 9). Indeed, we compared our Llama2-7b results to those of ASVD, SVD-Lora, and SliceGPT; all contemporary tensor decomposition compression works who have reported perplexity vs compression results for that model. This comparison showed that ESPACE noticeably advances the SOTA. Indeed, while lossless compression cannot be achieved beyond 5% compression (achieved by ASVD) for other methods, ESPACE maintains accuracy with 20% compression. Similarly, the 0.5 perplexity increase at 50% compression with ESPACE matches SliceGPT compression of 10%. Therefore, ESPACE generally advances tensor decomposition compressibility by a factor of 4-to-5x when matching other methods’ accuracy. It seems that this part of our discussion was missed in your original review. We apologize for not highlighting these comparisons more in our manuscript, this will be rectified in the revision.   \n* Response to Weaknesses #2 and #3 on application-specificness and robustness analyses.\n* * We appreciate the feedback on discussing application-specific compression and robustness analyses. These were beyond the scope of our work but we will add discussions for these topics in our revised manuscript, and specify directions for future work in these scopes. \n* Response to Weakness #4 on the use of GPT3 and Llama2 and the impact on generalizability.\n* * The reviewer is correct in highlighting the importance of generalizability of results. GPT3 and Llama are contemporary open-source and popular LLM architectures used broadly in various applications. This is why we elected to cover them in our paper. Furthermore, by virtue of having several model instances in these families, we did experiment on five models overall (GPT3-1.3B, GPT3-8B, GPT3-22B, Llama2-7B, Llama2-13B). Thus, our experiments do cover a wide range of model sizes. Nevertheless, we agree that it is desirable to cover more model architectures, and this is absolutely in our plan for future work, which we plan to discuss in the revision. \n* Response to Weakness #5 on the variety of tasks studied.\n* * We have evaluated ESPACE on a wide range on inference tasks, including several LM evaluation harness benchmarks (see Tables 1 and 2), as well as the complex MMLU benchmark (which we added as part of this rebuttal in response to Reviewer yYTB above). But the reviewer is correct that we have not studied specifically the issue of varying context length. This is a good direction for future work, which will be mentioned in our revision. Thank you! \n* Response to Question #1 on the impact of high precision and long inference times.\n* * This is a great question! With regards to precision, and as mentioned in our paper, we have used BF16 throughout our experiments and never had any numerical issues. Having said that, because our compression technique is algebraic, rather than numeric (e.g., such as works doing quantization and/or pruning), we do not anticipate issues should higher precision be required. With regards to workloads requiring long inference times, such scenarios are specifically instances where ESPACE’s benefits can be leveraged. Indeed, as shown in our paper, ESPACE leads to a reduction in 35-to-45% in GEMM latency. Thus, if GEMM latency is reduced, inference can run faster. In fact, we have recently generated end-to-end inference latency measurements to back this up. Please see our response to reviewer 1FAL on measured time-to-first-token: ESPACE helps significantly speed-up inference.\n* Response to Question #2 on the decision-making process for selecting specific compression settings and projection matrices\n* * We have used validation studies as described in Section 4.2 and further in-depth elaborated on in Appendix B. For the interested practitioner, we also summarize our strategy for these selections here. By using a validation set (which is mutually exclusive to training and test sets), we perform layer-wise studies where we analyzed various compression setting and projection matrix choice at each layer independently. We rank the accuracy profile of various candidate settings for each layer, and subsequently select the best choice to be further experimented on. Specifically, these become the compressed model configurations used for ESPACE in the experimental sections 4.3 and 4.4 in our paper.\n* Response to Question #3 on tests to evaluate ESPACE’s robustness under unusual conditions or edge cases\n* * We have not conducted such studies. We kindly ask the reviewer to provide examples of what qualifies as unusual conditions and edge cases. We will study those as future work and include a relevant discussion in the revision.\n\nOnce again, we thank the reviewer for the encouraging comments and productive feedback! We hope our responses above were satisfactory to the reviewer. It would be much appreciated if the reviewer would consider increasing their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2K9nA3bqld", "reviewer": "Reviewer_ndkj", "summary": "The paper introduces ESPACE (Eigen Static Principal Activation Component Estimation), a technique for compressing large language models (LLMs) by focusing on the dimensionality reduction of activation tensors rather than the traditional weight-centric tensor decomposition. The method involves projecting activation tensors onto a pre-calibrated set of principal components, which reduces the dimensionality of activations and results in weight compression during inference through matrix multiplication associativity.", "questions": "How does the calibration set impact the final performance?\n\nCan you provide a sensitivity profile vs layers in the Transformer model? I.e., how would the local ranks of projection in individual layers affect the global performance? (with the belief that a deep compression in early layers can hamper the output even higher ranks are used for later layers.) A portfolio of layers’ importance would be crucial for the compression strategy.\n\nHow does the projection influence the well-known outlier issue in activations and sometimes in the key values, which is observed in lots of LLM works?", "limitations": "The calibration set plays a crucial role in the quality of the principal components for projection, this needs to be analysed deeply to warrant good results.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces ESPACE for compressing LLMs through activation-centric dimensionality reduction. This differs from traditional weight-centric tensor decomposition techniques, which adds novelty to the research.\n\nThe authors provide a solid theoretical foundation for their approach, including the derivation of optimal constructions for projection matrices to minimize mean squared error and forward propagated noise metrics.\n\nESPACE can achieve up to 50% compression of models with a small increase in perplexity and a practical reduction in inference (GEMM) latency.", "weaknesses": "To me ESPACE is simply PCA projection (with a reinvented name) of activation tensor. Can the authors defend a bit?\n\nThe paper suggests that ESPACE can be used in conjunction with other compression techniques like pruning and quantization, as well as non-compressive acceleration methods such as speculative decoding. However, it does not provide empirical results for these combinations, which makes the claim unconvincing.\n\nAlthough the authors claim that ESPACE is orthogonal to other compression techniques, it would strengthen the paper to include comparisons with more baselines, such as pruning, to better demonstrate the significance of the proposed method. E.g., when compressed respectively & individually with ESPACE, pruning and quantization to the same model size, how do their perplexities compare to each other? You can demonstrate that on small models like llama-58M or GPT2-97MB.\n\nThe necessity for a calibration phase that involves forward-passing multiple batches of data to estimate activation auto-correlation matrices might not scale well with larger datasets or more complex models, potentially limiting the method’s applicability in diverse settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback provided, and for recommending acceptance of our paper. We also appreciate the positive comments highlighting the novelty of our approach, its solid theoretical foundation, and the promising empirical results. Here we provide answers to the concerns and questions raised:\n* Response to Weakness #1 on the similarity to PCA.\n* * We are happy to defend: we were indeed inspired by this classic algorithm and in fact did provide a comparison to PCA (lines 200-204). PCA’s goal is to extract low dimensional features having maximum correlation with input data. In contrast, ESPACE is derived such that the compression in eq. (3) is achieved while minimizing various accuracy metrics, which were theoretically derived in Section 3. Crucially, ESPACE leverages ergodicity of activation autocorrelation (see Section 3.1) to produce static projections enabling compression as described in eq (3) and Figure 2. This would not be feasible using PCA, as extracting low dimensional features of activations on the fly would require an online algorithm, which is expensive. In the revised manuscript we will further defend these differences between ESPACE and PCA. We thank the reviewer for bringing this up.\n* Response to Weaknesses #2 and #3 on comparisons between ESPACE and other compression techniques such as quantization or pruning.\n* * We have mentioned that ESPACE can be applied with other techniques because its implementation is orthogonal to others. For instance, ESPACE can be implemented using a quantized or sparse number format. However, while that is the case, we did mention compression fundamentally makes models more susceptible to noise and that a detailed study of combining ESPACE with other techniques was left for future work (Section 1.1; line 33-35). This claim of orthogonality will be toned down in our updated manuscript. With that said, we appreciate the suggestion to include studies with other compression techniques. As the reviewer appreciates, it requires time and efforts to generate such results. But for this rebuttal, we have studied the following: ESPACE and FP8 quantization. While sub-8-bit quantization has been explored, we chose FP8 quantization because it is popular among practitioners thanks to its versatility: it usually does not require QAT and is available in NVIDIA Hopper GPUs. Let us compare the following: baseline BF16, ESPACE BF16, baseline FP8, and ESPACE FP8. We use Hopper FP8 (E4M3) and employ per-tensor dynamic max scaling. Both weights and activations are quantized to FP8, and for ESPACE, additional tensors in eq (3) are quantized to FP8, i.e., projected activation, projection matrix, and precomputed weight-projection product. For brevity, we employ GPT3-8B and Llama2-7B as representative models. Complete results with all other models studied will be included in the revision. We evaluate Wikitext test perplexity; our results are as follows (these results are also included as Table B in the attachment to the common rebuttal):\n\n|Model (Compression)    |BF16 |FP8 |\n|-|-|-|\n|Baseline GPT3-8B       |7.38 |7.41|\n|ESPACE GPT3-8B (21%)   |7.00 |7.22|\n|ESPACE GPT3-8B (50%)   |7.66 |7.85|\n|-|-|-|\n|Baseline Llama2-7B     |5.06 |5.08|\n|ESPACE Llama2-7B (21%) |5.07 |5.13|\n|ESPACE Llama2-7B (50%) |5.67 |5.80|\n|-|-|-|\n* * It turns out that with ESPACE, susceptibility to quantization noise is slightly worse, where perplexity increases by ~0.1-0.2. This is reasonable since a compressed model is expected to be less robust to quantization noise (our prediction from lines 33-35). We thank the reviewer for the suggestion to add such results making our work stronger. We concede that ESPACE does makes quantization a bit harder, although we emphasize that the above are simple PTQ tests which could be improved with further optimizations such as SmoothQuant, GPTQ, or even QAT. \n* Response to Weakness #4, Question #2, and Limitation #1 on the impact of calibration set choice.\n* * We have found that the performance was not sensitive to the choice of calibration set. Specifically, we use 512 random sequences for calibration but found that consistent results were obtained when using smaller sets, e.g., 32 sequences (experiments on varying calibration set size were done for the GPT3-1.3B model). Thus, we believe calibration to be robust to the choice of calibration dataset. \n* Response to Question #2 on layer-wise sensitivity profiles.\n* * Layer-wise sensitivities to projections are included in detail in Appendix C, where the exact question is being answered for each layer of each model studied. The portfolio of layers’ importance was indeed utilized when selecting the compression configuration setting (see Section 4). To meet the page limit, we elected to keep the sensitivity study in Appendix C but did refer to it in the main text in Section 4.2.  We ask the reviewer to check that these parts of the submission did in fact provide answers to the reviewer’s question.\n* Response to Question #3 on the outlier issue.\n* * Since ESPACE is an algebraic compression technique, as opposed to numerical ones such as quantization and pruning, we have not had any issue with the presence of outliers. Also recall that our experiments use BF16 precision which could be shielding ESPACE from the outlier issue. Nevertheless, we are encouraged that ESPACE has circumvented the outlier issue, but we will need further studies to make definite claims, particularly when using low-precision quantization. So, this is a good direction for future work which blends naturally with our plans. Thank you! \n\nOnce again, we thank the reviewer for the encouraging comments and productive feedback! We hope our responses above were satisfactory to the reviewer. It would be much appreciated if the reviewer would consider increasing their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atAaaUWyNy", "reviewer": "Reviewer_1FAL", "summary": "The proposed method applies a PCA-inspired method to the activations of LLMs for model compression.", "questions": "- The paper could be made much stronger if you could show some practical benefits in terms of, e.g., inference time, VRAM requirement, or storage requirements. Right now you show compression ratios, but I believe more concrete numbers can make the method much more appealing.", "limitations": "The limitations are well addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- While most papers focus on quantization, pruning, or weight decomposition, the proposed approach goes in an interesting direction.\n- The paper is easy to follow and the proposed method is simple.\n- The studied problem is relevant.\n- The limitations of the method are described.\n- The experiments contain multiple ablations to verify the effectiveness of the method.", "weaknesses": "- Compression is never well-defined. Is it compression ratio w.r.t. W? What if you do not compress all weights of the model (line 112)?\n- Comparison with related baselines is missing. I understand your work is orthogonal to pruning and quantization methods, but it would make sense to compare with methods that use weight decomposition such as ASVD or the ones you mention in the paragraph starting on Line 41.\n- The practical benefit of the method can be better emphasized.\n- The quality of the figures can be improved, e.g., the text is grainy and Fig. 2 is too small to read.\n- Line 192, i==j -> i=j", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback provided, and for recommending acceptance of our paper. We also appreciate the reviewer underscoring the novelty, clarity, relevance, technicality, and substantiveness of our work. Here we provide answers to the concerns and questions raised:\n* Response to Weakness #1 on the definition of compression ratio.\n* * We apologize for the confusion. We had indeed properly defined compression ratio in the paragraph starting on line 276, but did not provide an explicit formula. The sizes of matrices $\\mathbf{W}$, $\\mathbf{P}$, and $\\mathbf{P}^T\\mathbf{W}$ were explicitly stated to be $K\\times N$, $L\\times K$, and $L\\times N$, respectively, at lines 161-166. We should have repeated those when introducing compression ratio at line 276 for better clarity, and this will be rectified in the revision. We also confirm that this compression refers to model size reduction and therefore does not consider layers involving cross-multiplications of activations (line 112). On lines 284-286, we had mentioned that it was beyond the scope of our work to evaluate the impact of ESPACE when taking into account non-GEMM layers. However, we’ve supplemented our results to provide preliminary results on those. Indeed, in the response below, we’ve added time-to-first-token measurements. \n* Response to Weakness #2 on a lack of comparison to SOTA tensor decomposition methods.\n* * We agree with the reviewer that a comparison with other SOTA methods is required to establish a benchmark of ESPACE’s effectiveness. In our submission, we did include this comparison in Figure 4 and section 4.4 (Page 9). Indeed, we compared our Llama2-7b results to those of ASVD, SVD-Lora, and SliceGPT; all contemporary tensor decomposition compression works who have reported perplexity vs compression results for that model. This comparison showed that ESPACE noticeably advances the SOTA. Indeed, while lossless compression cannot be achieved beyond 5% compression (achieved by ASVD) for other methods, ESPACE maintains accuracy with 20% compression. Similarly, the 0.5 perplexity increase at 50% compression with ESPACE matches SliceGPT compression of 10%. Therefore, ESPACE generally advances tensor decomposition compressibility by a factor of 4-to-5x when matching other methods’ accuracy. It seems that this part of our discussion was missed in your original review. We apologize for not highlighting these comparisons more in our manuscript, this will be rectified in the revision. \n* Response to Weaknesses #4 and #5. Thanks! The figures will be made bigger, and the equal sign will be fixed.\n* Response to Weakness #3 and Question #1 on metricizing practicality benefits.\n* * The reviewer is absolutely right about the importance of highlighting practical benefits. We first mention that in the submitted paper, we had included measurements on the speed-up in the matrix multiplication layers. These are included in Tables 1 and 2, and we refer the reviewer to check again if those were missed in the initial review. The findings were that with ~50% compression, ESPACE reduces the latency in GEMM layers by 35%-to-45%. In this rebuttal, per the reviewer’s suggestion, we’ve supplemented these results and measured the time-to-first-token (TTFT) for all models. The experimental setup is similar to that of GEMM latency measurements in Section 4, where we use NVIDIA A100 GPUs, a batch size of 1, and no tensor parallelism. These new results are included below (these results are also included as Table C in the attachment to the common rebuttal):\n\n|Model (Compression)    |Total GEMM Latency (From submission) | TTFT (New result) |\n|-|-|-|\n|Baseline GPT3-1.3B     |24.2ms            |39.8ms       |\n|ESPACE GPT3-1.3B (20%) |20.6ms (-15%)     |36.1ms (-9%) |\n|ESPACE GPT3-1.3B (47%) |15.9ms (-34%)     |31.7ms (-20%)|\n|-|-|-|\n|Baseline GPT3-8B       |136ms             |186ms        |\n|ESPACE GPT3-8B (21%)   |110ms (-19%)      |155ms (-16%) |\n|ESPACE GPT3-8B (50%)   |76.8ms (-44%)     |122ms (-35%) |\n|-|-|-|\n|Baseline GPT3-22B      |354ms             |457ms        |\n|ESPACE GPT3-22B (40%)  |229ms (-35%)      |313ms (-31%) |\n|ESPACE GPT3-22B (55%)  |181ms (-49%)      |261ms (-32%) |\n|-|-|-|\n|Baseline Llama2-7B     |210ms             |368ms        |\n|ESPACE Llama2-7B (21%) |169ms (-19%)      |322ms (-12%) |\n|ESPACE Llama2-7B (50%) |113ms (-46%)      |266ms (-28%) |\n|-|-|-|\n|Baseline Llama2-13B    |406ms             |643ms        |\n|ESPACE Llama2-13B (20%)|336ms (-17%)      |561ms (-13%) |\n|ESPACE Llama2-13B (50%)|259ms (-36%)      |447ms (-31%) |\n|-|-|-|\n* * As we can see, the speed-up in GEMM latency does lead to TTFT reduction. However, since some time is spent in non-GEMM layers (e.g., cross-multiplication of activations in attention), the relative speed-up compared to the baseline is slightly lower for ESPACE. The improvement in TTFT is still significant, and 50% compression with ESPACE leads to 20%-to-35% TTFT reduction. This shows that ESPACE practically improves inference time. In addition, we would like to note that storage requirements are also encapsulated by the previously reported model size reduction achieved via compression. In terms of VRAM requirements, since smaller matrices are loaded to memory, we expect some VRAM requirement reduction. However, a thorough architectural study is needed to quantify accurately what benefits ESPACE can lead. We defer this for future work, but we will include a summary of all of the above, as well as the new TTFT results in the revision. We thank the reviewer, as the above significantly strengthens our work. \n\nOnce again, we thank the reviewer for the encouraging comments and productive feedback! We hope our responses above were satisfactory to the reviewer. It would be much appreciated if the reviewer would consider increasing their score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, 
We would like to thank you for the useful feedback to our work. We’ve addressed every concern and question raised in the individual responses. In this common response we would like to emphasize a few points.

First, it seems that some parts of the paper were missed by some reviewers. We wanted to emphasize the presence of the following information in our original submission:

* Evaluations on a diverse set of downstream tasks from the LM evaluation harness (BoolQ, Hellaswag, Piqa, Race, Winogrande – see Tables 1 and 2). In fact, reviewer 1FAL explicitly states our “multiple ablations” as a strength of our work. However, Reviewer yYTB appeared to have missed those and mentioned we only evaluated our method on the Wikitext dataset.

* Information about training time can be found in Section 4 of our submission. Reviewer yYTB inquired about that.

* Results with a retrained Llama2 baseline (see Section 4.4. and Table 2) were included to address the issue of comparing a continuously trained model to a baseline having consumed the same amount of training data. Reviewer yYTB raised this important point but appears to have missed our inclusion of this result in our submission.

* Comparison to SOTA tensor decomposition works   is present in Section 4.4 and Figure 4. This was inquired by Reviewers maTu and 1FAL.

* Studies on layer-wise sensitivities are discussed in-depth in Appendix C and referenced in Section 4.2. This was inquired by Reviewer ndkj.
* A study on inference time proxied by GEMM latency measurements is included in Section 4 and Tables 1 and 2. This was inquired by Reviewer 1FAL.

More detailed responses to the above are provided in the individual rebuttals. We want to emphasize this in case any reviewer had provided lower scores due to missing any of the above information while reading our submission. We hope that the above clarifications can rectify that. In addition, in the individual rebuttals, we provide answers to all other concerns and questions raised by the reviewers.
We also wish to point out that, as part of the rebuttal, and thanks to some excellent points made by the reviewers, we have added three sets of results as follows (these results are included in the attached extra document and as inline tables in the individual responses to make the reading of rebuttals smoother):

* Additional results evaluating all models on the MMLU benchmark. For more details, please refer to the rebuttal to Reviewer yYTB and Table A in the attached document.

* Additional results combining ESPACE with FP8 quantization. For brevity, in the rebuttal we only included such results on GPT3-8B and Llama2-7B as representatives to all models studied. For more details, please refer to the response to Reviewer ndkj and Table B in the attached document.

* Additional latency measurements on the time-to-first-token for all models to supplement the GEMM latency measurements that were present in the original submission. For more details, please refer to the rebuttal to Reviewer 1FAL and Table C in the attached document.

The above set of new results significantly strengthens our work and will be included in the revision. We thank the reviewers again for the feedback.

Meta-review: Although barely above borderline, all reviewers supported acceptance, particularly after the authors' extensive responses clarified or improved some issues. So I'm happy to recommend acceptance.

The idea of compressing the activations by multiplying and reprojecting times a fixed matrix seems very similar to the idea of matrix sketches, and perhaps that is essentially what is going on here. Can the authors comment on this in the final version of the paper?
Decision: Accept (poster)