ESPACE: Dimensionality Reduction of Activations for Model Compression

Authors: charbel  sakr
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 22:32:00

Abstract: We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. Unlike prior works on weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of principal components. The activation-centrality of the approach enables retraining LLMs with no loss of expressivity; while at inference, weight decomposition is obtained as a byproduct of matrix multiplication associativity. Theoretical results on the construction of projection matrices with optimal computational accuracy are provided. Experimentally, we find ESPACE enables 50% compression of GPT3, Llama2, and Nemotron4 models with small accuracy degradation, as low as a 0.18 perplexity increase on GPT3-22B. At lower compression rates of 20% to 40%, ESPACE drives GPT3 models to outperforming their baseline, by up to a 0.38 decrease in perplexity for GPT3-8B. ESPACE also reduces GEMM execution time and prefill inference latency on existing hardware. Comparison with related works on compressing Llama2-7B via matrix factorization shows that ESPACE is a first step in advancing the state-of-the-art in tensor decomposition compression of LLMs.

Main paper: 
 introduction: 
 Capabilities of large language models (LLMs) have recently soared in natural language understanding and generative power. It is appreciated that there exists a correlation between model size and achievable accuracy. Indeed, as LLMs consume trillions of tokens during their training, a large parameter volume is required to capture intricate linguistic features [1] . This leads to a trade-off in LLMs: larger parameter counts improve accuracy but come with increased serving cost. However, it is also appreciated that the computational requirements of inference may be lower than those of training [2] . To that end, numerous studies have investigated compression of LLMs to reduce inference cost. The most popular LLM compression techniques are quantization [3] and pruning [4] . A less explored, but powerful technique is tensor decomposition, and in our work, we propose a novel, activation-centric way to decompose LLM tensors. Our proposal is to project activations onto a static set of components optimizing fidelity. The projection reduces activation dimensionality and leads to weight compression at inference as a byproduct of matrix multiplication associativity.Recent research has proposed many quantization and pruning techniques for compressing LLMs. Examples of advances in LLM quantization include SmoothQuant [3] , AWQ [5] , and GPTQ [6] ; while notable LLM pruning works include SparseGPT [4] , LLM-Pruner [7] , and ReLU-based masking [8] . These methods are conceptually orthogonal to our proposal for activation projection which can be implemented in low precision or sparse formats. Nevertheless, compression fundamentally introduces noise, and an open problem is to study the impact of combining different methods, e.g., quantization and matrix factorization. This is beyond the scope of our paper, but a good direction for future work.Our work is also orthogonal to non-compressive LLM serving acceleration such as continuous batching [9] or speculative decoding [10] , and attention optimizations such as PagedAttention [11] , RadixAttention [12] , and FlashAttention [13] . Our study is on matrix multiplication layers involving weights and activations, and hence is mutually exclusive to works improving cross multiplications of activation tensors in attention. In fact, all our experiments use FlashAttention.Finally, we turn to tensor decomposition, also known as factorization. Thus far, compression for LLM inference using factorization has been focused on weight decomposition. KnGPT [14] uses the Kronecker transform to pack a large matrix into two smaller ones. TSVD [15] performs iterative singular value decomposition (SVD) on weight matrices to produce high rank ternary components. TensorGPT [16] and HEAT [17] compress weight matrices into a cascade product of small matrices using the tensor-train algorithm. SVD-LoRa [18] uses a truncated SVD on weights and finetunes the model using LoRa [19] . The LoRa adapters are then merged to the main branch using bounds on the rank of sum of low rank matrices. ASVD [20] performs a truncated SVD on the weights after re-scaling them by a diagonal matrix and their inverse encapsulating activation statistics. This work realizes the importance of activation-awareness but still uses weight-centric compression. SliceGPT [21] extracts principal components in normalization layers to guide the deletion of rows and columns in weight matrices. The compression is achieved using a factorization made implicit via computational invariance. The statistical method employed by sliceGPT shares similarities with one of our results, but our problem formulation and solution are different. Factorization can also streamline LLM training and finetuning. For instance, LoRa [19] finetunes pretrained models using residual low rank adapters which are then absorbed into the main branch. Similarly, GaLore [22] applies a low rank approximation to gradients in back-propagation. These works do not modify inference parameter and operation count, and are hence orthogonal to ours. Our method could be applied in tandem with LoRa or GaLore, but this is beyond the scope of this paper.Since factorization increases the number of LLM tensors, achieving high compression rates requires the intermediate dimensions to be much smaller than that of original dot product. This breakage in computation usually necessitates a retraining or finetuning stage to be healed. Unfortunately, this healing process is impeded because factorized LLMs have fewer learnable parameters which decreases expressivity [17, 22] .To our knowledge, no prior art has explored activation decomposition. Indeed, applying factorization solvers (e.g., SVD) dynamically incurs large inference runtime overheads. Yet, activation decomposition has several desired features which we examine in Section 2 and motivate via the following insights: (a) weights stay uncompressed during retraining, preventing the aforementioned loss of expressivity; (b) large activation tensors contain inherent redundancies making them prime candidates for compression; and (c) since most LLM computation comprises multiplications of weights and activations, decomposing the latter can lead to compressing the former at inference.We propose Eigen Static Principal Activation Component Estimation (ESPACE), an LLM compression technique based on activation dimensionality reduction. Our contributions are as follows:• We project activation tensors onto a static and pre-calibrated orthonormal matrix. The projection lowers activation dimensionality but keeps weight matrices intact and fully available for training.At inference, leveraging matrix multiplication associativity, model compression is achieved through pre-computation of the product of weight and projection matrices. • We theoretically derive optimal constructions for activation dimensionality reduction. Specifically, the projection matrix is calibrated in a manner to minimize activation decomposition mean squared error and forward propagated noise metrics. The solution is based on an eigenvalue decomposition of activation auto-correlation and yields multiple candidate projections for each activaton tensor. • We empirically study compression of models in the GPT3, Llama2, and Nemotron4 families evaluated on the Wikitext-103 dataset for perplexity and the LM evaluation harness for downstream task accuracy. The amelioration in size versus perplexity2 trade-offs is summarized in Figure 1 . • We show that ESPACE can compress LLMs by ∼50% at the cost of a small accuracy loss, as low as 0.18 increase in perplexity on GPT3-22B. • At lower compression rates, we find encouraging empirical evidence that ESPACE filters out noise and improves accuracy; e.g., ∼20% compressed GPT3-8B lowers its baseline perplexity by 0.38. • As an additional benefit of ESPACE, tangible latency reduction of 35%-to-45% is obtained in matrix multiplication layers. This speed-up translates to up to ∼ 40% faster prefill inference latency metricized by the time to first token and measured on existing hardware. • By comparison to existing works on tensor decomposition, we determine that ESPACE is a first step in pushing the frontier of compression rate versus accuracy retention (see Figure 4 ).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We accurately report the paper's contribution in the abstract and introduction; this is why we chose to have a bulleted list in a standalone subsection in Section 1.2. For clarity, we listed all contributions in the paper, in the order in which they appear: our proposal, followed by theoretical results, and summaries of experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
dimensionality reduction & projections: 
 In this section, we introduce notation for matrix multiplication, review weight decomposition, and introduce our proposed mechanism of dimensionality reduction via activation projections.We consider general matrix multiplications (GEMMs) described in Figure 2 (a) of the formEQUATION)where W is a weight matrix of size K × N and X is an input activation tensor of size K × M so that the output activation tensor Y is of size N × M . Typically, K and N are defined by network topology and layer instance, they are commonly referred to as embedding or hidden size. In contrast, M stacks multiple dimensions in an activation tensors to obtain a 2D matrix view. Generally, these are the sequence and batch dimensions.Transformer-based LLMs have four GEMM layers per block: query-key-value (QKV), projection (Proj), fully connected 1 (FC1), and fully connected 2 (FC2) layers. Our study is concerned with these layers, while cross activation multiplication and embedding layers are untouched. For notational simplicity, in this paper, we do not include layer indices in our equations.The matrix W in (1) stores layer parameters and dictates the model's inference accuracy. To improve convergence of these parameters, an optimizer state is stored alongside weights during training and tracks historical values of gradients and updates [23, 24] . On the other hand, the activation tensor X depends on the input stimulus to the network, and is therefore generated on the fly.Thus, at inference, weights are fixed but activations are dynamic. As a consequence, prior work on tensor decomposition has focused on compressing frozen weight matrices. One way of doing so is breaking W T into a low-rank approximation using some form of truncated SVD [20, 18] , which is described in Figure 2 (b). Specifically, (1) is approximated as:EQUATIONwhere U and V are matrices of size N ×L and L×K, respectively, with L being the factorization rank.For the decomposition procedure to be useful, two conditions need to be met: (a) L << min(K, N ) for compression, and (b) the approximation W T ≈ UV should be accurate. However, achieving both conditions simultaneously may be challenging because a very low rank factorization usually leads to significant accuracy drop [22] . As with other compression techniques, e.g., quantization and pruning, retraining of the compressed model may be employed to recover accuracy. However, the decomposition in (2) introduces two training-related hurdles: (a) the effective number of trainable parameters has decreased significantly which reduces model expressivity, and (b) the breakage of spatial weight structure prevents the retraining procedure from loading the original optimizer state. Retraining a model without its optimizer state is known to introduce significant difficulty in convergence [17] .Since weight decomposition poses the above hurdles, we motivate the need for an activation-centric solution. In some measure, activation compression may be more achievable due to the large stack dimension M comprising batches and sequences. Statistically, the Central Limit Theorem claims that stacking data is likely to exhibit redundancies [25] . In the case of LLMs, such redundancies are further pronounced due to the likelihood of repeated tokens and information in natural language.Therefore, activations should be prime candidates for tensor decomposition. Nevertheless, prior arts have not explored activation decomposition due to one fundamental limitation: unlike weights, activations are generated on the fly; meaning that tensors must be compressed during inference, potentially incurring large runtime penalties.We propose to apply static dimensionality reduction on the activation tensor X in (1). Concretely, our proposal is to project X onto a pre-computed static orthonormal matrix P of size K × L, where crucially L << K. Reconstructing X requires a re-expansion using the transpose of the projection matrix, i.e., X ≈ PP T X. While P T P = I L×L , we note that PP T ̸ = I K×K since L << K. Thus, the proposed activation transformation is noisy, and in Section 3, we derive optimal conditions on the calibration of P to minimize the effects of this noise.Our proposal, described in Figure 2(c ) is to approximate the GEMM in (1) using the following:EQUATIONwhere we used associativity of matrix multiplication to highlight key aspects of our approach.During training/finetuning: we view our GEMM as W T PP T X where X has been replaced by its approximation. We emphasize that P is static and does not get updated during training.Meanwhile, W T is fully available for adaptation to the activation approximation. The availability of all learnable weights elides losing model expressivity. The structure of W T is also unchanged and can be mapped to the baseline's optimizer state. Thus, the proposed approach does not suffer from the same limitations as weight decomposition techniques. We do note that introducing P induces a small storage overhead at train time. However, when L << K, and the order of computation is properly compiled, the number of operations per iteration is lower than baseline training; and, though not central to our contribution, we did observe up to 15% reduction in training iteration time for 50% compressed models.During inference: we view our GEMM as P T W T P T X where the required matrices are P of size K × L and P T W of size N × L, which is pre-computed before deployment. Thus, per-layer parameter count required for inference has decreased from KN to L(K + N ), which, provided L << {K, N } presents an opportunity for significant model compression. For instance, if N = K, i.e., W T is square, and L = K /4, then our method yields 50% compression at inference time. This is one of the compression rates we target in Section 4.We emphasize that P is not shared across GEMM layers; rather, each GEMM layer decomposed according to (3) has its own pre-calibrated matrix P. Furthermore, by virtue of (3) not introducing dependencies across mini-batches, our method is fully compatible with data parallelism.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In our discussion on related works in Section 1.1, we have listed possible combinations of applying ESPACE alongside other methods (e.g., quantization or parameter efficient tuning), and have mentioned that this was not the scope of our paper. Furthermore, in the experimental Section 4, we have discussed the limitations of only metricizing compression via model size and weight times activation GEMM latency reductions. We argued that a more nuanced study is needed on the inference cost implications as Transformers comprise other GEMMs, and the regime (context pre-fill versus auto-regressive phase) needs to be taken into account. We have mentioned that a detailed study on inference cost with ESPACE is part of future work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
eigen static principal activation component estimation: 
 Our proposed activation decomposition induces an approximation error as X ̸ = PP T X. In this section, we first introduce an ergodic estimation of activation auto-correlation. This important statistic is then used for theoretical constructions of P with guarantees on computational accuracy. Multiple results are presented and then combined in our compression studies in Section 4.Let x be an arbitrary K-dimensional vector in X; we define the activation auto-correlation matrix of size K × K as C X = E xx T where expectation is taken over activation vectors. This matrix is symmetric positive semi-definite having a real eigenvalue decomposition (EVD) C X = VDV T where V is an orthonormal matrix whose columns are eigenvectors, and D is a diagonal matrix containing the corresponding non-negative eigenvalues, assumed to be sorted in decreasing order. The eigenvector corresponding to the i th largest eigenvalue is called i th principal eigenvector.This autocorrelation matrix can be empirically estimated using an instance of the activation tensor:EQUATION) However, evaluating (4) and its EVD dynamically introduces a prohibitive computational overhead. Thus, we estimate C X in a pre-deployment calibration process. Specifically, during calibration, we sample and forward pass B random input batches, and for each, calculate C (i) X = X (i) X (i) T /M, where superscript i denotes batch index. Then, we average our estimate of the auto-correlation matrix asC X = B i=1 C (i)X /B and use its eigenvalue decomposition for further optimizations. This ergodic approach of estimating activation statistics as part of a calibration process has been employed to great effect in other compression works on quantization [26, 27] and pruning [28] .Let us write X = PP T X; for a vector x ∈ X, its counterpart in x ∈ X is given by:EQUATIONwhere {p i } L i=1 are the orthonormal column vectors of P, i.e., ⟨p i , p j ⟩ = 1 {i==j} , ∀i, j ∈ 1 . . . L.We define the mean squared error (MSE) of the decomposition asEQUATION) with the L 2 -norm used throughout this paper. Our first result constructs P minimizing this MSE.Theorem 1. For an activation tensor X whose auto-correlation matrix has an eigenvalue decomposition given by C X = VDV T , the projection matrix P minimizing the mean squared error in (6) is given by P = [v 1 | . . . |v L ] where v i is the i th principal eigenvector in V.Proof. See Appendix A.1. The result is readily obtained by substituting x in (5) into (6) and minimizing the MSE which involves quadratic forms involving the positive semi-definite C X .Theorem 1 shares similarities with the Principal Component Analysis (PCA) algorithm [29] . PCA extracts low dimensional features having maximum correlation with input data. Unlike PCA, we omit input normalization to elide its computational cost. Still, we term the columns of P in Theorem 1 as Principal Activation Components. Since those are obtained using an EVD on a static estimation of C X , we call our method Eigen Static Principal Activation Component Estimation (ESPACE).The MSE in Theorem 1 is a strong indicator of the quality of an approximation technique, e.g., it is often employed in quantization studies [26, 27] . However, empirical data may contain large outliers which can dominate the optimization process; say a few high-magnitude vectors in (6) masking the contribution of small data on the solution. An alternate metric to the MSE can be employed to prevent such artifacts in averaging: the normalized MSE (NMSE) defined as:E ∥x-x∥ 2 /∥x∥ 2 (7)The solution of Theorem 1 can be slightly modified to minimize the NMSE in (7) .Corollary 2. For an activation tensor X, let ĈX = E ( x /∥x∥) ( x /∥x∥) T be its input-normalized auto-correlation matrix having an eigenvalue decomposition given by ĈX = VDV T , the projection matrix P minimizing the normalized mean squared error in (7) is given by P= [v 1 | . . . |v L ] where v i is the i th principal eigenvector in V.Proof. The proof in Appendix A.2 uses equivalence of NMSE and MSE with L 2 -normalized vectors.Corollary 2 applies to the decomposition in (3) with no activation normalization required at compute time. Rather, normalization is done during calibration, where ĈX is estimated instead of C X .Both solutions in Theorem 1 and Corollary 2 are options to be employed in ESPACE, where either may be more suitable on a layer-wise basis. Next we present further options for ESPACE based on the optimization of alternate metrics to the MSE and NMSE.While local fidelity metrics, such as the MSE and NMSE above, are good indicators of the quality of an approximation technique, it has been shown that better insights on a neural network's accuracy may be derived via the study of forward propagated noise [30, 31, 32] . In this section, we study the effects of the decomposition in (3) on the output of the GEMM, and the output loss of the model. At a given layer, let us write an arbitrary scalar in the GEMM output tensor in (1) as y ∈ Y. Note that y = ⟨w, x⟩ for some weight vector in w ∈ W and activation vector x ∈ X. We also let ỹ be the associated output when the GEMM is approximated by (3) , which is given by ỹ = ⟨w, x⟩ with xgiven by ( 5). We define the GEMM Output-referred MSE (GO-MSE) asE (y -ỹ) 2 .Similarly, given an input to the network, we write the output loss function (the vocab cross-entropy) as L. When one arbitrary activation tensor is transformed as per (3), a mismatch in computation is introduced and propagated all the way to the output. We let L be the resulting new value of the loss function. We define the Network Loss-referred MSE (NL-MSE) asE (L -L) 2 .A closed form solution for P in (3) minimizing the GO-MSE and NL-MSE is elusive to us. Therefore, we derive upper bounds on these metrics which we use as a proxy for optimization. Proposition 3. For a GEMM in (1) and its decomposition in (3), the GO-MSE is upper bounded by:EQUATIONand the NL-MSE is upper bounded byEQUATIONwhere a first order Taylor approximation on the loss function is assumed and its gradient with respect to vector x is denoted as ∇ x .Proof. The proof in Appendix A.3 first shows ∥x∥ 2 < ∥x∥ 2 and then uses the Cauchy Schwarz inequality to establish both bounds.Next, we provide closed form solutions for P in (3) minimizing the bounds in Proposition 3. Theorem 4. For a GEMM in (1) and its decomposition in (3), the projection matrix minimizing the bounds in Proposition 3 is given by P= [v 1 | . . . |v L ]where v i is the i th principal eigenvector in V obtained via eigenvalue decomposition on a matrix C = VDV T defined as:EQUATIONto minimize the upper bounds on GO-MSE in (8) and NL-MSE in (9), respectively.Proof. The proof is included in Appendix A.4, where we also include modifications required in calibration. Specifically, C X is reused and left/right multiplied by WW T /N to yield C in (10) minimizing the bound on GO-MSE. An additional backward pass is needed to properly scale activation vectors and their gradients when calibrating C in (10) minimizing the bound on NL-MSE.Theorem 4 augments Theorem 1 and Corollary 2 with two options for the design of P. Much like Corollary 2, we supplement our new solutions with L 2 -normalization to includeĈ = E (xx T ww T +ww T xx T ) /∥w∥ 2 •∥x∥ 2 and Ĉ = E (xx T ∇x∇ T x +∇x∇ T x xx T ) /∥∇x∥ 2 •∥x∥ 2as alternate choices for the calibrated matrices C in (10) . Unlike Corollary 2, L 2 -normalization in these two matrices does not correspond to a notable optimization. Nevertheless, these options are retained in the spirit of suppressing the influence of large data in calibration.Thus, overall we have six choices for P. Since each can be obtained as part of a fast and predeployment calibration phase, we may simply select the best one for each layer. In our experiments of Section 4, the best candidate is determined via a per-layer validation over all six choices. A sensitivity study on the impact of each of the six candidates is provided in Appendix B.4.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We provide detailed proofs in Appendix A for all results in the theoretical Section 3. Furthermore, following each Theorem, Proposition, and Corollary in Section 3, we also provide a teaser to the full proof in the main text, for the benefit of interested readers, where we try to provide the main intuition before referring to full proofs in Appendix A.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
model compression studies: 
 In this section, we report on experimental studies investigating LLM compression using ESPACE.We employ three sets of open source LLMs: GPT3 [33] , Llama2 [34] , and Nemotron4 [35] . Specifically, we experiment on GPT3-{1.3B, 8B, 22B}, Llama2-{7B, 13B}, and Nemotron4-15B. Accuracy is evaluated in two ways: perplexity measured on the Wikitext-103 dataset [36] and zero-shot downstream task accuracy of: BoolQ (BQ) [37] , Hellaswag (HS) [38] , PIQA (PQ) [39] , RACE (RA) [40] , and WinoGrande (WG) [41] .The Wikitext-103 dataset is split into train, validation, and test sets. We use 512 random sequences from the training set for calibrating projection matrices required by ESPACE. We use the validation set for layer-wise sensitivity studies. The test set is used to report perplexity results in this section.Our implementation uses NVIDIA's Megatron LM [33] and downstream task evaluation invokes Eleuther AI's LM evaluation harness [42] . For the latter, we report raw accuracy scores, and their average; we do not post process results or apply normalization to the scores.When ESPACE is applied, we retrain the models to adapt to the approximation error of activation projection as discussed in Section 2. Retraining simply extends the models' pre-training sessions and uses the 330B-token MTNLG dataset [43] , which was used to train GPT3 models. All implementation details are included in Appendix B to help reproducibility of our results.We metricize model size reduction via inference compression rate. Specifically, for layers decomposed per (3), we count the number of entries in P and P T W ; for other layers, we count those in W T . We also report the latency of executing all network GEMMs in (1) or ( 3), which we measure using a NVIDIA A100 GPU and a simple, un-optimized implementation (see Appendix B.4). We also report prefill inference latency, metricized via the Time to First Token (TTFT), and measured using the Megatron-LM implementation. In our measurements, we use a batch size of 1 and sequence length of 2048 and 4096 for GPT3 and Llama2/Nemotron4 models, respectively. The reported reductions in total GEMM latency and TTFT constitute evidence that compression improves inference throughput.It is beyond the scope of this paper to evaluate the impact of ESPACE on end-to-end token throughput and latency on LLM inference serving systems, since this requires a complex set of optimizations including but not limited to optimization of back-to-back GEMMs into fused kernels, KV caching, continuous batching, as well as thorough performance studies with varying input and output sequence lengths. Thus, we leave an evaluation of token generation throughput and energy savings and improvements to future work.Our experiments start with a calibration phase where we prepare the static projection matrix P for each layer. The dimension L in P is chosen as the lowest power of two such that layer compression is at least 50%. The power-of-two restriction ensures best tensor core utilization, and the resulting compression rate depends on the dimensions of the original layer (K and N ). Exact details of these values for all layers and models are included in Appendix B.2.We perform a sensitivity study on the Wikitext-103 validation perplexity when ESPACE is applied out-of-the-box (no retraining) one layer at a time. For each layer, we identify which of our six candidates projection matrices in Section 3 yields lowest validation perplexity. Layers are then sorted according to their impact on perplexity from least to most destructive. We then evaluate the validation perplexity when ESPACE is progressively applied to out-of-the-box to all layers according to this ranking. Fine-grained details of this exploration are included in Appendix C for all models. This exploration yields an interesting finding: as we progressively apply ESPACE to more layers, the perplexity marginally increases until an inflection point after which accuracy degradation accelerates. This inflection occurs at 20% to 40% compression depending on the model. Figure 3 depicts this phenomenon for GPT3-22B, and the same data for other models can be found in Appendix C.2.We find that out-of-the-box application of ESPACE works better for larger models; GPT3-22B, the largest model we experimented on, exhibits an inflection in perplexity at 40% compression, which is the highest in our results. This is consistent with many earlier works on general compression of neural networks [44, 45, 46, 47] . Interestingly, a 20% out-of-the-box compressed GPT3-22B is iso-accurate to its uncompressed counterpart (see Figure 3 ); without retraining, its test perplexity of 6.61 which is within 1% of the 6.55 baseline.After the above validation study is performed, we select two configurations for layers to be compressed using ESPACE: (a) layers corresponding to the inflection point, i.e., 20% to 40% compression, and (b) as many layers needed to achieve a compression of ∼50%. For both configurations, we retrain the compressed models and further evaluate their achievable accuracy.Once compression targets and layer configurations are set, we retrain GPT3 models on the MTNLG dataset. Although we use all of the 330B available tokens, we do observe the training loss quickly converging. We leave training hyperparameters unchanged except for one: we disable dropout. Our rationale is that activation projection is one form of deterministic and structured dropout such that additional regularization may not be needed. Results3 on GPT3 models are included in Table 1 .We find that ESPACE can compress GPT3 models by ∼50% at the cost of a small accuracy degradation. In the case of GPT3-22B, the perplexity increase is of only 0.18; in general, the gap decreases for larger overall model size. By and large, similar trends are observed for downstream task accuracies and we note that most scores of 50% compressed models fall within 5% of the baseline.For lower compression ratios (inflection points at 20% to 40%), ESPACE converges to an accuracy better than that of the baseline. The best improvement occurs for GPT3-8B, where ESPACE produces a 6B model with 0.38 lower perplexity than its 8B baseline. The improvements are observed both in terms of perplexity and downstream task accuracy. While GPT3 models may be over-parameterized, we posit that ESPACE acts a regularizer at moderate compression rates. Specifically, we believe that projection onto principal activation components filters out unnecessary information coming from small eigenvalue components.For all models, we observe an encouraging translation of compression to GEMM latency reduction by up to 49% which leads to noticeable speed-up in TTFT by up to 43%..For Llama2, we only retrain using 200B MTNLG tokens because we observed quick convergence for GPT3. Llama2 models were trained on an undisclosed dataset of 2T tokens [34] . Therefore, with 200B tokens, the healing phase of ESPACE constitutes no more than 10% of the original pre-training session. Since Llama2 pre-training details are not openly available, we re-used all hyperparameters from GPT3, which is likely to be sub-optimal. In spite of the two handicaps of dataset disparity and hyperparameter sub-optimality, we obtained promising results as reported in Table 1 .For Llama2-7B, we first retrained the uncompressed baseline. The purpose of this experiment is twofold: (a) ensure that our hyperparameters at least do not corrupt the model, and (b) verify that the healing process is not just an artifact of processing more tokens. Both hypotheses appeared to be valid: the retrained baseline has nearly identical accuracy compared to the original model.Generally, we find that the trends of ESPACE compression for Llama2 are similar to GPT3, albeit slightly less successful. Though the results are still promising, we attribute the slight shortcomings in accuracy to the handicaps above. We find that 50% ESPACE compression on Llama2 leads to ∼0.6 perplexity increase and similar degradation in terms of downstream task accuracy. Notably, compressing the Llama2-13B model to to a 6.3B model yields comparable accuracy to the Llama2-7B baseline which itself is a 6.5B model.In addition, for 20% compression, we find that ESPACE matches the accuracy of the baseline for Llama2 models. While not as impressive as the improvements observed with GPT3, ESPACE is able to produce 5B and 10B models matching the 7B and 13B baselines, which does push the pareto frontier of accuracy versus model size in the right direction as shown in Figure 1 . The Llama2-7B model has been used in related works on tensor decomposition mentioned in Section 1.1; specifically, ASVD [20] , SVD-LoRa [18] , and SliceGPT [21] .Both ASVD and sliceGPT have reported perplexity on Wikitext, but SVD-LoRa performed task-specific finetuning on a variety of datasets and averaged perplexities. Therefore, in Figure 4 , we compare our results to these works using perplexity increase over baseline, rather than raw perplexity, for maximum inclusivity.SVD-LoRa performed an SVD decomposition on the weights such that the intermediate dimension is half of dotproduct which leads to no compression. On the other hand, ASVD and sliceGPT can only achieve modest compression ratios of up to 25% with some loss in accuracy. Recall that these works apply factorization on weights which is the fundamental difference to ESPACE. As seen in Figure 4 , ESPACE is a step in the right direction towards improving the state-of-the-art in tensor decomposition of LLMs.Finally, we used ESPACE to compress Nemotron4-15B into 9.54 and 6.25 billion parameters, as reported in Table 1 . Retraining consumed 275B tokens which corresponds to ∼ 3% of this model's original training session. Once more, compression with ESPACE leads to minimal degradation in the moderate regime (25%) and yields a small accuracy drop in the aggressive regime (50%).Consistently with our findings for the above models, ESPACE reduces GEMM execution time by up to 46%. This, in turn, improves the TTFT by up to 26%. An interesting observation is that, for Llama2 and Nemotron4 models, the TTFT improvement is slightly less pronounced than for GPT3 models. This is simply due to the fact that the latter uses a sequence length of 2048, whereas the former two use 4096. A larger sequence length means more time is spent in attention cross-activation products which amortizes the speed-up in the GEMM layers.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: All details, including fine-grained configurations, software toolkit employed, and training recipes including hyperaparameters are included in Appendix B. These are also briefly mentioned in the main paper in Section 4, with references to Appendix B for the readers interested in more details needed for full reproducibility. 
conclusion: 
 We have presented ESPACE, a novel compression technique realizing tensor decomposition of LLMs in an activation-centric manner. A set of theoretical results were derived to guide the construction of activation projection which is done statically. Experimentally, we have shown promising results where ESPACE is able to ∼50% compress modern LLMs at the cost of a small accuracy degradation. Compared to related works, ESPACE is a first step in pushing the frontier of model size versus accuracy trade-offs. Future work includes combining ESPACE with alternate compression techniques such as quantization and pruning, and evaluating decomposition of activation tensors in attention. As potential extension to our algorithm, the use of matrix sketching and random projections may pave the way for better overall compressibility.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: As was mentioned in the previous question, we have provided all implementation details required to reproduce our results, including hyperparameters and a description of software implementation in Appendix B. We note that our implementation is based on the open-source Megatron-LM [33] and that additions made to this software toolkit needed to reproduce our results are included in Appendix B, where we also include an invitation to the reader to reach out to us (after blind reviewing is over) with any questions regarding reproducibility. As such, we believe the description of the work in the paper is sufficient for reproducibility; yet, we are happy to consider open sourcing our code in the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Yes, and this was already mentioned in our responses to the two questions above. We provide all fine-grained details of our implementation in Appendix B, which is referred to in the main text.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: While we do not report error bars, we do evaluate our method on a variety of downstream tasks, for the specific purpose of making conclusions with statistical significance; rather than relying on one number here and there. We also note that this approach of evaluating on several tasks is also widely adopted in the community for works on compression of LLMs.Guidelines: Justification: Yes, we are only working on a method to compress LLMs using tensor decomposition of activations. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: As per the guideline below, our work falls under the umbrella of optimization to the implementation of LLMs. Similar to the example provided in the guideline, we believe that there is no need to point out societal implications of making LLMs run faster.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: In this first appendix, we provide proofs for the various theoretical results in Section 3.For a pair of vectors x ∈ X and x ∈ X, and using (5), we have the squared error:EQUATIONFurthermore, note that∥x∥ 2 = xT x = L i=1 p i T x p i T L i=1 p i T x p iand since {p i } L i=1 are orthonromal, cross product terms vanish and we have:∥x∥ 2 = L i=1 p i T x 2which we plug back into the expression for the squared error:EQUATIONFurthermore, note that:p i T x 2 = p i T x p i T x = p i T x x T p i = p i T xx T p iwhere we used the commutativity of dot product and associativity of matrix multiplication. Thus the squared error is given by:∥x -x∥ 2 = ∥x∥ 2 - L i=1 p i T xx T p iFinally, we take expectation on both sides and obtain a formula for the MSE:E ∥x -x∥ 2 = E ∥x∥ 2 -E L i=1 p i T xx T p i = E ∥x∥ 2 - L i=1 p i T E xx T p iwhere we used linearity of expectation and the fact that {p i } L i=1 are not random. In this formula for the MSE, E ∥x∥ 2 does not depend on {p i } L i=1 , and therefore, minimizing the MSE is equivalent to maximizing the following expression involving the auto-correlation matrix:L i=1 p i T E xx T p i = L i=1 p i T C X p iwhere each term in the summation is a quadratic form on the positive semi-definite auto-correlation matrix C X . Since {p i } L i=1 are orthonormal, this is an equivalent form of the Rayleigh quotient [48] and the solution is to assign {p i } L i=1 as the L principal eigenvectors of C X . This concludes the proof of Theorem 1.The result can be readily obtained as a consequence of the following:E ∥x -x∥ 2 ∥x∥ 2 = E x ∥x∥ - x ∥x∥ 2 = E X   L i=1 ⟨p i , x⟩p i ∥x∥ - x ∥x∥ 2   ⇒ E ∥x -x∥ 2 ∥x∥ 2 = E   L i=1 ⟨p i , x ∥x∥ ⟩p i - x ∥x∥ 2  Therefore, the setup is identical to that of Theorem 1 and we may apply the same solution as Appendix A.1 above. The only difference is that activation vectors are L 2 -normalized which is why ĈX (which is also positive semi-definite) is used in lieu of C X in Corollary 2.A prelimnary result needed is to show that for any activation vector, we have ∥x∥ 2 < ∥x∥ 2 . We first note that while x = L i=1 ⟨p i , x⟩p i per (5), we also have that x = K i=1 ⟨p i , x⟩p i , where {p i } K i=1 extend the set of orthonormal vectors {p i } L i=1 to be complete, i.e., equivalent to no truncation of columns of the full rank matrix V when constructing P, regardless of the metric being optimized.Using orthonormality of projection vectors, similar to the proof of Theorem 1 in Appendix A.1 above, we obtain ∥x∥ 2 = L i=1 p i T x 2 and ∥x∥ 2 = K i=1 p i T x 2 . Therefore:∥x∥ 2 -∥x∥ 2 = K i=L+1 p i T x 2 ≥ 0 ⇒ ∥x∥ 2 < ∥x∥ 2where we used the fact that a sum of non-negative quantities is non-negative.Then for a scalar y ∈ Y and its counterpart ỹ ∈ Ỹ, we have:(y -ỹ) 2 = w T x -w T x 2 = w T x 2 + w T x 2 -2 w T xw T x ≤ ∥w∥ 2 • ∥x∥ 2 + ∥w∥ 2 • ∥x∥ 2 -2 w T xw T x ≤ ∥w∥ 2 • ∥x∥ 2 + ∥w∥ 2 • ∥x∥ 2 -2 w T xw T x = 2∥w∥ 2 • ∥x∥ 2 -2 w T xw T xwhere the first upper bound uses the Cauchy-Schwarz inequality while the second uses ∥x∥ 2 < ∥x∥ 2 which we proved above. Taking expectations on both sides of the inequality yields the upper bound on GO-MSE in (8) .Next, when a first order Taylor approximation on the loss function is assumed, we have the following relation between the unperturbed loss value L and its counterpart L when an activation vector x is projected to x per (5):L = L + ∇ T x (x -x) ⇒ L -L = ∇ T x x -∇ T x x ⇒ L -L 2 = ∇ T x x -∇ T x x 2 = ∇ T x x 2 + ∇ T x x 2 -2 ∇ T x x∇ T x xonce more, we use the Cauchy-Schwarz inequality and the fact that ∥x∥ 2 < ∥x∥ 2 to establish:∇ T x x 2 ≤ ∥∇ x ∥ 2 • ∥x∥ 2 & ∇ T x x 2 ≤ ∥∇ x ∥ 2 • ∥x∥ 2 ≤ ∇ x ∥ 2 • ∥x∥ 2which we plug into the difference in network losses above to obtain:L -L 2 ≤ 2∥∇ x ∥ 2 • ∥x∥ 2 -2 ∇ T x x∇ T x xTaking expectations on both sides yields the upper bound on NL-MSE in (9) . This completes the proof of Proposition 3.In order to minimize the upper bound on the GO-MSE in (8) , note that it suffices to maximize the quantity 2E [⟨w, x⟩ • ⟨w, x⟩] since 2E ∥w∥ 2 • ∥x∥ 2 does not depend on {p i } L i=1 . We have the following:2⟨w, x⟩ • ⟨w, x⟩ = 2w T xw T L i=1 p T i x p i = 2 L i=1 w T xw T p i p T i x = L i=1 w T xw T p i p T i x + w T xw T p i p T i xBut since the dot product is commutative, i.e., a T b = b T a for any two vectors a and b, we may rearrange each of the two identical terms inside the summation as follows:w T xw T p i p T i x = p T i xx T ww T p i & w T xw T p i p T i x = p T i ww T xx T p iTherefore, we obtain:2⟨w, x⟩ • ⟨w, x⟩ = L i=1 p T i xx T ww T p i + p T i ww T xx T p i = L i=1 p T i xx T ww T + ww T xx T p iTaking expectations, we find that the quantity that needs to be maximized in order to minimize the bound on GO-MSE in ( 8) is:L i=1 p T i E xx T ww T + ww T xx T p iSimilar to the proof of Theorem 1 in Appendix A.1, this is yet again a sum of a quadratic form over the orthonormal set of vectors {p i } L i=1 and the solution is therefore to assign these vectors as the L principal vectors of C = E xx T ww T + ww T xx T as per (10) .Note that the derivation above decomposed the dot products to obtain a quadratic form on the matrix xx T ww T + ww T xx T because its symmetry is required for real eigenvalue decomposition. Also note that if positive definiteness is not achieved, we sort absolute values of eigenvalues. Finally, observe that this solution requires no overhead on the calibration process. Indeed, assuming weights and activations are independent, we note thatE xx T ww T + ww T xx T = C X C W + C W C Xwhere C W = E ww T is the weight auto-correlation matrix, which can simply be calibrated asC W = WW T /N.Thus we only require left and right scaling of the calibrated auto-correlation matrix.Similarly, to minimize the upper bound on NL-MSE in (9) , it suffices to maximize2E [⟨∇ x , x⟩ • ⟨∇ x , x⟩].Using the exact same derivation as the above, replacing w by ∇ x , we obtain that the quantity to be maximized isL i=1 p T i E xx T ∇ x ∇ T x + ∇ x ∇ T x xx T p i which is done by assigning {p i } L i=1 as the L principal vectors of C = E xx T ∇ x ∇ T x + ∇ x ∇ Tx xx T as per (10) . Calibrating this matrix does require an extra step, where we perform a backward pass to estimate activation gradients. For ease of implementation, in our results, we make an approximation on the per-sequence independence of activation vectors and their gradients. This greatly reduces the memory requirements of the calibration process. And for each sample sequence in the calibration set, we computeC (i) = X (i) X (i) T G (i) G (i) T +G (i) G (i) T X (i) X (i) T /M 2where G (i) is the gradient tensor (whose vectors are instantiations of ∇ x ). As always, we end the calibration phase by averaging across samples: C =In this appendix, we discuss all details behind our implementation in Section 4. These details include per-model specific application of ESPACE as well as retraining recipes. We strive to provide excessive details such that independent reproducibility of our results is seamless. We also encourage readers to reach out to us (after blind reviewing) for any questions on implementations.As was mentioned in Section 4, our implementation is built on top of Megatron-LM [33] which itself is based on the Pytorch framework. We use Pytorch for all extra introductions needed by ESPACE except for eigenvalue decomposition. Our experiments were carried out in a cluster of A100 GPUs and use BF16 precision. Specifically, during the calibration process, we use Pytorch to track the required auto-correlation matrices; this simply done by averaging repeated instantiations of XX T as described in Section 3.Once the calibration of auto-correlation matrices is over, we use DLPack to transfer them from the Pytorch framework to RAPIDS framework. We then use the CUPY library in RAPIDS to perform fast (a few milliseconds per auto-correlation matrix) eigenvalue decomposition on GPUs. After truncating eigenvectors, we send back the projection matrix to Pytorch using DLPack.Once the projection matrix P is calibrated and inference/training is to be done using ESPACE, we simply insert a projection operation within the Megatron implementation to perform the operations in (3) as appropriate. The projection matrices are inserted as Pytorch buffers, rather than parameters, since they do not get updated during training.In Section 4, we mentioned that ESPACE was applied at each layer such that the number of components L satisfies two constraints: (a) be a power of two for best tensor core utilization, and (b) yield a compression of at least 50% at that layer. The exact values of L for each model and layer type are included in Table 2 . Note that the only exception corresponds to QKV layers in Llama2-13B and Nemotron4-15B, where we use a value of L = 2048 which corresponds to a compression of ∼ 45% instead of >50% at least. This is only because this amount of compression is already significant that we didn't feel the need to push for L = 1024, which would have lead to a compression of > 70%.By and large, we use the exact same recipe that was used to pretrain the open source GPT3 models [33] . As mentioned in Section 4, the only modification to hyperparameters is disabling dropout and weight decay, and identical hyperparameters are used for both sets of experiments on GPT3 and Llama2 families. The only arbitrary choices we had to make was on the selection of learning rate schedule and global batch size. We use a cosine decay for all runs, and remaining choicesa are as follows:• For GPT3-1.3B, the initial learning rate is set to 1.0 × 10 -4 , the final learning rate is set to 1.0 × 10 -5 , and the global batch size is set to 512. • For GPT3-8B, the initial learning rate is set to 5.0×10 -5 , the final learning rate is set to 5.0×10 -6 , and the global batch size is set to 512. • For GPT3-22B, the initial learning rate is set to 5.0 × 10 -5 , the final learning rate is set to 5.0 × 10 -6 , and the global batch size is set to 1024. • For Llama2-7B and Llama2-13B, training is done in two stages (each of 100B tokens). In the first stage, the initial learning rate is set to 5.0 × 10 -4 , and the final learning rate is set to 5.0 × 10 -5 .In the second stage, the initial learning rate is set to 5.0 × 10 -5 , and the final learning rate is set to 5.0 × 10 -6 . For both stages, the global batch size is set to 256. • For Nemotron4-15B, the initial learning rate is set to 1.0 × 10 -5 , the final learning rate is set to 0, and the global batch size is set to 512.We did not perform hyperparameter tuning, the above was purely arbitrary, but based on the following intuition: latency of computing QKV, Proj, FC1, and FC2 GEMMs with dimensions listed in Table 2 , and then add all results together for each transformer block of the corresponding model.We measure individual GEMM latencies in Pytorch. Specifically, for each configuration, we sample 1000 set of matrices of appropriate dimension and compute the appropriate GEMM. We synchronize before and after the computation occurs, and record times after synchronization. The elapsed times are averaged and then aggregated. Since the measurements were taken with native PyTorch code, we note that the implementation is un-optimized. Further improvements could be possible in future work from removing PyTorch overheads, implementing custom fused kernels, or other optimizations. In this appendix, we include additional experimental results that were not included in the main paper. These results are not essential to the description of our work nor its conclusion, and the main paper integrally contains all essential information related to our contribution. The additional results listed in this appendix are for the benefit of readers interested in going further and learning about fine-grained details behind the main results of Section 4.• For GPT3-1.3B, we exclude some layers from the application of ESPACE. These are the layers for which out-of-the-box application of ESPACE leads to a validation perplexity increase of more than 2% compared to the baseline. These layers can be found in Figure 7 and correspond to several FC1 and FC2 layers close to either ends of the model. For this reason, GPT3-1.3B is compressed to 47% instead of 50% in Section 4 and Table 1 . • For GPT3-22B, we apply ESPACE to all layers since validation perplexity increase is very small in Figure 7 . For this reason, the overall compression for GPT3-22B is slightly over 50%; it is 55% in Section 4 and Table 1 .NeurIPS Paper Checklist• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There are no such risks associated with our work, and as such the paper does not describe safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have done our best to provide credit to any prior work upon which we have built ours. This is mostly the case for Megatron-LM [33] , which we used for our experiments, and cited extensively throughout Section 4 and Appendix B.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We use the french pronunciation "espace", which means "space".38th Conference on Neural Information Processing Systems (NeurIPS2024).Perplexity depends on tokenizer so that comparisons across LLM families (GPT3 vs Llama2) are not useful.Boldface indicates best result per task. Italics indicates ESPACE results within 5% of the baselineB i=1 C (i)/B. This completes the proof of Theorem 4. 


Decision: Although barely above borderline, all reviewers supported acceptance, particularly after the authors' extensive responses clarified or improved some issues. So I'm happy to recommend acceptance.

The idea of compressing the activations by multiplying and reprojecting times a fixed matrix seems very similar to the idea of matrix sketches, and perhaps that is essentially what is going on here. Can the authors comment on this in the final version of the paper?

Reviews: review: paper_summary: paper_summary: This paper introduces ESPACE, a method that reduces activation in models via tensor decomposition, thereby aiding in the reduction of model size and GEMM latency.
strength_weakness: strength_weakness: - Wikitext-103 is a specific type of language dataset that focuses on knowledge-based content. However, models like GPT-3 and Llama2 are capable of handling a broader range of tasks, including math and question-answering. Therefore, it is necessary to evaluate perplexity (PPL) on a more diverse set of data.
- ESPACE requires ongoing training. It is important to understand the duration required for ESPACE's continual training process.
- In Table 1 and Table 2, since ESPACE is continually trained with more data than the original model, a direct comparison may not be entirely fair. It would be more convincing to compare the original model after it has been trained on the same amount of data as ESPACE.
- For generative models such as GPT-3 and Llama2 discussed in this paper, there is a need for more comprehensive performance validation, including metrics like mmlu and math benchmarks.
questions: questions: Please see the weaknesses.
limitations: limitations: The authors address the limitations.
paper_summary: paper_summary: The paper introduces a novel technique for compressing large language models (LLMs) by reducing the dimensionality of activation tensors. The ESPACE method differs from traditional weight-centric compression approaches by focusing on activation tensors instead. ESPACE projects these activations onto a pre-calibrated set of principal components, preserving expressivity during retraining while enabling weight compression at inference through associative matrix multiplication.
strength_weakness: strength_weakness: 1. More direct comparisons with other state-of-the-art activation-based and tensor decomposition methods could provide a clearer benchmark of ESPACE's effectiveness.

2. The paper could benefit from a more thorough discussion of the trade-offs in selecting compression settings and projection matrices for different application scenarios.

3. Including a robust error analysis would help identify potential weaknesses or failure modes of ESPACE, enhancing understanding of its limitations.

4. The study focuses primarily on GPT3 and Llama models, potentially limiting the generalizability of the results to other architectures.

5. The paper does not extensively explore ESPACE's performance across various inference tasks that differ in length or complexity, which could provide a deeper understanding of its practical implications.
questions: questions: 1. How does ESPACE perform on different types of inference workloads, particularly those requiring higher precision or longer inference times? Additional details on these scenarios would help assess the method’s practical applicability in diverse real-world settings.

2. Could you elaborate on the decision-making process for selecting specific compression settings and projection matrices? A detailed discussion on how to balance compression rate and model performance in various contexts would be highly valuable for practitioners.

3. Have you conducted any tests to evaluate ESPACE’s robustness under unusual conditions or edge cases? Sharing any findings or future plans for such analyses would provide a clearer understanding of the method’s limitations and potential areas for improvement.
limitations: limitations: 1. Include evaluations or discuss plans for testing ESPACE on various model architectures to better understand its generalizability.

2. Discuss potential challenges and solutions for deploying ESPACE in real-world systems, focusing on real-time performance and integration with existing workflows.
paper_summary: paper_summary: The paper introduces ESPACE (Eigen Static Principal Activation Component Estimation), a technique for compressing large language models (LLMs) by focusing on the dimensionality reduction of activation tensors rather than the traditional weight-centric tensor decomposition. The method involves projecting activation tensors onto a pre-calibrated set of principal components, which reduces the dimensionality of activations and results in weight compression during inference through matrix multiplication associativity.
strength_weakness: strength_weakness: To me ESPACE is simply PCA projection (with a reinvented name) of activation tensor. Can the authors defend a bit?

The paper suggests that ESPACE can be used in conjunction with other compression techniques like pruning and quantization, as well as non-compressive acceleration methods such as speculative decoding. However, it does not provide empirical results for these combinations, which makes the claim unconvincing.

Although the authors claim that ESPACE is orthogonal to other compression techniques, it would strengthen the paper to include comparisons with more baselines, such as pruning, to better demonstrate the significance of the proposed method. E.g., when compressed respectively & individually with ESPACE, pruning and quantization to the same model size, how do their perplexities compare to each other? You can demonstrate that on small models like llama-58M or GPT2-97MB.

The necessity for a calibration phase that involves forward-passing multiple batches of data to estimate activation auto-correlation matrices might not scale well with larger datasets or more complex models, potentially limiting the method’s applicability in diverse settings.
questions: questions: How does the calibration set impact the final performance?

Can you provide a sensitivity profile vs layers in the Transformer model? I.e., how would the local ranks of projection in individual layers affect the global performance? (with the belief that a deep compression in early layers can hamper the output even higher ranks are used for later layers.) A portfolio of layers’ importance would be crucial for the compression strategy.

How does the projection influence the well-known outlier issue in activations and sometimes in the key values, which is observed in lots of LLM works?
limitations: limitations: The calibration set plays a crucial role in the quality of the principal components for projection, this needs to be analysed deeply to warrant good results.
paper_summary: paper_summary: The proposed method applies a PCA-inspired method to the activations of LLMs for model compression.
strength_weakness: strength_weakness: - Compression is never well-defined. Is it compression ratio w.r.t. W? What if you do not compress all weights of the model (line 112)?
- Comparison with related baselines is missing. I understand your work is orthogonal to pruning and quantization methods, but it would make sense to compare with methods that use weight decomposition such as ASVD or the ones you mention in the paragraph starting on Line 41.
- The practical benefit of the method can be better emphasized.
- The quality of the figures can be improved, e.g., the text is grainy and Fig. 2 is too small to read.
- Line 192, i==j -> i=j
questions: questions: - The paper could be made much stronger if you could show some practical benefits in terms of, e.g., inference time, VRAM requirement, or storage requirements. Right now you show compression ratios, but I believe more concrete numbers can make the method much more appealing.
limitations: limitations: The limitations are well addressed.

Comments: title: Thank you!
Thank you
Thanks for your reply!
Thank you
comment: We sincerely thank the reviewer for acknowledging our rebuttal and updating their review to recommend acceptance. We also agree with the reviewer on the need to further improve the Llama2 results and are grateful for the suggestion to use SFT data such as Alpaca. We promise to work on this and show improved results in the revision.
Thanks for the response. The MMLU results of Llamma-2 are still weak as a significant performance reduction. I suggest using SFT data (like Alpaca) to improve the results to validate the possibility of ESPACE to improve the performance. 

As most of the weaknesses are addressed, I am willing to raise my score. However, I expect the authors to show better empirical results in the next version.
We are grateful for the reviewer's acknowledgment of our rebuttal and positive opinion towards our replies and additional experimental results. We kindly ask the reviewer to consider increasing the score in order to reflect their post-rebuttal opinion of our work.
I am happy with the authors' reply and the additional experimental results. I tend to keep my score.
We thank the reviewer for reading our rebuttal, acknowledging our answers, and sharing further positive comments on our results. We kindly ask the reviewer to consider increasing the score in order to reflect their post-rebuttal opinion of our work.
Thanks for your rebuttal, it answers my questions. It is indeed nice that the proposed method can reduce GEMM time. I will keep my score.