LeSTD: LLM Compression via Learning-based Sparse Tensor Decomposition
Authors: Yi Li
Zhichun Guo
Miao Yin
Bingzhe Li
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Compression
Post-training Compression
Tucker Decomposition
Sparsity
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) achieve remarkable success, but their massive parameter counts present significant deployment challenges. Post-training tensor decomposition offers a promising, data-free compression strategy by exploiting structural redundancies within the model weights. However, existing tensor methods face a critical limitation: the dense core tensor bottleneck. While these methods find a shared low-rank basis, the resulting dense core tensor grows polynomially with the chosen ranks, becoming a new storage bottleneck and capping the maximum achievable compression. To overcome this fundamental barrier, we introduce LeSTD (\textbf{Le}arning-based \textbf{S}parse \textbf{T}ensor \textbf{D}ecomposition), a novel two-stage framework for the high-ratio compression of Multi-Head Attention (MHA) blocks. LeSTD first employs an iterative algorithm to identify a high-quality, and shared orthogonal basis that jointly represents all attention heads. Subsequently, it introduces a principled, importance-based pruning algorithm that learns an ultra-sparse core tensor by systematically removing the least salient elements and refitting the remaining ones to preserve model fidelity. By decoupling basis optimization from core sparsification, LeSTD breaks the compression ceiling imposed by the dense core, enabling significantly higher compression ratios than prior methods.

Published as a conference paper at ICLR 2026
LESTD: LLM COMPRESSION VIA LEARNING-BASED SPARSE TENSOR DECOMPOSITION
Li', Zhichun Guo?, Miao Yin Bingzhe Li
1 University of Texas at Dallas, {Yi. {Yi.Li3, Li3, Bingzhe. Bathaaee.i@@tttttt LiJsutdallas.edu
2 Independent Researcher, zcguo work@gmail com
3 University of Texas at Arlington, miao aalingoonmmaa.yyinuuaaaeee yineuta.edu
ABSTRACT
Large language models (LLMs) deliver the impressive capability, while their pa- rameter scales hinder the deployment ability. Post-training matrix/tensor decom- position offers a promising strategy alleviate this by exploiting structural re- dundancies within model weights. However, it faces the critical dense core bot- tleneck. This bottleneck caps achievable compression level as dense core tensor becomes new storage burden. To solve this, we introduce LeSTD (Learning- based Sparse Tensor Decomposition), a two-stage, data-free compression frame- work. LeSTD first learns high-quality shared basis for model weights, then applics a theoretically-gruet grouund pruning mechanism: guided by a derived closed- form importance score, create ultra-sparse core tensor. Therefore, resulting a superior compression-accunt compression ratios than dense-core methods without sacrificing performance. Ex- periments on the LLMs up to 30B parameters confirm that LeSTD consistently attains lower perplexity and higher task accuracy matched compression levels, and critically, maintains strong performance under aggressive compression where prior methods degrade. Operationally, LeSTD executes the inference directly in its compressed domain, delivering significant throughput gains on standard hard- ware without requiring any custom kernels.
trade-off: LeSTD achieves substantially higher
INTRODUCTION
Transformer- based Large Language Models (LLMs) (Vaswani et al.) 2017) form the backhone of contemporary AI, demonstrating remarkable performance in natural language understanding (Nic al. 2019; Gatt & Krahmer 2018), code generation (Gu) 2023), and mathematical reasoning Zeng
jt al., 2023] Imani et al.) 2023). However, these advances are accompanied by an ever- wver-growing model scale, with models comprising hundreds of billions of parameters. This imposes formidable storage and memory demands (Kwon et al. 2023;; Hu et al. 2004)), limiting their deployment in resource-oonstrained environments and necessitating effective model compression strategies.
Among various paradigms, post-training LLM compression approach have emerged as practical so- lutions as can avoid the expensive retraining. This paradigm includes the pruning (Ma al. 2023 Fu et al. 2024), quantization Liu et al. (2023) 2024); Zhao et al. (2024), and low-rank decomposi- lion (Wang al. 2024b Yuan low-rank approach, e.g., Singular Value Decomposition (SVD), has shown the promise on individual weight matrices (Wang et al. 2024b), demonstrates strong application potential.
2023). Among all these different approaches, even the simplest
Most existing low-rank methods (Yuan et al.) 2023 Wang et al. 2024b) adopt matrix byy-matiix compression strategy. While this approach is intuitive, overlooks higher-order correlations and the structural redundancies spanning related weight matrices (Szekely et al.) (202). Consequently, it misses the opportunity to treat them as a unified object, which could further improve the achievable compression ratio. Moreover, vast studies have shown that multiple attention heads (MHA) within a single Transformer layer often learn the redundant patterns (Michel et al., 2019 Voita et al. 2019 Yang et al. 2024), highlighting additional opportunities for joint compression. Motivation. Achieving joint compression is non non-trivial. A fundamental question is whether a shared structure truly exists across heads within one layer? Intuitively, because all heads in a layer process

---

Published as a conference paper at ICLR 2026
sel
intra layer
Inter
explained energy
[3]
(b)
[c)
OPT-6 7B
0.00
600
800
1000
0.0
200
400
600
0.0
200
600
800
1000
explained energy
1.0 (d)
1.0 (e)
1.0
(f)
LLaMA 22-2
0.5
0.5
0.5
0.0
600
800
1000
0.0
400
600
0.0
800
1000
rank r (ane head)
rank
rank r (three heads)
(two heads)
Figure 1: Subspace-sharing diagnostic on OPT-6.7B and Llama2-7B across multiple heads. For each column, we compute a rank-r orthonormal basis from the concatenated weights [WQ|WA [ (W0) of layer-15 using: (Figure (a).(d) one head (head 0), (Figure (b), (e)) two heads (heads 0, 1), and (Figure (c), (f)) three heads (heads O, and 2), and evaluate the ex- plained energy on self, intra-layer (other heads in layer 15), and inter-layer (layer-25 head 0), with {2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]. The top row shows OPT-6.7B and the bottom row shows Llama2-7B.
the same input distribution, their weight matrices might lie in (or near) a common subspace. Diagnostic. To test this hypothesis, we design diagnostic that quantifies subspace sharing. Core idea is to learn basis from one attention head and assess how well it transfers to other heads. Concretely, for a given head in layer L, we construct its complete linear transformation by con- catenating its constituent Query (WQ), Key (WK), Value (WV) ) and Output (W²) weights into a single matrix: Hi,- = W W WV [W E Rddusa x 4-dhaad We then perform an SVD and extract top-r left singular vectors to form an orthonormal basis UG) This basis is used to project and reconstruct three targets: (i) the source matrix itself (self), (ii) another head j (j o i)'s matrix from the same layer (intra), and (iii) a head's 's matrix from a different layer & (k o 1) (inter). Re- construction quality is measured by the explained energy (Greenacre al.) 2022 Wiskott 2013),
defined as 1 - E², where E = |H-A is relative Frobenius reconstruction error (Tong et al,
H|]
2021 and H = (U()) T is reconstructed matrix (Eckart & Young 1936). Explained energy quantifies how much of target matrix's total information is retained when projected onto another head's 's basis, higher value indicates stronger overlap between two subspaces and thus greater sub- space sharing. The curves in Figure ddirectly address the central question posed our motivation: does a shared subspace exist across heads within the same layer? The answer is affirmative. As the subspace rank and the number of heads increase, explained energy for intra systematically rises and remain consistently above inter. This pattern shows that basis learned from one more heads capture nontrivial portions of the variation of other heads in the same layer, and that captured portion grows as
dimensions are allowed. Hence, there is real, layer-local structure that different heads share. At the same time, for a single head, the intra-layer curves saturate well below the self ceiling. However, by examining layers with a larger number of heads, we find that the shared similarity be- tween heads in terms of explained energy increases as the number of heads grows. Geometrically, this suggests that the corresponding subspaces exhibit greater overlap when more heads are present. This trend highlights additional untapped structure and indicates the potential for new methods that more effectively exploit the unexplained energy.
These two facts together validate our motivation and revealed research gap: shared structure does exist, but exploiting requires a mechanism that is richer than matrix-by-matrix (single-head) trans- fer. This mandates joint optimization approach to identify common subspace that can efficiently represent all heads simultaneously. Recent methods, such as the TensorLLM (Gu et al. 2025), apply global Tucker decomposition to the MHA block and successfully identify high-quality shared basis. While this represents a clear step forward beyond matrix- ba-rix- compression, it also in- troduces critical and unresolved limitation: Dense Core Bottleneck. Tucker decomposition factorizes a tensor into a set of compact factor ma- trices and tensor. In our setting, the factor matrices capture low-dimensional shared subspaces
2

---

Published as a conference paper at ICLR 2026
along each mode, while the core tensor encodes the remaining cross-mode interactions. While the factor matrices can be relative small, the core tensor remains fully dense (Ahmadi-Ast et al. 2021 Wang & Yang 2022b. To preserve model accuracy, the Tucker ranks must be sufficiently large, but the size of the core tensor grows polynomial with these ranks. This dense core rapidly becomes the new storage bottleneck, imposing hard limit the achievable compression ratio.
This limitation exposes a fundamental gap: current tensor-based methods reduce dimensionality but fail eliminate redundancy within the compressed latent space itself. To achieve the truly high-ratio compression, we must moving beyond low-rank approximation and into the domain of sparse tensor representation (Park al. 2021
To fill this critical gap. we propose LeSTD (Learning- Learning-based Sparse Tensor Decomposition), frame- work that synergistically combines iterative basis optimization with the learned core tensor sparsity. LeSTD operates in two stages: first, it optimizes high-quality shared basis for all attention heads; second, it learns an ultra-sparse representation for the core tensor within that basis. This integrated approach yields representation that is both more accurate and vastly more compact. Main contri- butions of LeSTD are as follows:
1.
The proposal of LeSTD, data-free post-training compression framework where Stage learns high-quality shared basis via iterative optimization, and Stage II introduces a principled pruning strategy to create ultra-sparse We provide a theoretical justification for the magnitude-basee based pruning in the Tucker-decomposed latent space. We derive closed-form importance score for each core element, directly linking its magnitude to its impact on the Frobenius reconstruction error. This allows for a principled, rather than purely heuristic, sparsification of the core
tensor.
3. We demonstrate how inference can operate directly on the compressed representation, avoiding the full weight reconstruction. It reduces arithmetic complexity and delivers practical throughput gains (tokens/sec) measured directly within the standard Transformers library (Wolf et al. 2020), requiring no specialized hardware or custom kernels.
4. Across GPT-J (6B). Llama2 (13B), and OPT (30B) on WikiText-2, MathQA, GSM8K, and Truth- fulQA, LeSTD consistently outperforms baselines at matched size fractions: maintaining higher accuracy under strong compression and delivering competitive-toosuperri throughput.
2 BACKGROUND AND PRELIMINARY
Notation. We denote tensors by uppercase cal- ligraphic letters, e.g., X' 1 ma- trices by uppercase bold letters, egg. X E 11 and vectors by the lowercase bold let- ters, e.g., x E R4, The Frobenius norm of a tensor (Defant Floret, 1992) ||
Tensorization MHA Block
XW
IFN Laysr
En "i.....i The mode-n unfolding of X is a matrix ((n) E f 1v) whose columns are the mode-n fibers of the tensor. Transformer Architecture. Modern LLMs typically employ a decoder-only Transformer architecture (Vaswani et al.) [217]] Fu al. 2023), composed of a stack identical layers, as shown at the right of Figure 2 Each layer contains a Multi-Head Attention (MHA) block and Feed-Forward Network (FFN). MHA block allows the model to jointly attend to information from different representation sub- spaces. For the i-th attention head, the query Q. key K, and value V inputs are linearly projected via weighted matrices WºO, w.K. W. Indde dread i h, and dread dunae is the per-head dimension:
Amseniinn
An illustration of the Transformer ar- chitecture (right) alongside the corresponding ten- sorization process used in our scheme (left).
Figure 2:
(QW
head; A KW.., vW!) softmax
(VW!) (1)
3

---

Published as a conference paper at ICLR 2026
Focker Descepsation
Loarnd Sparis Carn Tessers
Legend
Sparso Care Tenser
45
Figure 3: Overview of LeSTD. Left: MHA weights tensorized. Middle: An iterative optimiza- tion finds high-quality shared orthogonal factor matrices (Ur()) and dense core (G,) for each head. Right: A pruning mechanism enforces extreme sparsity on the core tensors.
To improve stability and training efficiency, residual connections (Xie et al. 2023) and layer nor- malization (Xiong et al. 2020) are usually applied, though omitted here for clarity. The outputs of attention heads are concatenated and projected back into the latent space via an output matrix Wo:
MultiHead (Q, K, V) Concat heads) Wo
Thus, MHA block involves four weight matrices: the query projection WQ, key projection WK value projection WV, and the output projection W We refer to these the MHA weight matri- ces, which constitute the target of our compression.
Tensor Operations and Decompositions. The n-mode product of tensor X' E R 12xxlxx"'xxx with a matrix U E R. II. is a tensor y E TR/1x--xxx xxxx"xxax-..1ll x..IIN denoted as y = X' x, U. Element- wise, this is defined as ())i.... Jo.....o.......
Σln Ti Pis...da....di j.. This operation can be under- stood multiplying the matrix U with every mode-n fiber of the tensor X. Tucker decomposition is fundamental higher-order generalization of SVD. It decomposes a tensor X into a (typically smaller) core tensor 9 E RR x.xx and a set of factor matrices (U(n) E RL, R., J.V.
In=1. one for each mode, The decomposition expressed as X R 9 x1 U(1) X2 U X U The core tensor G can be interpreted as containing the latent interactions between the factors, while the factor matrices (?) represent the principal components in each mode. The dimensions of the core tensor, (R1, Ry). are referred to as the Tucker ranks of the decomposition. By choosing Rn < Ins a compressed representation of X is achieved.
3 DESIGN OF LESTD
LeSTD is a two-stage, data-free post-training compression framework for the MHA blocks of LLMs. Its goal is to dramatically reduce the storage of MHA parameters while preserving model accu- racy. Figure 3e an overview of LeSTD. At a high level, LeSTD first learns a shared low-rank orthonormal subspace in which all heads and all Q/ Q/K/V K/V/O projections can be represented com- pactly (Stage D), and then sparsifies the resulting core tensor in this subspace with closed-form error control (Stage II). The final model performs inference directly in the compressed domain, without ever reconstructing the original dense weight matrices.
Two-stage design. In Stage (refer Section 3.1. we perform an iterative, shared-subspace Tucker decomposition (left sides of FigureZland Figure 3 on the MHA weight tensor. This procedure yields a set of shared orthogonal factor matrices (n) and, for each attention head i, a small but dense core tensor g,. By iteratively refining the shared factors to jointly represent all heads, Stage I captures the common structure across heads while allowing head-specific variation to reside in the cores. In Stage (refer to Section 3.2). we explicitly address the remaining dense-core bottleneck by introducing an importance-based pruning algorithm that learns an ultra-sparse core tensor. By systematically identifying and removing the least important core coefficients, and refitting the remaining ones in closed form, Stage II maximizes sparsity while tightly controlling the reconstruction error.
Both stages are executed sequentially (Stage followed by Stage II), resulting in a highly compact representation of the MHA block. Section 3.3 then describes how the compressed model carries out inference without reconstructing the original weight matrices, thereby reducing storage and trans- mission costs: all Q/K/V/O projections are evaluated directly from the shared factors and the sparse core, i.c., LeSTD enables LLMss to run inference directly on compressed weights without any decompression step.
4

---

Published as a conference paper at ICLR 2026
3.1
TUCKER DECOMPOSITION (STAGE I)
Tensorizing MHA Parameters. Following the recent works (Gu et al. 2025). we restructure MHA parameters into single 4th-order tensor (as Figure 3 Left). For each attention head we stack
it four weight matrices: W;O, W.K. W.V. E Rhoas dhoed where dhead dmodel/ hi and (W?9)
dddned x deead into 3D tensor: W; = W.O W.K, WWW.WWWW)
1R. xdhaax4 We then stack
these h tensors along fourth mode to form the final MHA weight tensor: Winual E RRiduasss X thess X xxxh Shared Low-Rank Decomposition. We perform standard Tucker decomposition (Malik & Becker 2018; Ahmadi-Asi et al., 2021] Wiotal that factorizes along modes I, 2, and 3, while leaving mode-4 (the head index) uncompressed. This means all heads share the same factor ma- trices (U("), = 2,)) but each retains its own slice of the core tensor (G,, E (,,2, k)). Topology of decomposited Wixal
uxal is shown in Figure Approximation is:
(3) where Grocal E R X is the dense core tensor produced in Stage and Ua), _2 U are the shared column-orthogonal factor matrices with column dimensions R1, R2, R3 respectively (Tucker ranks (R,, < In). Since we do not factorize the head mode, each head retains its own core slice G; = Gooal [....; E R R3 that can encode head-specific information, yet all slice G; live in joint coordinate system defined by the shared factors U (Z)2, (3).. Optimization Procedure. We determine U U(2) (3) and the gonal by minimizing reconstruction error between the original and factorized weights (Böttcher & Wenzel 2008):
W Motal 2 W Motal = Gwall X1 U X2 U X3 U
dhead
dheaad
| rotll Weocll s.t. (w TT() Im.
min ( (T(a) Iuul
toeat
(4)
We solve Eq. using the Higher-Order Orthogonal Iteration (HOOI) algorithm (Liu et al. 2014, Matstoms 1997.. Each sweep updates one factor by projecting W hotal onto the current subspaces of the other modes, taking a truncated SVD of the resulting mode- matricization, and setting ((1) to the top-Ri, left singular vec- tors. After updating modes 1,2,3, the core is recomputed G Wioial X1 (U(1)) X2 (U) X3 (U(3))T. This loop repeats until convergence, yielding the shared low-rank factors and the dense core tensor for all heads.
Amodel
Figure 4: The shape of W before (Left) and after (Right) decomposition.
3.2 LEARNING AN ULTRA-SPARSE CORE VIA IMPORTANCE- BASED PRUNING (STAGE II)
Stage (Eq. (3) in Section 3.1 yields a high-quality, but dense core tensor Giotal within a shared orthonormal subspace. We now aim to climinate the dense core bottleneck by enforcing extreme sparsity on the Gwall while pre- serving reconstruction fidelity. Our approach in Stage is ex- plicitly importance-driven: we prune the least important ele- ments of Gotal and set them to zero, thereby producing an ultra- sparse core 9 spurse that maintains the accuracy under orthonormal (Stage I). Overall procedure of Stage II is pre- sented in Algorithm[I] The criti- cal question is thus: how should we define and compute the im- portance of an individual core
Algorithm 1: Stage II Importance- based Core Pruning
Input Guotal Un))(n = 1, 3), W Wistal target sparsity
Suagets pruning rate
Output: Gspars
Initialize G <- Guxal
while | Surger do
# Step 1: Compute Importance
Calculate Imp(gs) for all non-zero ga E 9 using Eq. 5
# Step 2: Prune
k + | // number of non e elements // lowest Imp(-) score
A - k-
for 3 E A do
g3 t mark as pruned
shared
# Step 3: Refit
for each unpruned element gy E G do
Update gy using Eq. (6)
return 9 sparse + g
5

---

Published as a conference paper at ICLR 2026
element? We work entirely within the orthonormal subspace identified in Stage I (Eq. (3) Re- call that Stage I reconstruction is Wiotal tocal X1 U()) X2 U22) X3 U Let 3 = (r (2,, i) (n) the r,-th column of U(m). Define rank-1 basis tensor as- be a multi-index and denote by wi,
(3) sociated with B as the outer product: BB = un urz o Ury e, where e; is the i-th stan- dard basis vector along the head mode. Because factor matrices are column-orthonorraal the collection } is mutually orthonormal with unit Frobenius norms, i.e., (B B,) = 6g, and |B = Let R. W rotal W
total be Stage I residual. Since Westal rotal is the orthogonal projection of W motl onto the span of {Ba}, orthogonality of the decomposition implies that: (R. 33) = 0 and
[| - | + -3 s sß. where 9/ denotes the coefficient of B in Grotal-
Defining Importance. We quantify the importance of an element ge of the normalized increase in reconstruction error caused by zeroing it out while holding all other quantities fixed. Writing Stage I error as & = || consider the hypothetical reconstruction in which 93 is set to zero, the corre-
sponding reconstruction tensor is W (m = W go 93B3, because removing a single basis coefficient subtracts exactly its rank-1 contribution. The new residual therefore
0g Wiotll
(9=0) = R + BB
Corresponding reconstruction error is obtained by expanding Frobenius norm and the orthogonality:
{g)=0) = R + 9 (R + 93 + 93 = || + 2go (R. BB) + 93 1
E + 2gz 0 93 1 = ε + و
which yields the efficient expression (as shown in Algorithm D Step I):
E Eوgg 0) = 8 93
(5)
Hence the normalized importance is Imp(ga) = E(ga=0)-E E. Because the basis tensors are 8
orthonormal, ordering elements by Imp(ga) exactly equivalent ordering by |gal. It follows that, under the Frobenius loss, the best k-term approximation is achieved by keeping the k largest Igal coefficients (i.e., furd-thresholding). This gives principled, dlosed-form criterion for pruning: the least important elements (smallest magnitudes) are removed first and set to 0 (Algorithm [ Step 2). Refitting remaining coefficients. After pruning a subset of indices A (setting ga + 0 for B E A), we optionally refit remaining coefficients to counter numerical drift (Step 3). Fix all factors and the pruned pattern, and consider a surviving index 7 & A. We determine the optimal 9, by solving one- dimensional least-squares problem: g. - argmin 9; || - (g/ + Esgrser BB Ir. Ex- panding the squared norm and differentiating with respect to م'.. obtain the first-order optimality
condition: a (IW lotal 1/- 2//, (W lotal B,,) 2g/ Es g3 (3 3, BB) (04) B, 11/) O. Tg. Using BB B.,) 8g,, and ||? 1, the mixed terms vanish and we get - 22W total: +2g'
g/ (Wrotal, ioaal, B.). Thus the refitting update takes the closed form:
9- + (Wioral, B,)
(6)
which coincides with original coefficient when the factors are fixed and arithmetic is exact. Per- forming this refit after each pruning step helps to stabilize the reconstruction in finite precision.
3.3 INFERENCE WITHOUT RECONSTRUCTION
A key benefit of LeSTD is that inference operates directly in the compressed domain: the full dense matrices are never materialized. All computations are expressed through the shared low-rank factors (in Stage I) and the learned sparse core (in Stage II). The overall process is shown in Algorithm 2 we detailed the steps below:
One-off left projection (shared across heads). Let X E T'd be token representations (em- beddings) for a mini-batch (or T = in the autoregressive single-step). We project X once into the rank-R, R, subspace and reuse it for all heads and all of Q, K, V: Y X (1) E I' Core contraction and pea-head slices. Denote Stage II sparse core by & R Rexh and the third-mode factor by (3) E R where E (1:Q, 2:K, 3:V, 4:0) indexes the projection type. For each head i and type t, we define the slice, which is also typically sparse due to g sparse. as: ERz sparsel (r1, r2, r3, 1) (3)) ra), where M r2) IR R, x R2 and depends
M Marri, r2]

---

Published as a conference paper at ICLR 2026
only the compressed parameters and can be precomputed and cached offline. Input-side linear maps (Q, K, V) without reconstruction. With U E IR TRo R2, the three pro- jections (Eq. I for head i are computed by Q, = (U K; = ( and V, Y YM (UT)),, yielding Q. K, Vi E R'''dee without ever forming W W.KK W.Y.
Attention and output-side map without reconstruction. We scaled dot-product attention proceeds
as usual: A, = softmax Q.K, and O = A,V, R''has For output projection (Eq. 2 note Vihoal
that (W?) T = U ( = 0 = U M. (U))). Thus, projected output for head i is O O, (O, (U E R Xxukk Aggregating all attention heads (e.g., 2 gives the final output. An optimization is to delay the final multiplication by (U()): Σイ O (0,U ( (U()),, so that (U(I) is applied only once.
Concat in original Eq.
By performing above processes, the original matrices (W W WY, WP) are not instantiated. All products are composed from ( U precomputed (sparse) MI,, and activations. A formal theoretical complexity analysis is deferred to Appendix
Algorithm/ 2: Inference Without Reconstruction in LeSTD
Input: : Token representations X RT'xxuddi Shared factors U()) E R Rs,
U(22) Ra U E Rs; Sparse core 9 sparso IR Ri x RRxRRx
Output: MHA output Z E 2''xdu
# One-off left projection (shared across heads and types)
Y t XU TT
# Precompute per-head, per-type compressed matrices
for <-1 1 to do
for t E [1:Q, 3.V 4:0} do
for ri f- to Ri do for
1 to R2 do
Marrr,,
<
9 spars Sparserr,, [ri, v3.d] (1,
# side linear maps (Q, K, V) in compressed form
for +- 1 to h do
Q Y M, (U R xddhas
K, + M
E ddhad
V, + Y M (U E T'x'
Scaled dot- eot-product attention for i + 1 to h do
Q.K. v Creal
A < softmax
O, A.V, E x
Output-side projection in compressed form for f-1 1 to do
H, (2U E ERTRR
S H,M, T
S Σh S, T
Final projection back to model dimension
Z + ((UT E 7'dmon
return
EVALUATION
We compare LeSTD against SVD-based approaches
Yuan et al. 2023 Wang et al., 2024b: Gu
al. 2025) and pruning techniques (Ashkboos et al. 2024 We further consider quantization as

---

Inference Efficiency. As shown in the Figure 5 LeSTD delivers competitive- -erettivvv--osuupriioorr through- put while retaining accuracy across LLMs, datasets and compression rate. At mild compression (c.g.. 0.8), it is at least on par with fastest baselines and often ahead: on WikiText-2 it edges SVD- LLM on GPT-J (11571.82 11521.82) and improves ~ 3% on Llama2- 13BB. The gains over Published as a conference paper at ICLR 2026
another representative post-training compression paradigm (Huang al. 2024 Shang al. 2023 Specifically, we examine the direct comparisons in terms of compression ratio vs. model accuracy, with detailed results and analysis provided in Appendix
We conduct a series of internal ablation studies to validate LeSTD's s design choices. These studies analysis the individual contributions of two-stage approach and examine the model's sensitivity to key hyperparameters, such as Tucker ranks. The comprehensive results, which confirm effective- ness of each component, are presented in Appendix
4.1 EXPERIMENTAL SETTING
Baselines. We compare LeSTD with three post-training LLM compression schemes: (1) Low-rank
decomposition: ASVD (Yuan et al. 2023). SVD-LLM (Wang et al., 2024b), and TeMsorLLM Guu
ctt al.
)025); (2) Pruning: SliceGPT (Ashkboos et al. 2024); (3) Quantization: PB-LLM (Shang
et al.
2023) and BiLLM (Huang et al. 2024). Note that LeSTD is post-training compression
approaches, such as OneBit (Xu et al. 2024). fall outside scope of this work. Please refer to Appendix(Efoor detailed discussion of related works.
framework. Training-awar
LLM models and Datasets. We evaluate performance of LeSTD and baselines on three families of
LLM models at different scales (GPT-J (Wang 2021), Llama2- 13B (Touvron al. 2023), and OPT- 30B (Zhang et al. 2022)). ). Experiments spans four datasets covering diverse tasks: natural language
modeling (Wiki Text-2 (Merity et al. (0166). natural language understanding (MathQA (Amini al.)
2019)) math reasoning (GSM8K (Cobbe et al.) 2021)), and the natural language generation (Truth- fulQA (Lin et al. 202T)). For a detailed description of datasets, please refer to Appendix [A] Environment. All experiments are conducted on a server equipped with an Intel(R) (eon(R) Platinum 8470 CPU, 1024 GB of DDR4 memory, and one NVIDIA H100 GPU with 80 GB GPU memory, running Ubuntu 22.04.5 LTS. Implementation is based on Python 3.10.12 and PyTorch 2.7.0, with CUDA version of 12.9. The efficiency index (tokens/sec) were measured under the Transformer Library (Wolf et al. 2020 (version 4.53.0). For detailed setting of base- lines and LeSTD, such as hyperparameters and implementation, please refer to Appendix B Due to the presence of Stage II (Section (.2), compression ratio of LeSTD is not straightforward to compute as that of baselines. In Appendix G we present our approach for precisely controlling the compression ratio budget in LeSTD.
Holtware&Haadwanrd
4.2 OVERALL COMPARISON
WikiText-2, MathQA, GSMSK, and TruthfulQA across three LLMs with different scales/families, namely GPT-J (6B), Llama2-13B, and OPT-30B. Evaluation focuses on two aspects: trade-off between compression ratio and model accuracy, and inference efficiency, the latter quantified by tokens per second (tokens/sec).
We evaluate four datasets:
Compression Rate Fs. The Model Accuracy. As shown in the Table across all the three model scales/families and four benchmarks, same trend recurs: the compression strengthens, LeSTD preserves accuracy more faithfully than matrix-by-man matrixx SVD (ASVD and SVD-LLM) and than dense-core Tucker (Le) At a aggressive ratio (e.g., 0.2). LeSTD lowers WikiTeext-2 per- plexity by ~ 15% relative to SVD-LLM (e.g.. 42.70 vs. 50.25 on OPT-30B). Besides, when the core becomes the bottleneck for Tucker at very small ratio (e.g.. 0.2), LeSTD outperforms TensorLLM by ~ 10% (e.g., 80.37 vs. 89.52 on GPT-J). On MathQA and GSMSK, LeSTD typically matches or edges out the best baseline by 3 points, and TruthfulQA it remains stable where the pruning scheme (SliceGPT) or matrix-by- matrix SVD can collapse.
These results align with our design and analysis. Stage learns a single, shared subspace per layer (3) and Eq.44)) that captures cross-head regularities, explaining the flatter degradation relative to matrix-wiss wise SVD. Stage II then breaks the dense-core ceiling by pruning in an orthonormal latent basis, where reconstruction is an orthogonal projection and coefficient importance has closed form (5 and (q. This is why LeSTD sustains accuracy at compression levels where dense-core Tucker, for example, TensorLLM, must retain large ranks.
8

---

Published as a conference paper at ICLR 2026
The overall results of compression rate (defined as onggeaatd vs. model accuracy across four datasets. Some results marked "-", indicating that value is 0 when rounded to two decimal places. Uparrow "f" indicates higher measure value are better, and downarrow "L" indicates lower better. Columns with a compression ratio of refers the original model without compression.
Table I:
original
(a) WikiText2 (Metrie: Perplexity 4)
Ratio
CUTE-TOR
Mathal
0.2
0.4
0..6
0.8
0.2
0.4
0.5
0.5
0.2
0.4
0.5
0.8
ASVD
4412.47
3873.72
273.52
15.91
2826.88
139.20
49.05
14.97
2423.65
998.26
100-48
24.89
SVD-LLM
100.08
50.11
20.19
14.95
34.25
22.52
14.88
9.93
50.25
20.00
14.34
11.95
TM
89.52
49.70
9.90
8.92
8.86
14.97
14.09
9.56
5.01
4.88
49.53
19.97
14.90
11.95
9.56
SherCGP
107
59.09
12.1
1988
10.05
24.77
15:08
9.99
LuSTD (Ores)
80.37
49.36
8.92
13.99
11.94
7.93
42.70
20.13
14,02
9.08
(h) MathQA (Metric: Accuracy f)
Ratio
Llama2-138
OPT:30B
Method
0.2
0.4
0.6
0.8
0.2
0.4
0.5
0.8
0.2
0.4
0.6
0.8
1
ASVD
0.14
0.20
0.21
0.22
0.08
0.11
0.26
0.06
0.17
0.20
SVD-LLM
0.18
0.20
0.21
0.22
0.16
0,22
0.24
0.30
0.14
0.21
0.26
0,29
TensorLLM
0,16
0.19
0.20
0.21
0.15
0.26
0.29
0.10
0.24
0.31
SliceGPT
0.11
0.16
0.19
0.21
0.06
0.14
0.24
0.27
0.06
0.15
0.20
0.28
LeSTD (Ours)
0,14
0.19
0.20
0.22
0.16
0.23
0.26
0,30
0.13
0.22
0.25
0.29
(c) GSM8K (Metric: Exact-mate Accuracy f)
Ratio
GPT-J
Llama2-13B
OPT-30B
Mathod
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
ASVD
0.02
0.16
0.03
0.07
SVD-LLM
0.03
0.06
0.10
0.14
0.20
0.03
0.04
0.08
0.10
TensorL LLM
0.01
0.03
0.05
0.13
0.08
0.08
0.19
0.23
0.02
0.03
0.07
0.08
0.11
SPiceGPT
0.01
0.04
0.03
0.17
0.21
0.02
0.05
0.08
LeSTD
0.02
0.04
0.09
0.05
0.08
0.17
0.21
0.03
0.03
0.07
0.09
(d) TruthfulQA (Metric: BLEU-1 1)
Ratio
GPT-J
Llama2-138
OPT-30B
Methed
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1
0.2
0.4
0.6
0.8
0.02
0.03
0.04
SVD-LLM
0.01
0.01
0.05
0.03
0.03
0.04
0.06
0.03
0.03
0.06
0.07
Tensoel. I.M
0.02
0.04
0.06
0.02
0.02
0.05
0.07
0.08
0.02
0.03
0.05
0.07
0.08
SliceGPT
0.01
0.03
0.02
0.03
0.06
0.02
0.05
LeSTD
0.02
0.04
0.06
0.03
0.05
0.06
0.07
0.01
0.03
0.05
0.08
TensorLLM (dense-core Tucker) are large: up to ~ 40% improvement tokens/sec. For short-context workloads (Zhou et al.] 2025) such as GSM8K and TruthfulQA, LeSTD's advantage at an aggres- sive compression (e.g., 0.2) is consistent, yielding ~ 10% improvement averagely over SVD-LLM on GPT-J and OPT-30B. On MathQA, throughput 0.8 is mixed: small positive on GPT-J, near parity on Llama2- 13B, and modest deficit on OPT-30B, consistent with cases where heavily tuned GEMMs (General Matrix Multiplications) favor purely low-rank baselines (Sun et al. 2025).
At stronger compression (e.g.. 0.4), LeSTD maintains competitive throughput as dense-core Tucker degrades more sharply and matrix-by-matrix SVD increasingly pays for redundant per-head pro- jections. This behavior follows from LcSTD execution (Section 3.3 inference never reconstructs dense Q, K, V, O, a single shared left projection (1) is reused across heads and across Q, K, V, and the right side applies (T) with head-local, precomputable slices M Stage II's ultra- sparse core reduces arithmetic consumption, whereas dense-core Tucker (TensorLLM) retains poly- nomial many latent interactions. In summary, share once, sparsity what remains, and execute directly in compressed domain, the observed tokens/sec reflect exactly this design. To ensure our results reflect the real-world scenarios, all throughput benchmarks were conducted within the standard li- brary (Wolf et al. 2020) (v4.53.0) without any custom-written sparse kernels. Observed speedups (Figure 5) stem from the library's underlying Pytorch (Paszke et al. 2019) implementation, which can leverage the reduced computational load from the LeSTD's 's sparse core tensor (Section and inference compressed (Section 3.3). While performance of unstructured sparsity be sensitive to implementation and hardware details (Dave et al., 2021; WWang] 2020), our results demonstrate that LeSTD provides tangible wall-clock time improvements in widely-used environment.

---

Published as a conference paper at ICLR 2026
WikiText-2.
(b) MathQA.
(c) GSM8K.
(d) TruthfulQA.
(I) GPT-J
(c) WikiText-2.
(f) MathQA.
(g) GSM8K.
(h) TruthfulQA.
(II) Llama2-13B
ii WikiText-2.
ii) MathQA.
(k) GSM8K.
(l) TruthfulQA.
(III) OPT-30B
Figure 5: Throughput (tokens/sec) of LLMs across datasets, comparing baselines with our LeSTD
5 RELATED WORK
Recent research has explored post- vost-training compression techniques for LLMs (Zhu et al. 2024). ASVD (Yuan et al. 2023) introduces activation-awann scaling improve SVD- based low-rank ap- proximations without retraining whole model. SVD-LLLM (Wang al. 2024b) further refines this by incorporating truncation-aware whitening and closed-form updates, achieving superior accuracy under aggressive compression. TensorLLM (Gu al., 2025) applies Tucker decomposition to MHA, enforcing a shared subspace across heads for up to 250X compression. SliceGPT (Ashkboos et al., and columns of weight matrices, reduc- ing compute and memory while maintaining nearly full zero-shot accuracy. Our method, LeSTD, follows this post-training paradigm but departs from prior designs through its shared-subspen sparse Tucker formulation. A detailed description of related works, please refer to Appendix
2024) adopts structured pruning approach by deleting
6 CONCLUSION
We introduced LeSTD, a learning-based sparse tensor decomposition framework designed to solve the critical dense core bottleneck in tensor-based post-training I.IM compression. The key point is a principled, two-stage design, which decouples basis optimization from core sparsification. This allows LeSTD first learn high-quality shared latent space for all attention heads, and then ag- gressively yet safely sparsify the core tensor within that stable basis. It is this ability to eliminate redundancy within the compressed latent space itself that allows LeSTD to break the compression ceiling imposed by dense-core methods. Our extensive experiments confirm this powerful synergy. showing that LeSTD consistently outperforms existing methods and establishes a new state of the art in the compression-accurac trade-off for large language models.
ACKNOWLEDGMENT
This work was partially supported by NSF 2343863, 2413520, 2417747 and 2440611, 1. Any opinions, conclusions, or recommendations expressed in this material are those the authors and do not necessarily reflect the views of the NSF.
10

---

Published as a conference paper at ICLR 2026
LIMITATION
pruning produces an unstructured sparse core tensor. While this approach is highly effective at reducing theoretical FLOPs and storage (refer to Appendix F). leading to practical speedups we demonstrate using standard software libraries (Sec- 4.2n its performance on specialized hardware could be further amplified. Exploring methods to enforce structured sparsity (e.g., block-sparsity) within the core is a promising direction for better hardware co-design.
A limitation of LeSTD is that Stage (Section 3.2
ETHICS STATEMENT
In this paper, we present design grounded in sparse tensor factorization that addresses the growing scale of Large Language Models (LLMs). Our approach significantly reduces the storage and com- putational requirements of these models without compromising performance, thereby promoting the democratization and wider accessibility of powerful language technologies. This improvement holds potential for wide range of applications by enabling the deployment of LLMs resource- constrained devices. We believe that our method contributes positively to the advancement of ma- chine learning research by promoting computational efficiency and sustainability. Although do not anticipate any immediate negative ethical implications or societal concerns from our approach, it's important acknowledge that machine learning technologies, including LLMs, have broader impacts. The increased accessibility of these models underscores the need for ongoing research into their potential for misuse and the development of robust safeguards. Therefore, responsible implementation and continued dialogue are crucial to ensure that such technologies are applied in manner that promotes fairness, transparency, and beneficial socictal outcomes.
REFERENCES
Salman Ahmadi-Asl, Stanislav Abukhovich, Maame G Asante-Mensah, Andrzej Cichocki, Anh Huy Phan, Tohishisa Tanaka, and Ivan Oseledets. Randomized algorithms for computation of tucker decomposition and higher order svd (hosvd). IEEE Access, 222684-28700 2021.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms. arXiv preprint arXiv:
2019.
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:240.110024 2024.
Davide Bacciu and Danilo P. Mandic. Tensor decompositions in deep learning.
ArXiv,
abs/2002.11833, 2020.
URL https://app semanniccccooaat.ooooooooooupp
211532702
Abhinav Bandari, Lu Yin, Cheng-Yu Yu Hsieh, Ajay Kumar Jaiswal, Tianlong Chen, Li Shen, Ranjay Krishna, and Shiwei Liu. Is c4 dataset optimal for pruning? an investigation of calibration data for I1m pruning. arXiv preprint 2rXiv:2410.074611 2024.
Casey Battaglino, Grey Ballard, and Tamara G. Kolda. A practical randomized cp tensor decomposition. SIAM J. Matrix Anal. Appl., 39:876-901, 2017. URL /https://api.
semaant i cscho ar. org/ Corpus ID: 187688765555 69481.
Albrecht Böttcher and David Wenzel. The frobenius norm and the commutator. Linear algebra and its applications, 429(8 (0088-9):188 99:1864-1883 2008.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411 2021.
Shail Dave, Riyadh Baghdadi, Tony Nowatzki, Sasikanth Avancha, Aviral Shrivastava, and Baoxin
Li. Hardware acceleration of sparse and irregular tensor computations of ml models: A survey and insights. Proceedings of the IEFF, 109(10): 170-1172. 2021.

---

Published as a conference paper at ICLR 2026
Vin de Silva and Lek-Heng Lim. Tensor rank and the ill-posedness of the best low-rank ap- proximation problem. 2006. URL htLthtps::/// h/aps:://pp.. www...sssmattccca.r...coo http:/cchhc..aa.oooooorsrsppppaa
265039395
Andreas Defant and Klaus Floret. Tensor norms and operator ideals, volume 176. Elsevier, 1992.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- Irika, (3):211-215) 1936.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International conference machine learning, pp. 00323-10337 PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:22 22..:3333, 2022.
Qichen Fu, Minsik Cho, Thomas Merth, Sachin Mehta, Mohammad Rastegari, and Mahyar Na- jibi. Lazyllm: Dynamic token pruning for efficient long context Ilm inference. arXiv preprint arXiv:2407.14057, 2024.
Zihao Fu, Wai Lam, Qian Yu, Anthony Man-Cho So, Shengding Hu, Zhiyuan Liu, and Nigel Collier. Decoder-only encoder-decoder! interpreting language model regularized encoder-decoder. arXiv preprint arXiv:2304.044000 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Shawn Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jasmine Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9, 2021.
Shangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Disp-lIm: in Neural Information Processing Systems, 37:72219-722444 2024.
structural pruning for large language models. Advances
Albert Gatt and Emiel Krahmer. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65-170, 2018.
José Henrique de Morais Goulart, Mélanie Boizard, Remy Boyer, Gérard Favier, and Pierre Comon. Tensor cp decomposition with structured factor matrices: Algorithms and performance. IEEE Journal of Selected Topics in Signal Processing, 10(4):681-622, 2016. doi: 10. 1 1090/SSSSP....... 2562477.
Michael Greenacre, Patrick JF Groenen, Trevor Hastie, Alfonso lodice d'Enza, Angelos Markos, and Elena Tuzhilina. Principal component analysis. Nature Reviews Methods Primers, 2(1):100, 2022.
Qiuhan Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 2201-2203, 2023.
Yuxuan Gu, Wuyang Zhou, Giorgos lacovides, and Danilo Mandic. Tensorlim: Tensorising multi- head attention for enhanced reasoning and compression in llms. arXiv preprint arXiv:2501.15674, 2025.
Qiang Heng, Chi, and Yufeng Liu. Tucker-12e: Robust low-rank tensor decomposition with the 12 criterion. arXiv: 2208. / /806 v/, 2022.
Qiang Heng, Eric C Chi, and Yufeng Liu. Robust low-rank tensor decomposition with the crite- rion. Technometrics, 657555537-552, 2023.
Cunchen Hu, Heyang Huang, Junhao Hu, Jiang Xu, Xusheng Chen, Tao Xie, Chenxi Wang. Sa Wang. Yungang Bao, Ninghui Sun, et al. Memserve: Context caching for disaggregated Ilm serving with elastic memory pool. arXiv preprint arXiv:2406. 17565, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zit Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
12

---

Published as a conference paper at ICLR 2026
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for 1lms. arXiv preprint arXiv:2402.04291, 2024.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXtt:2303.053396 2023.
Jun-Gi Jang and Kang. D-tucker: Fast and memory-efficient tucker decomposition for dense tensors. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 1850- 1853. IEFE, 2020.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev. 51: 455-500, 2009. URL WtR.https:: /tte:///ppp. http://aa....eeneeeeeee.eee.e
ID: 6074199
Wim P. Krijnen, Theo K. Dijkstra, and Alwin Stegeman. On the non-existence of optimal solutions and the occurrence of "degeneracy" the candecomp/pprrfac model. Psychometrika, 73:431 439, 2008. URL https://: www.nt.eei..... escho ar hrr.gh/o......ggooooorri1:111111
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611 1-666, 2023.
Minhyeok Lee. Mathematical analysis and performance evaluation of the gelu activation function in deep learning. Journal of Mathematics, 2023(1):4229924, 2023.
Qi Li, Zhiping Qiu, and Xudong Zhang. Eigenvalue analysis of structures with interval parame- ters using the second-order taylor series expansion and the dea for qb. Applied Mathematical Modelling, 49:680-690, 2017.
Lieberman, Fidkowski, Willcox, and van Bloemen Waanders. Hessian- model re- duction: lcage-scale inversion and prediction. International Journal for Numerical Methods in Fluids, 71(2):33--150 2013.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation- aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 6:87-100, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 1rXiv:2109.07900 2021.
Yuanyuan Liu, Fanhua Shang, Wei Fan, James Cheng, and Hong Cheng, Generalized higher-order orthogonal iteration for tensor decomposition and completion. Advances neural information processing systems, 27, 2014.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 2023.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations. arXiv preprint arXiv:2405. 20044400066666 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Osman Asif Malik and Stephen Becker. Low-rank tucker decomposition of large tensors using tensorsketch. Advances in neural information processing systems, 31, 2018
Pontus Matstoms. Sparse linear least squares problems in optimization. Computational optimization and applications, 7(1):9--110, 1997.
13

---

Published as a conference paper at ICLR 2026
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv: 2016.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Morten Mørup, Lars Kai Hansen, and Sidse M Arnfred. Algorithms for sparse nonnegative tucker decompositions. Neural computation, 20 2008.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910. 14599, 2019.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. Tensorizing neural networks. In Neural Information Processing Systems, 2015. URL https://api. semanticscho haaplaorr/ooooooo ott/CorppsII::9900000
I. Oseledets. Tensor-train decomposition. SIAM J. Sci. Comput., 33:2295-2317, 2011a. URL https://apt www.settiiccchie ar. wwtpcchllarrrr//ooroooeooo www.or/o///oo....ooooooo..o..ooo 207059098
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- b0
Pentti Paatero. Construction and analysis of degenerate parafac models. Journal of Chemometrics, 14, 2000. URL http:://ppp.iii seemantt nt. cschol ar. www.ooaa.rrrrroooooooooooooooeoe CorpuIt:: 122299052
Moonjeong Park, Jun-Gi Jang, and Lee Sael. Vest: Very sparse tucker factorization of large-scale tensors. In 2021 IEEE International Conference Big Data and Smart Computing (BigComp), pp. 172-179. IEEE, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large lan- guage models. arXiv preprint arXiv:2310.00034 2023.
Qingquan Song, Hancheng Ge, James Caverlee, and Xia Hu. Tensor completion algorithms in big data analytics. ACM Transactions on Knowledge Discovery from Data (TKDD), 13:1 48, 2017. URL httph::tt//attt www.nt
org. wwtpooorp/ooor::o.oooooo 19060811
Mengyang Sun, Yihao Wang, Tao Feng, Dan Zhang, Yifan Zhu, and Jie Tang. A stronger mixture of low-rank experts for fine-tuning foundation models. arXiv preprint arXiv:2502. 15828, 2025.
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effecttve pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Eszter Szekely, Lorenzo Bardone, Federica Gerace, and Sebastian Goldt. Learning from higher- order correlations, efficiently: hypothesis tests, random features, and neural networks. Advances in Neural Information Processing Systems, 37.78479-78522. 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. htlps://crfm. stanford. edu/2023/33/1lllpaas html, 3(6):7, 2023.
Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150):1-63, 2021. 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
14

---

Published as a conference paper at ICLR 2026
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: (rXivv/905.09440 09418, 2019.
Ben Wang. Mesh-Traastfrmer-JAX Model-Parallel Implementation of Transformer Language Model with JAX. ortr:///.hhhhaooooo////o/oooooo/ http:///a.aaaaaooooooooaaaaaatar
wwt.
transformer jax
May 2021.
Jingeun Wang, Yu-Guang Chen, Ing-Chao Lin, Bing Li, and Grace Zhang. Basis sharing: Cross- layer parameter sharing for large language model compression. arXiv preprint arXiv:2410.03765, 2024a.
Xin Wang, Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377. 2024b.
Xin Wang, Samiul Alam, Zhongwei Wan, Hui Shen, and Mi Zhang. Svd-Ilm v2: Optimizing sin- gular value truncation for large language model compression. arXiv preprint 120:22003.13340, 2025.
Ying Wang and Yuning Yang. Hot-svd: higher order t-singular value decomposition for tensors based on tensor-tensor product. Computational and Applied Mathematics, 4 41(8))394, 2022.
Ziheng Wang. Sparsert: Accelerating unstructured sparsity on gpus for deep learning inference.
In Proceedings of the ACM international conference on parallel architectures and compilation techniques, pp. 31-42, 2020.
Laurenz Wiskott. Lecture notes on principal component analysis, 2013.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
Shufang Xie, Huishuai Zhang, Junliang Guo, Xu Tan, Jiang Bian, Hany Hassan Awadalla, Arul Menezes, Tao Qin, and Rui Yan. Residual: Transformer with dual residual connections. arXiv preprint arXiv:2304. 14802, 2023.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International conference on machine learning, pp. 10524-10533. PMLR, 2020.
Mingxue Xu, Yao Lei Xu, and Danilo P Mandic. Tensorgpt: Efficient compression of large language models based tensor-train decomposition. arXiv preprint arXiv:2307.00522. 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang. Shuo Wang. Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. Advances in Neural Information Processing Systems, 37:66377-66382. 2024.
Rumming Yang. Taiqiang Wu, Jiahao Wang, Pengfei Hu, Yik-Chung Wu, Ngai Wong, and Yujiu Yang. Llm-neo: Parameter efficient knowledge distillation for large language models. arXiv preprint S0t.::2411.06822.. 2024.
Yifan Yang, Kai Zhen, Bhavana Ganesh, Aram Galstyan, Goeric Huybrechts, Markus Müller, Jonas M Kübler, Rupak Vignesh Swaminathan, Athanasios Mouchtaris, Sravan Babu Bodap- ati, et al. Wanda++: Pruning large language models via regional gradients. arXiv preprint arXiv:2503.04991 2025.
Zhihang Yuan, Yuzhang Shang. Yue Song. Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-award singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
15

---

Published as a conference paper at ICLR 2026
Zhongshen Zeng, Pengguang Chen, Shu Liu, Haiyun Jiang, and Jiaya Jia. Mr-gsm8k: A meta- reasoning benchmark for large language model evaluation. arXiv preprint arXiv:2312.10000, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint wiXiv2205.010665 2022.
Zhengyan Zhang, Yixin Song, Guanghui Yu, Xu Han, Yankai Lin, Chaojun Xiao, Chenyang Song, Zhiyuan Liu, Zeyu Mi, and Maosong Sun. Relu wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXiv:2402.00 03.0040404.4 2024.
Jialin Zhao, Yingtao Zhang, and Carlo Vittorio Cannistraci. Pivoting factorization: compact meta low-rank represetttion of sparsity for efficient inference in large language models. arXiv preprint arXiv:2501.19099 2025.
Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate IIm serving. Proceedings of Machine Learning and Systems, 6:196-209, 2024.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your Ilms behave over infinitely increasing context length and reasoning complexity? arXiv preprint 25fv:25022.02522. 2025.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang- A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 1556-1577, 2024.
16

---

Published as a conference paper at ICLR 2026
A DATASET
WikiText-2 (Metrie: Perplexity 4). A word-level language-modeling corpus built from Wikipedia "Good/Featured" articles that preserves case, punctuation, and numbers. The standard split contains 2.09M tokens (train). 217.6k (validation), and 645.6k (test) with 33,278-word vocabulary, we eval- uate by perplexity (lower is better).
MathQA (Metric: Accuracy t). A multiple-choice math word-problem dataset (R:37k items) cre- ated by annotating AQuA problems with Seeration-programs. Authors use an 80/12/8 random split, i.e., 229.8k train/4.4k validation/3k test, and we report answer accuracy on the evaluation set. GSM8K (Metric: Accuracy 1). A grade-school math benchmark of 8.5k authored problems tar- geting multi-step reasoning. Canonical split is 7.5k train Ik test. We follow the original protocol and report exact-match accuracy of the final numeric answer.
TruthfulQA (Metric: BLEU-1 1). A truthfulness benchmark of 817 questions across 38 categories (e.g., health, law, finance, politics). There is no official train/val/test split, evaluation is conducted on the full question set via multiple-choice and/or free- tree-generation protocols. In our setup, we compute BLEU-1 (unigram BLEU) between model outputs and the provided references.
B BASELINES AND LESTD DETAILED SETTING
ASVD decomposes the attention weights with activation- aware scaling and a layerwise rank search, in the original paper, auhtor experiments use small, fixed calibration corpus drawn from WikiTexx-- 2: 32 calibration datasets each containing 2048 tokens. We keep the exact same calibration budget, and choose per-layer truncation ranks to meet the desired global compression ratio (e.g.. 0.2 0.8).
SVD-LLM introduces truncation-awarc whitening plus sequential low-rank updates (SLRA). Its im- plementation details specify that whitening//runcationn are fit on a random set of sentences with con- crete budge of 256 samples, SLRA recovery is performed with instruction data (Alpaca 50K (Taori ct al. 2023)). ). In our experiments, we use 256 calibration sentences (from WikiText-2) with sequence length 2028 for whitening and truncation, and enable SLRA with SVD-LLM's defaults.
SliceGPT performed column slicing guided by second-order/ layle criterion (Li al. 2017) and calibrates on either WikiText-2 or Alpaca. The paper fixes the main experiment calibration to 1024 sequences of length 2048, and uses the LM-Eval Harness (Gao et al. (02TT) defaults for the zero-shot tasks. Slicing levels reported across models are 10-30% with optional LoRA A-based recovery fine- tuning (with r = 32, a = 10). To remain the faithful across MathQA, GSM8K and TruthfulQA (not originally supported in the paper). we keep the same calibration budget (1024 x 2048 on WikiText- 2), set a target slicing fraction that match the compression grid (0..2 0.4, 0.6, 0.8] overall by solving for per-layer variance removal in the paper), and evaluate with LM-E defaults.
TensorLLM factorizes multi-head attention (MHA) with a shared-factor Tucker decomposition and is explicitly trstniraining, requiring no extra data, calibration, fine-tuning (Hu et al., (022). The only operative knobs are the multi-linear ranks (equivalently a per-layer MHA compression rate), which the paper sweeps broadly (e.g., on GPT-J, reported the MHA compression rates span 1= 247X depending on the layer/dataset) while keeping standard evalution. For all four datasets, we reuse our dense prompts and simply set Tucker ranks hit the target global compression ratios on the attetion block (start with 8- 16X on Q, K, V and 2 6X on output projections for decoder-only models, then adjust to match the overall ratio).
For two 1-bit quantization baselines (for experimental results, please refer to Appendix D). BiLLM and PB-LLM, emphasize ligh-weight post- post-training calibration on generic text. BiLLM quantize us- ing C4 (Bandari et al.) 2024) as the calibration source with blocksize 128 and the Hessian criterion for salient-weight selection. PB-LLM partially binarizes weights while preserving a small salient subset and use GPTQ-style (Frantar et al. (2022) bussian-guidee reconstruction. In our experiments, we use PTQ-only settings across four datasets, the choice is to mirror BiLLM: take 128- 512 cal- ibration sequences from C4 (length is 2048), use blocksize 128 and Hessian saliency for BiLLM, and for PB-LLM retain ~ 5 - 10% salient weights in higher precision with GPTQ-sttle Hessian reconstruction, and keep dataset prompts at LM-Eval defaults.
For our proposed LeSTD, we follow same evaluation setup as described above, but note that LeSTD is fully data-free post-training method (no calibration data retraining is used). We set the multi-
17

---

Published as a conference paper at ICLR 2026
linear Tucker ranks (R1. R2, Rs) in each attention layer so that the overall model meets the target compression ratio (0.2, 0.4, 0.6, 0.8). Concretely, for each desired global size fraction, we solve for the layer-wise ranks that yield approximately that compression of the MHA block. The Stage I (Section [31 shared Tucker factorization) is computed with HOOI using a fixed number of iterations (we typically run 5-10 iterations per layer, which reliably converges to low reconstruction error). After Stage I, Stage II (Section 3.2) performs importance-baseed pruning on the core tensor as fol- lows: we fix per-iteration prune fraction a 0.5 (i.e., remove 10% of the smallest-magniuud core elements each round). We continue pruning iteratively until the core has reached the target sparsity (chosen so that the final compressed size matches the target ratio). After each pruning step. we refit the remaining core coefficients by the closed-form update (6) to counteract numerical drift, this ensures optimal reconstruction with the fixed basis. summary. LeSTD requires no fine-tuning or task-specific data. All hyperparameters (Tucker ranks, pruning schedule) are determined by the global size target. Once the compression is done, inference proceeds directly on the compressed representation as described. In our experiments, we found that this setting reliably achieves the desired compression at the indicated performance levels, without using any extra data beyond what the evaluation tasks provide. Please also refer to Appendix for detailed explanation of how the compression ratio of LeSTD is precisely computed.
C ABLATION STUDIES
C.I RANK-SENSITIVITY
To gauge the sensitivity to Tucker ranks R2, Rs = 4),
target an overall MHA compression ratio around 0.6 and vary (R,, R₂) while keeping R3 fixed (since rank R3 rep- resents the Q, K, V axis of size 4). Table 2 reports sev- eral representative configurations (the "Ratio" column shows the actual compression fraction achieved), and perplexity is measured on WikiText-2 with GPT-J (6B). This is not an ex- haustive grid search, but it already covers a reasonably wide
Table 2: Perplexity under vary- ing Tucker ranks for GPT-J (6B) at 0.6 overall compression.
R1
R3
Ratio
Perplexity (U)
4096
256
8.86
1536
320
0.56
13.72
range of (Ra, R2) in the ~ 0.6 regime and illustrates clear
1536
384
4
0.66
11.91
1792
320
0.65
11,33
trends: larger ranks generally yield lower perplexity. In par- ticular, increasing R₁ R₂ consistently reduces perplexity, and configurations such as (R₃ = 2048, Rs = 256, R3 = 4) and (R1 2048, R2 288, R3 4) achieve near-original perplexity (down to 9.26) at compression ratios around 0.63-0.66 whereas smaller ranks (e.g., R, 1536, R₂ 320, R3 4) lead to noticeable degradation. Overall, the best perplexity in this sweep occurs at higher ranks near the 0.6 compression point, confirming that our reported setting (Table T lies in regime that balances model size and accuracy.
2048
224
0.56
10.53
2048
256
4
0.63
9.53
1920
256
0.60
2048
288
4
0.66
9.26
C.2 EFFECT OF STAGE-II ON CORE STRUCTURE
We adopt a shared-subspace Tucker decomposition with orthonormal factors (())U(()) U33) along the corresponding modes, respectively, while leaving the head mode unfac- torized. For display, as shown in Figure (. we keep R3 (Q, K, V) fixed and a particular head i 5, and show the resulting R1 x R2 core slice. Both panels share a diverging colormap centered at zero for strict comparability. performs best-k pruning in the same orthonormal basis by magnitude and refits the surviving coefficients. Because the basis tensors orthonormal, zeroing single coefficient ga increases the Frobenius error by exactly g so hard-thresholding by [gsl is optimal for best-k approximation. On this GPT-J slice, Stage II keeps
0.9
0.0
0.50
82
R2
Figure 6: Visualization of fixed Tucker-core slice before and after Stage II sparsification.
Stage
18

---

Published as a conference paper at ICLR 2026
the top 30% coefficients (nnz 87/288, 69.8% sparsity) while retaining 87.2% energy, visibly concentrating mass onto a small set of latent interactions.
C.3 WALL-CLOCK TIME BREAKDOWN
To quantify the practical overhead of our two- stage compression pipeline, we measure the of- fline wall-clock time of LeSTD on GPT-J and
12000
Wall-clock time (seconds)
Stage
OPT-30B under the same hardware and soft- ware setup as in Section In this experiment, apply LeSTD only the MHA blocks of all transformer layers (MHA-onlly) using the same
10000
Stage
8000
6000
rank and sparsity configuration in our main results (compression rate = 0.2).
4000
Figure summarize the breakdown, For GPT-
2000
J, Stage takes 883 s, while Stage II takes 436 for a total of 1319 For OPT-30B, Stage takes 9102s and Stage II takes 2385s, for a to- tal of 11487 s. The relative share shifts from roughly 67%/33% (Stage I/Stage II) GPT- J to about 79%/21 % on OPT-30B, confirming that Stage dominates the offline cost model depth and hidden dimension grow. Importantly, this cost is incurred only once as an offline pre- processing step. Even for OPT-30B, the total LeSTD runtime remains the order of few hours, which is negligible compared to the cost of pretraining or repeated fine-tuning runs on such models. After Stage I and II are completed, inference no additional runtime overhead beyond the speedups reported in Section 4
0
GPT-J
OPT-30B
Figure 7: Wall-clock time breakdown of LeSTD on GPT-J and OPT-30B.
the compressed weights directly and introduces
Table 3: Perplexity and throughput (tokens/sec) of GPT-J (6B) on WikiText-2 at compression ratio 0,6 over 10 runs. Results are reported as mean + standard deviation. Lower perplexity and higher throughput are better.
Method
Perplexity (L)
Tokens/sec (t)
SVD-LLM
20.28 0.14
11105.18 90.65
TensorLLM
9.87 + 0.10
8572.11 + 103.18
LeSTD (Ours)
9.48 + 0.09
12025.90 = 91.77
C.4 STATISTICAL SIGNIFICANCE
To evaluate the robustness our observations in main evaluation (Section 4.2 2, we repeat the com- pression procedure 10 times with different random seeds for GPT-J (6B) on WikiText-2 compression ratio of 0.6. We report the mean and standard deviation across runs for both perplex- ity and throughput (tokens/sec) in Table 3. comparing LeSTD against the strongest matrix-based baseline (SVD-LLM (Wang et al., 2024b)) and the dense-core Tucker method (TensorLLM (Gu al. 2025)). As shown Table LeSTD attains mean perplexity of 9.48 + 0.09, compared to 9.87 + 0.10 for TensorI.L and 20.28 + 0.14 for SVD-LL.M The variance across runs is small for all methods, and the gap between LeSTD and TensorI.L.M substantially larger than one stan- dard deviation, indicating that the accuracy improvements of LeSTD are not due randomness in the compression process. For efficiency, LeSTD achieves 12025.90 91.77 tokens/sec, which is comparable to slightly higher than SVD-LLM (11105.18 + 90.65) and significantly faster than TensorLLM (8572.11 1177.111:: 1112111+111 100.188. The standard deviations are again small relative to the mean values, showing that the throughput advantages of LeSTD are stable across seeds.
fixed
Overall, these results demonstrate that the performance gains in the both perplexity and throughput provided by LeSTD are consistent across multiple runs and not artifacts of particular random seed or single compression trial.
C.5 PRUNING RATE SENSITIVITY
19

---

Published as a conference paper at ICLR 2026
Stage II of LeSTD performs iterative importance--asse pruning on the shared core tensor g. At each iteration,
9.8
Perpiexity (Wik) 2)
compute importance score for every non-zero co- efficient g/ and remove the least important ones, followed by closed-form refitting step (Algorithm I The prun- ing rate a controls how aggressively we prune at each it- eration: given & non-zero coefficients, we remove [a- k]
9.7
9.6
of them and refit the remaining ones. Importantly, a does not determine the final sparsity of g, which is fixed by the target sparsity Surget derived from the desired compres- sion ratio. Instead, a trades off the number of pruning iterations and the aggressiveness of each step.
9.5
Pruning rate
Figure 8: Effect of pruning rate a in Stage II for GPT-J (6B) on WikiText-2 at compression ratio 0.6. We fix target core sparsity Starget and sweep a from 0.05 to 0.50. Reported numbers are WikiText-2 perplexity (1).
To evaluate the sensitivity to a, we fix the target core spar- sity according to a compression ratio of 0.6 for GPT-J (6B) on WikiText-2 and sweep a from 0.05 to 0,50 in steps of 0.05. Figure & reports the resulting perplexity on WikiText-2. We observe that LeSTD is very robust to the choice of a in broad range: for a E [0.05, 0.30), perplexity stays in the narrow range of 9.49-9.54, indicating that our importance-baset pruning and refitting procedure consistently con- verges to nearly the same solution under different per-iteration pruning rates. When is made more aggressive (c.g., 2 0.35), the perplexity gradually increases (up to 9.82 at = 0.50), which we attribute pruning larger fraction of coefficients before the refitting step has fully adjusted the remaining ones. Even in this regime, LeSTD remains noticeably better than matrix-wise SVD and dense-core Tucker at the same compression ratio.
Table Effect of the refitting step (Eq. (6) Stage for GPT-J (6B) on WikiTexx-2 at compression ratio 0.6. We compare the full LeSTD pipeline with variant that skips the refitting step while keeping the same target core sparsity Starget
Method
Perplexity (L)
Tokens/sec (t)
LcSTD (full)
9.57
11925.90
LeSTD (w/o refit)
11.25
11932.45
C.6 REFITTING STEP IMPORTANCE
Stage II of LeSTD performs importance-based pruning on the shared core tensor G in an iterative fashion (Algorithm D. At each iteration, we (i) compute an importance score for every non-zero coefficient ga. (ii) remove the least important [a k] coefficients among the current k non-zeros, and (iii) refit the remaining coefficients by solving a closed-form least-squares problem (Eq. (6). Thus, Eq. 66) corresponds to debiasing step on selected support: given the current sparsity pattern of 9, we recompute the values of the active entries so that the pruned core best approximates the original weights in the least-squares sense.
To quantify the effect of this refitting step, we compare full LeSTD pipeline with a variant that skips Eq. (6): in the ablated variant, we perform the same iterative importance-baset pruning but never refit g after pruning, instead, we simply zero out the selected coefficients and keep the remaining ones fixed. We fix the target core sparsity Surget according (6B) on WikiText-2, and we run both variants ten times with different random seeds. Table 4 reports perplexity and inference throughput. As shown in Table 4 skipping the refitting step consistently hurts accuracy: perplexity degrades from 9.57 for the full LeSTD to 11.25 for the variant without refitting, a gap that is larger than several standard deviations and thus statistically significant. This confirms that Eq. (6) is essential debiasing step that takes advantage of the selected sparse struc- ture to recover a better approximation of the dense core. In contrast, the inference throughput is essentially unchanged (11925.90 sparsity pattern and Tucker ranks; the difference between them lies purely in how well the non-zero coefficients are fitted.
compression rate of 0.6 for GPT-J
11932.45), since both variants produce cores with the same
Overall, these results show that the refitting step in Eq. 66) plays a crucial role in obtaining high- quality compressed models, while having negligible impact inference-time efficiency.
20

---

Published as a conference paper at ICLR 2026
Table Effect of replacing original used torch. mmatuul with torch. sparse mm LeSTD's sparse core tensors for GPT-J on WikiText-2 at MHA compression rate 0.6.
Kernel / Implementation
Throughput (tokens/sec)
torch matmul (Current Implementation) Unstructured sparse (torch. sparse mm))
11945.98 12592.54
C.7 SPEEDUP FACTORS ANALYSIS
In this subsection, we explore the question that whether the throughput gains reported in Figure might simply come from PyTorch (Paszke et al.] (2019) silently exploiting unstructured sparsity i our sparse cores. To isolate this factor, we replace original torch. ma maatmml--ae lbased implementa- tion of LeSTD's core contractions with PyToocch's unstructured sparse kernel torch, spa rse. ri,, while keeping all other components fixed (model: GPT-J (6B), dataset: WikiText-2, MHA com- pression rate: 0.6, hardware and batch/sequence settings identical to the main experiment, refer to Section for details).
Table 5 shows that the baseline dense-kernel implementation reaches 11945.98 tokens/sec, whereas the version that forces torch.. sparse se mm achieves 12592.54 tokens/sec, corresponding to a very modest ~ 5.4% relative speedup. In contrast, under the same setting LeSTD much larger throughput gain over the full-precision GPT-J baseline (cf. Figure 5). This indicates that the bulk of acceleration does not in PyTorch, but rather from the structural changes introduced by LeSTD, in particular, sharing the input-side projection across heads and performing attention much lower-dimensional, sparsified core space, which jointly reduce both FLOPs and memory traffic of the MHA block.
whole achieves
from any special native support for unstructured sparsity
D COMPARISONS WITH QUANTIZATION
Table 6: Throughput (tokens/sec) and accuracy (perplexity) comparison of our proposed LeSTD against quantization baselines PB-LLM and BiLLM the WikiText-2, evaluated with Llama2- 13B and OPT-30B.
Compression Rate
Llama2-13B Perplexity (J)
Llama2-13B Tokens/see (*)
OPT-30B Perplexity (L)
OPT-30B Tokens/sec (t)
Method
Sctting
(compeeseed orggina
LeSTD
0.2
13.99
15175.43
42.70
8545.22
LeSTD
0.4
0.4
11.94
11147.18
20.13
8447.63
LeSTD
0.6
0.6
7.93
12147.18
14.02
7139.44
LeSTD
0.8
6.98
11864.59
9.98
4122.30
BiLLM
PTQ 1.08-bit
~0007
9.96
9732.96
12.71
3856,80
PB-LLM
PTQ
salient 8-bit
~0.17
11.77
10839.19
14.75
4020.87
PB-LLM
PTQ 30% salient 8-bit
00.6
9.95
11666.06
13.70
3726.39
As shown in Table 6 putting the LeSTD side-by-side with SOTA post-training quantizers highlights a complementary trade-off rather than a simple win or lose. BiLLM drives weights to an average of 1.08 bits and reports strong perplexity extreme compression, while PB-LLM partially bina- rizes by reserving higher precision for salient weights. Both approaches achieve very small model footprints (7-16% of the FP16 size) with competitive accuracy on WikiText-2. The advantage in size does not always translate into throughput: at comparable accuracy targets, LeSTD's 's factorized and sparsified attention routinely runs faster. For example, at comparable or stronger compres- sion settings on Llama2-13B and OPT-30B, LeSTD's direct execution in the shared subspace yields 1.3-2.2XX higher tokens/sec than the 1.08-bit BiLLM and a consistent lead over PB-LLM. depsite the latter operating at lower bitwidths. This is consistent with the compute scheme of the methods: LeSTD replaces three large input-side GEMMs with a single shared projection and contracts head- local sparse cores, cutting memory traffic that often dominates short-context inference. Conversely, altra-loo-bii PTQ still incurs packing/unpacking overheads and kernel inefficiencies that can bound realized throughput even when the byte footprint is smaller.
21

---

Published as a conference paper at ICLR 2026
Accuracy-throughpe also differ in shape. Quantization reach lower perplexity at most aggres- sive size targets, especially when binarization is carefully structured (BiLLM's salient/non- when PB-LLM retains 8-bit precision subset of critical weights (refer to Appendix [B. LeSTD, by contrast, maintains perplexity close to full- precision baselines at moderate compressions while delivering higher tokens/sec with calibration data and without custom kernels. Because LeSTD's Stage II pruning operates in an orthonormal Tucker latent space, pruning criterion admits closed form and preserves the most influential inter- actions, empirically this sustains accuracy as compression strengthens, while it sparse core directly reduces runtime FLOPs and bandwidth. Taken together, results suggest the techniques are orthogo- nal: LeSTD offers a strong operating point for deployment scenarios prioritizing throughput under moderate size budgets, whereas extreme low-bit PTQ excels when model footprint first constraint.
treatment and binary residual approximation)
E DETAILED RELATED WORK
Data-free post-training compression methods for LLMs can be broadly grouped into low-rank, prun- ing, and quantization, typically in the calibration-lit setting. Recent surveys (Zhu et al. (024)) syn- thesize these lines and emphasize the practical importance of hardware-reallzable structure and end- to-end throughput when comparing methods. In what follows, we detail representative approaches in each category: SVD-style and tensorized decompositions, channe//feature-levd structured prun- ing, and weight/activation quantization including extreme 1-bit schemes, highlighting their design trade-offs and complementarity.
E. POST-TRAINING LOW-RANK DECOMPOSITION
A large body of research works compress LLM weights via training-free SVD-style factorizations. ASVD (Yuan et al., 2023) introduces snivattin-awarg reparameterization to absorb activation out- liers into weights and performs layer-sensitive truncation with lightweight calibration, thus yielding 10-30% model size reduction. SVD-LLM (Wang et al. (2024b) makes the truncation explicitly loss- aware by whitening activations so that the singular values align with approximation error, it further applies sequential low-rank updates mitigate post-truncation degradation, and its following tension, SVD-LLM-v2 /Wang et al. 2025). assigns per-layer compression via thcoretical truncation loss and improves the numerical pipeline. Beyond matrix SVD, tensorization exposes higher-order structure. TensorL.M tensorizes multi-head attention and applies Tucker decomposition with shared subspaces across heads, reporting substantial compression (up 250X in MHA weights) and ac- curacy benefits without specific dataset, related efforts (Xu et al. 2023) explore tensor-train (MPS) decompositions (Oseledets) 2b) for token embeddings cut memory and latency.
E.2 POST-TRAINING STRUCTURED PRUNING
SliceGPT (Ashkboos et al. 2024) replaces each weight matrix with smaller dense sub-matrix via feature/channel selection, producing storage and wall-clock gains without retraining, highlights that structured sparsity patterns are critical for realizing speedups on real kernels. DISP-LLM (Gao ct al., 2024) relaxes rigid sharing by allocating different sub-spaces to diffcrent layers, strength- ening the performance- cost trade-off for structural pruning at high sparsity ratio. Unstructured one-shot pruning: SparseGPT solves a sequence of layerwise sparse-regression problems to prune 50%+ of parameters in single pass very large GPT's (Frantar & Alistarh 2023). Wanda (Sun) eet al,, 2023) prioritizes weights by magnitude-activan subsequent work: Wanda++ (Yang et al. 2025) adds regional gradient signals for better robustness. While unstructured sparsity may under-utilize hardware, these works motivate structured designs (e.g.. block/column pruning) translate sparsity into the throughput, principle adopted by newer structural approaches, for example, SliceGPT (Ashkboos et al. 2024).
products for fast one-shot pruning, and
E.3 POST-TRAINING QUANTIZATION
Post-training quantization (PTQ) quantize weights and/or activations without task-specific finetun- ing. GPTQ (Frantar et al. 2022) performs one-shot, second-order weight-only quantization (3-4 4 bit) via approximate Hessian inversion (Lieberman et al. 2013) and demonstrates minimal accu- racy loss and end-to-end speedups on large GPTs. AWQ LLn al. 2024) is an activation-aware,
22

---

Published as a conference paper at ICLR 2026
hardware-friendle welght-only PTQ that protects salient channels using tiny calibration batches, abling low-bit deployment across families of LLMs with strong accuracy. Building on this line, PB- LLM (Shang et al. 2023) introduces partial binarization: binarizing most weights while reserving small salient subset at higher precision, and couples it with PTQ/QAT variants (e.g.. GPTQ-guided stabilize ultra-low-hit regimes. Pushing further, BiLLM (Huang al.) 2024) presents a 1-bit PTQ pipeline with salient weight selection, binary residual approximation, and optimal splitting of non-salient weights, reporting competitive perplexities (e.g., on LLaMA2) at 1-1.1 1 bit per weight and efficient binarization of multi-billin billion--aaaeeee parameter models, highlighting the role of saliency-awar awarre protection and structured binarization in preserving accuracy under aggres- sive compression.
Hessian reconstruction)
E.4 DIFFERENT TENSOR DECOMPOSITION FORMAT
Our method is built on a Tucker decomposition with a sparse core. In this section, we briefly discuss why adopt Tucker as the underlying topology and how it relates alternative tensor formats and structured Tucker variants.
CP decomposition. Given N-way tensor X' E R Rxxxxxxy the CANDECOMP/PARAFAC (CP) decomposition (Goulart et al. 2016 Battaglino et al. 2017)
writes
f
X R
a (N) 1""
where ( E 'R is the r-th column of a factor matrix (a) E. R..xR and o denotes the outer product. This representation is extremely parameter-efficient (all information is stored in the factor matrices), but each rank-1 term couples all modes through a single sealar coefficient, which limits expressiveness. It is well documented in the tensor-decomposition literature (Kolda & Bader [009 Song et al. 2017: de Silva Lim 2006; Krijnen et al.) 2008 Paatero 2000) that CP often requires substantially larger rank than Tucker to achieve comparable approximation error, and that CP fit- ting can be prone to degeneracy and ill ill-conditioning, especially for complex or nearly collinear data. In LeSTD, construct 4-way "joint MHA" tensor whose modes encode the model dimension, head dimension, projection dimension (Q/K/V/O), and head index. Our design goal is to learn shared subspaces along the first three modes (shared across all heads), while allowing per-head mixing to be captured in small core. A Tucker model
W x1 U() X2 U X3 (3) x, U
(8)
with (11), (2)), (3) shared and mode-4 slices of the core 9 playing the role of per-head param- eters, matches this pattern directly. In contrast, a CP model would express W as a sum of rank-1 terms where each component simultaneously couples all four modes. To preserve per-head struc- ture. one would either need to (i) tie the head-mode factors in a way that limits flexibility, or (ii) significantly increase the CP rank to recover head-specific patterns. Both options undermine the compression objective: the former sacrifices expressiveness and may hurt accuracy, while the latter expands the parameter budget and FLOPs. For this reason, we view Tucker as a more natural com- promise between expressiveness and compactness for our 4-way MHA tensor.
Tensor Train (TT) decomposition. The Tensor Train (TT) format is particularly attractive for very high-order tensors with many small modes (Oelleets
201 Given X' R TT writes
X (ii,..... (i1,...,iNy iN) ((1))) g()) (1, ... .....G())) ((((),)))))
where each ((1) 3-way core tensor with TT-ranks ro = 1, = 1. This long-chain topology is well suited for high-order problems such as quantum many-body systems or large struc- tured grids, where N is large and each I, is relatively small. In our setting, however, the joint MHA tensor is loweorder (four modes) but with semantically structured dimensions: model dimen- sion, head dimension, projection dimension, and head index. Mapping this tensor into a TT chain would require non-trivial reshaping and ordering choices (Novikov al., 2015 Bacciu & Mandic, 2020). and it becomes less clear how to (i) enforce a shared subspace across heads in a simple way, and (ii) maintain per-head cores with a straightForward interpretation. In contrast, the Tucker topology we adopt directly encodes a shared-basis and per-head-core structure: the factor matri- ces ( (2))), (3) span shared subspaces along the first three modes, and her-head variations are captured compactly in the mourth-mode slices of the dense core g. For these reasons, we prioritize
23

---

Published as a conference paper at ICLR 2026
Tucker over TT in the current work.
Structured and block-sparse Tucker variants. Beyond plain Tucker and CP/TT, there exist many structured Tucker variants that impose additional sparsity or block structure the core tensor or factor matrices (e.g., group-sparse (Mørup et al. 2008), block-diagonal (Jang & Kang 2020). or www-rank-plus-pparse cores (Heng et al.] |022|[2223 Our LeSTD framework can be viewed one such instance: we adopt a Tucker topology and impose structured sparsity on the core via Stage II, where the entries of 9 are pruned according an importance measure derived from the downstream contraction. This yields a compact, sparse core that is aligned with the semantics of the MHA ten- sor (shared factors and per-head slices). Orthogonal structured Tucker approaches that focus, for example, on block-diagonal or predefined block-sparse core patterns could in principle be combined with LeSTD: our Stage I can be replaced by or augmented with such variants, and our Stage II can be restricted to respect their block structure. A comprehensive empirical comparison between dif- ferent structured Tucker designs is beyond the scope of this paper. Here we adopt a minimal and model-aligned choice that already provides strong compression and speedup.
E.5 RECENT AND MORE ADVANCED SVD-BASED PRUNING METHODS
Basis Sharing. Basis Sharing (Wang et al. 2024a extends post-training SVD compression by ex- plicitly sharing low-rank bases across layers. Instead of decomposing each weight matrix indepen- dently, it factorizes selected matrices { (W(') } into a small set of shared singular vectors and layer- specific coefficient matrices, so that each layer weight can be reconstructed as linear combination of a layer-agnostic basis and layer-specific coefficients. The paper systematically studies which ma- trix types (c.g., self-attention projections versus feed-forward layers) and which layer groups can safely share bases without incurring large reconstruction error, and shows improved perplexity and zero-shot accuracy over prior SVD-based compression methods across 20%-50% compression tios (Wang et al. 2024a). Our LeSTD framework is conceptually related in that both approaches exploit shared subspaces reduce redundancy, but the scope and representation differ. Basis Shar- ing works at the matrix level and shares SVD bases across layers, its compressed representation is still a dense low-rank factorization of each matrix. In contrast, LeSTD first constructs a four-way MHA tensor that jointly encodes query, key, value, and output projections within a layer, then learns a shared Tucker basis across heads and performs sparsification in the latent core. This tensorial view lets LeSTD capture within-layer cross-head regularities that cross-layer SVD cannot see, and, im- portantly, it replaces the dense core with an ultra-sparse one via Stage II importance-bast bassdd pruning, rather than only reducing rank.
Pivoting Factorization and MPIFA. Pivoting Factorization (PIFA) and its end-to-end variant MPIFA (Zhao et al. [225]] start from standard low-rank decomposition and then learn a meta low- rank representation of the resulting factors. PIFA identifies a set of pivot rows (linearly independent rows) in each low-rank matrix and represents all remaining rows as linear combinations of the piv- ots, achieving additional parameter savings and speedups on top of low-rank pruning (e.g., around 24% extra memory reduction and 24% faster inference at r/d 0.5 over low-rank baseline). MPIFA further couples PIFA with an online reconstruction scheme that mitigates eтror accumula- tion across layers, enabling performance competitive with semi structured pruning while preserving the hardware-friendly nature of dense low-rank kernels. (Zhao et al. 2025). Compared with MPIFA, LeSTD operates at a different granularity and representation. MPIFA assumes low-rank matrix fac- tors and refines them with a lossless meta representation, it still ultimately applies dense low-rank linear maps during inference. LeSTD instead defines a Staared-subspace Tucker structure directly on the multi-head attention tensor and prunes in orthonormal latent basis, yielding an explicitly sparse core with a closed-form importance score and inference executed fully in the compressed domain. This allows LeSTD to break the dense-core bottleneck of tensor decompositions and to realize throughput gains even with standard dense libraries, whereas MPIFA focuses on improving the efficiency of misrix-wise low-rank pruning.
F COMPLEXITY ANALYSIS
Let dmooeed /h. In a standard MHA block, the primary cost comes from the four dense weight projections (for Q, K, V. O), each involving matrix multiplication with the input activations. Per token, this results in a complexity of approximately O(4 dmoda). ) In LeSTD, inference operates on the compressed factors, avoiding single large multiplication. Complexity breaks down follows:
24

---

Published as a conference paper at ICLR 2026
Input-side Projections (VQ.K, V). The initial shared projection of the input X by U costs O R1). This result is reused for all heads. For each of 3h projections (Q. K, V across h heads), the subsequent operations sor slice M. and a dense multiplication by U2. Let ( spanse) be the total number of non- zero elements in the sparse core. The average cost of the sparse multiplication per projection is (Spasee (44)). This is followed by dense multiplication costing O R2dhead).
sparse matrix multiplication involving the core ten-
Output-side Projection. Logic is symmetric. Cost involves h multiplications of cost (dhhaad followed by sparse multiplications related to the output slices of the core, and a final shared projec- tion by (U())) costing (RR, dhead).
When combining these steps, the total computational cost per token for LeSTD is dominated by: 20 (dmmodt R2) + (f + O(h Anead R2). Given that h dhead = dmodel- this simplifies to (dmoodal [R + R2)) + (nnzz (psass))). When the ranks R1, R2 < dmodel and the core tensor is highly sparse (low nnz (9 spars))) this is substantially lower than O (dmodd))
G DETAILED COMPUTATION OF COMPRESSION RATIO
For standard Transformer layer (omit the biases), the multi-heai attention (MHA) block has four dense projections: W WK, WV E Rodds X danddl and W E Re Xuaste given that dnead = dmwdee/ hh, so the original (uncompressed) parameter count per layer is P Poriginal = = dfanddr
LeSTD does a Tucker factorization along the first three modes (shared across heads) plus a head axis that is not factorized. Thus, for factor matrices: ((1) E IR U E R U E RIXR; and the dense core tensor (Stage I generated): Guoml E x For LeSTD with sparse core after Stage II, replace the dense core size by the number of nonzeros (9 So the compressed parameter count (ignoring sparse index overhead for the moment) is: Poompreased
dmodet R1 + dhead R.2 + 4R3 + nnz spunse). The ratio (p) defined in this paper is compressed thereby, original
it is equal to Aunaa &hea
4d
To fix p (e.g.. P 0.6) and search ranks:
1. Pick a candidate (Ri, R2, Ra) guided by TensorLLM (Gu et al. 2025). we start with a bit larger
(R1, R2, Ra) since Stage II will prune the core;
2. Run Stage I to get the dense core (refer to Section 3.1
3. In the Stage II (refer Section 3.2). we prune the dense core until the core hits a target nnz:
Enrget = 4dd
P (dimotee Ri + Chesa R2 + RR). Stop pruning when an= sparse) S Zurgel (or when further pruning would degrade accuracy too much);
4. If accuracy is insufficient at that p. adjust ranks and repeat, e.g., increase R4 or Rş (as shown in Table 2 these help for Perplexity) and tolerate a sparser core to keep p fixed.
H COMPARISONS WITH MORE ADVANCED SVD-BASED PRUNING METHODS
In this appendix, we provide an empirical comparison between LeSTD and two more recent SVD- based pruning methods: Basis Sharing (Wang et al.) (024a) and MPIFA (Zhao et al. (025). To keep the study focused and computationally manageable, we consider one single representative setting: GPT-J (6B) (Wang 202T) evaluated on Wiki'eext-2 (Merity et al.) 2016). Experiment in this ap- pendix are run on the same environment as in Scction 4. to ensure comparability. We focus on two global compression ratios, E {0.2, 0.8}. following the compression grid used in the main exper- iments (Appendix G). For each method, we tune its internal ranks or sparsity budget to match the target global parameter compression ratio as closely possible, using the same accounting scheme as for LeSTD. Specifically, at a given target ratio r, we adjust: (i) the matrix ranks and basis sharing groups in Basis Sharing, (ii) the low-rank rank r/d and PIFA/MPIFA pivot schedule for MPIFA, and (iii) the Tucker ranks and core sparsity level for LeSTD, until the overall parameter count of the compressed model matches the desired budget within a small tolerance.
Setting of Basis Sharing, For Basis Sharing (Wang et al. (024a). we adopt the official implementa- tionTan and follow the authors' recommended configuration for GPT-style decoder-only architectures. In particular, we enable cross-layer sharing for self-attention projection matrices (e.g., Wo. WK,
https: thub. com www..//ooo///a.... HWA 1/w.siii..ii... hat.saaashshhaaaa s__SS_Sharrr
25

---

Published as a conference paper at ICLR 2026
Table GPT-J on WikiText-2: comparison between LeSTD and advanced SVD-based compression methods (Basis Sharing (Wang et al. 2024a) and Pivoting Factorization/MPIIA (Zhao et al. 2025)
two global compression ratios (r (0.2,0.8))
Compression Rate r = 0.2
Compression Rate r = 0.8
Method
Perplexity())
Tokens/sec(1)
Perplexity())
Tokens/sec(1)
Basis Sharing
99.16
14025.69
12.30
10091.19
MPIFA
102.15
14086.19
12.88
9488.84
LeSTD (Ours)
80.37
15636.24
8.92
11514.42
Wv) while keeping output and feed-forward projections either unshared or less aggressively shared, as suggested by their reconstruction-ers analysis. We use the same type of small calibration cor- pus as in the original paper to estimate the SVD bases (sampling from WikiText-2) and then adjust the number of shared bases and per-layer coefficients so that the resulting global compression ratio matches E {0.2, 0.8).
Setting MPIFA (Pivoting Factorization). For Pivoting Factorization and its end-to-end variant MPIFA (haoo al. 2025), we again use the official code and default hyperparameters starting point. Each weight matrix is first approximated by a low-rank factorization (as in standard SVD- meta low-rank representation that selects pivot rows and reconstructs the remaining rows as linear combinations of these pivots. We select rank configu- rations corresponding overall densities close to r and apply MPIFA's reconstruction procedure to mitigate error accumulation across layers, in line with the original paper.
based pruning), and MPIFA is then applied
Table summarizes the comparison GPT-J with WikiText-2 at global compression ratios. At the more aggressive setting r = 0.2, all methods suffer from substantial perplexity increase, which is expected at such extreme compression levels. Basis Sharing and MPIFA reach validation perplexities of 99.16 16 and 102.15, respectively, while achieving throughput of around 14k tokens/sec. LeSTD attains a lower perplexity of 80.37 and higher throughput of 15636 tokens/sec, indicating that its tensor-level sparse Tucker representation remains more accurate and more efficient than the matrix-wise SVD-based baselines this extreme regime. At milder compression setting r 0.8, the differences become clearer. Both Basis Sharing and MPIFA incur noticeable accuracy drop, with perplexity increasing to 12.30 and 12.88, respectively. In contrast, LeSTD achieves a perplexity of 8.92, and improving throughput to 11514 tokens/sec.
Overall, these results validate that LeSTD is competitive with, and often stronger than, more ad- vanced SVD-based pruning methods. At moderate compression (r = 0.8), LeSTD preserves accu- racy almost perfectly while providing larger speedups. At very aggressive compression (r 0.2), all methods degrade substantially, but LeSTD still offers strietly better cefuuracc--ficiencyyy trade-off compared to Basis Sharing and MPIFA.
I EXTEND LESTD DESIGN TO FFN LAYER
In the main paper, we instantiate LeSTD on the Multi-Head Attention (MHA) block by tensorizing the four projection matrices into 4th-order tensor, and applying the two-stage procedure: (i) shared- subspace Tucker decomposition (Stage I) and (ii) importance-based core pruning (Stage II). Here we formally argue that exactly the same framework applies to the Feed-Forward Network (FFN) layers.
1.1 TENSORIZING FFN BLOCKS
Consider standard Transformer (as shown in Figure in the main content) FFN block in one layer, -> I daedd via intermediate hidden dimension dy:
which maps I. daoed
FFN(2))
W Woown (WiT. :+)
where W up E Ramda da W down R duase, and (6-) is pointwise nonlinearity (e.g.. GELU Lee
2023) or SwiGLU (Zhang al. 2024 Analogous the MHA case (Eq. 33) in Section 3.1 we
https: hhtp://gtttt..uecc cwm/bomm///oooooo omedi cal htt.://............ooooooooooooa
26

---

Published as a conference paper at ICLR 2026
group the FFN weight matrices associated with single layer into 3rd-order tensor
WFFN
R Rddaas
where the mode-3 index enumerates the p linear projections used in the FFN block (e.g., P 2 for vanilla two-layer FFN, p for StyleU-style FFN with gate/up/down matrices). Coneretely, (W denotes these FFN matrices arranged so that each W [ Iddnas X on the same input/ouiput spaces, we define
WEEN [AJ] w6) 1 isjsp
In this way, FFN weights admit a higher-order representation entirely analogous to the MHA tensor: each mode corresponds well-defined axis of variation (i.e., input model dimension, hidden FFN dimension, and projection type index).
I.2 APPLYING STAGE I: SHARED-SUREE SUBSPACE TUCKER DECOMPOSITION
Given the FFN tensor EYEN E composition used for MHA (Eq.
Rd Rduas Xduxp, we can apply the same shared-subspace Tucker de-
min
F g X1 U X2 (2) X3 U s.t. ((( = Ia. (10)
FPN
UM, 100, gma
Here
T R. U E R Rd U E PxRs
Sellmm-orthonormeg factor matrices, and F RRixRax fts is the FFN's dense core. Eq. (10) a direct specialization of the MHA objective (Section 3.1 to a 3rd-order tensor, it can solved using the same HOOI procedure employ for MHA.
Crucially, the derivation of the Stage does not depend on the interpretation each mode: it only re- quires that the factor matrices orthonormal and that the reconstruction is measured in Frobenius norm Böttcher & Wenzel (2008). Hence, all guarantees and intuitions for MHA Tucker decomposi- tion, e.g., capturing shared structure across projections, carry
the FFN tensor.
1.3 APPLYING STAGE II: IMPORTANCE- CORE PRUNING FOR FFN
Stage II (Section 3.2 derives (losed-form importance measure for each core element in the Tucker- decomposed MHA tensor. The key observation is that the derivation relies only on: (i) the orthonor- mality of the factor matrices {U(*), and (ii) use of Frobenius norm for reconstruction error. Both properties hold identically for the FFN decomposition in Eq. (TO). Therefore, the importance mea- sure and the pruning rule extend verbatim. Formally, let
W FFN
-HN
X1 (() X2 (2)
X3 (3)
be the Stage reconstruction of WIHHN and let R W w FFN
be the residual. Denote by the B (21,72, r3) multi-index into the core FFN and define the associated rank-1 1 basis tensor
B3 ui! u
(n) is the rn -th column of (()). By orthonormality, the collection {Ba} is orthonormal where urs with (B,, B..) = d3y and
| | = 1. As in Section 3.2 we can write
w FFN
B
with coefficients
IB R
3
and the Stage error satisfies
FFN = R
Σ
93
3
Now consider zeroing out single core coefficient gg. The new residual is
R (gs 0) FFFN
(N FFN
93 R gB
27

---

Published as a conference paper at ICLR 2026
and the corresponding reconstruction error becomes
E(ga = 0) = |R + 9 | = |R| 2 + 2g RR Bb) + 93 B ||- = - 93.
where E = ? is the original Stage error and we used (R. 33) = 0 and 113 = 1. exactly
as in Eq. ( of the main text. Therefore, the normalized importance of g
Imp(gs) Egg = 0) - o
E
E
and, again, ordering elements by Tmp(gg) is equivalent to ordering by lspl- Under the Frobenius loss, the best k-term approximation is obtained by keeping the largest-magnitude coefficients and pruning the rest, with optional refitting step using Eq. (6) Thus, Stage II's importance-based pruning generalizes directly to the FFN core G with no modification to the theory or algorithm.
1.4 PRELIMINARY FEN EXPERIMENT
To provide an initial empirical check of the FFN extension discussed above, we conduct a small- scale experiment on GPT-J (Wang 202T) evaluated on WikiText-2 (Merity et al.] 2016). We follow the same protocol as in Section (tokenization, context length, hardware, software, and evaluation metric). We consider four configurations:
1. Original. The original GPT-J model without any modification.
MHA-Only. LeSTD is applied only to the MHA block.
3. FFN-Only. LeSTD is applied only to the FFN block. MHA+FFN. LeSTD applied simultaneously to both MHA and FFN blocks.
Table reports validation perplexity and through- put on WikiText-2. The uncompressed GPT-J base- line attains a perplexity of 8.86 and a through- put of 7523 tokens/sec. Applying LeSTD only to the MHA block (MHA-only, compression rate- 0.6) increases perplexity modestly to 9.53 (a rela- tive change of about +7.6%) while already improv-
Table 8: GPT-J on Wikitext-2.
Configuration
Perplexity())
Throughput (loken/sec)
Original
8.86
7522.90
MHA-Oaly Only
9.53
12190.93
FFN-Only
12.44
16069.26
MHA-FFN
14.34
17604.24
ing throughput to 12,191 tokens/sec (approximately 1.62x over the baseline), consistent with our main GPT-J results in Section 4
When we apply LeSTD only to the FFN block (FFN-only, compression rate = 0.6), the throughput further increases to 16069 tokens/sec, reflecting the fact that FFN layers account for larger fraction of total FLOPs. As expected, this more aggressive compression of FFN weights leads to a larger accuracy drop, with perplexity rising to 12.44 (roughly +400% relative to the original model).
Compressing both MHA and FEN (MHA+FFN, with the same per-block compression rate) yields the highest throughput, 17604 tokens/sec (~ 2.34x speedup), at the cost of further perplexity increase to 14.34 (about +62%). Overall, this preliminary experiment confirms that the two-stage LeSTD procedure extends straightforwardly from MHA FFN: applying the same tensorization and core-sparsification pipeline to FFN layers provides substantial additional speedups, with tun- able securacy-effiieeng trade-off that behaves consistently with our theoretical analysis.
J DISCUSSION ON THE FEASIBILITY OF INCORPORATING ACTIVATION INFORMATION
A natural question is LeSTDD which is currently fully data-free, could be extended to exploit activation information (e.g., a small calibration set) in order to further improve compression fidelity. Existing post-training methods such as ASVD (Yuan et al. 2023) and SVD-LLM (Wang et al.) 2024b) demonstrate that activation-aware objectives can be beneficial matrix setting, especially when the activations exhibit strong anisotropy or heavy-tailed directions. However, our framework built around a shared Tucker decomposition and a sparse core defined in an orthonormal latent space. Below we explain why directly incorporating activations into this Tucker-based design is non-trivial, and why we deliberately keep LESTD data-free this work.
Activation-aware objectives in the matrix case. Consider single linear map with weight matrix
28

---

Published as a conference paper at ICLR 2026
W E Rdin ddows and imput activations z Rdn, Most data-free decompositions (including plain SVD and our Stage I objective) implicitly minimize a Frobenius reconstruction error
| - W
I - Wis)²
(11)
3,7
which corresponds to assuming an isotropic input distribution. By contrast, activation-awann matrix methods use a loss that reflects the output discrepancy calibration set. Let X E RNx collect N input activations (rows), and let Y X and Y XW be the corresponding outputs. The natural squared error is
Lan W W
XW - XW 1
X(W WW)-
(12)
N
N
Defining the empirical activation covariance
Σ N
x E Rdmxdn
(13)
we can rewrite
Laa W W) Tr (WW WW-
(14)
Thus activation-aware compression effectively replaces the Euclidean metric (A, B) =T(AA B) by a weighted inner product (A, B) = A Egg B) that emphasizes directions with large acti- vation variance. Methods such as ASVD and SVD-LLM can be interpreted performing low-rank
approximation in this weighted geometry, often via whitening transformations that absorb >//² into the weight matrices before applying SVD.
What an activation-awail Tucker objective would require. In LeSTD, the MHA parameters of one layer are reshaped into 4th-order tensor Wisul E R.landd that x4xh and Stage I solves the Tucker problem
min U F0re,, .rre) Weoual Groua X1 U()) X2 (2) x3 1
(15)
subject to (Ur())
I and shared (?)) across all heads, as detailed in Section 3.1 To
make this objective activation-awaree one would have to replace the Frobenius norm in Eq. (5))
by tensor analogue of the weighted metric Eq. (14). However, unlike the single-matrix case, an MHA block couples: (i.) multiple types of projections (Q/K/V/OK (ii.) multiple heads, and (iii.) multi-token activations flowing through attention. Formally, if we denote by X' the joint tensor of
pre-activation features feeding into all MHA projections, the natural activation-awe -wwrre loss would involve expectation of the form
Tucker acl
Ex [[] (X) - -fw... [x)]
(16)
where fw denotes the linear part of the MHA block parametrized by W. Even under strong lin- carization assumptions (ignoring softmax and non-lincarities), this expectation induces a 4th-order covariance operator over the tensorized weights:
Lact
Tacker
(W total
Wottl
C(W total
Wecca)
(17)
where C is a positive-defintt linear operator that depends on the joint statistics of Q/K///////// ac- tivations, heads, and tokens. To retain the shared-factor structure of Eq. (3) would need C to factorize approximately these mode-wise covariances would have to be compatible with a single set of orthogonal factors UT1)) U(2) (3) shared across all heads and projections. quite different distributions and scaling, and the covariance is neither separable nor shared across heads. Therefore, there is no obvious way to define single activation activation-weightee metric under which all rank-1 basis tensors remain mutually orthogonal.
Kronecker product over modes (e.g.. ( (2) & Σ(3), and
practice, Q/K/V K/V/O activations have
Impact on Stage II and sparse-core theory. Stage of LeSTD crucially relies on the fact that, un- der the Frobenius inner product, the rank-1 basis tensors (By) induced by (11), (?2), U(3) form an orthonormal system: (B,, B,))F ōpy. This orthonormality allows to express the recon- struction error as
W Iotal
W Wintall 11?
R
Σ g
29

---

Published as a conference paper at ICLR 2026
and to show that zeroing single coefficient 9a increases the enror by exactly 93 (Eq. (5)). Hence the optimal k-term approximation is obtained by hard-thresholding on [gal, and the refitting step admits a one-dimensional closed form (Eq. (6)) If switch to a general activation-weighted metric induced by C, the basis } no longer orthonormal:
(B By C(B,)) 68g
The error becomes
W lotal
W Lotal
112
E99g. (Bs, (Bs,
3,y
and the increase in loss associated with pruning single 9ø depends on all cross-terms with # 3. In this regime: (i.) magnitude-bant based ordering of [gal no longer the optimal, (ii.) identifying the best & coefficients is coupled combinatorial problem, and (ii..) refitting remaining coefficients no longer reduces to independent one-dimensional least squares. One can, in principle, define an activation-weighted sparse Tucker objective and derive generalized normal equations, but doing so would forfeit the key advantages of Stage II: data-free operation, closed-form importance scores, and efficient pruning with guaranteed monotone error accounting.
Practical considerations and our choice in this work. From a practical standpoint, activation- aware Tucker compression would also require: (i.) storing and streaming layer-wise activation tensors for each MHA block during calibration, (ii.) estimating high-order covariances (or at least their low-rank surrogates), and (iii.) calibrating separately for each model, dataset, and even down- stream task. Recent empirical studies on PTQ and pruning show that compression quality can be highly sensitive the choice of calibration corpus (Frantar al. 2022; Lin al.) 2024; Shang et al. 2023). and that calibration on mismatched data can even hurt generalization to new tasks. In contrast, LeSTD fully data-free and immediately applicable to any pretrained checkpoint without additional data curation or repeated activation collection.
In summary, while in principle one could attempt, but under our design constraints this is practically infeasible to define an activation-weigee Tucker problem, such a design would:
inner product and to formulate an activation-aware sparse
1. break the orthonormal structure that underpins our closed-form importance scores and refitting updates;
2. require strong and arguably unrealistic separability assumptions multi-head multi-projection activation statistics to preserve shared factors;
3. sacrifice the data-free nature of LeSTD by introducing the non-trivial calibration overhead and dataset dependence.
LLM USAGE K
During the preparation of this paper, we used OpenAl's ChatGPT-5 to assist with language polishing and improving readability of the manuscript. In addition, used OpenAl's Codex to help parse and understand baseline implementations by reading and decomposing publicly available source code. No part of the core technical contributions, experimental design, or results analysis was generated by LLMs; all scientific ideas, methodology, and validation remain the sole work of the authors.
30


Reviews:
[{"review_id": "N4Gj4WZh8x", "reviewer": "Reviewer_RLPz", "summary": "The paper introduces LeSTD, a two-stage, post-training compression framework for LLMs. Stage I performs a shared-subspace Tucker decomposition of the tensorized multi-head attention (MHA) weights: all heads in a layer share three orthonormal factor matrices, while each head retains its own small core, estimated via HOOI to minimize reconstruction error. Stage II sparsifies the per-head Tucker cores using an importance score equal to the coefficient magnitude, followed by a closed-form refit of the remaining coefficients. The method supports inference directly in the compressed domain where it reuses the shared projection and contracts the (sparse) per-head cores—without reconstructing dense weights. Empirical results demonstrate the effectiveness and efficiency of the proposed method across various tasks and models.", "questions": "* Can you report wall-clock for Stage I optimization?\n\n* Is it possible to consider structured pruning? Would this affect the performance greatly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is clearly written, with the methodology and experimental setup presented with detail in a organized way.\n* The pruning step is well justified.\n* Exploring tensor decomposition for post-training compression is an interesting and relatively underexplored direction in the domain.", "weaknesses": "* Figure 1 is not strong enough to justify the paper's motivation. The “shared subspace across heads within the same layer” claim is not well supported given the low explained energy, and the intra-layer explained energy is only marginally higher than the inter-layer case.\n* As the paper laid out in the limitation section, the current method does not handle FFN layers which constitute a large fraction of LLM parameters. Additionally, because the pruning is unstructured, actual storage and speed benefits would depend on the chosen sparse format and kernel support, so the practical gains may be smaller than the reported parameter reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vmZFFG2mdm", "reviewer": "Reviewer_y5qd", "summary": "In this submission a novel post-training compression framework for LLM called LeSTD is proposed. The approach is two-step: in the first step Tucker decomposition is applied to learn shared orthonormal factors across attention heads for each layer of the LLM. During the second step the ultra-sparse core tensor is created using importance-based iterative pruning. The importance score is derived from a reconstruction error.", "questions": "1.  Can you provide error bars across multiple experimental runs to assess statistical significance of the improvements over other methods?\n2.  How sensitive is performance to the pruning rate α? Only α=0.1 is mentioned; was this tuned?\n3.  What is the actual sparse indexing overhead at different sparsity levels? How does this affect real compression ratios?\n4.  Why does LeSTD sometimes lose to SVD-LLM on throughput (e.g., OPT-30B)? Can you characterize when your method wins vs. loses?\n5.  Can you provide results extending LeSTD to FFN layers, even if preliminary?\n6.  How important is the refitting step (Eq. 6)? How does skipping this step affect the performance?\n7.  What are the wall-clock compression times for Stage I and Stage II compared to baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The two-step approach is well-motivated and intuitive: first we learn a basis and then we sparsify within that basis\n2.  Theoretical justification for magnitude-based pruning is solid.\n3.  Extensive experiments with different models and different datasets.\n4.  LeSTD shows improvement over the competing methods across different compression ratios.\n5.  Paper is well-written, the presentation of the method is good with neat illustration;\n6.  The limitations are acknowledged and discussed (unstructured sparsity and MHA-only compression)", "weaknesses": "0.  The overall novelty of the submission is limited: Stage I is a known combination of the Tucker decomposition and HOOI, Stage II reduces to magnitude pruning (which is known optimal for orthonormal bases). The contribution is primarily in combining these for LLMs compression rather than theoretical/methodological novelty\n \n1.  Other sparse tensor methods are not considered (CP decomposition, Tensor-Train Decomposition, structured/block-sparse Tucker variants, etc)\n2.  Experiments do not include statistical significance (which is crucial in such works): no error bars, confidence intervals, or multiple runs reported;\n \n3.  Limited ablation studies: only 6 rank configurations tested (Table 2), no ablation on pruning rate α, refitting frequency, or HOOI convergence criteria\n \n4.  For the throughput, LeSTD sometimes loses to SVD-LLM (e.g., for OPT-30B on MathQA at 0.8 compression rate), but analysis of when/why is insufficient", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "61SmGhfCrk", "reviewer": "Reviewer_ee46", "summary": "This paper proposes a two-stage large language model compression framework based on Tucker decomposition. The first stage performs subspace decomposition to obtain low-rank latent representations, while the second stage compresses the core tensor via a closed-form sparse pruning method. The paper is well-organized, with sound theoretical analysis, detailed mathematical derivations, and extensive experimental validation.", "questions": "In the throughput analysis, the paper claims inference acceleration using standard PyTorch functions on sparse cores. However, it is uncertain whether PyTorch natively accelerates unstructured sparsity. Have the authors considered other possible factors that might contribute to the observed speedup?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper introduces a concise and elegant closed-form sparsification method in the decomposed latent space.\n2.\tThe mathematical derivations are thorough and rigorous, providing solid theoretical support for amplitude pruning in the Tucker latent space.\n3.\tThe motivation is clearly articulated, and the background is well presented.", "weaknesses": "1.\tThere are some formatting issues in the manuscript (e.g., line 100 references Figure 4 located at line 227).\n2.\tThe figures and their explanations are somewhat unclear. For example, in Figure 3, the components are scattered without clear annotation or explanation of what each parameter represents.\n3.\tThe introduction of the core tensor compression method—the paper’s key innovation—is somewhat disorganized. Although the mathematical derivations are complete, their presentation could be improved with more structured figures and explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g8YH2cKpFd", "reviewer": "Reviewer_W25T", "summary": "The paper introduces LeSTD (Learning-based Sparse Tensor Decomposition), a data-free, post-training compression framework for large language models. It addresses the dense core bottleneck in tensor decomposition methods by learning a shared basis across attention heads and then applying a theoretically grounded pruning mechanism to create an ultra-sparse core tensor. This yields higher compression ratios without accuracy loss. LeSTD performs inference directly in the compressed domain, providing throughput gains without custom kernels.", "questions": "1. The authors should clarify the distinction between LeSTD and TensorLLM. If my understanding is correct, Section 3.1 is identical with TensorLLM, while Section 3.2 is the main difference. It would be helpful to make this relationship explicit.\n\n2. As I understand, LeSTD does not require calibration data, which is a key advantage. Would it be possible to incorporate activation information into this framework to further improve performance? I suspect this might be challenging under Tucker decomposition, but a discussion on its feasibility would be valuable.\n\n3. The paper could include comparisons with more advanced SVD-based pruning methods, such as Basis Sharing [1] and Pivoting Factorization [2]. The concept of Basis Sharing appears somewhat related, since Tucker decomposition also captures inter-weight similarity. \n\n4. It would strengthen the presentation to include an inference algorithm, similar to Algorithm 1 for pruning. Section 3.3 is currently somewhat difficult to follow, and a concise pseudocode description would improve clarity.\n\n5. Will the code be released? If I understand correctly, the original linear layer requires one matrix multiplication, low-rank layers require two, and the proposed Tucker-based structure requires four. This may introduce additional I/O overhead. The authors claim that the method achieves speedup without custom kernels—i.e., purely using PyTorch—which is an impressive claim but also raises concerns. Providing the implementation during the rebuttal period would greatly improve credibility and reproducibility.\n\n[1] Wang, Jingcun, et al. \"Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression.\" The Thirteenth International Conference on Learning Representations.  \n[2] Zhao, Jialin, Yingtao Zhang, and Carlo Vittorio Cannistraci. \"Pivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models.\" Forty-second International Conference on Machine Learning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. If I understand correctly, prior methods such as SVD-LLM and ASVD require calibration data, whereas LeSTD operates entirely data-free—yet still outperforms them. This is quite impressive.  \n2. The idea of tensorizing all weights into a unified 4D structure and applying Tucker decomposition** is both intuitive and novel, offering a principled way to exploit inter-layer correlations that matrix-based methods overlook.", "weaknesses": "Check in questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)