BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference
Authors: Changwoo Lee
Soo Min Kwon
Qing Qu
Hun-Seok Kim
Venue: NeurIPS 2024 poster (2024)
Keywords: Efficiency
Compression
Low Rank
Pruning
Matrix Factorization
Structured Matrix
Acceleration
Optimization
Transformer
Large Language Model
Diffusion Model
Vision Model
Preconditioned Gradient Descent
Primary area: optimization_for_deep_networks
TL;DR: We improve DNN inference efficiency by learning low-dimensional weight structures through BLAST.

Abstract: Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70\% and 40\%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST.

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference
Changwoo Lee Soo Min Kwon
Qing Qu Hun-Seok Kim
University of Michigan
{cwoolee, kwonsm, qingqu, kingsm,
Qumi J ch, edu
Abstract
Large-scale foundation models have demonstrated exceptional performance language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, it represent various types of structures that either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium -ized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-78 7B and DiT-XL, the BLAST matrix achieves 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available https: / thub. com/ htt/:/chaaaee/lll.... BBLSS..
Introduction
Foundation models built on large deep neural networks (DNNs) have demonstrated remarkable performance in vision and language tasks. However, the size of these large networks poses both computational and storage challenges, especially resource-constrained environments such as edge devices. The size of a single DNN often exceeds the capacity of the supporting hardware devices [1-5]. For example, Llama-70B [1] demands least 140GB of memory solely for loading its weights in half-precisio floating point representation, while the state-of-the-arr commercial GPU only accommodates 80GB of memory. Furthermore, inference with these networks involves numerous dense matrix-vector operations, which can be limiting when computing power is constrained.
Fortunately, large (overparameterizee)) DNNs often exhibits parameter redundancy, where the intrinsic dimension of the weights is much lower than the ambient dimension. As such, the weights should be structured, possessing hidden properties such as low-rankness [6-9] or sparsity [10, 11]. Hence, possible to replace (or factorize) these dense existing weight matrices with structured ones without degrading performance [10-12]. However, using structured matrices that do not align with the true underlying structure of the weight matrices can result in significant performance degradation. We demonstrate this point in Figure where we attempt to capture the structure of a diffusion model transformer (DiT) [13] using the low-rank structure to generate synthetic images. In Figure we compress the model's 's linear layers by approximately 50% of the total number of parameters using low-rank weight matrices via singular value decomposition (SVD) and generate images with the compressed model (sec Section 4.2 and Appendix C.3 for details). As shown in Figure I (middle), simply using the low-rank structure introduces unwanted artifacts in the generated images.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original
Low-Rank
$:9
BLAST
Figure 1: Examples of generated images using DiT [13] starting from the same noise vectors and a deterministic solver. The original model is compressed by 50% through BLAST or low-rank matrices and re-trained for 10 epochs on ImageNet. The images from the model compressed via BLAST preserve the quality of the images of the original model, whereas the images generated by the low-rank model contain more undesired artifacts.
To address this issue, many flexible structures for modeling DNN weights have been proposed to minimize the misalignment between imposed and true low-dimensional structures. For example, Dao et al. [14] proposed the Monarch matrix, specific type of Block Low-Rank (BLR) structure [15), in which all blocks share the same rank, intended for use in the linear layers of transformers [16]. Matrix multiplication with Monarch matrix can be performed efficiently using batched matrix multiplication routines. Additionally, Chen et al. [17] investigated a block sparse plus low-rank structure. However, all of these methods still suffer from the fact that the underlying structure of each weight matrix is not known a priori. By imposing one of these structures, performance degradation may still occur due to misalignment. Recently Lee and Kim [12] introduced data-driven design called Generalized Block Rank (GBLR). This approach employs multiple rank-1 blocks with various sizes and locations learned from data via differentiable masks. Unfortunately. the GBLR matrix is optimized for custom custom-designed designed hardware, as the learned block patterns are random. It has limited usability on general GPUs as the computation of GBLR matrices does not accelerate well on them.
In this work, we introduce the Block-Level Adaptive Structured (BLAST) matrix, a versatile and efficient design tailored to uncover various low- -ow-dimensional structures in the weight matrices of DNNs for accelerated inference on GPUs. Our matrix structure leverages shared bases across block matrices with block-wise diagonal coupling factors. This structure encapsulates different structures such as low-rank, block low-rank, block-diagonal matrices, and their combinations. BLAST matrices can be applied to the training scenario from scratch or compression after training. For training from scratch, we let the linear layers of the DNN to directly adopt the BLAST structure and learn its factors from data. The factors of the BLAST matrix are constructed to have well-defined gradients, allowing them to be optimized using popular methods like stochastic gradient descent (SGD) or Adam [18]. For compressing existing weights, we propose factorization algorithm to learn the BLAST factors from pre- trained weights. The compression performance can be further improved by updating the BLAST factors using data, process we call "re-training"
We demonstrate the efficiency of BLAST by training Vision Transformers (ViT) [19] and GPT-2 [20] from scratch on various datasets, showing that it can reduce complexity by 70% and 40% respectively. We also compress existing ViT and Diffusion Transformer (DiT) [13] models with BLAST matrices by 70% and 50%, respectively, demonstrating that BLAST compression (and re-training) achieves higher accuracy / quality compared to existing methods for ViT and DiT (see Figure 1). For the language tasks, we compress Llama-7B [1] by 50% via BLAST and re-train on 0.49B tokens, showing the lowest accuracy degradation with significant inference speedup on a NVIDIA A 1100 GPU. Overall, our contributions can be summarized as follows:
2

---

U.
A UV
A.g Di, a....
Aa U.S.,
Monarch [14]
Low-Ramk
Block- Block-Diagonal
GBLR [12]
BLAST (Proposed)
Figure 2: Existing structured matrices and proposed BLAST matrix. The unique structure of BLAST allows for flexible matrix structures while enabling faster matrix multiplication compared to existing matrices.
We propose novel block-structured matrix called BLAST that encompasses a wide range of matrix structures, allowing for faster matrix multiplication. Various existing structured matrices such as Low-Rank, Monarch [14], and Block Diagonal matrices can be expressed using the BLAST matrix.
We provide gradient descent-based methods to find the BLAST factors for DNN weights. We empirically show that standard DNN training with the BLAST weight matrices effectively recovers the original accuracy while achieving up to a 70% reduction in computational complexity.
cases where pre-trained dense weights are available, we propose a preconditioned gradient descent factorization algorithm decompose the weights to BLAST factors for compression and further re-training. Our experimental results show that pre-trained foundation models for vision or language tasks can be compressed by 50% using BLAST matrices.
Notation and Organization. We use σ1 (X) ) to denote the largest singular value of the matrix X. The notation @ indicates Hadamard product.
The rest of the paper is organized as follows. In Section we introduce the BLAST matrix and discuss its properties. In Section 3, we propose methodology to train/compress DNNs with BLAST weight matrices. Section 4, we demonstrate the effectiveness the BLAST weights in improving efficiency without noticeable accuracy degradation. We discuss related works in Section 5, and conclude in Section 6.
Block-Level Adaptive Structured (BLAST) Matrix
Consider square matrix A RR for dimensional structure. We first equally partition the matrix into x blocks size x p where b.p E N are constants such that by:
N, which has unknown intrinsic low-
A
A
A..
A
A
A2.
A
Arj RPxP, i,j [6].
(1)
A
A
A
Then, the BLAST matrix parameterizes each block matrix A, using three factors:
A.j = U,
(2)
where U. V, E RPx are the left and the right factors, respectively, and St, diag(8:,) (8i is an r x diagonal matrix whose diagonal entries are 8 E Rr. We provide a visual representation on the rightmost side of Figure 2. and illustrate how this structure differs from other types of matrices. While the BLAST structure may appear similar to SVD, there are two notable differences: (i) the left and right factors do not need to be orthonormal, and (ii) the diagonal entries do not need to be positive. These distinctions make it more flexible in capturing different types of low-rank structures.
As illustrated in Figure 2, the BLAST matrix also comes with two unique properties:
For rectangular matrix, partition rows into chunks assuming that b divides well.

---

Factor Sharing: The left factor matrix U, of size is shared across b blocks at the 1th row, i.c., A....... A.
Likewise, the right factor V, is shared across the blocks at the 5thh column. On the other hand, the diagonal factor štjj of size is specific to each block A.j. Hence the total number of parameters of an n X BLAST matrix with b x number of blocks of rank is 2rph + rb² = 2nr + rb². This reduces the number of parameters b times by enforcing the blocks at the same row or column share the same bases.
Individual Diagonal Factors: The individual diagonal factors of each block matrix the source the adaptivity and ficxibility of the BLAST matrix. By changing the values of the diagonal factors, the BLAST matrix can encompass wide variety of matrix structures. These factors be estimated using gradient descent, since s is a real-valued vector and A., = U,diag (8t,)) V is linear ii.j
Now-Rank Matrices as Special Cases of BLAST capture different types of structures, we present example showing how the BLAST matrix can encompass low-rank matrix. Consider the case where the diagonal factors are ones, i.e. 8t.j = 1 write the block matrix as follows:
To demonstrate how the BLAST matrix can
for all i 1,2,.. ..... 6. Then,
U1
U.V U.V
U.
V U
U2
U2
UV =
V
V22
Val
U
UVV2"
UVV22"
U.V."
Hence, if the true underlying structure is low-rank, specific structure. Similarly, we show Section A.1 that the BLAST matrix can construct low-rank, block-diagonal, and block low-rank matrices through different diagonal parameters. A combination of these canonical structured matrices, such as a low-rank with block-diagona matrix, can also be achieved by simply concatenating the factors of each matrix.
expect the BLAST matrix to lear this
Matrix Multiplication DNNs involve
Algorithm 1 BLAST Matrix- Product
numerous matrix-vector (matrix-matrix) multiplications the form of y Ax (Y AX). Algorithm de- picts the BLAST matrix-vector multiplica- tion procedure. Consider the partitioned input vector x =I x.. ... " and the partitioned output vector y [BT ........................
Require: V T
1: [ 22 ... 2g
+
2: for j = 1,2,
do
#Parallel
3: 2, t -VVTa Zj
4: end for
5: for i 1, 2, do
#Parallel
.y. The zth partitioned output vector y: is then computed by the sum of the block-wise metrix-vector mul- tiplications along j 1,...,b. b:
6:
+ U, -5=1 S 3 zj
7: end for
8: return y 1y4 ..... yb 1
= A. = Σ U.S. = U.
Σ Suy (VF x,)
i i= 1,
(3)
3=1
The number of multiplications required to perform the matrix vector multiplication y Ax is (2n+66)r The matrix multiplication z; V æj, j =, b is computed once and shared across i == 1, b. whereas the matrix multiplications in Line and Line 6 of Algorithm can be executed in parallel, e.g., by torch. bmm in PyTorch [21]. An implementation of Algorithm I for general matrix or tensor inputs can be found in Appendix A.
Applications of BLAST Matrices
There are two main applications of BLAST matrices: (i) training from scratch with the BLAST structure and (ii) compression of pre-trained weights using BLAST factorization.
3.1 Training from Scratch using BLAST Matrices
To train DNN on dataset, parameters are typically initialized randomly and updated through stochastic gradient descent. In this setting, BLAST can replace dense weights to learn structures

---

from the training data. Instead of using random dense weight matrices, the model is initialized with random BLAST factors U, Si.j. Since the forward and the backward path of the linear layer involving the weight matrix is composed of three linear operations as in Equation (3), the derivatives of the minibatch loss can be back nock-propggtted by automatic differentiation frameworks [21]. Hence, all of the trainable parameters of BLAST can be updated using conventional optimizers (e.g.. Adam [18] or AdamW [22]) without additional treatment.
3.2 Compressing Weights via BLAST Factorization
BLAST Factorization via Gradient Descent Given pre-trained dense weights of a DNN, we can compress the weights using BLAST matrices. Let A denote the weight matrix and A., denote its blocks. We estimate the BLAST factors of Ai, by finding the factors of the BLAST matrix that minimize the Frobenius norm error between the original weight matrix and the BLAST structure:
b
E(U,, V., S.,))
Σ
1::5-UUn (s.,) || 2
F
where denotes the collection of all b components along the axis. This problem shares many characteristics with the classical matrix factorization problem [23-26], and hence we can solve for the factors using alternating gradient descent starting from small random initialization (c.g., Line I of Algorithm 2) [27. 8J. That is, the hthh gradient descent step is composed of three alternating updates with a step size r) > 0:
u.(t1) -U!1)) yuu! V Dari (U!+), v.(k), ..!.):
(5)
v(a+1) t v1(1) m Viare (U!+tt)) v.(h), .1.22
(6)
(k+1) (k)
1 a (U(++1) v((+1) sn.t
An i.
With properly chosen step sizes, Equations (5) to (7) always decrease the loss value whenever the current variables do not have any infinite entries and the gradient is non-zero. Using notations V.(A)
T to indicate the concatenation of the right and left factors scaled by the diagonal components, the loss is monotonically
[s(A] ( S b v (k))
T
and w)(²) (U +())))
(U yr
non-increasing as in the following theorem.
(k) R be factors of Theorem 1. Let Aig RPx be a target block and U.! (K) v(h E BRxx and s
a block in the BLAST matrix to be optimized. With the step sizes < Tores S $1/01 (V v()))
O (y I 1/0r (D (,(t)) < Tg I 1/or ((U o @ (V (K+JTT y 3 ((t))))
the gradient descent updates Equations (5) to (7) monotonically non-increase the loss:
((U(++1) (k+1) (k+1)) s ((U (())
The proof of Theorem is in Section B, which is application of the descent lemma from classical optimization theory.
Blast Factorization via Preconditioned Gradient Descent Recall that in order to estimate the BLAST factors given a pre-trained weight A, we need to choose a rank r. Since we do not know the rank of A priori, we may have to overestimate the rank. However, overestimating the rank may slow down the convergence rate for solving Equation (4). To illustrate this, we performed an a synthetically generated target low-rank matrix
empirical analysis of the convergence behavior A, whose dimension is 256 x 256 with a true rank of r' = 8. For the analysis, we computed the factors of a BLAST matrix with b 16 for various values of We used linearly decaying step sizes (A) (-) When
r* 8 (the ranks of the left and right factors U., V. match the actual rank of A). gradient descent finds low-rank solution with minimal error within 30 iterations, as shown by the blue curve Figure 3-left. However, in the case of r = 32 > y* where the BLAST factors are overparameterized, we observed slower convergence and substantial residual
(7(6) = (
5

---

10
10
10"4
10
Nomalized Fraberius
10"
Proberuus
to-3
10-1
10
10-4
20-3
10"7
20
40
60
80
100
20
40
60
20
100
Rerations
Berations
Figure 3: Convergence of the BLAST factorization with and without the preconditioning steps on noiseless low-rank matrix factorization with rank r*. Left: The BLAST parameter r*, Right: r > r*. When > r", the convergence rate of GD without the preconditioning is slowed down, while GD with the preconditioning (PrecGD) can recover the ground truth with small error.
Algorithm 2 Preconditioned BLAST Factorization (see Equations (8) and (9))
Require: A, (n) K-1
K, 8 0 1==0 C,
1: 0 v/00
N0,, e1), 8. Unif(0,
Vi,j 1,
2: for k = -..,...,
do
3: U. (k(t)) t U.!)) 1 (B)
( V - v. PU.
=i=
4: ((++))= + -V(A) n (k) U ( -A.j)
T o p(
Vj
(k+1) + 8ü.j n)) P (W!)))
(k)
U ((+))
An, v(hti))) Vi,j = I....,b
5:
(.j $1,5 diag
6: end for
7: return U. (K) (k) (K)
8,,
error after 100 steps as shown by the blue curve in Figure 3-right. This behavior is consistent with previous observations slow convergence in ill-conditioned matrix factorization problems [23, 24].
The convergence rate of solving the overparameterized low-rank factorization by gradient descent can be improved via inexpensive preconditioners [23, 24] which effectively decrease the condition number at each iteration. Inspired by the preconditioned gradient descent for low-rank factorization, we generalize the idea to solve our problem by multiplying preconditioning matrices to the gradients in Equations (5) to (7). We summarize the preconditioned gradient descent method for the BLAST factorization in Algorithm 2, where the following preconditioning matrices are used:
((k) (k)) (()))
+ 81) -1
PV,
(
T + 88
(8)
P
(x()) + 81)
w!A) (U ((+t)))) ( (
(9)
8 is proportional the square root of the error in Equation (4). The derivations are presented in Appendix A.2. Figure 3 shows that preconditioning improves the convergence of the overparame- terized BLAST factorization. The preconditioned gradient descent (yellow curve) finds the points with low error after 100 steps, whereas the gradient descent without preconditioning fails to achieve a small error. More empirical studies on the BLAST factorization with or without preconditioning can be found in Appendix D.I. We summarize the compression procedure in Algorithm 2. The computational complexity this compression algorithm is O(nr² - r²)) where the cubic dependency on is from the matrix inversion steps. We emphasize that these matrix inversion steps are not substantial computational bottlenecks because is smaller than n as in prior work [23, 24).
After BLAST compression outlined in Algorithm 2, the BLAST factors can be used directly for inference to save both storage and computational costs. However, we observe that, instead of using the estimated factors directly, using them initial points and refining the estimates by re-training the model with BLAST factors can further improve the performance of the compressed model. We refer to this process as "re-training" after BLAST compression.

---

VIT-S, CIFAR-10
ViT-S, CIFAR-100
70.0
Model
Accuracy (%)
Relative FLOPs (%)
07.0
Dense ViT-Base
78.7
1000
Acconcy (%)
66.0
Devse
Pank
Low-Rank
Monarch [14]
78.9
78.9
33.5
33.5
67.5
Manzrch
Moudi-GiBILRI (12)
78.5
32.8
Poeell
BLAST:
79.3
27.8
00.0
GoudiGGLRR
Table I: ImageNet validation accuracy and rel-
BLAST,
67.5
ative FLOPs of ViT-Base trained from scratch models with different structured weight matri- ces. The image and the patch sizes are 224 x 224 and 16 x 16, respectively. BLAST3 indicates the BLAST matrix with 3 x 3 number blocks.
FLOPs
FLOO'
CIFAR- 10/100 accuracy of ViT-S trained from scratch with different structured ma- trices.
Figure 4:
4 Experimental Results
We evaluate the BLAST matrix under two settings: (i) training from scratch with random initialization in the BLAST format, and (ii) re-training after compressing the dense weights BLAST matrices via Algorithm 2. We compare the performance of BLAST with both non-adaptive and adaptive structured matrices. Among the non-adaptive approaches. we include low-rank (LR) matrices, Monarch for block low-rank (BLR) [14], and Pixelfly [17] or Block-Diagonal for block sparse matrices. For the adaptive and learnable structured matrix category, we evaluate Gaudi-GBLR [12]. We report the number of floating point operations (FLOPs) by counting the number of multiplications. The BLAST matrix with r for every target weight matrix by setting it to meet the computational budget of the DNN. All experimental details can be found in Appendix C.
number of blocks denoted by BLAST, We used the same hyperparameter
4.1 Training from Scratch
Image Classification We train the reduced-size Vi- sion Transformers (ViT) [19] with BLAST; (BLAST with = 3) weight matrices on CIFAR-10, 100 [28], and ImageNet-Ik[29] for 310 epochs from random initializa- tion, and compare with other structured matrices. the CIFAR-10 and CIFAR- --0 benchmarks, BLAST outper- forms several non-adaptive baselines, such as Low-Rank, Pixelfly, and Monarch with higher accuracy at the same FLOOPs complexity (Figure 4). Gaudi-GBL.R presents the most favorable accuracy-to-Fil -F..OP tradeoff due to its capability of learning the adaptive resource/budget allocation for each weight matrix, which is a feature that our BLAST setting lacks in this particular evaluation (as we force it to use the same r for all matrices).
22.5-
Danse
LowRark
Parpleity
23.6
Block Diag
Munsech
Gandi-GBLR R
BLAST,
20.5
Relatine FLOPs
Figure 5: Pre-training result: WekiText 103 test perplexity-FL.Os trade-off curves from GPT-2 with different types of weight matrices.
However, in the context of ImageNet- 1k in Table 1, weight matrices trained using BLAST with b 3 attain the highest levels of accuracy with the least FLOPs. This superior performance of BLAST (despite the common r for all matrices) over Gaudi-GBLR be attributed to its simpler training process with fewer hyperparameters. In contrast, the more complex training requirements of Gaudi-GBL.R which involve smoothness annealing and proximal gradient descent, may lead to suboptimal results for large model such ViT-Base in Table
Language Model Evaluation We validate the training performance of BLAST weights on language models. We replace the weights of GPT-2 [20] with random BLAST, matrices and trained the network from scratch on the WikiText 103 [30] dataset for 100 epochs. Figure 5, we compare the test set perplexity of BLAST with the perplexity from low-rank, block-diagonal, Monarch, and Gaudi-GBL.R matrices. Similar to the ImageNet training, we found that BLAST achieves the best perplexity-FL.O0 trade-off. Compared to Gaudi-GBLR, BLAST obtains a significant perplexity gain. We attribute this
7

---

improvement to the simple training process of BLAST which requires less hyperparameter tuning than that of Gaudi-GBLR.
4.2 Compression and Re-training
In this section, we discuss the performance of BLAST weights when pre-trained dense weights are available. We first compress the dense weights using Algorithm and re-train the model on the training data with the cross-entropy loss.
ViT on ImageNet Classification We compress the weights of the vision transformer (ViT) trained on Im-
ageNet training set by BLAST, and BLAST using Al- gorithm 2 and re-train the models for 35 epochs. The
Imagelee Acunacy (N)
accuracy-FLOPs trade-off curve each model is pre- sented in Figure 6. Both BLAST compressed & re-trained models outperform other baselines, even though BLAST models did not use the adaptive budget allocation, like Gaudi-GBLR is observed that the accuracy of the BLAST models slightly increases from b 3 to b 12.
Lawy Rank
Monarch
Goudi GBLR
BLAST;
BLAST
0.96 0.28
0.31
0.30
Relative FLOPs
Figure 6: Compression and re-training result: ImageNet accuracy-FLOP. trade- off curves from ViT-B with different types of weight matrices.
Diffusion Models We compress the weights Dif- fusion Transformer (DiT) [13] pre-trained ImageNet using BLAST matrices and compare its performance to SVD-based low-rank approximation. For both techniques, we match the compression ratio such that both decrease the total number of model parameters by 50%, and re-train each model for 10 epochs on the ImageNet training set. We evaluate the models by generating total of 50,000 images using the original, low-rank, and BLAST com- pressed models, and compute the FID [31]. sFID [32], and IS [33] metrics with respect to the ImageNet validation set. The objective is observe the compressed model can generate images realistic as the original uncompressed model.
CR 0%
Method
FIDT))
sFI
[S(T)
Original
9.62
6.85
121.50
Low-Rank
48.07
11.44
26.09
50%
BLAST,
10.45
6.72
111.05
Table 2: Performance comparison for
compressing the weight matrices of diffusion model followed by re-training. FID and IS scores were computed with respect to validation dataset. CR stands for Compression Ratio.
In Table 2, we show quantitatively that the model compressed via BLAST significantly outperforms the model compressed via SVD, The low-rank compressed model often generates unrealistic images, leading to poor metrics such as the inception score. Figure also contrasts how the BLAST matrices contribute maintaining high perceptual quality well close instance-wise resemblance with the uncompressed model outputs. Due to space limitations, we defer additional qualitative results and experimental setup to Appendix D.2.
Large Language Models (LLMs) We compress the weights of Llama-7B [1] with BLAST matrices using Algorithm by 20% and 50% and re-train the models for 400 steps on a subset of SlimPajama [34] dataset using 0.49B tokens. The number of blocks b in the BLAST matrices is fixed at 16, and we use r 1024 for the attention modules and = 1488 for the MLP modules to achieve a 50% compression ratio. We test the WikiText-2 perplexity and the zero-shot task classification accuracy on common sense reasoning datasets including PIQA[35], HellaSwag| 36), WinoGrande(37), BoolQ[38), OpenBookQA[[9), ARC-easy and challenge [40]. We report the performance of Low-Rank, Monarch, and Block-Digonal weight matrices after compression at the same rate and re-training. In Table 3, the first row presents the performance of the original Llama-7B model. On 50% compression ratio in the last five rows, the Monarch and Block-Die -oiakDaaa matrices fail to recover the acceptable performance. Compared to Low-Rank weights, BLAST weights achieve the lowest performance degradation in WikiText-2 perplexity and zero-shot classification accuracy. The accuracy of each common sense reasoning benchmark and extended results can be found in Appendix D.3.
We provide analysis to quantify the performance impact of compression and re-training. We first quantify the weight compression performance at 20% compression ratio in Table 3. Although the compression ratio is moderate, Low-Rank and Monarch compression without re-training suffer from
8

---

CR
Method
# Params
Re Re-trained?
Wikilext-2 Perplexity (
Avg. 0-Shot Accuracy (1)
0%
Original Llama-78
6.74B
N/A
9.37
66.07
Low-Rank
5,41B
No
23.67 (-14.30)
59.37 (-6.50)
20%
Monarch [14] (BLR)
5.41B
No
47.18 (-37.81)
48.91(-17.17)
BLAST16
5,41B
No
12.13 (-2.76)
62.94 (-3.14)
Low-Rank
3,51B
Yes
26.33 (-16.96)
48,40 (-17.67)
Monarch [14] (BLR)
3,50B
Yes
7.53c5 (-7.53c5)
35.03 (-31.04)
50%
Block-Diss Diigonal
3,50B
Yes
5.21e6 (-5.21e6)
34,86 (-31.21)
BLAAST
3.56B
Yes
14.21 (-4.84)
56.22 (-9.84)
Table 3: Zero-shot performance of LLaMA-7B after compression and retraining. Avg. 0-Shot
Accuracy stands for the average accuracy of the zero-shot classification task. CR denotes compression
ratio. Bold indicates the best performance under the same compression ratio. BLAST, indicates the
BLAST matrix with b b number of blocks.
Ulma-7B. Avg. Zero-shot Accuracy
85-
Zero-shon Accuraey (%g
CR
L=
= 100
L= 1000
0%
N/A
0.41
-8c-5
3.82
49c4
41.23
=6c-3
60-
20%
2
0.35
-9c5
3.30
+2c-3
35,99
44e3
55
20%
16
0.36
8e-5
3,36
42e-3
36.48
=7e-3
Uncompressed
50%
16
0.31
44-4
41e-2
30.35 =2c-2
60
B (Comp. Only)
Table 4: Average runtime (in second) of Llama-7B with BLAST, weights from 10 runs of text gener- ation. =: standard deviation, L: the length of the generated sequence, CR: Compression Ratio. All models evaluated on single A100 GPU (40GB) using PyTorch [21] after torch. compile().
RLAST (Comp a Retrain)
50
Compression Ratio (N)
Figure 7: Average zero-shot accuracy vs. compression ratio curves of Llama-7B by BLAST before and after re- re-training.
significant performance loss, at most 4x higher perplexity and 25% accuracy degradation. On the other hand, 1LAST compression without re- re-training maintains reasonable accuracy and perplexity. This shows that the flexible and adaptive structure of BLAST matrices captures more information than other types of structured matrices. Yet, BLAST compression also exhibits noticeable performance degradation on a more intensive compression ratio (see the yellow curve in Figure 7). We provide more compression-only results Diffusion Models and LLMs in Appendix D.
We find that the re-training stage crucial for converting pre-trained model into an efficient version without losing significant accuracy when the compression ratio is high. In Figure 7, we show the average zero-shot classification accuracy of the compressed models with 50% compression. The models with BLAST weights before re-training (yellow curve) exhibit substantial accuracy degradation at higher compression ratios. However, re-training (blue curve) effectively recovers performance using only 0.49B tokens and 400 steps.
LLM Runtime Analysis We evaluate the runtime of the Llama-77 7B compressed by the BLAST
matrices the text generation task. For evaluation, we let the model generate the sequences of length L =100, 100, and 1000 ten times and report the average and the standard deviation of model runtime in Table 4. The instruction
use to generate the desired sequence length is "Increasing sequence: one," "" and the model generates the text sequence such as "two, three, " and so on, with a batch single 40GB NVIDIA A100 GPU after compiling the script using torch. compile (). The 20% compressed model shows 12%o~15% runtime reduction without any library function customization for BLAST matrix multiplication. The speedup when = 2 is higher than when = 16 because a larger number of blocks increases the computation overhead to perform Equation (3). Notably, the 50% compression ratio provides 32%~35% runtime reduction when = 16. We note that the test is highly memory-bandwind bounded. Thus the speedup reported in Table can be mostly attributed to the reduction of parameters (i.e., memory accesses) rather than the reduction in FLOPs due to BLAST compression.
size of 1. All runtime evaluation tests were conducted
5 Related Works
Structured Matrix with Shared Bases Sharing the bases of struc-st matrices has recently drawn interest due to its considerable memory savings. BLAST matrices exemplify this approach.

---

Ashcraft et al. [41] extend the BLR [15] format to BLR", incorporating shared bases and block-wise low-rank coupling factors determined through LQ or QR factorization. Similarly, Yen et al. [42] apply the concept of shared bases in the Block-Diagonal preconditioner for DNN weights. While BLAST also shares the bases of blocks, it is distinct in having a diagonal coupling matrix, as shown in Equation (2). The design of BLAST matrices aims to enhance efficiency and learn a variety of structures, from low-rank to high-rank block matrices. More importantly, identifying the diagonal coupling factors in BLAST matrices does not necessitate QR decomposition. Instead, they can be updated via gradient descent, making this approach well-suited for modeling the weight matrices in deep learning models.
DNN Weight Pruning and Decomposition Earlier work on DNN pruning [43 (3-45] identifies less important parameters from the Hessian or magnitude to sparsify the weight matrix. Unlike general sparse matrices, a group sparse matrix skips computation in group-wise manner by pruning channels [46, 47]. Sparse GPT [11] successfully prunes large language models with 50% sparsity without significantly degrading the performance of the original model. The model can achieve actual speedup utilizing 2:4 sparsity [48] on specific GPUs. However, 2:4 sparsity requires accelerators with specific architectures (e.g.. NVIDIA A100 GPUs) and supports only the 50% compression ratio. On the other hand, BLAST is device-agnostic since it can be implemented with basic matrix arithmetic operations and offers diverse compression ratios.
Low-rank matrices have been widely adopted for CNN compression [49, 50] and Transformer compression [51, 52]. Additionally, Butterfly [53] and Monarch [14] factorization methods model high-rank but low-dimensional structures of the weights. Specifically, Monarch matrix is a gener- alized version of a Butterffy matrix, yielding a wider spectrum of structured matrices. The number of blocks plays a key role in determining the rank of the Monarch matrix as whole and does not generalize to another Monarch matrix with fewer blocks. Unlike Monarch, BLAST with b x blocks can express Monarch matrices with the same or fewer number of blocks, including the global low-rank matrix, i.e., b 1. 1.
Learning Low-dimensional Structures of DNN Weights Similar to the BLAST matrix, Gaudi- GBLR matrix [12] enables learning diw-dimeniional structures by gradient descent in the generalized structured matrix space. Gaudi-GBLR overlaps variable number of zero-padded rank-1 1 blocks to model high-rank submatrices. Although Gaudi-GBI. can express wider spectrum of matrices than BLAST, the matrix-vector multiplication for Gaudi-GBLa is less efficient because GPUs and typical neural processors cannot handle zero-padded vectors efficiently. In contrast, the BLAST matrix-vector operation does not involve zero padding, allowing for more efficient execution in hardware for the same FLOPs (as shown in Figure 4, Figure 6, and Table I).
6 Conclusion and Future Work
In this work, we introduced the BLAST matrix designed to improve the inference efficiency of large DNNs. The BLAST matrix represents various low-dimensional structures of the weight matrices with fewer parameters, while enabling efficient matrix-vector products. The BLAST factors are either learnable from data estimated from existing weights using our preconditioned factorization algorithm. Our results on both language and vision tasks highlight the effectiveness of BLAST.
Limitations and Future Work The BLAST matrix-vector product consists of three steps, as detailed in Equation (3), which may degrade hardware-execution parallelism. In our evaluation, we used the same computational budget r for all matrices. Learning an adaptive budget per layer or matrix (c.g., via overparameterrizaton [54, 7J) could further improve BLAST performance. which is left for future work. The proposed method has not been evaluated on tiny (<100M parameters) or extremely large (>10B parameters) DNNs. Additionally, optimizing runtime and power consumption via BLAST matrices with customized library functions and/or hardware accelerators also remains as future work. Furthermore, a deeper theoretical investigation into the behaviors of BLAST matrices would provide more comprehensive understanding of their capabilities and limitations. Applying advanced re-training techniques, such as knowledge distillation [55] or iterative compression and distillation [56], to the BLAST compression pipeline is also left for future work. Finally, beyond the weight structures, we expect BLAST can also help understand and exploit Low-dimensional data manifolds [57-59] in future work.
10

---

Acknowledgments and Disclosure of Funding
The arXiv version of the paper can be found at https: /ttps://arxivv wttgg////g.....././/////// 21262. This work was supported in part by COGNISENSE, one of seven centers in JUMP a Semiconductor Research Corporation (SRC) program sponsored by DARPA. SMK and QQ acknowledge funding support from and NSF CCF-2212066, We thank Salar Fattahi, Reetuparna Das,
NSF CAREER CCF-214390 Mingyu Yang, Sara Shoouri, Shrikant Arvavasu, Jayeon Yi, Andrew Larson, Pierre Abillama, Alireza Khadem, Yufeng Gu, and Can Yaras for helpful discussion and feedback.
References
[1] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint (tt.:2330.13377// 2023.
[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yer Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2015-4026. 2023.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[5] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 9748-8763. PMLR, 2021.
[6] Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. Transactions on Machine Learning Research, 2022.
[7] Can Yaras, Peng Wang, Wei Hu, Zhihui Zhu, Laura Balzano, and Qing Qu. The law of parsimony in gradient descent for learning deep linear networks. arXiv preprint arXiv:2306. 154, 2023.
[8] Soo Min Kwon, Zekai Zhang, Dogyoon Song, Laura Balzano, and Qing Qu Efficient low- dimensional compression of overparameterized models. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1009-1017. PMLR, 02-04 May 2024. URL https: //pp::///rore //apps////oocceiiggss. mlr. 2238///238/ min-kwon24a. html.
[9] Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Under- standing deep representation learning via layerwise feature compression and discrimination. arXiv preprint arXiv:2311 (tt:::3/1.02966. 1.02960, 2023.
[10] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019. URL
https://openreval
[11] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337. PMLR. 2023.
[12] Changwoo Lee and Hun-Seok Kim. Differentiable learning of generalized structured matrices for efficient deep neural networks. In The Twelfth International Conference on Learning Representations, 2024. URL https::/ www.eerrriiee.net//ruutaaaaaeeae
11

---

[13] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[14] Tri Dao, Beidi Chen. Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference Machine Learning, pages 4690 4721. PMIR, 2022.
[15] Patrick Amestoy, Cleve Ashcraft, Olivier Boiteau, Alfredo Buttari, Jean-Yves L'Excellent, and Clément Weishecker. Improving multifrontal methods by means block low-rank representa- tions. SIAM Journal on Scientific Computing. 37(3):A1411-A1477) 2015.
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[17] Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations, 2022. URL https://openreviee net/forum? id-
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
[21] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL /ttppree//oerrrviieee nwt./frumm fttr///rrrrriieeeeeeoorourr id=Bkg6RiCqy7.
[23] Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150): 2021.
[24] Jialun Zhang, Salar Fattahi, and Richard Zhang. Preconditioned gradient descent for over- parameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems, 34:5985-5996, 2021.
[25] Xingyu Xu, Yandi Shen, Yuejie Chi, and Cong Ma. The power of preconditioning in overpa- rameterized low-rank matrix sensing. In International Conference on Machine Learning, pages 38611 1-36544 PMLR, 2023.
[26] Tian Ye and Simon Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. Advances in Neural Information Processing Systems, 34:1429-1439, 2021.
[27] Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems, 34:23831-23843, 2021.
[28] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL. https::///// ////:///wwwww /wwp///.. cs. toronto. weteeeduudae//////eee wttp://aeaenng-feeeeeeeeee cateeeess-00----- 2009-TR, Td.fff.
12

---

[29] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 11522:211-22,, 2015. doi: 10. 10.1000/s11200 1117111166----011666
[30] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprinl arXiv: 0tt:::/609.778888 2016.
[31] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[32] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia. Generating images with sparse representations. International Conference on Machine Learning, pages 7958-7968. PMLR, 2021.
[33] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
[34] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. s1 slimpaj ama- aamma-a-627b--o 2023. URL https:tt///// hnttps://uuggingfaaee / jama-m6622.....
https: //www/ cerebras net/blog/
and-
[35] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432-7439, 2020.
[36] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[37] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
[38] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 2019.
[39] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering, arXiv preprint 11X.v.1809.02789, 2018.
[40] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 2018
[41] Cleve Asheraft, Alfredo Buttari, and Theo Mary. Block low-rank matrices with shared bases: Potential and limitations of the blr "2 format. SIAM Journal on Matrix Analysis and Applications, 42(2):990-1010, 2021.
[42] Jui-Nan Yen, Sai Surya Duvvuri, Inderjit S Dhillon, and Cho-Jui Hsieh. Block low-rank preconditioner with shared basis for stochastic optimization. In -eirrth-eeenth Conference on Neural Information Processing Systems, 2023. URL htttp:://ppeeeeeeeeeeeeeeeeoeeee wet./orummifieeeeeeoooo JzQ1CqBm8d.
[43] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[44] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
13

---

[45] Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffiman coding. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICL. 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http:tt::/ /arxiv. org/ /b///////// 1ww1111101110
[46] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[47] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702-21722 2023.
[48] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprini arXiv:2/04.0837 2021.
[49] Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low- rank regularization. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:// http//aaxvvv..//. www.//gg///////e/eeoo 2
[50] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
[51] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL Bttptttt////ppprriiiee
[52] Habib Hajimolahoseini, Walid Ahmed, Mehdi Rezagholizadeh, Vahid Partovinia, and Yang Liu. Strategies for applying low rank decomposition to transformer- based models. In 36th Conference on Neural Information Processing Systems (NeurIPS2022), 2022.
[53] Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré. Learning fast algorithms for linear transforms using butterfll factorizations. In International conference on machine learning, pages 1517-1527. PMLR, 2019.
[54] Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overpa- rameterized low-rank learning & adaptation. In Forty-first International Conference on Machine Learning, 2024.
[55] Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 2015.
[56] Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marein Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation practice: The minitron approach. arXiv preprint arXiv:2408. 1796, 2024.
[57] Peng Wang. Huijie Zhang, Zekai Zhang, Siyi Chen, Yi Ma, and Qing Qu. Diffusion models learn low-dimensional distributions via subspace clustering. arXiv preprint arXiv:2409.02426, 2024.
[58] Gen Li and Yuling Yan. Adapting to unknown low-dimensional structures in score-based diffusion models. arXiv preprint 240.::2000..486... 2024.
[59] Siyi Chen, Huijie Zhang, Minzhe Guo, Yifu Lu, Peng Wang, and Qing Qu Exploring low- dimensional subspace in diffusion models for controllable image editing. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / //arxiv. wwg/abs/////oooooooootooot.eeert 2209/ 023744.
[60] Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
14

---

[61] Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends@ Machine Learning, 8(3-4):31-335,200
[62] Roger A Horn and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994.
[63] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 185 2018.
[64] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[65] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6857 2020.
[66] Runyu Peng, Yunhua Zhou, Qipeng Guo, Yang Gao, Hang Yan, Xipeng Qiu, and Dahua Lin. Data- freweegh compress and denoise for large language models. arXiv preprint arXiv:2402.16319. 2024.
[67] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https: attps://zenod:: org/ records/ 10256836.
15

---

A Details on BLAST Matrix and Factorization
Matrix Multiplication Code Implementation Following the conventional setting in Transformers [16], the input tensor is assumed to have batch, sequence, and channel dimensions. The left and right factors are multiplied using the batched matrix multiplication routine, whereas the diagonal factors are multiplied via broadcasting and summation.
def blast_matmul
x, # shape-(B, (B, n, q*b), size,, n-num_seq U, # shape=(b,
r), senum_blocks r=BLAST rank s, # shape=(b, b, r)
Vt, # shape=(b, I, q)
):
n) rearrange (X,, "B n b) -> (B nB q")
Y bmm (x, Vt.T) multiply right factor
2 Y. unsqueeee (0) u.unqqueez (2) # multiply diagonal factor Z.sum (1) aggregate, shape=(b, Ben, r)
Out bmm(2, U.T) multiply left factor Out = rearrange (Out, "b (B -> B (b p)")
return Out
Figure 8: Pseudocode of BLAST Matrix Multiplication. The function bmm stands for the batched matrix multiplication routine (e.g., torch, bmm (21]).
A.1
More Special Cases of BLAST Matrix
Block diagonal matrix block-diagonal matrix is a BLAST matrix when r = -p and 8ij =
{ 1,
if since
lo, otherwise
TA
Ujdiag($1,)
2
U222ag($2.2V(2
A
Undiag(st.) (8b V
When p. BLAST matrices model block-diagonal matrices which have low-rank diagonal blocks.
Block low-rank (BLR) matrix For ease understanding, let us consider BLR matrix of " rank-1 blocks. Each block is composed of the unique bases A., u Now consider U, [u:1 U422 ugaal and Vj [Ungj 22 Vg.g). Then, the BLR matrix BLAST matrix with
r = b 3:
U1 T
41,30 u2
u
UT USS.V U.S.V VT U U U USS.V UsS,V²
T U2
U2 22
T
U3,1 U,11
U3,20322
U3
where S = diag(() ([1, 0, 0)), S2 diag((0, [0, 1,)),, and S3 = diag((o, 0, 1]).
To model a m x n general b x partitioned BLR matrix where the rank of each block is 1, let us use the BLAST matrix with b blocks and r bt. The factors have the following shapes:
U,, Vj E (bt)
8i.j E Br
if if + + the BLAST matrix can model the BLR matrix.
By letting StJ.
otherwise
Note that the number of parameters of the BLAST matrix is 2nr + rd?, whereas that of the BLR matrix in this case 12 (p p)l 2(pb) 2nr. In other words, the BLASS matrix models various matrices with the cost of r
16

---

A.2 Derivation of Preconditioning Matrices
We rewrite the loss function in Equation (4) below:
e(U,, S.,x) 6
A U,d 2 F
We first derive gradients of U, Vj, and Big then discuss the preconditioning matrix for each factor.
A.2.1 Gradients
Here we derive the gradients of Equation (4) with respect to the BLAST factors. We begin with introducing the short-handed notation for the concatenated factors:
V." V ... V,,VV'), U, [U)))))) "F..)))""..(... (..((U,,.,))'''.
That is, the matrix Vi is composed by concatenating V horizontally along j = 2....... after scaling them with S.jj Uj is defined similarly by concatenating the scaled U,s vertically. Now we derive the gradients below.
Gradient of U; We only have to consider the loss term related to U, Therefore, we have the following gradient expression:
Vv,((U., V.,s..) Vv.
Σ
A U,diag(s:,, l
2
j=1
Ai.. - UU -V. 2
-U.V." A.,.
(10)
where for the second equality we used the concatenated version of the first line.
Gradient of V, follows the similar derivation as Equation (10):
b Vv, V.,), Vv,
Σ
Arj Unding (8i,) V
2
i=1
1
=Vv, [AA.J-U,V
= =ÜU,V/ - A..))
(11)
Gradient of stj
We consider the block-wise loss for the gradient:
Biy ((U,, Ve, s.,.) = V ex, 2 || - U,diag(s;.) V
(12)
Since the Frobenius norm can be expressed by matrix trace, the loss is written as follows:
| U.diag(S:.) -Tr (AA - U,S.oVy)) (A (.) TT UV -US + Aid)
= VS. VV 28,, V AT, U. + AT Aij),
where T)(X) is the trace of X. Note that the derivative of product in trace is given by V Tr(XY) YT for any two conformal matrices X and Y. Therefore, we have
1 S.... 2 | U,diag(s:. V 1
-
17

---

Now Equation (12) becomes as follows:
V a..
1 2 A - U
diag V,V
(13)
The first term on the right hand side is further arranged by using the fact that diag (XYT) = (X Y) for any two matrices X, of the same size.
diag U(U7 (V) Vj) [(0 U,diag ((,)) @ (V, Vi) 1,
= ((UTU.) o (V v;)) 81.
(14)
Hence, the gradient of $i, is now expressed as follows:
VaL. (U,,, V., s.,.) = (IU) U.) VV) 8t diag (U? A,, V2).
(15)
A.2.2 Preconditioning Matrices
Now let us derive the preconditioning matrices used Algorithm 2.
Preconditioning matrix Pu, for U, Let V, and S1,, are given. Let us consider the case when U; at the stationary point U which satisfies
1 Po514 A V
= A.
O,
where is the zero matrix. This gives us the normal equation
A, = ÜV. V.
(16)
Now consider a preconditioned gradient descent with Pu. E R'
U = U, (U.V." V. A..V.) Pe.
= =U, (U.V.V - Üv.TV. Po
= =U, (U--U) U) V.
= - = (U, - ú)) (I-
Suppose V." V, invertible. Then the ideal preconditioner is
Pt. V.)
(17)
since it brings U to the stationary point U. However, directly the inverse of V might result in numerical instability or complete breakdown of the algorithm when the matrix is singular. By following [24], we use the regularized version
Pu. = V.
(18)
where 5 chosen by
8 = 6n v (U,, V., 8.,)), 80 >0 0.
(19)
Preconditioning matrix Pv, for V, The preconditioner for V, can be derived by following the similar steps for Pu,. Here we present the result:
Pv, (070, +5) -1
(20)
where 8 is chosen by Equation (19).
18

---

Preconditioning matrix Ps.,, for stjj We again consider the stationary point & or the diagonal
version S = diag($) which has zero gradient:
V V - (U)) U.) (V, V)) diag
((UFU)) (V, 8- - diag (UT A V,) = 0
Therefore, Equation (15) can be written as follows:
V s., (U. V., s.,,) = U.) @ (V)VV))) (5.,) - -).
Now consider the preconditioned gradient descent:
| ši.j Pot., (LU) U.) о (V, (8t,) 8)
8/.5 - S.. - Pon. (UUTU) O V2)) (8L. - 8)
-I-- (I-Pa.. PAcs ((U) @ (V Vi))) (8i.)
The Hadamard product of two positive definite matrices are still positive definite (see [60, Theo-
rem 7. also known as Schur's Product Theorem). Hence, if both U, and V are positive definite so that invertible, U.) (V Vj) also invertible. The ideal preconditioner when both
matrices are invertible is therefore
P.,, = (UUTU.) o (V)
(21)
Same as for Pu. and Pv,, we also consider the regularized version:
Pe., = OV,
(22)
Here, 8 is chosen by Equation (19).
B Proof of Theorem 1
We first introduce the following properties:
Lemma 2. f61, Lemma 3.4] Assume f R.n TR. is convex and continuously differentiable, and its
gradient is T.-Lipschitz continuous. Then for any n. y R one has
L
f(y) f(x)((()))))))) - (V)xx) - x) S 2
y - 1
Proof. See [61, Lemma 3.4].
Lemma 3. Assume f R + R is convex and continuously differentiable, and its gradient is L-Lipschitz continuous. Consider a gradient descent update
x (k+1) (k) 17 (x)
Then, with the step size 0 < <nS S t, the following holds:
((x(h-1)) S (xx)
2
1
( ).
That is, the gradient descent update does not increase the function value.
Proof. By Lemma 2,
f(((()) (V)(x)))) x((+1) - (k)) + To [|o((++1)- - xx() $
2
= =f(x(()) .
(V
2
= )((((0) - - 1/h,)
[V)( >1
2
{1(((())) 377 | >13
(since T/L. S 1)
1
< {(((()))
V
11
2T.
19

---

Now we prove Theorem 1, which we restate below.
Theorem Let Aig E PPxp be a target block and U vi(h) E RPxr and 8,3
(k) RR' be factors of
a block in the BLAST matrix to be optimized. With the step sizes 0 < Tot S 11/0 (
0 ( 1/01 (
na S 1/01 ((U U ull+tl 1)
@ (V, (y v 1))
the gradient descent updates in Equations (5) to (7) monotonically non-increase the loss:
{(U!++1)
V (k+1) s.,t (++))) S =EUU!M) v.
Proof. To prove Theorem I, we first show that each step of Equations (5) to (7) satisfies Lemma under the given conditions. Then resemble the results to construct the bound.
Gradient Descent Update on U; Let us denote the loss term regarding U; by
((U.) - 1 - U.V.
Since (i) the Frobenius norm is convex, (ii) all linear mappings convex, and (iii) composition of two convex functions are convex, ||A U, I I
a convex function of U,. Also, the gradient we derived in Equation (10) always exists and has the following vectorized form:
vec vec( (V((U.)) vec(U, V Vi) vec re, Vi)
((V,"V)) & I)u; - vec A., Vi).
where & denotes the Kronecker product and u, VU, The Lipschitz constant of the gradient is the largest singular value of the matrix VV.'V)² @ I, which is the largest singular value of V." V; since the Kronecker product of two matrices of singular values Ej and Σg has the singular values of Σ1 @ Σn (see [62, Theorem 4.2.15).
Therefore, from Lemma obtain the following bound:
II (( M
((U < ((U
i I,...,b.
(23)
20 (((T)T((11)
Gradient Descent Update on Vi The loss function {(V)) = 2 - with respect to V, convex to Vj and the gradient of Vj in Equation (11) also always exists and can rewritten as follows:
vec V((V))) veo -eo vec
((0) 1 I)U; vec ( U,).
The Lipschitz constant of the gradient is again the largest singular value of UT We have the bound from Lemma 3 similar Equation (23):
IV (V (1 >1
((((-1)
((V
j == 1,,.....
(24)
201 (U 7,00)
Gradient Descent Update on 8;. The loss function
((81g) ||, U.ding(st,) (si
is also convex in súg since diag(.) is convex mapping, We know the gradient exists from Equa- tion (15):
V((s..)) diag (Uf UA.) diag (U. A V,) U(U @ V,TV,) 81,j diag (U, A, V2)
20

---

and the Lipschitz constant of the gradient is 01 (UU?U,) @ (VTV,)) The bound from Lemma
for the diagonal factors is follows:
II ((5 (
[()))) ( S ((5)5)
1 = I.....b.
(25)
201 (U, o (V)
Combining Equations (23) to (25), we retrieve the bound in Theorem 1:
E(U!i+)) Vi(a), s( = (lu! Vi s
!(U!ht)) v. (1+ sa.)) s ((U!ht)) v, s. ((U!tt)) v(k+1) 1 s Ae!ttt v (i
(U!t+ v((++1) 02, ) s (UU! v,(h), (!!
C Experimental Details
In this section, we provide the experimental details. Throughout the experiments, we used NVIDIA A40 GPUs 4 NVIDIA L40S GPUs for training and evaluation, and single NVIDIA A100 GPU with 40GB memory for runtime evaluation.
C.1 Datasets and Benchmarks
Image Datasets For image classification tasks, we use CIFAR-10 [28]. CIFAR-100 [28], and ImageNet-Ik [29] datasets our experiments. CIFAR-10 and 100 contain 50,000 training and 10,000 test images, each of which is 32 x 32 color images of 10 and 100 classes, respectively. ImageNet- 1k consists of 1,281,167 training and 50,000 validation images of 1,000 classes.
Common Sense Reasoning Benchmarks For our large language model evaluation, we use the following common sense reasoning benchmarks: Physical Interaction: Question Answering (PIQA) [35], HellaSwag[36) WinoGrande((37] (37], BoolQ[38),
(reeBBookAA(9)) A12's Reasoning Challenge (ARC)-easy and challenge [40]. PIQA targets the task of physical common sense reasoning, with 16,000 examples for training, 2,000 for development, and 3,000 for testing. HellaSwag is composed of 10k questions that are specifically hard for the machines, though trivial for humans (95% accuracy). Winogrande is a large-scale dataset of 44k pronoun resolution problems. BoolQ consists of 15,942 yes/no questions. OpenBookQA is modeled after open book exams for assessing human understanding of subject, containing 5,957 multiple-choit choice elementary- dev, 500 test). A12's s Reasoning Challenge (ARC) dataset consists of 7,787 multiple-choice science exam questions, and the questions are categorized into "easy" and "challenging" subsets.
science questions (4,957 train, 500
C.2 CIFAR-10/100 and ImageNet-1k Image Classification Training
For CIFAR- 10 and CIFAR-100 training, we trained the ViT-Small models with 4 4-sized patches [19]. We trained ViT-Base models with 16 x 16-sized patches for ImageNet- Ik training. All models trained by the AdamW [22] optimizer.
In the ViT models, we replaced the weight matrices of query, key, and value projection layers in one attention module and those in the feed-forward modules. In addition, we stacked the weights of query, key, and value weights and modeled them by one BLASS matrix.
The BLAST factors were randomly initialized to have the desired standard deviation = 0.02 while having zero-mean, where the standard deviation 0.02 was also used for initializing the weights the original-sized ViTs. Specifically, we initialized the factors as follows:
Vi = 1, ...,. U, (NO, vo.02I),
V, ~ ((0,, V0.02T), $ij Unif 2.0),
Vj = 1,.,.... Vi,j 1.......b
21

---

where (O.0 2.0) denotes uniform distribution on the interval [0, 2]. For all models, we applied AutoAugment [63]. Table 5.
We summarize the training hyperparameters in
Dutoses
Model
Epochs
Weight Decsy
Botch Size
Warmap Epochs
Warmup Start
Schoduler
LR
LR
LR Min
Dropout
Drogpath
BLAST
CIFAR-10
CUFAR-100
VT-S
310
cosine
Se-4
ImageNet
VIBB
310
0.05
1024
10
le-6
cosine
Ie-3
1e.5
0
3
Table 5: Hyperparameters used in training from scratch.
C.3 Compression and Re-training
VIT on ImageNet-1k For ImageNet-Ik Ik compression and re-training, we followed similar strategy to the ImageNet training with minor changes, summarized in Table 6. The ViT-Base with 16 x 16- sized patches was chosen baseline model. We decomposed the pre-trained weight matrices of ViT-Base by Algorithm 2 with K = 300 and du 0.1. Also, we linearly decayed the step size from 1.0 0.0. Then, all compressed models were trained by the AdamW [22] optimizer.
Model
Epochs
Weight Decay
Batch Size
Warmup Steps
Warmup Start
Scheduler
LR
LR
LR
Dropout
Droppath
BLAST
Net
ajma
La7e3
Table 6: Hyperparameters used in re-training.
Diffusion Model We compressed the DīT-XL model with 2 x 2-sized patches [13], pre-trained on the 256 x 256 ImageNet training samples. DiT model a variant of Vision Transformer [19] with additional udaptive layer normalization (adaLN) [64]. Here, we compressed the stacked query, key, and value weights. the first fully connected layer the feed-forward module, and the adaLÑ projection layers by BLAST, or low-rank matrices. All weights were compressed by Algorithm 2 with = 500 and 6n = 0.1. We linearly decayed the step size from 1.0 0.0. The parameters of set have the desired compression ratio in total, i.e., we remove 50% (or 20%) of the total parameters out of the network. We present the summary in Tables and 8. We generated 50,000 images using the original, the cow-rank-compressee and the BLAST-compressed DiT.
the BLAST and the low-rank matrices
Layer Type
71
b
r
Layer Indices
3456
1152
9
384
0-27
FC1 adaLN_proj
4608
1152
9
256
0-27
6912
1152
9
256
0-27
Table 7: Hyperparameters used for the BLASTo-compressed DiT-XL/2 with 50% compression ratio.
m, n: size of the original matrix, b: number of row/column partitions, r: BLAST rank parameter, Layer Indices: indices of layers that the BLAST matrix replaces the weight.
FID, sFID and IS Evaluation We sampled the novel images using DDPM sampler [65] for FID, sFID, and IS evaluation in Table 2. The step size was set to 250 for each model. Then, the FID, sFID, and IS were computed between each pool of generated images and the 50,000 ImageNet validation images to estimate the distributional discrepancy between the target and the generated samples.
Large Language Model For the large language model compression, we used the Llama-7B pre- trained model, publicly available at https: //huggingface co/cooo///// ho/huggyllama/ hala/lamm ammmmmmmmmmmmmmmmmmmmmaammmaalla 7b. For the 50% compression ratio, we compressed all the weights in the main modules of Llama-7B with BLAST with the parameters described in Table 9. For the 20% and 10% compression ratios, we compressed the weights of Q_proj, K_proj, gate_proj, up-proj layers to match the target compression ratio of the total parameter counts. Also, by following [66], we compress Q_proj and K_ K_proj layers for the first 10 attention modules. The parameters used the experiment are summarized in Tables 8, 9 and 11. All weights were compressed by Algorithm 2 with K 300 and
22

---

Layer Type
b
Layer Indices
QKV _proj
3456
1152
9
512
0-27
FC1
4608
1152
9
640
0-27
adaLN_proj
6912
1152
9
768
0-27
Table 8: Hyperparameters used for the comppessedd DiT-XL/2 with 20% compression ratio.
m, n. size of the original matrix, b: number of row/column partitions, BLAST rank parameter, Layer Indices: indices of layers that the BLAST matrix replaces the weight.
5n = 0.1. We linearly decayed the step size from 1.0 to 0.0. The factorization process takes 3.38 GPU hours for the BLAST weights of b 16 NVIDIA A40 GPUs.
To re-train the compressed Llama models, we used a subset of the SlimPajama dataset [34] for 400 steps using 0.47B tokens. The global batch size was set to 576, and the models were trained on NVIDIA L40S GPUs. See Table 6 for details.
We used Language Model Evaluation Harness [67] for the zero-shot classification accuracy evalua- tion.
Layer Type
b
Layer Indices
Q_proj
4096
4096
16
1024
0-31
K_proj
4096
4096
16
1024
0-31
V_proj
4096
4096
16
1024
0-31
0_proj
4096
4096
16
1024
0-31
gate_proj
11008
4096
16
1488
0-31
up-proj
11008
4096
16
1488
0-31
down_proj
4096
11008
16
1488
0-31
Table 9: Hyperparameters used the BLAST ceomppeseeee Llama-7B with 50% compression ratio.
m, n: size of the original matrix, b: number of row/column partitions, BLAST rank parameter,
Layer Indices: indices of layers that the BLAST matrix replaces the weight.
Layer Type
b
Layer Indices
Q_proj
4096
4096
16
496
0-31
K K_proj
4096
4096
16
496
0-31
gate_proj
11008
4096
16
2048
10-31
up-proj
11008
4096
16
2048
10-31
Table 10: Hyperparameters used for the BLASTTe-compressed Llama-7B with 20% compression
ratio. m, n: size of the original matrix, b: number of row/column partitions, r: BLAST rank parameter, Layer Indices: indices of layers that the BLAST matrix replaces the weight.
D Additional Experimental Results
D.1 Synthetic Experiments on BLAST Factorization
In this experiment, we test the factorization algorithms discussed in Section 3, similar to Figure but with a different target matrix. To be specific, we synthesize a 256 x 256-sized BLAST1 (i.e., b = 16) target matrix with r" 8. Then, we compare the eтror curve along the iterates the gradient descent without preconditioning (GD) in Equations (5) to (7), and the preconditioned gradient descent (PrecGD) Algorithm 2. Here, we consider the ver--prameterrieed setting 32 > r" In Figure 9, unlike the low-rank target matrix, GD does not converge in both cases. However, the preconditioned version easily finds the low-error solution for the exact parameterization. For the overparameterized case, the preconditioned gradient descent method in Algorithm 2 achieved in two orders of magnitude improvement from the simple GD.
exact parameterization setting when r = r* and
harness
23

---

Layer Type
b
Layer Indices
Q_ Q_proj
4096
4096
16
1024
0-31
K_ K_proj
4096
4096
16
1024
0-31
gate_proj
11008
4096
16
2368
10-31
Table Hyperparameters used for the BLAST compprssed Llama-7B with 10% compression
ratio. m, n: size of the original matrix, b: number of row/column partitions, r. BLAST rank parameter, Layer Indices: indices of layers that the BLAST matrix replaces the weight.
10a
GD
10a
GD
10
PrecGD
PrecGi
10
10-1
10
10-5
10
10"5
10"1
BLAST + BLAST.
Figure 9: Plots of normalized reconstruction errors using the BLAST factorization with GD and GD with preconditioning steps (PrecGD) in both exact and rank overparameterized settings, when the target matrix is BLAST16 Left: Reconstruction errors when r = r Right: Reconstruction errors when >> r*.
Original
Low Low-Rank Rank
BLAST
Figure 10: Examples of generated images using both low-rank and BLAST decompositions, Both methods compress the original model by 20%.
D.2 Additional Results on Diffusion Model Compression
We include extended experimental results of the diffusion model compression in Section 4.2
Compression-only In Figures 10 and 11, the additional image samples of original uncompressed, low-rank-comen compesse,,,s and BLASTy-compressee the same column were sampled using 250 DDIM steps, starting from the same noise vector. The compression ratio was set to 20% for both models. The figures show that the outputs of model compressed by BLAST maintain similar features and perceptual quality to the outputs of the original DiT.
DiT [13] models are presented. The images in
Compression and re-training We present additional samples from the low-rank and BLAST DIT models at the 50% compression ratio after re-truining in Figure 13. Similar to Figure the images generated by the low-rank DiT lose significant image quality, whereas the images from the BLAST DiT preserve the quality and semantics of the original samples.
24

---

Original
Low-Rank
BLAST
Figure 11: More examples of generated images using both low-rank and BLAST decompositions.
Both methods compress the original model by 20%.
BLAST
Low-Rank
5
Figure 12: Comparison of the images generated by the BLAST and low-rank compressed models. Overall, the low-rank approximated model often generates unrealistic images, which contributes to low scores evident in Table 2.
Evidence of low performance of low-rank-compresse model Some images generated by the 20%
model (Figure 12-1eft) are highly unrealistic and have inferior quality compared to the images generated by original and BLAAST-coppressed models (Figure 12-right). We observe that these samples contribute to the low scores Table 2. These samples were computed using 250 DDPM steps, done by the original DiT work [13].
D.3 Additional Results on Large Language Model Compression
Compression-only We report the performance of LLM-Pruned [47] and Joint Rank-k [66] for the same compression task. LLM-Pruner [47] identifies sparse weights with data to pinpoint unimportant neurons. Joint Rank-k [66] performs low-rank approximation jointly weight matrices with similar column spaces by stacking them and applying a truncated SVD.
In Table 12, compare the performance degradation of LLM-rruner, Joint Rank-k, Low-Rank. BLAST2, and BLASTIe. as well as their absolute performance. The first four rows represent the zero-shot performance of Llama-7B [1] from the literature, while the row marked with an asterisk (*) indicates our results.
For 10% compression, Joint Rank-k achieved the lowest performance degradation, although BLAST also exhibited a similar performance drop. When model is compressed by 20% BLAST surpasses Joint Rank-k [66], L.L.M-runer [47], and low-rank schemes. The zero-shot accuracy versus compression ratio curve in Figure shows that BLAST compression results in less performance drop for the same compression ratio.
25

---

Original
Low-Rank
BLAST
Figure 13: Examples of generated images using DiT [13] starting from the same noise vectors and a deterministic solver. The original model is compressed by 50% through BLAST or Low-Rank matrices and re-trained for 10 epochs on ImageNet. The images from the model compressed via BLAST preserves the quality of the images of the original model, whereas the images generated by the low-rank model contain artifacts.
CR Mothed
PIQA
HellaSwag
W Wiogrradd
BaolQ
OIRQA
ARC
ABC-C
Average
76.1
N0.1
57.2
72.8
41.5
68.59
11aMA-78(47]
63.25
[1166-7R[66]
77.64
43.40
66.11
L1AMA-7B
79.16
76.19
70.09
75.11
44.4
72.9
44.71
66.08
Jain [66]
(791(1125
44.00-0000
41.22122.9)
Monarch" [14]
77.37(11.7)
19.100-7.09
67.96-2.113
71.711-3.400 40)
3:30000:305
(9.57(3.33)
4..61)-4.10)
62.851-3.28)
BLAST
78.78/
74.22(-197)
20.24(-0.15)
76.12(11.01))
42.00
71.21-1109)
43.260-1.45)
68.121-0.96)
LLM--runcr(77
666
66
598357.18)
57.066-16.2)
10.0002.40
60941-6311)
36.52(-186)
66.69;66.6)
Foint Rank-A[66]
75.08-2.556 56)
643733311
60.66-1666)
62 20-7.11)
43.00-0.40)
61 61.73(4.559 5.00
34.24(-3.39)
57.33(-4.03)
2016
Join Kank-A [7166]
Now-Rank"
3.260-3266
75 33.30)33.6)
65.53110066 10.66)
(3.20X-12.9))
65.11(4.988
65.3533244)
67 5.65333333
06 (6.64)-8.42)
42.201-2.200
2.30
66.791-6.11
68.91-4.999
38.651-6.00
38.65)-606)
5.550
59.57(-6.51)
Monarch"[144 [14]
22.31(-685)
42.380-33811
14.85)-15.24)
31.401-13.00)
51.470221423)
21.731-16.95)
48.911-17.17)
BLAST;;
76 12)-3.0.)))
260-009))
19(-90)))
17(-2.944)
43.601-0
67.264-5.64)
61,55((4.53))
BLAST
27.48-1148)
69.74(-6.45)
(8.03-22006
72.45(-2.66)
44.001-0.40)
68.641-4.26)
40.27(-4.44)
62.94(-3.14)
Table 12: Zero-shot performance of LLaMA-7B with various compression methods without re-
training. All models are not post-trained. CR denotes compression ratio. Bold indicates the best
performance under the same compression ratio. Underline refers to the lowest performance drop. BLAST, indicates the BLAST matrix with b x number blocks. The mark represents the results from experiment.
Compression and Re-training Table 13, we present the performance of each common sense reasoning benchmark which report their average in Table 3.
PIQA
Winogrande
BoolO
Average
LLaMA-7B
79.16
76.23
69.93
75.14
72.9
41.71
66.07
20%
BLA
17.97-1119
12.88-1.35)
69 63)-06-663)
1285-2.51
21:00-2.0
41.3002300)
65-75-2.54)
Low Kank
66.181 13.00)
18.311
15.15)
65. 35.38.9.700
31008 13.40
21449
27.73-16350
48.00- 17.57)
50%
Monarch
Diagonal
50.711-88.48
50.381-28.78)
26.171-50.6)
26.241-49.99)
50.591-19.38
39.66)-20.60
28)
24.80X-19.60
26.40V-1800
26.35(-66.8
26.44
(6.55)
28,07(-
RL
73886-5.333)
63.59(-12.64)
(3.38666.5)
68.62(-6.52)
54,0099.80 80)
57.24-15666)
32.34(-12.37)
26.256-9844
Table 13: Zero-shot performance LLaMA-7B with various compression methods after re-training.
CR denotes compression ratio. BLAST,, indicates the BLAST matrix with b x b number of blocks.
E Broader Impact
Our proposed method targets improving the efficiency of the DNN inference. This might have a negative social impact by promoting the accessibility and usability of malicious DNNs such as DeepFake. However, at the same time, we expect the BLAST matrix will bring tremendous positive social impact. First, it contributes to sustainability by cutting down the energy consumption for DNN inference. Moreover, the BLAST matrix can improve the accessibility of Al-based medical, educational, and social services by providing a foundation for running those models on mobile devices. Therefore, we believe BLAST will give tangible benefits to our society.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
Section 3 and B.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix C and section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: Appendix C and section 4. Our code is available at https://githhb..co com/ changwoolee/BLAST.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
cnooolicyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/ccc/ppaac sac/public//ee Seriidee/CoeeSuue wwwsssssssslsooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix C and section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We reported the standard deviation in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section and appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / /ttp//euripp..e/ cc
Answer: [Yes]
Justification: Our paper is written upon widely-used publicly available datasets and models. We discuss potential harmful impact in Appendix E.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: Instead of providing decomposed Llama- 7B model, we provide the code file to reproduce the result if one can access the original safeguarded model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Appendix C.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation is available in the project directory at https://githhb: com/ changwoolee/BLAST
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did conduct experiments with human subjects croudsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did conduct experiments with human subjects croudsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "INvBC7kinJ", "reviewer": "Reviewer_V5GM", "summary": "This paper attempts to find efficient structures in weight matrices of deep learning models. The basic idea is to learn a group of block-wise low-rank matrices via gradient descent. The proposed method replaces original dense weight matrices and hence needs retraining.", "questions": "* How different is this work compared with block sparse + low-rank [23] and Gaudi-GBLR [24]? Why is adaptive and learnable a new contribution when these ideas have already been proposed? The paper [23] is not introduced before use in experimental results.\n* In Table 4, when L is 100, the execution time of 20% compression ratio increases compared with no compression, while other two columns show slight reduction of execution time. What is the reason? \n* Please provide more details, even if it is obvious, that the learned BLAST matrix structures do not need any library function customization. What is specifically used as the backend compute kernel?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper introduces an original approach on replacing weight matrices of DNNs to be block-wise low-rank through training.\n* The evaluation includes multiple models and different tasks, showing good generalizability of this work.", "weaknesses": "* The significance of the proposed method remains unclear. Compared with prior methods such as Gaudi-GBLR [14], the accuracy results of this work do not show consistent superior performance nor with a convincing explanation.\n* The method does not seem to improve efficiency much, showing limited significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. The significance of the proposed method remains unclear. Compared with prior methods such as Gaudi-GBLR \\[14\\], the accuracy results of this work do not show consistent superior performance nor with a convincing explanation.\n\nPlease refer to Q3 of “Global Comments.”\n\n### Q2. The method does not seem to improve efficiency much, showing limited significance.\n\nFirstly, the number of parameters in the ViT model for ImageNet classification is reduced by 72.2% with BLAST, while achieving **higher** accuracy than Gaudi-GBLR and even the dense baseline. This was done without any hyperparameter tuning. Also, unlike Gaudi-GBLR which requires specialized hardware and/or library to achieve inference speed-up (which has not been demonstrated), BLAST shows significant speed-up on off-the-shelf CUDA GPUs.\n\nSecondly, compressing the LLMs with structured sparsity (including structured matrix) is extremely challenging. The 20% compression ratio was chosen and accepted in the previous work on structured LLM compression (LLM-Pruner \\[30\\]). Note that LLM-Pruner is not data-free, whereas BLAST does not need *any* data to compress LLM. Still, BLAST outperforms LLM-Pruner in Table 3 with noticeable gaps. \n\nLastly, we conducted additional experiments on 30% compression ratio for diffusion models (Figure A in the attached pdf) and on 40\\~50% compression ratio for Llama-7B (Figure B in the attached pdf). Despite the difficulty of the data-free compression task beyond 20% ratio, the additional results show that BLAST preserves more accuracy on zero-shot classification and more details on image generation than the non-learnable baselines. \n\nFor the generative tasks  (e.g., Figure 1 and Table 3), we stress that the **goal of model compression is not necessarily to reproduce the outputs of the original model**. The main focus should be on  whether the compressed models’ outputs have details that look realistic (i.e., whether the ‘compressed’ generative model is still a valid model for the generative task). Based on the quantitative results in Table 2, we show that the perceptual quality of the BLAST-generated images is close to the quality of the images generated by the uncompressed model.\n\n### Q3. How different is this work compared with block sparse \\+ low-rank \\[23\\] and Gaudi-GBLR \\[14\\]? \n\nPlease refer to Q1 of “Global Comments.”\n\n### Q4. The paper \\[23\\] is not introduced before use in experimental results.\n\nThank you for pointing this out, we will introduce it before the experimental result section.\n\n### Q5. In Table 4, when L is 100, the execution time of 20% compression ratio increases compared with no compression. What is the reason? \n\nTable 4 actually shows that the execution time **decreases** with compression with L=100. This may have been an error made by the reviewer.\n\n### Q6. Please provide more details that the learned BLAST matrix structures do not need any library function customization. What is specifically used as the backend compute kernel?\n\nPlease refer to Q4 of “Global Comments.”\n\nWe truly appreciate Reviewer V5GM’s feedback. Please let us know whether our comments have successfully resolved all of your concerns or not. If so, we kindly request reconsidering the evaluation of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3y3kx9pCPy", "reviewer": "Reviewer_xrxa", "summary": "The authors propose a learnable compressed representation (BLAST) for weight matrices used in deep learning which enables lower complexity matrix multiplications which approximate the full, uncompressed operation.  A BLAST matrix decomposes the original matrix into a grid of blocks of diagonal matrices with shared low-rank factors along each (block) row and column, and these parameters can be learned directly from a random initialization.  Converting a dense matrix to a BLAST matrix by minimizing the Frobenius norm error between original and compressed matrices can be done through gradient descent factorization, and preconditioning can accelerate convergence.  The proposed technique is evaluated in pre-training, fine-tuning, and data-free compression scenarios, using a variety of networks and tasks (vision transformers and LLMs, image classification, image generation through diffusion, and zero-shot language modeling).", "questions": "1. Is matrix-matrix multiplication built by performing multiple matrix-vector products (Algorithm 1), or is there something more clever that can be done?\n\n2. What is the difference, in line 100, between \"learning\" and \"finding\" the structure of a weight matrix?\n\n3. What is the benefit (other than parameter reduction) in sharing the U and V factors along rows and columns?  Can it be quantified?\n\n4. What rank *r* is used for the experimental results?\n\n5. Regarding skipping some baselines for comparisons, were these techniques simply not designed for this task, or are they harder to adapt to this task, or are they entirely incompatible?  \n\n6. What batch size and input sequence length was used for the results in Table 4?", "limitations": "The authors' discussion is excellent, only missing one potential further limitation: this technique seems like it'd need extra work to apply to higher-dimensional tensors.  I could imagine applying it on matrix-slices of a tensor, but this may not work well.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Originality**: The proposed approach seems to be entirely novel, building on existing low rank and block sparse techniques.\n\n**Quality**: In general, the major claims are backed up with the necessary experiments - the impact of preconditioning on convergence speed is demonstrated, and the comparisons against the chosen baseline suggest that the BLAST format is superior in many cases, and at least comparable in most.  The breadth of evaluations is welcomed - different types of models and tasks can behave differently to different techniques, but the applicability of BLAST across this variety is demonstrated appropriately.\n\n**Clarity**: The writing is clear, and the organization is fantastic.  A reader can quickly understand what sets BLAST matrices apart from other techniques, if not the importance of its constituent components (see weaknesses below).\n\n**Significance**: Simplifying deep learning in practice, either by reducing memory or compute costs is an important area, and this submission seemingly approaches both angles.  The relative simplicity of the format and the immediate benefits (shown in Table 4) may make this an attractive technique for practitioners to adopt.", "weaknesses": "**Originality**: While the authors included a sparse-weight baseline (LLM-Pruner), [SparseGPT](https://arxiv.org/abs/2301.00774) deserves mention (and potentially inclusion as a baseline method to compare against).\n\n**Quality**\n\nThe authors claim in line 310 that BLAST matrices allow \"more efficient execution in hardware for the same FLOPs (in Figure 5, Figure 6, and Table 1),\" but these figures and table don't tell the reader anything about hardware efficiency.  While the authors have presented an algorithm for matrix-vector products, I'm not convinced that matrix-matrix multiplication is as efficient as optimized dense routines tensor execution units in any recent hardware.  (The runtime speedups in Table 4 may simply be due to the memory and bandwidth savings, not an computational gains.)\n\nThe authors stress that U and V are shared along rows and columns, but it's unclear why this is so important.  What if each block has its own U and V -- other than increasing the number of parameters, would the factorization process, quality of the results, multiplication algorithms, etc. change?\n\nIn line 245, the authors suggest that certain techniques \"have not been studied as data-free compression techniques\" and are therefore \"excluded from the comparison\" for this task.  I'm not sure if this is fair or not: were these techniques simply not designed for this task, or are they harder to adapt to this task, or are they entirely incompatible?  \n\n**Clarity**: there are some configuration details missing:\n\n- The rank *r* used for the experimental results.  (I see this detail is in the appendix, but I think it's useful to give at least a range of the values used in practice, as was done with *b*, in the main content.)\n\n- The batch size and input sequence length used in the LLM runtime analysis.\n\n\nFigure 4 is lacking vertical axis labels.\n\nI'm not sure of the difference between, in line 100, \"learning\" and \"finding\" the structure of a weight matrix.\n\n\n**Significance**: the quality of the results lets the submission down, in a way.  Compression rates of 10%-20% are fairly limited, and I'd be curious to know what happens, even if it's catastrophic, at higher compression rates -- even 50% would be informative.  Further, the quality of the BLAST diffusion model's output, while better than SVD's, still fall short of the baseline.  Fine detail is lost (in e.g. the water and rocks of the fourth column, the feathers in the fifth column, and the jersey in the sixth), and larger distortions appear (the hippo's snout no longer looks accurate, the bird's eye has disappeared, and the hot air balloon's shape is extremely lopsided).  Again, these results are better than the other Low-Rank technique, but practitioners may be hesitant to use the technique for this task without significant gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. SparseGPT deserves mention.\n\nWe agree that SparseGPT \\[Frantar & Alistarh, 2023\\] is an interesting related work that deserves mention. SparseGPT uses *unstructured pruning,* which makes it inefficient compared to other structured matrices (including BLAST) for GPU execution since the pruned weights are randomly placed without structured patterns. Also, the SparseGPT compression process is *not* a *data-free* method since it is based on the Hessian matrix for pruning. In contrast, BLAST can be applied to data-free compression as presented in Algorithm 2, and its matrix multiplications with structured sparsity can be accelerated on off-the-shelf CUDA GPUs with significant inference speed up.  \nWe will include SparseGPU discussion and comparison in the revision.\n\n### Q2. The result supports improved hardware efficiency but the runtime speedups in table 4 may be due to the memory and bandwidth savings.\n\nThank you for the insightful feedback\\! We agree that memory savings may be the dominant contributor to the runtime improvement in Table 4. We observed that the BLAST matrix multiplication is faster than the dense routine when the weight matrix is large enough. Yet, memory and bandwidth savings are also critical components of hardware efficiency, thus should be rewarded.   \nMoreover, there is room for improvement to further optimize the BLAST matrix multiplication kernel using optimized / modified CUDA functions, which we left for future work. The optimized kernel can be realized augmenting the gain from computation savings and the reduced memory access overhead.\n\n### Q3. The authors stress that U and V are shared along rows and columns, but it's unclear why this is so important. \n\nBy sharing U and V factors, we can **significantly reduce the number of parameters** while being able to model multiple structures by the diagonal factors. If the bases are not shared, more parameters are required to achieve a similar level of expressivity.   \nIndeed, a BLAST matrix without parameter sharing is identical to a Monarch (i.e., BLR) matrix. Under a similar number of parameters, BLAST matrices outperform BLR matrices in DNNs’ accuracy as we showed in Figures 5, 6, and Table 1. \n\n### Q5. Why are some baselines skipped for data-free compression? \n\nPlease refer to Q2 of “Global Comments.”\n\n### Q6. Compression rates of 10%-20% are fairly limited. What happens at higher compression rates? . Fine details are missing in the images generated by a diffusion model with BLAST weights, although BLAST shows better image quality than Low Rank.\n\nCompressing the LLMs with structured sparsity (including structured matrix) is extremely challenging. The 20% compression ratio was chosen and accepted in the previous work on structured LLM compression (LLM-Pruner \\[30\\]). Note that LLM-Pruner is not data-free, whereas BLAST does not need *any* data to compress LLM. Still, BLAST outperforms LLM-Pruner in Table 3 with noticeable gaps. \n\nWe conducted additional experiments on 30% compression ratio for diffusion models (Figure A in the attached pdf) and on 40\\~50% compression ratio for Llama-7B (Figure B in the attached pdf). Despite the difficulty of the data-free compression task beyond 20% ratio, the additional results show that BLAST preserves more accuracy on zero-shot classification and more details on image generation than the non-learnable baselines. \n\nFor the generative tasks  (e.g., Figure 1 and Table 3), we stress that the **goal of model compression is not necessarily to reproduce the outputs of the original model**. The main focus should be on  whether the compressed models’ outputs have details that look realistic (i.e., whether the ‘compressed’ generative model is still a valid model for the generative task). Based on the quantitative results in Table 2, we show that the perceptual quality of the BLAST-generated images is close to the quality of the images generated by the uncompressed model.\n\n### Q7. Is matrix-matrix multiplication built by performing multiple matrix-vector products (Algorithm 1), or is there something more clever that can be done?\n\nPlease refer to Q4 of “Global Comments.”\n\n### Q8. What is the difference, in line 100, between \"learning\" and \"finding\" the structure of a weight matrix?\n\nWe differentiate “learning” and “finding” in the following manner: “learning” indicates the structure of a weight matrix is inferred by minimizing the DNN’s prediction loss which involves *datasets*, whereas “finding” is reserved for the *data-free* compression in Algorithm 2\\.\n\n### Q9. What rank r is used for the experimental results?\n\nWe vary the rank of the BLAST matrix (from 48 to 192\\) to adjust the compression ratio of the models in Figure 5 and Figure 6\\. For ImageNet training experiments in Table 1, we use r=128. \n\n### Q10. What batch size and input sequence length was used for the results in Table 4?\n\nThe batch size was set to one, and the length of the input sequence was set to 3\\. Specifically, we used the following prompt to let the model generate the desired sequence length: “Increasing sequence: one,” and stopped the generation process when the generated sequence length approaches L=10, 100, or 1000\\.\n\n### Q11. Figure 4 is lacking vertical axis labels.\n\nThe normalized Frobenius norm error was used in the vertical axes of Figure 4\\. We will include the label.\n\n### Q12. This technique seems like it'd need extra work to apply to higher-dimensional tensors.\n\nWe do not see a straightforward approach to extend BLAST to high-dimensional tensors. We will discuss this as one of the limitations.\n\nWe truly appreciate Reviewer xrxa’s feedback. Please let us know whether our comments have successfully resolved all of your concerns or not. If so, we kindly request reconsidering the evaluation of our work.\n\n**Reference**  \n\nFrantar E, Alistarh D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning 2023 Jul 3 (pp. 10323-10337). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fNlwe4eNLx", "reviewer": "Reviewer_2dKz", "summary": "This paper proposes to compress linear layers by using a block low-rank structure with shared basis -- also known as the BLR^2 matrix structure. The basis U, V, and diagonal weights s of the low-rank block are computed through a gradient-based optimization that minimizes the Frobenius norm between the dense and low-rank matrix. They also use preconditioning to accelerate the convergence.", "questions": "Why does the paper repeatedly mention matrix-vector multiplications? The input to DNNs are not vectors.\n\nWhy do the authors mention that \"A block low-rank (BLR) [12], also known as Monarch [13] matrix\"? The Monarch matrix is a very different structure from BLR. \n\nWhy are Gaudi-GBLR, BSP+LR, and BLR excluded from the compression benchmark in Table 2? Just because they haven't been studied as data-free compression techniques doesn't mean they are not suitable for this task.", "limitations": "The limitations are stated clearly in a separate section.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Their experiments of ViT-s on ImageNet-1k show an improvement in validation accuracy compared to existing methods such as Gaudi-GBLR.", "weaknesses": "What is proposed as the BLAST matrix in this paper is an existing matrix structure known as BLR^2 [https://hal.science/hal-03070416]. The original BLR^2 structure uses LQ decomposition to compress the low-rank blocks, but the name of the structure is independent of the method used to compress the low-rank blocks. Therefore, the proposed method which uses a gradient based iterative method to compress the low-tank blocks is still a BLR^2 structure. The BLR^2 paper should be cited, and all mention of BLAST should be replaced with BLR^2. \n\nFor CIFAR-10 and CIFAR-100, Gaudi-GBLR seems to perform better than the proposed method. The authors mention that Gaudi-GBLR's \"capability of learning the adaptive resource/budget allocation for each weight matrix\" is the cause, but it is unclear why this is not the case for ImageNet-1k.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1. The BLAST matrix in this paper is an existing matrix structure known as BLR^2. The BLR^2 paper should be cited, and all mention of BLAST should be replaced with BLR^2.\n\nThank you for suggesting an important related work\\! We agree that BLR^2 \\[Ashcraft, Buttari & Mary, 2021\\] should be cited and carefully discussed.\n\nHowever, after carefully reviewing both approaches, we concluded that BLAST is **not identical to** BLR^2. One *cannot* replace BLAST with BLR^2 to obtain the same level of generality and efficiency that BLAST provides. Therefore, we think BLAST should be named as-is.\n\nThe BLAST matrix structure differs from the BLR^2 \\[Ashcraft, Buttari & Mary, 2021\\] method in its left, right, and diagonal factors.   \n(i) The matrix ($S\\_{i,j}$) of a BLAST block ($A\\_{i,j}=U\\_i S\\_{i,j} V\\_j^T$) is a diagonal matrix, whereas BLR^2 uses a *low-rank* matrix. Accordingly, a BLAST matrix can capture **high-rank blocks** with **fewer parameters** than BLR^2. (ii) BLR^2 restricts its shared left and right factors to have *orthonormal* bases, which introduces extra overhead and instability at each gradient descent update. In contrast, *none* of the BLAST factors have such orthonormal constraints so that the training process does not change from the conventional DNN training scheme. In revision, we will cite BLR^2 and discuss the work.\n\nWe argue that  BLAST is more suitable for efficient DNN inference and simple training than BLR^2 for the following reasons:\n\n**Expressiveness:** BLR^2 uses a low-rank coupling matrix whereas BLAST utilizes a diagonal matrix. As a consequence, BLAST can represent matrices that BLR^2 cannot. BLR^2's low-rank coupling factor fails to capture high-rank blocks, whereas BLAST's diagonal coupling factor can model high-rank blocks with fewer parameters.  \n**Efficiency:** Each low-rank block of a BLR^2 matrix can have a *different* rank, making it challenging to achieve optimal efficiency on the off-the-shelf GPUs due to *zero paddings*. On the other hand, the diagonal factor of a BLAST matrix *does not require zero padding*, allowing for faster matrix multiplication on GPUs, as illustrated in Algorithm 1 and the pseudocode in \"General Comments.\"   \n**Gradient Descent Compatibility:** BLR^2's orthonormality constraint adds computational cost and instability during gradient descent steps in training and fine-tuning. In contrast, BLAST avoids these issues as it does not require modifications to the training process.\n\nTherefore, unlike BLR^2, BLAST exhibits unique properties beneficial for efficient DNN inference as claimed in our paper. Once again, we truly appreciate Reviewer 2dKz's constructive suggestion to discuss BLR^2 in our work.\n\n### Q2. For CIFAR-10 and CIFAR-100, Gaudi-GBLR seems to perform better than the proposed method. \n\nPlease refer to Q3 of “Global Comments.”\n\n### Q3. Why does the paper repeatedly mention matrix-vector multiplications? The input to DNNs are not vectors.\n\nSince matrix-matrix multiplication is composed of multiple parallelized matrix-vector multiplication, we discussed BLAST at the matrix-vector multiplication level.   \nWe also thought the vector-shaped inputs were easier to express than the matrix-shaped or tensor-shaped inputs.   \nPlease refer to Q4 of “Global Comments” for the actual implementation.\n\n### Q4. Why do the authors mention that \"A block low-rank (BLR) \\[12\\], also known as Monarch \\[13\\] matrix\"? The Monarch matrix is a very different structure from BLR. \n\nThank you for pointing it out. We assumed the BLR matrix has the blocks of the same rank. In this case, BLR and Monarch are *equivalent* up to the row permutations. Indeed, the Monarch matrix is decomposed by the block-level SVD, which results in the block low-rank matrix of the same block rank. We will clarify that we assume the same block rank for BLR. \n\n### Q5. Why are Gaudi-GBLR, BSP+LR, and BLR excluded from the compression benchmark in Table 2?\n\nPlease refer to Q2 of “Global Comments.”\n\nWe truly appreciate Reviewer 2dKz’s feedback. Please let us know whether our comments have successfully resolved all of your concerns or not. If so, we kindly request reconsidering the evaluation of our work.\n\n**Reference**\n\nAshcraft C, Buttari A, Mary T. Block Low-Rank Matrices with Shared Bases: Potential and Limitations of the BLR ^2 Format. SIAM Journal on Matrix Analysis and Applications. 2021;42(2):990-1010.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers’ efforts and constructive suggestions. We are encouraged by the reviewers’ positive feedback, specifically in that “the proposed approach seems to be entirely novel” (Reviewer xrxa) and that our approach “shows an improvement in validation accuracy compared to existing methods such as Gaudi-GBLR” (Reviewer 2dKz). In the following, we answer a few concerns raised by multiple reviewers. We also attach a *PDF file summarizing the additional experimental results.*

### Q1. Contributions of BLAST?

Firstly, we clarify the structure of the BLAST matrix to emphasize its differences from existing methods such as LR, BLR, and BSP. The key difference is the flexibility of the BLAST matrix - its unique formulation can **learn the structure** of the weight matrix rather than **forcing it to have a specific (hand-crafted) structure**. For example, as we present in Section 3 and Appendix A.1, based on the diagonal terms $S\_{i, j}$, the BLAST matrix can capture multiple structures. This is done in an efficient manner that consists of equally partitioned blocks with structure-defining parameters, allowing for (i) better accuracy-efficiency tradeoff, (ii) inference speedup, and (iii) effective data-free model compression. BLAST also benefits from low training overhead – BLAST has much less hyperparameters that are less sensitive to training stability compared to those in Gaudi-GBLR and BSP+LR.

The BLAST matrix structure differs from the BLR^2 \[Ashcraft, Buttari & Mary, 2021\] (a related work suggested by Reviewer 2dKz) method in its left, right, and diagonal factors.   
(i) The middle factor ($S\_{i,j}$) of a BLAST block ($A\_{i,j}=U\_i S\_{i,j} V\_j^T$) is a diagonal matrix, whereas BLR^2 uses a *low-rank* matrix. Accordingly, a BLAST matrix can capture **high-rank blocks** with **fewer parameters** than BLR^2. (ii) BLR^2 restricts its shared left and right factors to have *orthonormal* bases, which introduces extra overhead and instability at each gradient descent update. In contrast, *none* of the BLAST factors have such orthonormal constraints so that the training process does not change from the conventional DNN training scheme. In revision, we will cite BLR^2 and discuss the work.

### Q2. Exclusion of (some) baselines in experimental results

We excluded BSP+LR (Pixelfly) \[23\] and Gaudi-GBLR \[14\] matrices from the compression benchmark since they do not have dedicated compression algorithms, nor can they be easily decomposed. Comparison with BLR is feasible as  a decomposition algorithm is available \[13\]. Hence, we conducted the data-free compression experiment with the BLR weights on Llama-7B. The BLR format did not outperform the other baselines (or BLAST), as presented in Table A in the attached PDF.

### Q3. Results are not significant, especially compared to Gaudi-GBLR

Firstly, we would like to state that methods such as Gaudi-GBLR (along with others such as BSP+LR) are not designed for **data-free model compression**. This makes our BLAST approach a unique contribution in this context.  
Secondly, for pre-training/fine-tuning experiments on CIFAR-10/100   and ImageNet, we had to extensively  tune Gaudi-GBLR’s 4-8 hyperparameters with consultation with the authors of Gaudi-GBLR. Because of this hyperparameter tuning difficulties, Gaudi-GBLR performs worse than our BLAST for the ImageNet benchmark (while the opposite is true for the simpler CIFAR-10/100 benchmark). The difficulty of tuning and training for Gaudi-GBLR makes BLAST weight matrices a stronger contribution.The BLAST weight matrices are much **easier to train** because no hyperparameters are introduced except for the rank and number of blocks. This explains why **BLAST outperforms Gaudi-GBLR on ImageNet** where hyperparameter tuning is more challenging. This indicates that the BLAST matrix is better suited for large-scale training.   
Thirdly, BLAST shows the significant inference speedup using an off-the-shelf GPU and standard CUDA library functions, whereas Gaudi-GBLR does not show such inference speedup gains on the same setup as it requires specialized hardware and/or optimized library functions to fully realize its claimed gain.

### Q4. How is BLAST matrix-matrix multiplication done?

The matrix-matrix multiplication between a BLAST matrix and an input X is done with **a few lines of** **basic tensor operations**–batched matrix-matrix multiplication (e.g., torch.bmm), reshaping, transpose, element-wise multiplication, and reduction by sum. We present a brief pseudocode below. 

```python
# Pseudocode of Blast Matrix-Matrix Multiplication

def blast_matmul(
        X,  # input,        shape=(B, n, q*b), B=batch_size, n=num_seq
        U,  # left factor,  shape=(b, p, r), b=num_blocks, r=rank 
        S,  # diag factor,  shape=(b, b, r)
        Vt, # right factor, shape=(b, r, q)
    ): # output = X @ A.T where A is a BLAST matrix of shape (b*p, b*q).
								 
    X = X.rearrange(b, B*n, q)
    Y = bmm(X, Vt.T) # multiply right factor
    Z = Y.unsqueeze(0) * S.unsqeeze(2) # multiply diagonal factor
    Z = Z.sum(1) # aggregate, shape=(b, B*n, r)
    Out = bmm(Z, U.T) # multiply left factor
    return Out.rearrange(B, n, b*p)
```

**Reference**

Ashcraft C, Buttari A, Mary T. Block Low-Rank Matrices with Shared Bases: Potential and Limitations of the BLR ^2 Format. SIAM Journal on Matrix Analysis and Applications. 2021;42(2):990-1010.

**PDF with additional experiments, Table and Figures**

Meta-review: Overall: this paper describes an interesting variation on low-rank/structured weight matrices with enough experimentation to suggest that the structure might be worth further investigation, and hence sharing with the NeurIPS audience.  The reviewers, particularly xrxa, draw attention to many deficiencies of experimentation and presentation, which it is strongly expected the authors will take into consideration when revising the paper.

On balance, however, I feel that the objections of reviewers V5gm and 2dKz are adequately rebutted, which puts this paper into the "borderline accept" category.  This is largely due to xrxa's observation that the "proposed approach seems to be entirely novel, building on existing low rank and block sparse techniques."  It feels as if it is valuable to have this approach described, even without the custom kernels, and  that would make its impact more widely 

As AC, I agree with xrxa: "the speedups should be rewarded, whether they are due to FLOPs or memory savings."  
However, this discussion does become relevant when rebutting the observation that a very short sequence length used, which artificially amplifies the performance gains by loading much more work onto the linear layers.  The rebuttal claims that "we observed the FLOPs saving is relevant to the runtime reduction", but this is more contentious given xrxa's observations.
Decision: Accept (poster)