Title: Matrix Compression via Randomized Low Rank and Low Precision Factorization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Matrix compression

Randomized low rank factorization

Randomized SVD

Sketching

Quantized embeddings

Random matrices
Abstract: Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can  involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix $\mathbf{A}$ as $\mathbf{A} \approx \mathbf{L}\mathbf{R}$, where $\mathbf{L}$ and $\mathbf{R}$ are the low rank factors. The total number of elements in $\mathbf{L}$ and $\mathbf{R}$ can be significantly less than that in $\mathbf{A}$. Furthermore, the entries of $\mathbf{L}$ and $\mathbf{R}$ are quantized to low precision formats -- compressing $\mathbf{A}$ by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of $\mathbf{A}$ by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of $\mathbf{A}$ onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-$7$b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.

Matrix Compression via Randomized Low Rank and Low Precision Factorization

Abstract

Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix A as A ≈ LR , where L and R are the low rank factors. The total number of elements in L and R can be significantly less than that in A . Furthermore, the entries of L and R are quantized to low precision formats - compressing A by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of A by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of A onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa7 b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.

1 Introduction

Low-rank structures for matrices have proven to be incredibly valuable and ubiquitous across numerous fields of study. Several real-world matrices approximately exhibit low-rank structure due to inherent redundancy or patterns, allowing them to be approximated using low-rank factors. Udell and Townsend [69] provide a potential justification by considering a generative model with latent variables for real-world matrices. Applications where low-rank structures in matrices are exploited include, but are not limited to, imaging (Lingala et al. [34]), fine-tuning large language models (Aghajanyan et al. [3], Hu et al. [25], Karimi Mahabadi et al. [30], Valipour et al. [70], Wang et al. [76]), compressing neural networks (Ben Noach and Goldberg [8], Idelbayev and Carreira-Perpinan [26], Mao et al. [39], Phan et al. [47], Swaminathan et al. [63], Tahaei et al. [64], Wang et al. [77], Winata et al. [78], Yu et al. [84]), obtaining efficient NN architectures (Jaderberg et al. [27], Tai et al. [65]), etc.

Given a matrix A ∈ R n × d , a low-rank approximation is given by A ≈ LR , where L ∈ R n × m , R ∈ R m × d denote the left and right low-rank factors with m ≪ min { n, d } . Since m ( n + d ) ≪ nd ,

(a) Original image

(b) Naïve quant.

(c) DSVD

(d)

LPLR

(ours)

(e)

LSVD

(ours)

八

00

the total number of entries in LR can be significantly smaller than A , enabling matrix compression. The need for compression is driven by the overwhelming storage demands and computational complexity linked to large matrices. In a comparable but distinct line of work, low-precision (LP) representations have also been studied extensively as a means to reduce the memory footprint. Quantization of a continuous valued variable using a small number of bits reduces storage requirement while trading off accuracy. In addition, they also facilitate low latency for real-time inference and low energy consumption (see Gholami et al. [21]). Works of Alimisis et al. [4], Safaryan et al. [55] have studied various matrix compression operators for distributed optimization.

In this work, we introduce LPLR : Low-Precision Low-Rank factorization - a general matrix compression algorithm that simultaneously exploits the low-rank structure of a matrix and quantizes it, to obtain a low-rank factorization, such that the elements of the factors are represented using a small number of bits. Although LPLR can generically refer to a class of algorithms whose main goal is to obtain low-precision representations and exploiting low-rank structures in matrices, in the rest of the paper, we reserve this acronym for our proposed algorithm in Alg. 1. Fig. 1 shows the effectiveness of LPLR in preserving the semantic features in image compression. Other algorithms designed towards achieving the same objective have been named and described appropriately throughout the paper.

1.1 Related Works

Low-rank approximation : The optimal rankk approximation of a matrix A ∈ R n × d , denoted as A k , can be obtained by performing the singular value decomposition (SVD) of A , which gives A = UΣV ⊤ . To obtain A k , we keep only the top k singular values along with their corresponding left and right singular vectors, setting the remaining singular values to zero. In other words, A k = U k Σ k V ⊤ k = ∑ k i =1 σ i u i v ⊤ i , where σ i , u i , and v i represent the i -th singular value, left singular vector, and right singular vector, respectively. The matrix A k minimizes the Frobenius norm ∥ A -̂ A ∥ F under the constraint that the rank of ̂ A is at most k . This result is known as the EckartYoung-Mirsky theorem [18], which states that ∥ A -A k ∥ 2 F = ∑ i>k σ 2 i . However, computing the SVD has a high computational complexity of O( nd 2 ) ( n ≥ d without loss of generality), which can be impractical for large matrices. Therefore, alternative methods such as in Jin et al. [28], Ye and Du [82], Zhang et al. [85] have been proposed which solve variants of the minimization problem. Chi et al. [11] provide a survey of optimization-based approaches for obtaining low-rank approximation.

Along a parallel line of work, randomized low-rank factorization algorithms have demonstrated their effectiveness in handling massive datasets efficiently while maintaining competitive accuracy levels. Several works, including those by Derezinski et al. [13], Drineas and Mahoney [16], Drineas et al. [17], Halko et al. [23], Ma and Solomonik [37], Mahoney [38], Martinsson et al. [43], Tropp et al. [67], Witten and Candès [79] have focused on reducing the complexity of computing SVD to approximately O( nm 2 ) , where m ≪ d . These algorithms aim to approximate the range space of matrix A by utilizing a sketched version AS , where S ∈ R d × m is a random matrix with m ≪ d . The columns of A are then projected onto this approximate range space. By choosing m = k + p , where p ≥ 2 is a small integer, the resulting low-rank approximation A ≈ LR provided by these algorithms can be proven to be within a small multiplicative factor of the optimal rankk approximation, stated as E ∥ LR -A ∥ 2 F ≤ (1 + δ ) ∥ A k -A ∥ 2 F . The sketching matrix S is typically selected from the

widely used class of Johnson-Lindenstrauss (JL) embeddings. A popular choice when A is dense is to sample a Gaussian matrix where each entry S ij ∼ N (0 , 1 m ) .

Randomized quantization : JL embeddings also have practical uses in vector quantization. In this context, when quantizing a vector x ∈ R d , instead of quantizing x directly, the encoder quantizes Sx , and then the decoder obtains an approximation of x through S ⊤ Q ( Sx ) ≈ x . This randomized transformation equalizes the coordinate values of x , allowing for a smaller range of values and higher precision for the quantizer Q under a fixed bit-budget. As a result, it leads to a smaller ℓ 2 error compared to independently quantizing each coordinate in a naive manner. Works such as Chen et al. [10], Lyubarskii and Vershynin [36], Mayekar and Tyagi [44], Safaryan et al. [54], Saha et al. [56, 57, 58], Studer et al. [60], Suresh et al. [61, 62], Vargaftik et al. [71, 72], Young et al. [83] explore different variations of this concept for different applications.

2 Proposed Algorithm

Prior to detailing our algorithm, we first review the characteristics of the uniformly dithered quantizer that we employ within our approach. Uniformly dithered quantization has been utilized in various prior works Alistarh et al. [5], Bernardo et al. [9], Gray and Stockham [22], Mayekar and Tyagi [44], Suresh et al. [61, 62], and is succinctly described in §(2.1) below.

2.1 Uniformly dithered quantizer

Let us consider quantizing a scalar x with | x | ≤ R . Given a bit-budget of B bits, the scalar quantizer with dynamic range R is described by first specifying the M = 2 B quantization points as:

Here, the resolution is given by ∆ = 2R M -1 , and the quantizer operation is defined as:

where k = arg max j { q j ≤ x } , i.e., x ∈ [ q k , q k +1 ) , and r = x -q k ∆ . Such a quantizer satisfies

i.e., it is unbiased and the quantization error variance is dictated by R and B . Here, the E ( · ) is over the randomness from dithering in (1) (ref. App. C). If the input x to the quantizer falls outside this range, i.e., x > R or x < -R , the quantizer is said to be saturated . Finally, to quantize any matrix X , we obtain Q R , B ( X ) by quantizing each entry independently, i.e., [Q R , B ( X )] ij ≜ Q R , B ( X ij ) .

2.2 Proposed algorithm: LPLR

In order to comprehend low precision and low rank representations, we first introduce a simple strategy, which we name as direct-SVD quant . This method involves two main steps: It first computes the optimal rankk decomposition A k = U k Σ k V ⊤ k , and then, it quantizes the low-rank factors independently, namely, L = Q( U k Σ k ) and R = Q ′ ( V ⊤ k ) . Here, Q and Q ′ are uniform scalar quantizers with respective bit-budgets B and B ′ . A detailed analysis of this direct-SVD quantization approach can be found in Appendix H, along with pseudocode provided in Algorithm 2. Despite its simplicity, this is not the optimal strategy due to two reasons. Firstly, it necessitates computing the SVD of A , which is computationally expensive at O( nd 2 ) . Secondly, let us consider improving the approximation to A when the first factor is fixed to Q( U k ) by solving the optimization problem:

This computes the projection of the columns of A onto the range space of Q( U k ) . Suppose the resulting projection coefficients are further quantized to get Q ′ (Q( U k ) † A ) . Since X ∗ is the solution of (3), for a sufficiently large value of B ′ , it is evident that ∥ ∥ Q( U k )Q ′ (Q( U k ) † A ) -A ∥ ∥ 2 F ≤ ∥ ∥ Q( U k Σ k )Q ′ ( V ⊤ k ) -A ∥ ∥ 2 F , and hence, better than direct-SVD quant. However, this approach of projecting onto the range space of Q( U k ) (which we refer to as LPLR-SVD and analyze in App. I), still requires the computation of U k , so we can replace U k by AS , i.e., an approximation of the

Input : Matrix A ∈ R n × d , sketch size m , Quantizers Q , Q ′ with dynamic ranges R Q , R Q ′ and bit-budgets B , B ′ respectively.

Output: Factorization: LR where L ∈ R n × m , R ∈ R m × d

1 Sample a Gaussian sketching matrix S ∈ R d × m with entries S ij ∼ N ( 0 , 1 m ) .

2 Compute an approximate basis of column space of A by forming the sketch: AS .

3 Quantize the approximate basis with Q to get Q( AS ) .

4 Find W ∗ = arg min W ∥ Q( AS ) W -A ∥ 2 F .

5 Quantize W ∗ using quantizer Q ′ to get Q ′ ( W ∗ ) .

6 return Low-rank and low-precision approximation LR where L = Q( AS ) , R = Q ′ ( W ∗ ) .

range space obtained through random linear combinations of the columns of A (also known as a randomized rangefinder [42]). This leads us to our LPLR algorithm, described in Alg. 1, which finds W ∗ = arg min W ∈ R k × d ∥ Q( AS ) W -A ∥ 2 F and forms the low-rank low-precision approximation Q( AS )Q ′ ( W ∗ ) where Q , Q ′ are quantization operators. While the solution of this problem is available in closed form as W ∗ = Q( AS ) † A , one can also use an approximation of W ∗ by solving this least-squares minimization using an iterative method such as conjugate gradient descent.

In addition to the above argument supporting the superiority of LPLR compared to other baselines (ref. to Tabs. 1 and 2), there exists another essential reason why LPLR outperforms them. This reason directly relates to the selection of S as a Gaussian matrix, which is an integral component of LPLR. Random Gaussian matrices are JL embeddings and possess an equalization property that enhances the precision of uniform quantizers. In particular, let us consider an arbitrary vector x ∈ R d and obtain an estimate as ̂ x = S ⊤ Q( Sx ) using a uniform quantizer Q . It can be shown that the vector quantization error remains constant and does not grow with the dimension d , expressed as E ∥ ̂ x -x ∥ 2 2 = O(1) . This represents a substantial improvement compared to the naïve strategy of independently quantizing each coordinate of x , which leads to a quantization error growth rate of O( d ) . We provide a detailed explanation of this phenomenon in App. D. Furthermore, even when the quantization is 1 -bit per coordinate, e.g., Q( Sx ) = Sign( Sx ) , this embedding provides strong near-isometric embedding properties due to the properties of random hyperplane tessellations [50].

While the strong equalization property of Gaussian embeddings is a known result, certain works such as Saha et al. [57, 58], Suresh et al. [61, 62] opt for using randomized Hadamard embeddings instead of Gaussian ones. The reason behind this choice is twofold: (i) Gaussian matrices are dense, requiring O( d 2 ) multiplications when computing Sx , and (ii) the entries of S are floating point numbers that must be stored in full precision, contradicting the objective of quantizing x using fewer bits. However, these concerns are not problematic for LPLR because the effects of S in the first low-rank factor are neutralized by the second low-rank factor, and S does not need to be stored. In fact, we exploit both the equalization property and the subspace approximation property of Gaussian matrices to derive a superior upper bound for the approximation error, as discussed next in §3.

3 Approximation Error Analysis

We are now in a position to state the approximation error guarantee of LPLR Let us denote the i th row of our input matrix A as a ( i ) . For convenience of analysis, we make the following assumption.

Assumption 3.1. Rows of matrix A have bounded norm, i.e., ∥ a ( i ) ∥ ≤ R for some known R > 0 .

The following result gives an informal upper bound on the expected Frobenius norm error of the factorization returned by Alg. 1.

Theorem 3.2. LPLR approximation error (Informal) Suppose our input matrix A ∈ R n × d with ∥ a ( i ) ∥ ≤ R = O(1) has singular values σ 1 , . . . , σ r with r = rank( A ) and target rank as k . Let κ ( A ) = σ 1 /σ r and κ ( A k ) = σ 1 /σ k respectively be the condition numbers of A and the best rankk approximation of A , and let us denote κ = min { κ ( A ) , κ ( A k ) (1 -c 4 σ k +1 /σ k ) -1 } . Furthermore, for a sufficiently small constant ϵ > 0 , suppose the dynamic range of Q is set to be c 1 √ log ( n/ϵ ) /m ,

and that of Q ′ is set to 2 κ √ m/d . Then, the LPLR factorization returned by Alg. 1 satisfies

while utilizing a total budget of log 2 ( cκ ( A k ) κ ϵ nm √ d √ log ( mn 2 ϵ ) ) bits for n ≈ d . Here, c 1 , c 2 , c 3 , and c 4 are constants that depend on R .

We provide a less formal version of our main result here, suitable for interpretation. The formal statement of this result, including specific constant values, can be found in Thm. G.2 of App. G. It does not necessitate the assumption n ≈ d and provides distinct thresholds for B and B ′ . Thm. G.2 asserts that, for a target rankk , as long as m ≥ k + 2 , one can ensure an arbitrarily small approximation error of ϵ by selecting the number of bits to be at least above a certain threshold budget. The threshold depends on the error tolerance ϵ , dimensions n and d , the sketch size m , and the spectrum of A . The value of κ is determined by taking the smaller of two quantities. In the case of matrices with a sharp decline in singular values (e.g., matrices of exact rankk ), where the ratio σ k /σ k +1 approaches zero, κ ≈ κ ( A k ) . For matrices with a smoother spectrum (e.g., all singular values are equal), κ = κ ( A ) , the condition number of the input matrix A .

Remark 1. We consider two distinct scenarios based on Asm. 3.1. The first case assumes that the row norms are bounded by a constant, represented by R = O(1) . This assumption is reasonable when the rows of A correspond to different normalized features of a data point. The second case assumes that the individual entries of A are bounded by a constant, i.e., A ij = O(1) . This implies that R = O( √ d ) , which is a reasonable assumption for scenarios like images, where it is known that each pixel value is bounded. In Tab. 1, we compare the performance of the algorithms when R = O(1) , while in Tab. 2, we assume R = O( √ d ) . Thm. 3.2 assumes that R = O(1) . The expressions in Tab. 2 can be obtained in a similar manner from the formal version in Thm G.2.

3.1 Analysis outline

The derivation of the upper bound on the approximation error of LPLR is presented in App. G. In this section we outline a brief proof sketch that highlights the main challenges of the proof. As mentioned already in §2.2, the analysis of LPLR utilizes the subspace embedding and equalization properties of random Gaussian matrices. A key component in Alg. 1 is the choice of dynamic range for quantizers Q and Q ′ . In our analysis, we assume that when either Q or Q ′ gets saturated in lines 3 and 5 of Alg. 1, a trivial factorization of LR = 0 is returned. We choose the dynamic ranges R Q and R Q ′ to be sufficiently high enough so that this happens with a very low probability. Formally, for quantizer Q , Lemma E.1 states the following:

Here, ∥ AS ∥ max is max-norm of the matrix AS , i.e., the coordinate with maximum magnitude. This concentration result is a consequence of the equalization property of Gaussian matrix S .

On the other hand, the input to the second quantizer is Q( AS ) † A and Lemma E.2 provides a concentration result for the max-norm of this matrix. Although in a general worst-case scenario, the coordinate values of the pseudo-inverse of a matrix with small entries can be large (Alon and Vu [6]), because we compute the pseudo-inverse of the matrix AS , which is rectangular ( m ≪ n ) and with random Gaussian entries, ∥ ( AS ) † ∥ 2 does not shoot up arbitrarily as shown by Rudelson and Vershynin [53]. We then show that ∥ Q( AS ) † ∥ 2 is not too far from ∥ ( AS ) † ∥ 2 , allowing us to derive

The second part of the analysis deals with upper bounding the approximation error when the second low-rank factor is unquantized, i.e., ∥ Q( AS )Q( AS ) † A -A ∥ 2 F conditioned on the event that Q and Q ′ are unsaturated. We reduce this problem to analyzing the solution of the following:

We refer to (4) as the sketched least squares problem with quantized response as it is a variant of the generalized least squares problem, X ∗ = arg min X ∥ ∥ X ⊤ A k -A ∥ ∥ 2 F . This is potentially a problem of independent interest, and we analyze the solution of (4) in detail in App. F. Exploiting the subspace embedding property of S we show that

This leads us to the proof of Lemma G.1 which gives the approximation error of LPLR when Q and Q ′ are unsaturated. Finally, taking into account the low-probability saturation events, for which the error is ∥ A ∥ 2 F , we derive our main result in Thm. G.2. Subsequently, we discuss the approximation made in App. G.2 and arrive at the informal result of Thm. 3.2.

3.2 Comparison with baselines

We are now in a position to compare the performance with baselines in Tabs. 1 and 2.

Naive quantization : The most straightforward baseline for matrix quantization is naïve quantization where each coordinate of the matrix is quantized independently, agnostic to any low-rank structure in the matrix A . In this, we allocate B bits to each coordinate of A and since there are nd entries in the matrix, from (1), the Frobenius norm error is upper bounded by R 2 nd (2 B -1) 2 . Note that this holds true irrespective of whether R = O(1) or O( √ d ) because ∥ a ( i ) ∥ ≤ R implies A ij ≤ R . To ensure that the error is within a certain tolerance ϵ , we then require 1 2 log 2 ( nd ϵ ) bits. In this, and also other expressions for bit-budget requirements of algorithms in Tabs. 1 and 2, we have ignored the multiplicative constant factors inside the log 2 ( · ) for simplicity of exposition. The exact expressions can be found in the corresponding appendices where we derive them.

One of the primary reasons why both direct-SVD and LPLR are expected to perform better than naïve is that the former strategies exploit the low-rank structure of the matrices to reduce the total number of parameters being quantized, i.e., k ( n + d ) for direct-SVD and m ( n + d ) for LPLR, vs. nd for naïve. Given a total bit-budget for the entire matrix A , since we now quantize fewer parameters than before, we can allocate a higher number of bits to each parameter, enabling higher precision. The price we pay for exploiting the low-rank structure of matrices is the additional ∥ A k -A ∥ 2 F dependent term, which is usually very small for matrices that can be well approximated by a low-rank structure. For matrices that are exactly rank k , this term is 0 . As we see in our numerical simulations in §4, several real-world matrices can be well-approximated by a low-rank structure.

Direct-SVD quant. : From Tab. 1, we see that to achieve an ϵ -quantization error, direct-SVD requires 1 2 log 2 ( k √ nd ) bits per entry, which is greater than 1 2 log 2 ( nm √ d ) required by LPLR (ignoring the

logarithmic terms). Evidently, LPLR demands fewer bits than direct-SVD because k, m ≪ min { n, d } for inherently low-rank matrices. For the regime presented in Tab. 2, the bit requirement for direct-SVD remains unchanged. However, LPLR now requires 1 2 log 2 ( nm √ d ) , slightly more than direct-SVD, due to the additional √ n factor. Thus, it makes sense to expect that direct-SVD can perform better in this regime. This is supported by our numerical experiments in Tabs. 4 to 7, where direct-SVD indeed outperforms LPLR in certain scenarios. Nevertheless, it is crucial to emphasize that direct-SVD necessitates computing the SVD, which can be prohibitive for very large matrices due to the current memory limitations of available GPUs, making LPLR the only viable option.

Computational complexity : Unsurprisingly, naïve quant. requires the least computation, i.e., O( nd ) , as it just does a single pass over all the elements of A . The O( nd 2 ) complexity of direct-SVD quant. stems from the requirement of computing SVD (assuming n ≥ d ). LPLR is the best of both worlds for the same bit-budget, LPLR has a smaller approximation error than both direct-SVD and naïve, and a complexity of O( ndm ) , arising from the requirement to compute AS , i.e., a product of two dense matrices of dimensions n × d and d × m , which is better than direct-SVD, since m ≪ d .

4 Numerical Simulations

4.1 Overview

We evaluate the robustness of LPLR on multiple tasks, namely, image compression, binary, and multi-class classification across disparate domains including vision, text and raw images, and neural network weight matrices. We consider a range of input configurations to showcase the performance and non linear effects of joint quantization and low rank approximation on a given dataset, especially at lower bit budgets. LPLR provides competitive results at bit budgets as low as a single bit, providing extreme model compression while maintaining non trivial performance for the task at hand.

Baselines. We employ naive quantization, which quantizes the input matrix by rounding to the nearest scalar in the underlying data type's quantization grid, as our primary benchmark. Naïve quant. and its variants - Dettmers et al. [14], Yao et al. [81], are the most popular method in use across domains, as their memory and computational run time requirements scale extremely well with model and dataset sizes. In addition, we also evaluate the performances of direct-SVD quantization and LPLR-SVD to disambiguate between the entangled effects of quantization and exploiting low rank structure.

Metrics. We evaluate LPLR performance using task specific goodness of fit metrics, as well as relative Frobenius norm error between the original and matrix reconstructed from its low-rank factors, i.e., ∥ LR -A ∥ 2 F . In addition to this, we enforce parity between the number of bits used by all quantization schemes, so that the total space required (in bits) for storing the approximated matrix is identical across LPLR, LPLR-SVD, direct-SVD quant., and naïve quant.

Notation. In all our experiments, we denote the bit budget for the left low rank factor L as B , for the right low rank factor R as B ′ , and the corresponding bit budget for naive quantization as B nq . For simplicity, we maintain equal bit budgets B = B ′ for both quantizers Q and Q ′ . Wherever necessary, we also abbreviate direct-SVD quant. as DSVD, LPLR-SVD as LSVD, and naïve quant. as NQ.

The main algorithm is implemented in Pytorch (Paszke et al. [46]), and utilizes Hugging Face [80] implementations of all datasets and large language models. All experiments were performed on a single GPU NVIDIA TITAN RTX. Further simulations and experimental details can be found in App. J. Our code is available at https://github.com/pilancilab/matrix-compressor .

4.2 Image Compression

Image compression is a prototypical application of low rank matrix compression, as images are known to be significantly rank deficient in many practical scenarios (Zhang et al. [86], Zhou et al. [87]). In this task, we apply LPLR on 1000 × 1000 dimensional Shepp Logan phantom images from Gach et al. [20]. These are a set of synthetic 2D images designed to simulate the typical characteristics and structures found in computed tomography (CT) scans. They consist of geometric shapes, including circles and ellipses, representing different tissues or organs within the scanned object.

The main results are summarized in Tab. 3. To ensure a fair comparison, we adjust the sketch size/target rank so that bit budgets are identical between naïve quant. and LPLR. This allows

us to preserve the original datatype of the image (consequently a large dynamic range), while substantially reducing the pixels used for representing the image to as low as 1 bit per pixel (on average). Specifically, in Figure 1, we can observe the least visual distortion in the case of LPLR, which preserves critical semantic features of the images, such as the small ellipses. It is clear that LPLR outperforms both techniques at lower naive quantization bit budgets. We attribute the better visual and quantitative performance to the higher dynamic range available to LPLR as well as structure preserved in the low rank decomposition.

B

32

28

24

20

16

12

8

Target Rank

15

17

20

25

31

41

62

B

nq

1

1

1

1

1

1

1

LPLR

0.610

0.557

0.540

0.485

0.447

0.402

0.340

DSVD

0.553

0.546

0.537

0.529

0.523

0.518

0.508

LSVD

0.506

0.490

0.454

0.426

0.391

0.360

0.326

NQ

0.532

0.532

0.532

0.532

0.532

0.532

0.532

4.3 Embeddings extracted from pre-trained models

The efficacy of pre-trained embeddings is well established in vision (Li et al. [33], Parisi et al. [45]), text (Qi et al. [51], Rezaeinia et al. [52]) for rapid feature computation as an input to a variety of downstream tasks. Embeddings also play a crucial role in a number of software applications, including but not limited to, open source vector search libraries (Liu [35], Marqo [41]), semantic search engines (Amazon AWS [7]), vector databases (Pinecone [49]). Since most applications rely on proximity in 'embedded space', it is essential that common operations on embeddings be computationally efficient. Specifically, one would like to optimize nearest neighbor (NN) searches which solve the optimization problem arg min i ∥ x i -y ∥ 2 2 (reducible to arg max i Xy where X is the matrix with training vectors { x i } as its rows, and y is the query vector). The time complexity of NN search scales linearly with dimensions of X ∈ R n × d and number of neighbors ( k ). By embedding the data matrix in a dimension m ≪ d , we directly speedup the run-time and reduce storage costs. Moreover, as datasets grow exponentially in size (especially document databases) and transfer learning becomes the dominant modality of training new models, embedding compression becomes a necessity for storing data without a corresponding exponential increase in hardware requirements.

4.3.1 Embedding Classification

In this experiment, we evaluate several embeddings of standard datasets, namely CIFAR-10, CIFAR100, IMDB and Emotion datasets. CIFAR-10 consists of 60,000 color images divided into 10 classes, with each class containing 6,000 images. The dataset is split into 50,000 training images and 10,000 test images, with a resolution of 32x32 pixels. CIFAR-100 increases the number of classes to 100 categories for an identical training and test size as CIFAR-10. The IMDB (mte [2]) dataset consists of 25,000 train and test sentences containing annotated binary sentiment labels for movie reviews, plot summaries and other rating information. The Emotion (mte [1]) dataset is a sentiment analysis dataset, containing 16,000 train and 2000 test sentences, each exemplifying a singular emotion, which represents the sentiment label for that sentence.

For CIFAR-10 and CIFAR-100, we embed the entire dataset using MobileNet v3 (Howard et al. [24]) pretrained on ImageNet (Deng et al. [12]) producing an embedding matrix of dimension 60000 × 1024 , which we compress using LPLR and compare with the baselines in §4.1. To evaluate the goodness of embeddings, we build a 3 -NN, a KNN Classifier using K = 3 nearest neighbors under Euclidean distance). We report the performance of the model using standard classification metrics - classification accuracy and weight averaged F1 score. We utilize a uniform bit budget B = B ′ = 8 bits for the quantizers Q , Q ′ across all cases. Tabs. 4 and 5 present our results under this setup. For each case we benchmark absolute performance using a 3 -NN classifier on the training set.

(

k

(

)

m

)

/

Similarly, we embed text sentences from IMDB and Emotion databases using BeRT (Devlin et al. [15]) into 512 dimensional vectors, and construct a 3 -NN classifier using Euclidean distance to perform binary and multi-class classification on the respective embeddings, and report classification metrics in Tabs. 6 and 7. We see that LPLR outperforms direct-SVD quant. and naïve quant. at lower bit budgets, and has performance parity as we increase B nq to 4 bits. We find that we match (and even exceed) the unquantized benchmark at single bit precision, which we attribute to the dominating low rank factorization, and its regularizing effect on data under extreme rank constraints. It is important to note that performance parity with direct-SVD quant. is also a successful outcome, since LPLR provides runtime improvements over taking an SVD to compress the data.

4.3.2 Compressing Weight Matrices of a Large Language Model

In this section, we present results on a major application of matrix compression - compressing the weight matrices of deep neural networks. We choose LlaMa by Touvron et al. [66], a popular foundation Large Language Model (LLM) as the network of our choice. LLMs are a natural candidate for matrix compression, due to their massive stacked transformer layers, rendering them difficult to deploy on several GPUs, let alone a single GPU. Many methods have emerged to quantize and compress these models in order to make them amenable to single GPU deployment and inference, including naive quantization with outlier exclusion (Dettmers et al. [14]), second order methods (Frantar et al. [19]), low rank parameter reduction (Hu et al. [25]), amongst others.

We apply LPLR to the 2 -dimensional weight tensors, i.e., matrices in LlaMa, leaving any other tensor, which does not lend itself to a low rank decomposition, unquantized. Figs. 2a and 2b (better

Layer

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

LPLR

Naive Quant

LPLR SVD

Layer

0.5

1.0

1.5

2.0

2.5

3.0

3.5

LPLR

Naive Quant

LPLR SVD

(a)

B = B

′

= 8

bits,

B

nq

= 4

bits

(b)

B = B

′

= B

nq

= 4

bits

resolution in App. J) showcase our results on applying LPLR and LPLR-SVD with bit budgets of 8 bits and 4 bits respectively, using relative Frobenius norm error as the metric. While it is clear that LPLR and LPLR-SVD perform significantly better across all layers (on average), there are outliers where naïve quant. is the better choice. We can a observe periodic structure in the error profile of naïve quant., implying that the low rank structure is a function of the index of attention layer in transformer blocks. It is important to note that a low Frobenius norm error is not a direct indicator of performance for other task specific metrics. It is possible to construct a holistic compression strategy using error profiles similar to Figs. 3 and 4 to adopt a per-layer quantization strategy, minimizing both task specific metrics as well as relative Frobenius norm error. We discuss this further in Appendix K.

5 Conclusions

In this work, we have considered the problem of obtaining a low-precision and low-rank factorization of a matrix. Such a factorization of a matrix into a product of tall and wide matrices has several advantages, including compression of the original matrix. We propose a fast randomized algorithm to obtain this factorization which requires O( nmd ) computations - considerably faster than alternative methods. Our algorithm employs a Gaussian sketch to estimate the range space of matrices that are approximately low-rank. By utilizing the properties of subspace approximation and equalization in Gaussian embeddings, we establish an upper bound on the approximation error attained by our algorithm, and show that it can be significantly smaller than its counterparts. Finally, we empirically evaluate our method on several vision and text datasets, where we show significant task performance at highly compressed bit budgets as low as a single bit. This provides a novel pragmatic approach to work with large datasets and models in real world settings, making them more accessible to researchers and deployment on regular consumer hardware.

Acknowledgments and Disclosure of Funding

This work was supported in part by the Air Force Office of Scientific Research (AFOSR) under Award #002484665; in part by National Science Foundation (NSF) CAREER Award under Grant CCF-2236829, Grant DMS-2134248 and Grant ECCS-2037304; in part by the U.S. Army Research Office Early Career Award under Grant W911NF-21-1-0242; in part by the Stanford Precourt Institute; and in part by the ACCESS AI Chip Center for Emerging Smart Systems through InnoHK, Hong Kong, SAR. The authors would also like to thank the anonymous reviewers whose comments and suggestions helped improve the presentation of this work.

References

mteb/emotion - datasets at hugging face. https://huggingface.co/datasets/mteb/ emotion , . (Accessed on 05/16/2023). (Cited on page 8)

mteb/imdb - datasets at hugging face. https://huggingface.co/datasets/mteb/imdb? doi=true , . (Accessed on 05/16/2023). (Cited on page 8)

A. Aghajanyan, L. Zettlemoyer, and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020. URL https://arxiv.org/abs/2012.13255 . (Cited on page 1)

F. Alimisis, P. Davies, and D. Alistarh. Communication-efficient distributed optimization with quantized preconditioners. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 196-206. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr. press/v139/alimisis21a.html . (Cited on page 2)

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems , volume 30, pages 1709-1720. Curran Associates, Inc., 2017. (Cited on page 3)

N. Alon and V. H. Vu. Anti-hadamard matrices, coin weighing, threshold gates, and indecomposable hypergraphs. J. Comb. Theory Ser. A , 79(1):133-160, jul 1997. ISSN 0097-3165. doi: 10.1006/jcta.1997.2780. URL https://doi.org/10.1006/jcta.1997.2780 . (Cited on page 5)

Amazon AWS. Enterprise Search Engine - Amazon Kendra - AWS, 2023. URL https: //aws.amazon.com/kendra/ . (Accessed on 05/16/2023). (Cited on page 8)

M. Ben Noach and Y. Goldberg. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing , pages 884-889, Suzhou, China, Dec. 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020.aacl-main.88 . (Cited on page 1)

N. I. Bernardo, J. Zhu, Y. C. Eldar, and J. Evans. Hardware-limited non-uniform task-based quantizers. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023. doi: 10.1109/ICASSP49357.2023.10094884. (Cited on page 3)

W.-N. Chen, P. Kairouz, and A. Ozgur. Breaking the communication-privacy-accuracy trilemma. In Neural Information Processing Systems (NeurIPS) 2020 , 2020. (Cited on page 3)

Y. Chi, Y. M. Lu, and Y. Chen. Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing , 67(20):5239-5269, 2019. doi: 10.1109/ TSP.2019.2937282. (Cited on page 2)

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. IEEE, 2009. (Cited on page 8)

M. Derezinski, F. T. Liang, Z. Liao, and M. W. Mahoney. Precise expressions for random projections: Low-rank approximation and randomized Newton. In Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/ d40d35b3063c11244fbf38e9b55074be-Paper.pdf . (Cited on page 2)

T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer. LLM.int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.07339 , 2022. (Cited on pages 7 and 9)

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. (Cited on page 9)

P. Drineas and M. W. Mahoney. RandNLA: Randomized numerical linear algebra. Commun. ACM , 59(6):80-90, may 2016. ISSN 0001-0782. doi: 10.1145/2842602. URL https: //doi.org/10.1145/2842602 . (Cited on page 2)

P. Drineas, R. Kannan, and M. W. Mahoney. Fast monte carlo algorithms for matrices II: Computing a low-rank approximation to a matrix. SIAM Journal on Computing , 36(1): 158-183, 2006. doi: 10.1137/S0097539704442696. URL https://doi.org/10.1137/ S0097539704442696 . (Cited on page 2)

C. Eckart and G. Young. The approximation of one matrix by another of lower rank. Psychometrika , 1:211-218, 1936. doi: https://doi.org/10.1007/BF02288367. URL https: //rdcu.be/dcbK1 . (Cited on page 2)

E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323 , 2022. (Cited on page 9)

H. M. Gach, C. Tanase, and F. Boada. 2D & 3D shepp-logan phantom standards for MRI. In 2008 19th International Conference on Systems Engineering , pages 521-526. IEEE, 2008. (Cited on page 7)

A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference, 2021. URL https://arxiv.org/abs/2103. 13630 . (Cited on page 2)

R. Gray and T. Stockham. Dithered quantizers. IEEE Transactions on Information Theory , 39 (3):805-812, 1993. doi: 10.1109/18.256489. (Cited on page 3)

N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review , 53(2):217-288, 2011. doi: 10.1137/090771806. URL https://doi.org/10.1137/090771806 . (Cited on pages 2 and 29)

A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam. Searching for mobilenetv3, 2019. URL https: //arxiv.org/abs/1905.02244 . (Cited on page 8)

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 . (Cited on pages 1 and 9)

Y. Idelbayev and M. A. Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8046-8056, 2020. doi: 10.1109/CVPR42600.2020.00807. (Cited on page 1)

M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions, 2014. URL https://arxiv.org/abs/1405.3866 . (Cited on page 1)

D. Jin, X. Bing, and Y. Zhang. Unique sparse decomposition of low rank matrices. In Advances in Neural Information Processing Systems , volume 34, pages 523-535. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 051928341be67dcba03f0e04104d9047-Paper.pdf . (Cited on page 2)

G. K. Karagiannidis and A. S. Lioumpas. An improved approximation for the gaussian Qfunction. IEEE Communications Letters , 11(8):644-646, 2007. doi: 10.1109/LCOMM.2007. 070470. (Cited on page 24)

R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1022-1035. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/ paper/2021/file/081be9fdff07f3bc808f935906ef70c0-Paper.pdf . (Cited on page 1)

X. Li and P. Li. Generalization error analysis of quantized compressive learning. Advances in Neural Information Processing Systems , 32, 2019. (Cited on page 45)

X. Li and P. Li. Random projections with asymmetric quantization. Advances in Neural Information Processing Systems , 32, 2019. (Cited on page 45)

X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16 , pages 121-137. Springer, 2020. (Cited on page 8)

S. G. Lingala, Y. Hu, E. DiBella, and M. Jacob. Accelerated dynamic MRI exploiting sparsity and low-rank structure: k-t SLR. IEEE Transactions on Medical Imaging , 30(5):1042-1054, 2011. doi: 10.1109/TMI.2010.2100850. (Cited on page 1)

J. Liu. LlamaIndex, 11 2022. URL https://github.com/jerryjliu/llama_index . (Cited on page 8)

Y. Lyubarskii and R. Vershynin. Uncertainty principles and vector quantization. IEEE Transactions on Information Theory , 56(7):3491-3501, 2010. (Cited on page 3)

L. Ma and E. Solomonik. Fast and accurate randomized algorithms for low-rank tensor decompositions. In Advances in Neural Information Processing Systems , volume 34, pages 24299-24312. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/ paper/2021/file/cbef46321026d8404bc3216d4774c8a9-Paper.pdf . (Cited on page 2)

M. W. Mahoney. Lecture notes on randomized linear algebra, 2016. URL https://arxiv. org/abs/1608.04481 . (Cited on page 2)

Y. Mao, Y. Wang, C. Wu, C. Zhang, Y. Wang, Q. Zhang, Y. Yang, Y. Tong, and J. Bai. LadaBERT: Lightweight adaptation of BERT through hybrid model compression. In Proceedings of the 28th International Conference on Computational Linguistics , pages 3225-3234, Barcelona, Spain (Online), Dec. 2020. International Committee on Computational Linguistics. doi: 10.18653/ v1/2020.coling-main.287. URL https://aclanthology.org/2020.coling-main.287 . (Cited on page 1)

K. Mardia, J. Kent, and J. Bibby. Multivariate Analysis . Academic Press, 1979. ISBN 978-0-12-471250-8. (Cited on page 21)

Marqo. marqo-ai/marqo: Vector search for humans, 2023. URL https://github.com/ marqo-ai/marqo . (Accessed on 05/16/2023). (Cited on page 8)

P.-G. Martinsson and J. A. Tropp. Randomized numerical linear algebra: Foundations and algorithms. Acta Numerica , 29:403-572, 2020. (Cited on page 4)

P.-G. Martinsson, V. Rokhlin, and M. Tygert. A randomized algorithm for the decomposition of matrices. Applied and Computational Harmonic Analysis , 30(1):47-68, 2011. ISSN 1063-5203. doi: https://doi.org/10.1016/j.acha.2010.02.003. URL https://www.sciencedirect.com/ science/article/pii/S1063520310000242 . (Cited on page 2)

P. Mayekar and H. Tyagi. RATQ: A universal fixed-length quantizer for stochastic optimization. IEEE Transactions on Information Theory , 67(5):3130-3154, 2021. doi: 10.1109/TIT.2021. 3058663. (Cited on page 3)

S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference on Machine Learning , pages 17359-17371. PMLR, 2022. (Cited on page 8)

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf . (Cited on page 7)

A.-H. Phan, K. Sobolev, K. Sozykin, D. Ermilov, J. Gusak, P. Tichavský, V. Glukhov, I. Oseledets, and A. Cichocki. Stable low-rank tensor decomposition for compression of convolutional neural network. In A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, editors, Computer Vision - ECCV 2020 , pages 522-539, Cham, 2020. Springer International Publishing. (Cited on page 1)

M. Pilanci. Lecture notes: Large scale matrix computation, optimization, and learning (ee 270), 2020. URL https://web.stanford.edu/class/ee270/Lecture18.pdf . (Cited on page 29)

Pinecone. Vector database for vector search | pinecone. https://www.pinecone.io/ , 2023. (Accessed on 05/16/2023). (Cited on page 8)

Y. Plan and R. Vershynin. Dimension reduction by random hyperplane tessellations. Discrete & Computational Geometry , 51(2):438-461, 2014. (Cited on page 4)

Y. Qi, D. S. Sachan, M. Felix, S. J. Padmanabhan, and G. Neubig. When and why are pre-trained word embeddings useful for neural machine translation? arXiv preprint arXiv:1804.06323 , 2018. (Cited on page 8)

S. M. Rezaeinia, R. Rahmani, A. Ghodsi, and H. Veisi. Sentiment analysis based on improved pre-trained word embeddings. Expert Systems with Applications , 117:139-147, 2019. (Cited on page 8)

M. Rudelson and R. Vershynin. Smallest singular value of a random rectangular matrix. Communications on Pure and Applied Mathematics , 62, 2008. (Cited on page 5)

M. Safaryan, E. Shulgin, and P. Richtárik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. Information and Inference: A Journal of the IMA , Apr 2021. (Cited on page 3)

M. Safaryan, R. Islamov, X. Qian, and P. Richtárik. FedNL: Making newton-type methods applicable to federated learning. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 18959-19010. PMLR, 2022. URL https://proceedings.mlr. press/v162/safaryan22a.html . (Cited on page 2)

R. Saha, M. Pilanci, and A. J. Goldsmith. Minimax optimal quantization of linear models: Information-theoretic limits and efficient algorithms, 2022. URL https://arxiv.org/abs/ 2202.11277 . (Cited on page 3)

R. Saha, M. Pilanci, and A. J. Goldsmith. Efficient randomized subspace embeddings for distributed optimization under a communication budget. IEEE Journal on Selected Areas in Information Theory , 3(2):183-196, 2022. doi: 10.1109/JSAIT.2022.3198412. (Cited on pages 3 and 4)

R. Saha, M. Pilanci, and A. J. Goldsmith. Low precision representations for high dimensional models. In ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023. doi: 10.1109/ICASSP49357.2023.10095529. (Cited on pages 3 and 4)

V. Siskind. Second moments of inverse wishart-matrix elements. Biometrika , 59(3):690-691, 12 1972. ISSN 0006-3444. doi: 10.1093/biomet/59.3.690. URL https://doi.org/10.1093/ biomet/59.3.690 . (Cited on page 21)

C. Studer, W. Yin, and R. G. Baraniuk. Signal representations with minimum ℓ ∞ -norm. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1270-1277, 2012. (Cited on page 3)

A. T. Suresh, F. X. Yu, S. Kumar, and H. B. McMahan. Distributed mean estimation with limited communication. In Proceedings of the 34th International Conference on Machine Learning Volume 70 , ICML'17, page 3329-3337. JMLR.org, 2017. (Cited on pages 3 and 4)

A. T. Suresh, Z. Sun, J. Ro, and F. Yu. Correlated quantization for distributed mean estimation and optimization. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 20856-20876. PMLR, 1723 Jul 2022. URL https://proceedings.mlr.press/v162/suresh22a.html . (Cited on pages 3 and 4)

S. Swaminathan, D. Garg, R. Kannan, and F. Andres. Sparse low rank factorization for deep neural network compression. Neurocomputing , 398:185-196, 2020. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2020.02.035. URL https://www.sciencedirect.com/ science/article/pii/S0925231220302253 . (Cited on page 1)

M. Tahaei, E. Charlaix, V. Nia, A. Ghodsi, and M. Rezagholizadeh. KroneckerBERT: Significant compression of pre-trained language models through kronecker decomposition and knowledge distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2116-2127, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.naacl-main.154. URL https://aclanthology.org/2022.naacl-main.154 . (Cited on page 1)

C. Tai, T. Xiao, Y. Zhang, X. Wang, and W. E. Convolutional neural networks with low-rank regularization, 2016. URL https://arxiv.org/abs/1511.06067 . (Cited on page 1)

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023. (Cited on page 9)

J. A. Tropp, A. Yurtsever, M. Udell, and V. Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications , 38(4):1454-1485, 2017. doi: 10.1137/17M1111590. URL https://doi.org/10.1137/17M1111590 . (Cited on page 2)

J. A. Tropp, A. Yurtsever, M. Udell, and V. Cevher. Streaming low-rank matrix approximation with an application to scientific simulation. SIAM Journal on Scientific Computing , 41(4):A2430A2463, 2019. doi: 10.1137/18M1201068. URL https://doi.org/10.1137/18M1201068 . (Cited on page 45)

M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science , 1(1):144-160, 2019. doi: 10.1137/18M1183480. URL https://doi.org/10.1137/18M1183480 . (Cited on page 1)

M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. DyLoRa: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation, 2023. URL https://arxiv.org/abs/2210.07558 . (Cited on page 1)

S. Vargaftik, R. Ben-Basat, A. Portnoy, G. Mendelson, Y. Ben-Itzhak, and M. Mitzenmacher. DRIVE: One-bit distributed mean estimation. In Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=KXRTmcv3dQ8 . (Cited on page 3)

S. Vargaftik, R. B. Basat, A. Portnoy, G. Mendelson, Y. B. Itzhak, and M. Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In International Conference on Machine Learning , pages 21984-22014. PMLR, 2022. (Cited on page 3)

R. Vershynin. Introduction to the non-asymptotic analysis of random matrices, 2011. URL https://arxiv.org/abs/1011.3027 . (Cited on pages 21 and 22)

R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596. (Cited on page 22)

M. J. Wainwright. Basic tail and concentration bounds , page 21-57. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/ 9781108627771.002. (Cited on page 21)

R. Wang, D. Tang, N. Duan, Z. Wei, X. Huang, J. Ji, G. Cao, D. Jiang, and M. Zhou. KAdapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021 , volume ACL/IJCNLP 2021 of Findings of ACL , pages 1405-1418. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.findings-acl.121. URL https: //doi.org/10.18653/v1/2021.findings-acl.121 . (Cited on page 1)

Z. Wang, J. Wohlwend, and T. Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.496. URL https://doi.org/10.18653%2Fv1%2F2020.emnlp-main.496 . (Cited on page 1)

G. I. Winata, A. Madotto, J. Shin, E. J. Barezi, and P. Fung. On the effectiveness of low-rank matrix factorization for LSTM model compression, 2019. URL https://arxiv.org/abs/ 1908.09982 . (Cited on page 1)

R. Witten and E. Candès. Randomized algorithms for low-rank matrix factorizations: Sharp performance bounds. Algorithmica , 72(1):264-281, may 2015. ISSN 0178-4617. doi: 10. 1007/s00453-014-9891-7. URL https://doi.org/10.1007/s00453-014-9891-7 . (Cited on page 2)

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 , 2019. (Cited on page 7)

Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems , 35:27168-27183, 2022. (Cited on page 7)

T. Ye and S. S. Du. Global convergence of gradient descent for asymmetric low-rank matrix factorization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 1429-1439. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/ 2021/file/0af854284f4ab0cfea8fcfd889cbb41a-Paper.pdf . (Cited on page 2)

S. Young, Z. Wang, D. Taubman, and B. Girod. Transform quantization for CNN compression. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2021. doi: 10.1109/tpami.2021.3084839. URL https://doi.org/10.11092%Ftpami.2021.3084839 . (Cited on page 3)

X. Yu, T. Liu, X. Wang, and D. Tao. On compressing deep models by low rank and sparse decomposition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 67-76, 2017. doi: 10.1109/CVPR.2017.15. (Cited on page 1)

H. Zhang, Y. Bi, and J. Lavaei. General low-rank matrix optimization: Geometric analysis and sharper bounds. In Advances in Neural Information Processing Systems , volume 34, pages 27369-27380. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper_files/paper/2021/file/e60e81c4cbe5171cd654662d9887aec2-Paper.pdf . (Cited on page 2)

Y. Zhang, Z. Jiang, and L. S. Davis. Learning structured low-rank representations for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 676-683, 2013. (Cited on page 7)

X. Zhou, C. Yang, H. Zhao, and W. Yu. Low-rank modeling and its applications in image analysis. ACM Computing Surveys (CSUR) , 47(2):1-33, 2014. (Cited on page 7)

Contents

K Limitations and further discussions

44

A Notations

We first gather some common notations in linear algebra and probability theory that have been used throughout the paper. Boldface upper and lowercase letters, A and a denote matrices and vectors respectively. I d denotes the d × d identity matrix. The subscript may be dropped if the dimension is clear from the context. For any matrix A , its i th row and j th column are denoted by a ( i ) and a j respectively. The singular values of A are denoted by σ max ( A ) = σ 1 ≥ σ 2 ≥ . . . ≥ σ r = σ min ( A ) , where r = rank( A ) . Similarly, the eigenvalues are denoted as λ 1 ( A ) , . . . , λ r ( A ) . The maxnorm of A is defined as ∥ A ∥ max = max i,j | A ij | , the spectral norm of A is defined as ∥ A ∥ 2 =

( ∑ k ∈ [ r ] σ 2 k ) 1 / 2 . For any vector x , ∥ x ∥ = ( x 2 i ) 1 / 2 denotes the ℓ 2 -norm, and ∥ x ∥ ∞ = max i | x i | denotes the ℓ ∞ -norm. We use the notations ≽ and ≼ for the positive semi-definite (PSD) cone ordering (or the Loewner ordering ) of symmetric matrices, i.e., for any symmetric matrices X and Y , X ≽ Y ⇐⇒ X -Y is PSD. X † denotes the Moore-Penrose pseudo-inverse of a matrix X . log( · ) denotes the natural logarithm, i.e., base e . log 2 ( · ) , i.e., with base 2 is specified explicitly. A normal distribution with mean µ and variance σ 2 is denoted as N (0 , σ 2 ) , while a multivariate normal distribution in R d is denoted as N ( µ , Σ ) . E [ · ] denotes expectation of a random variable. The probability measure over which the expectation is taken is described in text. We also use big-'oh' notation O( · ) that hides constants for asymptotic expressions, while ˜ O( · ) also hides terms that depends logarithmically on dimension.

We now list some notations that are either less commonly known or used specifically in this paper, along with some remarks such as their occurrence in the paper. Several of these notations have also been introduced in-context, but they are additionally collected here for easy reference.

B Preliminaries

B.1 Linear algebra inequalities

We state (with proofs) some standard inequalities from linear algebra that will be useful in proving our main results.

Lemma B.1. (Frobenius norm of matrix products) For any matrices A and B , we have

Proof. Note that:

Here, (i) follows from the definition of spectral norm of a matrix, and this completes the proof.

Lemma B.2. (Loewner ordering for matrix products) For any matrix A and B , we have

̸

Proof. For any vector x = 0 , we have,

The other direction holds similarly. This completes the proof.

Lemma B.3. (Max-norm spectral-norm inequality) For any matrix A ∈ R n × d , ∥ A ∥ max ≤ ∥ A ∥ 2 .

Proof. Let e i ∈ R n and ̂ e j ∈ R d denote the i th and j th canonical basis vectors in R n and R d respectively. Then, using Cauchy-Schwarz inequality, we have,

completing the proof.

Lemma B.4. (Submultiplicativity of spectral norm) For any two matrices A and B , we have ∥ .

Moreover, an analogous 'reverse-submultiplicativity" result for the minimum singular value of a matrix product also holds true, i.e.,

Proof. Using Lemma B.2, we have,

where (i) follows from Lemma B.2, completing the proof. The lower bound on σ min ( AB ) also follows a similar argument.

Lemma B.5. (Lower bound on minimum singular value of matrix sums) For any matrices A and B , we have

Proof. We have the following chain of inequalities,

where (i) is the reverse triangle inequality. This completes the proof.

Lemma B.6. (Rotation invariance of singular values) For a given matrix A , σ i ( A ) = σ i ( UA ) for all i = 1 , . . . , rank ( A ) for any unitary matrix U .

Proof. i = 1 , . . . , rank( A ) U ⊤ U = I

completing the proof.

Lemma B.7. (Rotation invariance of Frobenius norm) For a given matrix A , ∥ UA ∥ F = ∥ A ∥ F for any unitary matrix U .

Proof. We have,

B.2 Probability and random matrix theory

We restate some results from probability and random matrix theory which will be useful in deriving the main result of our paper.

B.2.1 Tail bound for Gaussian distribution

Gaussian distributions have strong concentration properties which we exploit in deriving the results of this paper. The following tail bound on a Gaussian random variable can be found in standard texts such as Wainwright [75, § 2 . 1 . 2 ], and is restated here.

Lemma B.8. (Chernoff bound for centered Gaussian) For X ∼ N ( 0 , σ 2 ) , we have,

The proof of this follows from a direct application of Chernoff bound.

B.2.2 Inverse Wishart distribution

Consider a matrix S ∈ R d × m , each row of which is drawn independently from the distribution N ( 0 , Ψ ) , where Ψ ∈ R m × m . Then, the probability distribution of the m × m random matrix S ⊤ S is called the Wishart distribution with d degrees of freedom, denoted as W ( Ψ , d ) . Moreover, the distribution of the matrix ( S ⊤ S ⊤ ) -1 is called the inverse Wishart distribution and is denoted by W -1 ( Ψ -1 , d ) . These distributions have been studied extensively and further details can be found in Mardia et al. [40] or Siskind [59].

Lemma B.9. (Expected trace of inverse Wishart matrix) Suppose S ∈ R d × m matrix, each entry of which is drawn independently from N ( 0 , 1 m ) . Then, the matrix S ⊤ S ∈ R m × m follows the Wishart distribution W ( 1 m I m , d ) that satisfies,

Proof. From Mardia et al. [40, eq. 3 . 8 . 3 ], if X ∼ W -1 ( Ψ -1 , d ) , then,

Here, Ψ = 1 m I m , implying Tr [ Ψ -1 ] = m 2 , which completes the proof.

B.2.3 Random Gaussian matrices

The spectral norm of random matrices with Gaussian entries have interesting concentration properties. In this section, we present a lemma from Vershynin [73] that formally states this result.

Lemma B.10. Let the entries of matrix S ∈ R d × m be distributed according to S ij ∼ N ( 0 , 1 )

Then for every t ≥ 0 , with probability at least 1 -2 e -mt 2

m . 2 , we have,

The above lemma is a straightforward modification of the result in Vershynin [73, Corr. 5.35], which states the concentration result when the entries are distributed according to N (0 , 1) . Note that given S with entries S ij ∼ N ( 0 , 1 m ) as above, the matrix ̂ S = √ m S will have entries ̂ S ij ∼ N (0 , 1) . Furthermore, σ i ( ̂ S ) = √ mσ i ( S ) and the conclusion is immediate.

B.2.4 Subgaussian random variables

Subgaussian random variables refer to a class of distributions that are dominated by the distribution of a centered Gaussian random variable. There are several equivalent ways to characterize subgaussian random variables which can be found in several textbooks (see for example, Vershynin [74, Prop. 2.5.2]). We will focus on the following definition. More formally, the distribution of a random variable X is subgaussian if the moment generating function of X 2 is bounded at some point, i.e.,

Definition B.11. (Subgaussian norm) The subgaussian norm of a subgaussian random variable X , denoted by ∥ X ∥ ψ 2 is defined to be the smallest K in (14). In other words,

It can be shown that any bounded random variable X is subgaussian, and satisfies,

We next present a result that upper bounds the spectral norm of a matrix with subgaussian entries.

Lemma B.12. (Vershynin [74, Thm 4.4.5]) Let X be a d × m random matrix whose entries X ij are independent, zero-mean, subgaussian random variables. Then, for any t > 0 , we have,

with probability exceeding 1 -2 e -t 2 . Here, K = max i,j ∥ A ij ∥ ψ 2 and C is an absolute constant.

C Quantization error of uniformly dithered scalar quantizer

For a scalar x ∈ [ -R , +R] , let us denote the quantization error of uniformly dithered scalar quantizer with a bit-budget of B bits as ϵ = Q R , B ( x ) -x . Clearly, the quantization error is bounded as | ϵ | ≤ ∆ . The following result further characterizes its mean and variance of this error.

Lemma C.1. The uniformly dithered scalar quantizer as described in (1) satisfies,

where the E ( · ) is over the randomness due to dithering in the quantizer operation.

Proof. Suppose x ∈ [ q k , q k +1 ) and q k +1 = q k +∆ , where ∆ = 2R B . Then,

To evaluate the variance,

This completes the proof.

D Gaussian embeddings: Application of equalization to vector quantizers

In this section, we show a result on how Gaussian embeddings help in reducing the ℓ 2 -quantization error of a uniformly dithered vector quantizer. We consider a clipped version of the uniform scalar quantizer with bit-budget B described in §2.1 and App. C. In order to quantize a vector x ∈ R d with ∥ x ∥ 2 ≤ R using a uniform scalar quantizer (as described above), the quantization operation is applied to each coordinate of the vector independently, i.e.,

Here, the subscript B is dropped because the bit-budget is evident from the context. Furthermore, since ∥ x ∥ 2 ≤ R implies x i ∈ [ -R , +R] for every i ∈ [ d ] , the quantizer Q R does not saturate. From Lemma C.1, the expected quantization error is given by,

Quantizing Gaussian embeddings : Suppose instead of quantizing x ∈ R d directly, we quantize u = Sx ∈ R m , where S ∈ R m × d with S ij ∼ N ( 0 , 1 m ) . Note that for every j ∈ [ m ] , we have u j ∼ N ( 0 , 1 m ∥ x ∥ 2 2 ) . Since u j can be anything in ( -∞ , + ∞ ) , there is a finite probability that the uniform scalar quantizer might get saturated. For this reason, for any scalar u ∈ ( -∞ , + ∞ ) , we define the clipped uniformly dithered quantizer with clipping parameter t as follows:

The dynamic range of the quantizer is parameterized by t , which is to be chosen appropriately. Note that it is just for this section for the purposes of illustration, that we choose the clipped variant of the quantizer. In LPLR, we choose the dynamic range to be high enough so in practice it remains unsaturated with a very high probability. The following proposition upper bounds the quantization error of quantizing Gaussian embeddings.

Proposition D.1. For a given vector x ∈ R d with ∥ x ∥ ≤ R , the clipped uniformly dithered quantizer described in (19) with t = R √ m satisfies

where the expectation is over the randomness in the construction of S and the quantization dither.

Proof. Since { u i } i ∈ [ m ] are independently and identically distributed, let us denote the distribution as f ( u ) = 1 σ √ 2 π e -u 2 2 σ 2 , where σ = ∥ x ∥ 2 2 m . The expected quantization error for the clipped quantizer is then given by,

Here, the expectation is over the stochasticity of the quantization dither, as well as the random matrix S . The factor of 2 in the second term appears due to symmetry of clipping and that of Gaussian distribution. Using (18), the first term on the R.H.S. can be upper bounded as,

To analyze the second term, note that,

where erf( z ) denotes the error function defined as,

Simple calculation would show,

Since ∂ Ψ( t,σ ) ∂σ ≥ 0 , Ψ( t, σ ) is a non-decreasing function of σ . Since σ 2 = ∥ x ∥ 2 2 m ≤ R 2 m , we have the upper bound, √

where erfc( z ) = 1 -erf( z ) is the complementary error function.

Quantization error of a scalar Gaussian sketch : Since x ∈ R d with ∥ x ∥ 2 ≤ R , let s ∼ N ( 0 , 1 m I d ) . We first consider the quantization of x ⊤ s , and upper bound the error E [ ( Q( x ⊤ s ) -x ⊤ s ) 2 2 ] . Clearly, x ⊤ s ∼ N (0 , R 2 m ) . Consequently, using (21) and (25), for any t ≥ 0 , we have,

Here, (i) follows from the upper bound erfc( z ) ≤ e -z 2 √ πz from Karagiannidis and Lioumpas [29].

Quantization error of a vector Gaussian sketch : We now consider for any t ≥ 0 , the expected quantization error for x ∈ R d with ∥ x ∥ 2 ≤ R and S ∈ R m × d with S ij ∼ N ( 0 , 1 m ) . Each row of S now independently plays the role of s in (26). Using (26), the vector quantization error is simply m times the scalar quantization error, and is now given by, √

Choice of dynamic range t : Setting t = R √ m in (27) yields,

This completes the proof.

Note that since E [ S ⊤ S ] is an identity matrix, an estimate ̂ x of x can be recovered from Q( Sx ) as S ⊤ Q( Sx ) . We can use the O(1) bound on ∥ Q( Sx ) -Sx ∥ 2 to get a bound on ∥ S ⊤ Q( Sx ) -x ∥ 2 as follows:

From properties of random Gaussian matrices, we know that σ 2 max ( S ) ≤ d m with high probability. Hence, the error ∥ S ⊤ Q( Sx ) -x ∥ 2 only depends on the aspect ratio d/m , and not the dimension d directly. Although the reconstruction error ∥ S ⊤ Q( Sx ) -x ∥ 2 scales as d/m , it does not necessarily increase with d if we choose the sketch size m to be proportional to d , i.e., m = O ( d ) .

Despite this, Gaussian embeddings are not used practically for vector quantization because S is dense matrix and computing Sx entails a complexity of O( d 2 ) . The entries of S themselves are floating point numbers that have to be stored in full precision, hence this defeats the whole purpose of quantizing x using fewer bits. However, this is not an issue for matrix compression because in LPLR, we do not explicitly compute the S ⊤ Q( SA ) anywhere. In other words, the effects of S in the first low-rank factor is nullified by the second low-rank factor. Unlike vector quantization, the corresponding sketch size m for LPLR only needs to be the same order as the inherent rank k , which can be much smaller than min { n, d } , i.e., the dimensions of the matrix being compressed.

E Dynamic range of quantizers

We now derive high probability upper bounds on the maximum magnitude of the input to uniform quantizers Q and Q ′ . Choosing these upper bounds to be the dynamic range of the uniform scalar quantizers will ensure that the quantizer remains unsaturated with a high probability, which is our desired regime of operation.

E.1 Quantization for the first low-rank factor

We first look at the choice of dynamic range for the quantizer Q , which is used to obtain the first low-rank factor. The input to the quantizer is AS ∈ R n × m where A ∈ R n × d and S ∈ R d × m , and the entries of S are i.i.d. as S ij ∼ N ( 0 , 1 m ) . Lemma E.1 below gives a high probability upper bound on the max norm of AS . This probability is computed over the randomness in the construction of S .

LemmaE.1. (Max norm of AS ) Given matrix A ∈ R n × d with bounded row norms, i.e., ∥ ∥ a ( i ) ∥ ∥ ≤ R , and S ∈ R d × m with entries distributed as S ij i.i.d. ∼ N ( 0 , 1 m ) , with probability exceeding 1 -ϵ 8 n R 2 , the max norm of AS satisfies,

Proof. Since ( AS ) ij = s ⊤ j a ( i ) , where a ( i ) ∈ R d is the i th row of A and s j ∈ R d is the j th column of S , we have ( AS ) ij ∼ N ( 0 , ∥ a i ∥ 2 2 m ) . Using Lemma B.8 and an application of union bound gives,

A subsequent application of union bound over all the entries of AS yields,

E.2 Quantization for the second low-rank factor

We now obtain a high-probability upper bound on the max norm of Q( AS ) † A , which is the input to the second quantizer Q ′ . Let us define Q to be the event that the quantizer Q does not saturate. From Lemma E.1, Q occurs with a sufficiently high probability. An appropriate choice of dynamic range for Q ′ will ensure that conditioned on the event that Q occurs, the quantizer Q ′ also does not saturate with a high probability. The following lemma states this formally.

Lemma E.2. (Max norm of Q( AS ) † A ) Let our input matrix A ∈ R n × d have non-zero singular values σ 1 , . . . , σ k , σ k +1 , . . . , σ r , where r = rank( A ) , and bounded row norms ∥ ∥ a ( i ) ∥ ∥ ≤ R . Let κ ( A ) = σ 1 /σ r and κ ( A k ) = σ 1 /σ k respectively be the condition numbers of A and the best rankk approximation of A , and for some small ϵ > 0 , let us denote

where γ = d/m is the aspect ratio of the sketching matrix S ∈ R d × m with S ij i.i.d. ∼ N ( 0 , 1 m ) √ ( 16R 2 n 2 m )

.

Furthermore, suppose the dynamic range of the quantizer Q is set to R 2 log ϵ m as dictated by Lemma E.1, and suppose for some absolute constant C , the bit-budget B satisfies,

Then, we have,

Proof. We have the following chain of inequalities:

where (i) and (ii) follow from Lemmas B.3 and B.4 respectively. We now need to upper bound ∥ ∥ ∥ Q( AS ) † ∥ ∥ ∥ 2 , which is done as follows:

Here, E = Q( AS ) -AS ∈ R n × m is the quantization error matrix from Q , (i) follows because the singular values of Q( AS ) † are inverses of the singular values of Q( AS ) (assuming Q( AS ) is invertible), and (ii) follows from Lemma B.5.

Lower bounding σ min ( AS ) : It now suffices to derive the a lower bound on σ min ( AS ) , which we do next. We derive two different lower bounds, either of which could be tighter depending on the singular value profile of A . The final lower bound will be the maximum of both.

For the first lower bound, let A = UΣV ⊤ be the full singular value decomposition of A , where U ∈ R n × n , Σ ∈ R n × d , and V ∈ R d × d . Then, denoting ˜ S = V ⊤ S , we have,

where A k is the best rankk approximation of A . Here, once again, (i) follows from Lemma B.5 and (ii) follows from Lemma B.4. In order to further lower bound σ min ( A k S ) , let us denote the SVD of A k as A k = UΣ k V ⊤ k . Since the best rankk approximation is obtained by retaining the topk singular values of A and zeroing out the rest, we have U ∈ R n × n , Σ k ∈ R n × k and V k ∈ R d × k . Let us further denote ˜ S = V ⊤ k S . Then we have

where (i) follows from Lemma B.6, and (ii) follows from Lemma B.4. Note that the ( i, j ) th entry of ˜ S is ˜ S ij = v ⊤ i s j , where v i is the i th column of V k . Clearly, ˜ S ij is a Gaussian random variable with mean, E [ ˜ S ij ] = ∑ ℓ V ℓi E [ S ℓj ] = 0 , and variance, Var ( ˜ S ij ) = ∑ ℓ V 2 ℓi Var ( S ℓj ) = 1 m ∑ ℓ V 2 ℓi = 1 m , where the last equality follows from the fact that the columns of V k are orthonormal. In other words, the entries of ˜ S ∈ R d × m are distributed according to ˜ S ij ∼ N ( 0 , 1 m ) . Using (39), (38) can be further lower bounded as

where (i) once again follows from the rotation invariance of spectrum, i.e., Lemma B.6.

On the other hand, let r = rank( A ) and σ r by the smallest non-zero singular value of A . Then, A = UΣV ⊤ = UΣ r V ⊤ r and using the same arguments as in (39), we also have

All we are left with now is to utilize the concentration bounds on the singular values of S . As a consequence of Lemma B.10 and recalling γ = d/m , with probability exceeding 1 -2 e -mt 2 2 ,

Substituting this in (42), with probability exceeding 1 -2 e -mt 2 2 , √ √

min

σ

(

AS

)

{

σ

r

(

γ

-

1

-

t

)

, σ

k

(

γ

-

1

-

t

)

-

σ

k

+1

(

γ

+1+

t

)

}

.

(44)

Upper bounding ∥ E ∥ 2 : Finally, conditioned on the event Q , the entries of E are bounded. Choosing the dynamic range of Q as dictated by the upper bound on ∥ AS ∥ max in Lemma E.1, we have, with probability exceeding 1 -ϵ 8 n R 2 ,

Since E ij is a bounded w.h.p., it is also subgaussian w.h.p. (ref. eq. (16)) with subgaussian norm given by,

From Lemma B.12 we get for some absolute constant C ,

Completing the proof : Finally, combining (36), (37), (44) and (47), we get with probability exceeding 1 -ϵ 8 n R 2 -2 e -mt 2 2 -2 e -m ˜ t 2 ,

where we denote µ = max { σ r , σ k -σ k +1 ( √ γ +1+ t √ γ -1 -t )} . Let us choose our bit-budget B of quantizer Q to be such that it satisfies ∥ E ∥ 2 ≤ µ 2 , i.e.,

Then,

√

Setting

it follows that with probability exceeding 1 -ϵ 4 n R 2 ,

This completes the proof.

F Sketched least squares with quantized response

Since the approximation error guarantees of sketched least squares with quantized response might be a problem of independent interest, this is a standalone section, and the notations used in this section are independent of the rest of the paper.

Consider the generalized least squares problem

where Φ ∈ R ℓ × p and Y ∈ R ℓ × q . The sketched variant of (54) with quantized response is given by,

where G ∈ R m × ℓ is a Gaussian sketch matrix with entries are distributed as G ij ∼ N ( 0 , 1 m ) , and Q ≡ Q R , B is the uniformly dithered quantizer as described in (1). We assume that the dynamic range R ≥ ∥ GY ∥ max so that Q is unsaturated. The solution of (55) can be obtained in closed form as,

The following lemma provides a characterization of the accuracy of ˜ X with respect to the original problem (54).

Lemma F.1. Let G ∈ R m × ℓ be a random Gaussian matrix with entries distributed as G ij ∼ N ( 0 , 1 m ) , and Q ≡ Q R , B be a uniformly dithered quantizer with dynamic range R and bit-budget B . Furthermore, suppose Φ ∈ R ℓ × p and Y ∈ R ℓ × q be given, and let us denote

Let E = Q( GY ) -GY be the quantization error matrix. Then, if R ≥ ∥ GY ∥ max , we have

where r = rank( Φ ) , and σ max and σ min are the maximum and minimum singular values of Φ respectively.

Proof. The solution of the generalized least squares problem (54) can be written as:

where y i ∈ R p denote the i th column of Y . Consequently, we will first analyze the standard least squares problem

and generalize the results by concatenating x ∗ i to obtain X ∗ . The sketched variant of (58) with quantized response is given by,

The solution of (59) is available in closed form as ˜ x = ( GΦ ) † Q( Gy ) . Let us denote the quantization error as ϵ ≜ Q( Gy ) -Gy ∈ R m . We then have,

This completes the proof.

Here, (i) follows as the cross term disappears because,

where the last equality follows from (2), since E Q [ ϵ ] = 0 when Q is a uniformly dithered quantizer.

Let us denote the quantization error matrix by E = Q( GY ) -GY ∈ R m × q . Generalizing (60) to the generalized least squares problem by treating each column y i separately and adding, we have,

Here, (i) follows from known results on the approximation error of sketched generalized least squares Halko et al. [23], Pilanci [48], and r is the rank of A .

Upper bounding E ∥ Φ ( GΦ ) † E ∥ 2 F : The remainder of the proof concerns upper bounding the final term in (62). Since ∥ X ∥ 2 F = Tr [ X ⊤ X ] for any matrix X , using the cyclic property of trace,

Since R ≥ ∥ GY ∥ max , from (2), the ( i, j ) th -entry of the quantization error matrix E satisfies,

So,

̸

In other words, E Q [ EE ⊤ ] is a diagonal matrix whose diagonal elements are upper bounded by q ∆ 2 4 . Let us denote Λ = ( ( GΦ ) † ) ⊤ Φ ⊤ Φ ( GΦ ) † . Then, (63) simplifies to,

Furthermore,

Here, (i) follows from Lemma B.1, (ii) is consequence of Lemma B.2, and (iii) follows from Lemma B.9. Combining (63), (66) and (67) yields,

G LPLR algorithm: Approximation error analysis

To prove Thm. 3.2, we will first prove the result when both quantizers Q and Q ′ are unsaturated. We first prove Lemma G.1, which gives us an upper bound on the approximation error when Q and Q ′ are unsaturated. Since we can ensure that they remain unsaturated with high probability (with appropriate choices of dynamic ranges), the final approximation error upper bound in Thm. 3.2 is slightly worse than Lemma G.1.

Lemma G.1. Let our matrix A ∈ R n × d have non-zero singular values σ 1 , . . . , σ k , σ k +1 , . . . , σ r , where r = rank( A ) , and bounded row norms ∥ ∥ a ( i ) ∥ ∥ ≤ R . Let κ ( A ) = σ 1 /σ r and κ ( A k ) = σ 1 /σ k respectively be the condition numbers of A and the best rankk approximation of A , and let us denote

for some sufficiently small ϵ that satisfies 0 < ϵ ≤ 4 n R 2 κ ( A ) 2 γ . Here γ = d/m is the aspect ratio of the sketching matrix S ∈ R d × m with S ij i.i.d. ∼ N ( 0 , 1 m ) . Suppose the dynamic ranges of the quantizers Q and Q ′ are set to R √ 2 log ( 16R 2 n 2 m ϵ ) m and 2 κ √ γ -1 -t as dictated by Lemmas E.1 and E.2 respectively, and suppose their bit-budgets B and B ′ satisfy

where,

and B 2 is equal to

Furthermore, let E be the event that both quantizers Q and Q ′ are unsaturated. Then,

where 1 ( · ) is the indicator function.

Proof. Let us denote the quantization error matrices from Q and Q ′ as E ∈ R n × m and E ′ ∈ R m × d respectively, i.e., E = Q( AS ) -AS and E ′ = Q ′ ( Q( AS ) † A ) -Q( AS ) † A . Since 1 E = 1 implies that Q and Q ′ are unsaturated, E [ E ] = 0 and E [ E ′ ] = 0 . Then,

E [ ∥ Q( AS )Q ′ ( Q( AS ) † A ) -A ∥ 2 F 1 E ] = E S [ E Q , Q ′ [ ∥ Q( AS )Q ′ ( Q( AS ) † A ) -A ∥ 2 F ] 1 E ] . Then,

The cross terms disappear in (i) because,

where, (ii) follows from E Q ′ [ E ′ ] = 0 as the quantizer Q ′ is unbiased when unsaturated.

Analyzing the term T 1 : Recall that A k is the best rankk approximation of A obtained using computing the full-SVD of A and subsequently making all singular values of A less than σ k as 0 . We next analyze the first term in (69) as

where ˜ X ≜ arg min X ∥ S ⊤ A ⊤ k X -Q( S ⊤ A ⊤ ) ∥ 2 F . This minimization problem is a Gaussian sketched variant of a generalized least squares problem with the response matrix quantized as seen in App. F. Corresponding to the notations in App. F, here we have S ⊤ instead of G , A ⊤ k instead of Φ , and A ⊤ instead of Y . As a consequence of Lemma F.1, we have the upper bound,

Analyzing the term T 2 : The term T 2 can be written as,

where, (i) follows once again since E Q ′ [ E ′ ] 1 E = 0 . The first term in (73) can be rewritten as

Similar to (65) in the proof of Lemma F.1, E Q ′ [ E ′ E ′⊤ ] 1 E is a diagonal matrix with

̸

Let us denote Γ ≜ S ⊤ A ⊤ AS . Then using the fact that 1 E ≤ 1 , we get,

Here, the last equality follows as E S [ SS ⊤ ] = I d . This is because the ( i, j ) th entry of E S [ SS ⊤ ] is

̸

Furthermore, the second term E Q , Q ′ ∥ ∥ EE ′⊤ ∥ ∥ 2 F 1 E can be simplified as follows,

Here, (i) follows since E Q ′ [ E ′ E ′⊤ ] 1 E is a diagonal matrix as seen before, and (ii) follows from the upper bound derived on E [( E ⊤ E ) ii ] 1 E in (65).

Combining all of the above, the approximation error of our low-precision low-rank approximation scheme can be upper bounded as,

Here, the first term T 3 is the low-rank approximation error, whereas the remaining terms, i.e., T 4 , T 5 and T 6 appear due to quantization. We now analyze each of them separately.

Analyzing the term T 4 : Since we choose the dynamic range of quantizer Q to be R √ 2 log ( 16R 2 n 2 m ϵ ) m (ref. Lemma E.1), we have

Here, we can ensure (i) holds true, i.e., that this term does not exceed ϵ/ 4 if we set the bit-budget B to satisfy,

Analyzing the term T 5 : Since we choose the dynamic range of the quantizer Q ′ to be equal to 2 κ √ γ -1 -t (ref. to Lemma E.2), where t and κ are defined as in (33), we have

where we have made use of Asm. 3.1. Once again, in order to ensure (i) , it suffices to choose the bit-budget B ′ to be

Note that in order for the dynamic range of quantizer Q ′ to hold true in Lemma E.2, we also require B to satisfy (34), i.e.,

Analyzing the term T 6 : Using bit-budgets B and B ′ as in (80) and (82), the final term is

Here, (i) is ensured by choosing ϵ to be sufficiently small - specifically, ϵ ≤ 4 n R 2 κ ( A ) 2 γ .

Tying it all together : Combining (79), (81), and (84), with the bit-budgets set appropriately as dictated by (80), (83), and (82), eq. (78) simplifies to

This completes the proof.

G.1 LPLR approximation error: Proof of Thm. 3.2

We now formally state our main approximation result.

Theorem G.2. (LPLR approximation error (formal)) Let our matrix A ∈ R n × d have non-zero singular values σ 1 , . . . , σ k , σ k +1 , . . . , σ r , where r = rank( A ) , and bounded row norms ∥ ∥ a ( i ) ∥ ∥ ≤ R . Let κ ( A ) = σ 1 /σ r and κ ( A k ) = σ 1 /σ k respectively be the condition numbers of A and the best rankk approximation of A , and let us denote

for some sufficiently small ϵ that satisfies 0 < ϵ ≤ 4 n R 2 κ ( A ) 2 γ . Here γ = d/m is the aspect ratio of the sketching matrix S ∈ R d × m with S ij i.i.d. ∼ N ( 0 , 1 m ) . Suppose the dynamic ranges of the √ 2 log ( 16R 2 n 2 m )

quantizers Q and Q ′ are set to R ϵ m and 2 κ √ γ -1 -t respectively, and suppose their bit-budgets B and B ′ satisfy

where,

and B 2 is equal to

Then, the low-precision and low-rank factorization returned by Alg. 1 satisfies

where the expectation is over the random sketching matrix S , as well as the inherent stochasticity from quantizers Q and Q ′ .

Proof. For the purpose of analysis, we assume that Alg. 1 returns L = 0 and R = 0 if either quantizer Q or Q ′ gets saturated. In practical implementation, it can easily be checked if either quantizer Q or Q ′ gets saturated or not, and the algorithm can be repeated again with a new realization of the sketching matrix S and stochastic quantizer Q . Since the choice of dynamic ranges for Q and Q ′ ensures that they remain unsaturated with a high probability, 'reasonably few" realizations of S would suffice to get at least one good realization in which Q and Q ′ are unsaturated.

However, in what follows, we assume that that if either quantizer Q or Q ′ gets saturated, then the algorithm returns 0 as an estimate of A , resulting in a Frobenius norm error of ∥ A ∥ F . Since this happen with a very small probability, we show that its contribution to the expected Frobenius norm error of Alg. 1 is small as well. With this in mind, the expected approximation error can be written as

Inequality (i) follows from Asm. 3.1 and the fact that the expectation of indicator function of an event is the probability of the event. From Lemma G.1, the first term can be upper bounded as:

Since Lemmas E.1 and E.2 state the probabilities of quantizers Q and Q ′ being unsaturated, Pr( 1 E C ) can be obtained by an application of union bound as

Then, (87) can be written as:

This completes the proof.

G.2 Informal version of Thm. G.2

From Thm. G.2, we can get a (simplified) asymptotic dependence of the bit-budgets B and B ′ . We have B ≥ max { B 1 , B 2 } ,

where we have ignored the constant terms inside the log 2 ( · ) in (i) and considered the regime m ≪ d in (ii) . Furthermore, B 2 is

where c ′ is approximately a constant. Here, (i) follows because when m ≪ d , we have

Similarly, √ γ +1+ t/ √ 2 √ γ -1 -t = O(1) . Comparing (91) and (92), we have

Moreover, the bit-budget B ′ is

where once again, to get (i) we have ignored the constants, and (ii) holds true when m ≪ d . Moreover, suppose n ≈ d (if they are not equal, then the total bit-budget should be computed as a

weighted average, since the L has nm elements and the second low-rank factor has md elements). Then, for some constant c 3 that depends on R , the total bit-budget is

Furthermore, the dynamic range of quantizer Q is c 1 √ log ( n ϵ ) m and that of quantizer Q ′ when m ≪ d is

for constants c 1 , c 2 and c 4 that depend on R , where additive logarithmic terms are ignored with respect to m . This gives us the result in Thm. 3.2.

H Direct-SVD Quantization: Approximation error analysis

In this section, we consider the baseline scheme for obtaining low-precision low-rank factorization by first computing the optimal low-rank factorization using SVD and individually quantizing the low-rank factors with uniform scalar quantizers. For any given matrix, A ∈ R n × d , we compute the full SVD as A = UΣV ⊤ . The best (unquantized) rankk approximation can be obtained from this by considering the singular vectors corresponding to the topk singular vectors, and constructing the matrix A k = ( UΣ ) k V ⊤ k . Here, ( UΣ ) k ∈ R n × k is the sub-matrix obtained by selecting the first k columns of UΣ , and V ⊤ k ∈ R k × d is the sub-matrix obtained by selecting the first k rows of V ⊤ . Subsequently, these low-rank factors are quantized with uniform scalar quantizers Q and Q ′ (having bit-budgets B and B ′ respectively). The algorithm pseudocode is given in Alg. 2 and we provide an upper bound to the approximation error in Proposition H.1. We use the notation ˜ U = UΣ ∈ R n × d .

Algorithm 2: Direct-SVD quant. : Directly quantizing the optimal low-rank factorization

Input : Matrix A ∈ R n × d , target rank k , Quantizers Q and Q ′ Output: Factorization: LR where L ∈ R n × k , R ∈ R k × d

1 Compute SVD and get A = UΣV ⊤

2 Extract the topk left (scaled) and right singular vectors and get ˜ U k = ( UΣ ) k and V ⊤ k

3 Quantize the factors individually and get L ← Q( ˜ U k ) and R ← Q ′ ( V k )

4 return L ∈ R n × k , R ∈ R k × d

Proposition H.1. (Direct-SVD quant. approximation error (formal)) Let our matrix A ∈ R n × d have maximum singular value σ 1 . Suppose our target rank is k , and for some small ϵ such that 0 < ϵ ≤ 3 kσ 2 1 , the dynamic ranges of quantizers Q and Q ′ are set to be σ 1 and 1 respectively with bit-budgets satisfying

Then, the factorization returned by direct-SVD quantization in Alg. 2 satisfies

where the expectation is over the inherent stochasticity of the quantizers Q and Q ′ .

Proof. Let us denote the quantization error matrices as E = Q( ˜ U k ) -˜ U k and E ′ = Q ′ ( V ⊤ k ) -V ⊤ k . The approximation error is given by

where the cross term disappears in (i) because when Q ′ is unsaturated, E Q ′ [ E ′ ] = 0 , and

Analyzing term T 1 : The first term in (96) can be upper bounded as

Here, in (i) , we utilize the fact that A k = ˜ U k V ⊤ k and the cross term vanishes once again as quantizer Q is unsaturated, i.e., E [ E ] = 0 and,

The second term in (97) is

Here, (i) follows because V ⊤ k V k = I k , and (ii) follows because E [ E ⊤ E ] is a diagonal matrix as

̸

So, (97) can be upper bounded as

Analyzing term T 2 : We upper bound the second term in (96) as

where the cross term vanishes in (i) because E Q ′ [ Tr ( ( ˜ U k E ′ ) ⊤ E Q [ E ] E ′ )] = 0 . The first term in (101) is

Here, (i) and (ii) follow because E [ E ′ E ′⊤ ] is a diagonal matrix following the same argument as (99), i.e.,

̸

Finally, (iii) follows because the i th column of ˜ U k is σ i u i , so ( ˜ U ⊤ k ˜ U k ) ii = ( σ i u i ) ⊤ ( σ i u i ) = σ 2 i ∥ u i ∥ 2 2 = σ 2 i .

The second term in (101) is

Here, (i) and (ii) follow because E [ E ′ E ′⊤ ] and E [ E ⊤ E ] are diagonal, which can be seen using similar arguments as in (99). So, (101) is

Choice of dynamic range for quantizers Q and Q ′ : Using Lemma B.3, the max-norm of the input to the first quantizer can be upper bounded as ∥ ˜ U k ∥ max ≤ ∥ ˜ U k ∥ 2 = σ 1 , and that of the second quantizer is ∥ V k ∥ max ≤ ∥ V k ∥ 2 = 1 . Therefore, choosing the dynamic ranges of Q and Q ′ to be σ 1 and 1 respectively, would ensure that they remain unsaturated, and (105) can be rewritten as

Choice of bit-budgets B and B ′ : For a given ϵ > 0 , suppose we choose B and B ′ so that

With such a choice, the second and third terms in (106) would not exceed ϵ . Furthermore, the last term of (106) will be k n ∆ 2 4 d ∆ ′ 2 4 ≤ dnk ϵ 3 nk ϵ 3 dkσ 2 1 ≤ ϵ 3 whenever ϵ < 3 kσ 2 1 . This completes the proof.

Prop. H.1 states that when n ≈ d , in order to guarantee an ϵ -additive error approximation with respect to the best rankk approximation, we require (ignoring constant multiplicative factors inside the log 2 ( · ) ), a total budget (ref. to Tab. 1) of

I LPLR-SVD algorithm: Approximation error analysis

LPLR-SVD is a simpler variant of our LPLR algorithm in which instead of approximating the range space of A using AS , we compute the full-SVD of A ∈ R n × d as A = UΣV ⊤ , where U ∈ R n × n , Σ ∈ R n × d , and V ∈ R d × d . The pseudocode of LPLR-SVD is provided in Alg. 3.

We first compute the SVD, A = UΣV ⊤ , followed by quantizing the singular vectors corresponding to the topk singular values, scaled by the singular values. We denote ˜ U = UΣ , and the sub-matrix formed by the first k columns as ˜ U k . We quantize this basis to get L = Q( ˜ U k ) as the first low-rank factor. Subsequently, we project the columns of A onto the subspace spanned by this quantized basis, i.e., we solve the optimization problem

The solution of this is given by W ∗ = Q( ˜ U k ) † A . The second low rank factor is given by quantizing it again as R = Q ′ ( Q( ˜ U k ) † A ) .

The following result provides an upper bound on the approximation error of LPLR-SVD. The proof of this result is similar to Thm. 3.2 without the complications arising from Gaussian concentration.

Theorem I.1. (LPLR-SVD approximation error (formal)) Let our matrix A ∈ R n × d have singular values σ 1 ≥ . . . σ k ≥ . . . , and bounded row norms ∥ ∥ a ( i ) ∥ ∥ ≤ R . Let κ ( A k ) = σ 1 /σ k be the condition number of the A k , i.e., the best rankk approximation of A . Consider some sufficiently small ϵ that satisfies 0 < ϵ < 4 kσ 2 1 , and suppose the dynamic ranges of quantizers Q and Q ′ are set to be σ 1 and

Input : Matrix A ∈ R n × d , target rank k , Quantizers Q and Q Output: Factorization: LR where L ∈ R n × k , R ∈ R k × d

1 Compute SVD and get A = UΣV ⊤

2 Extract the topk left to get an approximate basis for the range space of A as ˜ U k = ( UΣ ) k

3 Quantize and get an approximate basis Q( ˜ U k )

4 Find W ∗ = arg min W ∥ ∥ ∥ Q( ˜ U k ) W -A ∥ ∥ ∥ 2 F

5 Quantize W ∗ using quantizer Q ′ to get Q ′ ( W ∗ )

6 return Low-rank and low-precision approximation LR where L = Q( ˜ U k ) , R = Q ′ ( W ∗ ) .

2 κ ( A k ) respectively. Furthermore, suppose the bit-budgets of the quantizers satisfy

where, and,

Then, the low-precision and low-rank factorization returned by Alg. 3 satisfies

Proof. Since the solution of W ∗ = arg min W ∥ ∥ ∥ Q( ˜ U k ) W -A ∥ ∥ ∥ 2 F is available in closed form as W ∗ = Q( ˜ U k ) † A , we have the low rank factorization as A ≈ Q( ˜ U k )Q ′ ( Q( ˜ U k ) † A ) . Let us denote the quantization error matrices as E ′ = Q ′ ( Q( ˜ U k ) † A ) -Q( ˜ U k ) † A , and E = Q( ˜ U k ) -˜ U k . Then, the approximation error is

where the last equality follows the because the quantizer Q ′ is unbiased. Here, the E is over the stochasticity of the quantizers. This decomposition is similar to (69), and can be treated similarly. The term T 1 consists of the low rank approximation error and the error from the first quantizer, whereas the term T 2 consists of error from the second quantizer. We follow steps similar to the proof of Lemma G.1 which are detailed below.

Analyzing term T 1 : Using similar reasoning as (71), this term can be written as

This is the same as (100) in the analysis of direct-SVD quant., and hence can be upper bounded as

′

Analyzing term T 2 : This term is again the same as (101) in the analysis of direct-SVD quant., and can be upper bounded as

Using (111) and (112), we get:

Choice of dynamic range for quantizers Q : Since the input to the quantizer is ˜ U k , we need an upper bound on ∥ ˜ U k ∥ max . The i th column of ˜ U k is ( ˜ U k ) i = σ i u i , where 1 ≤ i ≤ k . From Lemma B.3,

Note that this upper bound is tight in the worst-case sense, i.e., if we consider all matrices with bounded spectral norm, we have sup X : X = UΣV ⊤ , ∥ X ∥ 2 ≤ r ∥ ˜ U k ∥ max = r . Hence, the dynamic range of quantizer Q is set as σ 1 .

Choice of dynamic range for quantizers Q ′ : On the other hand, the input to quantizer Q ′ is Q( ˜ U k ) † A . We adopt an approach similar to the proof of Lemma E.2 to upper bound the max-norm of Q( ˜ U k ) † A as follows:

where (i) and (ii) follow from Lemmas B.3 and B.4 respectively. We upper bound ∥ ∥ ∥ Q( ˜ U k ) † ∥ ∥ ∥ 2 as:

Here, (i) follows because the singular values of Q( ˜ U k ) † are inverses of the singular values of Q( ˜ U k ) , and (ii) follows from Lemma B.5.

We are now left with upper bounding ∥ E ∥ 2 . Since a choice of σ 1 for the dynamic range of Q ensures that quantizer Q remains unsaturated, the entries of E are bounded as:

Since E ij is a bounded w.h.p., it is also subgaussian w.h.p. (ref. eq. (16)) with subgaussian norm given by,

From Lemma B.12 we get for some absolute constant C ,

Let us choose our bit-budget B of quantizer Q to be such that ∥ E ∥ 2 ≤ σ k 2 , i.e.,

where κ ( A k ) = σ 1 /σ k is the condition number of the best rankk approximation of A . Then, using (116), we have:

So, if we choose the dynamic range of quantized Q ′ to be 2 κ ( A k ) , it will remain unsaturated with probability exceeding 1 -ϵ 4 n R 2 , provided that the bit-budget of first quantizer satisfies B satisfies (121).

Choice of bit-budgets B and B ′ : Referring to (113), the term T 3 is,

The term T 4 is:

With the above choices of B and B ′ , the term T 5 is:

Tying it all together : Similar to what is done in the proof of Thm. G.2, for the purpose of analysis, we assume that Alg. 3 returns L = 0 and R = 0 if quantizer Q ′ gets saturated. In practical implementation, it can easily be checked if either quantizer Q or Q ′ gets saturated or not, and the algorithm can be repeated again with a new realization of the stochastic quantizer Q . Since the choice of dynamic ranges for Q ensures that it remains unsaturated with a high probability, 'reasonably few" realizations would suffice to get at least one good realization in which Q is unsaturated.

However, in what follows, we assume that that if quantizer Q gets saturated, then the algorithm returns 0 as an estimate of A , resulting in a Frobenius norm error of ∥ A ∥ F . Since this happen with a very small probability, we show that its contribution to the expected Frobenius norm error of Alg. 3 is small as well. Let us denote the event that quantizer Q is unsaturated as E . Then, the expected approximation error can be written as

where inequality (i) follows from Asm. 3.1 and the fact that the expectation of indicator function of an event is the probability of the event, and (ii) follows from (113). Using appropriate choices of B and B ′ and small enough ϵ , from (122), (123), (124), and (125), we have

Note that from (123) and (121), the bit-budget of the quantizers must satisfy:

where, and,

This completes the proof.

I.1 Informal version of Thm. I.1

Consider n ≈ d . From Thm. I.1, we can get a (simplified) asymptotic dependence of the bit-budgets B and B ′ . In what follows, we ignore the constant multiplicative factors inside log 2 ( · ) . Comparing the expressions for B 1 and B 2 , we have

J Additional numerical simulations

J.1 Image Compression

In addition to Fig. 1 in the main paper, we also provide additional image compression experiments on images of different dimensions in Figs. 6, 5, 7, and 8. The values of B and B ′ are provided in the respective captions of these figures. For every image, the sketch size/target rank m and the bit-budgets B and B ′ are chosen so as to ensure parity with a naïve quantizer with bit budget B nq . In other words, for an n × d dimensional image, B · nm +B ′ · md = B nq · nd . The values of n and d are also provided in the corresponding captions.

For image compression, we also perform a normalize and shift operation of our DSVD/LPLR/LSVD output in order to adjust the scale and bias of the output image, as described next.

Normalize and shift : Let us denote the factorization obtained from executing DSVD/LPLR/LSVD on the input matrix X ∈ R n × d by Y = LR . Then, we can improve the estimate provided by this factorization by considering α Y + β I n × d . The overhead in storing these two additional scalars in full-precision is negligible when n and d are large. The optimal values of α and β can be found as:

Closed-form expressions for ( α ∗ , β ∗ ) can be obtained by equating the derivatives of ∥ α Y + β I -X ∥ 2 F to zero, and solving the resulting linear equations. Doing this algebra yields,

J.2 Wall-clock time for compressing image embeddings

Wenowreport the wall-clock time for computing the low-rank quantized factors via LPLR, LSVD, and DSVDfor embedding compression in Tab. 10. This reinforces the validity of our theoretical assertions concerning the computational advantage of LPLR when compared to its alternative methods.

J.3 High-resolution figures for Llama weight compression

We now provide higher resolutions figures of Figs. 2a and 2b as Figs. 3 and 4, respectively. The code to reproduce these numerical simulations is available at our github repository https://github.com/pilancilab/matrix-compressor .

and,

This gives a total bit-budget of

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

LPLR

Naive Quant

LPLR SVD

Layer

0.5

1.0

1.5

2.0

2.5

3.0

3.5

LPLR

Naive Quant

LPLR SVD

Layer

Dataset

LPLR

DSVD

LSVD

CIFAR-10

71

±

11

ms

306

±

26

ms

312

±

8

ms

CIFAR-100

14

±

3

ms

51

±

3

ms

56

±

3

ms

(a) Original image

(b) Naïve

(0

.

312)

(c) DSVD

(0

.

507)

(d)

LPLR

(0

.

264)

(e)

LSVD

(0

.

260)

八

八

(a) Original image

(b) Naïve

(0

.

508)

(c) DSVD

(0

.

508)

(d)

LPLR

(0

.

311)

(e)

LSVD

(0

.

301)

(a) Original image

(b) Naïve

(0

.

701)

(c) DSVD

(0

.

724)

(d)

LPLR

(0

.

346)

(e)

LSVD

(0

.

351)

(a) Original image

(b) Naïve

(0

.

711)

(c) DSVD

(0

.

556)

(d)

LPLR

(0

.

329)

(e)

LSVD

(0

.

321)

K Limitations and further discussions

On uniformly dithered quantizer : We would like to point out that the approximation error upper bound for our algorithm (derived in Thm. G.2), as well as for the baselines such as direct-SVD quantization (refer to Prop. H.1) holds true only for randomized rounding, or uniformly dithered quantizers. Dithering is preferred due to its ability to produce an unbiased estimate of the input. It offers an advantage by introducing a non-zero probability of quantizing an input to either its ceiling or floor, resulting in reduced variance when averaging multiple independent realizations. Moreover, the unbiasedness of the quantizer output simplifies the analysis. In our experiments, we did not observe any difference while using deterministic rounding instead of dithered rounding.

Comparison with LPLR with direct-SVD and LPLR-SVD : As we see from the numerical evaluations (ref. to Tab. 6 and Tab. 7), direct-SVD quant. and LPLR-SVD performs better than our proposed LPLR algorithm (that computes a Gaussian sketch). Intuitively, one would expect the performance of LPLR-SVD and LPLR (with Gaussian sketch) to be similar because the core idea behind both of them is the same - Quantize the first low-rank factor, and then project the columns of A onto the quantized (first) low-rank factor. For LPLR-SVD the basis being quantized (top-k singular vectors) is the exact top-k rangefinder (for the column space of A), whereas in LPLR, we sketch using the Gaussian matrix to get an approximate rangefinder. This sketching error might make LPLR worse over LPLR-SVD. But because sketching also introduces 'equalization effect of the Gaussian matrices", we can improve the resolution of the uniform scalar quantizers by choosing a smaller dynamic range. Hence, there is a tradeoff when comparing LPLR and LPLR-SVD - a tradeoff between sketching error and the quantization error.

Let us now focus on LPLR-SVD and direct-SVD quant. We will extrapolate this intuition to a comparison between LPLR and direct-SVD quant. The factorization obtained using direct-SVD quant. is A ≈ Q( ˜ U k )Q ′ ( V ⊤ k ) , where the approximation obtained using LPLR-SVD is A ≈ Q( ˜ U k )Q ′ ( Q( ˜ U k ) † A ) . Here, ˜ U k is the n × k matrix obtained using the first k columns of UΣ , where A = UΣV ⊤ is the full SVD of A . Let us assume B ′ →∞ so that we don't have to worry about the second quantizer. Then, we are left to compare the approximations: A ≈ Q( ˜ U k ) V ⊤ k and A ≈ Q( ˜ U k )Q( ˜ U k ) † A . Clearly, the latter has smaller Frobenius norm error because it is the minima of the minimization problem arg min W ∥ ∥ ∥ Q( ˜ U k ) W -A ∥ ∥ ∥ 2 F . So, for B ′ →∞ , it is evident that direct-SVD can only perform as good as LPLR-SVD, and not better than that. Furthermore, note that this difference in performance will be small when the bit-budget of the first quantizer, B is large, and it will be large when B is small. When B →∞ , this difference will be exactly zero. In other words, direct-SVD quant. and LPLR-SVD will be the same factorization when B →∞ and B ′ →∞ . When both B and B ′ are finite, direct-SVD can outperform LPLR-SVD depending on which one out of Q ′ ( Q( ˜ U k ) † A ) and Q ′ ( V ⊤ k ) is closer to the optimal solution, i.e., Q( ˜ U k ) † A . A limitation of our analysis is that we cannot always definitively predict which one will perform better, because we derive upper bounds on the approximation error.

Comparison of LPLR with naive quantization : In Tables 1 and 2 , we have compared LPLR with other benchmarks. In the column denoting approximation errors, aside from the common quantization error ϵ present across all rows, DSVD and LPLR introduce an additional term representing the error from low-rank approximation. Consequently, it might appear that naive quantization consistently outperforms these methods. However, it's essential to recognize that compression techniques based on low-rank factorization hold value exclusively when the matrix being compressed is inherently low-rank, meaning that ∥ A k -A ∥ 2 F starts off as small. If ∥ A k -A ∥ F = 0 , LPLR can achieve identical error levels as naive quantization, while demanding fewer bits than the latter. In other words, given the same bit-budget, LPLR can achieve a lower level of error compared to naive quantization.

Matrix compression with and without any computation constraints : In situations where computational resources are limited, employing the naive strategy emerges as the most cost-effective approach for matrix quantization. Nevertheless, due to its failure to capitalize on a matrix's inherent low-rank arrangement, naive quantization may prove considerably suboptimal. As matrices increase in dimension, accommodating them in memory becomes impractical, rendering approaches like DSVD or LSVD unviable. In such scenarios, our LPLR method stands as a viable alternative,

demanding slightly more computational effort than naive quantization, yet capable of harnessing the low-rank structure for improved approximation accuracy.

If there is no scarcity of computation resources in being able to compute the SVD, and our goal is to just compress an input matrix A , then it is possible to compress the matrix with all the strategies, and choose the best one. However, it must be kept in mind that LPLR provides significant savings with respect to computational complexity, i.e., O( ndm ) , compared to LPLR-SVD, which has a complexity of O( nd 2 ) . Consequently, if the matrix being compressed is very large, it might not even be feasible to do direct-SVD quant. or LPLR-SVD, given the current memory limitations of available GPUs, and LPLR (with Gaussian sketch) becomes our only option.

Furthermore, a careful examination of Tabs. 6 and 7 shows that even though LPLR-SVD and directSVD quant. have a smaller Frobenius norm error than LPLR, the accuracy and weighted F1 scores are still comparable. One aspect of quantization that we have not studied in detail is its inherent regularization effect when testing the performance on test datasets. Whether LPLR leads to a better regularization effect that the other baselines is an open question that will be studied in future works.

Comparison with existing works on quantized random projections for approximate nearest neighbor search : Prior works of Li and Li [31] and Li and Li [32] consider the idea of quantized random projections specifically for the the application of approximate nearest neighbor search. Although related, the goals of these works are different from matrix compression considered in our work. These prior works approximate the matrix vector product Ax by Q( AS )Q ′ ( S ⊤ x ) . For their setup, in addition to Q( AS ) , the random matrix S needs to be stored in full-precision, in order to process an incoming query x . So, their storage requirement is nm quantized and dm full-precision parameters. In contrast, LPLR stores only the quantized entries of L and R , i.e., m ( n + d ) quantized entries, and A ≈ LRx , implying a smaller storage requirement. Furthermore, since LPLR involves computation of Rx , where R consists of md quantized values, it can leverage modern advancements in hardware primitives for speeding up low-precision computations, such as half and mixed-precision compute.

Extension to streaming settings : LPLR as presented in this paper, is used for compression when the entire matrix A is available. An interesting extension is to consider a variant of LPLR to streaming data settings in order to handle new data points. It is possible to do so using sketching based low rank approximations [68]. The left factor L is updated with the concatenation of an additional row corresponding to the newly arriving datapoint. The second factor R , which is the solution of a new least squares minimization problem, can be updated using Woodbury matrix inversion lemma.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of computing a low rank approximation when the low rank factors are under a bit budget constraint, that is, we must output factors L and R with bounded bits such that LR approximates a given input matrix A in the Frobenius norm. The authors show that by incorporating sketching into the quantization procedure, one can get improved bounds, due to the fact that a Gaussian sketch can “flatten” the entries of a vector, which is advantageous when rounding (Appendix D). Empirical results show that this algorithm indeed gives improved results over other naive implementations such as directly rounding SVD factors.\n\nSTRENGTHS:\nThe problem of efficiently quantizing low rank approximations is an extremely important problem given that both low rank approximations and low precision is gaining popularity for compressing massive neural networks (https://papers.nips.cc/paper/2020/file/13b919438259814cd5be8cb45877d577-Paper.pdf, https://arxiv.org/abs/2302.03764). This work offers an interesting new method which takes advantage of the “flattening” property of Gaussian sketches in order to obtain improved results for quantization in the context of low rank approximation. This idea is conceptually simple yet interesting. Empirical results are also convincing.\n\nWEAKNESSES:\nThe contribution is already quite nice, but there are several followup investigations that could strengthen this work much more:\n* Are there any lower bounds on the trade-off between the approximation accuracy and the bit budget?\n* There are structured sketching transforms such as the Subsampled Randomized Hadamard Transform (see Theorem 2.4 of http://www.cs.cmu.edu/afs/cs/user/dwoodruf/www/wNow3.pdf) which also have the “equalizing” or “flattening” type of behavior, yet can be applied in much faster time, and furthermore also save on the storage of the sketching matrix since it is an integer matrix. Can this be used to get faster implementations in theory/practice? \n* Can you report the running time of the experiments?\n* Can you comment on whether the bit bounds imply actual savings in the memory usage, or are the practical implementations of bit complexity too crude to capture these improvements?\n\nQUESTIONS:\n* Should put A \\approx LR in the abstract\n* Some of the references in the Related Works section for Low-rank approximation seem inappropriate for the context. Candes-Tao and Recht et al are not focused on faster algorithms for SVD, but rather a related but different problem of matrix completion. Gradient descent on the low rank factors is indeed one way to solve approximate SVD, which I agree Zhang et al captures, but there may be more appropriate references such as https://arxiv.org/abs/2106.14289 and references within.\n* Line 58: you should first establish that n >= d, or else the SVD time should be O(min{nd^2, dn^2})\n* Typo (line 77): should be “instead of quantizing x directly”\n* Is the randomized rounding in Section 2.1 necessary? Can you get better results with naive rounding?\n* Line 177: what do you mean by “saturated” and “unsaturated”? Please define this term, it is unfamiliar to non-experts.\n\nLIMITATIONS:\nDiscussion of limitations is limited. This work is mainly theoretical, and there are no potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper introduces a novel low-rank matrix factorization algorithm that is using sketching matrix idea and quantization, such as they do:

1. Use Gaussian RV to generate sketch of the matrix and compute the approximate basis
2. Use Quantization with Q - to get Q(AS)
3. Use Q(AS) and Q' to get  Q'(W)
4. Return Q(AS) and Q'(W)

Authors provide theoretical and numerical analysis of their idea.

STRENGTHS:
- introduction section written very well & and very informative and to the point, such as introducing LPLR algorithm briefly, talking about Low-rank approximation and Randomized quantization.

- the introduced algorithm clearly communicated and results are theoretically sound.

WEAKNESSES:
- Abstract seem to be a bit wordy - it would be nice if there were formulations and numbers that grabs readers attention with resutls.

- i like the way motivating the work - with memory constraints - but i am curious is there any application in real life for low rank decomposition that actually saves memory. some examples would be good

- the paper doesn't exactly introduce a new direction - rather seems to be using existing ideas and put them together.

QUESTIONS:
Table 1 & 2  is a bit puzzling me - it seems that Approximataion Error of Naive Uniform is smaller than LPLR and with cheaper computation cost. Do you think that you can clarify in the table itself  - why LPLR is better ? because it seems to be confusing in the first glance.

LIMITATIONS:
The are many sketching / matric factorization algorithms, i am puzzled why we only presented Naive Uniform and Direct SVD algorithms as comparison points.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate combining low-rank matrix factorization and (uniform scalar) quantization.\nThrough theoretical analysis and experiments they demonstrate that this can yield much higher accuracy than directly quantizing the input matrix. One natural choice is to compute the SVD of the matrix and quantize the two factors independently. It's shown that quantizing the left factor first and then computing a new right factor that best approximates the input when multiplied with the quantized left factor yields much better result. The choice of SVD is not critical and could be replaced by randomly mixing the columns of the input matrix, i.e. random sketching. In fact sketching has provable benefits as the entries of sketched matrices are bounded with very high probability. This bounded range improves quantization theoretically. The authors prove several theorems (their proofs are in the extensive appendix) and conduct detailed empirical evaluation.\n\nSTRENGTHS:\n1) All the key ideas of the paper (low-rank approximation, sketching) are sound.\n2) Detailed theoretical analysis with rigorous proofs.\n3) Extensive experiments.\n4) Compression of neural network weights and embeddings via low rank approximation and quantization are popular and impactful topics both to reduce memory usage and to speed up training and inference.\n\nWEAKNESSES:\n1) Neither the ideas nor the analysis are particularly inventive in my opinion. It's self evident that optimizing the second factor after quantizing the first (LSVD) is superior to independent quantization (DSVD), in fact the process could be iterated further. While I appreciate the 30+ pages of proofs provided by the authors it seems as if they rely on chaining known results and techniques for sketching and random matrices combining with patient algebra.\n2) Despite its seemingly weaker theoretical bounds LSVD is always one of the most accurate method in the experiments (see Tables 4-7). This is also clearly highlighted by the authors in the limitations section. \n3) Quantization aware training (not considered in the paper) is highly likely to produce equivalent or better results.\n\nQUESTIONS:\n1) Line 226 and Theorem 3.2: For (1+eps') relative Frobenius error approximation LPLR requires m=O(k/eps') >> k columns, which the text seems to overlook. Could you please discuss?\n2) Table 3: Could you also add LSVD to this experiment too? DSVD's Frobenius error is always the same 0.496, and LPLR's error is the same  the bottom of the table, when the bit budget is doubled, as in the top of the table. Could you discuss why? What was the rank (k) of DVSD, was it the same as m of LPLR? I.e. could you make Rank column header precise?\n3) Matrix entries are typically rescaled to [0,1] before quantization as it's cheap to store their min/max (or some very low/high quantiles), even for each row (or column) of a matrix. How would such rescaling change your theoretical and empirical results?\n\n4) Line 77: Sx -> x\n5) Table 1, row of LPLR: delta is undefined\n6) Lines 219-222: missing log2(), 4 times\n\nLIMITATIONS:\nYes, limitation section is adequate.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
The paper studies compression of low-rank matrices by simultaneous low-rank factorization and quantization. It proposes a method that first quantizes the randomized rangefinder as the first low-rank factor and then quantize the minimizer of reconstruction error with respect to the remaining factor as the second. Randomized rangefinder uses random Gaussian matrix which possesses the equalization property to maintain low quantization error, compared to naïve quant. Experiments are provided to demonstrate the benefits of the proposed algorithm.

STRENGTHS:
1. provide a low rank factorization algorithm that come with quantization for further reducing memory footprint.
2. experiments demonstrate the advantage of the algorithm.

WEAKNESSES:
1. experimental settings should be made clearer. the current description is a bit confusing.

QUESTIONS:
1. what is the difference between LPLR and LPLR-SVD in experiments?
2. Accuracy seems to decrease with bits for some cases in experiments. Why?

LIMITATIONS:
No limitations are addressed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a memory efficient approach to approximate a matrix $A$ by: low-rank approximation $A=LR$ and quantization. The LPLR algorithm first applies a quantized random projection (RP) as the $L$, and then solve a minimization problem for the right loew-rank factor $R$, which is also quantized afterwards. Theoretical approximation error is obtained and compared with an alternative approach that quantized SVD low rank factors instead of RP. Experiments are conducted on image approximation and embedding classification, to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The paper is well-organized and easy to follow. The theoretical analysis seems rigorous.\n\n2. Experiments on multiple ML tasks and datasets are provided which make the results more grounded.\n\nWEAKNESSES:\nIn my understanding, the main idea of the paper is to waive the need to compute the SVD of A, by using random projection (RP) as a surrogate. I have the following concerns and suggestions:\n\n1. At line 220, the authors wrote that the bits per entry for SVD-quant is $O(nd\\sqrt k)$, but in Table 1 and Table 2, it is $O(k\\sqrt{nd})$. Please double check and clarify. Also, the authors simply stated that LPLR is better than SVD-quant in terms of bits per query, which is not true with some n, d, m, k (comparing the results in the table). I suggest the authors to carefully compare the results and state the regimes when LPLR is better, and when it is worse.\n\n2. In the main Theorem 3.2, $\\kappa$ could be negative, right? Is $1-c_4\\sigma_k/\\sigma_{k-1}$ bounded? Or do we need to further assume an eigen gap for this result to hold?\n\n3. I understand that the main usage of LPLR is for matrix (data) approximation, so the first experiments (Figure 1 and Table 3) make sense to me. However, for the second set of experiments on classification, why not directly using $Q(AS)$ (i.e., the quantized random projections)? This saves the storage for W (in other words, we may increase the sketch size m when using Q(AS) only). Some recent references on this include\n\nRandom projections with asymmetric quantization, Li and Li NeurIPS 2019\n\nGeneralization error analysis of quantized compressive learning, Li and Li, NeurIPS 2019 \n\nIndeed, the research on QRP is highly related to this submission, since LPLR essentially does an optimization on W to recover the data from QRP. I suggest to add some discussion on this direction in the paper and some empirical comparisons.\n\n4. Also, if LPLR is used for processing or storing the data for classification or search tasks, it might be inconvenient to handle new data points (e.g., in a streaming setting). Thus, it may not be suitable for such tasks. On the other hand, recently people are using low rank approximation in LLM fine-tuning frequently. Experiments related to fine-tuning language models could be a better application scenario for the proposed method.\n\n5. Some references on similar results are missing. A similar result as in Appendix D that $S^TQ(AS)$ with uniform quantization has approximation error independent of $d$ has been established in [EDEN: Communication-Efficient and Robust Distributed\nMean Estimation for Federated Learning, Vargaftik et al., ICML 2022] (or maybe some even earlier paper) for rotation matrix $S$. This related result should be cited. Also, Eq. (2) is a standard result for uniform stochastic rounding. A reference should be added there.\n\nIn all, I think the paper proposes a simple but intuitive method for low-rank low-precision matrix approximation from QRPs. The idea is clear, the analysis seems sufficient (despite the above and below questions).The experiments can be improved, but the current results on several tasks and datasets are convincing enough to show the effectiveness of LPLR in matrix approximation. For now, I would recommand borderline accept.\n\nQUESTIONS:\n1. Proposition D.1 only bounds the error between $Q(Sx)$ and $Sx$. How does it imply $||S^TQ(Sx)-x||^2$ is also a constant?\n\n2. What solver is used to solve Algorithm 1 line 4? This should be clarified. Is closed-form feasible?\n\nLIMITATIONS:\nNA.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}

comments: SUMMARY:
The paper studies the low-rank factorization of the matrix in the low-precision setting and proposes a new algorithm which is a combination of randomized low-rank approximation method and quantization.  The paper formally analyzes the guarantee of the proposed algorithms and also give experiments on real world dataset which demonstrate the advantage of the proposed algorithm.

STRENGTHS:
1. The presentation of the paper is good. The writing of the paper is clear and easy to follow.

2. To get the formal guarantee, the authors do a careful analysis, which I think is non-trivial.

WEAKNESSES:
1. Technical novelty: the main algorithm (Algorithm 1) seems to just be the standard way in randomized numerical linear algebra then plusing the quantization. Can you authors give more explanations about the technical novelty? (though the analysis I think is not standard)

2. Experiment: I have some questions about the setting and details of the experiments section. See the next question.

QUESTIONS:
1. Theorem 3.2 shows the guarantee of the proposed algorithms. However, would the sketch size $m$ also have the requirement given the accuracy parameter $\epsilon$ and $k$? Also in experiment it may be a factor that affects the runtime and accuracy a lot, is there some place indicating the choice of $m$?

2. I am a little confused about the naive quantization baseline. Can this way make the matrix be low-rank?

3. As mentioned in the paper, the sketching-based method is popular in randomized low-rank approximation. I think in the experiment the baselines should also include it (with the naive way of quantization).

4. The paper discusses the runtime complexity. Hence, I think it would be better to include the runtime in the experiment section.

LIMITATIONS:
See the above questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a low rank, quantized/low precision matrix factorization which decomposes an n x d matrix A in the form A= LR, where L (of size n x m) and R (of size m x d) are low rank factors. L and R are computed using a random projection matrix S in the form L = Q(AS) and R = Q'(W^*) where W^* is the matrix minimizing the squared Frobenius norm ||Q(AS)W− A||. Q and Q' are two independent quantizers with specified budgets. \n\nThe authors contrast their method with an SVD based method for computing the quantized low rank approximation which instead sets L = Q(U_k S_k) and R = Q'(V_k), where U_k/V_k and S_k are the singular vectors/values respectively. The paper has a theorem deriving a bound on the Frobenious norm of the factorization eror. They apply the approximation on image data (for image compression) and embedding matrices (for an embedding classifation task).\n\nSTRENGTHS:\n+ The paper is very well written and very clear in its presentation\n+ Clear technical presentation incl. theorems, algorithms etc.\n+ Novel idea, providing good review of relevant literature (different matrix sketching approaches, quantization etc.)\n+ Thoughtful experiments to demonstrate real world application (using embedding compression)\n\nWEAKNESSES:\nI don't see any weaknesses that need to be addressed at this moment.\n\nQUESTIONS:\n* In Table 3, why are the SVD and Naive quant. Frobenious norm errors the same for different rank choice? Is it because the matrices are really low (<15) rank? \n\n* I think Table 3 may be expanded to a set of figures to better convey the message (with potentially more bit budgets).\n\nLIMITATIONS:\nThis is an optimization paper and as such the limitations are not immediately obvious, though the eventual use of the model for downstream compression/computation speed up could be quite broad. The societal impact could also be positive given the matrix compression can lead to compute/energy savings. The paper doesn't have a dedicated discussion/section on any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-03"}