SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rajarshi Saha, Varun Srivastava, Mert Pilanci
Stanford, CA 94305, USA
{rajsaha, vsriva, pilanci}@stanford.edu
Matrices are exceptionally useful in various fields of study as they provide a convenient framework to organize and manipulate data in a structured manner. However, modern matrices can involve billions of elements, making their storage and processing quite demanding in terms of computational resources and memory usage. Although prohibitively large, such matrices are often approximately low rank. We propose an algorithm that exploits this structure to obtain a low rank decomposition of any matrix  $\mathbf{A}$  as  $\mathbf{A} \approx \mathbf{L}\mathbf{R}$ , where  $\mathbf{L}$  and  $\mathbf{R}$  are the low rank factors. The total number of elements in  $\mathbf{L}$  and  $\mathbf{R}$  can be significantly less than that in  $\mathbf{A}$ . Furthermore, the entries of  $\mathbf{L}$  and  $\mathbf{R}$  are quantized to low precision formats - compressing  $\mathbf{A}$  by giving us a low rank and low precision factorization. Our algorithm first computes an approximate basis of the range space of  $\mathbf{A}$  by randomly sketching its columns, followed by a quantization of the vectors constituting this basis. It then computes approximate projections of the columns of  $\mathbf{A}$  onto this quantized basis. We derive upper bounds on the approximation error of our algorithm, and analyze the impact of target rank and quantization bit-budget. The tradeoff between compression ratio and approximation accuracy allows for flexibility in choosing these parameters based on specific application requirements. We empirically demonstrate the efficacy of our algorithm in image compression, nearest neighbor classification of image and text embeddings, and compressing the layers of LlaMa-7b. Our results illustrate that we can achieve compression ratios as aggressive as one bit per matrix coordinate, all while surpassing or maintaining the performance of traditional compression techniques.
Low-rank structures for matrices have proven to be incredibly valuable and ubiquitous across numerous fields of study. Several real-world matrices approximately exhibit low-rank structure due to inherent redundancy or patterns, allowing them to be approximated using low-rank factors. Udell and Townsend [69] provide a potential justification by considering a generative model with latent variables for real-world matrices. Applications where low-rank structures in matrices are exploited include, but are not limited to, imaging (Lingala et al. [34]), fine-tuning large language models (Aghajanyan et al. [3], Hu et al. [25], Karimi Mahabadi et al. [30], Valipour et al. [70], Wang et al. [76]), compressing neural networks (Ben Noach and Goldberg [8], Edelbayev and Carreira-Perpinan [26], Mao et al. [39], Phan et al. [47], Swaminathan et al. [63], Tahaei et al. [64], Wang et al. [77], Winata et al. [78], Yu et al. [84]), obtaining efficient NN architectures (Jaderberg et al. [27], Tai et al. [65]), etc.
Given a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , a low-rank approximation is given by  $\mathbf{A} \approx \mathbf{L}\mathbf{R}$ , where  $\mathbf{L} \in \mathbb{R}^{n \times m}$ ,  $\mathbf{R} \in \mathbb{R}^{m \times d}$  denote the left and right low-rank factors with  $m \ll \min\{n, d\}$ . Since  $m(n + d) \ll nd$ ,
(a) Original image
(b) Naive quant.
(c) DSVD
Figure 1: Compression of Shepp-Logan phantom (a standard test image for medical image reconstruction). Naive quant. was done with 2-bits per pixel of this  $10^{3} \times 10^{3}$  image. Quantizing the SVD factors "directly" (i.e., DSVD) and (our) LPLR/LSVD algorithms, factorize the image into a product of tall & wide matrices which reduces the total number of elements, allowing each entry to be represented using upto 8-bits of precision per pixel. Despite the increase in precision per pixel, the total number of bits remains the same at  $2 \cdot 10^{6}$ .
(d) LPLR (ours)
(e) LSVD (ours)
the total number of entries in LR can be significantly smaller than A, enabling matrix compression. The need for compression is driven by the overwhelming storage demands and computational complexity linked to large matrices. In a comparable but distinct line of work, low-precision (LP) representations have also been studied extensively as a means to reduce the memory footprint. Quantization of a continuous valued variable using a small number of bits reduces storage requirement while trading off accuracy. In addition, they also facilitate low latency for real-time inference and low energy consumption (see Gholami et al. [21]). Works of Alimisis et al. [4], Safaryan et al. [55] have studied various matrix compression operators for distributed optimization.
In this work, we introduce LPLR: Low-Precision Low-Rank factorization – a general matrix compression algorithm that simultaneously exploits the low-rank structure of a matrix and quantizes it, to obtain a low-rank factorization, such that the elements of the factors are represented using a small number of bits. Although LPLR can generically refer to a class of algorithms whose main goal is to obtain low-precision representations and exploiting low-rank structures in matrices, in the rest of the paper, we reserve this acronym for our proposed algorithm in Alg. 1. Fig. 1 shows the effectiveness of LPLR in preserving the semantic features in image compression. Other algorithms designed towards achieving the same objective have been named and described appropriately throughout the paper.
Low-rank approximation: The optimal rank-  $k$  approximation of a matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ , denoted as  $\mathbf{A}_k$ , can be obtained by performing the singular value decomposition (SVD) of  $\mathbf{A}$ , which gives  $\mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^{\top}$ . To obtain  $\mathbf{A}_k$ , we keep only the top  $k$  singular values along with their corresponding left and right singular vectors, setting the remaining singular values to zero. In other words,  $\mathbf{A}_k = \mathbf{U}_k \boldsymbol{\Sigma}_k \mathbf{V}_k^{\top} = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^{\top}$ , where  $\sigma_i$ ,  $\mathbf{u}_i$ , and  $\mathbf{v}_i$  represent the  $i$ -th singular value, left singular vector, and right singular vector, respectively. The matrix  $\mathbf{A}_k$  minimizes the Frobenius norm  $\| \mathbf{A} - \widehat{\mathbf{A}} \|_F$  under the constraint that the rank of  $\widehat{\mathbf{A}}$  is at most  $k$ . This result is known as the Eckart-Young-Mirsky theorem [18], which states that  $\| \mathbf{A} - \mathbf{A}_k \|_F^2 = \sum_{i > k} \sigma_i^2$ . However, computing the SVD has a high computational complexity of  $O(nd^2)$  ( $n \geq d$  without loss of generality), which can be impractical for large matrices. Therefore, alternative methods such as in Jin et al. [28], Ye and Du [82], Zhang et al. [85] have been proposed which solve variants of the minimization problem. Chi et al. [11] provide a survey of optimization-based approaches for obtaining low-rank approximation.
Along a parallel line of work, randomized low-rank factorization algorithms have demonstrated their effectiveness in handling massive datasets efficiently while maintaining competitive accuracy levels. Several works, including those by Derezinski et al. [13], Drineas and Mahoney [16], Drineas et al. [17], Halko et al. [23], Ma and Solomonik [37], Mahoney [38], Martinsson et al. [43], Tropp et al. [67], Witten and Candès [79] have focused on reducing the complexity of computing SVD to approximately  $\mathrm{O}(nm^2)$ , where  $m \ll d$ . These algorithms aim to approximate the range space of matrix  $\mathbf{A}$  by utilizing a sketched version  $\mathbf{AS}$ , where  $\mathbf{S} \in \mathbb{R}^{d \times m}$  is a random matrix with  $m \ll d$ . The columns of  $\mathbf{A}$  are then projected onto this approximate range space. By choosing  $m = k + p$ , where  $p \geq 2$  is a small integer, the resulting low-rank approximation  $\mathbf{A} \approx \mathbf{LR}$  provided by these algorithms can be proven to be within a small multiplicative factor of the optimal rank-  $k$  approximation, stated as  $\mathbb{E}\| \mathbf{LR} - \mathbf{A}\|_{\mathrm{F}}^{2} \leq (1 + \delta)\| \mathbf{A}_{k} - \mathbf{A}\|_{\mathrm{F}}^{2}$ . The sketching matrix  $\mathbf{S}$  is typically selected from the
widely used class of Johnson-Lindenstrauss (JL) embeddings. A popular choice when  $\mathbf{A}$  is dense is to sample a Gaussian matrix where each entry  $S_{ij} \sim \mathcal{N}(0, \frac{1}{m})$ .
Randomized quantization: JL embeddings also have practical uses in vector quantization. In this context, when quantizing a vector  $\mathbf{x} \in \mathbb{R}^d$ , instead of quantizing  $\mathbf{x}$  directly, the encoder quantizes  $\mathbf{Sx}$ , and then the decoder obtains an approximation of  $\mathbf{x}$  through  $\mathbf{S}^\top \mathbf{Q}(\mathbf{Sx}) \approx \mathbf{x}$ . This randomized transformation equalizes the coordinate values of  $\mathbf{x}$ , allowing for a smaller range of values and higher precision for the quantizer  $\mathbf{Q}$  under a fixed bit-budget. As a result, it leads to a smaller  $\ell_2$  error compared to independently quantizing each coordinate in a naive manner. Works such as Chen et al. [10], Lyubarskii and Vershynin [36], Mayekar and Tyagi [44], Safaryan et al. [54], Saha et al. [56, 57, 58], Studer et al. [60], Suresh et al. [61, 62], Vargaftik et al. [71, 72], Young et al. [83] explore different variations of this concept for different applications.
2 Proposed Algorithm
Prior to detailing our algorithm, we first review the characteristics of the uniformly dithered quantizer that we employ within our approach. Uniformly dithered quantization has been utilized in various prior works Alistarh et al. [5], Bernardo et al. [9], Gray and Stockham [22], Mayekar and Tyagi [44], Suresh et al. [61, 62], and is succinctly described in §(2.1) below.
2.1 Uniformly dithered quantizer
Let us consider quantizing a scalar  $x$  with  $|x| \leq \mathrm{R}$ . Given a bit-budget of B bits, the scalar quantizer with dynamic range R is described by first specifying the  $M = 2^{\mathrm{B}}$  quantization points as:
$$
q _ {1} = - \mathrm {R}, q _ {2} = - \mathrm {R} + \Delta , q _ {3} = - \mathrm {R} + 2 \Delta , \dots , q _ {M} = - \mathrm {R} + (M - 1) \Delta .
$$
Here, the resolution is given by  $\Delta = \frac{2\mathrm{R}}{M - 1}$ , and the quantizer operation is defined as:
$$
\mathrm {Q} _ {\mathrm {R}, \mathrm {B}} (x) = \left\{ \begin{array}{l l} q _ {k + 1} & \text {w i t h p r o b a b i l i t y} r, \\ q _ {k} & \text {w i t h p r o b a b i l i t y} 1 - r, \end{array} \right. \tag {1}
$$
where  $k = \arg \max_{j} \{ q_{j} \leq x \}$ , i.e.,  $x \in [q_{k}, q_{k + 1})$ , and  $r = \frac{x - q_k}{\Delta}$ . Such a quantizer satisfies
$$
\mathbb {E} \left[ \mathrm {Q} _ {\mathrm {R}, \mathrm {B}} (x) \right] = x \quad \text {a n d} \quad \mathbb {E} \left(\mathrm {Q} _ {\mathrm {R}, \mathrm {B}} (x) - x\right) ^ {2} \leq \frac {\Delta^ {2}}{4} = \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}}, \tag {2}
$$
i.e., it is unbiased and the quantization error variance is dictated by R and B. Here, the  $\mathbb{E}(\cdot)$  is over the randomness from dithering in (1) (ref. App. C). If the input  $x$  to the quantizer falls outside this range, i.e.,  $x > \mathrm{R}$  or  $x < -\mathrm{R}$ , the quantizer is said to be saturated. Finally, to quantize any matrix  $\mathbf{X}$ , we obtain  $\mathrm{Q}_{\mathrm{R,B}}(\mathbf{X})$  by quantizing each entry independently, i.e.,  $[\mathrm{Q}_{\mathrm{R,B}}(\mathbf{X})]_{ij} \triangleq \mathrm{Q}_{\mathrm{R,B}}(X_{ij})$ .
2.2 Proposed algorithm: LPLR
In order to comprehend low precision and low rank representations, we first introduce a simple strategy, which we name as direct-SVD quant. This method involves two main steps: It first computes the optimal rank-  $k$  decomposition  $\mathbf{A}_k = \mathbf{U}_k\pmb {\Sigma}_k\mathbf{V}_k^\top$ , and then, it quantizes the low-rank factors independently, namely,  $\mathbf{L} = \mathrm{Q}(\mathbf{U}_k\pmb {\Sigma}_k)$  and  $\mathbf{R} = \mathrm{Q}'(\mathbf{V}_k^\top)$ . Here,  $\mathrm{Q}$  and  $\mathrm{Q}'$  are uniform scalar quantizers with respective bit-budgets B and B'. A detailed analysis of this direct-SVD quantization approach can be found in Appendix H, along with pseudocode provided in Algorithm 2. Despite its simplicity, this is not the optimal strategy due to two reasons. Firstly, it necessitates computing the SVD of  $\mathbf{A}$ , which is computationally expensive at  $\mathrm{O}(nd^2)$ . Secondly, let us consider improving the approximation to  $\mathbf{A}$  when the first factor is fixed to  $\mathrm{Q}(\mathbf{U}_k)$  by solving the optimization problem:
$$
\mathbf {X} ^ {*} = \underset {\mathbf {X} \in \mathbb {R} ^ {k \times d}} {\arg \min } \| Q (\mathbf {U} _ {k}) \mathbf {X} - \mathbf {A} \| _ {\mathrm {F}} ^ {2} = Q \left(\mathbf {U} _ {k}\right) ^ {\dagger} \mathbf {A}. \tag {3}
$$
This computes the projection of the columns of  $\mathbf{A}$  onto the range space of  $\mathrm{Q}(\mathbf{U}_k)$ . Suppose the resulting projection coefficients are further quantized to get  $\mathrm{Q}'(\mathrm{Q}(\mathbf{U}_k)^\dagger\mathbf{A})$ . Since  $\mathbf{X}^*$  is the solution of (3), for a sufficiently large value of  $\mathbf{B}'$ , it is evident that  $\left\| \mathrm{Q}(\mathbf{U}_k)\mathrm{Q}'(\mathrm{Q}(\mathbf{U}_k)^\dagger\mathbf{A}) - \mathbf{A}\right\|_{\mathrm{F}}^2 \leq \left\| \mathrm{Q}(\mathbf{U}_k\Sigma_k)\mathrm{Q}'(\mathbf{V}_k^\top) - \mathbf{A}\right\|_{\mathrm{F}}^2$ , and hence, better than direct-SVD quant. However, this approach of projecting onto the range space of  $\mathrm{Q}(\mathbf{U}_k)$  (which we refer to as LPLR-SVD and analyze in App. I), still requires the computation of  $\mathbf{U}_k$ , so we can replace  $\mathbf{U}_k$  by  $\mathbf{AS}$ , i.e., an approximation of the
Input :Matrix  $\mathbf{A}\in \mathbb{R}^{n\times d}$  , sketch size  $m$  , Quantizers Q,  $\mathrm{Q^{\prime}}$  with dynamic ranges  $\mathrm{R_Q}$ $\mathrm{R_{Q^{\prime}}}$  and bit-budgets B,  $\mathrm{B^{\prime}}$  respectively.
Output: Factorization: LR where  $\mathbf{L} \in \mathbb{R}^{n \times m}$ ,  $\mathbf{R} \in \mathbb{R}^{m \times d}$
1 Sample a Gaussian sketching matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$  with entries  $S_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ .
2 Compute an approximate basis of column space of  $\mathbf{A}$  by forming the sketch: AS.
3 Quantize the approximate basis with  $\mathbf{Q}$  to get  $\mathrm{Q}(\mathbf{AS})$
4 Find  $\mathbf{W}^{*} = \arg \min_{\mathbf{W}}\| Q(\mathbf{A}\mathbf{S})\mathbf{W} - \mathbf{A}\|_{\mathrm{F}}^{2}$
5 Quantize  $\mathbf{W}^*$  using quantizer  $\mathrm{Q}'$  to get  $\mathrm{Q}'(\mathbf{W}^*)$ .
6 return Low-rank and low-precision approximation LR where  $\mathbf{L} = \mathrm{Q}(\mathbf{A}\mathbf{S})$ ,  $\mathbf{R} = \mathrm{Q}'(\mathbf{W}^*)$ .
range space obtained through random linear combinations of the columns of  $\mathbf{A}$  (also known as a randomized rangefinder [42]). This leads us to our LPLR algorithm, described in Alg. 1, which finds  $\mathbf{W}^{*} = \arg \min_{\mathbf{W} \in \mathbb{R}^{k \times d}} \| \mathbf{Q}(\mathbf{AS})\mathbf{W} - \mathbf{A}\|_{\mathrm{F}}^{2}$  and forms the low-rank low-precision approximation  $\mathrm{Q}(\mathbf{AS})\mathrm{Q}'(\mathbf{W}^{*})$  where  $\mathrm{Q}, \mathrm{Q}'$  are quantization operators. While the solution of this problem is available in closed form as  $\mathbf{W}^{*} = \mathbf{Q}(\mathbf{AS})^{\dagger}\mathbf{A}$ , one can also use an approximation of  $\mathbf{W}^{*}$  by solving this least-squares minimization using an iterative method such as conjugate gradient descent.
In addition to the above argument supporting the superiority of LPLR compared to other baselines (ref. to Tabs. 1 and 2), there exists another essential reason why LPLR outperforms them. This reason directly relates to the selection of  $\mathbf{S}$  as a Gaussian matrix, which is an integral component of LPLR. Random Gaussian matrices are JL embeddings and possess an equalization property that enhances the precision of uniform quantizers. In particular, let us consider an arbitrary vector  $\mathbf{x} \in \mathbb{R}^d$  and obtain an estimate as  $\widehat{\mathbf{x}} = \mathbf{S}^\top \mathbf{Q}(\mathbf{S}\mathbf{x})$  using a uniform quantizer  $\mathbf{Q}$ . It can be shown that the vector quantization error remains constant and does not grow with the dimension  $d$ , expressed as  $\mathbb{E}\left\| \widehat{\mathbf{x}} - \mathbf{x} \right\|_2^2 = \mathrm{O}(1)$ . This represents a substantial improvement compared to the naive strategy of independently quantizing each coordinate of  $\mathbf{x}$ , which leads to a quantization error growth rate of  $\mathrm{O}(d)$ . We provide a detailed explanation of this phenomenon in App. D. Furthermore, even when the quantization is 1-bit per coordinate, e.g.,  $\mathrm{Q}(\mathbf{S}\mathbf{x}) = \mathrm{Sign}(\mathbf{S}\mathbf{x})$ , this embedding provides strong near-isometric embedding properties due to the properties of random hyperplane tessellations [50].
While the strong equalization property of Gaussian embeddings is a known result, certain works such as Saha et al. [57, 58], Suresh et al. [61, 62] opt for using randomized Hadamard embeddings instead of Gaussian ones. The reason behind this choice is twofold: (i) Gaussian matrices are dense, requiring  $\mathrm{O}(d^2)$  multiplications when computing  $\mathbf{S}\mathbf{x}$ , and (ii) the entries of  $\mathbf{S}$  are floating point numbers that must be stored in full precision, contradicting the objective of quantizing  $\mathbf{x}$  using fewer bits. However, these concerns are not problematic for LPLR because the effects of  $\mathbf{S}$  in the first low-rank factor are neutralized by the second low-rank factor, and  $\mathbf{S}$  does not need to be stored. In fact, we exploit both the equalization property and the subspace approximation property of Gaussian matrices to derive a superior upper bound for the approximation error, as discussed next in §3.
3 Approximation Error Analysis
We are now in a position to state the approximation error guarantee of LPLR. Let us denote the  $i^{\mathrm{th}}$  row of our input matrix  $\mathbf{A}$  as  $\mathbf{a}^{(i)}$ . For convenience of analysis, we make the following assumption.
Assumption 3.1. Rows of matrix  $\mathbf{A}$  have bounded norm, i.e.,  $\| \mathbf{a}^{(i)}\| \leq R$  for some known  $R > 0$ .
The following result gives an informal upper bound on the expected Frobenius norm error of the factorization returned by Alg. 1.
Theorem 3.2. LPLR approximation error (Informal) Suppose our input matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  with  $\| \mathbf{a}^{(i)} \| \leq \mathrm{R} = \mathrm{O}(1)$  has singular values  $\sigma_1, \ldots, \sigma_r$  with  $r = \operatorname{rank}(\mathbf{A})$  and target rank as  $k$ . Let  $\kappa(\mathbf{A}) = \sigma_1 / \sigma_r$  and  $\kappa(\mathbf{A}_k) = \sigma_1 / \sigma_k$  respectively be the condition numbers of  $\mathbf{A}$  and the best rank- $k$  approximation of  $\mathbf{A}$ , and let us denote  $\kappa = \min \left\{\kappa(\mathbf{A}), \kappa(\mathbf{A}_k) (1 - c_4 \sigma_{k+1} / \sigma_k)^{-1}\right\}$ . Furthermore, for a sufficiently small constant  $\epsilon > 0$ , suppose the dynamic range of  $\mathbf{Q}$  is set to be  $c_1 \sqrt{\log(n / \epsilon) / m}$ ,
Algorithm 1: LPLR: Randomized Low-Precision Low-Rank factorization
and that of  $\mathbf{Q}'$  is set to  $2\kappa \sqrt{m / d}$ . Then, the LPLR factorization returned by Alg. 1 satisfies
$$
\mathbb {E} \left\| \mathbf {L R} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \left(1 + \frac {k}{m - k - 1}\right) \left\| \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} + \epsilon ,
$$
while utilizing a total budget of  $\log_2\left(\frac{c\kappa(\mathbf{A}_k)\kappa}{\epsilon}\frac{nm}{\sqrt{d}}\sqrt{\log\left(\frac{mn^2}{\epsilon}\right)}\right)$  bits for  $n\approx d$ . Here,  $c_{1},c_{2},c_{3},$  and  $c_{4}$  are constants that depend on R.
We provide a less formal version of our main result here, suitable for interpretation. The formal statement of this result, including specific constant values, can be found in Thm. G.2 of App. G. It does not necessitate the assumption  $n \approx d$  and provides distinct thresholds for B and B'. Thm. G.2 asserts that, for a target rank-  $k$ , as long as  $m \geq k + 2$ , one can ensure an arbitrarily small approximation error of  $\epsilon$  by selecting the number of bits to be at least above a certain threshold budget. The threshold depends on the error tolerance  $\epsilon$ , dimensions  $n$  and  $d$ , the sketch size  $m$ , and the spectrum of A. The value of  $\kappa$  is determined by taking the smaller of two quantities. In the case of matrices with a sharp decline in singular values (e.g., matrices of exact rank-  $k$ ), where the ratio  $\sigma_k / \sigma_{k+1}$  approaches zero,  $\kappa \approx \kappa(\mathbf{A}_k)$ . For matrices with a smoother spectrum (e.g., all singular values are equal),  $\kappa = \kappa(\mathbf{A})$ , the condition number of the input matrix A.
Remark 1. We consider two distinct scenarios based on Asm. 3.1. The first case assumes that the row norms are bounded by a constant, represented by  $\mathrm{R} = \mathrm{O}(1)$ . This assumption is reasonable when the rows of  $\mathbf{A}$  correspond to different normalized features of a data point. The second case assumes that the individual entries of  $\mathbf{A}$  are bounded by a constant, i.e.,  $A_{ij} = \mathrm{O}(1)$ . This implies that  $\mathrm{R} = \mathrm{O}(\sqrt{d})$ , which is a reasonable assumption for scenarios like images, where it is known that each pixel value is bounded. In Tab. 1, we compare the performance of the algorithms when  $\mathrm{R} = \mathrm{O}(1)$ , while in Tab. 2, we assume  $\mathrm{R} = \mathrm{O}(\sqrt{d})$ . Thm. 3.2 assumes that  $\mathrm{R} = \mathrm{O}(1)$ . The expressions in Tab. 2 can be obtained in a similar manner from the formal version in Thm G.2.
3.1 Analysis outline
The derivation of the upper bound on the approximation error of LPLR is presented in App. G. In this section we outline a brief proof sketch that highlights the main challenges of the proof. As mentioned already in §2.2, the analysis of LPLR utilizes the subspace embedding and equalization properties of random Gaussian matrices. A key component in Alg. 1 is the choice of dynamic range for quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ . In our analysis, we assume that when either  $\mathrm{Q}$  or  $\mathrm{Q}'$  gets saturated in lines 3 and 5 of Alg. 1, a trivial factorization of  $\mathbf{LR} = \mathbf{0}$  is returned. We choose the dynamic ranges  $\mathrm{R}_{\mathrm{Q}}$  and  $\mathrm{R}_{\mathrm{Q'}}$  to be sufficiently high enough so that this happens with a very low probability. Formally, for quantizer  $\mathrm{Q}$ , Lemma E.1 states the following:
$$
\| \mathbf {A S} \| _ {\max } \leq R \sqrt {\frac {2 \log \left(\frac {1 6 \mathrm {R} ^ {2} n ^ {2} m}{\epsilon}\right)}{m}} \quad \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - \frac {\epsilon}{8 n \mathrm {R} ^ {2}}.
$$
Here,  $\| \mathbf{AS} \|_{\max}$  is max-norm of the matrix  $\mathbf{AS}$ , i.e., the coordinate with maximum magnitude. This concentration result is a consequence of the equalization property of Gaussian matrix  $\mathbf{S}$ .