SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset	LPLR	DSVD	LSVD
CIFAR-10	71 ± 11 ms	306 ± 26 ms	312 ± 8 ms
CIFAR-100	14 ± 3 ms	51 ± 3 ms	56 ± 3 ms
(b) Naive (0.312)
(c) DSVD (0.507)
(d) LPLR (0.264)
(e) LSVD (0.260)
Figure 5: Compression of Shepp-Logan phantom (a standard test image for medical image reconstruction, courtesy: Phantomator, PyPI).  $\mathrm{B} = 4$ ,  $\mathrm{B}' = 8$ ,  $\mathrm{B}_{\mathrm{np}} = 2$ ,  $m = 166$ . Orig. image dim.:  $1000 \times 1000$
(a) Original image
(b) Naive (0.508)
(c) DSVD (0.508)
(d) LPLR (0.311)
(e) LSVD (0.301)
Figure 6: Compression of Shepp-Logan phantom (a standard test image for medical image reconstruction, courtesy: Phantomator, PyPI).  $\mathrm{B} = 4$ ,  $\mathrm{B}' = 8$ ,  $\mathrm{B}_{\mathrm{np}} = 1$ ,  $m = 83$ . Orig. image dim.:  $1000 \times 1000$
(a) Original image
(b) Naive (0.701)
(c) DSVD (0.724)
(d) LPLR (0.346)
(e) LSVD (0.351)
Figure 7: Compression of a Jupiter image showing its Great Red Spot and Ganymede's shadow (courtesy: NASA/ESA Hubble Space Telescope).  $\mathrm{B} = 2, \mathrm{~B}' = 8, \mathrm{~B}_{\mathrm{rq}} = 1, m = 110$ . Orig. image dim.:  $1102 \times 1102$
(a) Original image
Figure 8: Compression of an MR image of the human brain (courtesy: User Asnaebsa, Wikipedia). B = 4, B' = 8, B_{\mathrm{sq}} = 1, m = 124 maintains parity. Orig. image dim.: 1534 × 1433
(b) Naive (0.711)
(c) DSVD (0.556)
(d) LPLR (0.329)
(e) LSVD (0.321)
K Limitations and further discussions
On uniformly dithered quantizer: We would like to point out that the approximation error upper bound for our algorithm (derived in Thm. G.2), as well as for the baselines such as direct-SVD quantization (refer to Prop. H.1) holds true only for randomized rounding, or uniformly dithered quantizers. Dithering is preferred due to its ability to produce an unbiased estimate of the input. It offers an advantage by introducing a non-zero probability of quantizing an input to either its ceiling or floor, resulting in reduced variance when averaging multiple independent realizations. Moreover, the unbiasedness of the quantizer output simplifies the analysis. In our experiments, we did not observe any difference while using deterministic rounding instead of dithered rounding.
Comparison with LPLR with direct-SVD and LPLR-SVD: As we see from the numerical evaluations (ref. to Tab. 6 and Tab. 7), direct-SVD quant. and LPLR-SVD performs better than our proposed LPLR algorithm (that computes a Gaussian sketch). Intuitively, one would expect the performance of LPLR-SVD and LPLR (with Gaussian sketch) to be similar because the core idea behind both of them is the same - Quantize the first low-rank factor, and then project the columns of A onto the quantized (first) low-rank factor. For LPLR-SVD the basis being quantized (top-k singular vectors) is the exact top-k rangefinder (for the column space of A), whereas in LPLR, we sketch using the Gaussian matrix to get an approximate rangefinder. This sketching error might make LPLR worse over LPLR-SVD. But because sketching also introduces "equalization effect of the Gaussian matrices", we can improve the resolution of the uniform scalar quantizers by choosing a smaller dynamic range. Hence, there is a tradeoff when comparing LPLR and LPLR-SVD - a tradeoff between sketching error and the quantization error.
Let us now focus on LPLR-SVD and direct-SVD quant. We will extrapolate this intuition to a comparison between LPLR and direct-SVD quant. The factorization obtained using direct-SVD quant. is  $\mathbf{A} \approx \mathrm{Q}(\widetilde{\mathbf{U}}_k)\mathrm{Q}'(\mathbf{V}_k^\top)$ , where the approximation obtained using LPLR-SVD is  $\mathbf{A} \approx \mathrm{Q}(\widetilde{\mathbf{U}}_k)\mathrm{Q}'\left(\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger\mathbf{A}\right)$ . Here,  $\widetilde{\mathbf{U}}_k$  is the  $n \times k$  matrix obtained using the first  $k$  columns of  $\mathbf{U}\boldsymbol{\Sigma}$ , where  $\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$  is the full SVD of  $\mathbf{A}$ . Let us assume  $\mathrm{B}' \to \infty$  so that we don't have to worry about the second quantizer. Then, we are left to compare the approximations:  $\mathbf{A} \approx \mathrm{Q}(\widetilde{\mathbf{U}}_k)\mathbf{V}_k^\top$  and  $\mathbf{A} \approx \mathrm{Q}(\widetilde{\mathbf{U}}_k)\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger\mathbf{A}$ . Clearly, the latter has smaller Frobenius norm error because it is the minima of the minimization problem  $\arg \min_{\mathbf{W}}\left\|\mathrm{Q}(\widetilde{\mathbf{U}}_k)\mathbf{W}-\mathbf{A}\right\|_{\mathrm{F}}^2$ . So, for  $\mathrm{B}' \to \infty$ , it is evident that direct-SVD can only perform as good as LPLR-SVD, and not better than that. Furthermore, note that this difference in performance will be small when the bit-budget of the first quantizer, B is large, and it will be large when B is small. When  $\mathrm{B} \to \infty$ , this difference will be exactly zero. In other words, direct-SVD quant. and LPLR-SVD will be the same factorization when  $\mathrm{B} \to \infty$  and  $\mathrm{B}' \to \infty$ . When both B and B' are finite, direct-SVD can outperform LPLR-SVD depending on which one out of  $\mathrm{Q}'\left(\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger\mathbf{A}\right)$  and  $\mathrm{Q}'(\mathbf{V}_k^\top)$  is closer to the optimal solution, i.e.,  $\mathrm{Q}(\widetilde{\mathbf{U}}_k)^\dagger\mathbf{A}$ . A limitation of our analysis is that we cannot always definitively predict which one will perform better, because we derive upper bounds on the approximation error.
Comparison of LPLR with naive quantization: In Tables 1 and 2, we have compared LPLR with other benchmarks. In the column denoting approximation errors, aside from the common quantization error  $\epsilon$  present across all rows, DSVD and LPLR introduce an additional term representing the error from low-rank approximation. Consequently, it might appear that naive quantization consistently outperforms these methods. However, it's essential to recognize that compression techniques based on low-rank factorization hold value exclusively when the matrix being compressed is inherently low-rank, meaning that  $\| \mathbf{A}_k - \mathbf{A}\|_{\mathrm{F}}^2$  starts off as small. If  $\| \mathbf{A}_k - \mathbf{A}\|_{\mathrm{F}} = 0$ , LPLR can achieve identical error levels as naive quantization, while demanding fewer bits than the latter. In other words, given the same bit-budget, LPLR can achieve a lower level of error compared to naive quantization.
Matrix compression with and without any computation constraints: In situations where computational resources are limited, employing the naive strategy emerges as the most cost-effective approach for matrix quantization. Nevertheless, due to its failure to capitalize on a matrix's inherent low-rank arrangement, naive quantization may prove considerably suboptimal. As matrices increase in dimension, accommodating them in memory becomes impractical, rendering approaches like DSVD or LSVD unviable. In such scenarios, our LPLR method stands as a viable alternative,
demanding slightly more computational effort than naive quantization, yet capable of harnessing the low-rank structure for improved approximation accuracy.
If there is no scarcity of computation resources in being able to compute the SVD, and our goal is to just compress an input matrix  $\mathbf{A}$ , then it is possible to compress the matrix with all the strategies, and choose the best one. However, it must be kept in mind that LPLR provides significant savings with respect to computational complexity, i.e.,  $O(ndm)$ , compared to LPLR-SVD, which has a complexity of  $O(nd^2)$ . Consequently, if the matrix being compressed is very large, it might not even be feasible to do direct-SVD quant. or LPLR-SVD, given the current memory limitations of available GPUs, and LPLR (with Gaussian sketch) becomes our only option.
Furthermore, a careful examination of Tabs. 6 and 7 shows that even though LPLR-SVD and directSVD quant. have a smaller Frobenius norm error than LPLR, the accuracy and weighted F1 scores are still comparable. One aspect of quantization that we have not studied in detail is its inherent regularization effect when testing the performance on test datasets. Whether LPLR leads to a better regularization effect that the other baselines is an open question that will be studied in future works.
Comparison with existing works on quantized random projections for approximate nearest neighbor search: Prior works of Li and Li [31] and Li and Li [32] consider the idea of quantized random projections specifically for the application of approximate nearest neighbor search. Although related, the goals of these works are different from matrix compression considered in our work. These prior works approximate the matrix vector product  $\mathbf{A}\mathbf{x}$  by  $\mathrm{Q}(\mathbf{AS})\mathrm{Q}'(\mathbf{S}^\top \mathbf{x})$ . For their setup, in addition to  $\mathrm{Q}(\mathbf{AS})$ , the random matrix  $\mathbf{S}$  needs to be stored in full-precision, in order to process an incoming query  $\mathbf{x}$ . So, their storage requirement is  $nm$  quantized and  $dm$  full-precision parameters. In contrast, LPLR stores only the quantized entries of  $\mathbf{L}$  and  $\mathbf{R}$ , i.e.,  $m(n + d)$  quantized entries, and  $\mathbf{A} \approx \mathbf{LR}\mathbf{x}$ , implying a smaller storage requirement. Furthermore, since LPLR involves computation of  $\mathbf{R}\mathbf{x}$ , where  $\mathbf{R}$  consists of  $md$  quantized values, it can leverage modern advancements in hardware primitives for speeding up low-precision computations, such as half and mixed-precision compute.
Extension to streaming settings: LPLR as presented in this paper, is used for compression when the entire matrix  $\mathbf{A}$  is available. An interesting extension is to consider a variant of LPLR to streaming data settings in order to handle new data points. It is possible to do so using sketching based low rank approximations [68]. The left factor  $\mathbf{L}$  is updated with the concatenation of an additional row corresponding to the newly arriving datapoint. The second factor  $\mathbf{R}$ , which is the solution of a new least squares minimization problem, can be updated using Woodbury matrix inversion lemma.