SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the other hand, the input to the second quantizer is  $\mathrm{Q}(\mathbf{AS})^{\dagger}\mathbf{A}$  and Lemma E.2 provides a concentration result for the max-norm of this matrix. Although in a general worst-case scenario, the coordinate values of the pseudo-inverse of a matrix with small entries can be large (Alon and Vu [6]), because we compute the pseudo-inverse of the matrix  $\mathbf{AS}$ , which is rectangular ( $m \ll n$ ) and with random Gaussian entries,  $\| (\mathbf{AS})^\dagger \|_2$  does not shoot up arbitrarily as shown by Rudelson and Vershynin [53]. We then show that  $\| \mathrm{Q}(\mathbf{AS})^\dagger \|_2$  is not too far from  $\| (\mathbf{AS})^\dagger \|_2$ , allowing us to derive an expression for  $\mathrm{R}_{\mathrm{Q'}}$ . We get for  $\gamma = \frac{d}{m}$  and  $t = \sqrt{\frac{2\log\left(\frac{32nR^2}{\epsilon}\right)}{m}}$ ,
$$
\left\| \mathrm {Q} (\mathbf {A S}) ^ {\dagger} \mathbf {A} \right\| _ {\max } \leq \frac {2 \kappa}{\sqrt {\gamma} - 1 - t} \text {w i t h p r o b a b i l i t y e x c e e d i n g} 1 - \frac {\epsilon}{4 n \mathrm {R} ^ {2}}.
$$
The second part of the analysis deals with upper bounding the approximation error when the second low-rank factor is unquantized, i.e.,  $\| Q(\mathbf{AS})Q(\mathbf{AS})^\dagger\mathbf{A} - \mathbf{A}\|_{\mathrm{F}}^2$  conditioned on the event that  $Q$  and  $Q'$  are unsaturated. We reduce this problem to analyzing the solution of the following:
$$
\widetilde {\mathbf {X}} = \underset {\mathbf {X}} {\arg \min } \left\| \mathbf {X} ^ {\top} \mathbf {A} _ {k} \mathbf {S} - \mathrm {Q} (\mathbf {A} \mathbf {S}) \right\| _ {\mathrm {F}} ^ {2}. \tag {4}
$$
Table 1: Comparison with baselines (row-norm bound is constant, i.e.,  $\| \mathbf{a}^{(i)}\| = \mathrm{O}(1)$ ).  $k,m\ll \min \{d,n\}$ .  $n$ : no. of rows,  $d$ : no. of columns,  $m$ : sketch size,  $\epsilon$ : error tolerance,  $\delta = k / (m - k - 1)$ . The expressions for bit-budget (per entry) ignores constant multiplicative factors inside the  $\log_2(\cdot)$ . We assume  $n\geq d$ .
Algorithms	Approximation error	Bit-budget (per entry)	Computation
Naïve uniform	ε	1/2 log2(nd/ε)	O(nd)
Direct-SVD	||Ak-A||2F+ε	1/2 log2(ks2/ε√nd)	O(nd2)
LPLR (ours)	(1+δ)||Ak-A||2F+ε	1/2 log2(κ(Ak)/εnm/√d√log(mn2/ε))	O(ndm)
Table 2: Comparison with baselines (individual entries of  $\mathbf{A}$  are bounded by a constant, i.e.,  $A_{ij} = \mathrm{O}(1)$ ). Dimension dependent terms are color highlighted for ease of comparison with Tab. 1.
Algorithms	Approximation error	Bit-budget (per entry)	Computation
Naïve uniform	ε	1/2 log2(nd/ε)	O(nd)
Direct-SVD	||Ak-A||2F+ε	1/2 log2(kσ12/ε√nd)	O(nd2)
LPLR (ours)	(1+δ)||Ak-A||2F+ε	1/2 log2(κ(Ak)/εnm√d log(mn2d2/ε))	O(ndm)
We refer to (4) as the sketched least squares problem with quantized response as it is a variant of the generalized least squares problem,  $\mathbf{X}^{*} = \arg \min_{\mathbf{X}}\left\| \mathbf{X}^{\top}\mathbf{A}_{k} - \mathbf{A}\right\|_{\mathrm{F}}^{2}$ . This is potentially a problem of independent interest, and we analyze the solution of (4) in detail in App. F. Exploiting the subspace embedding property of  $\mathbf{S}$  we show that
$$
\left\| \mathbf {X} ^ {* ^ {\top}} \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \mathbb {E} \left\| \widetilde {\mathbf {X}} ^ {\top} \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} \leq \frac {m - 1}{m - k - 1} \left\| \mathbf {X} ^ {* ^ {\top}} \mathbf {A} _ {k} - \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} + \text {q u a n t i z a t i o n e r r o r t e r m}.
$$
This leads us to the proof of Lemma G.1 which gives the approximation error of LPLR when Q and  $Q^{\prime}$  are unsaturated. Finally, taking into account the low-probability saturation events, for which the error is  $\| \mathbf{A}\|_{\mathrm{F}}^{2}$ , we derive our main result in Thm. G.2. Subsequently, we discuss the approximation made in App. G.2 and arrive at the informal result of Thm. 3.2.
3.2 Comparison with baselines
We are now in a position to compare the performance with baselines in Tabs. 1 and 2.
Naive quantization: The most straightforward baseline for matrix quantization is naive quantization where each coordinate of the matrix is quantized independently, agnostic to any low-rank structure in the matrix  $\mathbf{A}$ . In this, we allocate B bits to each coordinate of  $\mathbf{A}$  and since there are  $nd$  entries in the matrix, from (1), the Frobenius norm error is upper bounded by  $\frac{\mathrm{R}^2nd}{(2^{\mathrm{B}} - 1)^2}$ . Note that this holds true irrespective of whether  $\mathrm{R} = \mathrm{O}(1)$  or  $\mathrm{O}(\sqrt{d})$  because  $\| \mathbf{a}^{(i)}\| \leq \mathrm{R}$  implies  $A_{ij} \leq \mathrm{R}$ . To ensure that the error is within a certain tolerance  $\epsilon$ , we then require  $\frac{1}{2}\log_2\left(\frac{nd}{\epsilon}\right)$  bits. In this, and also other expressions for bit-budget requirements of algorithms in Tables 1 and 2, we have ignored the multiplicative constant factors inside the  $\log_2(\cdot)$  for simplicity of exposition. The exact expressions can be found in the corresponding appendices where we derive them.
One of the primary reasons why both direct-SVD and LPLR are expected to perform better than naive is that the former strategies exploit the low-rank structure of the matrices to reduce the total number of parameters being quantized, i.e.,  $k(n + d)$  for direct-SVD and  $m(n + d)$  for LPLR, vs.  $nd$  for naive. Given a total bit-budget for the entire matrix  $\mathbf{A}$ , since we now quantize fewer parameters than before, we can allocate a higher number of bits to each parameter, enabling higher precision. The price we pay for exploiting the low-rank structure of matrices is the additional  $\| \mathbf{A}_k - \bar{\mathbf{A}} \|_{\mathrm{F}}^2$  dependent term, which is usually very small for matrices that can be well approximated by a low-rank structure. For matrices that are exactly rank  $k$ , this term is 0. As we see in our numerical simulations in §4, several real-world matrices can be well-approximated by a low-rank structure.
Direct-SVD quant.: From Tab. 1, we see that to achieve an  $\epsilon$ -quantization error, direct-SVD requires  $\frac{1}{2}\log_2(k\sqrt{nd})$  bits per entry, which is greater than  $\frac{1}{2}\log_2\left(\frac{nm}{\sqrt{d}}\right)$  required by LPLR (ignoring the
logarithmic terms). Evidently, LPLR demands fewer bits than direct-SVD because  $k, m \ll \min\{n, d\}$  for inherently low-rank matrices. For the regime presented in Tab. 2, the bit requirement for direct-SVD remains unchanged. However, LPLR now requires  $\frac{1}{2} \log_2 \left( nm \sqrt{d} \right)$ , slightly more than direct-SVD, due to the additional  $\sqrt{n}$  factor. Thus, it makes sense to expect that direct-SVD can perform better in this regime. This is supported by our numerical experiments in Tables 4 to 7, where direct-SVD indeed outperforms LPLR in certain scenarios. Nevertheless, it is crucial to emphasize that direct-SVD necessitates computing the SVD, which can be prohibitive for very large matrices due to the current memory limitations of available GPUs, making LPLR the only viable option.
Computational complexity: Unsurprisingly, naive quant. requires the least computation, i.e.,  $\mathrm{O}(nd)$ , as it just does a single pass over all the elements of  $\mathbf{A}$ . The  $\mathrm{O}(nd^2)$  complexity of direct-SVD quant. stems from the requirement of computing SVD (assuming  $n \geq d$ ). LPLR is the best of both worlds – for the same bit-budget, LPLR has a smaller approximation error than both direct-SVD and naive, and a complexity of  $\mathrm{O}(ndm)$ , arising from the requirement to compute AS, i.e., a product of two dense matrices of dimensions  $n \times d$  and  $d \times m$ , which is better than direct-SVD, since  $m \ll d$ .
We evaluate the robustness of LPLR on multiple tasks, namely, image compression, binary, and multi-class classification across disparate domains including vision, text and raw images, and neural network weight matrices. We consider a range of input configurations to showcase the performance and non linear effects of joint quantization and low rank approximation on a given dataset, especially at lower bit budgets. LPLR provides competitive results at bit budgets as low as a single bit, providing extreme model compression while maintaining non trivial performance for the task at hand.
Baselines. We employ naive quantization, which quantizes the input matrix by rounding to the nearest scalar in the underlying data type's quantization grid, as our primary benchmark. Naive quant. and its variants - Dettmers et al. [14], Yao et al. [81], are the most popular method in use across domains, as their memory and computational run time requirements scale extremely well with model and dataset sizes. In addition, we also evaluate the performances of direct-SVD quantization and LPLR-SVD to disambiguate between the entangled effects of quantization and exploiting low rank structure.
Metrics. We evaluate LPLR performance using task specific goodness of fit metrics, as well as relative Frobenius norm error between the original and matrix reconstructed from its low-rank factors, i.e.,  $\| \mathbf{LR} - \mathbf{A}\|_{\mathrm{F}}^2$ . In addition to this, we enforce parity between the number of bits used by all quantization schemes, so that the total space required (in bits) for storing the approximated matrix is identical across LPLR, LPLR-SVD, direct-SVD quant., and naive quant.
Notation. In all our experiments, we denote the bit budget for the left low rank factor  $\mathbf{L}$  as  $\mathrm{B}$ , for the right low rank factor  $\mathbf{R}$  as  $\mathrm{B}'$ , and the corresponding bit budget for naive quantization as  $\mathrm{B}_{\mathrm{sq}}$ . For simplicity, we maintain equal bit budgets  $\mathrm{B} = \mathrm{B}'$  for both quantizers  $\mathrm{Q}$  and  $\mathrm{Q}'$ . Wherever necessary, we also abbreviate direct-SVD quant. as DSVD, LPLR-SVD as LSVD, and naive quant. as NQ.
The main algorithm is implemented in Pytorch (Paszke et al. [46]), and utilizes Hugging Face [80] implementations of all datasets and large language models. All experiments were performed on a single GPU NVIDIA TITAN RTX. Further simulations and experimental details can be found in App. J. Our code is available at https://github.com/pilancilab/matrix-compressor.
4.2 Image Compression
Image compression is a prototypical application of low rank matrix compression, as images are known to be significantly rank deficient in many practical scenarios (Zhang et al. [86], Zhou et al. [87]). In this task, we apply LPLR on  $1000 \times 1000$  dimensional Shepp Logan phantom images from Gach et al. [20]. These are a set of synthetic 2D images designed to simulate the typical characteristics and structures found in computed tomography (CT) scans. They consist of geometric shapes, including circles and ellipses, representing different tissues or organs within the scanned object.
The main results are summarized in Tab. 3. To ensure a fair comparison, we adjust the sketch size/target rank so that bit budgets are identical between naive quant. and LPLR. This allows
us to preserve the original datatype of the image (consequently a large dynamic range), while substantially reducing the pixels used for representing the image to as low as 1 bit per pixel (on average). Specifically, in Figure 1, we can observe the least visual distortion in the case of LPLR, which preserves critical semantic features of the images, such as the small ellipses. It is clear that LPLR outperforms both techniques at lower naive quantization bit budgets. We attribute the better visual and quantitative performance to the higher dynamic range available to LPLR as well as structure preserved in the low rank decomposition.
Table 3: Comparison of LPLR and LPLR-SVD (LSVD) Frobenius norm errors with baselines, for different input LPLR bit budgets. Each triplet  $(\mathrm{B},\mathrm{B}^{\prime},\mathrm{B}_{\mathrm{sq}})$  of configurations has an identical compression ratio. Here,  $\mathrm{B} = \mathrm{B}^{\prime}$ . The second column specifies the sketch size  $m$  for LPLR, and target rank  $k$  for DSVD or LSVD. We provide results for input bit budgets at a finer granularity to identify regimes where naive quant. is outperformed.
B	Target Rank (k) / Sketch Size (m)	\( B_{\text{nq}} \)	LPLR	DSVD	LSVD	NQ
32	15	1	0.610	0.553	0.506	0.532
28	17	1	0.557	0.546	0.490	0.532
24	20	1	0.540	0.537	0.454	0.532
20	25	1	0.485	0.529	0.426	0.532
16	31	1	0.447	0.523	0.391	0.532
12	41	1	0.402	0.518	0.360	0.532
8	62	1	0.340	0.508	0.326	0.532
B	Target Rank (k) / Sketch Size (m)	\( {\mathrm{B}}_{\mathrm{{nq}}} \)	LPLR	DSVD	LSVD	NQ
32	31	2	0.447	0.523	0.392	0.312
28	35	2	0.434	0.521	0.380	0.312
24	41	2	0.401	0.517	0.358	0.312
20	50	2	0.371	0.513	0.331	0.312
16	62	2	0.341	0.509	0.308	0.312
12	83	2	0.310	0.506	0.286	0.312
8	125	2	0.267	0.499	0.284	0.312
4.3 Embeddings extracted from pre-trained models
The efficacy of pre-trained embeddings is well established in vision (Li et al. [33], Parisi et al. [45]), text (Qi et al. [51], Rezaeinia et al. [52]) for rapid feature computation as an input to a variety of downstream tasks. Embeddings also play a crucial role in a number of software applications, including but not limited to, open source vector search libraries (Liu [35], Marqo [41]), semantic search engines (Amazon AWS [7]), vector databases (Pinecone [49]). Since most applications rely on proximity in "embedded space", it is essential that common operations on embeddings be computationally efficient. Specifically, one would like to optimize nearest neighbor (NN) searches which solve the optimization problem  $\arg \min_{i} \| \mathbf{x}_{i} - \mathbf{y}\|_{2}^{2}$  (reducible to  $\arg \max_{i}\mathbf{X}\mathbf{y}$  where  $\mathbf{X}$  is the matrix with training vectors  $\{\mathbf{x}_i\}$  as its rows, and  $\mathbf{y}$  is the query vector). The time complexity of NN search scales linearly with dimensions of  $\mathbf{X}\in \mathbb{R}^{n\times d}$  and number of neighbors  $(k)$ . By embedding the data matrix in a dimension  $m\ll d$ , we directly speedup the run-time and reduce storage costs. Moreover, as datasets grow exponentially in size (especially document databases) and transfer learning becomes the dominant modality of training new models, embedding compression becomes a necessity for storing data without a corresponding exponential increase in hardware requirements.
4.3.1 Embedding Classification
In this experiment, we evaluate several embeddings of standard datasets, namely CIFAR-10, CIFAR-100, IMDB and Emotion datasets. CIFAR-10 consists of 60,000 color images divided into 10 classes, with each class containing 6,000 images. The dataset is split into 50,000 training images and 10,000 test images, with a resolution of  $32 \times 32$  pixels. CIFAR-100 increases the number of classes to 100 categories for an identical training and test size as CIFAR-10. The IMDB (mte [2]) dataset consists of 25,000 train and test sentences containing annotated binary sentiment labels for movie reviews, plot summaries and other rating information. The Emotion (mte [1]) dataset is a sentiment analysis dataset, containing 16,000 train and 2000 test sentences, each exemplifying a singular emotion, which represents the sentiment label for that sentence.
For CIFAR-10 and CIFAR-100, we embed the entire dataset using MobileNet v3 (Howard et al. [24]) pretrained on ImageNet (Deng et al. [12]) producing an embedding matrix of dimension  $60000 \times 1024$ , which we compress using LPLR and compare with the baselines in §4.1. To evaluate the goodness of embeddings, we build a 3-NN, a KNN Classifier using  $K = 3$  nearest neighbors under Euclidean distance). We report the performance of the model using standard classification metrics - classification accuracy and weight averaged F1 score. We utilize a uniform bit budget  $\mathrm{B} = \mathrm{B}' = 8$  bits for the quantizers Q,  $\mathrm{Q}'$  across all cases. Tabs. 4 and 5 present our results under this setup. For each case we benchmark absolute performance using a 3-NN classifier on the training set.
Similarly, we embed text sentences from IMDB and Emotion databases using BeRT (Devlin et al. [15]) into 512 dimensional vectors, and construct a 3-NN classifier using Euclidean distance to perform binary and multi-class classification on the respective embeddings, and report classification metrics in Tabs. 6 and 7. We see that LPLR outperforms direct-SVD quant. and naive quant. at lower bit budgets, and has performance parity as we increase  $\mathrm{B_{nq}}$  to 4 bits. We find that we match (and even exceed) the unquantized benchmark at single bit precision, which we attribute to the dominating low rank factorization, and its regularizing effect on data under extreme rank constraints. It is important to note that performance parity with direct-SVD quant. is also a successful outcome, since LPLR provides runtime improvements over taking an SVD to compress the data.
Table 4: CIFAR10 embeddings generated by MobileNetV3 with an unquantized accuracy and F1 score  $91\%$  :Results on LPLR and LPLR-SVD with  $\mathrm{B} = \mathrm{B}' = 8$  bits
Frobenius Norm Error	Accuracy (%)	Weighted F1 Score (%)
Bnq	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ
1	1.05	1.08	1.09	7.17	92	92	92	11	92	92	92	4
2	1.08	1.1	1.1	2.29	92	92	91	30	92	92	91	23
4	1.1	1.11	1.11	1.15	91	92	91	91	91	92	91	91
Table 5: CIFAR100 embeddings generated by MobileNetV3 with an unquantized accuracy and F1 score  $76\%$  :Results on LPLR and LPLR-SVD with  $\mathrm{B} = \mathrm{B}' = 8$  bits
Frobenius Norm Error	Accuracy (%)	Weighted F1 Score (%)
Bnq	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ
1	1.04	1.08	1.09	6.75	79	82	82	1	79	82	82	0
2	1.08	1.1	1.12	2.18	80	80	80	1.7	80	80	80	1.3
4	1.11	1.12	1.14	1.17	79	78	77	75	79	78	78	75
Table 6: IMDB embeddings generated by BERT with an unquantized accuracy and F1 score 75% and 74% respectively: Results on LPLR and LPLR-SVD with  $\mathrm{B} = {\mathrm{B}}^{\prime } = 8$  bits
Frobenius Norm Error	Accuracy (%)	Weighted F1 Score (%)
Bnq	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ
1	0.313	0.241	0.229	6.63	73	74	75	50	74	74	75	33
2	0.235	0.178	0.161	1.016	74	74	74	50	74	74	74	50
4	0.148	0.122	0.098	0.417	75	74	75	73	74	74	75	73
Table 7: Emotion embeddings generated by BERT with an unquantized accuracy and F1 score  $43\%$  and  $40\%$  respectively: Results on LPLR and LPLR-SVD with  $\mathrm{B} = \mathrm{B}' = 8$  bits
Frobenius Norm Error	Accuracy (%)	Weighted F1 Score (%)
Bnq	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ	LPLR	LSVD	DSVD	NQ
1	0.383	0.296	0.286	5.109	41	43	43	29	38	40	40	13
2	0.291	0.215	0.202	1.561	42	43	43	33	39	40	40	30
4	0.187	0.137	0.121	0.367	42	43	43	41	39	40	40	38
4.3.2 Compressing Weight Matrices of a Large Language Model
In this section, we present results on a major application of matrix compression – compressing the weight matrices of deep neural networks. We choose LlaMa by Touvron et al. [66], a popular foundation Large Language Model (LLM) as the network of our choice. LLMs are a natural candidate for matrix compression, due to their massive stacked transformer layers, rendering them difficult to deploy on several GPUs, let alone a single GPU. Many methods have emerged to quantize and compress these models in order to make them amenable to single GPU deployment and inference, including naive quantization with outlier exclusion (Dettmers et al. [14]), second order methods (Frantar et al. [19]), low rank parameter reduction (Hu et al. [25]), amongst others.
We apply LPLR to the 2-dimensional weight tensors, i.e., matrices in LlaMa, leaving any other tensor, which does not lend itself to a low rank decomposition, unquantized. Figs. 2a and 2b (better
$^{1}$ LPLR-SVD used in the simulations computes the left low-rank factor  $\mathbf{L}$  as  $\mathrm{Q}(\mathbf{U}_k\mathbf{S})$  where  $S_{ij} \sim \mathcal{N}(0, 1/m)$  instead of simply  $\mathbf{U}_k$  in order to exploit the equalization property of Gaussian embeddings.
(a)  $\mathrm{B} = \mathrm{B}' = 8$  bits,  $\mathrm{B_{nq}} = 4$  bits
(b)  $\mathrm{B} = \mathrm{B}' = \mathrm{B}_{\mathrm{sq}} = 4$  bits
Figure 2: Comparison of LPLR and LPLR SVD on LlaMa weights, ordered by the original sequence of layers
Table 8: Average relative Frobenius norm error on LlaMa weight matrices
B = B' = 8 bits, Bnq = 4 bits
Metric	LPLR	LPLR-SVD	Naive Quant.
Mean	0.672	0.537	0.836
Std Dev	0.080	0.079	0.470
B = B' = Bnq = 4 bits
Metric	LPLR	LPLR-SVD	Naive Quant.
Mean	0.548	0.540	0.836
Std Dev	0.053	0.055	0.470
resolution in App. J) showcase our results on applying LPLR and LPLR-SVD with bit budgets of 8 bits and 4 bits respectively, using relative Frobenius norm error as the metric. While it is clear that LPLR and LPLR-SVD perform significantly better across all layers (on average), there are outliers where naive quant. is the better choice. We can a observe periodic structure in the error profile of naive quant., implying that the low rank structure is a function of the index of attention layer in transformer blocks. It is important to note that a low Frobenius norm error is not a direct indicator of performance for other task specific metrics. It is possible to construct a holistic compression strategy using error profiles similar to Figs. 3 and 4 to adopt a per-layer quantization strategy, minimizing both task specific metrics as well as relative Frobenius norm error. We discuss this further in Appendix K.
In this work, we have considered the problem of obtaining a low-precision and low-rank factorization of a matrix. Such a factorization of a matrix into a product of tall and wide matrices has several advantages, including compression of the original matrix. We propose a fast randomized algorithm to obtain this factorization which requires  $\mathrm{O}(nmd)$  computations – considerably faster than alternative methods. Our algorithm employs a Gaussian sketch to estimate the range space of matrices that are approximately low-rank. By utilizing the properties of subspace approximation and equalization in Gaussian embeddings, we establish an upper bound on the approximation error attained by our algorithm, and show that it can be significantly smaller than its counterparts. Finally, we empirically evaluate our method on several vision and text datasets, where we show significant task performance at highly compressed bit budgets as low as a single bit. This provides a novel pragmatic approach to work with large datasets and models in real world settings, making them more accessible to researchers and deployment on regular consumer hardware.
This work was supported in part by the Air Force Office of Scientific Research (AFOSR) under Award #002484665; in part by National Science Foundation (NSF) CAREER Award under Grant CCF-2236829, Grant DMS-2134248 and Grant ECCS-2037304; in part by the U.S. Army Research Office Early Career Award under Grant W911NF-21-1-0242; in part by the Stanford Precourt Institute; and in part by the ACCESS AI Chip Center for Emerging Smart Systems through InnoHK, Hong Kong, SAR. The authors would also like to thank the anonymous reviewers whose comments and suggestions helped improve the presentation of this work.
1.1 Related Works 2
2 Proposed Algorithm 3
2.1 Uniformly dithered quantizer 3
2.2 Proposed algorithm: LPLR 3
3 Approximation Error Analysis 4
3.1 Analysis outline 5
3.2 Comparison with baselines 6
4 Numerical Simulations 7
4.1 Overview 7
4.2 Image Compression 7
4.3 Embeddings extracted from pre-trained models 8
4.3.1 Embedding Classification 8
4.3.2 Compressing Weight Matrices of a Large Language Model 9
A Notations 19
B Preliminaries 20
B.1 Linear algebra inequalities 20
B.2 Probability and random matrix theory 21
B.2.1 Tail bound for Gaussian distribution 21
B.2.2 Inverse Wishart distribution 21
B.2.3 Random Gaussian matrices 21
B.2.4 Subgaussian random variables 22
C Quantization error of uniformly dithered scalar quantizer 22
D Gaussian embeddings: Application of equalization to vector quantizers 23
E Dynamic range of quantizers 25
E.1 Quantization for the first low-rank factor 25
E.2 Quantization for the second low-rank factor 25
F Sketched least squares with quantized response 28