SlowGuess/NeurIPS_2023_OCR huggingface dataset:
G LPLR algorithm: Approximation error analysis 30
G.1 LPLR approximation error: Proof of Thm. 3.2 33
G.2 Informal version of Thm. G.2 34
H Direct-SVD Quantization: Approximation error analysis 35
I LPLR-SVD algorithm: Approximation error analysis 37
I.1 Informal version of Thm. I.1 41
J Additional numerical simulations 41
J.1 Image Compression 41
J.2 Wall-clock time for compressing image embeddings 41
J.3 High-resolution figures for Llama weight compression 41
K Limitations and further discussions 44
We first gather some common notations in linear algebra and probability theory that have been used throughout the paper. Boldface upper and lowercase letters,  $\mathbf{A}$  and a denote matrices and vectors respectively.  $\mathbf{I}_d$  denotes the  $d\times d$  identity matrix. The subscript may be dropped if the dimension is clear from the context. For any matrix  $\mathbf{A}$ , its  $i^{\mathrm{th}}$  row and  $j^{\mathrm{th}}$  column are denoted by  $\mathbf{a}^{(i)}$  and  $\mathbf{a}_j$  respectively. The singular values of  $\mathbf{A}$  are denoted by  $\sigma_{\max}(\mathbf{A}) = \sigma_1\geq \sigma_2\geq \ldots \geq \sigma_r = \sigma_{\min}(\mathbf{A})$ , where  $r = \mathrm{rank}(\mathbf{A})$ . Similarly, the eigenvalues are denoted as  $\lambda_{1}(\mathbf{A}),\dots,\lambda_{r}(\mathbf{A})$ . The max-norm of  $\mathbf{A}$  is defined as  $\| A\|_{\max} = \max_{i,j}|A_{ij}|$ , the spectral norm of  $\mathbf{A}$  is defined as  $\| \mathbf{A}\| _2 = \sup_{\| \mathbf{x}\| = 1}\| \mathbf{Ax}\| = \sigma_{\max}(\mathbf{A})$ , and the Frobenius norm is  $\| \mathbf{A}\|_{\mathrm{F}} = \left(\sum_{i,j}A_{ij}^{2}\right)^{1 / 2} = \operatorname {Tr}\left[\mathbf{A}^{\top}\mathbf{A}\right] = \left(\sum_{k\in [r]}\sigma_k^2\right)^{1 / 2}$ . For any vector  $\mathbf{x}$ ,  $\| \mathbf{x}\| = (x_i^2)^{1 / 2}$  denotes the  $\ell_2$ -norm, and  $\| \mathbf{x}\|_{\infty} = \max_{i}|x_{i}|$  denotes the  $\ell_{\infty}$ -norm. We use the notations  $\succcurlyeq$  and  $\preccurlyeq$  for the positive semi-definite (PSD) cone ordering (or the Loewner ordering) of symmetric matrices, i.e., for any symmetric matrices  $\mathbf{X}$  and  $\mathbf{Y}$ ,  $\mathbf{X}\succcurlyeq \mathbf{Y}\iff \mathbf{X} - \mathbf{Y}$  is PSD.  $\mathbf{X}^{\dagger}$  denotes the Moore-Penrose pseudo-inverse of a matrix  $\mathbf{X}$ .  $\log (\cdot)$  denotes the natural logarithm, i.e., base  $e$ .  $\log_2(\cdot)$ , i.e., with base 2 is specified explicitly. A normal distribution with mean  $\mu$  and variance  $\sigma^2$  is denoted as  $\mathcal{N}(0,\sigma^2)$ , while a multivariate normal distribution in  $\mathbb{R}^d$  is denoted as  $\mathcal{N}(\boldsymbol {\mu},\boldsymbol {\Sigma})$ .  $\mathbb{E}[\cdot ]$  denotes expectation of a random variable. The probability measure over which the expectation is taken is described in text. We also use big-'oh' notation  $\mathrm{O}(\cdot)$  that hides constants for asymptotic expressions, while  $\widetilde{\mathrm{O}} (\cdot)$  also hides terms that depends logarithmically on dimension.
We now list some notations that are either less commonly known or used specifically in this paper, along with some remarks such as their occurrence in the paper. Several of these notations have also been introduced in-context, but they are additionally collected here for easy reference.
Table 9: Notations used in this paper
Notation	Description	Remarks
A, L, R	Input matrix, left and right LPLR factors	LR is an approximation of A. Entries of L and R are represented in low-precision formats
n, d	Dimensions of input matrix	A ∈ Rn×d.
k	Target rank	Often, k << min{n, d}, but not necessarily.
A = UΣV^T	Full SVD of the matrix A ∈ Rn×d	U ∈ Rn×n, Σ ∈ Rn×d, and V ∈ Rd×d.
A_k = U_kΣ_kV_k^T	Best rank-k approximation of A	U ∈ Rn×k and V ∈ Rn×k consists of top-k left and right singular vectors respectively. Σ ∈ Rk×k consists of top-k singular values.
m	Sketch size	m = k + p, where k is the target rank, and p is the oversampling factor.
S ∈ Rd×m	Sketching matrix	In this work, Sij ~ N(0, 1/m).
Q, Q'	Quantizers for the first and second low-rank factors with bit-budgets B and B', and dynamic ranges R_Q and R_Q' respectively.
κ(A)	Condition number of input matrix A	κ(A) = σ1, σr
κ(A_k)	Condition number of best rank-k approximation A_k	κ(A_k) = σ1/σk
κ	A quantity that depends on the spec-trum of A	κ = min{κ(A), κ(A_k)/(1-O(1)σk+1/σk} , defined in Thm. G.2
γ	Aspect ratio of the sketch	γ = d/m, defined in Thm. G.2.
δ	Target rank and sketch-size depen-dent quantity	δ = km-k-1
W(Ψ, d)	Wishart distribution with covariance matrix Ψ and degree of freedom d	Refer to §B.2.2.
B.1 Linear algebra inequalities
We state (with proofs) some standard inequalities from linear algebra that will be useful in proving our main results.
Lemma B.1. (Frobenius norm of matrix products) For any matrices  $\mathbf{A}$  and  $\mathbf{B}$ , we have
$$
\| \mathbf {A B} \| _ {\mathrm {F}} \leq \| \mathbf {A} \| _ {2} \| \mathbf {B} \| _ {\mathrm {F}}.
$$
Proof. Note that:
$$
\left\| \mathbf {A B} \right\| _ {\mathrm {F}} ^ {2} = \sum_ {j} \| (\mathbf {A B}) _ {j} \| _ {2} ^ {2} \leq \sum_ {j} \| \mathbf {A B} _ {j} \| _ {2} ^ {2} \stackrel {(i)} {\leq} \| \mathbf {A} \| _ {2} ^ {2} \sum_ {j} \| \mathbf {B} _ {j} \| _ {2} ^ {2} = \| \mathbf {A} \| _ {2} ^ {2} \| \mathbf {B} \| _ {\mathrm {F}} ^ {2}. \tag {5}
$$
Here, (i) follows from the definition of spectral norm of a matrix, and this completes the proof.
Lemma B.2. (Loewner ordering for matrix products) For any matrix  $\mathbf{A}$  and  $\mathbf{B}$ , we have
$$
\sigma_ {\min} ^ {2} (\mathbf {A}) \mathbf {B} ^ {\top} \mathbf {B} \preccurlyeq \mathbf {B} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A} \mathbf {B} \preccurlyeq \sigma_ {\max} ^ {2} (\mathbf {A}) \mathbf {B} ^ {\top} \mathbf {B}.
$$
Proof. For any vector  $\mathbf{x} \neq \mathbf{0}$ , we have,
$$
\mathbf {x} ^ {\top} \mathbf {B} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A B x} = \| \mathbf {A B x} \| _ {2} ^ {2} \geq \sigma_ {\min } ^ {2} (\mathbf {A}) \| \mathbf {B x} \| _ {2} ^ {2} = \mathbf {x} ^ {\top} \left(\sigma_ {\min } ^ {2} (\mathbf {A}) \mathbf {B} ^ {\top} \mathbf {B}\right) \mathbf {x}. \tag {6}
$$
The other direction holds similarly. This completes the proof.
Lemma B.3. (Max-norm spectral-norm inequality) For any matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$ ,  $\| \mathbf{A} \|_{\max} \leq \| \mathbf{A} \|_2$ .
Proof. Let  $\mathbf{e}_i\in \mathbb{R}^n$  and  $\widehat{\mathbf{e}}_j\in \mathbb{R}^d$  denote the  $i^{\mathrm{th}}$  and  $j^{\mathrm{th}}$  canonical basis vectors in  $\mathbb{R}^n$  and  $\mathbb{R}^d$  respectively. Then, using Cauchy-Schwarz inequality, we have,
$$
\left\| \mathbf {A} \right\| _ {\max } = \max  _ {i, j} \left| \mathbf {e} _ {i} ^ {\top} \mathbf {A} \widehat {\mathbf {e}} _ {j} \right| \leq \max  _ {i, j} \left\| \mathbf {e} _ {i} \right\| \left\| \mathbf {A} \widehat {\mathbf {e}} _ {j} \right\| = \max  _ {j} \left\| \mathbf {A} \widehat {\mathbf {e}} _ {j} \right\| \leq \sup  _ {\| \mathbf {x} \| = 1} \| \mathbf {A x} \| = \| \mathbf {A} \| _ {2}, \tag {7}
$$
Lemma B.4. (Submultiplicativity of spectral norm) For any two matrices  $\mathbf{A}$  and  $\mathbf{B}$ , we have
$$
\left\| \mathbf {A B} \right\| _ {2} \leq \left\| \mathbf {A} \right\| _ {2} \left\| \mathbf {B} \right\| _ {2}.
$$
Moreover, an analogous "reverse-submultiplicativity" result for the minimum singular value of a matrix product also holds true, i.e.,
$$
\sigma_ {\min } (\mathbf {A B}) \geq \sigma_ {\min } (\mathbf {A}) \sigma_ {\min } (\mathbf {B}).
$$
Proof. Using Lemma B.2, we have,
$$
\left\| \mathbf {A B} \right\| _ {2} ^ {2} = \lambda_ {\max } \left(\mathbf {B} ^ {\top} \mathbf {A} ^ {\top} \mathbf {A B}\right) \stackrel {(i)} {\leq} \left\| \mathbf {A} \right\| _ {2} ^ {2} \cdot \lambda_ {\max } \left(\mathbf {B} ^ {\top} \mathbf {B}\right) = \left\| \mathbf {A} \right\| _ {2} ^ {2} \left\| \mathbf {B} \right\| _ {2} ^ {2}, \tag {8}
$$
where (i) follows from Lemma B.2, completing the proof. The lower bound on  $\sigma_{\mathrm{min}}(\mathbf{AB})$  also follows a similar argument.
Lemma B.5. (Lower bound on minimum singular value of matrix sums) For any matrices  $\mathbf{A}$  and  $\mathbf{B}$ , we have
$$
\sigma_ {\min } (\mathbf {A} + \mathbf {B}) \geq \sigma_ {\min } (\mathbf {A}) - \| \mathbf {B} \| _ {2}.
$$
Proof. We have the following chain of inequalities,
$$
\begin{array}{l} \sigma_ {\min } (\mathbf {A} + \mathbf {B}) = \inf  _ {\| \mathbf {x} \| = 1} \| (\mathbf {A} + \mathbf {B}) \mathbf {x} \| \overset {\text {(i)}} {\geq} \inf  _ {\| \mathbf {x} \| = 1} \left(\| \mathbf {A x} \| - \| \mathbf {B x} \|\right) \\ \geq \inf  _ {\| \mathbf {x} \| = 1} \| \mathbf {A x} \| - \sup  _ {\| \mathbf {x} \| = 1} \| \mathbf {B x} \| = \sigma_ {\min } (\mathbf {A}) - \| \mathbf {B} \| _ {2}, \\ \end{array}
$$
where (i) is the reverse triangle inequality. This completes the proof.
Lemma B.6. (Rotation invariance of singular values) For a given matrix  $\mathbf{A}$ ,  $\sigma_i(\mathbf{A}) = \sigma_i(\mathbf{U}\mathbf{A})$  for all  $i = 1, \ldots, \mathrm{rank}(\mathbf{A})$  for any unitary matrix  $\mathbf{U}$ .
Proof. For  $i = 1, \dots, \operatorname{rank}(\mathbf{A})$ , since  $\mathbf{U}^{\top} \mathbf{U} = \mathbf{I}$ , we have
$$
\sigma_ {i} (\mathbf {U A}) = \sqrt {\lambda_ {i} \left(\mathbf {A} ^ {\top} \mathbf {U} ^ {\top} \mathbf {U A}\right)} = \sqrt {\lambda_ {i} \left(\mathbf {A} ^ {\top} \mathbf {A}\right)} = \sigma_ {i} (\mathbf {A}),
$$
Lemma B.7. (Rotation invariance of Frobenius norm) For a given matrix  $\mathbf{A}$ ,  $\| \mathbf{U} \mathbf{A} \|_{\mathrm{F}} = \| \mathbf{A} \|_{\mathrm{F}}$  for any unitary matrix  $\mathbf{U}$ .
$$
\left\| \mathbf {U} \mathbf {A} \right\| _ {\mathrm {F}} ^ {2} = \operatorname {T r} \left[ \mathbf {A} ^ {\top} \mathbf {U} ^ {\top} \mathbf {U} \mathbf {A} \right] = \operatorname {T r} \left[ \mathbf {A} ^ {\top} \mathbf {A} \right] = \left\| \mathbf {A} \right\| _ {\mathrm {F}} ^ {2}. \tag {9}
$$
B.2 Probability and random matrix theory
We restate some results from probability and random matrix theory which will be useful in deriving the main result of our paper.
B.2.1 Tail bound for Gaussian distribution
Gaussian distributions have strong concentration properties which we exploit in deriving the results of this paper. The following tail bound on a Gaussian random variable can be found in standard texts such as Wainwright [75, §2.1.2], and is restated here.
Lemma B.8. (Chernoff bound for centered Gaussian) For  $X \sim \mathcal{N}(0, \sigma^2)$ , we have,
$$
\Pr \left(| X | \geq t\right) \leq 2 e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}}. \tag {10}
$$
The proof of this follows from a direct application of Chernoff bound.
B.2.2 Inverse Wishart distribution
Consider a matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$ , each row of which is drawn independently from the distribution  $\mathcal{N}(\mathbf{0}, \boldsymbol{\Psi})$ , where  $\boldsymbol{\Psi} \in \mathbb{R}^{m \times m}$ . Then, the probability distribution of the  $m \times m$  random matrix  $\mathbf{S}^{\top} \mathbf{S}$  is called the Wishart distribution with  $d$  degrees of freedom, denoted as  $\mathcal{W}(\boldsymbol{\Psi}, d)$ . Moreover, the distribution of the matrix  $(\mathbf{S}^{\top} \mathbf{S}^{\top})^{-1}$  is called the inverse Wishart distribution and is denoted by  $\mathcal{W}^{-1}(\boldsymbol{\Psi}^{-1}, d)$ . These distributions have been studied extensively and further details can be found in Mardia et al. [40] or Siskind [59].
Lemma B.9. (Expected trace of inverse Wishart matrix) Suppose  $\mathbf{S} \in \mathbb{R}^{d \times m}$  matrix, each entry of which is drawn independently from  $\mathcal{N}\left(0, \frac{1}{m}\right)$ . Then, the matrix  $\mathbf{S}^{\top} \mathbf{S} \in \mathbb{R}^{m \times m}$  follows the Wishart distribution  $\mathcal{W}\left(\frac{1}{m} \mathbf{I}_m, d\right)$  that satisfies,
$$
\mathbb {E} \operatorname {T r} \left[ \left(\mathbf {S} ^ {\top} \mathbf {S}\right) ^ {- 1} \right] = \frac {m ^ {2}}{d - m - 1}. \tag {11}
$$
Proof. From Mardia et al. [40, eq. 3.8.3], if  $\mathbf{X} \sim \mathcal{W}^{-1}\left(\Psi^{-1}, d\right)$ , then,
$$
\mathbb {E} \left[ \left(\mathbf {S} ^ {\top} \mathbf {S}\right) ^ {- 1} \right] = \frac {\boldsymbol {\Psi} ^ {- 1}}{d - m - 1}. \tag {12}
$$
Here,  $\Psi = \frac{1}{m}\mathbf{I}_m$ , implying  $\operatorname{Tr}[\Psi^{-1}] = m^2$ , which completes the proof.
B.2.3 Random Gaussian matrices
The spectral norm of random matrices with Gaussian entries have interesting concentration properties. In this section, we present a lemma from Vershynin [73] that formally states this result.
Lemma B.10. Let the entries of matrix  $\mathbf{S} \in \mathbb{R}^{d \times m}$  be distributed according to  $S_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ . Then for every  $t \geq 0$ , with probability at least  $1 - 2e^{-\frac{mt^2}{2}}$ , we have,
$$
\sqrt {\frac {d}{m}} - 1 - t \leq \sigma_ {\min } (\mathbf {S}) \leq \sigma_ {\max } (\mathbf {S}) \leq \sqrt {\frac {d}{m}} + 1 + t. \tag {13}
$$
The above lemma is a straightforward modification of the result in Vershynin [73, Corr. 5.35], which states the concentration result when the entries are distributed according to  $\mathcal{N}(0,1)$ . Note that given  $\mathbf{S}$  with entries  $S_{ij} \sim \mathcal{N}\left(0,\frac{1}{m}\right)$  as above, the matrix  $\widehat{\mathbf{S}} = \sqrt{m}\mathbf{S}$  will have entries  $\widehat{S}_{ij} \sim \mathcal{N}(0,1)$ . Furthermore,  $\sigma_i(\widehat{\mathbf{S}}) = \sqrt{m}\sigma_i(\mathbf{S})$  and the conclusion is immediate.
B.2.4 Subgaussian random variables
Subgaussian random variables refer to a class of distributions that are dominated by the distribution of a centered Gaussian random variable. There are several equivalent ways to characterize subgaussian random variables which can be found in several textbooks (see for example, Vershynin [74, Prop. 2.5.2]). We will focus on the following definition. More formally, the distribution of a random variable  $\mathbf{X}$  is subgaussian if the moment generating function of  $X^2$  is bounded at some point, i.e.,
$$
\mathbb {E} \left[ e ^ {X ^ {2} / K ^ {2}} \right] \leq 2. \tag {14}
$$
Definition B.11. (Subgaussian norm) The subgaussian norm of a subgaussian random variable  $X$ , denoted by  $\| X\|_{\psi_2}$  is defined to be the smallest  $K$  in (14). In other words,
$$
\left\| X \right\| _ {\psi_ {2}} \triangleq \inf  \left\{t \geq 0 \mid \mathbb {E} \left[ e ^ {X ^ {2} / t ^ {2}} \right] \leq 2 \right\}. \tag {15}
$$
It can be shown that any bounded random variable  $X$  is subgaussian, and satisfies,
$$
\| X \| _ {\psi_ {2}} \leq \frac {\| X \| _ {\infty}}{\log 2}. \tag {16}
$$
We next present a result that upper bounds the spectral norm of a matrix with subgaussian entries.
Lemma B.12. (Vershynin [74, Thm 4.4.5]) Let  $\mathbf{X}$  be a  $d \times m$  random matrix whose entries  $X_{ij}$  are independent, zero-mean, subgaussian random variables. Then, for any  $t > 0$ , we have,
$$
\left\| \mathbf {X} \right\| _ {2} \leq C K \left(\sqrt {d} + \sqrt {m} + t\right)
$$
with probability exceeding  $1 - 2e^{-t^2}$ . Here,  $K = \max_{i,j} \| A_{ij} \|_{\psi_2}$  and  $C$  is an absolute constant.
C Quantization error of uniformly dithered scalar quantizer
For a scalar  $x \in [-\mathrm{R}, +\mathrm{R}]$ , let us denote the quantization error of uniformly dithered scalar quantizer with a bit-budget of B bits as  $\epsilon = \mathrm{Q}_{\mathrm{R,B}}(x) - x$ . Clearly, the quantization error is bounded as  $|\epsilon| \leq \Delta$ . The following result further characterizes its mean and variance of this error.
Lemma C.1. The uniformly dithered scalar quantizer as described in (1) satisfies,
$$
\mathbb {E} \left[ \epsilon \right] = 0 a n d V a r (\epsilon) \leq \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}},
$$
where the  $\mathbb{E}(\cdot)$  is over the randomness due to dithering in the quantizer operation.
Proof. Suppose  $x \in [q_k, q_{k+1})$  and  $q_{k+1} = q_k + \Delta$ , where  $\Delta = \frac{2\mathrm{R}}{2^{\mathbb{B}} - 1}$ . Then,
$$
\mathbb {E} \mathrm {Q} _ {\mathrm {R}, \mathrm {B}} (x) = q _ {k + 1} \frac {x - q _ {k}}{\Delta} + q _ {k} \left(1 - \frac {x - q _ {k}}{\Delta}\right) = \frac {(q _ {k} + \Delta) (x - q _ {k}) + q _ {k} (\Delta - x + q _ {k})}{\Delta} = x.
$$
To evaluate the variance,
$$
\begin{array}{l} \operatorname {V a r} \left(\mathrm {Q} _ {\mathrm {R}, \mathrm {B}} (x) - x\right) ^ {2} = \left(q _ {k + 1} - x\right) ^ {2} \frac {(x - q _ {k})}{\Delta} + \left(q _ {k} - x\right) ^ {2} \left(1 - \frac {x - q _ {k}}{\Delta}\right) \\ \leq \left(q _ {k + 1} - x\right) \left(x - q _ {k}\right) \\ \leq \sup  _ {x \in \left[ q _ {k}, q _ {k + 1}\right)} \left(q _ {k + 1} - x\right) \left(x - q _ {k}\right) \\ = \left(q _ {k + 1} - \frac {q _ {k} + q _ {k + 1}}{2}\right) \left(\frac {q _ {k} + q _ {k + 1}}{2} - q _ {k}\right) = \frac {\Delta^ {2}}{4} = \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}}. \\ \end{array}
$$
D Gaussian embeddings: Application of equalization to vector quantizers
In this section, we show a result on how Gaussian embeddings help in reducing the  $\ell_2$ -quantization error of a uniformly dithered vector quantizer. We consider a clipped version of the uniform scalar quantizer with bit-budget B described in §2.1 and App. C. In order to quantize a vector  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \|_2 \leq R$  using a uniform scalar quantizer (as described above), the quantization operation is applied to each coordinate of the vector independently, i.e.,
$$
\mathrm {Q} _ {\mathrm {R}} (\mathbf {x}) = \left[ \mathrm {Q} _ {\mathrm {R}} \left(x _ {1}\right), \dots , \mathrm {Q} _ {\mathrm {R}} \left(x _ {d}\right) \right]. \tag {17}
$$
Here, the subscript B is dropped because the bit-budget is evident from the context. Furthermore, since  $\| \mathbf{x} \|_2 \leq R$  implies  $x_i \in [-R, +R]$  for every  $i \in [d]$ , the quantizer  $Q_R$  does not saturate. From Lemma C.1, the expected quantization error is given by,
$$
\mathbb {E} \left\| Q _ {R} (\mathbf {x}) - \mathbf {x} \right\| _ {2} ^ {2} = \sum_ {i \in [ d ]} \mathbb {E} \left[ \left(Q _ {R} \left(x _ {i}\right) - x _ {i}\right) ^ {2} \right] \leq \frac {\mathrm {R} ^ {2} d}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}}. \tag {18}
$$
Quantizing Gaussian embeddings: Suppose instead of quantizing  $\mathbf{x} \in \mathbb{R}^d$  directly, we quantize  $\mathbf{u} = \mathbf{S}\mathbf{x} \in \mathbb{R}^m$ , where  $\mathbf{S} \in \mathbb{R}^{m \times d}$  with  $S_{ij} \sim \mathcal{N}\left(0, \frac{1}{m}\right)$ . Note that for every  $j \in [m]$ , we have  $u_j \sim \mathcal{N}\left(0, \frac{1}{m} \| \mathbf{x} \|_2^2\right)$ . Since  $u_j$  can be anything in  $(- \infty, + \infty)$ , there is a finite probability that the uniform scalar quantizer might get saturated. For this reason, for any scalar  $u \in (-\infty, +\infty)$ , we define the clipped uniformly dithered quantizer with clipping parameter  $t$  as follows:
$$
\mathrm {Q} (u) = \left\{ \begin{array}{l l} \mathrm {Q} _ {t} (u) & \text {i f} | u | \leq t \\ t & \text {i f} u > t \\ - t & \text {i f} u <   - t. \end{array} \right. \tag {19}
$$
The dynamic range of the quantizer is parameterized by  $t$ , which is to be chosen appropriately. Note that it is just for this section for the purposes of illustration, that we choose the clipped variant of the quantizer. In LPLR, we choose the dynamic range to be high enough so in practice it remains unsaturated with a very high probability. The following proposition upper bounds the quantization error of quantizing Gaussian embeddings.
Proposition D.1. For a given vector  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \| \leq \mathrm{R}$ , the clipped uniformly dithered quantizer described in (19) with  $t = \frac{\mathrm{R}}{\sqrt{m}}$  satisfies
$$
\mathbb {E} \left\| \mathrm {Q} (\mathbf {S x}) - \mathbf {S x} \right\| _ {2} ^ {2} \leq \frac {\mathrm {R} ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} + \frac {\mathrm {R} ^ {2} \sqrt {2}}{\sqrt {\pi e}},
$$
where the expectation is over the randomness in the construction of  $\mathbf{S}$  and the quantization dither.
Proof. Since  $\{u_i\}_{i\in [m]}$  are independently and identically distributed, let us denote the distribution as  $f(u) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{u^2}{2\sigma^2}}$ , where  $\sigma = \frac{\|\mathbf{x}\|_2^2}{m}$ . The expected quantization error for the clipped quantizer is then given by,
$$
\mathbb {E} _ {\mathbf {S}, \mathbf {Q} _ {t}} \left[ (\mathrm {Q} (u) - u) _ {2} ^ {2} \right] = \int_ {| u | \leq t} \mathbb {E} _ {\mathbf {Q} _ {t}} \left[ (\mathrm {Q} (u) - u) _ {2} ^ {2} \right] f (u) d u + 2 \int_ {u > t} (t - u) ^ {2} f (u) d u \tag {20}
$$
Here, the expectation is over the stochasticity of the quantization dither, as well as the random matrix S. The factor of 2 in the second term appears due to symmetry of clipping and that of Gaussian distribution. Using (18), the first term on the R.H.S. can be upper bounded as,
$$
\int_ {| u | \leq t} \mathbb {E} _ {\mathrm {Q} _ {t}} \left[ (\mathrm {Q} (u) - u) _ {2} ^ {2} \right] f (u) d u \leq \frac {t ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}} \int_ {| u | \leq t} f (u) d u \leq \frac {t ^ {2}}{\left(2 ^ {\mathrm {B}} - 1\right) ^ {2}}. \tag {21}
$$
To analyze the second term, note that,
$$
\frac {1}{\sigma \sqrt {2 \pi}} \int_ {t} ^ {\infty} (t - u) ^ {2} e ^ {- \frac {u ^ {2}}{2 \sigma^ {2}}} d u = \frac {1}{2} (\sigma^ {2} + t ^ {2}) \left(1 - \operatorname {e r f} \left(\frac {t}{\sigma \sqrt {2}}\right)\right) - \frac {\sigma t}{\sqrt {2 \pi}} e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} \triangleq \Psi (t, \sigma), \tag {22}
$$
where  $\operatorname{erf}(z)$  denotes the error function defined as,
$$
\operatorname {e r f} (z) = \frac {2}{\sqrt {\pi}} \int_ {0} ^ {x} e ^ {- x ^ {2}} d x \tag {23}
$$
Simple calculation would show,
$$
\begin{array}{l} \begin{array}{c} \frac {\partial \Psi (t , \sigma)}{\partial \sigma} \end{array} \\ = \sigma \left(1 - \mathrm {e r f} \left(\frac {t}{\sigma \sqrt {2}}\right)\right) - \frac {\sigma t}{\sqrt {2 \pi}} e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} + \frac {t}{\sqrt {2 \pi}} \left(1 + \frac {t ^ {2}}{\sigma^ {2}}\right) e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} - \frac {t}{\sqrt {2 \pi}} e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} - \frac {t ^ {3}}{\sigma^ {2} \sqrt {2 \pi}} e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} \\ = \sigma \left(1 - \operatorname {e r f} \left(\frac {t}{\sigma \sqrt {2}}\right)\right) \geq 0. \tag {24} \\ \end{array}
$$
Since  $\frac{\partial\Psi(t,\sigma)}{\partial\sigma}\geq 0$ $\Psi (t,\sigma)$  is a non-decreasing function of  $\sigma$ . Since  $\sigma^2 = \frac{\|\mathbf{x}\|_2^2}{m}\leq \frac{\mathrm{R}^2}{m}$ , we have the upper bound,
$$
\begin{array}{l} 2 \int_ {u > t} (t - u) ^ {2} f (u) d u \leq \left(\frac {\mathrm {R} ^ {2}}{m} + t ^ {2}\right) \left(1 - \operatorname {e r f} \left(\frac {t \sqrt {m}}{\mathrm {R} \sqrt {2}}\right)\right) - \frac {\mathrm {R} t \sqrt {2}}{\sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}} \\ = \left(\frac {\mathrm {R} ^ {2}}{m} + t ^ {2}\right) \operatorname {e r f c} \left(\frac {t \sqrt {m}}{\mathrm {R} \sqrt {2}}\right) - \frac {\mathrm {R} t \sqrt {2}}{\sqrt {\pi m}} e ^ {- \frac {m t ^ {2}}{2 \mathrm {R} ^ {2}}}, \tag {25} \\ \end{array}
$$
where  $\operatorname{erfc}(z) = 1 - \operatorname{erf}(z)$  is the complementary error function.
Quantization error of a scalar Gaussian sketch: Since  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \|_2 \leq R$ , let  $\mathbf{s} \sim \mathcal{N}(\mathbf{0}, \frac{1}{m}\mathbf{I}_d)$ . We first consider the quantization of  $\mathbf{x}^\top \mathbf{s}$ , and upper bound the error  $\mathbb{E}\left[\left(\mathrm{Q}(\mathbf{x}^\top \mathbf{s}) - \mathbf{x}^\top \mathbf{s}\right)_2^2\right]$ . Clearly,  $\mathbf{x}^\top \mathbf{s} \sim \mathcal{N}(0, \frac{\mathrm{R}^2}{m})$ . Consequently, using (21) and (25), for any  $t \geq 0$ , we have,