SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hilal Asi
hilal.asi94@gmail.com
Vitaly Feldman
vitaly.edu@gmail.com
Jelani Nelson
minilek@berkeley.edu
Huy L. Nguyen
hu.nguyen@northeastern.edu
Kunal Talwar
kunal@kunaltalwar.org
We study the problem of locally private mean estimation of high-dimensional vectors in the Euclidean ball. Existing algorithms for this problem either incur suboptimal error or have high communication and/or run-time complexity. We propose a new algorithmic framework, ProjUnit, for private mean estimation that yields algorithms that are computationally efficient, have low communication complexity, and incur optimal error up to a  $1 + o(1)$ -factor. Our framework is deceptively simple: each randomizer projects its input to a random low-dimensional subspace, normalizes the result, and then runs an optimal algorithm such as PrivUnitG in the lower-dimensional space. In addition, we show that, by appropriately correlating the random projection matrices across devices, we can achieve fast server run-time. We mathematically analyze the error of the algorithm in terms of properties of the random projections, and study two instantiations. Lastly, our experiments for private mean estimation and private federated learning demonstrate that our algorithms empirically obtain nearly the same utility as optimal ones while having significantly lower communication and computational cost.
Distributed estimation of the mean, or equivalently the sum, of vectors  $v_{1}, \ldots, v_{n} \in \mathbb{R}^{d}$  is a fundamental problem in distributed optimization and federated learning. For example, in the latter, each of  $n$  devices may compute some update to parameters of a machine learning model based on its local data, at which point a central server wishes to apply all updates to the model, i.e. add  $\sum_{i=1}^{n} v_{i}$  to the vector of parameters. The typical desire to keep local data private necessitates methods for computing this sum while preserving privacy of the local data on each individual device, so that the central server essentially only learns the noisy sum and (almost) nothing about each individual summand  $v_{i}$  [20, 11].
The gold standard for measuring privacy preservation is via the language of differential privacy [21]. In this work, we study this problem in the setting of local differential privacy (LDP). We consider (one-round) protocols for which there exists some randomized algorithm  $\mathcal{R}:\mathbb{R}^d\to \mathcal{M}$  (called the local randomizer), such that each device  $i$  sends  $\mathcal{R}(v_i)$  to the server. We say the protocol is  $\varepsilon$ -differentially private if for any  $v,v^{\prime}\in \mathbb{R}^{d}$  and any event  $S\subseteq \mathcal{M}$ ,  $\operatorname *{Pr}(\mathcal{R}(v)\in S)\leq e^{\varepsilon}\operatorname *{Pr}(\mathcal{R}(v^{\prime})\in S)$ . If  $\varepsilon = 0$  then the distribution of  $\mathcal{R}(v)$  is independent of  $v$ , and hence the output of  $\mathcal{R}(\cdot)$  reveals nothing about the local data (perfect privacy); meanwhile if  $\varepsilon = \infty$  then the distributions of  $\mathcal{R}(v)$  and  $\mathcal{R}(v^{\prime})$  can be arbitrarily far, so that in fact one may simply set  $\mathcal{R}(x) = x$  and reveal local data in the clear (total lack of privacy). Thus,  $\varepsilon \geq 0$  is typically called the privacy loss of a protocol.
There has been much previous work on private protocols for estimating the mean  $\mu \coloneqq \frac{1}{n}\sum_{i = 1}^{n}v_{i}$  in the LDP setting. Henceforth we assume each  $v_{i}$  lives on the unit Euclidean sphere  ${}^{1}\mathbb{S}_{d - 1}\subset \mathbb{R}^{d}$ . Let  $\hat{\mu}$  be the estimate computed by the central server based on the randomized messages  $\mathcal{R}(v_1),\ldots ,\mathcal{R}(v_n)$  it receives. Duchi and Rogers [18] showed that the asymptotically optimal expected mean squared error  $\mathbb{E}\| \mu -\hat{\mu}\| _2^2$  achievable by any one-round protocol must be at least  $\Omega (\frac{d}{n\min(\varepsilon,\varepsilon^2)})$ , which is in fact achieved by several protocols [19, 9, 14, 24]. These protocols however achieved empirically different errors, with some having noticeably better constant factors than others.
Recent work of Asi et al. [7] sought to understand the optimal error achievable by any protocol. Let  $\mathcal{R}$  be any local randomizer satisfying  $\varepsilon$ -DP, and  $\mathcal{A}$  be an aggregation algorithm for the central server such that it computes its mean estimate as  $\hat{\mu} \coloneqq \mathcal{A}(\mathcal{R}(v_1), \ldots, \mathcal{R}(v_n))$ . Furthermore, suppose that the protocol is unbiased, so that  $\mathbb{E} \hat{\mu} = \mu$  for any inputs  $v_1, \ldots, v_n$ . Lastly, let  $\mathcal{A}_{\mathrm{PrivUnit}_\varepsilon}, \mathcal{R}_{\mathrm{PrivUnit}_\varepsilon}$  denote the  $\mathrm{PrivUnit}_\varepsilon$  protocol of [9] (parameterized to satisfy  $\varepsilon$ -DP<sup>2</sup>). Let  $\mathrm{Err}_{n,d}(\mathcal{A}, \mathcal{R})$  denote
$$
\sup  _ {v _ {1}, \dots , v _ {n} \in S ^ {d - 1}} \| \mathcal {A} (\mathcal {R} (v _ {1}), \dots , \mathcal {R} (v _ {n})) - \mu \| _ {2} ^ {2}.
$$
Asi et al. [7] proved the remarkable theorem that for any  $n, d$ :
$$
\operatorname {E r r} _ {n, d} (\mathcal {A}, \mathcal {R}) \geq \operatorname {E r r} _ {n, d} \left(\mathcal {A} _ {\text {P r i v U n i t} _ {\varepsilon}}, \mathcal {R} _ {\text {P r i v U n i t} _ {\varepsilon}}\right).
$$
Thus, PrivUnit is not only asymptotically optimal, but in fact actually optimal in a very strong sense (at least, amongst unbiased protocols).
While this work thus characterizes the optimal error achievable for  $\varepsilon$ -LDP mean estimation, there are other desiderata that are important in practice. The most important amongst them are the device runtime (the time to compute  $\mathcal{R}(v)$ ), the server runtime (the time to compute  $\mathcal{A}$  on  $(\mathcal{R}(v_1),\dots,\mathcal{R}(v_n))$ ), and the communication cost ( $|\log_2|\mathcal{M}|\|$  bits for each device to send its  $\mathcal{R}(v)$  to the server). The known error-optimal algorithms (PrivUnit [9] and PrivUnitG [7]) either require communicating  $d$  floats or have a slower device runtime of  $\Omega(e^{\varepsilon}d)$ . As mean estimation is often used as a subroutine in high-dimensional learning settings, this communication cost can be prohibitive and this has led to a large body of work on reducing communication cost [2, 27, 14, 26, 24, 12]. Server runtimes of these optimal algorithms are also slow, scaling as  $nd$ , whereas one could hope for nearly linear time  $\tilde{O}(n + d)$  (see Table 1).
Chen et al. [14] recently studied this tradeoff and proposed an algorithm called SQKR, which has an optimal communication cost of  $\varepsilon$  bits and device runtime only  $O(d\log^2 d)$ . However, this algorithm incurs error that is suboptimal by a constant factor, which can be detrimental in practice. Indeed our experiments in Section 4 demonstrate the significance of such constants as the utility of these algorithms does not match that of optimal algorithms even empirically, resulting for example in  $10\%$  degradation in accuracy for private federated learning over MNIST with  $\varepsilon = 10$ .
Feldman and Talwar [24] give a general approach to reducing communication via rejection sampling. When applied to PrivUnitG, it yields a natural algorithm that we call Compressed PrivUnitG. While it yields optimal error and near-optimal communication, it requires device run time that is  $O(e^{\varepsilon}d)$ . These algorithms are often used for large  $d$  (e.g. in the range  $10^{5} - 10^{7}$ ) corresponding to large model sizes. The values of  $\varepsilon$  are often in the range 4-12 or more, which may be justifiable due to privacy being improved by aggregation or shuffling [10, 15, 22, 25]. For this range of values, the  $\Theta(e^{\varepsilon}d)$  device runtime is prohibitively large and natural approaches to reduce this in Feldman and Talwar [24] lead to increased error. To summarize, in the high-dimensional setting, communication-efficient local randomizers are forced to choose between high device runtime or suboptimal error (see Table 1).
Another related line of work is non-private communication efficient distributed mean estimation where numerous papers have recently studied the problem due to its importance in federated learning [37, 4, 29, 2, 26, 23, 32, 38, 39]. Similarly to our paper, multiple works in this line of work have used random rotations to design efficient algorithms [37-39]. However, the purpose of these works is to
One often considers the problem for the vectors being of norm at most 1, rather than exactly 1. It is easy to show that vectors  $v$  in the unit ball in  $\mathbb{R}^d$  can be mapped to  $\mathbb{S}_d \subseteq \mathbb{R}^{d + 1}$ , simply as  $(v, 1 - \| v\| _2^2)$ . Thus up to changing  $d$  to  $d + 1$ , the two problems are the same. Since we are interested in the case of large  $d$ , we choose the version that is easier to work with.
There are multiple ways to set parameters of PrivUnit to achieve  $\varepsilon$ -DP; we assume the setting described by Asi et al. [7], which optimizes the parameters to minimize the expected mean squared error.
develop better quantization schemes for real-valued vectors to reduce communication to  $(1 + o(1))\cdot d$  bits. This is different from our goal, which is to send  $k\ll d$  parameters while still obtaining the statistically optimal bounds up to a  $1 + o(1)$  factor. Moreover, in order to preserve privacy, our algorithms require new techniques to handle the norms of the projected vectors, and post-process them using different normalization schemes, in order to guarantee that the projection error after post-processing is negligible.
Our main contribution is a new framework, ProjUnit, for private mean estimation which results in near-optimal, efficient, and low-communication algorithms. Our algorithm obtains the same optimal utility as PrivUnit and PrivUnitG but with a significantly lower polylogarithmic communication complexity, and device runtime that is  $O(d\log d)$  and server runtime  $\tilde{O}(n + d)$ . We also implement our algorithms and show that both the computational cost and communication cost are small empirically as well. Figure 1 plots the error as a function of  $\varepsilon$  for several algorithms and demonstrates the superiority of our algorithms compared to existing low-communication algorithms (see more details in Section 4). Moreover, we show that the optimal error bound indeed translates to fast convergence in our private federated learning simulation.
At a high level, each local randomizer in our algorithm first projects the vector to a randomly chosen lower-dimensional subspace, and then runs an optimal local randomizer in this lower-dimensional space. At first glance, this is reminiscent of the use of random projections in the Johnson-Lindenstrauss (JL) transform or the use of various embeddings in prior work (such as [14]). However, unlike the JL transform and embeddings in prior work, in our application, each point uses its own projection matrix. The JL transform is designed to preserve distances between points, not the points themselves. In our application, a random projection is used to obtain a low-dimensional unbiased estimator for a point; however the variance of this estimator is quite large (of the order of  $d / k$ ). Our main observation is that while large, this variance is small compared to the variance added due to privacy when  $k$  is chosen appropriately. This fact allows us to use the projections as a pre-processing step. With some care needed to control the norm of the projected vector which is no longer fixed, we then run a local randomizer in the lower dimensional space. We analyze the expected squared error of the whole process and show that as long as the random projection ensemble satisfies certain specific properties, the expected squared error of our algorithm is within  $(1 + o(1))$  factors of the optimal; the  $o(1)$  term here falls with the projection dimension  $k$ . The required properties are easily shown to be satisfied by random orthogonal projections. We further show that more structured projection ensembles, that allow for faster projection algorithms, also satisfy the required properties, and this yields even faster device runtime.
Although these structured projections result in fast device runtime, they still incur an expensive computational cost for the server which needs to apply the inverse transformation for each client, resulting in runtime  $O(nd\log d)$ . Specifically, each device sends a privatized version  $\hat{v}_i$  of  $W_i v_i$ , and the server must then compute  $\sum_{i} W^{\top} \hat{v}_i$ . To address this, we use correlated transformations in order to reduce server runtime while maintaining optimal accuracy up to  $1 + o(1)$  factors. In our correlated ProjUnit protocol, the server pre-defines a random transformation  $W$ , which all devices then use to define  $W_i = S_i W$  where  $S_i$  is a sampling matrix. The advantage is then that  $\sum_{i} W_i^{\top} \hat{v}_i$  is replaced with  $\sum_{i} W^{\top} \hat{v}_i = W^{\top}(\sum_{i} S_i \hat{v}_i)$ , which can be computed more quickly as it requires only a single matrix-vector multiplication. The main challenge with correlated transformations is that the correlated client transformations result in increased variance. However, we show that the independence in choosing the sampling matrices  $S_i$  is sufficient to obtain optimal error.
Finally, we note without correlating projections each client using its own projection would imply that each projection needs to be communicated to the server. Doing this naively would require communicating  $kd$  real values completely defeating the benefits of our protocol. However the projection matrix does not depend on the input and therefore can be communicated cheaply using a seed to an appropriate pseudorandom generator.
2 A random projection framework for low-communication private algorithms
In this section we propose a new algorithm, namely ProjUnit, which has low communication complexity and obtains near-optimal error (namely, up to a  $1 + o(1)$  factor of optimum). The starting point
Utility	Run-time (client)	Run-time (server)	Communication
Repeated PrivHS [19, 24]	O(OPT)	εd	n[ε]	ε·poly(log d)
PrivUnit [9]	OPT	d	nd	d
SQKR [14]	O(OPT)	d log2d	nε log d + d log2d	ε log d
CompPrivUnit [24]	(1 + o(1))·OPT	εd	nd log d	ε·poly(log d)
FastProjUnit (Section 2)	(1 + o(1))·OPT	d log d	nd log d	ε log2d
FastProjUnit-corr (Section 3)	(1 + o(1))·OPT	d log d	n log3d + nε log d + d log d	ε log2d
Table 1: Comparison of Error-Runtime-Communication trade-offs for different algorithms for private mean estimation. The last two rows use our algorithms from Section 2 and Section 3 with a communication budget  $k \approx \varepsilon \log d$ . PrivUnitG is omitted as it has the same guarantees as PrivUnit except utility where it has  $(1 + o(1)) \cdot OPT$ . We omit constant factors from the run-time and communication complexities.
Figure 1: Squared error of different algorithms as a function of  $\varepsilon$  for  $d = 32768$  averaged over 50 runs with  $90\%$  confidence intervals. The lines for the top three algorithms are almost identical.
of our algorithms is a randomized projection map in  $\mathbb{R}^{k\times d}$  which we use to project the input vectors to a lower-dimensional space. The algorithm then normalizes the vector as a necessary pre-processing step. Finally, the local randomizer applies PrivUnitG [7] (see Algorithm 8 in Appendix) over the normalized projected vector and sends the response to the server. The server then applies the inverse transformation and aggregates the responses in order to estimate the mean. We present the full details of the client and server algorithms in Algorithm 1 and Algorithm 2.
To analyze this algorithm, we first present our general framework for an arbitrary distribution over projections. In the next sections we utilize different instances of the framework using different random projections such as random rotations and more structured transforms.
Algorithm 1 ProjUnit (client)
Require: Input vector  $v \in \mathbb{R}^d$ , Distribution over projections  $\mathcal{W}$ .
1: Randomly sample transform  $W \in \mathbb{R}^{k \times d}$  from  $\mathcal{W}$
2: Project the input vector  $v_{p} = Wv$
3:Normalize  $u = \frac{v_p}{\|v_p\|_2}$
4: Let  $\hat{u} = \operatorname{PrivUnitG}(u)$  (as in Algorithm 8)
5: Send  $\hat{u}$  and (encoding of)  $W$  to server
The following theorem states the privacy and utility guarantees of ProjUnit for a general distribution over transformation  $\mathcal{W}$  that satisfies certain properties. For ease of notation, let  $\mathcal{R}_{\mathrm{PU}}$  denote the ProjUnit local randomizer of the client (Algorithm 1), and  $\mathcal{A}_{\mathrm{PU}}$  denote the server aggregation of
Algorithm 2 ProjUnit (server)
1:Receive  $\hat{u}_1,\dots ,\hat{u}_1$  from clients with (encodings of) transforms  $W_{1},\ldots ,W_{n}$
2: Return the estimate  $\hat{\mu} = \frac{1}{n}\sum_{i = 1}^{n}W_{i}^{\top}\hat{u}_{i}$
ProjUnit (Algorithm 2). To simplify notation, we let
$$
\operatorname {E r r} _ {n, d} \left(\operatorname {P r i v U n i t G}\right) = \operatorname {E r r} _ {n, d} \left(\mathcal {A} _ {\operatorname {P r i v U n i t G} _ {\varepsilon}}, \mathcal {R} _ {\operatorname {P r i v U n i t G} _ {\varepsilon}}\right) = c _ {d, \varepsilon} \frac {d}{n \varepsilon}
$$
denote the error of the PrivUnitG  $\varepsilon$ -DP protocol where  $A_{\mathrm{PrivUnitG}_{\varepsilon}}$ ,  $\mathcal{R}_{\mathrm{PrivUnitG}_{\varepsilon}}$  denote the PrivUnitG protocol with optimized parameters (see Algorithm 8) and  $c_{d,\varepsilon} = O(1)$  is a constant [7]. We defer the proof to Appendix B.1.
Theorem 1. Let  $k \leq d$  and assume that the transformations  $W_{i} \in \mathbb{R}^{k \times d}$  are independently chosen from a distribution  $\mathcal{W}$  that satisfies:
1. Bounded operator norm:  $\mathbb{E}\left[\left\| W_i^\top \right\|^2\right] \leq d / k + \beta_{\mathcal{W}}$ .
2. Bounded bias:  $\left\| \mathbb{E}\left[\frac{W_i^\top W_i v}{\|W_i v\|_2}\right] - v\right\|_2 \leq \sqrt{\alpha_{\mathcal{W}}} \text{ for all unit vectors } v \in \mathbb{R}^d.$
Then for all unit vectors  $v_{1},\ldots ,v_{n}\in \mathbb{R}^{d}$ , setting  $\hat{\mu} = \mathcal{A}_{\mathsf{PU}}(\mathcal{R}_{\mathsf{PU}}(v_1),\dots,\mathcal{R}_{\mathsf{PU}}(v_n))$ , the local randomizers  $\mathcal{R}_{\mathsf{PU}}$  are  $\varepsilon$ -DP and
$$
\mathbb {E} \left[ \left\| \hat {\mu} - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| _ {2} ^ {2} \right] \leq \operatorname {E r r} _ {n, d} (\operatorname {P r i v U n i t} G) \cdot \left(1 + \frac {k \beta_ {\mathcal {W}}}{d} + O \left(\frac {\varepsilon + \log k}{k}\right)\right) + \alpha_ {\mathcal {W}}.
$$
2.1 ProjUnit using Random Rotations
Building on the randomized projection framework of the previous section, in this section we instantiate it with a random rotation matrix. In particular, we sample  $W \in \mathbb{R}^{k \times d}$  with the structure
$$
W _ {H} = \sqrt {\frac {d}{k}} S U, \tag {1}
$$
where  $U \in \mathbb{R}^{d \times d}$  is a random rotation matrix such that  $U^{\top} U = I$ , and  $S \in \mathbb{R}^{k \times d}$  is a sampling matrix where each row has a single 1 in a uniformly random location (without repetitions). The following theorem states our guarantees for this distribution.
Theorem 2. Let  $k \leq d$  and  $W \in \mathbb{R}^{k \times d}$  be a random rotation matrix sampled as described in (1). Then for all unit vectors  $v_{1},\ldots ,v_{n} \in \mathbb{R}^{d}$ , setting  $\hat{\mu} = \mathcal{A}_{\mathsf{PU}}(\mathcal{R}_{\mathsf{PU}}(v_1),\dots,\mathcal{R}_{\mathsf{PU}}(v_n))$ , the local randomizers  $\mathcal{R}_{\mathsf{PU}}$  are  $\varepsilon$ -DP and
$$
\mathbb {E} \left[ \left\| \hat {\mu} - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| _ {2} ^ {2} \right] \leq \mathsf {E r r} _ {n, d} (\mathsf {P r i v U n i t G}) \cdot \left(1 + O \left(\frac {\varepsilon + \log k}{k}\right)\right) + O \left(\frac {1}{k ^ {2}}\right).
$$
The proof follows directly from Theorem 1 and the following proposition which proves certain properties of random rotations. We defer the proof to Appendix B.2.
Proposition 1. Let  $W \in \mathbb{R}^{k \times d}$  be a random rotation matrix sampled as described in (1). Then
1. Bounded operator norm:  $\| W^{\top}\| \leq \sqrt{\frac{d}{k}}$
2. Bounded bias: for every unit vector  $v \in \mathbb{R}^d$ :  $\left\| \mathbb{E}\frac{W^\top Wv}{\|Wv\|} - v \right\| = O(1 / k)$ .
We also have similar analysis for Gaussian transforms with an additional  $O(\sqrt{k / d})$  factor in the first term. We include the analysis in Appendix D.
2.2 Fast ProjUnit using the SRHT
While the random rotation based randomizer enjoys near-optimal error and low communication complexity, its runtime complexity is somewhat unsatisfactory as it requires calculating  $Wv$  for  $W \in \mathbb{R}^{k \times d}$ , taking time  $O(kd)$ . In this section, we propose a ProjUnit algorithm using the Subsampled Randomized Hadamard transform (SRHT), which is closely related to the fast JL transform [3]. We show that this algorithm has the same optimality and low-communication guarantees as the random rotations version, and additionally has an efficient implementation with  $O(d\log d)$  client runtime.
The SRHT ensemble contains matrices  $W_{H}\in \mathbb{R}^{k\times d}$  with the following structure:
$$
W _ {H} = \sqrt {\frac {d}{k}} S H D, \tag {2}
$$
where  $S \in \mathbb{R}^{k \times d}$  is a sampling matrix where each row has a single 1 in a uniformly random location sampled without replacement,  $H \in \mathbb{R}^{d \times d}$  is the Hadamard matrix, and  $D \in \mathbb{R}^{d \times d}$  is a diagonal matrix where  $D_{ii}$  are independent samples from the Rademacher distribution, that is,  $D_{ii} \sim \mathrm{Unif}\{-1, +1\}$ . The main advantage of the SRHT is that there exist efficient algorithms for matrix-vector multiplication with  $H$ .
The following theorem presents our main guarantees for the SRHT-based ProjUnit algorithm.
Theorem 3. Let  $k \leq d$  and  $W$  be sampled from the SRHT ensemble as described in (2). Then for all unit vectors  $v_{1},\ldots ,v_{n} \in \mathbb{R}^{d}$ , setting  $\hat{\mu} = \mathcal{A}_{\mathsf{PU}}(\mathcal{R}_{\mathsf{PU}}(v_1),\dots,\mathcal{R}_{\mathsf{PU}}(v_n))$ , the local randomizers  $\mathcal{R}_{\mathsf{PU}}$  are  $\varepsilon$ -DP and
$$
\mathbb {E} \left[ \left\| \hat {\mu} - \frac {1}{n} \sum_ {i = 1} ^ {n} v _ {i} \right\| _ {2} ^ {2} \right] \leq \operatorname {E r r} _ {n, d} (\operatorname {P r i v U n i t} G) \cdot \left(1 + O \left(\frac {\varepsilon + \log k}{k}\right)\right) + O \left(\frac {\log^ {2} d}{k}\right).
$$
Remark 1. The communication complexity of SRHT-based ProjUnit can be reduced to  $O(k \log d + \log^2 d)$ . To see this, note that  $\hat{u}$  is a  $k$ -dimensional vector. Moreover, the matrix  $W = \sqrt{d / k} \cdot SHD$  can be sent in  $O(k \log d)$  as follows:  $S$  has  $k$  rows, each with a single entry that contains 1, hence we can send the indices for each row using  $\log d$  bits for each row. Moreover,  $H$  is the Hadamard transform and need not be sent. Finally,  $D$  is a diagonal matrix with entries  $D_{ii} \sim \mathrm{Unif}\{-1, +1\}$ . By standard techniques [36, 5], we only need the entries of  $D$  to be  $O(\log (d))$ -wise independent for Proposition 2 to hold. Thus  $O(\log^2 d)$  bits suffice to communicate a sampled  $D$ .
The proof of the theorem builds directly on the following two properties of the SHRT.
Proposition 2. Let  $W$  be sampled from the SRHT ensemble as described in (2). Then we have
1. Bounded operator norm:  $\mathbb{E}\left[\| W^{\top}\|\right] = \mathbb{E}\left[\| W\|\right]\leq \sqrt{d / k}.$
2. Bounded bias: for every unit vector  $v \in \mathbb{R}^d$ :  $\left\| \mathbb{E}\frac{W^\top Wv}{\|Wv\|} - v \right\| = O(\log(d) / \sqrt{k})$ .
Theorem 3 now follows directly from the bounds of Theorem 1 using  $\alpha_{\mathcal{W}} = O(\log^2 (d) / k)$ . We defer the proof to Appendix B.3.