SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Main Differences from [DKPP22] For the streaming setting, we change our viewpoint and instead of denoting by  $P = (1 - \epsilon)G + \epsilon B$  the uniform distribution over a fixed dataset, we now denote by  $P$  the underlying data distribution itself (in our case  $G = \mathcal{N}(\mu, \mathbf{I})$  or more generally any distribution satisfying the goodness conditions of Definition 2.1). Consequently, the quantities  $\Sigma_t, \mathbf{B}_t, \mathbf{M}_t, \Sigma_t^\perp, \mathbf{B}_t^\perp, \mathbf{M}_t^\perp$ , mentioned in the pseudocode and in our proofs, are all population-level quantities. Instead of having direct access to them (as in Section 3), the algorithm is free to use the stream of fresh i.i.d. samples from  $P$  and build estimators to approximate them. The estimators needed for this are the same as in [DKPP22], for which we will give an overview later on. Apart from this difference, the algorithm from Section 3 is easy to implement in the streaming setting with low memory: The operation of multiplying a vector  $z$  by an empirical covariance matrix  $\sum_x (x - \mu)(x - \mu)^\top$  (which was used frequently in Section 3) can be implemented in the streaming setting without storing the entire covariance matrix, if one calculates first the inner product  $(x - \mu)^\top z$  in the sum and then multiply that scalar result with  $(x - \mu)$ . Second, the weight function  $w_t(x)$  that the algorithm maintains for re-weighting points is now defined over the entire  $\mathbb{R}^d$  (and not just over a fixed dataset, since there is no such dataset anymore). The algorithm can store a representation of  $w_t$  by storing the matrices  $\mathbf{U}_t$  and how many times Line 7 of the filter has been executed (having these in memory then given any point  $x \in \mathbb{R}^d$  the algorithm can calculate  $w_t(x)$ ). Since the number of iterations  $t_{\max} = O(\mathrm{polylog}(d / \epsilon))$  and the size of each  $\mathbf{U}_t$  is  $O(d\mathrm{polylog}(d / \epsilon))$ , the memory needed to store the weights is  $O(d\mathrm{polylog}(d / \epsilon))$  overall.
The proof of correctness of the resulting streaming algorithm follows the same arguments as before, but needs to account extra errors for the approximations taking place. Instead of giving a complete formal proof, we state the new deterministic conditions that are needed in Condition E.3. Then, we discuss why each of them is needed and how it can be obtained.
Condition E.3 (Deterministic Conditions for Algorithm 6). Let  $S_{\mathrm{cover}}$  be the cover set of [DKPP22, Lemma 4.9] For all  $t \in [t_{\mathrm{max}}]$ , the following hold:
(i) Scores: For every  $x \in S_{\mathrm{cover}}$ , it holds  $\| \mathbf{M}_t^\perp (x - \mu_t)\| _2^2 \gtrsim \| \mathbf{U}_t(x - \mu_t)\| _2^2 -\| \mathbf{M}_t^\perp \| _\mathrm{F}^2$
(ii) Let  $T \coloneqq \widehat{\lambda_t} \| \mathbf{U}_t\|_{\mathrm{F}}^2$ . For every  $w:\mathbb{R}^d\to [0,1]$ , the algorithm has access to an estimator  $f(w)$  for the quantity  $\mathbf{E}_{X\sim P}[w(X)\tilde{\tau}_t(X)]$ , where  $\tilde{\tau}_t(X) = \| \mathbf{U}_t(x - \mu_t)\| _2^2\mathbb{1}(\| \mathbf{U}_t(x - \mu_t)\| _2^2 >100\mathrm{tr}(\mathbf{U}_t^\top \mathbf{U}_t))$  such that  $f(w) > T_{t} / 2$  whenever  $\mathbf{E}_{X\sim P}[w(X)\tau_t(X)] > T_t$ .
(iii) Spectral norm of  $\mathbf{B}_t^\perp$ :  $\widehat{\lambda}_t \in [0.1\|\mathbf{B}_t^\perp\|_{\mathrm{op}}, 10\|\mathbf{B}_t^\perp\|_{\mathrm{op}}]$ .
(iv) Frobenius norm  $\| \mathbf{U}_t\|_{\mathrm{F}}^2\in [0.8k\| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2,1.2k\| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2 ]$
(v) Maximum norm:  $\max_{j\in [k]}\| v_{t,j}\| _2^2\leq \frac{\log k}{k}\mathrm{tr}(\mathbf{U}_t^\top \mathbf{U}_t)$
(vi) Probability estimate:  $|\widehat{q}_t - q_t| \leq 0.1 / (k^2 t_{\max})$
(vii)  $\mathbf{U}_t$  almost orthogonal: For every pair  $u_i, u_j$  of distinct rows of  $\mathbf{U}_t$  it holds  $|\langle u_i, u_j \rangle| \leq \| u_i \|_2 \| u_j \|_2 / k^2$ .
(viii) Approximate eigenvector: The vector  $u_{t}$  from Line 19 satisfies  $u_{t}^{\top}\mathbf{B}_{t}^{\perp}u_{t} \geq (1 - 1 / p)\| \mathbf{B}_{t}^{\perp}\|_{\mathrm{op}}$ .
Condition (i) This condition is used instead of Condition (iii) in Condition C.1 to go from Equation (20) to Equation (21). The issue with the old condition condition is that in the streaming setting there is no fixed dataset for which we can require the approximation to hold. Instead, we can use the technique from [DKPP22] and define an appropriate cover  $S_{\mathrm{cover}}$  for which  $\mathbf{E}_{X \sim B}[w_{t+1}(X) \| \mathbf{M}_t^\perp(X^\perp - \mu^\perp)] \approx \mathbf{E}_{X \sim S}[w_{t+1}(X) \| \mathbf{M}_t^\perp(X^\perp - \mu^\perp)]$  (see Section 4.2.1 of [DKPP22] for more details on the cover argument). Finally, we remark that the additional term  $\| \mathbf{M}_t^\perp \|_{\mathrm{F}}^2$  is due to the fact that the matrix  $\mathbf{U}_t$  is calculated in Line 17 based on approximations of  $\mathbf{M}_t$  and not  $\mathbf{M}_t$  itself (see [DKPP22, Lemma 4.15]). That error term can be tolerated in the steps of the correctness proof that we provided in Appendix C.
Condition (ii) Since the stopping condition of the filtering procedure (Algorithm 3) depends on population-level quantities, it needs to be estimated within sufficient accuracy. The estimator for that and the correctness of the resulting filter has been analyzed in [DKPP22].
Condition (iii) An approximate top eigenvector for Condition (iii) is obtained by multiplying a random Gaussian vector by  $\log d$  fresh sample-based estimators of  $\mathbf{B}_t^\perp$  (i.e., multiply  $z \sim \mathcal{N}(0, \mathbf{I})$  by  $\widehat{\mathbf{M}}_t^\perp$ , where  $\widehat{\mathbf{M}}_t^\perp$  as in Line 14). This estimator has been analyzed in [DKPP22, page 24].
Conditions (v) and (vi) These follow similarly to the corresponding conditions of Appendix C. For Condition (vi), the estimator now will use fresh estimates  $\widehat{\mathbf{M}}_t^\perp$  of the form of Line 14, instead of the population-level  $\mathbf{M}_t^\perp$ .
Condition (viii) This condition can be obtained by [DKPP23, Lemma 4.3] and [DKPP22, Lemma 4.15].
Algorithm 6 Robust Mean Estimation Under Huber Contamination In Streaming Model (Stage 1)
Input: Parameter  $\epsilon \in (0,1 / 2)$  stream of i.i.d. points from the distribution  $P = (1 - \epsilon)G + \epsilon B$  where  $G$  satisfies Definition 2.1 with appropriate parameters. Output: An approximation of the mean in a subspace  $\nu^{\perp}$  and the orthogonal subspace  $\nu$ .  
1: Let  $C$  be a sufficiently large constant,  $k = C\log^{2}(n + d)$ ,  $t_{\mathrm{max}} = (\log (d / \epsilon))^{C}$ .  
2: Initialize  $V_{1} \gets \emptyset$  and  $w_{1}(x) = 1$  for all  $x \in \mathbb{R}^{d}$ .  
3: for  $t = 1, \ldots, t_{\mathrm{max}}$  do  
4: Let  $\mathcal{V}_{t}$  be the subspace spanned by the vectors in  $V_{t}$ , and  $\mathcal{V}_{t}^{\perp}$  be the perpendicular subspace.  
5: Let  $P_{t}$  be the distribution  $P$  re-weighted by  $w_{t}$ , i.e.,  $P_{t}(x) = w_{t}(x)P(x) / \mathbf{E}_{X \sim P}[w(X)]$ .  
6: Let  $\mu_{t}^{\perp}, \Sigma_{t}^{\perp}$  be the mean and covariance of  $\operatorname{Proj}_{\mathcal{V}_{t}}(X)$  when  $X \sim P_{t}$   
7: Define  $\mathbf{B}_{t}^{\perp} = (\mathbf{E}_{X \sim P}[w_{t}(X)])^{2}\mathbf{\Sigma}_{t}^{\perp} - (1 - C_{1}\epsilon)\Pi_{\mathcal{V}_{t}^{\perp}}$ , where  $\Pi_{\mathcal{V}_{t}^{\perp}}$  is the orthogonal projection matrix for  $\mathcal{V}_{t}^{\perp}$ , and  $\mathbf{M}_{t}^{\perp} := (\mathbf{B}_{t}^{\perp})^{p}$  for  $p = \log(d)$ .  
8: Calculate  $\widehat{\lambda}_{t}$  such that  $\widehat{\lambda}_{t} / \| \mathbf{B}_{t}^{\perp}\|_{\mathrm{op}} \in [0.1 / 10]$  using power iteration. ▷ cf. Appendix B  
9: If  $\widehat{\lambda}_{t} \leq C\epsilon$  then return  $\mu_{t}$  and  $V_{t}$ .  
10: Let  $q_{t} := \mathbf{Pr}_{z,z' \sim \mathcal{N}(0,\mathbf{I})}[\|\mathbf{M}_{t}^{\perp}z, \mathbf{M}_{t}^{\perp}z'\| > \| \mathbf{M}_{t}^{\perp}z\|_{2}\|\mathbf{M}_{t}^{\perp}z'\|_{2}/k^{2}]$ .  
11: Calculate an estimate  $\widehat{q}_{t}$  such that  $|\widehat{q}_{t} - q_{t}| \leq \frac{1}{10(k^{2}t_{\mathrm{max}})}$ .  
12: if  $\widehat{q}_{t} \leq 1/(k^{2}t_{\mathrm{max}})$  then ▷ Case 1 (cf. Section 3.1)  
13: for  $j \in [k]$  do  
14: Let  $\widehat{\mathbf{B}}_{t,j,\ell}^{\perp}$  for  $\ell \in [\log d]$  sample-based versions of  $\mathbf{B}_{t}^{\perp}$ .  
15: Let  $\widehat{\mathbf{M}}_{t}^{\perp} := \prod_{\ell=1}^{\log d}\widehat{\mathbf{B}}_{t,j,\ell}^{\perp}$ .  
16:  $v_{t,j} \gets \widehat{\mathbf{M}}_{t}^{\perp}z_{t,j}$  for  $z_{t,j} \sim \mathcal{N}(0,\mathbf{I})$ .  
17:  $\mathbf{U}_{t} \gets [v_{t,1},\dots,v_{t,k}]^{\top}$  i.e., the matrix with rows  $v_{t,j}$  for  $j \in [k]$ .  
18:  $w_{t+1} \gets \text{MULTI-DIRECTIONALFILTER}(P,w,\epsilon,\mathbf{U}_{t})$  ▷ cf. [DKPP22, Algorithm 4]  
19: else ▷ Case 2 (cf. Section 3.2)  
20: Find a vector  $u_{t}$  such that  $u_{t}^{\top}\mathbf{B}_{t}^{\perp}u_{t} \geq (1-1/p)\|\mathbf{B}_{t}^{\perp}\|_{\mathrm{op}}$  ▷ Power iteration  
21:  $V_{t+1} \gets V_{t} \cup \{u_{t}\}$ .  
22: Let  $\mu_{\mathcal{V}_t} = \mathbf{E}_{X \sim P_t}[\operatorname{Proj}_{\mathcal{V}_t}(X)]$  be the mean of  $P_t$  after projection to  $\mathcal{V}_t$ .  
23: return  $\mu_{\mathcal{V}_t}$  and  $V_t$ .