SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Furthermore, applying Lemma 5 with  $M = Q_k^\top \Pi_{\widehat{U}}^\perp \in \mathbb{R}^{k\times d}$  and  $N = \Pi_{\widehat{U}}^\perp V \in \mathbb{R}^{d\times k}$ , a sufficient condition for having  $\left\| Q_k^\top \Pi_{\widehat{U}}^\perp (\widehat{\Sigma} - \Sigma)\Pi_{\widehat{U}}^\perp V\right\|_2 \leq \frac{\delta\Delta_{k,\kappa}}{20\sqrt{dk}}$  with probability at least  $1 - \delta$  is that
$$
\mathcal {B} \geq \frac {1 6 0 0 d k (\mathcal {V} + V ^ {2})}{\delta^ {2} \Delta_ {k , \kappa} ^ {2}} \log \frac {8 k}{\delta} + \frac {4}{9} \frac {\mathcal {M} ^ {2}}{\mathcal {V} + V ^ {2}} \log \frac {8 k}{\delta}.
$$
Combining all bounds for  $\mathcal{B}$ , we conclude that a sufficient block size  $\mathcal{B}$  to have our desired statement is
$$
\mathcal {B} = \Omega \left(\frac {\mathcal {V} + V ^ {2}}{\Delta_ {k , \kappa} ^ {2}} \left(\frac {d k}{\delta^ {2}} \log \frac {k}{\delta} + \frac {1}{\epsilon^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} + V ^ {2}} \log \frac {d}{\delta}\right).
$$
Proof of Lemma 5. Note that  $\tilde{\pmb{x}}_j$ 's are i.i.d. samples from  $\mathcal{D}$  ( $j \in \{(\tau - 1)\mathcal{B} + 1, \dots, \tau \mathcal{B}\}$ ). Let
$$
\boldsymbol {Y} _ {j} = M \left(\tilde {\boldsymbol {x}} _ {j} \tilde {\boldsymbol {x}} _ {j} ^ {\intercal} - \boldsymbol {\Sigma}\right) \boldsymbol {N}.
$$
Then,  $\mathbf{Y}_i$  are i.i.d. random zero-mean  $d \times k$  matrices across  $j$ . In addition, with Assumption 1,
$$
\mathbb {P} \left[ \| \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} - \boldsymbol {\Sigma} \| _ {2} \leq \mathcal {M} \right] = \sum_ {a ^ {\prime} \in \{0, 1 \}} \mathbb {P} \left[ \| \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} - \boldsymbol {\Sigma} \| _ {2} \leq \mathcal {M} \mid a = a ^ {\prime} \right] \mathbb {P} [ a = a ^ {\prime} ] = 1,
$$
$$
\begin{array}{l} \operatorname {V a r} (\tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal}) = \left\| \mathbb {E} \left[ \mathbb {E} [ (\tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal}) ^ {2} | a ] \right] - \mathbb {E} \left[ \mathbb {E} [ \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a ] ^ {2} \right] \right\| _ {2} \\ \leq \left\| \mathbb {E} \left[ \mathbb {E} \left[ (\tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal}) ^ {2} | a \right] - \mathbb {E} [ \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a ] ^ {2} \right] \right\| _ {2} + \left\| \mathbb {E} \left[ \mathbb {E} [ \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a ] ^ {2} \right] - \mathbb {E} [ \mathbb {E} [ \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a ] ] ^ {2} \right\| _ {2} \\ \leq \sum_ {a \in [ 0, 1 ]} p _ {a} \underbrace {\left\| \mathbb {E} \left[ (\tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal}) ^ {2} | a \right] - \mathbb {E} \left[ \tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a \right] ^ {2} \right\| _ {2}} _ {\operatorname {V a r} (\tilde {\boldsymbol {x}} \tilde {\boldsymbol {x}} ^ {\intercal} | a)} + \left\| \mathbb {E} \left[ (\boldsymbol {\Sigma} _ {a} + \boldsymbol {\mu} _ {a} \boldsymbol {\mu} _ {a} ^ {\intercal}) ^ {2} \right] - \mathbb {E} \left[ \boldsymbol {\Sigma} _ {a} + \boldsymbol {\mu} _ {a} \boldsymbol {\mu} _ {a} ^ {\intercal} \right] ^ {2} \right\| _ {2} \\ \leq \mathcal {V} + p _ {0} p _ {1} \| \boldsymbol {\Sigma} _ {0} + \boldsymbol {\mu} _ {0} \boldsymbol {\mu} _ {0} ^ {\intercal} - \boldsymbol {\Sigma} _ {1} + \boldsymbol {\mu} _ {1} \boldsymbol {\mu} _ {1} ^ {\intercal} \| _ {2} ^ {2} \\ \leq \mathcal {V} + 4 p _ {0} (1 - p _ {0}) V ^ {2} \\ <   \mathcal {V} + V ^ {2}. \\ \end{array}
$$
Applying the matrix Bernstein inequality (Proposition 1), we obtain the following: when  $0 \leq x \leq \frac{3(\mathcal{V} + V^2)}{\mathcal{M}}$ ,
$$
\mathbb {P} \left(\left\| \frac {1}{\mathcal {B}} \sum_ {i = 1} ^ {\mathcal {B}} \boldsymbol {Y} _ {i} \right\| _ {2} \geq x\right) \leq (m + n) \exp \left(- \frac {\mathcal {B} x ^ {2}}{4 (\mathcal {V} + V ^ {2})}\right) \leq \frac {\delta}{4}.
$$
Solving for  $x$ , we have
$$
\frac {4 (\mathcal {V} + V ^ {2})}{\mathcal {B}} \log \frac {4 (m + n)}{\delta} \leq x ^ {2} \leq 9 \left(\frac {(\mathcal {V} + V ^ {2})}{\mathcal {M}}\right) ^ {2},
$$
from which the statement naturally follows.
E.5 Bounding the Estimation Error of Mean Difference
Lemma 6. For any  $\delta >0$ , choose  $b\geq \frac{2}{Tp_{\mathrm{min}}}\log \frac{4d}{\delta}$ . Then, the following holds with probability at least  $1 - \delta$ :
$$
\left\| \widehat {\boldsymbol {f}} - \boldsymbol {f} \right\| _ {2} \leq \mathcal {E} ^ {(f)} \triangleq \sqrt {\frac {8 \mathfrak {c} ^ {2} \sigma^ {2}}{T b p _ {\operatorname* {m i n}}} \log \frac {4 d}{\delta}},
$$
where  $c$
Proof. Again, note that  $(a_{i},\pmb{x}_{i})$  's are i.i.d. samples  $(i\in [Tb])$  . Consider independent random vectors
$$
\boldsymbol {y} _ {i} ^ {(a)} = \boldsymbol {x} _ {i} - \boldsymbol {\mu} _ {a}.
$$
For each  $A \subseteq [Tb]$ ,  $E_A = \{a_i = \mathbb{1}[i \in A] \forall i \in [b]\}$  is an event satisfying  $\mathbb{P}(E_A) = p_0^{Tb - |A|}p_1^{|A|}$ . To exploit this, we now apply the peeling argument as follows: for any  $c_0, c_1 \in (0,1)$  with  $c_0 + c_1 = 1$ ,
$$
\begin{array}{l} \mathbb {P} \left(\left\| \widehat {\boldsymbol {f}} - \boldsymbol {f} \right\| _ {2} \geq x\right) \\ = \sum_ {r = 0} ^ {T b} \sum_ {A \in \binom {[ T b ]} {r}} \mathbb {P} \left(\left\| \widehat {\boldsymbol {f}} - \boldsymbol {f} \right\| _ {2} \geq x \mid E _ {A}\right) \mathbb {P} \left(E _ {A}\right) \\ = \sum_ {r = 1} ^ {T b - 1} \sum_ {A \in \binom {\left(^ {[ T b ]}\right)} {r}} \mathbb {P} \left(\left\| \frac {1}{r} \sum_ {i \in A} \boldsymbol {y} _ {i} ^ {(1)} - \frac {1}{T b - r} \sum_ {i \in A ^ {\mathrm {c}}} \boldsymbol {y} _ {i} ^ {(0)} \right\| _ {2} \geq x \mid E _ {A}\right) \mathbb {P} \left(E _ {A}\right) \\ + \mathbb {P} \left(\left\| \frac {1}{T b} \sum_ {i = 1} ^ {T b} \boldsymbol {y} _ {i} ^ {(0)} \right\| _ {2} \geq x \Bigg | E _ {\emptyset}\right) \mathbb {P} (E _ {\emptyset}) + \mathbb {P} \left(\left\| \frac {1}{T b} \sum_ {i = 1} ^ {T b} \boldsymbol {y} _ {i} ^ {(1)} \right\| _ {2} \geq x \Bigg | E _ {[ T b ]}\right) \mathbb {P} (E _ {[ T b ]}) \\ \leq \sum_{r = 1}^{Tb - 1}\sum_{A\in \binom {[Tb]}{r}}\left\{\mathbb{P}\left(\left\| \frac{1}{Tb - r}\sum_{i\in A^{\mathfrak{c}}} \boldsymbol{y}_{i}^{(0)}\right\|_{2}\geq c_{0}x\bigg|E_{A}\right) + \mathbb{P}\left(\left\| \frac{1}{r}\sum_{i\in A}\boldsymbol{y}_{i}^{(1)}\right\|_{2}\geq c_{1}x\bigg|E_{A}\right)\right\} p_{0}^{Tb - r}p_{1}^{r} \\ + \mathbb {P} \left(\left\| \frac {1}{T b} \sum_ {i = 1} ^ {T b} \boldsymbol {y} _ {i} ^ {(0)} \right\| _ {2} \geq x \mid E _ {\emptyset}\right) p _ {0} ^ {T b} + \mathbb {P} \left(\left\| \frac {1}{T b} \sum_ {i = 1} ^ {T b} \boldsymbol {y} _ {i} ^ {(1)} \right\| _ {2} \geq x \mid E _ {[ T b ]}\right) p _ {1} ^ {T b} \\ \end{array}
$$
Observe that conditioned on the event  $E_{A}$ , both  $\left\{\pmb{y}_i^{(1)}\right\}_{i\in A}$  and  $\left\{\pmb{y}_i^{(0)}\right\}_{i\in A^c}$  are sets of i.i.d. random zero-mean  $d$ -dimensional vectors. In addition, with Assumption 1, we have that for each  $i\in A$ ,  $\pmb{y}_i^{(1)}\in \mathrm{nSG}(\sigma_1)$  (and analogously for each  $i\in A^c$ ). Thus, applying the vector Bernstein inequality (Proposition 2), we obtain the following: for  $x > 0$
$$
\begin{array}{l} \mathbb{P}\left(\left\| \widehat{\boldsymbol{f}} -\boldsymbol {f}\right\|_{2}\geq x\right)\leq 2d\sum_{r = 1}^{Tb - 1}p_{0}^{Tb - r}p_{1}^{r}\sum_{A\in \binom {\left[Tb\right]}{r}}\left\{\exp \left(-\frac{(Tb - r)c_{0}^{2}x^{2}}{\mathfrak{c}^{2}\sigma_{0}^{2}}\right) + \exp \left(-\frac{rc_{1}^{2}x^{2}}{\mathfrak{c}^{2}\sigma_{1}^{2}}\right)\right\} \\ + 2 d p _ {0} ^ {T b} \exp \left(- \frac {T b x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {0} ^ {2}}\right) + 2 d p _ {1} ^ {T b} \exp \left(- \frac {T b x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right) \\ \leq 2 d \sum_ {r = 0} ^ {T b} \binom {T b} {r} p _ {0} ^ {T b - r} p _ {1} ^ {r} \left\{\exp \left(- \frac {(T b - r) c _ {0} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {0} ^ {2}}\right) + \exp \left(- \frac {r c _ {1} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right) \right\} \\ = 2 d \left(p _ {0} + p _ {1} \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right)\right) ^ {T b} + 2 d \left(p _ {1} + p _ {0} \exp \left(- \frac {c _ {0} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right)\right) ^ {T b} <   \delta . \\ \end{array}
$$
It suffices for each of both terms to be bounded by  $\frac{\delta}{2}$ . Let us consider only the first term, as the second term follows from symmetry. Taking the log, we have
$$
\log \left(\left(1 - p _ {1}\right) + p _ {1} \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right)\right) \leq \frac {1}{T b} \log \frac {\delta}{4 d}.
$$
$$
\frac {1}{T b} \log \frac {4 d}{\delta} \leq p _ {1} \left(1 - \exp \left(- \frac {c _ {1} ^ {2} x ^ {2}}{\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}\right)\right).
$$
Using  $y \leq 1 - e^{-2y}$  for  $y \leq 1/2$ , it now suffices to have
$$
\frac {2 \mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}{T b p _ {1} c _ {1} ^ {2}} \log \frac {4 d}{\delta} \leq x ^ {2} \leq \frac {\mathfrak {c} ^ {2} \sigma_ {1} ^ {2}}{c _ {1} ^ {2}}.
$$
Combining this with the other term and, for simplicity (see the footnote on pg. 25), choosing  $c_{0} = c_{1} = 1 / 2$ , we have our desired statement.
E.6 Proof of Theorem 1 - Sample Complexity for PAFO-learnability
Let us fix  $\varepsilon_{\mathrm{f}}, \varepsilon_{\mathrm{o}}, \delta \in (0,1)$ . From the definition of PAFO-learnability (Definition 2), with a large enough sample size, we must guarantee the following with probability at least  $1 - \delta$ :
$$
\operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V}\right) \geq \operatorname {t r} \left(\boldsymbol {V} ^ {\star^ {\intercal}} \boldsymbol {\Sigma} \boldsymbol {V} ^ {\star}\right) - \varepsilon_ {\mathrm {f}}, \quad \| \boldsymbol {\Pi} _ {\boldsymbol {U}} \boldsymbol {V} \| _ {2} \leq \varepsilon_ {\mathrm {o}}.
$$
Let  $\widehat{\pmb{U}}$  be the final estimate of the unfair subspace  $\pmb{U}$  from Algorithm 1, and let  $\pmb{V}$  be the final estimate of  $\pmb{Q}_k = \pmb{V}^\star$ , whose columns are top  $k$  eigenvectors of  $\Pi_U^\perp\Sigma\Pi_U^\perp$ .
We first recall the von Neumann trace inequality (Mirsky, 1975; von Neumann, 1937):
Proposition 3 (Theorem H.1.g of Marshall et al. (2011)). If  $A, B$  are two  $n \times n$  Hermitian matrices, then
$$
\operatorname {t r} (\boldsymbol {A B}) \leq \sum_ {i = 1} ^ {n} \lambda_ {i} (\boldsymbol {A}) \lambda_ {i} (\boldsymbol {B}),
$$
where  $\lambda_{i}(\cdot)$  is the  $i$ -th smallest eigenvalue.
Also, the following lemma implies that the sine angle between subspaces (with the same dimension) and the corresponding Grassmannian (projection) distance are the same.
Lemma 7. Let  $A_1 \in \mathbb{R}^{d \times k}$  and  $B_1 \in \mathbb{R}^{d \times \ell}$  be semi-orthogonal matrices:  $A_1^\top A_1 = I_k$  and  $B_1^\top B_1 = I_\ell$ . Define
$$
\operatorname {d i s t} (\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)) \triangleq \left\| \boldsymbol {\Pi} _ {\boldsymbol {A} _ {1}} - \boldsymbol {\Pi} _ {\boldsymbol {B} _ {1}} \right\| _ {2},
$$
$$
\sin \left(\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)\right) \triangleq \left\| \boldsymbol {\Pi} _ {\boldsymbol {A} _ {1}} ^ {\perp} \boldsymbol {B} _ {1} \right\| _ {2}.
$$
Then, (i) if  $k = \ell$
$$
\begin{array}{l} \operatorname {d i s t} \left(\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)\right) = \sin \left(\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)\right) \\ = \sin \left(\operatorname {c o l} \left(\boldsymbol {B} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {A} _ {1}\right)\right) = \sqrt {1 - \sigma_ {\operatorname* {m i n}} \left(\boldsymbol {A} _ {1} ^ {\intercal} \boldsymbol {B} _ {1}\right) ^ {2}}, \\ \end{array}
$$
where  $\sigma_{\mathrm{min}}(M)$  is the minimum singular value of  $M$ . On the other hand, (ii) if  $k < \ell$
$$
\begin{array}{l} \operatorname {d i s t} \left(\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)\right) = \sin \left(\operatorname {c o l} \left(\boldsymbol {A} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {B} _ {1}\right)\right) = 1 \\ \geq \sin \left(\operatorname {c o l} \left(\boldsymbol {B} _ {1}\right), \operatorname {c o l} \left(\boldsymbol {A} _ {1}\right)\right) = \sqrt {1 - \sigma_ {\min} \left(\boldsymbol {A} _ {1} ^ {\intercal} \boldsymbol {B} _ {1}\right) ^ {2}}. \\ \end{array}
$$
Now we can write the optimality (the first inequality) as follows:
$$
\begin{array}{l} \operatorname {t r} \left(\boldsymbol {Q} _ {k} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {Q} _ {k}\right) - \operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V}\right) = \operatorname {t r} \left(\boldsymbol {\Sigma} \left(\boldsymbol {Q} _ {k} \boldsymbol {Q} _ {k} ^ {\intercal} - \boldsymbol {V} \boldsymbol {V} ^ {\intercal}\right)\right) \\ \stackrel {(a)} {\leq} \sum_ {i = 1} ^ {d} \lambda_ {i} (\pmb {\Sigma}) \lambda_ {i} (\pmb {Q} _ {k} \pmb {Q} _ {k} ^ {\mathsf {T}} - \pmb {V} \pmb {V} ^ {\mathsf {T}}) \\ \stackrel {(b)} {\leq} 2 k \| \boldsymbol {\Sigma} \| _ {2} \| \boldsymbol {Q} _ {k} \boldsymbol {Q} _ {k} ^ {\intercal} - \boldsymbol {V} \boldsymbol {V} ^ {\intercal} \| _ {2} \\ \stackrel {(c)} {\leq} 2 k V \| \Pi_ {V} ^ {\perp} Q _ {k} \| _ {2} \leq \varepsilon_ {\mathrm {f}}. \\ \end{array}
$$
Here,  $(a)$  follows from the von Neumann trace inequality,  $(b)$  follows from the fact that  $Q_{k}Q_{k}^{\mathsf{T}} - VV^{\mathsf{T}}$  is a symmetric matrix of rank at most  $2k$ , and  $(c)$  follows from Lemma 7. Thus, for the optimality, it suffices to ensure that  $\| \Pi_{\overline{V}}^{\perp}Q_{k}\|_{2}\leq \frac{\varepsilon_{f}}{2kV}$ .
Similarly, for the fairness (the second inequality), it suffices to ensure that  $\| \Pi_U\Pi_{\widehat{U}}^\perp \| _2\leq \varepsilon_{\mathrm{o}}$  or  $\| \Pi_{\widehat{U}} - \Pi_U\| _2\leq \varepsilon_{\mathrm{o}}$ , as
$$
\left\| \boldsymbol {\Pi} _ {U} \boldsymbol {V} \right\| _ {2} \stackrel {(*)} {=} \left\| \boldsymbol {\Pi} _ {U} \boldsymbol {\Pi} _ {\widehat {U}} ^ {\perp} \boldsymbol {V} \right\| _ {2} \leq \left\| \boldsymbol {\Pi} _ {U} \boldsymbol {\Pi} _ {\widehat {U}} ^ {\perp} \right\| _ {2} \stackrel {(\star)} {\leq} \left\| \boldsymbol {\Pi} _ {\widehat {U}} - \boldsymbol {\Pi} _ {U} \right\| _ {2} \leq \varepsilon_ {\mathrm {o}},
$$
where  $(\ast)$  follows from  $V = \Pi_{\widehat{U}}^{\perp} V$  (due to the design of Algorithm 4) and  $(\star)$  follows from Lemma 7.
From Lemma 1 (letting  $\mathcal{T} = \Theta\left(\frac{K_{k,\kappa}}{\Delta_{k,\kappa}}\log \frac{d}{\epsilon\delta}\right)$  and 3 (substituting  $\epsilon \mapsto \frac{\varepsilon_{\mathrm{f}}}{2kV}$ ), given that
$$
\left\| \boldsymbol {\Pi} _ {\hat {U}} - \boldsymbol {\Pi} _ {U} \right\| _ {2} \leq \eta_ {k} = \eta_ {k} \left(\varepsilon_ {\mathrm {f}}, \varepsilon_ {\mathrm {o}}, \delta\right) \triangleq \min  \left(\varepsilon_ {\mathrm {o}}, \frac {\Delta_ {k , \kappa} \varepsilon_ {\mathrm {f}}}{4 0 V ^ {2} k}, \frac {\Delta_ {k , \kappa} \delta}{1 6 0 V \sqrt {d k}}\right), \tag {17}
$$
a total of  $N_{2} = \mathcal{T}\mathcal{B}$  samples are sufficient in Algorithm 4 for ensuring  $\varepsilon_{\mathrm{f}}$ -optimality with probability at least  $1 - \frac{\delta}{2}$ , where
$$
N _ {2} \gtrsim \left(\frac {K _ {k , \kappa} (\mathcal {V} + V ^ {2})}{\Delta_ {k , \kappa} ^ {3}} \left(\frac {d k}{\delta^ {2}} \log \frac {k}{\delta} + \frac {k ^ {2} V ^ {2}}{\varepsilon_ {\mathrm {f}} ^ {2}} \log \frac {d}{\delta}\right) + \frac {K _ {k , \kappa} \mathcal {M} ^ {2}}{\Delta_ {k , \kappa} (\mathcal {V} + V ^ {2})} \log \frac {d}{\delta}\right) \log \frac {d k V}{\varepsilon_ {\mathrm {f}} \delta}. \tag {18}
$$
We now focus on obtaining the sample complexity for Algorithm 3 to satisfy Eqn. (17) with probability at least  $1 - \frac{\delta}{2}$ . Then combining those with a union bound gives us the desired statement.
From hereon and forth, we write
$$
\boldsymbol {g} \triangleq \Pi_ {\boldsymbol {P} _ {m}} ^ {\perp} \boldsymbol {f}, \quad \boldsymbol {h} \triangleq \frac {1}{\| \boldsymbol {g} \| _ {2}} \boldsymbol {g},
$$
$$
\widehat {\boldsymbol {g}} \triangleq \Pi_ {\boldsymbol {W} _ {T}} ^ {\perp} \widehat {\boldsymbol {f}}, \quad \widehat {\boldsymbol {h}} \triangleq \frac {1}{\| \widehat {\boldsymbol {g}} \| _ {2}} \widehat {\boldsymbol {g}}.
$$
Also, we introduce the following lemma that provides a general bound of sine angle between a pair of vectors with a small  $\ell_2$ -distance.
Lemma 8. Consider two vectors  $\mathbf{a}, \mathbf{b} \in \mathbb{R}^d$ . If we denote the (acute) angle between  $\mathbf{a}$  and  $\mathbf{b}$  as  $\theta(\mathbf{a}, \mathbf{b})$ ,
$$
\sin \theta (\boldsymbol {a}, \boldsymbol {b}) = \sin (\operatorname {s p a n} (\boldsymbol {a}), \operatorname {s p a n} (\boldsymbol {b})) = \left\| \frac {1}{\| \boldsymbol {a} \| _ {2} ^ {2}} \boldsymbol {a} \boldsymbol {a} ^ {\intercal} - \frac {1}{\| \boldsymbol {b} \| _ {2} ^ {2}} \boldsymbol {b} \boldsymbol {b} ^ {\intercal} \right\| _ {2}.
$$
Consider any  $\epsilon \in \left(0, \frac{\|\pmb{a}\|_2}{2}\right]$  and suppose  $\| \pmb{a} - \pmb{b}\|_{2} \leq \epsilon$ . Then,
$$
\sin \theta (\boldsymbol {a}, \boldsymbol {b}) \leq \frac {\sqrt {2} \epsilon}{\| \boldsymbol {a} \| _ {2}}.
$$
Case 1. Assume  $\pmb{g} \neq \mathbf{0}$ . In this case,  $\pmb{U} = [P_{m}|\pmb{h}]$ ,  $\| \pmb{g} \|_{2} \geq g_{\min}$ , and  $\| \pmb{f} \|_{2} \leq f_{\max}$  by Assumption 1. We want to upper-bound the following probability to be  $< \delta / 2$ :
$$
\begin{array}{l} \mathbb {P} \left[ \left\| \boldsymbol {\Pi} _ {U} \boldsymbol {\Pi} _ {\widehat {U}} ^ {\perp} \right\| _ {2} > \eta_ {k} \right] \\ = \mathbb {P} \left[ \left(\widehat {\boldsymbol {g}} \neq \mathbf {0} \text {a n d} \left\| \boldsymbol {\Pi} _ {\boldsymbol {U}} - \boldsymbol {\Pi} _ {\widehat {\boldsymbol {U}}} \right\| _ {2} > \eta_ {k}\right) \text {o r} \left(\widehat {\boldsymbol {g}} = \mathbf {0} \text {a n d} \left\| \boldsymbol {\Pi} _ {\boldsymbol {U}} \boldsymbol {\Pi} _ {\widehat {\boldsymbol {U}}} ^ {\perp} \right\| _ {2} > \eta_ {k}\right) \right] \\ \leq \mathbb {P} \left[ \left\| \boldsymbol {\Pi} _ {[ \boldsymbol {P} _ {m} | \boldsymbol {h} ]} - \boldsymbol {\Pi} _ {[ \boldsymbol {W} _ {T} | \widehat {\boldsymbol {h}} ]} \right\| _ {2} > \eta_ {k} \Big | \widehat {\boldsymbol {g}} \neq \boldsymbol {0} \right] + \mathbb {P} [ \widehat {\boldsymbol {g}} = \boldsymbol {0} ]. \\ \end{array}
$$
Let us obtain a sample complexity to bound the first probability term to be  $< \frac{\delta}{4}$ . Note that
$$
\left\| \Pi_ {\left[ P _ {m} | \boldsymbol {h} \right]} - \Pi_ {\left[ \boldsymbol {W} _ {T} | \widehat {\boldsymbol {h}} \right]} \right\| _ {2} \leq \| \Pi_ {\boldsymbol {P} _ {m}} - \Pi_ {\boldsymbol {W} _ {T}} \| _ {2} + \left\| \boldsymbol {h} \boldsymbol {h} ^ {\intercal} - \widehat {\boldsymbol {h}} \widehat {\boldsymbol {h}} ^ {\intercal} \right\| _ {2}.
$$
Firstly, to yield
$$
\left\| \boldsymbol {\Pi} _ {\boldsymbol {P} _ {m}} - \boldsymbol {\Pi} _ {\boldsymbol {W} _ {T}} \right\| _ {2} = \left\| \boldsymbol {\Pi} _ {\boldsymbol {W} _ {T}} ^ {\perp} \boldsymbol {P} _ {m} \right\| _ {2} \leq \frac {\eta_ {k}}{2}
$$
with probability at least  $1 - \frac{\delta}{12}$ , it is sufficient that
$$
T = \Theta \left(\frac {K _ {m , \nu}}{\Delta_ {m , \nu}} \log \left(\frac {d}{\eta_ {k} \delta}\right)\right), \tag {19}
$$
$$
b = \Omega \left(\frac {\mathcal {V}}{\Delta_ {m , \nu} ^ {2} p _ {\operatorname* {m i n}}} \left(\frac {d m}{\delta^ {2}} \log \frac {m}{\delta} + \frac {1}{\eta_ {k} ^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} p _ {\operatorname* {m i n}}} \log \frac {d}{\delta}\right), \tag {20}
$$
due to application of Lemma 1 and 2.
Secondly, we aim to make the second term  $\left\| h h^{\mathsf{T}} - \widehat{h} \widehat{h}^{\mathsf{T}} \right\|_{2}$  small with high probability. We first claim that, if  $\| g - \widehat{g} \|_{2} \leq \frac{g_{\min}}{2} \min \left\{ \frac{\eta_{k}}{\sqrt{2}}, 1 \right\}$ , then  $\left\| h h^{\mathsf{T}} - \widehat{h} \widehat{h}^{\mathsf{T}} \right\|_{2} \leq \frac{\eta_{k}}{2}$ . Here is the proof of the claim: Since  $\| g - \widehat{g} \|_{2} \leq \frac{\| g \|_{2}}{2}$ , we have
$$
\left\| \boldsymbol {h} \boldsymbol {h} ^ {\intercal} - \widehat {\boldsymbol {h}} \widehat {\boldsymbol {h}} ^ {\intercal} \right\| _ {2} \stackrel {{\text {L e m . 8}}} {{\leq}} \frac {\sqrt {2}}{\| \boldsymbol {g} \| _ {2}} \| \boldsymbol {g} - \widehat {\boldsymbol {g}} \| _ {2} \leq \frac {\sqrt {2}}{g _ {\min }} \| \boldsymbol {g} - \widehat {\boldsymbol {g}} \| _ {2} \leq \frac {\eta_ {k}}{2}.
$$
Thus, since
$$
\begin{array}{l} \| \boldsymbol {g} - \widehat {\boldsymbol {g}} \| _ {2} = \left\| \left(\Pi_ {P _ {m}} ^ {\perp} - \Pi_ {\boldsymbol {W} _ {T}} ^ {\perp}\right) \boldsymbol {f} + \Pi_ {\boldsymbol {W} _ {T}} ^ {\perp} (\boldsymbol {f} - \widehat {\boldsymbol {f}}) \right\| _ {2} \\ \leq \left\| \boldsymbol {\Pi} _ {\boldsymbol {P} _ {m}} - \boldsymbol {\Pi} _ {\boldsymbol {W} _ {T}} \right\| _ {2} f _ {\max } + \left\| \boldsymbol {f} - \widehat {\boldsymbol {f}} \right\| _ {2}, \tag {21} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} \left[ \left\| \boldsymbol {h} \boldsymbol {h} ^ {\intercal} - \widehat {\boldsymbol {h}} \widehat {\boldsymbol {h}} ^ {\intercal} \right\| _ {2} > \frac {\eta_ {k}}{2} \mid \widehat {\boldsymbol {g}} \neq \mathbf {0} \right] \\ \leq \mathbb {P} \left[ \| \boldsymbol {g} - \widehat {\boldsymbol {g}} \| _ {2} > \frac {g _ {\min }}{2} \min  \left\{\frac {\eta_ {k}}{\sqrt {2}}, 1 \right\} \Big | \widehat {\boldsymbol {g}} \neq \mathbf {0} \right] \\ \leq \mathbb {P} \left[ \| \boldsymbol {\Pi} _ {\boldsymbol {P} _ {m}} - \boldsymbol {\Pi} _ {\boldsymbol {W} _ {T}} \| _ {2} > \frac {g _ {\operatorname* {m i n}}}{4 f _ {\operatorname* {m a x}}} \min  \left\{\frac {\eta_ {k}}{\sqrt {2}}, 1 \right\} \Bigg | \widehat {\boldsymbol {g}} \neq \mathbf {0} \right] + \mathbb {P} \left[ \left\| \boldsymbol {f} - \widehat {\boldsymbol {f}} \right\| _ {2} > \frac {g _ {\operatorname* {m i n}}}{4} \min  \left\{\frac {\eta_ {k}}{\sqrt {2}}, 1 \right\} \Bigg | \widehat {\boldsymbol {g}} \neq \mathbf {0} \right]. \\ \end{array}
$$
$$
\left\| \boldsymbol {\Pi} _ {\boldsymbol {P} _ {m}} - \boldsymbol {\Pi} _ {\boldsymbol {W} _ {T}} \right\| _ {2} \leq \frac {g _ {\min }}{4 f _ {\max }} \min  \left\{\frac {\eta_ {k}}{\sqrt {2}}, 1 \right\}
$$
$$
T = \Theta \left(\frac {K _ {m , \nu}}{\Delta_ {m , \nu}} \log \left(\frac {d}{\eta_ {k} \delta} \frac {f _ {\operatorname* {m a x}}}{g _ {\operatorname* {m i n}}}\right)\right), \tag {22}
$$
$$
b = \Omega \left(\frac {\mathcal {V}}{\Delta_ {m , \nu} ^ {2} p _ {\operatorname* {m i n}}} \left(\frac {d m}{\delta^ {2}} \log \frac {m}{\delta} + \frac {f _ {\operatorname* {m a x}} ^ {2}}{\eta_ {k} ^ {2} g _ {\operatorname* {m i n}} ^ {2}} \log \frac {d}{\delta}\right) + \frac {\mathcal {M} ^ {2}}{\mathcal {V} p _ {\operatorname* {m i n}}} \log \frac {d}{\delta}\right). \tag {23}
$$
Moreover, to have
$$
\left\| \boldsymbol {f} - \widehat {\boldsymbol {f}} \right\| _ {2} \leq \frac {g _ {\operatorname* {m i n}}}{4} \min  \left\{\frac {\eta_ {k}}{\sqrt {2}}, 1 \right\}
$$
$$
T b \geq \frac {1 2 8 \mathfrak {c} ^ {2} \sigma^ {2}}{p _ {\min } g _ {\min } ^ {2}} \max  \left\{\frac {2}{\eta_ {k} ^ {2}}, 1 \right\} \log \frac {4 8 d}{\delta}. \tag {24}
$$
Next, we obtain a sample complexity to bound the second probability term  $\mathbb{P}[\widehat{\pmb{g}} = \mathbf{0}] < \frac{\delta}{4}$ . Because of Eqn. (21),