SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Pr \left(\bigcup_ {1 \leq i, j \leq n} \left\{\left| \mathcal {E} _ {i} \cap \mathcal {A} _ {j} \right| - n p q ^ {2} \right| \geq \epsilon n p q ^ {2} \}\right) \leq 2 n ^ {2} \exp \left(- \frac {3}{8} \epsilon^ {2} n p q ^ {2}\right) \leq \frac {2}{n}, \tag {27}
$$
$$
\Pr \left(\bigcup_ {1 \leq i, j \leq n} \left\{\left| \mathcal {A} _ {i j} \right| - n p ^ {2} q ^ {2} \right| \geq \epsilon n p ^ {2} q ^ {2} \}\right) \leq 2 n ^ {2} \exp \left(- \frac {3}{8} \epsilon^ {2} n p ^ {2} q ^ {2}\right) \leq \frac {2}{n}. \tag {28}
$$
Hence, we complete the proof of Lemma 1 once  $n \geq 4$ .
We can first compute
$$
\begin{array}{l} \sum_ {1 \leq i, j \leq n} \| X _ {i} - X _ {j} \| _ {F} \leq \sum_ {1 \leq i, j \leq n} \frac {1}{| \mathcal {A} _ {i j} |} \sum_ {k \in \mathcal {A} _ {i j}} \left(\| X _ {i} - X _ {k} \| _ {F} + \| X _ {j} - X _ {k} \| _ {F}\right) \tag {29} \\ \leq \frac {1}{(1 - \epsilon) n p ^ {2} q ^ {2}} \sum_ {1 \leq i, j \leq n} \sum_ {k \in \mathcal {A} _ {i j}} \left(\| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {k} \| _ {F} + \| \boldsymbol {X} _ {j} - \boldsymbol {X} _ {k} \| _ {F}\right). \\ \end{array}
$$
Here, the first inequality comes from the triangle inequality, while the second one follows from Lemma 1. Now, invoking Lemma 1, which tells  $|\mathcal{A}_k| \leq (1 + \epsilon)npq$ , gives
$$
\begin{array}{l} \sum_ {1 \leq i, j \leq n} \sum_ {k \in \mathcal {A} _ {i j}} \| X _ {i} - X _ {k} \| _ {F} = \sum_ {1 \leq i \leq n} \sum_ {k \in \mathcal {A} _ {i}} \sum_ {j \in \mathcal {A} _ {k}} \| X _ {i} - X _ {k} \| _ {F} \\ \leq (1 + \epsilon) n p q \sum_ {1 \leq i \leq n} \sum_ {k \in \mathcal {A} _ {i}} \| X _ {i} - X _ {k} \| _ {F} \\ \end{array}
$$
$$
= (1 + \epsilon) n p q \sum_ {(i, k) \in \mathcal {A}} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {k} \| _ {F}.
$$
By symmetry, we conclude that
$$
\sum_ {1 \leq i, j \leq n} \sum_ {k \in \mathcal {A} _ {i j}} \left(\| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {k} \| _ {F} + \| \boldsymbol {X} _ {j} - \boldsymbol {X} _ {k} \| _ {F}\right) \leq 2 (1 + \epsilon) n p q \sum_ {(i, j) \in \mathcal {A}} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {j} \| _ {F}. \tag {30}
$$
Furthermore, we claim the following bound for any  $X \in \mathrm{SO}(d)^n$ :
$$
\sum_ {1 \leq i, j \leq n} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {j} \| _ {F} \geq \frac {n}{2} \operatorname {d i s t} _ {1} (\boldsymbol {X}, \boldsymbol {X} ^ {\star}). \tag {31}
$$
Combining (29), (30), (31), and the fact  $g(\pmb{X}) = \sum_{(i,j) \in \mathcal{A}} \| \pmb{X}_i - \pmb{X}_j \|_F$  establishes Proposition 1.
Hence, it remains to show (31). First of all, according to the triangle inequality, we have
$$
\sum_ {1 \leq i, j \leq n} \| X _ {i} - X _ {j} \| _ {F} \geq \sum_ {1 \leq i \leq n} \left\| n X _ {i} - \sum_ {1 \leq j \leq n} X _ {j} \right\| _ {F} = n \sum_ {1 \leq i \leq n} \| X _ {i} - \bar {X} \| _ {F}, \tag {32}
$$
where  $\overline{X} = \frac{1}{n}\sum_{1\leq j\leq n}X_{j}$  can be taken as a diagonal matrix (since the Frobenius norm is invariant up to a global rotation) with its first  $(d - 1)$  diagonal entries being positive. Finally, applying the following lemma to (32) provides (31).
Lemma 7. For any  $\mathbf{A} \in \mathrm{SO}(d)$  and  $\mathbf{B} = \operatorname{Diag}(b_1, \ldots, b_d)$  satisfying  $b_1, \ldots, b_{d-1} \in [0,1]$  and  $b_d \in [-1,1]$ , we have
$$
\| \boldsymbol {A} - \boldsymbol {B} \| _ {F} \geq \frac {1}{2} \| \boldsymbol {A} - \boldsymbol {I} \| _ {F}.
$$
Proof. It is equivalent to show that  $\| \pmb{A} - \pmb{B}\| _F^2\geq \frac{1}{4}\| \pmb {A} - \pmb {I}\| _F^2$  . Since  $A\in \mathrm{SO}(d)$  , we simplify as
$$
d + \langle \boldsymbol {A}, \boldsymbol {I} - 4 \boldsymbol {B} \rangle + 2 \| \boldsymbol {B} \| _ {F} ^ {2} = \sum_ {1 \leq i \leq d} 1 + \boldsymbol {A} _ {i i} (1 - 4 b _ {i}) + 2 b _ {i} ^ {2} \geq 0. \tag {33}
$$
To prove (33) holds for all  $\mathbf{A} \in \mathrm{SO}(d)$ , we choose  $\bar{\mathbf{A}} = \operatorname*{argmin}_{\mathbf{A} \in \mathrm{SO}(d)} \langle \mathbf{A}, \mathbf{I} - 4\mathbf{B} \rangle = \mathcal{P}_{\mathrm{SO}(d)}(4\mathbf{B} - \mathbf{I})$ . It is easy to see that  $\bar{\mathbf{A}}$  is also a diagonal matrix. For any  $1 \leq i \leq d - 1$ , we have
$$
1 + \boldsymbol {A} _ {i i} (1 - 4 b _ {i}) + 2 b _ {i} ^ {2} = \left\{ \begin{array}{l l} 2 (b _ {i} - 1) ^ {2}, & \boldsymbol {A} _ {i i} = 1; \\ 2 b _ {i} ^ {2} + 4 b _ {i}, & \boldsymbol {A} _ {i i} = - 1. \end{array} \right.
$$
So it is always nonnegative since  $b_{i} \geq 0$  for any  $1 \leq i \leq d - 1$ . For the last summation in (33), on the one hand, if  $\bar{A}_{dd} = 1$ , then  $1 + A_{dd}(1 - 4b_d) + 2b_d^2 = 2(b_d - 1)^2 \geq 0$ . On the other hand, if  $\bar{A}_{dd} = -1$ , then there exist another  $1 \leq k \leq d - 1$  such that  $\bar{A}_{kk} = -1$ . So we have  $(1 + A_{kk}(1 - 4b_k) + 2b_k^2) + (1 + A_{dd}(1 - 4b_d) + 2b_d^2) = 2b_k^2 + 2b_d^2 + 4(b_k + b_d) \geq 0$ . The last inequality holds because  $|b_d| \leq b_k$ . We complete the proof.
Based on (11) and our simplification that  $X_{i}^{\star} = I$ ,  $1 \leq i \leq n$ , we have
$$
h (\boldsymbol {X}) - h \left(\boldsymbol {X} ^ {\star}\right) \geq \sum_ {(i, j) \in \mathcal {A} ^ {c}} \left\langle \frac {\boldsymbol {I} - \boldsymbol {O} _ {i j}}{\| \boldsymbol {I} - \boldsymbol {O} _ {i j} \| _ {F}}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \right\rangle . \tag {34}
$$
Let  $\mathbf{A} = \mathbb{E}\left\{\frac{\mathbf{I} - \mathbf{O}_{ij}}{\|\mathbf{I} - \mathbf{O}_{ij}\|_F}\right\}, \mathbf{Z}_{ij} = \frac{\mathbf{I} - \mathbf{O}_{ij}}{\|\mathbf{I} - \mathbf{O}_{ij}\|_F} \cdot \mathbf{1}_{\mathcal{A}^c}(ij) - (1 - p)q\mathbf{A}$ , and  $\mathbf{Z} \in \mathbb{R}^{nd \times nd}$  collects each  $\mathbf{Z}_{ij}$  in its  $(i,j)$ -th block. We can compute
$$
\begin{array}{l} \left| \sum_ {(i, j) \in \mathcal {A} ^ {\mathrm {c}}} \left\langle \frac {\boldsymbol {I} - \boldsymbol {O} _ {i j}}{\| \boldsymbol {I} - \boldsymbol {O} _ {i j} \| _ {F}}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \right\rangle \right| = \left| \sum_ {1 \leq i, j \leq n} \left\langle \boldsymbol {Z} _ {i j} + (1 - p) q \boldsymbol {A}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \right\rangle \right| \\ \leq \left| \sum_ {1 \leq i, j \leq n} \left\langle (1 - p) q \boldsymbol {A}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \right\rangle \right| \tag {35} \\ \end{array}
$$
$$
+ \left| \sum_ {1 \leq i, j \leq n} \left\langle \boldsymbol {Z} _ {i j}, (\boldsymbol {X} _ {i} - \boldsymbol {I}) ^ {\top} (\boldsymbol {X} _ {j} - \boldsymbol {I}) + (\boldsymbol {X} _ {i} - \boldsymbol {I}) ^ {\top} + (\boldsymbol {X} _ {j} - \boldsymbol {I}) \right\rangle \right|.
$$
To bound the first term in (35), we can proceed as
$$
\begin{array}{l} \left| \sum_ {1 \leq i, j \leq n} \left\langle (1 - p) q \boldsymbol {A}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \right\rangle \right| = \left| \left\langle (1 - p) q \boldsymbol {A}, \left(\sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i}\right) ^ {\top} \left(\sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i}\right) - n ^ {2} \boldsymbol {I} \right\rangle \right| \\ = \left| \left\langle (1 - p) q \boldsymbol {A}, \left(\sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} + n \boldsymbol {I}\right) ^ {\top} \left(\sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} - n \boldsymbol {I}\right) \right\rangle \right| \\ \leq \left\| (1 - p) q \boldsymbol {A} \left(\sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} + n \boldsymbol {I}\right) \right\| _ {F} \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} - n \boldsymbol {I} \right\| _ {F} \\ \leq \| (1 - p) q \boldsymbol {A} \| _ {F} \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} + n \boldsymbol {I} \right\| _ {2} \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} - n \boldsymbol {I} \right\| _ {F} \\ \leq 2 (1 - p) q n \| \boldsymbol {A} \| _ {F} \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} - n \boldsymbol {I} \right\| _ {F}. \tag {36} \\ \end{array}
$$
Here, the second equality is true since  $\sum_{i=1}^{n} X_i$  can be taken as a diagonal matrix. According to [39, Lemma A.1], we know that  $\| \mathbf{A} \|_F = \left\| \mathbb{E}\left\{\frac{\mathbf{I} - \mathbf{O}_{ij}}{\|\mathbf{I} - \mathbf{O}_{ij}\|_F}\right\} \right\|_F \leq \frac{1}{\sqrt{2}}$  for all  $d \geq 2$ . On the other hand, we know that
$$
\left. \operatorname {d i s t} \left(\boldsymbol {X}, \boldsymbol {X} ^ {\star}\right) ^ {2} = \sum_ {i = 1} ^ {n} \| \boldsymbol {X} _ {i} - \boldsymbol {I} \| _ {F} ^ {2} = 2 \operatorname {t r a c e} \left(n \boldsymbol {I} - \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i}\right) \geq 2 \left\| \sum_ {i = 1} ^ {n} \boldsymbol {X} _ {i} - n \boldsymbol {I} \right\| _ {F}, \right. \tag {37}
$$
where the last inequality holds because  $n\mathbf{I} - \sum_{i=1}^{n}\mathbf{X}_i$  is a nonnegative diagonal matrix. Therefore, we obtain
$$
\begin{array}{l} \left| \sum_ {1 \leq i, j \leq n} \langle (1 - p) q \boldsymbol {A}, \boldsymbol {X} _ {i} ^ {\top} \boldsymbol {X} _ {j} - \boldsymbol {I} \rangle \right| \leq \frac {(1 - p) q n}{\sqrt {2}} \operatorname {d i s t} (\boldsymbol {X}, \boldsymbol {X} ^ {\star}) ^ {2} \tag {38} \\ \leq \frac {(1 - p) q n}{\sqrt {2}} \max  _ {i} \| \boldsymbol {X} _ {i} - \boldsymbol {I} \| _ {F} \sum_ {i = 1} ^ {n} \| \boldsymbol {X} _ {i} - \boldsymbol {I} \| _ {F}. \\ \end{array}
$$
To further bound the second term in (35), we can proceed as
$$
\begin{array}{l} \left| \sum_ {1 \leq i, j \leq n} \left\langle \mathbf {Z} _ {i j}, (\mathbf {X} _ {i} - \mathbf {I}) ^ {\top} (\mathbf {X} _ {j} - \mathbf {I}) + (\mathbf {X} _ {i} - \mathbf {I}) ^ {\top} + (\mathbf {X} _ {j} - \mathbf {I}) \right\rangle \right| \\ \leq \left(\boldsymbol {X} - \boldsymbol {I} ^ {n}\right) \boldsymbol {Z} \left(\boldsymbol {X} - \boldsymbol {I} ^ {n}\right) ^ {\top} + 2 \left| \sum_ {1 \leq i \leq n} \left\langle \sum_ {1 \leq j \leq n} \boldsymbol {Z} _ {i j}, \boldsymbol {X} _ {i} - \boldsymbol {I} \right\rangle \right| \tag {39} \\ \leq \| \boldsymbol {Z} \| _ {o p} \| \boldsymbol {X} - \boldsymbol {I} ^ {n} \| _ {F} ^ {2} + 2 \sum_ {1 \leq i \leq n} \left\| \sum_ {1 \leq j \leq n} \boldsymbol {Z} _ {i j} \right\| _ {F} \| \boldsymbol {X} _ {i} - \boldsymbol {I} \| _ {F} \\ \leq \left(2 \sqrt {d} \| Z \| _ {\mathrm {o p}} + 2 \max  _ {1 \leq i \leq n} \left\| \sum_ {1 \leq j \leq n} Z _ {i j} \right\| _ {F}\right) \sum_ {1 \leq i \leq n} \| X _ {i} - I \| _ {F}, \\ \end{array}
$$
where  $\pmb{I}^n\in \mathbb{R}^{nd\times d}$  collects  $n$  identity matrix together and the last inequality holds because  $\| X - I^n\| _F^2 = \sum_{1\leq i\leq n}\| X_i - I\| _F^2\leq \sum_{1\leq i\leq n}(\| X_i\| _F + \| I\| _F)\| X_i - I\| _F = 2\sqrt{d}\sum_{1\leq i\leq n}\| X_i - I\| _F.$
Using the randomness of  $O_{ij}$ , we claim that with probability at least  $1 - 4d / n$ , we have
$$
\| \boldsymbol {Z} \| _ {\mathrm {o p}} \leq \sqrt {8 n (1 - p) q \log n}, \quad \text {a n d} \quad \max  _ {1 \leq i \leq n} \left\| \sum_ {1 \leq j \leq n} \boldsymbol {Z} _ {i j} \right\| _ {F} \leq \sqrt {8 n (1 - p) q \log n}. \tag {40}
$$
Combining the above bounds gives
$$
\begin{array}{l} h (\boldsymbol {X}) - h \left(\boldsymbol {X} ^ {\star}\right) \\ \geq - \left(2 (\sqrt {d} + 1) \sqrt {8 n (1 - p) q \log n} + \frac {(1 - p) q n}{\sqrt {2}} \max  _ {i} \| X _ {i} - I \| _ {F}\right) \sum_ {1 \leq i \leq n} \| X _ {i} - I \| _ {F} \tag {41} \\ \geq - \frac {n p q}{1 6} \cdot \operatorname {d i s t} _ {1} (\boldsymbol {X}, \boldsymbol {X} ^ {\star}), \\ \end{array}
$$
where the last inequality holds because we assume  $p^2 q^2 = \Omega \left( \frac{\log n}{n} \right)$ ,  $\max_i \| X_i - I \|_F = \mathrm{dist}_\infty(X, X^\star) = \mathcal{O}(p)$ , and  $\sum_{1 \leq i \leq n} \| X_i - I \|_F = \mathrm{dist}_1(X, X^\star)$ .
Finally, it remains to show that the two inequalities in (40) holds with probability at least  $1 - 4d / n$ . It is quick to verify that  $\mathbb{E}(\pmb{Z}_{ij}) = \mathbf{0}$ ,  $\| \pmb{Z}_{ij}\|_{op} \leq 1 + (1 - p)q\|\pmb{A}\|_F \leq 2$ , and
$$
\begin{array}{l} \mathbb {E} \left(\boldsymbol {Z} ^ {2}\right) = \operatorname {B l k D i a g} \left(\sum_ {j} \mathbb {E} \left(\boldsymbol {Z} _ {1 j} \boldsymbol {Z} _ {1 j} ^ {\top}\right), \dots , \sum_ {j} \mathbb {E} \left(\boldsymbol {Z} _ {n j} \boldsymbol {Z} _ {n j} ^ {\top}\right)\right) \\ = \sum_ {j} \mathbb {E} \left(\boldsymbol {Z} _ {1 j} \boldsymbol {Z} _ {1 j} ^ {\top}\right) \otimes \boldsymbol {I} _ {n} \\ = n (1 - p) q \mathbb {E} \left(\frac {\left(\boldsymbol {I} - \boldsymbol {O} _ {i j}\right) \left(\boldsymbol {I} - \boldsymbol {O} _ {i j}\right) ^ {\top}}{\| \boldsymbol {I} - \boldsymbol {O} _ {i j} \| _ {F} ^ {2}} - (1 - p) ^ {2} q ^ {2} \boldsymbol {A} \boldsymbol {A} ^ {\top}\right) \otimes \boldsymbol {I} _ {n}, \\ \end{array}
$$
where  $\mathrm{BlkDiag}(\cdot)$  means the block diagonal matrix,  $\otimes$  means the Kronecker product and  $I_{n}$  denotes the  $n$ -by- $n$  identity matrix. Thus, we have  $\| \mathbb{E}(Z^2)\|_{op} \leq n(1 - p)q$ . According to the Matrix Bernstein inequality [38], we have
$$
\begin{array}{l} \Pr \left(\| Z \| _ {\mathrm {o p}} \leq \sqrt {8 n (1 - p) q \log n}\right) \geq 1 - 2 n d \exp \left(\frac {- 4 n (1 - p) q \log n}{n (1 - p) q + 2 \sqrt {8 n (1 - p) q \log n} / 3}\right) \\ \geq 1 - \frac {2 d}{n}. \tag {42} \\ \end{array}
$$
Here, the last inequality holds because we assume  $2\sqrt{8n(1 - p)q\log n} /3 \leq n(1 - p)q$ , which is true as long as  $p = \Omega (\log n / n)$ .
For any fixed  $1 \leq i \leq n$ , a similar argument based on Matrix Bernstein inequality [38] shows that
$$
\begin{array}{l} \Pr \left(\| \sum_ {j} Z _ {i j} \| _ {F} \leq \sqrt {8 n (1 - p) q \log n}\right) \geq 1 - 2 d \exp \left(\frac {- 4 n (1 - p) q \log n}{n (1 - p) q + \sqrt {8 n (1 - p) q \log n} / 3}\right) \\ \geq 1 - \frac {2 d}{n ^ {2}}, \tag {43} \\ \end{array}
$$
which implies  $\operatorname*{Pr}\left(\max_i\| \sum_j\mathbf{Z}_{ij}\| _F\leq \sqrt{8n(1 - p)q\log n}\right)\geq 1 - 2d / n.$
C Full Proof of Theorem 4
C.1 Proof of Corollary 1
Combining the convexity of  $f$  and Theorem 3, we have
$$
- \frac {n p q}{8} \sum_ {1 \leq i \leq n} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F} \geq f (\boldsymbol {X} ^ {\star}) - f (\boldsymbol {X}) \geq \left\langle \widetilde {\nabla} f (\boldsymbol {X}), \boldsymbol {X} ^ {\star} - \boldsymbol {X} \right\rangle , \quad \forall \widetilde {\nabla} f (\boldsymbol {X}) \in \partial f (\boldsymbol {X}). \tag {44}
$$
for all  $\mathbf{X} \in \mathrm{SO}(d)^n$  satisfying  $\mathrm{dist}_{\infty}(\mathbf{X}, \mathbf{X}^{\star}) = \mathcal{O}(p)$ . For any  $\widetilde{\nabla}_{\mathcal{R}}^{\perp} f(\mathbf{X}) \in \partial_{\mathcal{R}}^{\perp} f(\mathbf{X})$ , we can further compute
$$
\begin{array}{l} \left\langle \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (\boldsymbol {X}), \boldsymbol {X} - \boldsymbol {X} ^ {\star} \right\rangle = \sum_ {1 \leq i \leq n} \left\langle \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (\boldsymbol {X} _ {i}), \mathcal {P} _ {\mathrm {T} _ {\overline {{\boldsymbol {X}}} _ {i}}} (\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}) \right\rangle \\ \leq \sum_ {1 \leq i \leq n} \| \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (X _ {i}) \| _ {F} \cdot \| \mathcal {P} _ {\mathrm {T} _ {X _ {i}} ^ {\perp}} (X _ {i} - X _ {i} ^ {\star}) \| _ {F}. \\ \end{array}
$$
On the one hand, notice that
$$
\begin{array}{l} \mathcal {P} _ {\mathrm {T} _ {\pmb {X} _ {i}} ^ {\perp}} (\pmb {X} _ {i} - \pmb {X} _ {i} ^ {\star}) = \pmb {X} _ {i} - \pmb {X} _ {i} ^ {\star} - \mathcal {P} _ {\mathrm {T} _ {\pmb {X} _ {i}}} (\pmb {X} _ {i} - \pmb {X} _ {i} ^ {\star}) \\ = \boldsymbol {X} _ {i} \left(\boldsymbol {X} _ {i} ^ {\top} \left(\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}\right) + \left(\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}\right) ^ {\top} \boldsymbol {X} _ {i}\right) / 2 \\ = \boldsymbol {X} _ {i} \left(\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}\right) ^ {\top} \left(\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}\right) / 2, \\ \end{array}
$$
$$
\| \mathcal {P} _ {\mathrm {T} _ {\boldsymbol {X}} ^ {\perp}} (\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}) \| _ {F} = \frac {1}{2} \| \boldsymbol {X} _ {i} (\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}) ^ {\top} (\boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star}) \| _ {F} \leq \frac {1}{2} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F} ^ {2}.
$$
On the other hand, according to Lemma 1, with probability at least  $1 - \mathcal{O}(1 / n)$ , for any  $i\in [n]$
$$
\| \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (\boldsymbol {X} _ {i}) \| _ {F} \leq \| \widetilde {\nabla} f (\boldsymbol {X} _ {i}) \| _ {F} = \left\| \sum_ {j \in \mathcal {E} _ {i}} \widetilde {\nabla} f _ {i, j} (\boldsymbol {X} _ {i}) \right\| _ {F} \leq \sum_ {j \in \mathcal {E} _ {i}} \left\| \widetilde {\nabla} f _ {i, j} (\boldsymbol {X} _ {i}) \right\| _ {F} \leq | \mathcal {E} _ {i} | \leq 2 n q,
$$
where  $\widetilde{\nabla} f_{i,j}(\pmb{X}_i) \in \partial f_{i,j}(\pmb{X}_i)$  satisfies  $\| \widetilde{\nabla} f_{i,j}(\pmb{X}_i)\|_F \leq 1$ . Hence, we have
$$
\left\langle \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (\boldsymbol {X}), \boldsymbol {X} - \boldsymbol {X} ^ {\star} \right\rangle \leq n q \sum_ {1 \leq i \leq n} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F} ^ {2} \leq \frac {n p q}{1 6} \sum_ {i} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F}
$$
for any  $X$  such that  $\mathrm{dist}_{\infty}(\pmb {X},\pmb{X}^{\star})\leq \frac{p}{16}$ . Invoking the above bounds into (44) yields the desired result
$$
\left\langle \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X}), \boldsymbol {X} - \boldsymbol {X} ^ {\star} \right\rangle = \left\langle \widetilde {\nabla} f (\boldsymbol {X}) - \widetilde {\nabla} _ {\mathcal {R}} ^ {\perp} f (\boldsymbol {X}), \boldsymbol {X} - \boldsymbol {X} ^ {\star} \right\rangle \geq \frac {n p q}{1 6} \sum_ {1 \leq i \leq n} \| \boldsymbol {X} _ {i} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F}.
$$
C.2 Proof of Contraction
Let us first present some preliminary results, which will be used in our later derivations. By noticing that  $\widetilde{\nabla} f(\mathbf{X}_i) = \sum_{j\in \mathcal{E}_i}\widetilde{\nabla} f_{i,j}(\mathbf{X}_i)$  where  $\widetilde{\nabla} f_{i,j}(\mathbf{X}_i)\in \partial f_{i,j}(\mathbf{X}_i)$ , we define
$$
\widetilde {\nabla} g (\boldsymbol {X} _ {i}) = \sum_ {j \in \mathcal {A} _ {i}} \widetilde {\nabla} f _ {i, j} (\boldsymbol {X} _ {i}), \quad \widetilde {\nabla} h (\boldsymbol {X} _ {i}) = \sum_ {j \in \mathcal {E} _ {i} \backslash \mathcal {A} _ {i}} \widetilde {\nabla} f _ {i, j} (\boldsymbol {X} _ {i}).
$$
Recall that  $\widetilde{\nabla}_{\mathcal{R}}g(\boldsymbol{X}_i) = \mathcal{P}_{\mathrm{T}_{\boldsymbol{X}_i}}(\widetilde{\nabla} g(\boldsymbol{X}_i))$  and  $\widetilde{\nabla}_{\mathcal{R}}h(\boldsymbol{X}_i) = \mathcal{P}_{\mathrm{T}_{\boldsymbol{X}_i}}(\widetilde{\nabla} h(\boldsymbol{X}_i))$ . Similarly, we have  $\widetilde{\nabla} f(\boldsymbol{X}_i) = \widetilde{\nabla} g(\boldsymbol{X}_i) + \widetilde{\nabla} h(\boldsymbol{X}_i)$  and  $\widetilde{\nabla}_{\mathcal{R}}f(\boldsymbol{X}_i) = \widetilde{\nabla}_{\mathcal{R}}g(\boldsymbol{X}_i) + \widetilde{\nabla}_{\mathcal{R}}h(\boldsymbol{X}_i)$ . Furthermore, the QR decomposition-based retraction satisfies the second-order boundedness property, i.e., there exists some  $M \geq 1$  such that
$$
\begin{array}{l} \left\| \boldsymbol {X} _ {i} ^ {k + 1} - \boldsymbol {X} _ {i} ^ {\star} \right\| _ {F} = \left\| \operatorname {R e t r} _ {\boldsymbol {X} _ {i} ^ {k}} \left(- \mu_ {k} \widetilde {\nabla} _ {\mathcal {R}} f \left(\boldsymbol {X} _ {i} ^ {k}\right)\right) - \boldsymbol {X} _ {i} ^ {\star} \right\| _ {F} \tag {45} \\ \leq \left\| \boldsymbol {X} _ {i} ^ {k} - \mu_ {k} \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} _ {i} ^ {k}) - \boldsymbol {X} _ {i} ^ {\star} \right\| _ {F} + M \cdot \mu_ {k} ^ {2} \left\| \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} _ {i} ^ {k}) \right\| _ {F} ^ {2}. \\ \end{array}
$$
Recall that  $\widetilde{\nabla}_{\mathcal{R}}f(\pmb{X}_i^k) = \widetilde{\nabla}_{\mathcal{R}}g(\pmb{X}_i^k) + \widetilde{\nabla}_{\mathcal{R}}h(\pmb{X}_i^k)$ , we have
$$
\begin{array}{l} \| \widetilde {\nabla} _ {\mathcal {R}} f \left(\boldsymbol {X} _ {i} ^ {k}\right) \| _ {F} \leq \| \widetilde {\nabla} _ {\mathcal {R}} g \left(\boldsymbol {X} _ {i} ^ {k}\right) \| _ {F} + \| \widetilde {\nabla} _ {\mathcal {R}} h \left(\boldsymbol {X} _ {i} ^ {k}\right) \| _ {F} \tag {46} \\ \leq 5 \sqrt {2 \delta_ {0}} n q + (1 + \epsilon) n p q \leq (1 + 2 \epsilon) n p q, \\ \end{array}
$$
where the second inequality comes from
$$
\| \widetilde {\nabla} _ {\mathcal {R}} g (\boldsymbol {X} _ {i} ^ {k}) \| _ {F} = \left\| \sum_ {j \in \mathcal {A} _ {i}} \widetilde {\nabla} _ {\mathcal {R}} f _ {i, j} (\boldsymbol {X} _ {i}) \right\| _ {F} \leq \sum_ {j \in \mathcal {A} _ {i}} \left\| \widetilde {\nabla} _ {\mathcal {R}} f _ {i, j} (\boldsymbol {X} _ {i}) \right\| _ {F} \leq | \mathcal {A} _ {i} | \leq (1 + \epsilon) n p q,
$$
and Lemma 10 and the last inequality is due to the choice  $\delta_0\leq \epsilon^2 p^2 /50$  (i.e.,  $\delta_0 = \mathcal{O}(p^2)$ ). Thus, by choosing  $\mu_{k} = \mathcal{O}\bigl (\frac{\delta_{k}}{n}\bigr)$ , and  $\delta_0 = \mathcal{O}(p^2)$ , the second-order term  $M\cdot \mu_k^2\| \widetilde{\nabla}_{\mathcal{R}}f(\boldsymbol {X}_i^k)\| _F^2 = \mathcal{O}(p^6 q^2)$  which is a very high-order error. In the following analysis, we will ignore this term to simplify our derivations.
Using the above preliminaries and Corollary 1, we are ready to establish two key lemmas, which show that if  $\pmb{X}^k \in \mathcal{N}_F^k \cap \mathcal{N}_\infty^k$ , then  $\pmb{X}^{k+1} \in \mathcal{N}_F^{k+1}$  (Lemma 8) and  $\pmb{X}^{k+1} \in \mathcal{N}_\infty^{k+1}$  (Lemma 9), respectively. This completes the proof of Theorem 4.
Lemma 8. With high probability, suppose that  $\pmb{X}^{k} \in \mathcal{N}_{F}^{k} \cap \mathcal{N}_{\infty}^{k}$ ,  $\mu_{k} = \mathcal{O}\left(\frac{\delta_{k}}{n}\right)$ , and
$$
\delta_ {0} = \mathcal {O} \left(p ^ {2}\right), \quad a n d \quad \xi_ {0} = \Theta (\sqrt {n p q} \delta_ {0}), \tag {47}
$$
then  $X^{k + 1}\in \mathcal{N}_F^{k + 1}$
Proof. By ignoring the high-order error term in (45), in order to bound  $\| \pmb{X}^{k + 1} - \pmb{X}^{\star}\|_{F}^{2}$  we can first compute
$$
\begin{array}{l} \left\| \boldsymbol {X} ^ {k + 1} - \boldsymbol {X} ^ {\star} \right\| _ {F} ^ {2} = \sum_ {1 \leq i \leq n} \left\| \boldsymbol {X} _ {i} ^ {k + 1} - \boldsymbol {X} _ {i} ^ {\star} \right\| _ {F} ^ {2} \leq \sum_ {1 \leq i \leq n} \left\| \boldsymbol {X} _ {i} ^ {k} - \mu_ {k} \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} _ {i} ^ {k}) - \boldsymbol {X} _ {i} ^ {\star} \right\| _ {F} ^ {2} \\ = \| \boldsymbol {X} ^ {k} - \boldsymbol {X} ^ {\star} \| _ {F} ^ {2} - 2 \mu_ {k} \left\langle \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} ^ {k}), \boldsymbol {X} ^ {k} - \boldsymbol {X} ^ {\star} \right\rangle + \mu_ {k} ^ {2} \| \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} ^ {k}) \| _ {F} ^ {2}. \\ \end{array}
$$
Then, according to Corollary 1, we have
$$
\begin{array}{l} \left\langle \widetilde {\nabla} _ {\mathcal {R}} f (\boldsymbol {X} ^ {k}), \boldsymbol {X} ^ {k} - \boldsymbol {X} ^ {\star} \right\rangle \geq \frac {n p q}{1 6} \sum_ {1 \leq i \leq n} \| \boldsymbol {X} _ {i} ^ {k} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F} \geq \frac {n p q}{1 6 \delta_ {k}} \sum_ {1 \leq i \leq n} \| \boldsymbol {X} _ {i} ^ {k} - \boldsymbol {X} _ {i} ^ {\star} \| _ {F} ^ {2} \\ = \frac {n p q}{1 6 \delta_ {k}} \| \boldsymbol {X} ^ {k} - \boldsymbol {X} ^ {\star} \| _ {F} ^ {2}, \\ \end{array}
$$
where the second inequality holds because  $\| X_i^k - X_i^\star \|_F \leq \delta_k$  (i.e.,  $X^k \in \mathcal{N}_\infty^k$ ). Combining the above two inequalities gives