SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The remaining term  $2\eta R\langle G_t, D \rangle + \langle D, D \rangle$  might be negative and cancel some of our progress. A natural attempt is to try to bound it using  $2\eta R\langle G_t, D \rangle + \langle D, D \rangle \geq -\eta^2 R^2\langle G_t, G_t \rangle$ . However, the wRIP condition of  $\mathcal{A}$  does not provide any guarantee on  $\|G_t\|_F^2$ . (In fact, we can derive that  $\|G\|_2 \leq L$  from wRIP, but the best we can hope for is  $\|G\|_F^2 \leq \mathrm{rank}(G) \cdot \|G\|_2^2$  where  $\mathrm{rank}(G) \leq d$ .) This motivates the decomposition property in the dRIP Condition 2.3 and in the weight oracle. The idea is that even though we cannot directly bound  $\|G_t\|_F$ , we can in fact lower bound  $2\eta R\langle G_t, D \rangle + \langle D, D \rangle$  the term by decomposing  $G_t$  into a Frobenius-norm-bounded matrix  $S_t$ , and an operator-norm-bounded matrix  $E_t$ . Specifically, we will use the decomposition guarantee of the  $(1, 12L^2)$ -oracle, which states that there exists norm-decomposition of  $G_t = S_t + E_t$  where  $\|S_t\|_F \leq 12L^2$  and  $\|E_t\|_2 \leq \frac{1}{6\sqrt{r}}$ . As our movement is confined in  $\mathcal{X}$ ,  $D = X_{t+1} - X_t$  is nuclear-norm-bounded so the inner product  $\langle E_t, D \rangle$  can be bounded by generalized Holder's inequality. Recall  $\eta = \frac{1}{288L^4}$ .
$$
\begin{array}{l} 2 \eta R \langle G _ {t}, D \rangle + \langle D, D \rangle = 2 \eta R \langle E _ {t}, D \rangle + 2 \eta R \langle S _ {t}, D \rangle + \langle D, D \rangle \\ \geq - 2 \eta R \| E _ {t} \| _ {2} \cdot \| D \| _ {*} - \eta^ {2} R ^ {2} \langle S _ {t}, S _ {t} \rangle \\ \geq - 2 \eta R \cdot \frac {1}{6 \sqrt {r}} \cdot 2 \sqrt {2 r} R - \eta^ {2} R ^ {2} \cdot 1 4 4 L ^ {4} \\ \geq - \frac {3}{2} \eta R ^ {2}. \tag {5} \\ \end{array}
$$
Putting inequalities 3,4 and 5 together:
$$
\left\| D _ {t} \right\| _ {F} ^ {2} - \left\| D _ {t + 1} \right\| _ {F} ^ {2} \geq 2 \eta R ^ {2} - \frac {3}{2} \eta R ^ {2} \geq \frac {\eta}{2} \cdot \left\| D _ {t} \right\| _ {F} ^ {2} \Longrightarrow \left\| D _ {t + 1} \right\| _ {F} ^ {2} \leq \left(1 - \frac {\eta}{2}\right) \cdot \left\| D _ {t} \right\| _ {F} ^ {2}.
$$
In the case that the algorithm terminates after  $T = \frac{6}{\eta}$  iterations,
$$
\left\| X _ {T} - X ^ {*} \right\| _ {F} ^ {2} \leq \left(1 - \frac {\eta}{2}\right) ^ {T} \cdot \left\| X _ {\text {i n}} - X ^ {*} \right\| _ {F} ^ {2} \leq \exp \left(- \frac {\eta T}{2}\right) \cdot R ^ {2} \leq \frac {1}{1 6} R ^ {2},
$$
$$
\left\| X _ {\text {o u t}} - X ^ {*} \right\| _ {F} \leq \left\| X _ {\text {o u t}} - X _ {T} \right\| _ {F} + \left\| X _ {T} - X ^ {*} \right\| _ {F} \leq 2 \left\| X _ {T} - X ^ {*} \right\| _ {F} \leq \frac {1}{2} R.
$$
The last inequality comes from  $X_{\mathrm{out}}$  being the best rank-  $r$  approximation of  $X_{T}$ .
In the case that the algorithm terminates early at Line 11, we can assume with probability at least  $1 - \frac{\delta}{T}$  that the weight oracle would have succeeded given applicable input  $u_{t}$ . Then failure to satisfy the progress and decomposition guarantees means that  $u_{t}$  is not an applicable input, which means  $V_{t}$  does not satisfy the norm constraint in the weight oracle.  $\| V_t\|_*\leq 2\sqrt{2r}$  is guaranteed because  $X_{t}\in \mathcal{X}$ , and  $\| V_t\| _F = \frac{1}{R}\| X_t - X^*\| _F$  is decreasing in each round, so we must have  $\| V_t\| _F < \frac{1}{4}$  which means  $\| X_t - X^*\| _F < \frac{1}{4} R$ . By the same argument as above,  $\| X_{\mathrm{out}} - X^{*}\|_{F}\leq \frac{1}{2} R$ .
Finally, by a union bound on the failure probability of the weight oracle, the algorithm succeeds with probability at least  $1 - \delta$ .
Proof. Algorithm 2 has a for-loop that's repeated for  $T = O(L^4)$  times.
Inside the loop, line 5 and 7 takes linear time  $O(nd^2)$ . Computing  $w_t$  using the oracle (line 6) runs in time  $O(nd^\omega \log(d) \log \frac{L}{\delta} r \rho^2)$  according to Lemma 5.4. Line 8 through line 12 are all upper
bounded by time of SVD, which is on the same order of matrix multiplication  $O(d^{\omega})$  [DDH07], with current best of  $O(d^{2.373})$  [Wil14]. In particular, verifying the oracle guarantees (line 8) can be solved as an eigenvalue problem. Finding  $X_{t + 1}$  (line 9) is equivalent to projecting  $X_{t + 1}' := \frac{X_t - \eta RG_t - X_{\mathrm{in}}}{\sqrt{2rR}}$  onto the unit nuclear norm ball. We first perform SVD on  $X_{t + 1}'$  then binary search for the largest truncation from its singular values to reach the nuclear norm sphere in time  $O(d\log d)$ , and the entire projection step is dominated by SVD. Finally the output step consists of SVD and matrix multiplication.
The overall running time of the algorithm is dominated by the weight oracle, so the total running time is  $O(nd^{\omega} \log (d) \log \frac{L}{\delta} r \rho^{2} L^{4})$ .
B Omitted proofs from Section 5
First we state a couple of lower bounds related to the decomposition potential function.
Claim B.1.  $\mu^2\log [F(E)]\geq \mu^2\log (d)$  and  $\mu^2\log [F(E)]\geq \| E\| _2^2$
Proof. For the first lower bound,  $\exp \left(\frac{\sigma_j^2(E)}{\mu^2}\right) \geq 1$  for all  $j \in [d]$  so  $F(E) = \sum_{j=1}^{d} \exp \left(\frac{\sigma_j^2(E)}{\mu^2}\right) \geq d$ .
For the second lower bound,  $F(E) = \sum_{j=1}^{d} \exp \left( \frac{\sigma_j^2(E)}{\mu^2} \right) \geq \exp \left( \frac{\sigma_1^2(E)}{\mu^2} \right) = \exp \left( \frac{\|E\|_2^2}{\mu^2} \right)$ .
Lemma B.1 (Correctness of Algorithm 3). If Algorithm 3 terminates from the inner loop, the output satisfies the progress and decomposition guarantees as defined in 2.5.
Proof. We start with  $w_0 = 0$ , which means  $\Phi_{\mathrm{prog}}(w_0) = 0$ ,  $\Phi_{\mathrm{dc}}(w_0) = \mu^2 \log d$ , and  $\Phi(w_0) = 0 - Cr\mu^2 \log d = -1$ .
At each round, since  $s_t$  is chosen to maximize  $\Phi(w_t + s_t e_i)$ , in particular if we choose  $s_t = 0$  then  $\Phi(w_{t+1}) = \Phi(w_t)$ , so  $\Phi(w_{t+1}) \geq \Phi(w_t)$  which is non-decreasing. By definition  $\Phi_{\mathrm{prog}}(w_t)$  is also non-decreasing, and increases by at most 1 each round, because  $\Phi_{\mathrm{prog}}(w_{t+1}) - \Phi_{\mathrm{prog}}(w_t) = s_t u_i^2 \leq \eta(\|A_i\|_2 \|V\|_*)^2 \leq 8 \eta r \rho^2 \leq \frac{8}{K \log d} \leq 1$ .  $\Phi_{\mathrm{dc}}(w_t)$  may not be monotone, but we have  $\Phi_{\mathrm{dc}}(w_t) \geq \mu^2 \log d$ .
Suppose the algorithm terminates at round  $t$  during one of the inner loops, which means  $\Phi_{\mathrm{prog}}(w_{t - 1}) < 1$  and  $1 \leq \Phi_{\mathrm{prog}}(w_t) < 2$ .
Progress guarantee:  $\Phi_{\mathrm{prog}}(w_t) = \sum_{i=1}^n (w_t)_i u_i^2 \geq 1$  is satisfied upon termination.
Decomposition guarantee:
$$
\begin{array}{l} \Phi (w _ {t}) = \Phi_ {\mathrm {p r o g}} (w _ {t}) - C r \Phi_ {\mathrm {d c}} (w _ {t}) \geq \Phi (w _ {0}) = - 1, \\ \Rightarrow \min  _ {\| S \| _ {F} \leq L \| w \| _ {1}} \left(\mu^ {2} \log \left[ F \left(G _ {w} - S\right) \right]\right) + \frac {\| w _ {t} \| _ {1}}{4 C L r} = \Phi_ {\mathrm {d c}} (w _ {t}) \leq \frac {\Phi_ {\operatorname* {p r o g}} (w _ {t}) + 1}{C r} \leq \frac {3}{C r}, \\ \end{array}
$$
$$
\begin{array}{l} \min  _ {\| S \| _ {F} \leq L \| w _ {t} \| _ {1}} \left(\mu^ {2} \log \left[ F \left(G _ {w _ {t}} - S\right) \right]\right) \leq \frac {3}{C r} \Rightarrow \exists \| S \| _ {F} \leq L \| w _ {t} \| _ {1} s. t. \| G _ {w _ {t}} - S \| _ {2} ^ {2} \leq \frac {3}{C r}, \\ \text {a n d} \frac {\| w _ {t} \| _ {1}}{4 C L r} \leq \frac {3}{C r} \Rightarrow \| w _ {t} \| _ {1} \leq 1 2 L. \\ \end{array}
$$
So there exist  $\| S\| _F\leq 12L^2$  and  $\| E\| _2 = \| G - S\| _2\leq \frac{\sqrt{3}}{\sqrt{Cr}} = \frac{1}{6\sqrt{r}}$  which satisfy the decomposition guarantee.
Notice that for any round  $t' < t$ ,  $\Phi_{\mathrm{prog}}(w_{t'}) < 1$ , we also have  $\Phi_{\mathrm{dc}}(w_{t'}) \leq \frac{\Phi_{\mathrm{prog}}(w_{t'}) + 1}{Cr} \leq \frac{2}{Cr}$ , so  $\Phi_{\mathrm{dc}}(w_t) \leq \frac{3}{Cr}$  throughout the algorithm, which is a fact we will use later in Lemma B.3.
Lemma B.2 (Success probability of Algorithm 3). Given  $\mathcal{A}$  satisfying dRIP Condition and applicable input  $u$ , Algorithm 3 terminates from the inner loop with probability at least  $1 - \delta$ .
Proof. We first show the probability that the algorithm terminates from the inner loop is at least  $\frac{1}{2}$ , i.e.,  $\operatorname*{Pr}\left[\Phi_{\mathrm{prog}}(w_t) \geq 1\right] \geq \frac{1}{2}$  for some  $t \leq N$ .
Notice that  $\Phi(w_{t}) = \Phi_{\mathrm{prog}}(w_{t}) - Cr\Phi_{\mathrm{dc}}(w_{t}) \geq 0 \implies \Phi_{\mathrm{prog}}(w_{t}) \geq Cr\Phi_{\mathrm{dc}}(w_{t}) \geq Cr\Phi_{\mathrm{dc}}(w_{0}) = 1$ , therefore the algorithm starts with  $\Phi(w_{0}) = -1$  and will terminate once  $\Phi(w_{t}) \geq 0$ . Also notice that throughout the algorithm  $\Phi(w_{t}) < 1$  because  $\Phi_{\mathrm{prog}}(w_{t}) < 2$  and  $Cr\Phi_{\mathrm{dc}}(w_{t}) \geq 1$  (from proof of Lemma B.1).
To prove by contradiction, assume that  $\operatorname{Pr}[\Phi_{\mathrm{prog}}(w_t) \geq 1] < \frac{1}{2}$  for all  $t \leq N$ , i.e.,  $\operatorname{Pr}$  [continue]  $\geq \frac{1}{2}$  for all rounds. We will lower bound the expected increase in  $\Phi(w_t)$  each round, and we will show that with sufficiently large  $N$ ,  $\mathbb{E}[\Phi(w_N)] \geq 1$  contradicting  $\Phi(w_t) < 1$  for all  $t \leq N$ .
Recall that  $\Phi(w_{t}) = \Phi_{\mathrm{prog}}(w_{t}) - Cr\Phi_{\mathrm{dc}}(w_{t})$ , the lower bound for increase in  $\Phi_{\mathrm{prog}}(w_{t})$  is provided by dRIP condition on  $\mathcal{A}$  and applicable input  $u$ . The upper bound for expected increase in  $\Phi_{\mathrm{dc}}(w_{t})$  is provided by Lemma B.3.
Given the algorithm continues at round  $t \leq N$ , consider choosing  $s_t = \eta w_i^*$  so that  $w' = w_t + \eta w_i^* e_i$ , then the expected increase in  $\Phi$  is at least:
$$
\begin{array}{l} \mathbb {E} \left[ \Phi \left(w _ {t + 1}\right) - \Phi \left(w _ {t}\right) \mid \text {c o n t i n u e} \right] = \mathbb {E} \left[ \Phi_ {\operatorname {p r o g}} \left(w _ {t + 1}\right) - \Phi_ {\operatorname {p r o g}} \left(w _ {t}\right) \right] - C r \mathbb {E} \left[ \Phi_ {\mathrm {d c}} \left(w _ {t + 1}\right) - \Phi_ {\mathrm {d c}} \left(w _ {t}\right) \right] \\ \geq \mathbb {E} \left[ \Phi_ {\operatorname {p r o g}} \left(w ^ {\prime}\right) - \Phi_ {\operatorname {p r o g}} \left(w _ {t}\right) \right] - C r \mathbb {E} \left[ \Phi_ {\mathrm {d c}} \left(w ^ {\prime}\right) - \Phi_ {\mathrm {d c}} \left(w _ {t}\right) \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \eta w _ {i} ^ {*} u _ {i} ^ {2} - C r \left(\mathbb {E} \left[ \Phi_ {\mathrm {d c}} \left(w ^ {\prime}\right) \right] - \Phi_ {\mathrm {d c}} \left(w _ {t}\right)\right) \\ \geq \frac {\eta}{L n} - C r \cdot \frac {\eta}{2 C L n r} \\ = \frac {\eta}{2 L n} . \\ \end{array}
$$
Given the algorithm stops after round  $t$ ,  $\mathbb{E}[\Phi(w_{t+1}) - \Phi(w_t) \mid \text{stop}] = 0$ . Overall:
$$
\begin{array}{l} \mathbb {E} \left[ \Phi \left(w _ {t + 1}\right) - \Phi \left(w _ {t}\right) \right] = \mathbb {E} \left[ \Phi \left(w _ {t + 1}\right) - \Phi \left(w _ {t}\right) \mid \text {c o n t i n u e} \right] \cdot \Pr [ \text {c o n t i n u e} ] + 0 \\ \geq \frac {\eta}{2 L n} \cdot \Pr [ \text {c o n t i n u e} ]. \\ \end{array}
$$
By choosing a sufficiently large  $N = \frac{8Ln}{\eta}$
$$
\mathbb {E} \left[ \Phi \left(w _ {N}\right) \right] \geq \Phi \left(w _ {0}\right) + \frac {\eta N}{2 L n} \cdot \Pr [ \text {c o n t i n u e} ] \geq - 1 + \frac {\eta N}{4 L n} \geq 1,
$$
contradicting  $\Phi(w_t) < 1$ . This means each inner loop of the algorithm will terminate with probability greater than  $\frac{1}{2}$ . Finally, we boost the success probability to  $(1 - \delta)$  using the outer loop with  $N' = \log_2 \frac{1}{\delta}$  iterations.
Lemma B.3 provides a crucial bound used in the proof. Even though it achieves similar result as Lemma 13  $\left[\mathrm{KLL}^{+}23\right]$  on the potential functions defined for vectors, analyzing the potential function defined for matrices involves very different techniques.
Lemma B.3 (Potential Increase Upper Bound). Given  $w \in \mathbb{R}^n$  s.t.  $\Phi_{dc}(w) \leq \frac{3}{Cr}$ , by choosing a sufficiently large value for  $K$ , for  $w' = w + \eta w_i^* e_i$ , we have:
$$
\mathbb {E} _ {i \in [ n ]} \left[ \Phi_ {d c} \left(w ^ {\prime}\right) \right] \leq \Phi_ {d c} (w) + \frac {\eta}{2 C L n r}.
$$
The assumption  $\Phi_{dc}(w) \leq \frac{3}{Cr}$  is justified in the proof of Lemma B.1.
Proof. First we introduce some notation:
Denote  $\Phi_{\mathrm{op}}(w) = \min_{\| S\| _F\leq L\| w\| _1}\left(\mu^2\log \left[F(G_w - S)\right]\right)$ , so that  $\Phi_{\mathrm{dc}}(w) = \Phi_{\mathrm{op}}(w) + \frac{\|w\|_1}{4CLr}$ . Let  $G^{*} = \sum_{i = 1}^{n}w_{i}^{*}u_{i}A_{i}$ , and by dRIP Condition 2.3, we know  $\exists (L,\frac{1}{K\sqrt{r}})$ -norm-decomposition of  $G^{*} = S^{*} + E^{*}$ , where  $\| S^{*}\|_{F}\leq L$  and  $\| E^{*}\|_{2}\leq \frac{1}{K\sqrt{r}}$ . Let  $G = \sum_{i = 1}^{n}w_{i}u_{i}A_{i}$ , and  $S =$
$\operatorname{argmin}\Phi_{\mathrm{op}}(w)$  so that  $\Phi_{\mathrm{op}}(w) = \mu^2\log [F(G - S)]$  and let  $E = G - S$ . Let  $G' = \sum_{i=1}^{n} w_i' u_i A_i$ . Using these notation and  $\sum_{i=1}^{n} w_i^* = 1$ :
$$
\begin{array}{l} \underset {i \in [ n ]} {\mathbb {E}} \left[ \Phi_ {\mathrm {d c}} \left(w ^ {\prime}\right) \right] = \underset {i \in [ n ]} {\mathbb {E}} \left[ \Phi_ {\mathrm {o p}} \left(w ^ {\prime}\right) + \frac {\| w ^ {\prime} \| _ {1}}{4 C L r} \right] = \underset {i \in [ n ]} {\mathbb {E}} \left[ \Phi_ {\mathrm {o p}} \left(w ^ {\prime}\right) + \frac {\| w \| _ {1} + \eta w _ {i} ^ {*}}{4 C L r} \right] \\ = \underset {i \in [ n ]} {\mathbb {E}} \left[ \Phi_ {\mathrm {o p}} \left(w ^ {\prime}\right) \right] + \frac {\| w \| _ {1}}{4 C L r} + \frac {\eta}{4 C L n r}. \\ \end{array}
$$
We need to show  $\mathbb{E}_{i\in [n]}\big[\Phi_{\mathrm{dc}}(w^{\prime})\big]\leq \Phi_{\mathrm{dc}}(w) + \frac{\eta}{2CLnr} = \Phi_{\mathrm{op}}(w) + \frac{\|w\|_1}{4CLr} +\frac{\eta}{2CLnr}$ , equivalently
$$
\mathbb {E} _ {i \in [ n ]} \left[ \Phi_ {\mathrm {o p}} \left(w ^ {\prime}\right) \right] \leq \Phi_ {\mathrm {o p}} (w) + \frac {\eta}{4 C L n r}.
$$
Consider  $S' = S + \eta w_i^* S^*$ . We have  $\| S' \|_F \leq \| S \|_F + \eta w_i^* \| S^* \|_F \leq L \cdot \| w \|_1 + \eta w_i^* L = L \cdot \| w' \|_1$ , so  $S'$  is a valid argument for  $\Phi_{\mathrm{op}}(w') = \min_{\| S \|_F \leq L \| w' \|_1} \left( \mu^2 \log [F(G' - S)] \right)$ , therefore  $\Phi_{\mathrm{op}}(w') \leq \mu^2 \log [F(G' - S')]$ . Let  $E' = G' - S' = G + \eta w_i^* u_i A_i - S - \eta w_i^* S^* = E + Z^{(i)}$  where  $Z^{(i)} = \eta w_i^* u_i A_i - \eta w_i^* S^*$ . Using these and the concavity of the log function, we have
$$
\begin{array}{l} \mathbb {E} _ {i \in [ n ]} \left[ \Phi_ {\mathrm {o p}} \left(w ^ {\prime}\right) \right] \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mu^ {2} \log \left[ F \left(G ^ {\prime} - S ^ {\prime}\right) \right] = \frac {1}{n} \sum_ {i = 1} ^ {n} \mu^ {2} \log \left[ F \left(E + Z ^ {(i)}\right) \right] \\ \leq \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left(F (E + Z ^ {(i)})\right) \right]. \\ \end{array}
$$
It suffices to show
$$
\mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \Big (F (E + Z ^ {(i)}) \Big) \right] \leq \Phi_ {\mathrm {o p}} (w) + \frac {\eta}{4 C L n r} = \mu^ {2} \log [ F (E) ] + \frac {1}{4 C L} \frac {\eta}{n r}.
$$
Expanding the left hand side:
$$
\begin{array}{l} \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} F (E + Z ^ {(i)}) \right] \\ = \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {t r} \exp \left(\frac {E ^ {\top} E + Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right] \\ \leq \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \operatorname {t r} \left[ \exp \left(\frac {E ^ {\top} E}{\mu^ {2}}\right) \cdot \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right] \right] \\ = \mu^ {2} \log \left[ \operatorname {t r} \left[ \exp \left(\frac {E ^ {\top} E}{\mu^ {2}}\right) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right] \right] \\ \leq \mu^ {2} \log \left[ \operatorname {t r} \exp \left(\frac {E ^ {\top} E}{\mu^ {2}}\right) \cdot \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \right] \\ = \mu^ {2} \log [ F (E) ] + \mu^ {2} \log \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \\ \end{array}
$$
The first inequality uses Golden-Thompson Inequality (stated as Lemma B.6), and the second inequality follows from Lemma B.4. Finally it suffices to show
$$
\mu^ {2} \log \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \leq \frac {1}{4 C L} \frac {\eta}{n r}.
$$
We will use the approximation  $\exp(X) \preccurlyeq I + X + X^2$  for symmetric  $X$  with  $\|X\|_2 \leq 1$ . The argument in the exponential satisfies this condition as justified in Claim B.2. We will also use
$\log (1 + x)\leq x\forall x\geq 0.$
$$
\begin{array}{l} \mu^ {2} \log \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \\ \leq \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \right] \\ \leq \mu^ {2} \log \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \left\| I + \frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}} + \frac {(Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E) ^ {2}}{\mu^ {4}} \right\| _ {2} \right] \\ \leq \mu^ {2} \log \left[ \| I \| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {Z ^ {(i) \top} Z ^ {(i)}}{\mu^ {2}} \right\| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}} \right\| _ {2} \right. \\ \left. + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {\left(Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E\right) ^ {2}}{\mu^ {4}} \right\| _ {2} \right] \\ \leq \mu^ {2} \log \left[ \| I \| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {Z ^ {(i) \top} Z ^ {(i)}}{\mu^ {2}} \right\| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}} \right\| _ {2} \right. \\ \left. + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {2 (Z ^ {(i) \top} Z ^ {(i)}) ^ {2}}{\mu^ {4}} \right\| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {2 (E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E) ^ {2}}{\mu^ {4}} \right\| _ {2} \right] \\ \leq \mu^ {2} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {Z ^ {(i) \top} Z ^ {(i)}}{\mu^ {2}} \right\| _ {2} + \mu^ {2} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}} \right\| _ {2} \\ + 2 \mu^ {2} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {(Z ^ {(i) \top} Z ^ {(i)}) ^ {2}}{\mu^ {4}} \right\| _ {2} + 2 \mu^ {2} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {(E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E) ^ {2}}{\mu^ {4}} \right\| _ {2} \\ = \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} Z ^ {(i) \top} Z ^ {(i)} \right\| _ {2} + \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E \right\| _ {2} \\ + \frac {2}{\mu^ {2}} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} (Z ^ {(i) \top} Z ^ {(i)}) ^ {2} \right\| _ {2} + \frac {2}{\mu^ {2}} \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} (E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E) ^ {2} \right\| _ {2} \\ \end{array}
$$
These four terms are bounded by Claims B.3,B.4, B.5 and B.6 respectively, notice that the second term dominates the first and the third, and the forth term dominates the second. So finally we have:
$$
\begin{array}{l} \mu^ {2} \log \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp \left(\frac {Z ^ {(i) \top} Z ^ {(i)} + E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E}{\mu^ {2}}\right) \right\| _ {2} \\ \leq \left(\frac {3 \times 4}{\sqrt {C} K} + \frac {9 6 L ^ {2}}{K}\right) \frac {\eta}{n r} \\ \leq \frac {9 7 L ^ {2}}{K} \frac {\eta}{n r} \\ = \frac {1}{4 C L} \frac {\eta}{n r}, \text {w i t h c h o i c e o f} K = 3 8 8 C L ^ {3} = O \left(L ^ {3}\right). \\ \end{array}
$$
Lemma B.4. If  $0 \preccurlyeq A$ , then  $\operatorname{tr}(AB) \leq \operatorname{tr}(A) \cdot \|B\|_2$ .
Proof. Since  $0 \preccurlyeq A$ ,  $A = \sum_{j} \sigma_{j} u_{j} u_{j}^{\top}$  with  $\sigma_{j} \geq 0$ .
$$
\begin{array}{l} \operatorname {t r} (A B) = \operatorname {t r} \left(\sum_ {j} \sigma_ {j} u _ {j} u _ {j} ^ {\top} B\right) = \sum_ {j} \sigma_ {j} \operatorname {t r} (u _ {j} u _ {j} ^ {\top} B) = \sum_ {j} \sigma_ {j} u _ {j} ^ {\top} B u _ {j} \\ \leq \sum_ {j} \sigma_ {j} \cdot \| B \| _ {2} \\ = \operatorname {t r} (A) \cdot \| B \| _ {2}. \\ \end{array}
$$
Lemma B.5.  $(A + B)^{\top}(A + B) \preccurlyeq 2A^{\top}A + 2B^{\top}B$ , and  $\left[(A + B)^{\top}(A + B)\right]^{2} \preccurlyeq 8(A^{\top}A)^{2} + 8(B^{\top}B)^{2}$ .
$$
\begin{array}{l} 2 A ^ {\top} A + 2 B ^ {\top} B - (A + B) ^ {\top} (A + B) = A ^ {\top} A + B ^ {\top} B - A ^ {\top} B - B ^ {\top} A \\ = (A - B) ^ {\top} (A - B) \\ \succcurlyeq 0. \\ \end{array}
$$
$$
\begin{array}{l} \left[ (A + B) ^ {\top} (A + B) \right] ^ {2} \preccurlyeq \left(2 A ^ {\top} A + 2 B ^ {\top} B\right) ^ {2} \\ \prec 2 \left[ 2 \left(A ^ {\top} A\right) \right] ^ {2} + 2 \left[ 2 \left(B ^ {\top} B\right) \right] ^ {2} \\ \prec 8 (A ^ {\top} A) ^ {2} + 8 (B ^ {\top} B) ^ {2}. \\ \end{array}
$$
The following claims were used in Lemma B.3. Recall that  $\mathcal{A}$  satisfies dRIP Condition 2.3,  $u = \mathcal{A}[V] \in \mathbb{R}^n$  for some  $V \in \mathbb{R}^{d_1 \times d_2}$  satisfying  $\| V \|_F \in [\frac{1}{4}, 1]$ ,  $\| V \|_* \leq 2\sqrt{2r}$ ,  $Z^{(i)} = \eta w_i^*(u_i A_i - S^*)$ , and  $\Phi_{\mathrm{dc}}(w) \leq \frac{3}{Cr}$  by assumption of Lemma B.3. We have the following:
$\| A_i\| _2\leq \rho$  by boundedness property of dRIP Condition 2.3,
$|u_i| = |\langle A_i, V \rangle| \leq \|A_i\|_2 \|V\|_* \leq \rho 2\sqrt{2r} \leq L\sqrt{r}\rho$  assuming  $2\sqrt{2} \leq L$ ,
$\| S^{*}\|_{2}\leq \| S^{*}\|_{F}\leq L,\| E^{*}\|_{2}\leq \frac{1}{K\sqrt{r}}$  by decomposition property of dRIP Condition 2.3,
$\| E\| _2^2\leq \Phi_{\mathrm{op}}(w)\leq \Phi_{\mathrm{dc}}(w)\leq \frac{3}{Cr}$  by Claim B.1.
Claim B.2.  $\left\| \frac{Z^{(i)\top}Z^{(i)} + E^{\top}Z^{(i)} + Z^{(i)\top}E}{\mu^2}\right\| _2\leq 1.$
$$
\begin{array}{l} \left\| Z ^ {(i) \top} Z ^ {(i)} \right\| _ {2} = \eta^ {2} w _ {i} ^ {* 2} \left\| (u _ {i} A _ {i} - S ^ {*}) ^ {\top} (u _ {i} A _ {i} - S ^ {*}) \right\| _ {2} \\ \leq 2 \eta^ {2} w _ {i} ^ {* 2} \left(u _ {i} ^ {2} \left\| A _ {i} ^ {\top} A _ {i} \right\| _ {2} + \left\| S ^ {* \top} S ^ {*} \right\| _ {2}\right) (L e m m a B. 5) \\ \leq 2 \eta^ {2} w _ {i} ^ {* 2} \left(L ^ {2} r \rho^ {4} + L ^ {2}\right) \\ \leq 4 \eta^ {2} w _ {i} ^ {* 2} L ^ {2} r \rho^ {4} \\ \end{array}
$$
$$
\begin{array}{l} \left\| E ^ {\top} Z ^ {(i)} + Z ^ {(i) \top} E \right\| _ {2} \leq 2 \left\| E ^ {\top} Z ^ {(i)} \right\| _ {2} \leq 2 \| E \| _ {2} \cdot \left\| Z ^ {(i)} \right\| _ {2} \\ = 2 \eta w _ {i} ^ {*} \| E \| _ {2} \cdot \| u _ {i} A _ {i} - S ^ {*} \| _ {2} \\ \leq 2 \eta w _ {i} ^ {*} \| E \| _ {2} \cdot \left(| u _ {i} | \| A _ {i} \| _ {2} + \| S ^ {*} \| _ {2}\right) \\ \leq 2 \eta w _ {i} ^ {*} \frac {2}{\sqrt {C r}} \left(L \sqrt {r} \rho^ {2} + L\right) \\ \leq 8 \eta w _ {i} ^ {*} \frac {L \rho^ {2}}{\sqrt {C}} \\ \end{array}
$$
Putting them together: