SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A crucial component in our algorithm is a weight oracle that produces a nonnegative weight on each sensing matrix (the weights are in general different in each iteration), such that the weighted gradient step moves the current solution closer to  $X^{*}$ . The oracle should output weights that satisfy certain properties which we term progress and decomposition guarantees. The purpose of these two guarantees is further explained in the proof of Lemma 4.2 in Appendix A.
Definition 2.5 (Weight Oracle). We say an algorithm  $\mathcal{O}$  is a  $(C_{\mathrm{prog}}, C_F)$ -oracle, if given as input  $n$  matrices  $\mathcal{A} = \{A_i\}_{i=1}^n \subset \mathbb{R}^{d_1 \times d_2}$  and an vector  $u = \mathcal{A}[V] \in \mathbb{R}^n$  where  $V \in \mathbb{R}^{d_1 \times d_2}$ ,  $\|V\|_F \in [\frac{1}{4}, 1]$ , and  $\|V\|_* \leq 2\sqrt{2r}$ , the algorithm  $\mathcal{O}(\mathcal{A}, u, \delta)$  returns a weight vector  $w \in \mathbb{R}_{\geq 0}^n$  such that the following conditions hold with probability at least  $1 - \delta$ :
1. Progress guarantee:  $\sum_{i=1}^{n} w_i u_i^2 \geq C_{\mathrm{prog}}$ ;
2. Decomposition guarantee:  $\exists (C_F, \frac{C_{\mathrm{prog}}}{6\sqrt{r}})$  norm-decomposition of  $G = \sum_{i=1}^{n} w_i u_i A_i$ .
Note that the progress guarantee is equivalent to  $\langle G,V\rangle \geq C_{\mathrm{prog}}$
Finally we define numerical rank which we use in our analysis. Numerical rank serves as a lower bound for the rank of a matrix based on its nuclear norm and Frobenius norm. That is, we always have  $\mathrm{Rank}_{\mathrm{n}}(A) \leq \mathrm{rank}(A)$ .
Definition 2.6 (Numerical Rank). The numerical rank of  $A$  is  $\operatorname{Rank}_{\mathrm{n}}(A) = \frac{\|A\|_{*}^{2}}{\|A\|_{F}^{2}}$ .
3 Semi-Random Matrix Sensing
In this section, we present our main algorithm (Algorithm 1) for semi-random matrix sensing. Algorithm 1 with high probability recovers the ground-truth matrix  $X^{*}$  to arbitrary accuracy.
Algorithm 1: SemiRandomMatrixSensing  $(R_0,\epsilon ,\delta ,\mathcal{A},b)$
1: Input:  $R_0 \geq \|X^*\|_F, b = \mathcal{A}[X^*], \epsilon > 0, \delta \in (0,1)$ ;  
2: Output:  $X_{\mathrm{out}}$  s.t.  $\|X_{\mathrm{out}} - X^*\|_F \leq \epsilon$ .  
3:  $X_0 \gets 0, T \gets \log \frac{R_0}{\epsilon}, \delta' \gets \frac{\delta}{T}, R \gets R_0$ ;  
4: for  $0 \leq t \leq T$  do  
5:  $X_{t+1} \gets \text{HalveError}(X_t, R, \mathcal{O}, \delta', \mathcal{A}, b), R \gets \frac{R}{2}$ ;  
6: end for  
7: Return  $X_{\mathrm{out}} \gets X_T$ ;
The performance guarantee and runtime of Algorithm 1 are formally stated in the following theorem.
Theorem 3.1 (Matrix Sensing under wRIP). Suppose the ground-truth matrix  $X^{*} \in \mathbb{R}^{d_{1} \times d_{2}}$  satisfies  $\mathrm{rank}(X^{*}) \leq r$  and  $\| X^{*}\|_{F} \leq R_{0}$ . Suppose the sensing matrices  $\mathcal{A} = (A_{i} \in \mathbb{R}^{d_{1} \times d_{2}})_{i=1}^{n}$  satisfy  $(r, L, \rho)$ -wRIP (as in Definition 2.2). Let  $b = \mathcal{A}[X^{*}] \in \mathbb{R}^{n}$  be the corresponding measurements.
For any  $\epsilon, \delta > 0$ , Algorithm 1 can output  $X \in \mathbb{R}^{d_1 \times d_2}$  such that  $\| X - X^* \|_F \leq \epsilon$  with probability at least  $1 - \delta$ . Algorithm 1 runs in time  $O(nd^\omega \log(d) \log\left(\frac{L}{\delta} \log \frac{R_0}{\epsilon}\right) r \rho^2 L^4 \log \frac{R_0}{\epsilon})$  where  $d = \max(d_1, d_2)$  and  $\omega < 2.373$  is the matrix multiplication exponent.
Theorem 1.1 is a direct corollary of Theorem 3.1 under Gaussian design.
Proof of Theorem 1.1. When there are  $\Omega (dr)$  sensing matrices with i.i.d. standard Gaussian entries, the input sensing matrices satisfy  $(r,L,\rho)$ -wRIP for  $L = O(1)$  and  $\rho = O(d^{1 / 2})$  with probability at least  $1 - \frac{1}{\mathrm{poly}(d)}$ . This follows from a standard proof for RIP and the fact that we can ignore any sensing matrices with  $\| A_i\| _2\gg d^{1 / 2}$ . We assume that the wRIP condition is satisfied.
By Theorem 3.1, when  $L = O(1)$ ,  $\rho = O(d^{1/2})$ ,  $R_0 = \mathrm{poly}(d)$  and  $\delta = \frac{1}{\mathrm{poly}(d)}$ , Algorithm 1 can output a solution  $X$  such that  $\| X - X^* \|_F \leq \epsilon$  with high probability. The runtime of Algorithm 1 can be simplified to  $\widetilde{O}(nd^{\omega + 1}r\log(1/\epsilon))$ .
We first provide a road map for our analysis for proving Theorem 3.1:
- Our main algorithm runs a "halving" subroutine for  $\log \frac{R_0}{\epsilon}$  iterations to reduce the error to  $\epsilon$ . Each call to this subroutine reduces the upper bound on the distance between the current solution and the ground truth  $X^*$  by half. This halving subroutine runs in time  $O(nd^\omega \log (d)\log (\frac{L}{\delta}\cdot \log \frac{R_0}{\epsilon})r\rho^2 L^4)$  according to Lemmas 4.2 and 4.3.
- In Section 4, we present the halving algorithm (Algorithm 2). It depends on a  $(\Omega(1), O(1))$ -oracle, and Lemma 4.1 shows that the oracle guarantees can be easily verified. The algorithm's correctness and running time are analyzed in Lemma 4.2 and Lemma 4.3.
- In Section 5 we present the weight oracle required by the halving algorithm. We first show in Lemma 5.1 that the wRIP condition implies that the sensing matrices satisfy the dRIP condition tailored to the design of the oracle. Then we present an implementation of the oracle in Algorithm 3 based on the dRIP condition, and analyze its correctness and running time in Lemma 5.3 and Lemma 5.4.
4 Algorithm for Halving the Error
In this section, we present Algorithm 2 (HalveError). Algorithm 2 takes an estimate  $X_{\mathrm{in}}$  with  $\| X_{\mathrm{in}} - X^{*}\|_{F} \leq R$  and outputs  $X_{\mathrm{out}}$  such that  $\| X_{\mathrm{out}} - X^{*}\|_{F} \leq \frac{R}{2}$ . This is the matrix version of the HalfRaidusSparse [KLL+23] algorithm for vectors.
Algorithm 2: HalveError(Xin, R, O, δ, A, b)
1: Input: Rank-  $r$  matrix  $X_{\mathrm{in}}\in \mathbb{R}^{d_1\times d_2}$ $\| X_{\mathrm{in}} - X^{*}\|_{F}\leq R,\mathcal{O}$  is a  $(1,12L^2)$  -oracle for  $\mathcal{A}$  with failure probability  $\delta \in (0,1)$  , linear measurements  $b = \mathcal{A}[X^{*}]$  .   
2: Output:  $X_{\mathrm{out}}\in \mathbb{R}^{d_1\times d_2}$  s.t.  $\| X_{\mathrm{out}} - X^{*}\|_{F}\leq \frac{R}{2} w.p.\geq 1 - \delta$  and rank  $(X_{\mathrm{out}})\leq r$  .   
3:  $X_0\gets X_{\mathrm{in}},\mathcal{X} = \{X\in \mathbb{R}^{d_1\times d_2}\mid \| X - X_{\mathrm{in}}\|_*\leq \sqrt{2r} R\}$ $\eta = \frac{1}{288L^4},T = \frac{6}{\eta}$  .   
4: for  $0\leq t\leq T$  do   
5:  $u_{t}\leftarrow \frac{1}{R} (\mathcal{A}[X_{t}] - b);$  /\*  $u_{t} = \mathcal{A}[\frac{X_{t} - X^{*}}{R} ]$  where  $(u_t)_i = \frac{1}{R}\langle A_i,X_t - X^*\rangle *\bigcirc$    
6:  $w_{t}\gets \mathcal{O}(\mathcal{A},u_{t},\frac{\delta}{T})$  .   
7:  $G_{t}\gets \sum_{i = 1}^{n}(w_{t})_{i}(u_{t})_{i}A_{i}$  .   
8: if O output satisfies the progress and decomposition guarantees on  $u_{t}$  then   
9:  $X_{t + 1}\gets \operatorname {argmin}_{X\in \mathcal{X}}\| X - (X_t - \eta RG_t)\| _F^2$  .   
10: else   
11: Return  $X_{\mathrm{out}}\gets (X_t)_{(r)}$  ; /\* Rank-r approximation of  $X_{t}*$    
12: end if   
13: end for   
14: Return  $X_{\mathrm{out}}\gets (X_T)_{(r)}$  .
A crucial requirement of the algorithm is a  $(\Omega(1), O(1))$ -oracle for  $\mathcal{A}$ . In each iteration, the oracle takes a vector  $u_t = \frac{\mathcal{A}[X_t] - b}{R}$ , which is the (normalized) "measured deviation" between current estimate  $X_t$  and  $X^*$ , and computes a weight vector  $w_t$ . The algorithm then tries to minimize the weighted objective function by gradient descent:
$$
\text {O b j e c t i v e :} f _ {t} (X) = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \left\langle A _ {i}, \frac {X - X ^ {*}}{R} \right\rangle^ {2}, \text {i . e .} f _ {t} (X _ {t}) = \frac {1}{2} \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \left(u _ {t}\right) _ {i} ^ {2},
$$
$$
\text {G r a d i e n t :} \nabla_ {X} f _ {t} (X) = \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \langle A _ {i}, \frac {X - X ^ {*}}{R} \rangle A _ {i} , \text {i . e .} G _ {t} = \nabla_ {X} f _ {t} (X) | _ {X _ {t}} = \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \left(u _ {t}\right) _ {i} A _ {i}.
$$
Ideally in the next iteration, we would like to make a step from  $X_{t}$  in the opposite direction of the gradient  $G_{t}$  with the goal of minimizing the deviation in the next iteration. However, we cannot take a step exactly in the direction of  $G_{t}$ , and our movement is constrained within a ball of (nuclear norm) radius  $\sqrt{2r} R$  centered at  $X_{\mathrm{in}}$ , namely the region  $\mathcal{X} = \{X \mid \| X - X_{\mathrm{in}} \|_{*} \leq \sqrt{2r} R\}$ . Nuclear norm is used as a proxy to control the rank and Frobenius norm of  $X_{t}$  simultaneously throughout the algorithm: firstly, since  $\| X_{\mathrm{in}} - X^{*} \|_{F} \leq R$ , it makes sense that in each iteration  $\| \dot{X}_{t} - X_{\mathrm{in}} \|_{F} \leq R$  as well; secondly, while trying to minimize the difference between  $X_{t}$  and  $X^{*}$ , we also want to ensure the rank of  $X_{t}$  is relatively small, i.e.  $\operatorname{rank}(X_t) \leq O(r)$ . To tie things together, we use the following relationship between rank and numerical rank:
$$
\operatorname {r a n k} (X _ {t} - X _ {\mathrm {i n}}) \geq \operatorname {R a n k} _ {\mathrm {n}} (X _ {t} - X _ {\mathrm {i n}}) = \frac {\| X _ {t} - X _ {\mathrm {i n}} \| _ {*} ^ {2}}{\| X _ {t} - X _ {\mathrm {i n}} \| _ {F} ^ {2}}.
$$
Assuming  $\mathrm{rank}(X_t) \geq \mathrm{rank}(X_{\mathrm{in}})$  and  $\| X_t - X_{\mathrm{in}} \|_F \leq R$  throughout, then  $\mathrm{rank}(X_t) \geq \frac{\| X_t - X_{\mathrm{in}} \|_*^2}{2R^2}$ . Roughly speaking, in order for  $\mathrm{rank}(X_t) \leq O(r)$ , it is necessary that  $\| X_t - X_{\mathrm{in}} \|_* \leq O(\sqrt{r} R)$ , i.e.  $X_t$  is inside some nuclear norm ball  $\mathcal{X}$  of radius  $O(\sqrt{r} R)$  centered at  $X_{\mathrm{in}}$ . We set the radius of  $\mathcal{X}$  to be  $\sqrt{2r} R$  so that  $X^* \in \mathcal{X}$ , since  $\| X_{\mathrm{in}} - X^* \|_F \leq R$ ,  $\mathrm{rank}(X_{\mathrm{in}} - X^*) \leq 2r$  therefore  $\| X^* - X_{\mathrm{in}} \|_* \leq \sqrt{2r} R$ . Thus we confine our movement within this nuclear norm ball of radius  $\sqrt{2r} R$  centered at  $X_{\mathrm{in}}$  throughout the algorithm, and take the rank- $r$  approximation of the last  $X_t$  to ensure  $\mathrm{rank}(X_{\mathrm{out}}) \leq r$  upon the termination of the algorithm.
To analyze the algorithm, first we show how to check whether the weight oracle output satisfies the progress and decomposition guarantees. The progress condition  $\sum_{i=1}^{n} w_i u_i^2 \geq 1$  is trivial to verify, and we check whether  $G$  is  $(C_F, C_2)$ -decomposable using Lemma 4.1, with details and proof deferred to Appendix A.
Lemma 4.1 (Verify Norm Decomposition). Given a matrix  $G = U\Sigma V^{\top} = \sum_{i=1}^{d}\sigma_{i}u_{i}v_{i}^{\top}$  and  $C_2 > 0$ , suppose  $\sigma_1 \geq \ldots \geq \sigma_k > C_2 \geq \sigma_{k+1} \ldots \geq \sigma_d$ , then for all  $\|E\|_2 \leq C_2$ , we have  $\|G - E\|_F^2 \geq \sum_{i=1}^{k}(\sigma_i - C_2)^2$ .
The following lemmas analyze the algorithm's correctness and show that it terminates with the desired distance contraction, as well as its running time. The proof is deferred to Appendix A.
Lemma 4.2 (Algorithm 2: HalveError). Given  $a$  ( $1, 12L^2$ )-oracle for  $\mathcal{A}$  with failure probability  $\delta \in (0, 1)$ , where  $\mathcal{A}$  satisfies the dRIP Condition 2.3, and  $b = \mathcal{A}[X^*]$ , Algorithm 2 succeeds with probability at least  $1 - \delta$ .
Lemma 4.3 (Algorithm 2 Running Time). Algorithm 2 with failure probability  $\delta$  runs in time  $O(nd^{\omega}$  polylog  $(d)\log \frac{L}{\delta} r\rho^{2}L^{4})$ .
The crucial part of Lemma 4.2 shows that if current estimate  $X_{t}$  is sufficiently far from  $X^{*}$ , i.e.  $\| X_{t} - X^{*}\|_{F}\geq \frac{1}{4} R$ , then according to Lemma 5.3 with high probability the weight oracle produces an output satisfying the progress and decomposition guarantees, and each iteration of Algorithm 2 decreases the distance to  $X^{*}$  by a constant factor:  $\| X_{t + 1} - X^{*}\|_{F}^{2}\leq \left(1 - \frac{\eta}{2}\right)\cdot \| X_{t} - X^{*}\|_{F}^{2}$ , thus after sufficient number of iterations the distance to  $X^{*}$  will be halved. On the other hand, if the weight oracle fails, with high probability the current estimate  $X_{t}$  is already sufficiently close to  $X^{*}$ , thus the algorithm can terminate early.
5 Oracle for Reweighting the Input
In this section, we present an algorithm (Algorithm 3) that serves as the weight oracle required by the error-halving algorithm (Algorithm 2). Algorithm 3 is the matrix version of the StepOracle  $\left[\mathrm{KLL}^{+}23\right]$  algorithm for vectors. We first state that, given proper choices of parameters within a constant factor, the wRIP Condition 2.2 implies the dRIP Condition 2.3, which is a more suitable property for our oracle implementation. The proof is deferred to Appendix C.
Lemma 5.1 ( $wRIP \Rightarrow dRIP$ ). If  $\mathcal{A}$  satisfies wRIP Condition 2.2 with parameters  $r', L', \rho$ , then  $\mathcal{A}$  satisfies the dRIP Condition 2.3 with parameters  $L, K, r, \rho$  such that  $L = \Theta(L')$ ,  $r = \Theta(r')$ , and some constant  $K \geq 1$ .
Now we are ready to present an implementation of the weight oracle in Algorithm 3 based on the dRIP condition. This algorithm takes as inputs the dRIP sensing matrices  $\mathcal{A}$  and a vector  $u$ . If  $u$  is an applicable input to the oracle, with high probability the algorithm outputs a weight vector  $w$  satisfying the progress and decomposition guarantees as in Definition 2.5.
First we introduce some potential functions used in the algorithm.
Definition 5.2 (Potential Functions in Algorithm 3). For sensing matrices  $\mathcal{A} = \{A_i\}_{i=1}^n$  and input  $u \in \mathbb{R}^n$  to the oracle, we define the following potential functions on weight vector  $w \in \mathbb{R}^n$ :
- Progress potential:  $\Phi_{\mathrm{prog}}(w) = \sum_{i=1}^{n} w_i u_i^2$ .
- Decomposition potential:  $\Phi_{\mathrm{dc}}(w) = \min_{\| S \|_F \leq L \| w \|_1} \left( \mu^2 \log [F(G_w - S)] \right) + \frac{\| w \|_1}{4CLr}$ , where  $G_w = \sum_{i=1}^{n} w_i u_i A_i$  and  $F(E) = \operatorname{tr} \exp \left( \frac{E^\top E}{\mu^2} \right)$ .
- Overall potential:  $\Phi (w) = \Phi_{\mathrm{prog}}(w) - Cr\Phi_{\mathrm{dc}}(w)$ .
Note that  $F(E) = \sum_{j=1}^{d} \exp \left( \frac{\sigma_j^2(E)}{\mu^2} \right)$  where  $\sigma_j(E)$  is the  $j^{\text{th}}$  singular value of  $E$ , due to properties of the exponential of a diagonalizable matrix.
The progress and decomposition potential functions control the progress and decomposition guarantees respectively, and later we will show that the termination condition is implied by the overall potential  $\Phi \geq 0$ . Consequently, by maximizing the overall potential each round, the algorithm tries to make as much progress as possible while ensuring  $G$  is decomposable.
Algorithm 3:  $\mathcal{O}(\mathcal{A},u,\delta)$
1: Input: Sensing operator  $\mathcal{A}$  satisfying dRIP Condition 2.3,  $u\in \mathbb{R}^n$
2: Output:  $w \in \mathbb{R}^n$  such that the algorithm is a  $(1, 12L^2)$ -oracle as in Definition 2.5 with probability  $\geq (1 - \delta)$ .
3:  $C\gets 108,\mu \gets \frac{1}{\sqrt{Cr\log d}},\eta \gets \frac{1}{Kr\rho^2\log d},N'\gets \log_2\frac{1}{\delta},N\gets \frac{8Ln}{\eta}.$
4: for  $0 < k < N'$  do
5:  $w_0\gets 0;$
6: for  $0 \leq t \leq N$  do
7: if  $\Phi_{\mathrm{prog}}(w_t) \geq 1$  then
8: Return  $w \gets w_t$
9: else
0: Sample  $i\in [n]$  uniformly random;
1:  $s_t \gets \operatorname{argmax}_{s \in [0, \eta]} \Phi(w_t + se_i)$ ;
2:  $w_{t + 1}\gets w_t + s_t e_i;$
3: end if
4: end for
5: end for
6: Return  $w \gets 0$
Lemma 5.3 (Correctness of Algorithm 3). Suppose  $\mathcal{A}$  satisfies dRIP Condition 2.3 and  $u$  is an applicable input to the weight oracle (that is,  $u = \mathcal{A}[V] \in \mathbb{R}^n$  for some  $V \in \mathbb{R}^{d_1 \times d_2}$  satisfying  $\| V \|_F \in$
$\left[\frac{1}{4}, 1\right]$  and  $\|V\|_{*} \leq 2\sqrt{2r}$ . Then, Algorithm 3 is a  $(1,12L^{2})$ -oracle for  $\mathcal{A}$  (as in Definition 2.5) with failure probability at most  $\delta$ .
We prove this lemma in two steps: first we show in Lemma B.1 that the output is valid; then in Lemma B.2 we show that the oracle achieves the success probability. Our weight oracle is inspired by the step oracle in  $\mathrm{[KLL^{+}23]}$ . It is worth noting that Lemma B.3, a key component used in the proof of Lemma B.2, is significantly different in the matrix case compared to the vector case. Lemma B.3 upper bounds the increase in  $\Phi_{\mathrm{dc}}$  each round, which is then used to provide a lower bound for the increase in  $\Phi$ . Combining Lemma B.3 with our earlier remark that the algorithm terminates when  $\Phi \geq 0$  gives us the number of iterations needed to terminate with high probability.
The running time of Algorithm 3 is stated in the following lemma, and the proof is deferred to Appendix B.
Lemma 5.4 (Algorithm 3 Running Time). Algorithm 3 with failure probability  $\delta$  runs in time  $O(nd^{\omega}$  polylog  $(d)\log \frac{1}{\delta} r\rho^{2})$ .
In this paper, we pose and study the matrix sensing problem in a natural semi-random model. We relax the standard RIP assumption on the input sensing matrices to a much weaker condition where an unknown subset of the sensing matrices satisfies RIP while the rest are arbitrary.
For this semi-random matrix sensing problem, existing non-convex objectives can have bad local optima. In this work, we employ an iterative reweighting approach using a weight oracle to overcome the influence of the semi-random input. Our solution is inspired by previous work on semi-random sparse vector recovery, where we exploit the structural similarities between linear regression on sparse vectors and matrix sensing on low-rank matrices.
Looking forward, we believe our approach can serve as a starting point for designing more efficient and robust algorithms for matrix sensing, as well as for other low-rank matrix and sparse vector problems in the semi-random model.
We thank Rong Ge for helpful discussions. Xing Gao is supported in part by NSF awards ECCS-2217023 and CCF-2307106. Yu Cheng is supported in part by NSF Award CCF-2307106.
[Tho65] Colin J Thompson. Inequality with applications in statistical mechanics. Journal of Mathematical Physics, 6(11):1812-1813, 1965.
[Wil14] Virginia Vassilevska Williams. Multiplying matrices in o (n2. 373) time. preprint, 2014.
A Omitted Proofs from Section 4
To check whether  $G$  is  $(C_F, C_2)$ -decomposable using Lemma 4.1, we first consider the following construction. Suppose the SVD of  $G$  is  $G = U\Sigma V^\top = \sum_{i=1}^d \sigma_i u_i v_i^\top$ . Let  $S = \sum_{i=1}^d \mu_i u_i v_i^\top$  with  $\mu_i = \max(\sigma_i - C_2, 0)$  and let  $E = \sum_{i=1}^d \lambda_i u_i v_i^\top$  with  $\lambda_i = \min(C_2, \sigma_i)$ . In other words, suppose  $\sigma_1 \geq \ldots \geq \sigma_k > C_2 \geq \sigma_{k+1} \ldots \geq \sigma_d$ . For  $i \leq k$  (i.e.,  $\sigma_i > C_2$ ), let  $\mu_i = \sigma_i - C_2$  and  $\lambda_i = C_2$ ; for  $i > k$  (i.e.,  $\sigma_i \leq C_2$ ), let  $\mu_i = 0$  and  $\lambda_i = \sigma_i$ . We have  $G = S + E$  with  $\|E\|_2 \leq C_2$ , and  $\|S\|_F = \sqrt{\sum_{i=1}^k (\sigma_i - C_2)^2}$ . Then by Lemma 4.1,  $G$  is  $(C_F, C_2)$ -norm-decomposable if and only if  $\|S\|_F \leq C_F$  because:
- if  $\| S\| _F\leq C_F$  , we have a valid  $(C_F,C_2)$  -norm-decomposition for  $G$
- if  $\| S\| _F > C_F$  , a valid  $(C_F,C_2)$  -norm-decomposition does not exist for  $G$
Proof. Fix any  $E$  with  $\| E \|_2 \leq C_2$ . Observe that for all  $1 \leq i \leq k$ , we have  $u_i^\top G v_i = \sigma_i$  and  $-C_2 \leq u_i^\top E v_i \leq C_2$ . Consequently, for all  $i \leq k$ , we have  $\sigma_i > C_2$  and  $u_i^\top (G - E) v_i \geq \sigma_i - C_2 > 0$ .
Let  $S = G - E$ , we have
$$
\begin{array}{l} \left\| S \right\| _ {F} ^ {2} = \left\| U ^ {\top} S V \right\| _ {F} ^ {2} = \sum_ {i, j} \left(U ^ {\top} S V\right) _ {i j} ^ {2} \\ \geq \sum_ {i = 1} ^ {d} \left(U ^ {\top} S V\right) _ {i i} ^ {2} \geq \sum_ {i = 1} ^ {k} \left(U ^ {\top} S V\right) _ {i i} ^ {2} \\ = \sum_ {i = 1} ^ {k} \left(u _ {i} ^ {\top} S v _ {i}\right) ^ {2} \geq \sum_ {i = 1} ^ {k} \left(\sigma_ {i} - C _ {2}\right) ^ {2}. \\ \end{array}
$$
Proof. We will first show that the distance to  $X^{*}$  decreases by a constant factor after each iteration:
$$
\left\| X _ {t + 1} - X ^ {*} \right\| _ {F} ^ {2} \leq \left(1 - \frac {\eta}{2}\right) \cdot \left\| X _ {t} - X ^ {*} \right\| _ {F} ^ {2}.
$$
Consider iteration  $t$  in Algorithm 2:  $X_{t + 1} = \operatorname{argmin}_{X\in \mathcal{X}}\| X - (X_t - \eta RG_t)\| _F^2$ . Taking the gradient of  $\| X - (X_t - \eta RG_t)\| _F^2$  at  $X = X_{t + 1}$ , we get  $2[X_{t + 1} - (X_t - \eta RG_t)]$ . Since  $X^{*}\in \mathcal{X}$  and  $X_{t + 1}$  is the local minimizer in  $\mathcal{X}$ :
$$
2 \left\langle X _ {t + 1} - \left(X _ {t} - \eta R G _ {t}\right), X _ {t + 1} - X ^ {*} \right\rangle \leq 0.
$$
Rearranging the terms gives
$$
2 \eta R \left\langle G _ {t}, X _ {t + 1} - X ^ {*} \right\rangle \leq - 2 \left\langle X _ {t + 1} - X _ {t}, X _ {t + 1} - X ^ {*} \right\rangle .
$$
To simplify, let  $D = X_{t + 1} - X_t$ ,  $D_t = X_t - X^*$ ,  $D_{t + 1} = X_{t + 1} - X^*$ . Note that  $D + D_t = D_{t + 1}$ .
$$
\begin{array}{l} 2 \eta R \langle G _ {t}, D _ {t + 1} \rangle \leq - 2 \langle D, D _ {t + 1} \rangle = \langle D - D _ {t + 1}, D - D _ {t + 1} \rangle - \langle D, D \rangle - \langle D _ {t + 1}, D _ {t + 1} \rangle \\ = \left\langle D _ {t}, D _ {t} \right\rangle - \left\langle D, D \right\rangle - \left\langle D _ {t + 1}, D _ {t + 1} \right\rangle , \\ \end{array}
$$
$$
\left\| D _ {t} \right\| _ {F} ^ {2} - \left\| D _ {t + 1} \right\| _ {F} ^ {2} \geq 2 \eta R \langle G _ {t}, D _ {t + 1} \rangle + \langle D, D \rangle = 2 \eta R \langle G _ {t}, D _ {t} \rangle + 2 \eta R \langle G _ {t}, D \rangle + \langle D, D \rangle . \tag {3}
$$
Inequality (3) provides a lower bound on the distance contraction after each iteration. We break the right-hand side into two parts. The first term  $2\eta R\langle G_t, D_t \rangle$  corresponds to the magnitude of the step  $G_t$  in the direction  $D_t = X_t - X^*$ , which is the progress made by this step. To lower bound it, we
will use the progress guarantee of the  $(1,12L^2)$ -oracle. Recall  $u_{t} = \frac{1}{R} (\mathcal{A}[X_{t}] - b)$  and consider  $V_{t} = \frac{1}{R} (X_{t} - X^{*}) = \frac{1}{R} D_{t}$  so that  $u_{t} = \mathcal{A}[V_{t}]$ . Given that the oracle's output satisfies the progress guarantee, which states that  $\sum_{i=1}^{n}(w_{t})_{i}(u_{t})_{i}^{2} \geq 1$ , we have:
$$
2 \eta R \langle G _ {t}, D _ {t} \rangle = 2 \eta R ^ {2} \langle G _ {t}, V _ {t} \rangle = 2 \eta R ^ {2} \left\langle \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \left(u _ {t}\right) _ {i} A _ {i}, V _ {t} \right\rangle = 2 \eta R ^ {2} \sum_ {i = 1} ^ {n} \left(w _ {t}\right) _ {i} \left(u _ {t}\right) _ {i} ^ {2} \geq 2 \eta R ^ {2}. \tag {4}
$$