SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For a matrix  $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$  and  $i \in [n_1]$ , we set  $\rho_i(\mathbf{X}) = \| (\mathbf{X})_{i',\cdot} \|_2$  where  $i'$  is a row index corresponding to the  $i$ -th largest row of  $\mathbf{X}$  in  $\ell_2$ -norm. More precisely, if  $\overline{\mathbf{X}}$  is a decreasing rearrangement of  $\mathbf{X}$  with rows ordered by magnitude in  $\ell_2$ -norm, then  $\rho_i(\mathbf{X}) = \| (\overline{\mathbf{X}})_{i,\cdot} \|_2$ . As the following lemma shows, the quantity  $\rho_s(\mathbf{X})$  determines a local neighborhood of  $\mathbf{X}$  on which  $\mathsf{H}_s$  preserves the row-support.
Lemma B.2. Let  $\mathbf{X} \in \mathbb{R}^{n_1 \times n_2}$  be a matrix with row-support  $S \subset [n_1]$  and  $|S| = s$ . Then, for any  $\mathbf{Z} \in \mathbb{R}^{n_1 \times n_2}$  with  $\| \mathbf{X} - \mathbf{Z} \|_{\infty, 2} \coloneqq \max_{i \in [n_1]} \| \mathbf{X}_{i,:} - \mathbf{Z}_{i,:} \|_2 \leq \frac{1}{2} \rho_s(\mathbf{X})$  the matrix  $\mathsf{H}_s(\mathbf{Z})$  has row-support  $S$ .
Proof: Note that
$$
\max  _ {i \in [ n _ {1} ]} \| (\mathbf {Z}) _ {i,:} - (\mathbf {X}) _ {i,:} \| _ {2} \leq \frac {1}{2} \rho_ {s} (\mathbf {X})
$$
implies that any non-zero row of  $\mathbf{X}$  corresponds to a non-zero row of  $\mathsf{H}_s(\mathbf{Z})$  and hence yields the claim.
A first important observation is that if  $\mathcal{A}$  has the  $(r,s)$ -RIP, then the norm of kernel elements of  $\mathcal{A}$  is bounded in the following way.
Lemma B.3. If  $\mathcal{A}$  has the  $(r,s)$ -RIP with  $\delta \in (0,1)$  and  $\mathbf{U} \in \mathbb{R}^{n_1 \times r}, \mathbf{V} \in \mathbb{R}^{n_2 \times r}$  with  $\mathrm{supp}(\mathbf{U}) = S$ ,  $|S| \leq s$ , then
$$
\left\| \boldsymbol {\Xi} \right\| _ {F} \leq \sqrt {1 + \frac {\left\| \mathcal {A} \right\| _ {2 \rightarrow 2} ^ {2}}{(1 - \delta)}} \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, s}} (\boldsymbol {\Xi}) \right\| _ {F},
$$
for all  $\Xi \in \ker (\mathcal{A})$
The proof of Lemma B.3 is presented in Appendix C.3.
Remark B.4. If  $\mathcal{A}$  is a Gaussian operator with standard deviation  $\sqrt{\frac{1}{m}}$ , one has with high probability that  $\| \mathcal{A} \|_{2 \to 2}^2 \approx \frac{n_1 n_2}{m}$ .
We use of the following lemma to characterize the solution of the weighted least squares problem (11). Its proof is analogous to [54, Lemma B.7] and [26, Lemma 5.2].
Lemma B.5. Let  $\mathcal{A}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  and  $\mathbf{y}\in \mathbb{R}^m$ . Let  $W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  be the weight operator (8) defined based on the information of  $\mathbf{X}^{(k)}\in \mathbb{R}^{n_1\times n_2}$ . Then the solution of the weighted least squares step (11) of Algorithm 1
$$
\mathbf {X} ^ {(k + 1)} = \underset {\mathcal {A} (\mathbf {X}) = \mathbf {y}} {\arg \min } \langle \mathbf {X}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} (\mathbf {X}) \rangle , \tag {21}
$$
is unique and solves (21) if and only if
$$
\mathcal {A} \left(\mathbf {X} ^ {(k + 1)}\right) = \mathbf {y} \quad a n d \quad \langle W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} \left(\mathbf {X} ^ {(k + 1)}\right), \boldsymbol {\Xi} \rangle = 0 \text {f o r a l l} \boldsymbol {\Xi} \in \ker \mathcal {A}. \tag {22}
$$
For any iterate  $\mathbf{X}^{(k)}$  of Algorithm 1, we furthermore abbreviate the tangent space (20) of the fixed rank- $r$  manifold  $\mathcal{M}_r$  restricted to  $S$  at  $\mathsf{H}_s(\mathbf{X}^{(k)})$  by
$$
T _ {k} = T _ {\tilde {\mathbf {U}}, \tilde {\mathbf {V}}, S}, \tag {23}
$$
where  $\widetilde{\mathbf{U}}\in \mathbb{R}^{n_1\times r}$  and  $\widetilde{\mathbf{V}}\in \mathbb{R}^{n_2\times r}$  are matrices with leading  $^4$ $r$  singular vectors of  $\mathsf{H}_s(\mathbf{X}^{(k)})$  as columns, and  $S\in [n_{1}]$  is the support set of the  $s$  rows of  $\mathbf{X}^{(k)}$  with largest  $\ell_2$  -norm.
The following lemma is the first crucial tool for showing local quadratic convergence of Algorithm 1.
Lemma B.6. Let  $\mathbf{X}_{\star} \in \mathcal{M}_{r,s}$  and let  $\mathbf{X}^{(k)}$  be the  $k$ -th iterate of Algorithm 1 with rank and sparsity parameters  $\widetilde{r} = r$  and  $\widetilde{s} = s$ , let  $\delta_k, \varepsilon_k$  be such that  $s_k$  and  $r_k$  from Definition 2.1 satisfy  $s_k \geq s$  and  $r_k \geq r$ . Assume that there exists a constant  $c > 1$  such that
$$
\left\| \boldsymbol {\Xi} \right\| _ {F} \leq c \left\| \mathbb {P} _ {T _ {k} ^ {\perp}} (\boldsymbol {\Xi}) \right\| _ {F} \quad \text {f o r a l l} \quad \boldsymbol {\Xi} \in \ker (\mathcal {A}), \tag {24}
$$
where  $T_{k} = T_{\widetilde{\mathbf{U}},\widetilde{\mathbf{V}},S}$  is as defined in (23) for matrices  $\widetilde{\mathbf{U}} \in \mathbb{R}^{n_1 \times r}$  and  $\widetilde{\mathbf{V}} \in \mathbb{R}^{n_2 \times r}$  of leading  $r$  left and right singular vectors of  $\mathsf{H}_s(\mathbf{X}^{(k)})$  and  $S \subset [n_1]$  is the support set of  $\mathsf{H}_s(\mathbf{X}^{(k)})$ . Assume furthermore that
$$
\left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| \leq \min  \left\{\frac {1}{2} \rho_ {s} \left(\mathbf {X} _ {\star}\right), \min  \left\{\frac {1}{4 8}, \frac {1}{1 9 c} \right\} \sigma_ {r} \left(\mathbf {X} _ {\star}\right) \right\}. \tag {25}
$$
$$
\begin{array}{l} \left\| \mathbf {X} ^ {(k + 1)} - \mathbf {X} _ {\star} \right\| \leq 4 c ^ {2} \min  \left\{\frac {\sigma_ {r + 1} (\mathbf {X} ^ {(k)})}{\varepsilon_ {k}}, \frac {\rho_ {s + 1} (\mathbf {X} ^ {(k)})}{\delta_ {k}} \right\} ^ {2} \\ \cdot \left(\left\| W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} \left(\mathbf {X} _ {\star}\right) \right\| _ {*} + \left\| \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \cdot \mathbf {X} _ {\star} \right\| _ {1, 2}\right), \\ \end{array}
$$
where  $\| \mathbf{M}\|_{1,2} = \sum_{i}\| \mathbf{M}_{i,:}\|_{2}$  denotes the row-sum norm of a matrix  $\mathbf{M}$ , and  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}$  and  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}$  are the weight operators (9) and (10) from Definition 2.1.
The proof of Lemma B.6 is presented in Appendix C.4.
Remark B.7. By revisiting the proof of Lemma B.6 (omit the bound in (48) and keep the term  $\langle \Xi, \overline{W} \Xi \rangle$  until the end), one can show under the same assumptions as in Lemma B.6 that
$$
\| \Xi \| _ {F} ^ {2} \leq 4 c ^ {2} \min \left\{\frac {\sigma_ {r + 1} (\mathbf {X} ^ {(k)})}{\varepsilon_ {k}}, \frac {\rho_ {s + 1} (\mathbf {X} ^ {(k)})}{\delta_ {k}} \right\} ^ {2} \left\langle \Xi , \left(\mathbb {P} _ {\mathbf {U}, \mathbf {V}} ^ {\perp} W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} \mathbb {P} _ {\mathbf {U}, \mathbf {V}} ^ {\perp} + \mathbb {P} _ {S ^ {c}} \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \mathbb {P} _ {S ^ {c}}\right) \Xi \right\rangle ,
$$
where  $\mathbf{U}$  and  $\mathbf{V}$  are containing the left and right singular vectors of  $\mathbf{X}^{(k)}$ , see Definition 2.1.
As  $\widetilde{\mathbf{U}}$  and  $\widetilde{\mathbf{V}}$  might not be unique, any set of  $r$  leading singular vectors can be chosen in this definition.
The contribution of the norms of the weighted  $\mathbf{X}_{\star}$  terms in Lemma B.6 can be controlled by Lemmas B.8 and B.9 below.
Lemma B.8. Let  $W_{\mathbf{X}^{(k)}, \varepsilon_k}^{lr} : \mathbb{R}^{n_1 \times n_2} \to \mathbb{R}^{n_1 \times n_2}$  be the rank-based weight operator (9) that uses the spectral information of  $\mathbf{X}^{(k)}$  and let  $\mathbf{X}_{\star} \in \mathbb{R}^{n_1 \times n_2}$  be a rank- $r$  matrix. Assume that there exists  $0 < \zeta < \frac{1}{2}$  such that
$$
\max  \left\{\varepsilon_ {k}, \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| \right\} \leq \zeta \sigma_ {r} \left(\mathbf {X} _ {\star}\right). \tag {26}
$$
Then for each  $1 \leq q \leq \infty$ ,
$$
\left\| W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {X} _ {\star}) \right\| _ {S _ {q}} \leq \frac {r ^ {1 / q}}{(1 - \zeta) \sigma_ {r} (\mathbf {X} _ {\star})} \left(\frac {1}{1 - \zeta} \varepsilon_ {k} ^ {2} + \varepsilon_ {k} K _ {q} \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| + 2 \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| ^ {2}\right)
$$
$$
\left\| W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {X} _ {\star}) \right\| _ {S _ {q}} \leq \frac {1}{(1 - \zeta) \sigma_ {r} (\mathbf {X} _ {\star})} \left(\frac {r ^ {1 / q}}{1 - \zeta} \varepsilon_ {k} ^ {2} + 2 \left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| _ {S _ {q}} \left(\varepsilon_ {k} + \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \|\right)\right)
$$
where  $K_{q}$  is such that  $K_{q} = 2^{1 / q}$  for  $1 \leq q \leq 2$  and  $4 \leq q$ ,  $K_{q} = \sqrt{2}$  for  $2 < q \leq 4$  and  $K_{q} = 1$  for  $q = \infty$ .
Lemma B.9. Let  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}\in \mathbb{R}^{n_1\times n_1}$  be the row-sparsity-based weight operator (10) that uses the current iterate  $\mathbf{X}^{(k)}$  with  $\delta_{k} = \min \left(\delta_{k - 1},\rho_{s + 1}(\mathbf{X}^{(k)})\right)$  and let  $\mathbf{X}_{\star}\in \mathbb{R}^{n_1\times n_2}$  be an  $s$ -row-sparse matrix. Assume that there exists  $0 < \zeta < \frac{1}{2}$  such that
$$
\left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| _ {\infty , 2} = \max  _ {i \in [ n _ {1} ]} \| \left(\mathbf {X} ^ {(k)}\right) _ {i,:} - \left(\mathbf {X} _ {\star}\right) _ {i,:} \| _ {2} \leq \zeta \rho_ {s} \left(\mathbf {X} _ {\star}\right), \tag {27}
$$
where  $\rho_s(\mathbf{M})$  denotes the  $\ell_2$ -norm of the in  $\ell_2$ -norm  $s$ -largest row of  $\mathbf{M}$ . Then
$$
\left\| \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \cdot \mathbf {X} _ {\star} \right\| _ {1, 2} \leq \frac {s \delta_ {k} ^ {2}}{(1 - \zeta) ^ {2} \rho_ {s} (\mathbf {X} _ {\star})}
$$
Lemma B.8 is a refined version of [54, Lemma B.9] the proof of which we omit here. The proof of Lemma B.9 is provided in Appendix C.5. Finally, the following lemma will allow us to control the decay of the IRLS parameters  $\delta_{k}$  and  $\varepsilon_{k}$ .
Lemma B.10 ([54, Lemma B.5]). Let  $\mathbf{X}_{\star} \in \mathcal{M}_{r,s}$ , assume that  $\mathcal{A}$  has the  $(r,s)$ -RIP with  $\delta \in (0,1)$  and let us abbreviate  $n = \min \{n_1, n_2\}$ .
Assume that the  $k$ -th iterate  $\mathbf{X}^{(k)}$  of Algorithm 1 with  $\widetilde{r} = r$  and  $\widetilde{s} = r$  updates the smoothing parameters in (12) such that one of the statements  $\varepsilon_{k} = \sigma_{r + 1}(\mathbf{X}^{(k)})$  or  $\delta_{k} = \rho_{s + 1}(\mathbf{X}^{(k)})$  is true, and that  $r_k\geq r$  and  $s_k\geq s$ . Furthermore, let
$$
\varepsilon_ {k} \leq \frac {1}{4 8} \sigma_ {r} (\mathbf {X} _ {\star})
$$
with  $c_{\| \mathcal{A}\|_{2\to 2}} = \sqrt{1 + \frac{\|\mathcal{A}\|_{2\to 2}^2}{(1 - \delta)}}$ , let  $\Xi^{(k)} := \mathbf{X}^{(k)} - \mathbf{X}_{\star}$  satisfy
$$
\left\| \boldsymbol {\Xi} ^ {(k)} \right\| \leq \min  \left\{\frac {1}{2} \rho_ {s} \left(\mathbf {X} _ {\star}\right), \min  \left\{\frac {1}{4 8}, \frac {1}{2 1 c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}}} \right\} \sigma_ {r} \left(\mathbf {X} _ {\star}\right)\right\}. \tag {28}
$$
$$
\| \Xi^ {(k)} \| _ {F} \leq 2 \sqrt {2} \sqrt {n} c _ {\| \mathcal {A} \| _ {2 \to 2}} \sqrt {4 \varepsilon_ {k} ^ {2} + \delta_ {k} ^ {2}}.
$$
The proof of Lemma B.10 is provided in Appendix C.6. We finally have all the tools to prove Theorem 2.5. Note that (14) implies
$$
\left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| \leq \min  \left\{\frac {1}{4 8 c _ {\parallel \mathcal {A} \parallel_ {2 \rightarrow 2}} ^ {2}} \min  \left\{\frac {\sigma_ {r} (\mathbf {X} _ {\star})}{r}, \frac {\rho_ {s} (\mathbf {X} _ {\star})}{s} \right\}, \frac {1}{4 \mu \sqrt {5 n} c _ {\parallel \mathcal {A} \parallel_ {2 \rightarrow 2}}} \right\}, \tag {29}
$$
This result is a technical result of an unpublished paper. In this paper, we only use that result as a tool. If the reviewers think that adding the proof is relevant here, we are happy to provide it.
$$
\varepsilon_ {k} \leq \frac {1}{4 8} \sigma_ {r} \left(\mathbf {X} _ {\star}\right) \tag {30}
$$
which we will use in the proof below. The latter follows from the fact that for  $\tilde{r} = r$
$$
\varepsilon_ {k} = \min  \left(\varepsilon_ {k - 1}, \sigma_ {r + 1} (\mathbf {X} ^ {(k)})\right) \leq \sigma_ {r + 1} (\mathbf {X} ^ {(k)}) \leq | | \mathbf {X} ^ {(k)} - \mathbf {X} _ {*} | | \leq \sigma_ {r} (\mathbf {X} _ {*}) / 4 8.
$$
Proof of Theorem 2.5: First note, that by assumption  $\tilde{r} = r$  and  $\tilde{s} = s$ . Furthermore, since  $\frac{1}{48c_{\|A\|_{2\to 2}}^2}\leq \frac{1}{2}$ , the closeness assumption (29) implies that  $\mathsf{H}_s(\mathbf{X}^{(k)})$  and  $\mathbf{X}_{\star}$  share the same support due to Lemma B.2.
Let  $\mathbf{X}^{(k)}$  be the  $k$ -th iterate of Algorithm 1. Since the operator  $\mathcal{A}\colon \mathbb{R}^{n_1\times n_2}\to \mathbb{R}^m$  has the  $(r,s)$ -RIP with  $\delta \in (0,1)$ , Lemma B.3 yields for all  $\mathbf{U}\in \mathbb{R}^{n_1\times r},\mathbf{V}\in \mathbb{R}^{n_2\times r}$  with  $\mathrm{supp}(\mathbf{U}) = S$ ,  $|S|\leq s$ , that
$$
\left\| \boldsymbol {\Xi} \right\| _ {F} \leq c _ {\left\| \boldsymbol {\mathcal {A}} \right\| _ {2 \rightarrow 2}} \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} ^ {\perp} (\boldsymbol {\Xi}) \right\| _ {F},
$$
for any  $\Xi \in \ker (\mathcal{A})$ . Furthermore, due to our assumption that  $\tilde{s} = s$  and  $\tilde{r} = r$ , the smoothing parameter update rules in (12), i.e.,  $\delta_{k} = \min \left(\delta_{k - 1},\rho_{s + 1}(\mathbf{X}^{(k)})\right)$  and  $\varepsilon_{k} = \min \left(\varepsilon_{k - 1},\sigma_{r + 1}(\mathbf{X}^{(k)})\right)$ , imply that  $r_k\geq r$  and  $s_k\geq s$  for all  $k$ . We can thus apply Lemma B.6 for  $\Xi^{(k)}\coloneqq \mathbf{X}^{(k)} - \mathbf{X}_{\star}$  (note at this point that (29) implies the closeness assumption (25) of Lemma B.6) and obtain
$$
\begin{array}{l} \left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| = \left\| \mathbf {X} ^ {(k + 1)} - \mathbf {X} _ {\star} \right\| \\ \leq 4 c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}} ^ {2} \min  \left\{\frac {\sigma_ {r + 1} \left(\mathbf {X} ^ {(k)}\right)}{\varepsilon_ {k}}, \frac {\rho_ {s + 1} \left(\mathbf {X} ^ {(k)}\right)}{\delta_ {k}} \right\} ^ {2} \left(\left\| W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} \left(\mathbf {X} _ {\star}\right)\right\| _ {*} + \left\| \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \cdot \mathbf {X} _ {\star} \right\| _ {1, 2}\right), \tag {31} \\ \end{array}
$$
where  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  is the low-rank promoting part (9) of the weight operator associated to  $\mathbf{X}^{(k)}$  and  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}\in \mathbb{R}^{n_1\times n_1}$  the sparsity promoting part (10). Since by assumption
$$
\max  (\varepsilon_ {k}, \| \boldsymbol {\Xi} ^ {(k)} \|) \leq \frac {1}{4 8} \sigma_ {r} (\mathbf {X} _ {\star}),
$$
Lemma B.8 yields
$$
\left\| W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} \left(\mathbf {X} _ {\star}\right) \right\| _ {*} \leq 0. 9 9 5 \frac {4 2}{4 0 \sigma_ {r} \left(\mathbf {X} _ {\star}\right)} \left(\varepsilon_ {k} ^ {2} r + 2 \varepsilon_ {k} \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| + 2 \| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \| ^ {2}\right). \tag {32}
$$
Similarly, by assumption
$$
\| \boldsymbol {\Xi} ^ {(k)} \| _ {\infty , 2} \leq \| \boldsymbol {\Xi} ^ {(k)} \| \leq \frac {1}{4 8 s} \rho_ {s} (\mathbf {X} _ {\star}) \leq \frac {1}{4 8} \rho_ {s} (\mathbf {X} _ {\star}),
$$
such that Lemma B.9 yields
$$
\left\| \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \cdot \mathbf {X} _ {\star} \right\| _ {1, 2} \leq 0. 9 9 5 \frac {2 1 s \delta_ {k} ^ {2}}{2 0 \rho_ {s} (\mathbf {X} _ {\star})}. \tag {33}
$$
Inserting (32) and (33) into (31) we obtain that
$$
\begin{array}{l} \left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| \leq 0. 9 9 5 \cdot 4. 2 c _ {| | \mathcal {A} | | _ {2 \to 2}} ^ {2} \min \left\{\frac {\sigma_ {r + 1} (\mathbf {X} ^ {(k)})}{\varepsilon_ {k}}, \frac {\rho_ {s + 1} (\mathbf {X} ^ {(k)})}{\delta_ {k}} \right\} ^ {2} \\ \cdot \left(\frac {r}{\sigma_ {r} (\mathbf {X} _ {\star})} \left(\varepsilon_ {k} ^ {2} + 2 \varepsilon_ {k} \| \boldsymbol {\Xi} ^ {(k)} \| + 2 \| \boldsymbol {\Xi} ^ {(k)} \| ^ {2}\right) + \frac {2 s}{\rho_ {s} (\mathbf {X} _ {\star})} \delta_ {k} ^ {2}\right). \\ \end{array}
$$
Due to the assertion that  $r_k \geq r$ , it holds that  $\varepsilon_k \leq \sigma_{r+1}(\mathbf{X}^{(k)})$ . Therefore, Lemma C.3 yields that
$$
\left(\varepsilon_ {k} ^ {2} + 2 \varepsilon_ {k} \| \boldsymbol {\Xi} ^ {(k)} \| + 2 \| \boldsymbol {\Xi} ^ {(k)} \| ^ {2}\right) \leq 5 \| \boldsymbol {\Xi} ^ {(k)} \| ^ {2}.
$$
and, since  $s_k \geq s$ , also that
$$
\delta_ {k} ^ {2} \leq \| \Xi^ {(k)} \| _ {\infty , 2} ^ {2} \leq \| \Xi^ {(k)} \| ^ {2},
$$
since  $\delta_k \leq \rho_{s+1}(\mathbf{X}^{(k)})$  in this case.
Thus, using the assertion that one of the statements  $\varepsilon_{k} = \sigma_{r + 1}(\mathbf{X}^{(k)})$  or  $\delta_k = \rho_{s + 1}(\mathbf{X}^{(k)})$  is true, we obtain from (34) that
$$
\left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| \leq 0. 9 9 5 \cdot 4. 2 c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}} ^ {2} \cdot \left(\frac {5 r}{\sigma_ {r} (\mathbf {X} _ {\star})} + \frac {2 s}{\rho_ {s} (\mathbf {X} _ {\star})}\right)\left\| \boldsymbol {\Xi} ^ {(k)} \right\| ^ {2}. \tag {35}
$$
For  $\left\| \Xi^{(k)}\right\| < \frac{1}{48} c_{\|\mathcal{A}\|_{2\to 2}}^{-2}\min \left\{\frac{\sigma_r(\mathbf{X}_\star)}{r},\frac{\rho_s(\mathbf{X}_\star)}{s}\right\}$  (as implied by (29)), this yields
$$
\left\| \Xi^ {(k + 1)} \right\| <   0. 9 \left\| \Xi^ {(k)} \right\| \tag {36}
$$
and the quadratic error decay
$$
\| \Xi^ {(k + 1)} \| \leq \mu \| \Xi^ {(k)} \| ^ {2}
$$
if we define  $\mu = 4.179c_{\| \mathcal{A}\|_{2\to 2}}^2\big(\frac{5r}{\sigma_r(\mathbf{X}_\star)} +\frac{2s}{\rho_s(\mathbf{X}_\star)}\big)$
To show the remaining statement, we need to argue that the assertions of Theorem 2.5 are satisfied not only for  $k$ , but for any  $k + \ell$  with  $\ell \geq 1$ . For this, it is sufficient to show that
1.  $r_{k + 1}\geq r$
2.  $s_{k + 1}\geq s$
3.  $\varepsilon_{k + 1}\leq \frac{1}{48}\sigma_r(\mathbf{X}_\star)$
4. (29) holds for  $\mathbf{X}^{(k + 1)}$ , and that
5. one of the statements  $\varepsilon_{k + 1} = \sigma_{r + 1}(\mathbf{X}^{(k + 1)})$  or  $\delta_{k + 1} = \rho_{s + 1}(\mathbf{X}^{(k + 1)})$  is true,
as in this case,  $\mathbf{X}^{(k + \ell)}\stackrel {\ell \to \infty}{\rightarrow}\mathbf{X}_{\star}$  follows by induction due to successive application of (36).
For 1. and 2., we see that this follows from the smoothing parameter update rules (12) which imply that  $\varepsilon_{k + 1} \leq \sigma_{r + 1}(\mathbf{X}^{(k + 1)})$  and  $\delta_{k + 1} \leq \rho_{s + 1}(\mathbf{X}^{(k + 1)})$ .
3. follows from (30) and the fact that due to (12),  $(\varepsilon_{k})_{k\geq 1}$  is non-increasing. 4. is satisfied due to (36) and (29). To show 5., we note that due to (29), the assertion (28) is satisfied, and therefore it follows from (35) and Lemma B.10 that
$$
\left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| \leq 4. 1 7 9 c _ {\| \mathcal {A} \| _ {2 \to 2}} ^ {2} \left(\frac {5 r}{\sigma_ {r} (\mathbf {X} _ {\star})} + \frac {2 s}{\rho_ {s} (\mathbf {X} _ {\star})}\right) \left\| \boldsymbol {\Xi} ^ {(k)} \right\| \cdot 2 \sqrt {2} \sqrt {n} c _ {\| \mathcal {A} \| _ {2 \to 2}} \sqrt {4 \varepsilon_ {k} ^ {2} + \delta_ {k} ^ {2}}.
$$
We now distinguish the case (i)  $\delta_k < \varepsilon_k$  and the case (ii)  $\delta_k \geq \varepsilon_k$ .
In case (i), it holds that
$$
\begin{array}{l} \sigma_ {r + 1} \left(\mathbf {X} ^ {(k + 1)}\right) \leq \left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| \leq 4. 1 7 9 c _ {\parallel \mathcal {A} _ {\parallel_ {2} \rightarrow 2}} ^ {2} \left(\frac {5 r}{\sigma_ {r} \left(\mathbf {X} _ {\star}\right)} + \frac {2 s}{\rho_ {s} \left(\mathbf {X} _ {\star}\right)}\right) 2 \sqrt {1 0 n} c _ {\parallel , \mathcal {A} _ {\parallel_ {2} \rightarrow 2}} \left\| \boldsymbol {\Xi} ^ {(k)} \right\| \varepsilon_ {k} \\ = \mu 2 \sqrt {1 0 n} c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}} \left\| \boldsymbol {\Xi} ^ {(k)} \right\| \varepsilon_ {k} \\ <   \varepsilon_ {k}, \\ \end{array}
$$
where the last inequality holds since by (29) the  $k$ -th iterate  $\mathbf{X}^{(k)}$  additionally satisfies
$$
\left\| \mathbf {X} ^ {(k)} - \mathbf {X} _ {\star} \right\| <   \frac {1}{2 \mu \sqrt {1 0} c _ {\| \mathcal {A} \| _ {2 \rightarrow 2}}}. \tag {37}
$$
In this case, due to the smoothing parameter update rule (12), we have that  $\varepsilon_{k + 1} = \sigma_{r + 1}(\mathbf{X}^{(k + 1)})$ .
In case (ii), we have likewise that
$$
\rho_ {s + 1} (\mathbf {X} ^ {(k + 1)}) \leq \left\| \boldsymbol {\Xi} ^ {(k + 1)} \right\| \leq \mu 2 \sqrt {1 0 n} c _ {\| \mathcal {A} \| _ {2 \to 2}} \left\| \boldsymbol {\Xi} ^ {(k)} \right\| \delta_ {k} <   \delta_ {k},
$$
due to (35), Lemma B.10, and (37). Hence,  $\delta_{k + 1} = \rho_{s + 1}\bigl (\mathbf{X}^{(k + 1)}\bigr)$  which shows the remaining statement 5. and concludes the proof of Theorem 2.5.
B.2 Proof of Theorem 2.6
1.) Let  $\varepsilon, \delta > 0$  be arbitrary. Due to the additive structure of  $\mathcal{F}_{\varepsilon, \delta}(\cdot)$ , cf. (4), it is sufficient to establish that
$$
\mathcal {F} _ {s p, \delta} (\mathbf {Z}) \leq \mathcal {Q} _ {s p, \delta} (\mathbf {Z} | \mathbf {X}) = \mathcal {F} _ {s p, \delta} (\mathbf {X}) + \left\langle \nabla \mathcal {F} _ {s p, \delta} (\mathbf {X}), \mathbf {Z} - \mathbf {X} \right\rangle + \frac {1}{2} \left\langle \mathbf {Z} - \mathbf {X}, \mathbf {W} _ {\mathbf {X}, \delta} ^ {s p} (\mathbf {Z} - \mathbf {X}) \right\rangle \tag {38}
$$
for any  $\mathbf{Z},\mathbf{X}\in \mathbb{R}^{n_1\times n_2}$ , where  $\mathbf{W}_{\mathbf{X},\delta}^{sp}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  is defined analogously to (10) and
$$
\mathcal {F} _ {l r, \varepsilon} (\mathbf {Z}) \leq \mathcal {Q} _ {l r, \varepsilon} (\mathbf {Z} | \mathbf {X}) = \mathcal {F} _ {l r, \varepsilon} (\mathbf {X}) + \left\langle \nabla \mathcal {F} _ {l r, \varepsilon} (\mathbf {X}), \mathbf {Z} - \mathbf {X} \right\rangle + \frac {1}{2} \left\langle \mathbf {Z} - \mathbf {X}, W _ {\mathbf {X}, \varepsilon} ^ {l r} (\mathbf {Z} - \mathbf {X}) \right\rangle , \tag {39}
$$
for any  $\mathbf{Z},\mathbf{X}\in \mathbb{R}^{n_1\times n_2}$ , where  $W_{\mathbf{X},\varepsilon}^{lr}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  is defined analogously to (9).
The argument for (38) is standard in the IRLS literature [2, 73, 76] and is based on the facts that both  $\mathcal{Q}_{sp,\delta}(\mathbf{Z}|\mathbf{X})$  and  $\mathcal{F}_{sp,\delta}(\mathbf{Z})$  are row-wise separable, and that  $t\mapsto f_{\sqrt{\tau}}(\sqrt{t})$  is concave and therefore majorized by its linearization: indeed, let  $g_{\tau}:\mathbb{R}\to \mathbb{R}$  be such that
$$
g _ {\tau} (t) := \left\{ \begin{array}{l l} \frac {1}{2} \tau \log (e | t | / \tau), & \text {i f} | t | > \tau , \\ \frac {1}{2} | t |, & \text {i f} | t | \leq \tau . \end{array} \right.
$$
The function  $g_{\tau}(\cdot)$  is continuously differentiable with derivative  $g_{\tau}'(t) = \frac{\tau}{2\max(|t|, \tau)} \mathrm{sign}(t)$  and furthermore, concave restricted to the non-negative domain  $\mathbb{R}_{\geq 0}$ .
Therefore, it holds for any  $t,t^{\prime}\in \mathbb{R}_{\geq 0}$  that
$$
g _ {\tau} (t) \leq g _ {\tau} \left(t ^ {\prime}\right) + g _ {\tau} ^ {\prime} \left(t ^ {\prime}\right) \left(t - t ^ {\prime}\right).
$$
We recall the definition  $f_{\tau}(t) = \frac{1}{2}\tau^{2}\log (et^{2} / \tau^{2})$  for  $|t| > \tau$  and  $f_{\tau}(t) = \frac{1}{2} t^2$  for  $|t|\leq \tau$  from (2) with derivative  $f_{\tau}'(t) = \frac{\max(t^2,\tau^2)t}{t^2} = \frac{\tau^2t}{\max(t^2,\tau^2)}$ . Thus, for any  $x,z\in \mathbb{R}$ , it follows that