SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f _ {\tau} (z) = g _ {\tau^ {2}} \left(z ^ {2}\right) \leq g _ {\tau^ {2}} \left(x ^ {2}\right) + g _ {\tau^ {2}} ^ {\prime} \left(x ^ {2}\right) \left(z ^ {2} - x ^ {2}\right) \\ = f _ {\tau} (x) + \frac {\tau^ {2}}{2 \max (x ^ {2} , \tau^ {2})} (z ^ {2} - x ^ {2}), \\ \end{array}
$$
and inserting  $\tau = \delta$ ,  $z = \|\mathbf{Z}_{i,:}\|_2$ ,  $x = \|\mathbf{X}_{i,:}\|_2$  and summing over  $i = 1, \ldots, n_1$  implies that
$$
\begin{array}{l} \mathcal {F} _ {s p, \delta} (\mathbf {Z}) = \sum_ {i = 1} ^ {n _ {1}} f _ {\delta} \left(\| \mathbf {Z} _ {i,:} \| _ {2}\right) \leq \mathcal {F} _ {s p, \delta} (\mathbf {X}) + \sum_ {i = 1} ^ {n _ {1}} \frac {\delta^ {2}}{2 \max \left(\| \mathbf {X} _ {i , :} \| _ {2} ^ {2} , \delta^ {2}\right)} \left(\| \mathbf {Z} _ {i,:} \| _ {2} ^ {2} - \| \mathbf {X} _ {i,:} \| _ {2} ^ {2}\right) \\ = \mathcal {F} _ {s p, \delta} (\mathbf {X}) + \sum_ {i = 1} ^ {n _ {1}} \frac {\delta^ {2}}{\max (\| \mathbf {X} _ {i , :} \| _ {2} ^ {2} , \delta^ {2})} \langle \mathbf {X} _ {i,:}, \mathbf {Z} _ {i,:} - \mathbf {X} _ {i,:} \rangle + \frac {1}{2} \sum_ {i = 1} ^ {n _ {1}} \frac {\left\| \mathbf {Z} _ {i , :} - \mathbf {X} _ {i , :} \right\| _ {2} ^ {2}}{\max (\| \mathbf {X} _ {i , :} \| _ {2} ^ {2} / \delta^ {2} , 1)} \\ \end{array}
$$
From the chain rule, it follows that for all  $i = 1,\dots ,n_{1}$  for which  $\mathbf{X}_{i,:}\neq 0$
$$
\frac {d}{d \mathbf {X} _ {i , :}} f _ {\delta} \left(\| \mathbf {X} _ {i,:} \| _ {2}\right) = f _ {\delta} ^ {\prime} \left(\| \mathbf {X} _ {i,:} \| _ {2}\right) \frac {d \| \mathbf {X} _ {i , :} \| _ {2}}{d \mathbf {X} _ {i , :}} = \frac {\delta^ {2} \| \mathbf {X} _ {i ,} \| _ {2}}{\max \left(\| \mathbf {X} _ {i , :} \| _ {2} ^ {2} , \delta^ {2}\right)} \frac {\mathbf {X} _ {i , :}}{\| \mathbf {X} _ {i , :} \| _ {2}}, = \frac {\delta^ {2} \mathbf {X} _ {i , :}}{\max \left(\| \mathbf {X} _ {i , :} \| _ {2} ^ {2} , \delta^ {2}\right)} \tag {40}
$$
$$
\mathcal {F} _ {s p, \delta} (\mathbf {Z}) \leq \mathcal {F} _ {s p, \delta} (\mathbf {X}) + \langle \nabla \mathcal {F} _ {s p, \delta} (\mathbf {X}), \mathbf {Z} - \mathbf {X} \rangle + \frac {1}{2} \langle \mathbf {Z} - \mathbf {X}, \mathbf {W} _ {\mathbf {X}, \delta} ^ {s p} (\mathbf {Z} - \mathbf {X}) \rangle
$$
which shows the majorization of (38), recalling the definition  $\mathbf{W}_{\mathbf{X},\delta}^{sp}$  = diag  $\left(\max \left(\| \mathbf{X}_{i,:}\| _2^2 /\delta^2,1\right)_{i = 1}^{d_1}\right)^{-1}$  of (10).
The majorization of (39) is non-trivial but follows in a straightforward way from [53, Theorem 2.4] as the objective  $\mathcal{F}_{lr,\varepsilon}(\mathbf{Z})$  corresponds to the one of [53, Theorem 2.4] up to a multiplicative factor of  $\varepsilon^2$  and constant additive factors, and since the weight operator  $W_{\mathbf{X},\varepsilon}^{lr}$  corresponds to the weight operator used in [53, Chapter 2] for  $p = 0$ .
2.) Due to the definition (3) of  $\mathcal{F}_{sp,\delta_k}(\cdot)$  and the derivative computation of (40), we observe that
$$
\nabla \mathcal {F} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k)}) = \mathrm {d i a g} \left(\left(\max \left(\| (\mathbf {X} ^ {(k)}) _ {i,:} \| _ {2} ^ {2} / \delta_ {k} ^ {2}, 1\right) ^ {- 1}\right) _ {i = 1} ^ {d _ {1}}\right) \mathbf {X} ^ {(k)} = \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \cdot \mathbf {X} ^ {(k)},
$$
comparing the resulting term with the definition of (10) of  $\mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}$ . Furthermore, an analogue equality follows from the formula
$$
\nabla \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)}) = [ \mathbf {U} \quad \mathbf {U} _ {\perp} ] \mathrm {d i a g} \left(\left(\sigma_ {i} ^ {(k)} \max \left((\sigma_ {i} ^ {(k)}) ^ {2} / \varepsilon_ {k} ^ {2}, 1\right) ^ {- 1}\right) _ {i = 1} ^ {d}\right) \left[ \begin{array}{c} \mathbf {V} ^ {*} \\ \mathbf {V} _ {\perp} ^ {*} \end{array} \right]
$$
with  $\sigma_{i}^{(k)} = \sigma_{i}(\mathbf{X}^{(k)})$  for any  $i\leq d$ , which is a direct consequence from the calculus of spectral functions Lemma C.1, and inserting into the low-rank promoting weight operator formula (9)
$$
W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {X} ^ {(k)}) = [ \mathbf {U} \quad \mathbf {U} _ {\perp} ] \boldsymbol {\Sigma} _ {\varepsilon_ {k}} ^ {- 1} \mathrm {d i a g} \left(\left(\sigma_ {i} ^ {(k)}\right) _ {i = 1} ^ {d}\right) \boldsymbol {\Sigma} _ {\varepsilon_ {k}} ^ {- 1} \left[ \begin{array}{c} \mathbf {V} ^ {*} \\ \mathbf {V} _ {\perp} ^ {*} \end{array} \right] = \nabla \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)})
$$
Inserting  $\nabla \mathcal{F}_{sp,\delta_k}(\mathbf{X}^{(k)}) = \mathbf{W}_{\mathbf{X}^{(k)},\delta_k}^{sp}\cdot \mathbf{X}^{(k)}$  and  $\nabla \mathcal{F}_{lr,\varepsilon_k}(\mathbf{X}^{(k)}) = W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}(\mathbf{X}^{(k)})$  into the definitions of  $\mathcal{Q}_{lr,\varepsilon_k}(\mathbf{Z}|\mathbf{X}^{(k)})$  and  $\mathcal{Q}_{sp,\delta_k}(\mathbf{Z}|\mathbf{X}^{(k)})$ , we see that it holds that
$$
\mathcal {Q} _ {l r, \varepsilon_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) = \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)}) + \frac {1}{2} \left(\langle \mathbf {Z}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {Z}) \rangle - \langle \mathbf {X} ^ {(k)}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r} (\mathbf {X} ^ {(k)}) \rangle\right)
$$
$$
\mathcal {Q} _ {s p, \delta_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) = \mathcal {F} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k)}) + \frac {1}{2} \left(\langle \mathbf {Z}, \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \mathbf {Z} \rangle - \langle \mathbf {X} ^ {(k)}, \mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p} \mathbf {X} ^ {(k)} \rangle\right).
$$
Therefore, we see that the weighted least squares solution  $\mathbf{X}^{(k + 1)}$  of (11) for  $k + 1$  coincides with the minimizer of
$$
\begin{array}{l} \min  _ {\mathbf {Z}: \mathcal {A} (\mathbf {Z}) = \mathbf {y}} \left[ \mathcal {Q} _ {l r, \varepsilon_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) + \mathcal {Q} _ {s p, \delta_ {k}} (\mathbf {Z} | \mathbf {X} ^ {(k)}) \right] \\ = \min  _ {\mathbf {Z}: \mathcal {A} (\mathbf {Z}) = \mathbf {y}} \left[ \mathcal {F} _ {l r, \varepsilon_ {k}} \left(\mathbf {X} ^ {(k)}\right) + \mathcal {F} _ {s p, \delta_ {k}} \left(\mathbf {X} ^ {(k)}\right) \right. \tag {41} \\ \left. + \frac {1}{2} \left(\langle \mathbf {Z}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} (\mathbf {Z}) \rangle - \langle \mathbf {X} ^ {(k)}, W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k)}) \rangle\right) \right] \\ \end{array}
$$
with the weight operator  $W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}$  of (8), which implies that
$$
\mathcal {Q} _ {l r, \varepsilon_ {k}} \left(\mathbf {X} ^ {(k + 1)} \mid \mathbf {X} ^ {(k)}\right) + \mathcal {Q} _ {s p, \delta_ {k}} \left(\mathbf {X} ^ {(k + 1)} \mid \mathbf {X} ^ {(k)}\right) \leq \mathcal {Q} _ {l r, \varepsilon_ {k}} \left(\mathbf {X} ^ {(k)} \mid \mathbf {X} ^ {(k)}\right) + \mathcal {Q} _ {s p, \delta_ {k}} \left(\mathbf {X} ^ {(k)} \mid \mathbf {X} ^ {(k)}\right). \tag {42}
$$
Using the majorization (16) established in Statement 1 of Theorem 2.6 and (42), it follows that
$$
\begin{array}{l} \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k + 1)}) \leq \mathcal {Q} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k + 1)} | \mathbf {X} ^ {(k)}) + \mathcal {Q} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k + 1)} | \mathbf {X} ^ {(k)}) \\ \leq \mathcal {Q} _ {l r, \varepsilon_ {k}} \left(\mathbf {X} ^ {(k)} \mid \mathbf {X} ^ {(k)}\right) + \mathcal {Q} _ {s p, \delta_ {k}} \left(\mathbf {X} ^ {(k)} \mid \mathbf {X} ^ {(k)}\right) \tag {43} \\ = \mathcal {F} _ {l r, \varepsilon_ {k}} (\mathbf {X} ^ {(k)}) + \mathcal {F} _ {s p, \delta_ {k}} (\mathbf {X} ^ {(k)}) = \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k)}), \\ \end{array}
$$
using in the third line that  $\mathcal{Q}_{lr,\varepsilon_k}(\mathbf{X}^{(k)}|\mathbf{X}^{(k)}) = \mathcal{F}_{lr,\varepsilon_k}(\mathbf{X}^{(k)})$  and  $\mathcal{Q}_{sp,\delta_k}(\mathbf{X}^{(k)}|\mathbf{X}^{(k)}) = \mathcal{F}_{sp,\delta_k}(\mathbf{X}^{(k)})$ .
To conclude, it suffices to show that  $\varepsilon \mapsto \mathcal{F}_{\varepsilon,\delta_k}(\mathbf{X}^{(k+1)})$  and  $\delta \mapsto \mathcal{F}_{\varepsilon_k,\delta}(\mathbf{X}^{(k+1)})$  are non-decreasing functions, since (43) then extends to
$$
\mathcal {F} _ {\varepsilon_ {k + 1}, \delta_ {k + 1}} (\mathbf {X} ^ {(k + 1)}) \leq \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k + 1}} (\mathbf {X} ^ {(k + 1)}) \leq \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k + 1)}) \leq \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k)}),
$$
where we used that the sequences  $\varepsilon_{k}$  and  $\delta_{k}$  defined in Algorithm 1 are decreasing. So let us prove this last claim. We define for  $t\in \mathbb{R}$  the function  $h_t:\mathbb{R}_{>0}\to \mathbb{R}$  such that  $h_t(\tau) = f_\tau (t)$ , i.e.,
$$
h _ {t} (\tau) = \left\{ \begin{array}{l l} \frac {1}{2} t ^ {2}, & \text {i f} \tau \geq | t |, \\ \frac {1}{2} \tau^ {2} \log (e t ^ {2} / \tau^ {2}), & \text {i f} \tau <   | t |. \end{array} \right.
$$
This function is continuously differentiable with  $h_t'(\tau) = 0$  for all  $\tau > |t|$  and
$$
h _ {t} ^ {\prime} (\tau) = \tau \left(\log (e t ^ {2} / \tau^ {2}) - 1\right)
$$
for  $\tau < |t|$ , which implies that  $h_t'(\tau) \geq 0$  for all  $\tau \geq 0$  and thus shows that  $\varepsilon \mapsto \mathcal{F}_{\varepsilon, \delta_k}(\mathbf{X}^{(k+1)})$  and  $\delta \mapsto \mathcal{F}_{\varepsilon_k, \delta}(\mathbf{X}^{(k+1)})$  are non-decreasing functions due to the additive structure of  $\mathcal{F}_{\varepsilon, \delta}(\mathbf{X}^{(k+1)})$  and (4).
3.) First, we argue that  $(\mathbf{X}^{(k)})_{k\geq 1}$  is a bounded sequence: Indeed, if  $\overline{\varepsilon} := \lim_{k\to \infty}\varepsilon_k > 0$  and  $\overline{\delta} := \lim_{k\to \infty}\delta_k > 0$ , we note that
$$
\begin{array}{l} \frac {1}{2} \bar {\varepsilon} ^ {2} \log (e \| \mathbf {X} ^ {(k)} \| ^ {2} / \bar {\varepsilon} ^ {2}) + \frac {1}{2} \bar {\delta} ^ {2} \log (e \max _ {i} \| \mathbf {X} ^ {(k)} \| _ {\infty , 2} / \bar {\delta} ^ {2}) \\ = \frac {1}{2} \bar {\varepsilon} ^ {2} \log (e \sigma_ {1} ^ {2} (\mathbf {X} ^ {(k)}) / \bar {\varepsilon} ^ {2}) + \frac {1}{2} \bar {\delta} ^ {2} \log (e \max _ {i} \| \mathbf {X} _ {i,:} ^ {(k)} \| _ {2} / \bar {\delta} ^ {2}) \\ \leq \frac {1}{2} \varepsilon_ {k} ^ {2} \log (e \sigma_ {1} ^ {2} (\mathbf {X} ^ {(k)}) / \varepsilon_ {k} ^ {2}) + \frac {1}{2} \delta_ {k} ^ {2} \log (e \max _ {i} \| \mathbf {X} _ {i,:} ^ {(k)} \| _ {2} / \delta_ {k} ^ {2}) \\ \leq \mathcal {F} _ {l r, \varepsilon_ {k}} \left(\mathbf {X} ^ {(k)}\right) + \mathcal {F} _ {s p, \delta_ {k}} \left(\mathbf {X} ^ {(k)}\right) = \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} \left(\mathbf {X} ^ {(k)}\right) \leq \mathcal {F} _ {\varepsilon_ {1}, \delta_ {1}} \left(\mathbf {X} ^ {(1)}\right) \\ \leq \frac {1}{2} \min  (d _ {1}, d _ {2}) \sigma_ {1} ^ {2} (\mathbf {X} ^ {(1)}) + \frac {1}{2} d _ {1} \max  _ {i} \| \mathbf {X} _ {i,:} ^ {(1)} \| _ {2} ^ {2} =: C _ {\mathbf {X} ^ {(1)}}, \\ \end{array}
$$
which implies that  $\{\| \mathbf{X}^{(k)}\| \}_{k\geq 1}$  is bounded by a constant that depends on  $C_{\mathbf{X}^{(1)}}$
Furthermore, we note that the optimality condition of (11) (see Lemma B.5) implies that  $\mathbf{X}^{(k + 1)}$  satisfies
$$
\langle W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k + 1)}), \boldsymbol {\Xi} \rangle = 0 \text {f o r a l l} \boldsymbol {\Xi} \in \ker \mathcal {A} \text {a n d} \mathcal {A} (\mathbf {X} ^ {(k + 1)}) = \mathbf {y}.
$$
Choosing  $\Xi = \mathbf{X}^{(k + 1)} - \mathbf{X}^{(k)}$  and using the notation  $W^{(k)} = W_{\mathbf{X}^{(k)},\varepsilon_k,\delta_k}$  we see that
$$
\begin{array}{l} \langle \mathbf {X} ^ {(k + 1)}, W ^ {(k)} (\mathbf {X} ^ {(k + 1)}) \rangle - \langle \mathbf {X} ^ {(k)}, W ^ {(k)} (\mathbf {X} ^ {(k)}) \rangle \\ = \langle \mathbf {X} ^ {(k + 1)}, W ^ {(k)} (\mathbf {X} ^ {(k + 1)}) \rangle - \langle \mathbf {X} ^ {(k)}, W ^ {(k)} (\mathbf {X} ^ {(k)}) \rangle - 2 \langle W ^ {(k)} (\mathbf {X} ^ {(k + 1)}), \mathbf {X} ^ {(k + 1)} - \mathbf {X} ^ {(k)} \rangle \\ = - \left(\langle \mathbf {X} ^ {(k + 1)}, W ^ {(k)} \left(\mathbf {X} ^ {(k + 1)}\right) \rangle - 2 \langle W ^ {(k)} \left(\mathbf {X} ^ {(k)}\right), \mathbf {X} ^ {(k + 1)} \rangle + \langle \mathbf {X} ^ {(k)}, W ^ {(k)} \left(\mathbf {X} ^ {(k)}\right) \rangle\right) \tag {44} \\ = - \left\langle \left(\mathbf {X} ^ {(k + 1)} - \mathbf {X} ^ {(k)}\right), W ^ {(k)} \left(\mathbf {X} ^ {(k + 1)} - \mathbf {X} ^ {(k)}\right) \right\rangle . \\ \end{array}
$$
Due to the definition of  $W^{(k)}$ , we note that its smallest singular value (interpreted as matrix operator) can be lower bounded by
$$
\begin{array}{l} \sigma_ {\min } \left(W ^ {(k)}\right) \geq \sigma_ {\min } \left(W _ {\mathbf {X} ^ {(k)}, \varepsilon_ {k}} ^ {l r}\right) + \sigma_ {\min } \left(\mathbf {W} _ {\mathbf {X} ^ {(k)}, \delta_ {k}} ^ {s p}\right) \geq \delta_ {k} ^ {2} / \max  _ {i} \left\| \mathbf {X} _ {i,:} ^ {(k)} \right\| _ {2} ^ {2} + \varepsilon_ {k} ^ {2} / \sigma_ {1} ^ {2} (\mathbf {X} ^ {(k)}) \\ \geq \overline {{\delta}} ^ {2} / c _ {\mathrm {s p}, \mathbf {X} ^ {(1)}} + \overline {{\varepsilon}} ^ {2} / c _ {\mathrm {l r}, \mathbf {X} ^ {(1)}}, \\ \end{array}
$$
where  $c_{\mathrm{sp},\mathbf{X}^{(1)}}$  and  $c_{\mathrm{lr},\mathbf{X}^{(1)}}$  are constants that satisfy  $c_{\mathrm{sp},\mathbf{X}^{(1)}} \leq \overline{\delta}^2 \exp(C_{\mathbf{X}^{(1)}} / \overline{\delta}^2 - 1)$  and  $c_{\mathrm{lr},\mathbf{X}^{(1)}} \leq \overline{\varepsilon}^2 \exp(C_{\mathbf{X}^{(1)}} / \overline{\varepsilon}^2 - 1)$ .
Combining this with (44), the monotonicity according to Statement 2 of Theorem 2.6, and (41), it follows that
$$
\begin{array}{l} \mathcal {F} _ {\varepsilon_ {k}, \delta_ {k}} (\mathbf {X} ^ {(k)}) - \mathcal {F} _ {\varepsilon_ {k + 1}, \delta_ {k + 1}} (\mathbf {X} ^ {(k + 1)}) \geq \frac {1}{2} \langle (\mathbf {X} ^ {(k)} - \mathbf {X} ^ {(k + 1)}), W ^ {(k)} (\mathbf {X} ^ {(k)} - \mathbf {X} ^ {(k + 1)}) \rangle \\ \geq \frac {1}{2} \left(\bar {\delta} ^ {2} / c _ {\mathrm {s p}, \mathbf {X} ^ {(1)}} + \bar {\varepsilon} ^ {2} / c _ {\mathrm {l r}, \mathbf {X} ^ {(1)}}\right) \| \mathbf {X} ^ {(k + 1)} - \mathbf {X} ^ {(k)} \| _ {F} ^ {2}. \\ \end{array}
$$
Summing over all  $k$ , this implies that  $\lim_{k\to \infty}\| \mathbf{X}^{(k + 1)} - \mathbf{X}^{(k)}\| _F = 0$
Since  $(\mathbf{X}^{(k)})_{k\geq 1}$  is bounded, each subsequence of  $(\mathbf{X}^{(k)})_{k\geq 1}$  has a convergent subsequence. Let  $(\mathbf{X}^{(k_{\ell})})_{\ell \geq 1}$  be such a sequence with  $\lim_{\ell \to \infty}\mathbf{X}^{(k_{\ell})} = \bar{\mathbf{X}}$  , i.e.,  $\bar{\mathbf{X}}$  is an accumulation point of the sequence. As the weight operator  $W^{(k_{\ell})}$  depends continuously on  $\mathbf{X}^{(k_{\ell})}$  , there exists a weight operator  $\bar{W}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  such that  $\bar{W} = \lim_{\ell \to \infty}W^{(k_{\ell})}$
Since  $\lim_{k\to \infty}\| \mathbf{X}^{(k + 1)} - \mathbf{X}^{(k)}\| _F = 0$ , it also holds that  $\mathbf{X}^{(k_\ell +1)}\rightarrow \bar{\mathbf{X}}$  and therefore
$$
\langle \nabla \mathcal {F} _ {\bar {\varepsilon}, \bar {\delta}} (\bar {\mathbf {X}}), \boldsymbol {\Xi} \rangle = \langle \bar {W} (\bar {\mathbf {X}}), \boldsymbol {\Xi} \rangle = \lim _ {\ell \rightarrow \infty} \langle W ^ {(k _ {\ell})} (\mathbf {X} ^ {(k _ {\ell} + 1)}), \boldsymbol {\Xi} \rangle = 0
$$
for all  $\Xi \in \ker \mathcal{A}$ . The statement is shown as this is equivalent to  $\bar{\mathbf{X}}$  being a stationary point of  $\mathcal{F}_{\overline{\varepsilon},\overline{\delta}}(\cdot)$  subject to the linear constraint  $\{\mathbf{Z} \in \mathbb{R}^{n_1 \times n_2} : \mathcal{A}(\mathbf{Z}) = \mathbf{y}\}$ .
C Technical addendum
C.1 Auxiliary Results
In the proof of Theorem 2.6, we use the following result about the calculus of spectral functions.
Lemma C.1 ([63],[35, Proposition 7.4]). Let  $F: \mathbb{R}^{d_1 \times d_2} \to \mathbb{R}$  be a spectral function  $F = f \circ \sigma$  with an associated function  $f: \mathbb{R}^d \to \mathbb{R}$  that is absolutely permutation symmetric. Then,  $F$  is differentiable at  $\mathbf{X} \in \mathbb{R}^{d_1 \times d_2}$  if and only if  $f$  is differentiable at  $\sigma(\mathbf{X}) \in \mathbb{R}^d$ .
In this case, the gradient  $\nabla F$  of  $F$  at  $\mathbf{X}$  is given by
$$
\nabla F (\mathbf {X}) = \mathbf {U} \operatorname {d i a g} \left(\nabla f (\sigma (\mathbf {X})) \mathbf {V} ^ {*} \right.
$$
if  $\mathbf{X} = \mathbf{U}\mathrm{diag}\bigl (\sigma (\mathbf{X})\bigr)\mathbf{V}^*$  for unitary matrices  $\mathbf{U}\in \mathbb{R}^{d_1\times d_1}$  and  $\mathbf{V}\in \mathbb{R}^{d_2\times d_2}$ .<sup>6</sup>
Here, for  $\mathbf{v} \in \mathbb{R}^{\min(d_1, d_2)}$ ,  $\mathrm{diag}(\mathbf{v}) \in \mathbb{R}^{d_1 \times d_2}$  refers to the matrix with diagonal elements  $\mathbf{v}_i$  on its main diagonal and zeros elsewhere.
C.2 Proof of Lemma B.1
The projection operators for  $\mathcal{M}_r^{n_1,n_2}$ ,  $\mathcal{N}_s^{n_1,n_2}$ , and  $T_{\mathbf{U},\mathbf{V}}$  are well-known, see e.g. [8]. To see the final statement assume that  $\mathbf{U}$  has row-support  $S$  and note that  $\mathbb{P}_{\mathbf{U},\mathbf{V},S}$  is idempotent, i.e.,
$$
\begin{array}{l} \mathbb {P} _ {\mathbf {U}, \mathbf {V}, S} \mathbb {P} _ {\mathbf {U}, \mathbf {V}, S} \mathbf {Z} \\ = \mathbf {U} \mathbf {U} ^ {*} (\mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*}) + \mathbb {P} _ {S} (\mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*}) \mathbf {V} \mathbf {V} ^ {*} \\ - \mathbf {U} \mathbf {U} ^ {*} (\mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*}) \mathbf {V} \mathbf {V} ^ {*} \\ = \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbf {U} \mathbf {U} ^ {*} \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} + \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} \\ - \mathbf {U U} ^ {*} \mathbf {Z V V} ^ {*} - \mathbf {U U} ^ {*} \mathbb {P} _ {S} \mathbf {Z V V} ^ {*} + \mathbf {U U} ^ {*} \mathbf {Z V V} ^ {*} \\ = \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} + \mathbb {P} _ {S} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} - \mathbf {U} \mathbf {U} ^ {*} \mathbf {Z} \mathbf {V} \mathbf {V} ^ {*} = \mathbb {P} _ {\mathbf {U}, \mathbf {V}, S} \mathbf {Z}. \\ \end{array}
$$
One can easily check that  $\mathbb{P}_{\mathbf{U},\mathbf{V},S}$  acts as identity when applied to matrices in  $T_{\mathbf{U},\mathbf{V},S}$  and that  $\mathbb{P}_{\mathbf{U},\mathbf{V},S} = \mathbb{P}_{\mathbf{U},\mathbf{V},S}^{*}$  since  $\langle \mathbf{Z}',\mathbb{P}_{\mathbf{U},\mathbf{V},S}\mathbf{Z}\rangle_F = \langle \mathbb{P}_{\mathbf{U},\mathbf{V},S}\mathbf{Z}',\mathbf{Z}\rangle_F$  for any  $\mathbf{Z},\mathbf{Z}'$ . This proves the claim.
C.3 Proof of Lemma B.3
Let  $\Xi \in \ker (\mathcal{A})$ . Note that
$$
0 = \| \mathcal {A} (\boldsymbol {\Xi}) \| _ {2} = \left\| \mathcal {A} \left(\mathbb {P} _ {T _ {\mathbf {U}}, \mathbf {v}, s} (\boldsymbol {\Xi}) + \mathbb {P} _ {T _ {\mathbf {U}}, \mathbf {v}, s} ^ {\perp} (\boldsymbol {\Xi})\right) \right\| _ {2} \geq \left\| \mathcal {A} \left(\mathbb {P} _ {T _ {\mathbf {U}}, \mathbf {v}, s} (\boldsymbol {\Xi})\right) \right\| _ {2} - \left\| \mathcal {A} \left(\mathbb {P} _ {T _ {\mathbf {U}}, \mathbf {v}, s} ^ {\perp} (\boldsymbol {\Xi})\right) \right\| _ {2}
$$
By the RIP we hence get that
$$
\begin{array}{l} \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} (\boldsymbol {\Xi}) \right\| _ {F} ^ {2} \leq \frac {1}{1 - \delta} \left\| \mathcal {A} (\mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} (\boldsymbol {\Xi})) \right\| _ {2} ^ {2} \leq \frac {1}{1 - \delta} \left\| \mathcal {A} (\mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} ^ {\perp} (\boldsymbol {\Xi})) \right\| _ {2} ^ {2} \\ \leq \frac {\left\| \mathcal {A} \right\| _ {2 \rightarrow 2} ^ {2}}{(1 - \delta)} \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} ^ {\perp} (\boldsymbol {\Xi}) \right\| _ {F} ^ {2}. \\ \end{array}
$$
$$
\| \Xi \| _ {F} ^ {2} = \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} (\Xi) \right\| _ {F} ^ {2} + \left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} ^ {\perp} (\Xi) \right\| _ {F} ^ {2} \leq \left(1 + \frac {\| \mathcal {A} \| _ {2 \rightarrow 2} ^ {2}}{(1 - \delta)}\right)\left\| \mathbb {P} _ {T _ {\mathbf {U}, \mathbf {V}, S}} ^ {\perp} (\Xi) \right\| _ {F} ^ {2}.
$$
C.4 Proof of Lemma B.6
In the proof of Lemma B.6 we will use the following fact.
Lemma C.2. Let  $W_{\mathbf{X}^{(k)}, \varepsilon_k}^{lr}$  be the weight operator defined in (9), which is based on the matrices  $\mathbf{U} \in \mathbb{R}^{n_1 \times r_k}$  and  $\mathbf{V} \in \mathbb{R}^{n_2 \times r_k}$  of leading  $r_k$  left and right singular vectors of  $\mathbf{X}^{(k)}$ . If  $\mathbf{M} \in T_{\mathbf{U},\mathbf{V}}$ , then  $W_{\mathbf{X}^{(k)}, \varepsilon_k}^{lr}(\mathbf{M}) \in T_{\mathbf{U},\mathbf{V}}$ . If  $\mathbf{M} \in T_{\mathbf{U},\mathbf{V}}^{\perp}$ , then  $W_{\mathbf{X}^{(k)}, \varepsilon_k}^{lr}(\mathbf{M}) \in T_{\mathbf{U},\mathbf{V}}^{\perp}$ .
Proof: If  $\mathbf{M} \in T_{\mathbf{U},\mathbf{V}}$ , there exist  $\mathbf{M}_1 \in \mathbb{R}^{r_k \times r_k}$ ,  $\mathbf{M}_2 \in \mathbb{R}^{r_k \times (n_2 - r_k)}$ ,  $\mathbf{M}_3 \in \mathbb{R}^{(n_1 - r_k) \times r_k}$  such that
$$
\mathbf {M} = \left[ \begin{array}{c c} \mathbf {U} & \mathbf {U} _ {\perp} \end{array} \right] \left[ \begin{array}{c c} \mathbf {M} _ {1} & \mathbf {M} _ {2} \\ \mathbf {M} _ {3} & 0 \end{array} \right] \left[ \begin{array}{c} \mathbf {V} ^ {*} \\ \mathbf {V} _ {\perp} ^ {*} \end{array} \right],
$$
e.g., see [90, Proposition 2.1]. We thus observe that the weight operator  $W_{\mathbf{X}^{(k)},\varepsilon_k}^{lr}:\mathbb{R}^{n_1\times n_2}\to \mathbb{R}^{n_1\times n_2}$  from (18) satisfies