SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\max  _ {\alpha \in \mathbb {R} ^ {n}, \beta \in \mathbb {R} ^ {n}} - \mathcal {L} _ {\eta_ {k}} (\mathbf {x}, \pi^ {k}) \quad \text {i s t h e d u a l f o r m u l a t i o n o f} \quad \min  _ {\pi \in \Pi (r, c)} h (\pi , U). \tag {42}
$$
Therefore, we can write
$$
q (U) = - \min  _ {\pi \in \Pi (r, c)} h (\pi , U). \tag {43}
$$
Since the entropy function is 1-strongly convex with respect to the  $\ell_1$ -norm over the probability simplex, the minimization problem in (43) has a unique solution  $\pi_U^* = \operatorname{argmin}_{\pi \in \Pi(r,c)} h(\pi, U)$ . Using [34, Lemma B.1], we thus have
$$
\operatorname {g r a d} q (U) = \operatorname {P r o j} _ {\mathrm {T} _ {U} U} (- 2 V _ {\pi_ {U} ^ {*}} U). \tag {44}
$$
By (5), (7), (12), and recalling  $[C(U)]_{ij} = \langle M_{ij},UU^{\top}\rangle$ , we have
$$
[ C (U) ] _ {i j} + \eta_ {k} \log \phi_ {\eta_ {k}} \left(\mathbf {x} _ {U} ^ {*}, \pi^ {k}\right) _ {i j} - \eta_ {k} \log \pi_ {i j} ^ {k} = - \left[ \alpha_ {U} ^ {*} \right] _ {i} - \left[ \beta_ {U} ^ {*} \right] _ {j} - \eta_ {k} \log \left(\| \zeta_ {\eta_ {k}} \left(\mathbf {x} _ {U} ^ {*}, \pi^ {k}\right) \| _ {1}\right). \tag {45}
$$
By the optimality of  $\alpha_{U}^{*}$  and  $\beta_{U}^{*}$ , we know that  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}_U^*,\pi^k) = \mathsf{e}_{\eta_k}^2 (\mathbf{x}_U^*,\pi^k) = 0$  and thus  $\phi_{\eta_k}(\mathbf{x}_U^*,\pi^k)\mathbf{1} = r$  and  $\phi_{\eta_k}(\mathbf{x}_U^*,\pi^k)^\top \mathbf{1} = c$ . With (45), (41), and the definition of  $\mathcal{L}_{\eta_k}(\cdot ,\cdot)$  in (11), by some calculations, we have
$$
h (\phi_ {\eta_ {k}} (\mathbf {x} _ {U} ^ {*}, \boldsymbol {\pi} ^ {k}), U) = - r ^ {\mathsf {T}} \alpha_ {U} ^ {*} - c ^ {\mathsf {T}} \beta_ {U} ^ {*} - \eta_ {k} \log (\| \zeta_ {\eta_ {k}} (\mathbf {x} _ {U} ^ {*}, \boldsymbol {\pi} ^ {k}) \| _ {1}) = - \mathcal {L} _ {\eta_ {k}} (\mathbf {x} _ {U} ^ {*}, \boldsymbol {\pi} ^ {k}).
$$
By the dual formulation (42), we have  $-\mathcal{L}_{\eta_k}(\mathbf{x}_U^*,\pi^k)\leq h(\pi_U^*,U)$ . Hence, we know that  $h(\phi_{\eta_k}(\mathbf{x}_U^*,\pi^k),U)\leq h(\pi_U^*,U)$ , which, together with the optimality and uniqueness of  $\pi_U^*$ , yields  $\phi_{\eta_k}(\mathbf{x}_U^*,\pi^k) = \pi_U^*$ . Finally, using (44), we can complete the proof.
We first establish the iteration complexity result of iRBBS, which is based on the following nice properties of Sinkhorn iteration (21). Note that the sufficient decrease property in Lemma B.2 is quite different from that in [29, Lemmas 4.10 & 4.11] since we can establish the non-increasing property of the feasibility violation of  $\pi^{(l)}$  in Lemma B.1.
Lemma B.1. Let  $\{(\alpha^{\ell},\beta^{\ell})\}$  be the sequence generated by (21) and consider  $\{\pi^{(\ell)}\}$  and  $\{\pi^{(\ell +\frac{1}{2})}\}$  in (22). Then we have  $\| \pi^{(0)}\mathbf{1} - r\| _1 + \| (\pi^{(0)})^\top \mathbf{1} - c\| _1\geq \| (\pi^{(\frac{1}{2})})^\top \mathbf{1} - c\| _1\geq \| \pi^{(1)}\mathbf{1} - r\| _1$  and for each  $\ell \geq 1$
$$
\left\| \pi^ {(\ell)} \mathbf {1} - r \right\| _ {1} \geq \left\| \left(\pi^ {(\ell + \frac {1}{2})}\right) ^ {\mathrm {T}} \mathbf {1} - c \right\| _ {1} \geq \left\| \pi^ {(\ell + 1)} \mathbf {1} - r \right\| _ {1}. \tag {46}
$$
Proof. By the optimality of  $\alpha^{(\ell +1)}$  in (20) and the first equation in (22), we have  $\pi^{(\ell +\frac{1}{2})}\mathbf{1} - r = 0$  for  $\ell \geq 0$ . Therefore, we have
$$
\begin{array}{l} \| \pi^ {(\ell + 1)} \mathbf {1} - r \| _ {1} = \| \pi^ {(\ell + \frac {1}{2})} \mathbf {1} - \pi^ {(\ell + 1)} \mathbf {1} \| _ {1} \leq \| \pi^ {(\ell + \frac {1}{2})} - \pi^ {(\ell + 1)} \| _ {1} \\ = \sum_ {i j} \pi_ {i j} ^ {(\ell + \frac {1}{2})} \left| 1 - \exp \left(- \frac {\beta_ {j} ^ {(\ell + 1)} - \beta_ {j} ^ {(\ell)}}{\eta_ {k}}\right) \right| \\ = \sum_ {i j} \frac {\pi_ {i j} ^ {(\ell + \frac {1}{2})}}{[ (\pi^ {(\ell + \frac {1}{2})}) ^ {\mathsf {T}} \mathbf {1} ] _ {j}} \left| [ (\pi^ {(\ell + \frac {1}{2})}) ^ {\mathsf {T}} \mathbf {1} ] _ {j} - c _ {j} \right| = \| (\pi^ {(\ell + \frac {1}{2})}) ^ {\mathsf {T}} \mathbf {1} - c \| _ {1}, \\ \end{array}
$$
where the first inequality uses the Cauchy-Schwarz inequality, the second equality comes from (7), (22), and the definitions of  $\zeta^{(\ell)}$  and  $\zeta^{(\ell +\frac{1}{2})}$  after (21), and the third equality is due to the second equation in (21) and the first equation in (22). This proves  $\| (\pi^{(\ell +\frac{1}{2})})^{\mathsf{T}}\mathbf{1} - c\| _1\geq \| \pi^{(\ell +1)}\mathbf{1} - r\| _1$
for  $\ell \geq 0$ . On the other hand, by the optimality of  $\beta^{(\ell +1)}$  in (20) and the second equation in (22), we have  $(\pi^{(\ell)})^{\mathsf{T}}\mathbf{1} - c = 0$  for  $\ell \geq 1$ . Using a similar argument, we can prove  $\| \pi^{(\ell)}\mathbf{1} - r\| _1\geq$ $\| (\pi^{(\ell +\frac{1}{2})})^{\mathsf{T}}\mathbf{1} - c\|_{1}$  for  $\ell \geq 1$  and  $\| \pi^{(0)}\mathbf{1} - r\| _1 + \| (\pi^{(0)})^{\mathsf{T}}\mathbf{1} - c\| _1\geq \| (\pi^{(\frac{1}{2})})^{\mathsf{T}}\mathbf{1} - c\| _1$ . The proof is completed.
Lemma B.2 (Sufficient decrease of  $\mathcal{L}_{\eta_k}(\cdot ,\cdot)$  in  $(\alpha ,\beta))$  . Let  $\{(\alpha^{\ell},\beta^{\ell})\}$  be the sequence generated by (21) and consider  $\{\pi^{(\ell)}\}$  in (22). Then, for each  $\ell \geq 0$  , we have
$$
\begin{array}{l} \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}\right) \\ \leq \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell)}, \beta^ {(\ell)}, U ^ {t + 1}, \pi^ {k}\right) - \frac {\eta_ {k}}{2} \left(\| \pi^ {(\ell)} \mathbf {1} - r \| _ {1} ^ {2} + \| \pi^ {(\ell + 1)} \mathbf {1} - r \| _ {1} ^ {2}\right). \tag {47} \\ \end{array}
$$
Proof. By [29, Lemma 4.11], for  $\ell \geq 0$ , we have
$$
\mathcal {L} _ {\eta_ {k}} (\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}) \leq \mathcal {L} _ {\eta_ {k}} (\alpha^ {(\ell + 1)}, \beta^ {(\ell)}, U ^ {t + 1}, \pi^ {k}) - \frac {\eta_ {k}}{2} \| (\pi^ {(\ell + \frac {1}{2})}) ^ {\mathsf {T}} \mathbf {1} - c \| _ {1} ^ {2}.
$$
By Lemma B.1, for  $\ell \geq 0$ , we further have
$$
\mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}\right) \leq \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell)}, U ^ {t + 1}, \pi^ {k}\right) - \frac {\eta_ {k}}{2} \| (\pi^ {(\ell + 1)}) \mathbf {1} - r \| _ {1} ^ {2}. \tag {48}
$$
$$
\begin{array}{l} \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell)}, U ^ {t + 1}, \pi^ {k}\right) - \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell)}, \beta^ {(\ell)}, U ^ {t + 1}, \pi^ {k}\right) = \langle r, \alpha^ {(\ell + 1)} - \alpha^ {(\ell)} \rangle \\ = - \eta_ {k} \left\langle r, \log r - \log \left(\pi^ {(\ell)} \mathbf {1}\right) \right\rangle = - \eta_ {k} \mathrm {K L} (r \| \pi^ {(\ell)} \mathbf {1}) \leq - \frac {\eta_ {k}}{2} \| \pi^ {(\ell)} \mathbf {1} - r \| _ {1} ^ {2}, \tag {49} \\ \end{array}
$$
where the first equality uses the first equation in (21), the second equality comes from the second equation in (22), and the last inequality is due to Pinsker's inequality [33]. Here, for  $p, q \in \Delta^n$ , the Kullback-Leibler divergence between  $p$  and  $q$  is  $\mathsf{KL}(p\|q) = \sum_{i} p_i \log(p_i / q_i)$ . Combining (48) and (49), we obtain (47). The proof is completed.
We need to use the following elementary result.
Lemma B.3. Let  $\vartheta_{1}$  and  $\vartheta_{2}$  be two given positive constants. Consider two sequences  $\{a_{\ell}\}, \{b_{\ell}\} \subseteq \mathbb{R}_{+}$  with  $\ell \geq 0$ . If they obey:
$$
a _ {\ell} - a _ {\ell + 1} \geq \vartheta_ {1} \left(b _ {\ell} ^ {2} + b _ {\ell + 1} ^ {2}\right), \tag {50a}
$$
$$
a _ {\ell} \leq \vartheta_ {2} b _ {\ell} \tag {50b}
$$
for all  $\ell \geq 1$ , then we have  $b_{1} \leq \vartheta_{2} / \vartheta_{1}$  and
$$
\min  _ {1 \leq i \leq \ell + 1} b _ {i} \leq \frac {\vartheta_ {2}}{\vartheta_ {1}} \cdot \frac {1}{\ell + \sqrt {2} - 1}, \quad \forall   \ell \geq 1.
$$
Proof. Applying (50) on  $\ell = 1$ , it is easy to see that  $\vartheta_2 b_1 \geq a_1 \geq \vartheta_1 b_1^2$ . Therefore, we have  $b_1 \leq \vartheta_2 / \vartheta_1$  and  $a_1 \leq \vartheta_2^2 / \vartheta_1$ . For  $\ell \geq 1$ , by (50a), we have
$$
\frac {1}{a _ {\ell + 1}} - \frac {1}{a _ {\ell}} = \frac {a _ {\ell} - a _ {\ell + 1}}{a _ {\ell} a _ {\ell + 1}} \geq \frac {\vartheta_ {1} (b _ {\ell} ^ {2} + b _ {\ell + 1} ^ {2})}{\vartheta_ {2} ^ {2} b _ {\ell} b _ {\ell + 1}} \geq \frac {2 \vartheta_ {1}}{\vartheta_ {2} ^ {2}},
$$
which, together with  $a_1 \leq \vartheta_2^2 / \vartheta_1$ , implies
$$
a _ {\ell + 1} \leq \frac {\vartheta_ {2} ^ {2}}{\vartheta_ {1}} \cdot \frac {1}{2 (\ell + 1) - 1}, \quad \forall \ell \geq 0.
$$
Let  $\lfloor \ell /2\rfloor$  be the largest integer smaller than  $\ell /2$ . For  $\ell \geq 1$ , summing (50a) over  $i = \lfloor \ell /2\rfloor +1,\ldots ,\ell$ , we have
$$
2 (\ell - \lfloor \ell / 2 \rfloor) \min _ {1 \leq i \leq \ell + 1} b _ {i} ^ {2} \leq \sum_ {i = \lfloor \ell / 2 \rfloor + 1} ^ {\ell} (b _ {i} ^ {2} + b _ {i + 1} ^ {2}) \leq \frac {a _ {\lfloor \ell / 2 \rfloor + 1}}{\vartheta_ {1}} \leq \frac {\vartheta_ {2} ^ {2}}{\vartheta_ {1} ^ {2}} \cdot \frac {1}{2 (\lfloor \ell / 2 \rfloor + 1) - 1},
$$
which immediately implies  $\min_{1\leq i\leq l + 1}b_i^2\leq \frac{\vartheta_2^2 / \vartheta_1^2}{\ell(\ell + 1)}$  . With the fact that  $\ell (\ell +1)\geq (\ell +\sqrt{2} -1)^2$  for  $\ell \geq 1$  and  $b_{1}\leq \vartheta_{2} / \vartheta_{1}$  , we thus complete the proof.
With the help of the above lemmas, we can provide a new analysis of the iteration complexity of the classical Sinkhorn iteration based on the decreasing properties developed in Lemma B.1 and B.2. It differs from the proof technique in [19], wherein a switching strategy is adopted to establish the complexity.
Lemma B.4. The total number of Sinkhorn iterations to find a point  $\pi^{(\ell +1)}$  satisfying (23) is at most  $\lceil 2R^{t + 1} / (\eta_k\theta_{t + 1}) + 2 - \sqrt{2}\rceil$ , where
$$
R ^ {t + 1} = \left\| C \left(U ^ {t + 1}\right) \right\| _ {\text {v a r}} + \eta_ {k} \Psi , \tag {51}
$$
where  $\Psi = \| \log \pi^k\|_{\mathrm{var}} + \max \{\| \log r\|_{\mathrm{var}},\| \log c\|_{\mathrm{var}}\}$ . Here,  $\lceil a\rceil$  is the smallest nonnegative integer larger than  $a\in \mathbb{R}^n$ .
Proof. Let  $(\alpha_{U^{t + 1}}^*,\beta_{U^{t + 1}}^*)\in \mathrm{argmin}_{\alpha \in \mathbb{R}^n,\beta \in \mathbb{R}^n}\mathcal{L}_{\eta_k}(\alpha ,\beta ,U^{t + 1},\pi^k)$ . By refining the proof in [19] and [36], we can prove
$$
\max  \left\{\| \alpha^ {(\ell + 1)} \| _ {\text {v a r}}, \| \alpha_ {U ^ {t + 1}} ^ {*} \| _ {\text {v a r}}, \| \beta^ {(\ell + 1)} \| _ {\text {v a r}}, \| \beta_ {U ^ {t + 1}} ^ {*} \| _ {\text {v a r}} \right\} \leq R ^ {t + 1}, \quad \forall \ell \geq 0. \tag {52}
$$
Since  $\nabla_{\alpha}\mathcal{L}_{\eta_k}(\alpha^{(\ell +1)},\beta^{(\ell +1)},U^{t + 1},\pi^k) = r - \pi^{(\ell +1)}\mathbf{1},$
$$
\nabla_ {\beta} \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}\right) = c - \left(\pi^ {(\ell + 1)}\right) ^ {\top} \mathbf {1} = 0,
$$
and  $\mathcal{L}_{\eta_k}(\alpha, \beta, U^{t+1}, \pi^k)$  is jointly convex with respect to  $\alpha$  and  $\beta$ , we have
$$
\mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}\right) - \mathcal {L} _ {\eta_ {k}} \left(\alpha_ {U ^ {t + 1}} ^ {*}, \beta_ {U ^ {t + 1}} ^ {*}, U ^ {t + 1}, \pi^ {k}\right) \leq \langle \pi^ {(\ell + 1)} \mathbf {1} - r, \alpha_ {U ^ {t + 1}} ^ {*} - \alpha^ {(\ell + 1)} \rangle . \tag {53}
$$
Given  $x \in \mathbb{R}^n$ , let  $x_{\mathrm{m}} = \| x\|_{\mathrm{var}} / 2 + x_{\mathrm{min}}$ , it holds that  $\| x - x_{\mathrm{m}}\mathbf{1}\|_{\infty} = \| x\|_{\mathrm{var}} / 2$ . For  $y \in \mathbb{R}^n$  with  $\langle y, \mathbf{1} \rangle = 0$ , we further know that
$$
\langle y, x \rangle = \langle y, x - x _ {\mathrm {m}} \mathbf {1} \rangle \leq \| y \| _ {1} \| x - x _ {\mathrm {m}} \mathbf {1} \| _ {\infty} = \frac {\| x \| _ {\mathrm {v a r}}}{2} \| y \| _ {1}.
$$
Applying this assertion with  $y = \pi^{(\ell +1)}\mathbf{1} - r, x = \alpha_{U^{t + 1}}^*$  or  $x = \alpha^{(\ell +1)}$ , we obtain from  $\| \pi^{(\ell +1)}\| _1 = \| r\| _1 = 1$ , (52), and (53) that
$$
\mathcal {L} _ {\eta_ {k}} \left(\alpha^ {(\ell + 1)}, \beta^ {(\ell + 1)}, U ^ {t + 1}, \pi^ {k}\right) - \mathcal {L} _ {\eta_ {k}} \left(\alpha_ {U ^ {t + 1}} ^ {*}, \beta_ {U ^ {t + 1}} ^ {*}, U ^ {t + 1}, \pi^ {k}\right) \leq R ^ {t + 1} \| \pi^ {(\ell + 1)} \mathbf {1} - r \| _ {1}, \quad \forall \ell \geq 0. \tag {54}
$$
Applying Lemma B.3 with  $a_{\ell} = \mathcal{L}_{\eta_k}(\alpha^{(\ell)},\beta^{(\ell)},U^{t + 1},\pi^k) - \mathcal{L}_{\eta_k}(\alpha^{*}_{U^{t + 1}},\beta^{*}_{U^{t + 1}},U^{t + 1},\pi^k)$  and  $b_{\ell} = \| \pi^{(\ell)}\mathbf{1} - r\| _1$  and using (46), (47), and (54), we have  $\| \pi^{(1)}\mathbf{1} - r\| _1\leq 2R^{t + 1} / \eta_k$  and  $\| \pi^{(\ell +1)}\mathbf{1}-$ $r\| _1\leq 2R^{t + 1} / (\eta_k(\ell +\sqrt{2} -1))$  for all  $\ell \geq 1$ . Letting  $2R^{t + 1} / (\eta_k(\ell +\sqrt{2} -1))\leq \theta_{t + 1}$ , we can complete the proof of Lemma B.4.
We now can establish the convergence result of iRBBS, namely, Algorithm 2.
Theorem B.5. Let  $\{\mathbf{x}^t\}$  be the sequence generated by Algorithm 2. If  $\epsilon_1 = \epsilon_2 = 0$ , we have  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^t,\pi^k)\to 0$  and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^t,\pi^k)\to 0$ . If  $\epsilon_{1} > 0$  and  $\epsilon_{2} > 0$ , then Algorithm 2 stops within at most  $\lceil \Upsilon \max \{\epsilon_1^{-2},\epsilon_2^{-2}\} \rceil$  iterations for any  $\theta_{t + 1}\geq 0$ , where  $\Upsilon$  is a constant defined in (64).
Proof. Given  $\alpha \in \mathbb{R}^n$  and  $\beta \in \mathbb{R}^n$ , by [29, Lemma 4.8] for any  $\iota \in [0,1]$  and  $U_1, U_2 \in \mathcal{U}$ ,
$$
\begin{array}{l} \left\| \nabla_ {U} \mathcal {L} _ {\eta_ {k}} \left(\alpha , \beta , U _ {2}, \pi^ {k}\right) - \nabla_ {U} \mathcal {L} _ {\eta_ {k}} \left(\alpha , \beta , \iota U _ {1} + (1 - \iota) U _ {2}, \pi^ {k}\right) \right\| _ {F} \tag {55} \\ \leq 2 \left(\| C \| _ {\infty} + 2 \| C \| _ {\infty} ^ {2} / \eta\right) \iota \| U _ {1} - U _ {2} \| _ {\mathbf {F}}. \\ \end{array}
$$
From [38, 10], we know that the retraction on the Stiefel manifold has the following nice properties, namely, there exist positive constants  $L_{1}$  and  $L_{2}$  such that
$$
\left\| \operatorname {R e t r} _ {U} (\xi) - U \right\| _ {\mathbb {F}} \leq L _ {1} \| \xi \| _ {\mathbb {F}} \quad \text {a n d} \quad \left\| \operatorname {R e t r} _ {U} (\xi) - (U + \xi) \right\| _ {\mathbb {F}} \leq L _ {2} \| \xi \| _ {\mathbb {F}} ^ {2} \tag {56}
$$
hold for all  $\xi \in \mathrm{T}_U\mathcal{U}$  and  $U\in \mathcal{U}$ . With (55) and (56), following the proof of Lemma 3 in [10], for each  $U\in \mathcal{U}$  and  $\xi \in \mathrm{T}_U\mathcal{U}$ , we have
$$
\mathcal {L} _ {\eta_ {k}} (\alpha , \beta , \operatorname {R e t r} _ {U} (\xi), \pi^ {k}) \leq \mathcal {L} _ {\eta_ {k}} (\alpha , \beta , U, \pi^ {k}) + \left\langle \operatorname {g r a d} _ {U} \mathcal {L} _ {\eta_ {k}} (\mathbf {x}, \pi^ {k}), \xi \right\rangle + \frac {L}{2} \| \xi \| _ {\mathsf {F}} ^ {2}, \tag {57}
$$
$$
L = 2 \left(L _ {1} ^ {2} + L _ {2}\right) \| C \| _ {\infty} + 4 L _ {1} ^ {2} \| C \| _ {\infty} ^ {2} / \eta_ {k}. \tag {58}
$$
Applying (57) with  $\alpha = \alpha^t$ ,  $\beta = \beta^t$ ,  $U = U^t$ , and  $\xi = -\tau_t \operatorname{grad}_U \mathcal{L}_{\eta_k}(\mathbf{x}^t, \pi^k) = -\tau_t \xi^t$ , with (19a), we have
$$
\mathcal {L} _ {\eta_ {k}} \left(\alpha^ {t}, \beta^ {t}, U ^ {t + 1}, \pi^ {k}\right) \leq \mathcal {L} _ {\eta_ {k}} \left(\mathbf {x} ^ {t}, \pi^ {k}\right) - \tau_ {t} \left(1 - \frac {L \tau_ {t}}{2}\right) \| \xi^ {t} \| _ {\mathsf {F}} ^ {2}. \tag {59}
$$
In addition, by the fact that  $\alpha^{t + 1} = \alpha^{(\ell +1)}$ ,  $\beta^{t + 1} = \beta^{(\ell +1)}$ , and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^{t + 1},\pi^k) = \| \pi^{(\ell +1)}\mathbf{1} - r\| _1$  for some  $\ell$  and  $\alpha^{(0)} = \alpha^t,\beta^{(0)} = \beta^t$ , we have from (47) that
$$
\mathcal {L} _ {\eta_ {k}} \left(\mathbf {x} ^ {t + 1}, \pi^ {k}\right) \leq \mathcal {L} _ {\eta_ {k}} \left(\alpha^ {t}, \beta^ {t}, U ^ {t + 1}, \pi^ {k}\right) - \frac {\eta_ {k}}{2} \left(\mathrm {e} _ {\eta_ {k}} ^ {2} \left(\mathbf {x} ^ {t + 1}, \pi^ {k}\right)\right) ^ {2}. \tag {60}
$$
Recalling (24), it follows from (59) and (60) that
$$
E _ {\rho} \left(\mathbf {x} ^ {t + 1}\right) \leq E _ {\rho} \left(\mathbf {x} ^ {t}\right) - \tau_ {t} \left(1 - \frac {L \tau_ {t}}{2}\right) \| \xi^ {t} \| _ {\mathsf {F}} ^ {2} - \left(\frac {\eta_ {k}}{2} - \rho\right) \left(\mathrm {e} _ {\eta_ {k}} ^ {2} \left(\mathbf {x} ^ {t + 1}, \pi^ {k}\right)\right) ^ {2} \tag {61}
$$
holds with  $\rho \in [0,\eta_k / 2)$
Recalling that  $\tau_{t}$  takes the form  $\tau_{t} = \tau_{t}^{(0)}\sigma^{s}$  for some nonnegative integer  $s$ , with (26), we know from (61) that if  $s = \left\lceil \log \frac{2(1 - \delta_1)}{\tau_{\max}L} / \log \sigma \right\rceil$ , there must hold that
$$
E _ {\rho} (\mathbf {x} ^ {t + 1}) \leq E _ {\rho} (\mathbf {x} ^ {t}) - \delta_ {1} \tau_ {t} \| \xi^ {t} \| _ {\mathsf {F}} ^ {2} - \left(\frac {\eta_ {k}}{2} - \rho\right) \bigl (\mathsf {e} _ {\eta_ {k}} ^ {2} (\mathbf {x} ^ {t + 1}, \pi^ {k}) \bigr) ^ {2}.
$$
Using similar arguments as [54, Lemma 1.1], this also shows that the backtracking line search in Algorithm 2 terminates in at most  $\left\lceil \log \frac{2(1 - \delta_1)}{\tau_{\max L}} / \log \sigma \right\rceil$  trials and  $E_t^r \geq E_\rho(\mathbf{x}^t)$ . Moreover, since  $\tau_t^{(0)} \geq \tau_{\min}$  from (26), we further know that  $\tau_t = \tau_t^{(0)}\sigma^s \geq \underline{\tau} := \frac{2\sigma(1 - \delta_1)\tau_{\min}}{\tau_{\max L}}$ . Therefore, we know that the nonmonotone line search condition (25) is satisfied for some  $\tau_t \geq \underline{\tau}$ .
Let  $\mathbf{x}^{*} = (\alpha^{*},\beta^{*},U^{*})$  be the minimizer of problem (11). Using a similar argument as [28, Lemma 4.7], by (34) and  $-\log \pi_{ij}^k\leq \| \log \pi^k\|_\infty$ , we have
$$
\mathcal {L} _ {\eta_ {k}} \left(\mathbf {x} ^ {*}, \pi^ {k}\right) \geq - \| C \| _ {\infty} - \eta_ {k} \| \log \pi^ {k} \| _ {\infty}. \tag {62}
$$
Let  $\omega_{t} = E_{t}^{r} - E_{\rho}(\mathbf{x}^{t})$ . By (62) and using the argument in [44, Theorem 5], we know that such  $\omega_{t} \geq 0$  satisfies
$$
\sum_ {t = 0} ^ {+ \infty} \omega_ {t} \leq \frac {\gamma}{1 - \gamma} (E _ {\eta_ {k} / 2} (\mathbf {x} ^ {0}) + \| C \| _ {\infty} + \eta_ {k} \| \log \pi^ {k} \| _ {\infty}) <   + \infty .
$$
By (24), the fact that (25) holds with  $\tau_t \geq \underline{\tau}$ , and  $\|\xi^t\|_{\mathsf{F}} = \mathsf{e}_{\eta_k}^1(\mathbf{x}^t, \pi^k)$ , we have
$$
\begin{array}{l} \delta_ {1} \underline {{\mathbf {\Pi}}} \left(\mathrm {e} _ {\eta_ {k}} ^ {1} \left(\mathbf {x} ^ {t}, \pi^ {k}\right)\right) ^ {2} + \left(\frac {\eta_ {k}}{2} - \rho\right) \left(\mathrm {e} _ {\eta_ {k}} ^ {2} \left(\mathbf {x} ^ {t + 1}, \pi^ {k}\right)\right) ^ {2} \tag {63} \\ \leq \mathcal {L} _ {\eta_ {k}} (\mathbf {x} ^ {t}, \boldsymbol {\pi} ^ {k}) - \mathcal {L} _ {\eta_ {k}} (\mathbf {x} ^ {t + 1}, \boldsymbol {\pi} ^ {k}) + \rho ((\mathsf {e} _ {\eta_ {k}} ^ {2} (\mathbf {x} ^ {t}, \boldsymbol {\pi} ^ {k})) ^ {2} - (\mathsf {e} _ {\eta_ {k}} ^ {2} (\mathbf {x} ^ {t + 1}, \boldsymbol {\pi} ^ {k})) ^ {2}) + \omega_ {t}. \\ \end{array}
$$
Summing (63) over  $t = 0,\dots ,T$  and adding the term  $(\eta_k / 2 - \rho)(\mathbf{e}_{\eta_k}^2 (\mathbf{x}^0))^2$  on both sides of the obtained equation, and then combining them with  $\mathcal{L}_{\eta_k}(\mathbf{x}^{T + 1},\pi^k)\geq \mathcal{L}_{\eta_k}(\mathbf{x}^*,\pi^k)$  and (62), we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {T} \left(\delta_ {1} \underline {{\tau}} \left(\mathsf {e} _ {\eta_ {k}} ^ {1} \left(\mathbf {x} ^ {t}, \pi^ {k}\right)\right) ^ {2} + \left(\frac {\eta_ {k}}{2} - \rho\right) \left(\mathsf {e} _ {\eta_ {k}} ^ {2} \left(\mathbf {x} ^ {t}, \pi^ {k}\right)\right) ^ {2}\right) \\ \leq \widetilde {\Upsilon} := \frac {1}{1 - \gamma} \left(E _ {\eta_ {k} / 2} (\mathbf {x} ^ {0}) + \| C \| _ {\infty} + \eta_ {k} \| \log \pi^ {k} \| _ {\infty}\right) \\ \end{array}
$$
for any  $T\geq 1$  . This further implies
$$
\sum_ {t = 0} ^ {T} \left(\mathrm {e} _ {\eta_ {k}} ^ {1} \left(\mathbf {x} ^ {t}, \pi^ {k}\right)\right) ^ {2} + \left(\mathrm {e} _ {\eta_ {k}} ^ {2} \left(\mathbf {x} ^ {t}, \pi^ {k}\right)\right) ^ {2} \leq \Upsilon := \frac {\widetilde {\Upsilon}}{\min  \left\{\delta_ {1} \underline {{\tau}}, \eta_ {k} / 2 - \rho \right\}}. \tag {64}
$$
Hence, we have  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^t,\pi^k)\to 0$  and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^t,\pi^k)\to 0$  as  $t\to \infty$
Suppose  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^t,\pi^k)\leq \epsilon_1$  and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^t,\pi^k)\leq \epsilon_2$  are fulfilled for  $t = \overline{T}$  but not fulfilled for all  $t <   \overline{T}$  Then there hold that  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^\overline{T},\pi^k)\leq \epsilon_1,\mathsf{e}_{\eta_k}^2 (\mathbf{x}^\overline{T},\pi^k)\leq \epsilon_2$  , and  $(\mathsf{e}_{\eta_k}^1 (\mathbf{x}^t,\pi^k))^2 +(\mathsf{e}_{\eta_k}^2 (\mathbf{x}^t,\pi^k))^2 >$ $\min \{\epsilon_1^2,\epsilon_2^2\}$  for all  $t <   \overline{T}$  . Setting  $T$  in (64) as  $\overline{T}$  yields  $\overline{T}\min \{\epsilon_1^2,\epsilon_2^2\} \leq \Upsilon$  , which completes the proof.
We are now ready to complete the proof of Theorem 3.2.
Proof of Theorem 3.2. Given  $\epsilon_{1} > 0$  and  $\epsilon_{2} > 0$ , by Theorem B.5, we know that Algorithm 2 will return a point  $\mathbf{x}^T$  with  $T \leq \lceil \Upsilon \max \{\epsilon_1^{-2}, \epsilon_2^{-2}\}\rceil$  satisfying satisfying  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^t,\pi^k) \leq \epsilon_1$  and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^t,\pi^k) \leq \epsilon_2$ . Let  $\hat{\pi}^T \coloneqq \mathrm{Round}(\phi_{\eta_k}(\mathbf{x}^T,\pi^k),\Pi (r,c))$ , where "Round" is the rounding procedure given in [3, Algorithm 2]. By the second equation in (22) and  $\mathbf{x}^T = (\alpha^T,\beta^T,U^T)$  with  $\alpha^{T} = \alpha^{(\ell +1)},\beta^{T} = \beta^{(\ell +1)}$  for some  $\ell$ , we know that  $\| \zeta_{\eta_k}(\mathbf{x}^T,\pi^k)\| _1 = 1$ . By  $\| C(U)\|_{\infty} \leq \| C\|_{\infty}$  for all  $U \in \mathcal{U}$ , (58), (52), (51), we have  $\| \alpha^T\|_{\mathrm{var}} + \| \beta^T\|_{\mathrm{var}} + \| C\|_{\infty} \leq 3\| C\|_{\infty} + 2\eta_k\Psi$ . Therefore, we obtain from (13) that
$$
\begin{array}{l} \left\| \operatorname {P r o j} _ {\mathrm {T} _ {U} U} \left(- 2 V _ {\hat {\pi} ^ {T}} U ^ {T}\right) \right\| _ {\mathbf {F}} \leq \epsilon_ {1} + 2 \| C \| _ {\infty} \epsilon_ {2}, \\ \left\langle \hat {\pi} ^ {T}, Z (\mathbf {x} ^ {T}) \right\rangle \leq (2 \log n + \| \log \pi^ {k} \| _ {\text {v a r}}) \eta_ {k} + (3 \| C \| _ {\infty} + 2 \eta_ {k} \Psi) \epsilon_ {2}. \\ \end{array}
$$
With the choices of  $\eta_{k},\epsilon_{1}$  and  $\epsilon_{2}$  in Theorem 3.2, we can immediately know from (13) that such  $(\mathbf{x}^T, - \varphi (\mathbf{x}^T)_{\mathrm{min}})$  is an  $(\epsilon_1',\epsilon_2')$  -stationary point of problem (6). With the expression of  $\Upsilon$  in (64), Lemma B.4 and Remark 3.5, we can complete the proof. Note that the  $\mathcal{O}(\cdot)$  in this theorem hides the constants related to  $E_{\eta_k / 2}(\mathbf{x}^0)$ $\Psi ,L_1,L_2,\log n$  , and  $\| C\|_{\infty}$
C Experimental Settings
We implemented all methods in MATLAB 2021b and performed all the experiments on a MacBook Pro with a  $2.3\mathrm{GHz}$  8-core Intel Core i9. We follow the ways in [34] to generate the following real dataset.
Datasets C.1. We consider six Shakespeare operas. Each script is preprocessed according to the way in [34] and corresponds to a matrix over  $X \in \mathbb{R}^{300 \times n_X}$ . The values  $n_X$  of H5, H, JC, MV, O, and RJ are 1303, 1481, 910, 1008, 1148, and 1141, respectively. The weight vector  $r$  or  $c$  is taken as  $1_{n_X} / n_X$ . We set  $k = 2$ .