SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \tau_ {1} d _ {\mathbf {K L}} \left(\left(\pi_ {c}\right) _ {1}, \alpha\right) + \tau_ {2} d _ {\mathbf {K L}} \left(\left(\pi_ {c}\right) _ {2}, \beta\right) \\ \mathrm {s . t .} d _ {\mathbf {K L}} (\mu , \alpha) \leq \rho_ {1}, d _ {\mathbf {K L}} (\nu , \beta) \leq \rho_ {2} \\ = \operatorname {G W} (\mu_ {c}, \nu_ {c}) + \inf  _ {\alpha \in \mathcal {P} (X), d _ {\mathbf {K L}} (\mu , \alpha) \leq \rho_ {1}} d _ {\mathbf {K L}} (\mu_ {c}, \alpha) + \inf  _ {\beta \in \mathcal {P} (Y), d _ {\mathbf {K L}} (\nu , \beta) \leq \rho_ {2}} d _ {\mathbf {K L}} (\nu_ {c}, \beta). \\ \end{array}
$$
To establish an upper bound for  $\mathrm{GW}_{\rho_1,\rho_2}^{\mathrm{rob}}(\mu ,\nu)$ , let us begin by addressing the following problem:
$$
\inf  _ {\alpha \in \mathcal {P} (X), d _ {\mathbf {K L}} (\mu , \alpha) \leq \rho_ {1}} d _ {\mathbf {K L}} \left(\mu_ {c}, \alpha\right) \tag {8}
$$
We consider the distribution of the form  $(1 - \gamma)\mu +\gamma \mu_{c}$ , for  $\gamma \in [0,1]$ . Then we prove that if  $\gamma \leq \min \left(\frac{\rho_1}{\epsilon_1d_{\mathbf{KL}}(\mu_a,\mu_c)},1\right)$ , then  $(1 - \gamma)\mu +\gamma \mu_{c}$  is a feasible solution for problem (8).
By the joint convexity of KL divergence, we have
$$
\begin{array}{l} d _ {\mathbf {K L}} (\mu , (1 - \gamma) \mu + \gamma \mu_ {c}) \leq \gamma d _ {\mathbf {K L}} (\mu , \mu_ {c}) \\ = \gamma d _ {\mathbf {K L}} \left(\left(1 - \epsilon_ {1}\right) \mu_ {c} + \epsilon_ {1} \mu_ {a}, \mu_ {c}\right) \\ \leq \gamma \epsilon_ {1} d _ {\mathbf {K L}} \left(\mu_ {a}, \mu_ {c}\right) \\ \leq \rho_ {1}. \\ \end{array}
$$
$$
\begin{array}{l} d _ {\mathbf {K L}} \left(\mu_ {c}, (1 - \gamma) \mu + \gamma \mu_ {c}\right) \leq (1 - \gamma) d _ {\mathbf {K L}} \left(\mu_ {c}, \mu\right) \\ = (1 - \gamma) d _ {\mathbf {K L}} \left(\mu_ {c}, (1 - \epsilon_ {1}) \mu_ {c} + \epsilon_ {1} \mu_ {a}\right) \\ \leq (1 - \gamma) \epsilon_ {1} d _ {\mathbf {K L}} \left(\mu_ {c}, \mu_ {a}\right) \\ \end{array}
$$
The largest value  $\gamma$  can take is  $\frac{\rho_1}{\epsilon_1 d_{\mathbf{KL}}(\mu_a, \mu_c)}$ . This gives
$$
\inf_{\alpha \in \mathcal{P}(X),  d_{\mathbf{KL}}(\mu ,\alpha)\leq \rho_{1}}d_{\mathbf{KL}}(\mu_{c},\alpha)\leq \max \left(0,1 - \frac{\rho_{1}}{\epsilon_{1}d_{\mathbf{KL}}(\mu_{a},\mu_{c})}\right)\epsilon_{1}d_{\mathbf{KL}}(\mu_{c},\mu_{a}).
$$
Similarly, we can prove that
$$
\inf_{\beta \in \mathcal{P}(Y),  d_{\mathbf{KL}}(\nu ,\beta)\leq \rho_{2}}d_{\mathbf{KL}}(\nu_{c},\beta)\leq \max \left(0,1 - \frac{\rho_{2}}{\epsilon_{2}d_{\mathbf{KL}}(\nu_{a},\nu_{c})}\right)\epsilon_{2}d_{\mathbf{KL}}(\nu_{c},\nu_{a}).
$$
B.2 Proof and Discussion of Remark 2.4
Utilizing the same notations as in the proof of Theorem 2.3, and considering that  $\pi_c$  is a feasible solution to the UGW problem, substituting it directly leads to the desired result.
Furthermore, referring to [40, Theorem 1] and adapting the notations to our framework, we have the following property: specifically, when considering only  $\mu$  is corrupted with outliers, characterized by  $\mu = (1 - \epsilon_1)\mu_c + \epsilon_1\mu_a$ , while  $\nu = \nu_{c}$ . If we let  $\delta = 2(\tau_{1} + \tau_{2})\epsilon_{1}$ , and  $K = M + \frac{1}{M}\mathrm{UGW}(\mu_c,\nu) + \delta$ , where  $M$  represents the transported mass between clean data and  $\Delta_{\infty}$  signifies the maximal deviation between the contaminated source and the target, then the following inequality holds:
$$
\mathrm {U G W} (\mu , \nu) \leq (1 - \epsilon_ {1}) \mathrm {U G W} (\mu_ {c}, \nu) + \delta M \left[ 1 - \exp \left(- \frac {\Delta_ {\infty} (1 + M) + K}{\delta M}\right) \right].
$$
Given that  $\mathrm{UGW}(\mu_c,\nu)\leq \mathrm{GW}(\mu_c,\nu)$ , we can derive the following relationship:
$$
\operatorname {U G W} (\mu , \nu) \leq (1 - \epsilon_ {1}) \mathrm {G W} (\mu_ {c}, \nu) + \delta M \left[ 1 - \exp \left(- \frac {\Delta_ {\infty} (1 + M) + M + \frac {1}{M} \mathrm {G W} (\mu_ {c} , \nu) + \delta}{\delta M}\right) \right]. \tag {9}
$$
Comparing this result to Remark 2.4, (9) also incorporates  $\mathrm{GW}(\mu_c,\nu)$  in the exponential term and  $M$ , the transported mass between clean data. These variables may be influenced by the marginal parameters  $\tau_{1}$  and  $\tau_{2}$ . As a result, finding an optimal choice for  $\tau_{1}$  and  $\tau_{2}$  to establish a tight upper bound for UGW using the GW distance between clean samples, as presented in (9), proves challenging. As outlined in Remark 2.4, it is worth noting that setting  $\tau_{1} = \tau_{2} = 0$  represents the sole means to attain a tight upper bound. Nevertheless, in real-world applications, this approach is impractical, as it could lead to over-relaxation and compromise the performance.
C Proof Details of Bregman Proximal Alternating Linearized Minimization Method for Robust GW
Given a vector  $\mathbf{x}$ , we use  $\| x \|_2$  to denote its  $\ell_2$  norm. We use  $\| X \|_F$  to denote the Frobenius norm of matrix  $X$ . For a convex set  $C$  and a point  $x$ , we define the distance between  $C$  and  $x$  as
$$
\operatorname {d i s t} (x, C) = \min  _ {y \in C} \| x - y \| _ {2}.
$$
Proof. Problem (7) can be written as
$$
\min  _ {\alpha \in \Delta^ {n}} \sum_ {i = 1} ^ {n} \left((\pi_ {1} ^ {k + 1}) _ {i} \log \left(\frac {(\pi_ {1} ^ {k + 1}) _ {i}}{\alpha_ {i}}\right) - (\pi_ {1} ^ {k + 1}) _ {i} + \alpha_ {i}\right) + \frac {1}{c _ {k}} \sum_ {i = 1} ^ {k} \alpha_ {i} ^ {k} \log \left(\frac {\alpha_ {i} ^ {k}}{\alpha_ {i}}\right) + w \left(\sum_ {i = 1} ^ {n} \mu_ {i} \log \left(\frac {\mu_ {i}}{\alpha_ {i}}\right) - \rho_ {1}\right).
$$
We first consider a relaxed problem of the problem above:
$$
\min  _ {\alpha^ {T} \mathbf {1} _ {n} = 1} \sum_ {i = 1} ^ {n} \left((\pi_ {1} ^ {k + 1}) _ {i} \log \left(\frac {(\pi_ {1} ^ {k + 1}) _ {i}}{\alpha_ {i}}\right) - (\pi_ {1} ^ {k + 1}) _ {i} + \alpha_ {i}\right) + \frac {1}{c _ {k}} \sum_ {i = 1} ^ {n} \alpha_ {i} ^ {k} \log \left(\frac {\alpha_ {i} ^ {k}}{\alpha_ {i}}\right) + w \left(\sum_ {i = 1} ^ {n} \mu_ {i} \log \left(\frac {\mu_ {i}}{\alpha_ {i}}\right) - \rho_ {1}\right).
$$
Consider the Lagrangian function of the relaxed problem
$$
\begin{array}{l} L (\alpha , \lambda) = \sum_ {i = 1} ^ {n} \left((\pi_ {1} ^ {k + 1}) _ {i} \log \left(\frac {(\pi_ {1} ^ {k + 1}) _ {i}}{\alpha_ {i}}\right) - (\pi_ {1} ^ {k + 1}) _ {i} + \alpha_ {i}\right) + \frac {1}{c _ {k}} \sum_ {i = 1} ^ {n} \alpha_ {i} ^ {k} \log \left(\frac {\alpha_ {i} ^ {k}}{\alpha_ {i}}\right) + \\ w \left(\sum_ {i = 1} ^ {n} \mu_ {i} \log \left(\frac {\mu_ {i}}{\alpha_ {i}}\right) - \rho_ {1}\right) + \lambda \left(\alpha^ {T} \mathbf {1} _ {n} - 1\right). \\ \end{array}
$$
$$
\frac {\partial L}{\partial \alpha} = \left( \begin{array}{c} - \frac {(\pi_ {1} ^ {k + 1}) _ {1}}{\alpha_ {1}} + 1 \\ - \frac {(\pi_ {1} ^ {k + 1}) _ {2}}{\alpha_ {2}} + 1 \\ \vdots \\ - \frac {(\pi_ {1} ^ {k + 1}) _ {n}}{\alpha_ {n}} + 1 \end{array} \right) + \frac {1}{c _ {k}} \left( \begin{array}{c} - \frac {\alpha_ {1} ^ {k}}{\alpha_ {1}} \\ - \frac {\alpha_ {2} ^ {k}}{\alpha_ {2}} \\ \vdots \\ - \frac {\alpha_ {n} ^ {k}}{\alpha_ {n}} \end{array} \right) + w \left( \begin{array}{c} - \frac {\mu_ {1}}{\alpha_ {1}} \\ - \frac {\mu_ {2}}{\alpha_ {2}} \\ \vdots \\ - \frac {\mu_ {n}}{\alpha_ {n}} \end{array} \right) + \left( \begin{array}{c} \lambda \\ \lambda \\ \vdots \\ \lambda \end{array} \right) = 0.
$$
$$
\alpha_ {i} = \frac {\left(\pi_ {1} ^ {k + 1}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + w \mu_ {i}}{1 + \lambda}.
$$
Since  $\sum_{i=1}^{n} \alpha_{i} = 1$ ,  $\sum_{i=1}^{n} \alpha_{i}^{k} = 1$ , and  $\sum_{i=1}^{n} \mu_{i} = 1$ , then  $1 + \lambda = \sum_{ij} \pi_{ij}^{k+1} + \frac{1}{c_{k}} + w$ . Thus, the optimal solution to the relaxed problem is
$$
\hat {\alpha} (w) = \frac {\pi^ {k + 1} \mathbf {1} _ {m} + \frac {1}{c _ {k}} \alpha^ {k} + w \mu}{\sum_ {i , j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w}.
$$
We can see that  $\hat{\alpha} (w)\geq 0$ . Hence,  $\hat{\alpha} (w)$  is also the optimal solution to problem (7).
Since if  $w$  satisfies (i) or (ii),  $(\hat{\alpha}(w), w)$  is a solution to KKT conditions of problem (5), therefore,  $\hat{\alpha}(w)$  is an optimal solution to problem (5). Next, we prove that  $p$  is differentiable when  $h$  is relative entropy. Problem (7) admits the closed-form solution
$$
\hat {\alpha} (w) = \frac {\pi^ {k + 1} \mathbf {1} _ {m} + \frac {1}{c _ {k}} \alpha^ {k} + w \mu}{\sum_ {i , j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w}. \tag {10}
$$
By substituting (10) into  $p, p(w)$  can be written as
$$
p (w) = \sum_ {i = 1} ^ {n} \mu_ {i} \log \left(\frac {\mu_ {i} \left(\sum_ {i , j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w\right)}{\left(\pi^ {k + 1} \mathbf {1} _ {m}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + w \mu_ {i}}\right) - \rho_ {1}.
$$
Thus,  $p$  is twice differentiable. The first-order derivative and second-order of  $p$  are
$$
p ^ {\prime} (w) = \sum_ {i = 1} ^ {n} \mu_ {i} \frac {\left(\left(\pi^ {k + 1} \mathbf {1} _ {m}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + \mu_ {i} w\right) - \mu_ {i} \left(\sum_ {i j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w\right)}{\left(\sum_ {i j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w\right) \left(\left(\pi^ {k + 1} \mathbf {1} _ {m}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + \mu_ {i} w\right)},
$$
$$
p ^ {\prime \prime} (w) = - \sum_ {i = 1} ^ {n} \mu_ {i} \frac {\left(\left(\pi^ {k + 1} \mathbf {1} _ {m}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + \mu_ {i} w\right) ^ {2} - \mu_ {i} ^ {2} \left(\sum_ {i j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w\right) ^ {2}}{\left(\sum_ {i j} \pi_ {i j} ^ {k + 1} + \frac {1}{c _ {k}} + w\right) ^ {2} \left(\left(\pi^ {k + 1} \mathbf {1} _ {m}\right) _ {i} + \frac {1}{c _ {k}} \alpha_ {i} ^ {k} + \mu_ {i} w\right) ^ {2}}.
$$
Then we prove that  $p'(w) \leq 0$  and  $p''(w) \geq 0$  for  $w \geq 0$ , so  $p$  is monotonically non-increasing and convex on  $\mathbb{R}_+$ . Let  $s_i = (\pi^{k+1} \mathbf{1}_m)_i + \frac{1}{c_k} \alpha_i^k + \mu_i w$  and  $s = (\pi^{k+1} \mathbf{1}_m)_i + \frac{1}{c_k} \alpha_i^k + \mu_i w$ . Note that  $s = \sum_{i=1}^n s_i$ . Then  $p'$  and  $p''$  can be written as
$$
p ^ {\prime} (w) = \sum_ {i = 1} ^ {n} \mu_ {i} \frac {s _ {i} - \mu_ {i} s}{s _ {i} \cdot s} = \frac {1}{s} \left(1 - \sum_ {i = 1} ^ {n} \mu_ {i} \frac {\mu_ {i} s}{s _ {i}}\right),
$$
$$
p ^ {\prime \prime} (w) = - \sum_ {i = 1} ^ {n} \mu_ {i} \frac {s _ {i} ^ {2} - \mu_ {i} ^ {2} s ^ {2}}{s _ {i} ^ {2} \cdot s ^ {2}} = - \frac {1}{s ^ {2}} \left(1 - \sum_ {i = 1} ^ {n} \mu_ {i} \frac {\mu_ {i} ^ {2} s ^ {2}}{s _ {i} ^ {2}}\right).
$$
Therefore, it is equivalent to show  $\sum_{i=1}^{n} \mu_i \frac{\mu_i s}{s_i} \geq 1$  and  $\sum_{i=1}^{n} \mu_i \frac{\mu_i^2 s^2}{s_i^2} \geq 1$ .
Recall that  $\frac{1}{x}$  and  $\frac{1}{x^2}$  are convex on  $\mathbb{R}_{++}$ , then
$$
\sum_ {i = 1} ^ {n} \mu_ {i} \frac {\mu_ {i} s}{s _ {i}} = \sum_ {i = 1} ^ {n} \frac {1}{\mu_ {i} \frac {s _ {i}}{\mu_ {i} s}} \geq \frac {1}{\sum_ {i = 1} ^ {n} \mu_ {i} \frac {s _ {i}}{\mu_ {i} s}} = 1,
$$
$$
\sum_ {i = 1} ^ {n} \mu_ {i} \frac {\mu_ {i} ^ {2} s ^ {2}}{s _ {i} ^ {2}} = \sum_ {i = 1} ^ {n} \mu_ {i} \frac {1}{\left(\frac {s _ {i}}{\mu_ {i} s}\right) ^ {2}} \geq \frac {1}{\left(\sum_ {i = 1} ^ {n} \mu_ {i} \frac {s _ {i}}{\mu_ {i} s}\right) ^ {2}} = 1.
$$
C.2 Proof of Proposition 3.2
Proof. First, prove that  $p$  only has one root on  $I$ . Since  $p$  is continuous on  $I$  and there exists  $\tilde{x}, \bar{x} \in I$  such that  $p(\tilde{x}) > 0$  and  $p(\bar{x}) < 0$ ,  $p$  contains at least one root on  $[\tilde{x}, \bar{x}]$ . Since  $p$  is non-increasing,  $p$  cannot have roots outside  $[\tilde{x}, \bar{x}]$ . Suppose that  $p$  has two different roots  $z_1$  and  $z_2$  on  $[\tilde{x}, \bar{x}]$  and  $z_1 < z_2$ . By convexity of  $p$ , we have
$$
0 = p (z _ {2}) = p \left(\frac {\bar {x} - z _ {2}}{\bar {x} - z _ {1}} z _ {1} + \frac {z _ {2} - z _ {1}}{\bar {x} - z _ {1}} \bar {x}\right) \leq \frac {\bar {x} - z _ {2}}{\bar {x} - z _ {1}} p (z _ {1}) + \frac {z _ {2} - z _ {1}}{\bar {x} - z _ {1}} p (\bar {x}) = \frac {z _ {2} - z _ {1}}{\bar {x} - z _ {1}} p (\bar {x}) <   0.
$$
This is a contradiction. So  $p$  has a unique root on  $I$ .
$p^{\prime}(x)\leq 0$  since  $p$  is non-increasing on  $I$ . Denote the root of  $p$  as  $r$ . Claim that  $p^\prime (x) < 0$  for  $x\in [\tilde{x},r]$ . Otherwise, there exist  $x\in [\tilde{x},r]$  such that  $p^\prime (x) = 0$ , then
$$
0 > p (\bar {x}) \geq p (x) + p ^ {\prime} (x) (\bar {x} - x) = p (x) \geq 0,
$$
which leads to a contradiction. Especially,  $p'(\tilde{x}) < 0$ , and we can set  $\tilde{x}$  as the initial point of Newton's method.
The update of Newton's method is
$$
x _ {k + 1} = x _ {k} - \frac {p \left(x _ {k}\right)}{p ^ {\prime} \left(x _ {k}\right)}.
$$
Therefore,  $x_{k + 1}\geq x_k$  and  $\{x_{k}\}_{k\geq 1}$  is an increasing sequence. Since  $p$  is convex,
$$
p \left(x _ {k + 1}\right) \geq p \left(x _ {k}\right) + p ^ {\prime} \left(x _ {k}\right) \left(x _ {k + 1} - x _ {k}\right) = p \left(x _ {k}\right) - p \left(x _ {k}\right) = 0.
$$
$x_{k} \leq r$  because  $p$  is a monotonically non-increasing function.  $\{x_{k}\}_{k \geq 1}$  is an increasing sequence with an upper bound, so it has a limit  $x^{*}$  and  $\lim_{k \to \infty} (x_{k} - x_{k+1}) = 0$ . Also,  $p'$  is bounded on  $[\tilde{x}, r]$  since it is continuous. Therefore,
$$
p \left(x ^ {*}\right) = \lim  _ {k \rightarrow + \infty} p \left(x _ {k}\right) = \lim  _ {k \rightarrow + \infty} p ^ {\prime} \left(x _ {k}\right)\left(x _ {k} - x _ {k + 1}\right) = 0.
$$
Hence, the sequence generated by Newton's method converges to a root of  $p$ .
Assumption C.1. The critical point set  $\mathcal{X}$  is non-empty.
Before the proof of Theorem 3.3, we first prove that sequence  $\{\pi^k\}_{k\geq 0}$  generated by BPALM lies in a compact set.
Proposition C.2. Sequence  $\{\pi^k\}_{k\geq 0}$  generated by BPALM lies in a compact set.
Proof. We prove that  $\{\pi^k\}_{k\geq 0}$  lies in the compact set  $\mathcal{A} \coloneqq \{\pi \in \mathbb{R}^{n\times m}: 0 \leq \pi_{ij} \leq 1\}$  by mathematical induction. For  $k = 0$ , we can initialize  $\pi^0$  with  $0 \leq \pi_{ij}^0 \leq 1$ . Suppose that  $\pi^k \in \mathcal{A}$  and  $\pi^{k+1} \notin \mathcal{A}$ . Then there exist  $i \in [n]$  and  $j \in [m]$  such that  $\pi_{ij}^{k+1} > 1$ . Recall that  $\pi^{k+1}$  is the optimal solution to the problem
$$
\min  _ {\pi \geq 0} \varphi (\pi) := \left\langle \mathcal {L} (D, \bar {D}) \otimes \pi^ {k}, \pi \right\rangle + \tau_ {1} d _ {\mathbf {K L}} \left(\pi \mathbf {1} _ {m}, \alpha^ {k}\right) + \tau_ {2} d _ {\mathbf {K L}} \left(\pi^ {T} \mathbf {1} _ {n}, \beta^ {k}\right) + \frac {1}{t _ {k}} d _ {\mathbf {K L}} \left(\pi , \pi^ {k}\right), \tag {11}
$$
Observe that function  $\phi(x) \coloneqq x \log \frac{x}{a} - x + a$  is a unimodal function on  $\mathbb{R}_+$  and achieve its minimum at  $x = a$ . Since  $\alpha_i, \beta_j$ , and  $\pi_{ij}^k$  are smaller than or equal to  $1, \alpha_i, \beta_j$ , and  $\pi_{ij}^k$  are strictly smaller than  $\pi_{ij}^{k+1}$ . Let  $\tilde{\pi} \in \mathbb{R}^{n \times m}$ ,
$$
\tilde {\pi} _ {k l} = \left\{ \begin{array}{l} \max  \{\alpha_ {i}, \beta_ {j}, \pi_ {i j} ^ {k} \}, \quad (k, l) = (i, j), \\ \pi_ {k l} ^ {k + 1}, \quad \text {o t h e r w i s e}. \end{array} \right.
$$
Then  $\varphi (\tilde{\pi}) <   \varphi (\pi^k)$  , this contradicts to  $\pi^{k + 1}$  is the optimal solution to problem (11). Thus,  $\pi^{k + 1}\in \mathcal{A}.$
For the proof of Theorem 3.3, we first prove the sufficient decrease property of BPALM, i.e., there exist a constant  $\kappa_{1} > 0$  and an index  $k_{1}\geq 0$  such that for  $k\geq k_{1}$
$$
F \left(\pi^ {k + 1}, \alpha^ {k + 1}, \beta^ {k + 1}\right) - F \left(\pi^ {k}, \alpha^ {k}, \beta^ {k}\right) \leq - \kappa_ {1} \left(\left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + \left\| \alpha^ {k + 1} - \alpha^ {k} \right\| _ {2} ^ {2} + \left\| \beta^ {k + 1} - \beta^ {k} \right\| _ {2} ^ {2}\right).
$$
And then we prove the subsequence convergence result.
Proof. It is worth noting that  $f(\pi)$  is a quadratic function, i.e.,  $f(\pi) = \left\langle \mathcal{L}(D,\bar{D})\otimes \pi ,\pi \right\rangle$ , then  $f(\pi)$  is gradient Lipschitz continuous with the constant  $\max_{i,j}\left(\sum_{k,l}\mathcal{L}(D,\bar{D})_{i,j,k,l}^2\right)^{1 / 2}$ . To simplify the notation, let  $L_{f} = \max_{i,j}\left(\sum_{k,l}\mathcal{L}(D,\bar{D})_{i,j,k,l}^{2}\right)^{1 / 2}$ .
$$
F \left(\pi^ {k + 1}, \alpha^ {k + 1}, \beta^ {k + 1}\right) - F \left(\pi^ {k}, \alpha^ {k}, \beta^ {k}\right)
$$
$$
\leq \left\langle \nabla f (\pi^ {k}), \pi^ {k + 1} - \pi^ {k} \right\rangle + \frac {L _ {f}}{2} \left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + q (\pi^ {k + 1}) + g _ {1} (\pi^ {k + 1}, \alpha^ {k + 1}) + g _ {2} (\pi^ {k + 1}, \beta^ {k + 1}) +
$$
$$
h _ {1} \left(\alpha^ {k + 1}\right) + h _ {2} \left(\beta^ {k + 1}\right) - \left(q \left(\pi^ {k}\right) + g _ {1} \left(\pi^ {k}, \alpha^ {k}\right) + g _ {2} \left(\pi^ {k}, \beta^ {k}\right) + h _ {1} \left(\alpha^ {k}\right) + h _ {2} \left(\beta^ {k}\right)\right)
$$
$$
\stackrel {(\diamond)} {\leq} \left\langle \nabla f (\pi^ {k}), \pi^ {k + 1} - \pi^ {k} \right\rangle + \frac {L _ {f}}{\sigma} d _ {\mathbf {K L}} \left(\pi^ {k + 1}, \pi^ {k}\right) + q \left(\pi^ {k + 1}\right) + g _ {1} \left(\pi^ {k + 1}, \alpha^ {k + 1}\right) + g _ {2} \left(\pi^ {k + 1}, \beta^ {k + 1}\right) +
$$
$$
= \left\langle \nabla f (\pi^ {k}), \pi^ {k + 1} \right\rangle + q (\pi^ {k + 1}) + g _ {1} (\pi^ {k + 1}, \alpha^ {k}) + g _ {2} (\pi^ {k + 1}, \beta^ {k}) + \frac {1}{t _ {k}} d _ {\mathbf {K L}} (\pi^ {k + 1}, \pi^ {k}) -
$$
$$
\left\langle \nabla f (\pi^ {k}), \pi^ {k} \right\rangle - q \left(\pi^ {k}\right) - g _ {1} \left(\pi^ {k}, \alpha^ {k}\right) - g _ {2} \left(\pi^ {k}, \beta^ {k}\right) + g _ {1} \left(\pi^ {k + 1}, \alpha^ {k + 1}\right) + h _ {1} \left(\alpha^ {k + 1}\right) + \frac {1}{c _ {k}} d _ {\mathbf {K L}} \left(\alpha^ {k}, \alpha^ {k + 1}\right) -
$$
$$
g _ {1} \left(\pi^ {k}, \alpha^ {k}\right) - h _ {1} \left(\alpha^ {k}\right) + g _ {2} \left(\pi^ {k + 1}, \beta^ {k + 1}\right) + h _ {2} \left(\beta^ {k + 1}\right) + \frac {1}{r _ {k}} d _ {\mathbf {K L}} \left(\beta^ {k}, \beta^ {k + 1}\right) - g _ {2} \left(\pi^ {k}, \beta^ {k}\right) - h _ {2} \left(\beta^ {k}\right) -
$$
$$
\left(\frac {1}{t _ {k}} - \frac {L _ {f}}{\sigma}\right) d _ {\mathbf {K L}} \left(\pi^ {k + 1}, \pi^ {k}\right) - \frac {1}{c _ {k}} d _ {\mathbf {K L}} \left(\alpha^ {k}, \alpha^ {k + 1}\right) - \frac {1}{r _ {k}} d _ {\mathbf {K L}} \left(\beta^ {k}, \beta^ {k + 1}\right)
$$
$$
\leq - \left(\frac {1}{t _ {k}} - \frac {L _ {f}}{\sigma}\right) d _ {\mathbf {K L}} \left(\pi^ {k + 1}, \pi^ {k}\right) - \frac {1}{c _ {k}} d _ {\mathbf {K L}} \left(\alpha^ {k}, \alpha^ {k + 1}\right) - \frac {1}{r _ {k}} d _ {\mathbf {K L}} \left(\beta^ {k}, \beta^ {k + 1}\right).
$$
$$
\stackrel {(\diamond)} {\leq} - \frac {\sigma}{2} \left(\left(\frac {1}{t _ {k}} - \frac {L _ {f}}{\sigma}\right) \left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + \frac {1}{c _ {k}} \left\| \alpha^ {k + 1} - \alpha^ {k} \right\| _ {2} ^ {2} + \frac {1}{r _ {k}} \left\| \beta^ {k + 1} - \beta^ {k} \right\| _ {2} ^ {2}\right).
$$
$(\diamondsuit)$  is because as  $x \log x$  is  $\sigma$ -strongly convex, we have
$$
d _ {\mathbf {K L}} (\pi^ {k + 1}, \pi^ {k}) \geq \frac {\sigma}{2} \| \pi^ {k + 1} - \pi^ {k} \| _ {F} ^ {2}, \quad d _ {\mathbf {K L}} (\alpha^ {k}, \alpha^ {k + 1}) \geq \frac {\sigma}{2} \| \alpha^ {k + 1} - \alpha^ {k} \| _ {2} ^ {2}, \quad d _ {\mathbf {K L}} (\beta^ {k}, \beta^ {k + 1}) \geq \frac {\sigma}{2} \| \beta^ {k + 1} - \beta^ {k} \| _ {2} ^ {2}.
$$
By letting  $\kappa_{1} = \frac{\sigma}{2}\min \left(\left(\frac{1}{t_{k}} -\frac{L_{f}}{\sigma}\right),\frac{1}{\bar{r}}\right) > 0$  we get
$$
F \left(\pi^ {k + 1}, \alpha^ {k + 1}, \beta^ {k + 1}\right) - F \left(\pi^ {k}, \alpha^ {k}, \beta^ {k}\right) \leq - \kappa_ {1} \left(\left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + \left\| \alpha^ {k + 1} - \alpha^ {k} \right\| _ {2} ^ {2} + \left\| \beta^ {k + 1} - \beta^ {k} \right\| _ {2} ^ {2}\right). \tag {12}
$$
Summing up (12) from  $k = 0$  to  $+\infty$ , we obtain
$$
\kappa_ {1} \sum_ {k = 0} ^ {\infty} \left(\left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + \left\| \alpha^ {k + 1} - \alpha^ {k} \right\| _ {2} ^ {2} + \left\| \beta^ {k + 1} - \beta^ {k} \right\| _ {2} ^ {2}\right) \leq F (\pi^ {0}, \alpha^ {0}, \beta^ {0}) - F (\pi^ {\infty}, \alpha^ {\infty}, \beta^ {\infty}).
$$
As  $F$  is coercive and  $\left\{\left(\pi^k,\alpha^k,\beta^k\right)\right\}$  is a bounded sequence, it follows that the left-hand side is bounded. This implies
$$
\sum_ {k = 0} ^ {\infty} \left(\left\| \pi^ {k + 1} - \pi^ {k} \right\| _ {F} ^ {2} + \left\| \alpha^ {k + 1} - \alpha^ {k} \right\| _ {2} ^ {2} + \left\| \beta^ {k + 1} - \beta^ {k} \right\| _ {2} ^ {2}\right) <   + \infty ,
$$
$$
\lim  _ {k \rightarrow + \infty} \left(\| \pi^ {k + 1} - \pi^ {k} \| _ {F} + \| \alpha^ {k + 1} - \alpha^ {k} \| _ {2} + \| \beta^ {k + 1} - \beta^ {k} \| _ {2}\right) = 0.
$$
Let  $l(x) = \sum_{i} x_{i} \log x_{i}$ . Recall the optimality condition of BPALM, we have
$$
0 \in \nabla f (\pi^ {k + 1}) + \partial q (\pi^ {k + 1}) + \nabla_ {\pi} g _ {1} (\pi^ {k + 1}, \alpha^ {k}) + \nabla_ {\pi} g _ {2} (\pi^ {k + 1}, \beta^ {k}) + \frac {1}{t _ {k}} (\nabla l (\pi^ {k + 1}) - \nabla l (\pi^ {k})) \tag {13}
$$
$$
0 \in \nabla_ {\alpha} g _ {1} (\pi^ {k + 1}, \alpha^ {k + 1}) + \partial h _ {1} (\alpha^ {k + 1}) + \frac {1}{c _ {k}} \nabla^ {2} l (\alpha^ {k + 1}) (\alpha^ {k + 1} - \alpha^ {k}), \tag {14}
$$
$$
0 \in \nabla_ {\beta} g _ {2} (\pi^ {k + 1}, \beta^ {k + 1}) + \partial h _ {2} (\beta^ {k + 1}) + \frac {1}{r _ {k}} \nabla^ {2} l (\beta^ {k + 1}) (\beta^ {k + 1} - \beta^ {k}). \tag {15}
$$
Let  $(\pi^{\infty},\alpha^{\infty},\beta^{\infty})$  be a limit point of the sequence  $\left\{(\pi^{k},\alpha^{k},\beta^{k})\right\}_{k\geq 0}$ . Then, there exists a sequence  $\{n_k\}_{k\geq 0}$  such that  $\{(\pi^{n_k},\alpha^{n_k},\beta^{n_k})\}_{k\geq 0}$  converges to  $(\pi^{\infty},\alpha^{\infty},\overline{\beta}^{\infty})$ . Since we assume that  $h$  is twice continuous differentiable and  $\alpha^k$  and  $\beta^k$  are in a compact set, then  $\nabla^2 l(\alpha^k)$  and  $\nabla^2 l(\beta^k)$  are bounded. Therefore,  $\lim_{k\to \infty}\nabla^{2}l(\alpha^{k + 1})(\alpha^{k + 1} - \alpha^{k}) = 0$  and  $\lim_{k\to \infty}\nabla^2 l(\beta^{k + 1})(\beta^{k + 1} - \beta^k) = 0$ . Replacing the  $k$  by  $n_k$  in (13), (14), and (15), taking limits on both sides as  $k\to \infty$ , we obtain that
$$
0 \in \nabla f (\pi^ {\infty}) + \partial q (\pi^ {\infty}) + \nabla_ {\pi} g _ {1} (\pi^ {\infty}, \alpha^ {\infty}) + \nabla_ {\pi} g _ {2} (\pi^ {\infty}, \beta^ {\infty}),
$$
$$
0 \in \nabla_ {\alpha} g _ {1} (\pi^ {\infty}, \alpha^ {\infty}) + \partial h _ {1} (\alpha^ {\infty}),
$$
$$
0 \in \nabla_ {\beta} g _ {2} \left(\pi^ {\infty}, \beta^ {\infty}\right) + \partial h _ {2} \left(\beta^ {\infty}\right).
$$
Thus  $(\pi^{\infty},\alpha^{\infty},\beta^{\infty})$  belongs to  $\mathcal{X}$
C.4 Discussion of Computational Complexity of PGW, UGW, and RGW