SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Initial points and retraction. We choose  $\alpha^{-1} = 0$ ,  $\beta^{-1} = 0$  for iRBBS and  $\alpha^0 = 0$ ,  $\beta^0 = 0$  for R(A)BCD. As for  $U^0$ , we choose  $U^0 \in \mathrm{argmax}_{U \in \mathcal{U}} \langle V_{\pi^0}, U U^\top \rangle$  with  $\pi^0 \in \Pi(r, c)$  for all methods. Here,  $\pi^0$  is formulated by firstly generating a matrix  $\tilde{\pi}^0$  with each entry randomly drawn from the standard uniform distribution on  $(0, 1)$  and then rounding  $\tilde{\pi}^0 / \| \tilde{\pi}^0 \|_1$  to  $\pi^0$  via [3, Algorithm 2]. The retraction operator in all the above methods is taken as the QR-retraction  $\mathrm{Retr}_U^{\mathrm{qr}}(\xi) = \mathrm{qf}(X + \xi)$ , where  $\mathrm{qf}(X + \xi) \in \mathcal{U}$  satisfying  $X + \xi = \mathrm{qf}(X + \xi) \mathrm{upp}(X + \xi)$  with  $\mathrm{upp}(X + \xi)$  being an upper triangular  $k$ -by- $k$  matrix with positive diagonal elements.
Parameters of iRBBS in Section 4.1. We choose  $\tau^{\mathrm{min}} = 10^{-10} / L$ ,  $\tau^{\mathrm{max}} = 10^{10} / L$ ,  $\gamma = 0.85$ ,  $\tau_0 = 10^{-3}$ ,  $\sigma = 1/2$ ,  $\delta_1 = 10^{-4}$ , and  $\rho = 0.49\eta_k$ .
Parameters of  $R(A)BCD$  in Section 4.1. As stated in Remark 4.1 in [28], to achieve the best performance of  $R(A)BCD$ , one has to spend some efforts in tuning the step sizes. Here, we adopt the step sizes used therein (with a slight modification, marked in italic type, to have better performance for some cases). We choose  $\tau_{\mathrm{RBCD}} = 0.09$  if the instance is X/RJ and  $\tau_{\mathrm{RBCD}} = 0.1$  otherwise;  $\tau_{\mathrm{RABCD}} = 0.0015$  if the instance is X/RJ with  $\mathrm{X} \neq \mathrm{H}$ ,  $\tau_{\mathrm{RABCD}} = 0.001$  if the instance is H/RJ and  $\tau_{\mathrm{RABCD}} = 0.0025$  otherwise. Moreover, we stop  $R(A)BCD$  when  $\mathsf{e}_{\eta_k}^1 (\alpha^t,\beta^{t - 1},U^t)\leq \epsilon_1$  and  $\mathsf{e}_{\eta_k}^2 (\alpha^t,\beta^{t - 1},U^t)\leq \epsilon_2$  or the maximum iteration number reaches 5000.
Parameters of REALM in Section 4.2. Once  $\eta_{k}$  becomes  $\eta_{\mathrm{min}}$ , we set the corresponding  $\epsilon_{k,1} = \epsilon_{1}$  and  $\epsilon_{k,2} = \epsilon_{2}$  and stop the algorithm if  $\mathsf{e}_{\eta_k}^1 (\mathbf{x}^k,\pi^k)\leq \epsilon_1$  and  $\mathsf{e}_{\eta_k}^2 (\mathbf{x}^k,\pi^k)\leq \epsilon_2$ . We choose  $\epsilon_{1,2} = 10^{-1}\max \{\| r\|_{\infty},\| c\|_{\infty}\}$  and  $\epsilon_{1,1} = 2\| C\|_{\infty}\epsilon_{1,2}$ .
(a)  $n = 100$
(b)  $d = 50$
(c)  $n = d$
Figure 2: Averaged results for Dataset D.1. The scaled time means the time of each method over the minimum time of all methods. For (a),  $d \in \{20, 100, 250, 500\}$ ; for (b),  $n \in \{100, 250, 500, 1000\}$ ; for (c),  $d \in \{20, 50, 250, 500\}$ ; for (d),  $d \in \{10, 20, 100, 250\}$ .
(d)  $n = 10d$
D Additional Numerical Results
In this section, we present additional numerical results to evaluate the performance of our proposed methods to compute the PRW distance (1). We follow the ways in [34, 28, 41] to generate the synthetic and real datasets.
Datasets D.1 (Synthetic dataset: Fragmented hypercube [41]). Define a map  $P(x) = x + 2\mathrm{sign}(x) \odot \sum_{k=1}^{k^*} e_k$ , where  $\mathrm{sign}(\cdot)$  is taken elementwise,  $k^* \in \{1, \dots, d\}$  and  $\{e_1, \dots, e_{k^*}\}$  is the canonical basis of  $\mathbb{R}^d$ . Let  $\mu = \mathcal{U}([-1,1]^d)$  be the uniform distribution over an hypercube and  $\nu = P_{\#}(\mu)$  be the pushforward of  $\mu$  under the map  $P$ . We set  $k = k^* = 2$  and take both the weight vectors  $r$  and  $c$  as  $1/n$ .
Datasets D.2 (Real datasets: digits from MNIST datasets [34]). For each digit  $0,1,\ldots ,9$ , we extract the 128-dimensional features from a pre-trained convolutional neural network. Each digit corresponds to a matrix  $X\in \mathbb{R}^{128\times n_X}$ . The values  $n_{X}$  of  $0,1,\ldots ,9$  are 980, 1135, 1032, 1010, 982, 892, 958, 1028, 974, and 1009, respectively. The weight vector  $r$  or  $c$  is taken as  $\mathbf{1}_{n_x} / n_X$ . We choose  $k = 2$ .
D.1 Comparison with R(A)BCD on Solving Subproblem (11)
We follow the same settings of  $\eta$  in [28]. For Dataset D.1, we choose  $\eta = 0.2$  when  $d < 250$  and  $\eta = 0.5$  otherwise. For Dataset D.2, we set  $\eta = 8$ .
Parameters of  $R(A)BCD$  in this section. For Dataset D.1, we choose  $\tau_{\mathrm{RBCD}} = \tau_{\mathrm{RABCD}} = 0.001$  as done in [28]. For Dataset D.2, we choose  $\tau_{\mathrm{RBCD}} = 0.004$ , and we do not test RABCD for this dataset since the well-chosen stepsize is not provided in [28].
For Dataset D.1, we randomly generate 10 instances for each  $(n,d)$  pair, each equipped with 5 randomly generated initial points. The comparison results for Dataset D.1 are plotted as Figure 2. Note that the values  $\widehat{\mathcal{P}}_k^2$  returned by different methods are almost the same for this dataset. Therefore, we do not report the values in the figure. From the figure, we can draw the following observations. (i) Among iRBBS methods, iRBBS with smaller  $\theta$  always has less nGrad but more nSk. This means that computing grad  $q(U^{t})$  in a relatively high precision can help to reduce the whole iteration number of updating  $U$ . In particular, iRBBS-0, which computes (almost) the exact grad  $q(U^{t})$ , takes the least nGrad. Because the complexity of one Sinkhorn iteration is much less than that of updating  $U$  (see Remark 3.5), iRBBS with a moderate  $\theta$  generally achieves the best overall performance. The last rows of Figure 2 show that iRBBS-0.1 is almost the fastest among all iRBBS methods. (ii) Our iRBBS- $\theta$  is better than R(A)BCD in terms that the former always have smaller nGrad and nSk and are always faster. Particularly, for Dataset D.1, iRBBS-0.1 is always more than 5x faster than RABCD and is about more than 10x faster than RBCD. For the largest instance  $d = 250$ ,  $n = 2500$ , iRBBS-0.1 (1.6s) is more than 7.2x faster than RABCD (11.1s) and about 28.6x faster than RBCD (44.4s). It
should also be mentioned that for all 800 problem instances, there are 5/21 instances for which RBCD/RABCD meet the maximum iteration numbers and return solutions not satisfying the stopping criteria.
For each instance of Dataset D.2, we randomly generate 20 initial points. The comparison results for Dataset D.2 are reported in Table 3. From the table, we have the following observations: (i) as for the solution quality, iRBBS-0 and iRBBS-0.1 perform the best among all methods; (ii) among iRBBS methods, iRBBS-0.1 and iRBBS-0 take the least nGrad and iRBBS-inf takes the most nGrad
Table 4: The average results of REALM for Dataset D.1. In this table, “a” and “b” stand for REALM-(0.02, 0) and REALM-(0.055, 0.9), respectively.
n/d	\(\widehat{P}_k^2\)	nGrad	\(nS k_{exp}/nS k_{log}\)	time	iter
a	b	a	b	a	b	a	b	a	b
100/20	8.34297	8.34296	182	138	3208/8342	13258/0	1.1	0.1	0.0/7.0	8.0/14.0
100/100	9.17320	9.17340	287	241	3837/3354	11369/0	0.5	0.1	0.0/7.0	8.0/14.0
100/250	10.89561	10.89689	467	348	3200/8858	10550/279	1.3	0.2	0.0/7.0	8.0/14.1
100/500	13.31111	13.31524	680	469	2192/3782	9844/213	0.8	0.2	0.0/7.0	8.0/14.0
100/50	8.59993	8.60005	321	189	3945/13096	11700/0	1.8	0.1	0.0/7.0	8.0/14.0
250/50	8.23339	8.23336	150	156	2480/4168	10828/0	2.2	0.3	0.0/7.0	7.9/13.9
500/50	8.12987	8.12986	122	141	1944/4508	11578/0	7.2	0.8	0.0/7.0	8.0/14.0
1000/50	8.07096	8.07095	111	130	1716/5307	12344/0	30.8	2.3	0.0/7.0	8.0/14.0
20/20	9.30977	9.31036	591	221	4016/23861	11906/0	0.5	0.0	0.0/7.1	8.0/14.0
50/50	9.36270	9.36292	386	248	4656/9416	11819/0	0.5	0.1	0.0/7.0	7.9/13.9
250/250	9.17441	9.17476	282	257	2942/4136	11371/248	2.2	0.5	0.0/7.0	8.0/14.0
500/500	9.11498	9.11540	258	262	2391/4884	9927/873	7.5	2.2	0.0/7.0	7.6/13.6
100/10	8.16197	8.16193	405	173	4635/33570	65772/0	4.6	0.3	0.0/7.0	8.0/14.0
200/20	8.13310	8.13309	180	128	2861/10259	15007/0	3.8	0.3	0.0/7.0	8.0/14.0
1000/100	8.11874	8.11873	124	145	2096/5157	12153/141	30.5	3.1	0.0/7.0	8.0/14.0
2500/250	8.11640	8.11639	117	146	2416/5499	11482/624	373.8	103.3	0.0/7.0	7.9/13.9
AVG	9.01557	9.01602	291	212	3033/9262	15057/149	29.3	7.1	0.0/7.0	8.0/14.0
while iRBBS-0 takes the most nSk and iRBBS-inf takes the least nSk; Compared with iRBBS-inf, iRBBS-0.1 is about 2.7x faster for Dataset D.2. Compared with iRBBS-0, iRBBS-0.1 is about 1.4x faster for Dataset D.2; (iii) compared with R(A)BCD, iRBBS-0.1 can take significantly less nGrad and may take a bit more nSk. This makes it about 7x faster than RABCD and about 12.8x faster than RBCD for Dataset D.2. Besides, for instances D1/D8, D2/D4, D3/D4, D4/D8, RBCD meets the maximum iteration number.
From the above results, we can conclude that iRBBS generally performs much better than R(A)BCD for the two datasets. More importantly, our methods adopt the adaptive stepsize without needing to tune the best stepsize as done in R(A)BCD.
D.2 Comparison on Computing the PRW Distance (1)
In this subsection, we present more numerical results on Datasets D.1 and D.2 to illustrate the effectiveness and efficiency of our proposed REALM, namely, Algorithm 1. The subproblem (11) is solved by our developed iRBBS algorithm, namely, Algorithm 2, with  $\theta_t$  chosen as in Section 4.1.
The results for Dataset D.1 are presented in Table 4. For each  $(n,d)$  pair, we randomly generate 10 instances, each equipped with 5 randomly generated initial points. We consider REALM-(0.055, 0.9) and REALM-(0.02, 0) both with  $\eta_{1} = 1$ ,  $\gamma_{\epsilon} = 0.25$ , and  $\gamma_{\eta} = 0.5$ . Note that the latter admits a smaller  $\eta_{\mathrm{min}}$  and does not update the multiplier matrix. From Table 4, we can observe that REALM-(0.055, 0.9) can not only return better solutions than REALM-(0.02, 0.9) but also is about  $4\mathrm{x}$  faster. This shows that updating the multiplier matrix does help. In fact, on average REALM-(0.055, 0.9) updates the multiplier matrix 8 times in 14 total iterations.
The results over 20 runs on the real Dataset D.2 are reported in Table 5. We consider REALM-(1, 0) and REALM-(3, 0.9) both with  $\eta_{1} = 200$  and  $\gamma_{\eta} = \gamma_{\epsilon} = 0.25$ . From Table 5, we can see that updating the multiplier matrix also helps. Compared with REALM-(1, 0), REALM-(3, 0.9) can not only return better solutions but is about  $2.4x$  faster. On average, REALM-(3, 0.9) updates the multiplier matrix 6.7 times in 11.7 total iterations.