SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ahn, K. and Sra, S. From nesterov's estimate sequence to riemannian acceleration. In Conference on Learning Theory, pp. 84-118. PMLR, 2020.
Alimisis, F., Orvieto, A., Bécigneul, G., and Lucchi, A. A continuous-time perspective for modeling acceleration in riemannian optimization. In International Conference on Artificial Intelligence and Statistics, pp. 1297-1307. PMLR, 2020.
Antonakopoulos, K., Belmega, E. V., and Mertikopoulos, P. Online and stochastic optimization beyond lipschitz continuity: A riemannian approach. In ICLR 2020-International Conference on Learning Representations, pp. 1-20, 2020.
Bacák, M. Convex analysis and optimization in Hadamard spaces. de Gruyter, 2014.
Bécigneul, G. and Ganea, O. Riemannian adaptive optimization methods. In 7th International Conference on Learning Representations, 2019.
Borisenko, A. Convex sets in hadamard manifolds. Differential Geometry and its Applications, 17 (2-3):111-121, 2002.
Bridson, M. R. and Haefliger, A. Metric spaces of non-positive curvature, volume 319. Springer Science & Business Media, 2013.
Carson, T., Mixon, D. G., and Villar, S. Manifold optimization for k-means clustering. In 2017 International Conference on Sampling Theory and Applications (SampTA), pp. 73-77. IEEE, 2017.
Criscitiello, C. and Boumal, N. Negative curvature obstructs acceleration for strongly geodesically convex optimization, even with exact first-order oracles. In Conference on Learning Theory, pp. 496-542. PMLR, 2022.
Garber, D. and Kretzu, B. New projection-free algorithms for online convex optimization with adaptive regret guarantees. arXiv preprint arXiv:2202.04721, 2022.
Grotschel, M., Lovász, L., and Schrijver, A. Geometric algorithms and combinatorial optimization, volume 2. Springer Science & Business Media, 2012.
Hazan, E. and Kale, S. Projection-free online learning. arXiv preprint arXiv:1206.4657, 2012.
Hazan, E. and Minasyan, E. Faster projection-free online learning. In Conference on Learning Theory, pp. 1877-1893. PMLR, 2020.
Hazan, E. and Seshadhri, C. Efficient learning algorithms for changing environments. In Proceedings of the 26th annual international conference on machine learning, pp. 393-400, 2009.
Hazan, E. et al. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2(3-4):157-325, 2016.
Hirai, H., Nieuwboer, H., and Walter, M. Interior-point methods on manifolds: theory and applications. arXiv e-prints, pp. arXiv-2303, 2023.
Hu, Z., Wang, G., and Abernethy, J. Minimizing dynamic regret on geodesic metric spaces. arXiv preprint arXiv:2302.08652, 2023.
Jaggi, M. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning, pp. 427-435. PMLR, 2013.
Jézéquel, R., Ostrovskii, D. M., and Gaillard, P. Efficient and near-optimal online portfolio selection. arXiv preprint arXiv:2209.13932, 2022.
Kowalski, O. and Vanhecke, L. Ball-homogeneous and disk-homogeneous riemannian manifolds. Mathematische Zeitschrift, 180:429-444, 1982.
Kretzu, B. and Garber, D. Revisiting projection-free online learning: the strongly convex case. In International Conference on Artificial Intelligence and Statistics, pp. 3592-3600. PMLR, 2021.
Kristály, A., Li, C., López-Acedo, G., and Nicolae, A. What do 'convexities' imply on hadamard manifolds? Journal of Optimization Theory and Applications, 170:1068-1074, 2016.
Lee, J. M. Riemannian manifolds: an introduction to curvature, volume 176. Springer Science & Business Media, 2006.
Lee, J. M. Introduction to Riemannian manifolds, volume 2. Springer, 2018.
Levy, K. and Krause, A. Projection free online learning over smooth sets. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1458-1466. PMLR, 2019.
Li, Y., Chen, X., and Li, N. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. Advances in Neural Information Processing Systems, 32, 2019.
Lin, Q., Yi, H., Pang, J., Chen, M., Wierman, A., Honig, M., and Xiao, Y. Competitive online optimization under inventory constraints. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 3(1):1-28, 2019.
Lu, Z., Brukhim, N., Gradu, P., and Hazan, E. Projection-free adaptive regret with membership oracles. In International Conference on Algorithmic Learning Theory, pp. 1055-1073. PMLR, 2023.
Mhammedi, Z. Efficient projection-free online convex optimization with membership oracle. In Conference on Learning Theory, pp. 5314-5390. PMLR, 2022a.
Mhammedi, Z. Exploiting the curvature of feasible sets for faster projection-free online learning. arXiv preprint arXiv:2205.11470, 2022b.
Molinaro, M. Online and random-order load balancing simultaneously. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 1638-1650. SIAM, 2017.
Montanari, A. and Richard, E. Non-negative principal component analysis: Message passing algorithms and sharp asymptotics. IEEE Transactions on Information Theory, 62(3):1458-1484, 2015.
Petersen, P. Riemannian geometry, volume 171. Springer, 2006.
Rusciano, A. A riemannian corollary of helly's theorem. arXiv preprint arXiv:1804.10738, 2018.
Sakai, T. Riemannian geometry, volume 149. American Mathematical Soc., 1996.
Silva Louzeiro, M., Bergmann, R., and Herzog, R. Fenchel duality and a separation theorem on hadamard manifolds. SIAM Journal on Optimization, 32(2):854-873, 2022.
Udriste, C. Convex functions and optimization methods on Riemannian manifolds, volume 297. Springer Science & Business Media, 2013.
Wan, Y. and Zhang, L. Projection-free online learning over strongly convex sets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10076-10084, 2021.
Wan, Y., Tu, W.-W., and Zhang, L. Online frank-wolfe with arbitrary delays. Advances in Neural Information Processing Systems, 35:19703-19715, 2022.
Wan, Y., Zhang, L., and Song, M. Improved dynamic regret for online frank-wolfe. In Neu, G. and Rosasco, L. (eds.), Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 3304-3327. PMLR, 12-15 Jul 2023. URL https://proceedings.mlr.press/v195/wan23a.html.
Wang, X., Li, C., and Yao, J.-C. On some basic results related to affine functions on riemannian manifolds. Journal of Optimization Theory and Applications, 170(3):783-803, 2016.
Wang, X., Tu, Z., Hong, Y., Wu, Y., and Shi, G. No-regret online learning over riemannian manifolds. Advances in Neural Information Processing Systems, 34:28323-28335, 2021.
Wang, X., Tu, Z., Hong, Y., Wu, Y., and Shi, G. Online optimization over riemannian manifolds. Journal of Machine Learning Research, 24(84):1-67, 2023.
Weber, M. and Sra, S. Projection-free nonconvex stochastic optimization on riemannian manifolds. IMA Journal of Numerical Analysis, 42(4):3241-3271, 2022a.
Weber, M. and Sra, S. Riemannian optimization via frank-wolfe methods. Mathematical Programming, pp. 1-32, 2022b.
Zhang, H. and Sra, S. First-order methods for geodesically convex optimization. In Conference on Learning Theory, pp. 1617-1638. PMLR, 2016.
A Omitted Proofs in Section 4
Proof. Fix  $t$ , since  $\tilde{\mathbf{y}}_{t+1}$  is an infeasible projection of  $\mathbf{y}_{t+1}$  onto  $\tilde{\mathcal{K}}$ , and  $\mathbf{y}_{t+1} = \mathrm{Exp}_{\tilde{\mathbf{y}}_t}(-\eta_t\nabla_t)$ , by Lemma 29, we have for any  $\mathbf{x} \in \tilde{\mathcal{K}}$ ,
$$
\begin{array}{l} d \left(\tilde {\mathbf {y}} _ {t + 1}, \mathbf {x}\right) ^ {2} \leq d \left(\mathbf {y} _ {t + 1}, \mathbf {x}\right) ^ {2} \leq \zeta d \left(\mathbf {y} _ {t + 1}, \tilde {\mathbf {y}} _ {t}\right) ^ {2} + d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {x}\right) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {y} _ {t + 1}, \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \tag {4} \\ \leq \zeta \eta_ {t} ^ {2} \| \nabla_ {t} \| ^ {2} + d (\tilde {\mathbf {y}} _ {t}, \mathbf {x}) ^ {2} + 2 \eta_ {t} \left\langle \nabla_ {t}, \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle . \\ \end{array}
$$
To verify that  $\zeta$  represents the correct geometric distortion, we need to show  $d(\tilde{\mathbf{y}}_t,\mathbf{x})\leq 2R$  holds for any  $\tilde{\mathbf{y}}_t\in \mathbb{B}_{\mathbf{p}}(R)$  and  $\mathbf{x}\in \tilde{\mathcal{K}}$ . Since  $\tilde{\mathcal{K}}\subseteq \mathcal{K}\subseteq \mathbb{B}_{\mathbf{p}}(R)$ , we can demonstrate this by the triangle inequality:
$$
d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {x}\right) \leq d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {p}\right) + d (\mathbf {p}, \mathbf {x}) \leq 2 R.
$$
Rearranging Equation (4), we have
$$
\left\langle \nabla_ {t}, - \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \leq \frac {d (\tilde {\mathbf {y}} _ {t} , \mathbf {x}) ^ {2}}{2 \eta_ {t}} - \frac {d (\tilde {\mathbf {y}} _ {t + 1} , \mathbf {x}) ^ {2}}{2 \eta_ {t}} + \frac {\zeta \eta_ {t} \| \nabla_ {t} \| ^ {2}}{2}. \tag {5}
$$
Fix  $1 \leq s \leq e \leq T$  and sum over  $[s, e]$ , we have
$$
\sum_ {t = s} ^ {e} \left\langle \nabla_ {t}, - \mathrm {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \leq \frac {d (\tilde {\mathbf {y}} _ {s} , \mathbf {x}) ^ {2}}{2 \eta_ {s}} + \sum_ {t = s + 1} ^ {e} \left(\frac {1}{2 \eta_ {t}} - \frac {1}{2 \eta_ {t - 1}}\right) d (\tilde {\mathbf {y}} _ {t}, \mathbf {x}) ^ {2} + \sum_ {t = s} ^ {e} \frac {\zeta \eta_ {t} \| \nabla_ {t} \| ^ {2}}{2}.
$$
Using gsc-convexity of  $f_{t}(\cdot)$  and set  $\eta_t = \eta$ , we have
$$
\sum_ {t = s} ^ {e} f _ {t} (\tilde {\mathbf {y}} _ {t}) - f _ {t} (\mathbf {x}) \leq \frac {d (\tilde {\mathbf {y}} _ {s} , \mathbf {x}) ^ {2}}{2 \eta} + \frac {\eta \zeta \sum_ {t = s} ^ {e} \| \nabla_ {t} \| ^ {2}}{2}.
$$
In case all losses are  $\alpha$ -strongly gsc-convex, using  $f_{t}(\mathbf{y}) - f_{t}(\mathbf{x}) \leq \left\langle \nabla f_{t}(\mathbf{y}), -\mathrm{Exp}_{\mathbf{y}}^{-1}\mathbf{x} \right\rangle - \frac{\alpha d(\mathbf{x},\mathbf{y})^{2}}{2}$  and set  $(s,e) = (1,T)$ , we have
$$
\sum_ {t = 1} ^ {T} f _ {t} (\tilde {\mathbf {y}} _ {t}) - f _ {t} (\mathbf {x}) \leq \sum_ {t = 1} ^ {T} \frac {\eta_ {t} \| \nabla_ {t} \| ^ {2}}{2} + \left(\frac {1}{2 \eta_ {1}} - \frac {\alpha}{2}\right) d (\tilde {\mathbf {y}} _ {1}, \mathbf {x}) ^ {2} + \sum_ {t = 2} ^ {T} \zeta \left(\frac {1}{2 \eta_ {t}} - \frac {1}{2 \eta_ {t - 1}} - \frac {\alpha}{2}\right) d (\tilde {\mathbf {y}} _ {t}, \mathbf {x}) ^ {2}.
$$
Setting  $\eta_t = \frac{1}{\alpha t}$ , we get the guarantee for the strongly gsc-convex case.
Proof. By Lemma 29,
$$
\begin{array}{l} d (\tilde {\mathbf {y}}, \mathbf {z}) ^ {2} \leq \zeta d (\mathbf {y}, \tilde {\mathbf {y}}) ^ {2} + d (\mathbf {y}, \mathbf {z}) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \tilde {\mathbf {y}}, \operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z} \right\rangle \\ \leq \zeta \gamma^ {2} C ^ {2} + d (\mathbf {y}, \mathbf {z}) ^ {2} - 2 \left\langle - \gamma \mathbf {g}, \operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z} \right\rangle \\ \leq \zeta \gamma^ {2} C ^ {2} + d (\mathbf {y}, \mathbf {z}) ^ {2} - 2 \gamma Q \\ = d (\mathbf {y}, \mathbf {z}) ^ {2} - \frac {Q ^ {2}}{\zeta C ^ {2}}, \\ \end{array}
$$
where the first inequality relies on Lemma 29 and  $d(\mathbf{y}, \mathbf{z}) \leq 2R$ , the third inequality holds because  $-\left\langle \mathrm{Exp}_{\mathbf{y}}^{-1}\mathbf{z}, \mathbf{g} \right\rangle \geq Q$ , and for the last line, we plug in  $\gamma = \frac{Q}{\zeta C^2}$ .
B Omitted Proofs in Section 5
The technical difficulty lies in constructing an infeasible projection oracle using a separation oracle. The following lemma is instrumental to the later results.
Proof. Let  $\mathbf{x} = \mathrm{Exp}_{\mathbf{y}}\left(\delta \bar{r}\frac{\mathbf{g}}{\|\mathbf{g}\|} +\mathrm{Exp}_{\mathbf{y}}^{-1}\mathbf{z}\right)$ , if we can show  $\mathbf{x}\in \mathcal{K}$ , then by the definition of the separation oracle, we have
$$
\left\langle - \mathrm {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {x}, \mathbf {g} \right\rangle = - \delta \bar {r} \| \mathbf {g} \| - \left\langle \mathrm {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z}, \mathbf {g} \right\rangle > 0,
$$
which in turn implies  $\langle -\mathrm{Exp}_{\mathbf{y}}^{-1}\mathbf{z},\mathbf{g}\rangle >\delta \bar{r}\| \mathbf{g}\|$ . Note that when  $\bar{r} = 0$ ,  $\mathbf{x} = \mathbf{z}\in (1 - \delta)\mathcal{K}\subset \mathcal{K}$  thus there exists positive  $\bar{r}$  such that  $\mathbf{x}\in \mathcal{K}$  for any  $\mathbf{z}\in (1 - \delta)\mathcal{K}$ . Let  $\tilde{r} = \frac{\sqrt{-\kappa}(R + r)}{\sinh(\sqrt{-\kappa}(R + r))}\cdot r$ . By Lemma 19, we have  $\mathbb{B}_{\mathbf{z}}(\delta \tilde{r})\subset \mathcal{K}$ . Thus we only need to ensure  $d(\mathbf{x},\mathbf{z})\leq \delta \tilde{r}$  because this implies  $\mathbf{x}\in \mathbb{B}_{\mathbf{z}}(\delta \tilde{r})\subset \mathcal{K}$ . We define an admissible family of curves:
$$
\Gamma : [ 0, 1 ] \times [ 0, 1 ] \to \mathcal {M} (t, s) \to \mathrm {E x p} _ {\mathbf {y}} (t (\mathrm {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z} + s \delta \bar {r} \mathbf {v}))
$$
where  $\mathbf{v} = \frac{\mathbf{g}}{\|\mathbf{g}\|}$ . Then  $\Gamma(1,0) = \mathbf{z}$  and  $\Gamma(1,1) = \mathbf{x}$ . Let  $J_{s}(t) := \frac{\partial\Gamma(t,s)}{\partial s}$ , which is a variation field of the variation  $\Gamma$ , and is thus a Jacobi field. We can compute  $J_{s}(0) = \frac{\partial\Gamma(0,s)}{\partial s} = 0$  and by Lemma 14.
$$
\left. D _ {t} J _ {s} (t) \right| _ {t = 0} = \delta \bar {r} \mathbf {v}.
$$
To apply Lemma 15, we need to reparametrize  $\Gamma$  to make it unit speed. Let  $\Gamma(t,s) = \tilde{\Gamma}\left(\frac{t}{R(s)},s\right)$  where
$$
R (s) = d (\Gamma (1, s), \Gamma (0, s)) = \| \operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z} + s \delta \bar {r} \mathbf {v} \| \leq \| \operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \mathbf {z} \| + \delta \bar {r} \leq 2 R + \delta r \leq 2 R + r.
$$
Then we have  $J_{s}(t) = \tilde{J}_{s}(R(s)t)$  and  $D_{t}J_{s}(t) = R(s)D_{t}\tilde{J}_{s}(R(s)t)$ . Now we can apply Lemma 15:
$$
\begin{array}{l} \| J _ {s} (1) \| = \| \tilde {J} _ {s} (R (s)) \| \leq \frac {\sinh (\sqrt {- \kappa} R (s)) \| D _ {t} \tilde {J} _ {s} (0) \|}{\sqrt {- \kappa}} \\ = \frac {\sinh (\sqrt {- \kappa} R (s)) \| D _ {t} J _ {s} (0) \|}{\sqrt {- \kappa} R (s)} = \frac {\sinh (\sqrt {- \kappa} R (s)) \delta \bar {r}}{\sqrt {- \kappa} R (s)} \\ \end{array}
$$
We would like to have  $d(\mathbf{x},\mathbf{z})\leq \delta \tilde{r}$ , and it suffices to choose  $\bar{r}$  such that
$$
d (\mathbf {x}, \mathbf {z}) \leq \int_ {0} ^ {1} \| J _ {s} (1) \| d s \leq \frac {\sinh (\sqrt {- \kappa} (2 R + r))}{\sqrt {- \kappa} (2 R + r)} \delta \bar {r} \leq \delta \tilde {r}.
$$
We then find a valid  $\bar{r}$  is
$$
\bar {r} = \frac {\sqrt {- \kappa} (2 R + r)}{\sinh (\sqrt {- \kappa} (2 R + r))} \tilde {r} = \frac {\sqrt {- \kappa} (2 R + r)}{\sinh (\sqrt {- \kappa} (2 R + r))} \cdot \frac {\sqrt {- \kappa} (R + r)}{\sinh (\sqrt {- \kappa} (R + r))} \cdot r.
$$
Proof. We use  $k$  to denote the number of iterations in Algorithm 2, then for any  $i < k$ , we have  $\mathbf{y}_i \notin \mathcal{K}$ . By Lemma 3, we have  $\left\langle -\mathrm{Exp}_{\mathbf{y}_i}^{-1}\mathbf{z}, \mathbf{g}_i \right\rangle \geq \delta \bar{r} \| \mathbf{g}_i\|$  for any  $\mathbf{z} \in (1 - \delta)\mathcal{K}$ . We then invoke Lemma 2 with  $C = \| \mathbf{g}_i\|$ ,  $Q = \delta \bar{r}\| \mathbf{g}_i\|$  to get
$$
d \left(\mathbf {y} _ {i + 1}, \mathbf {z}\right) ^ {2} \leq d \left(\mathbf {y} _ {i}, \mathbf {z}\right) ^ {2} - \frac {\delta^ {2} \bar {r} ^ {2}}{\zeta} \tag {7}
$$
holds for any  $\mathbf{z} \in (1 - \delta)\mathcal{K}$ . To ensure the geometric distortion  $\zeta$  is valid here, we need to prove  $d(\mathbf{y}_i, \mathbf{z}) \leq 2R$  holds for any  $i \geq 1$  and  $\mathbf{z} \in (1 - \delta)\mathcal{K}$ , which can be guaranteed by induction. The case of  $i = 1$  is straightforward. As  $\mathbf{y}_1 \in \mathbb{B}_{\mathbf{p}}(R)$ ,  $\mathbf{z} \in (1 - \delta)\mathcal{K} \subseteq \mathbb{B}_{\mathbf{p}}(R)$ , we have
$$
d (\mathbf {y} _ {1}, \mathbf {z}) \leq d (\mathbf {y} _ {1}, \mathbf {p}) + d (\mathbf {z}, \mathbf {p}) \leq 2 R.
$$
Now assume  $d(\mathbf{y}_i,\mathbf{z})\leq 2R$  holds for some  $i\geq 1$  and any  $\mathbf{z}\in (1 - \delta)\mathcal{K}$ , then by Lemma 29 and Lemma 2,
$$
\begin{array}{l} d \left(\mathbf {y} _ {i + 1}, \mathbf {z}\right) ^ {2} \leq \zeta (\kappa , d \left(\mathbf {y} _ {i}, \mathbf {z}\right)) \cdot d \left(\mathbf {y} _ {i + 1}, \mathbf {y} _ {i}\right) ^ {2} + d \left(\mathbf {y} _ {i}, \mathbf {z}\right) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {y} _ {i + 1}, \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {z} \right\rangle \\ \leq \zeta (\kappa , 2 R) \cdot d \left(\mathbf {y} _ {i + 1}, \mathbf {y} _ {i}\right) ^ {2} + d \left(\mathbf {y} _ {i}, \mathbf {z}\right) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {y} _ {i + 1}, \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {z} \right\rangle \\ = \zeta d (\mathbf {y} _ {i + 1}, \mathbf {y} _ {i}) ^ {2} + d (\mathbf {y} _ {i}, \mathbf {z}) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {y} _ {i + 1}, \operatorname {E x p} _ {\mathbf {y} _ {i}} ^ {- 1} \mathbf {z} \right\rangle \\ \leq d (\mathbf {y} _ {i}, \mathbf {z}) ^ {2} - \frac {\delta^ {2} \bar {r} ^ {2}}{\zeta} \leq (2 R) ^ {2}. \\ \end{array}
$$
Thus, we know  $d(\mathbf{y}_i,\mathbf{z})\leq 2R$  holds for any  $i\geq 1$  and  $\mathbf{z}\in (1 - \delta)\mathcal{K}$
By Equation (7),  $d(\mathbf{y}, \mathbf{z})^2 \leq d(\mathbf{y}_1, \mathbf{z})^2$  for all  $\mathbf{z} \in (1 - \delta)\mathcal{K}$ . Since  $\mathbf{y}_1$  is the projection of  $\mathbf{y}_0$  onto  $\mathbb{B}_{\mathbf{p}}(R)$ , we have  $d(\mathbf{y}_1, \mathbf{z})^2 \leq d(\mathbf{y}_0, \mathbf{z})^2$  holds for  $\mathbf{z} \in \mathbb{B}_{\mathbf{p}}(R)$ . Thus, we indeed have  $d(\mathbf{y}, \mathbf{z})^2 \leq d(\mathbf{y}_0, \mathbf{z})^2$  for any  $\mathbf{z} \in (1 - \delta)\mathcal{K}$ .
It remains to bound the number of iterations in Algorithm 2. We must be careful because, on manifolds,  $(1 - \delta)\mathcal{K}$  is not guaranteed to be gsc-convex (Theorem 6). Note that  $\mathrm{argmin}_{\mathbf{x} \in (1 - \delta)\mathcal{K}} d(\mathbf{y}_i, \mathbf{x})^2$  is consistently non-empty because  $(1 - \delta)\mathcal{K}$  is a closed set. Let  $\mathbf{x}_i^* \in \mathrm{argmin}_{\mathbf{x} \in (1 - \delta)\mathcal{K}} d(\mathbf{y}_i, \mathbf{x})^2$  and invoke Equation (7), we have
$$
\begin{array}{l} d \left(\mathbf {y} _ {i + 1}, (1 - \delta) \mathcal {K}\right) ^ {2} = d \left(\mathbf {y} _ {i + 1}, \mathbf {x} _ {i + 1} ^ {*}\right) ^ {2} \\ \leq d \left(\mathbf {y} _ {i + 1}, \mathbf {x} _ {i} ^ {*}\right) ^ {2} \leq d \left(\mathbf {y} _ {i}, \mathbf {x} _ {i} ^ {*}\right) ^ {2} - \frac {\delta^ {2} \bar {r} ^ {2}}{\zeta} = d \left(\mathbf {y} _ {i}, (1 - \delta) \mathcal {K}\right) ^ {2} - \frac {\delta^ {2} \bar {r} ^ {2}}{\zeta}. \\ \end{array}
$$
The first inequality is due to the definition of  $\mathbf{x}_{i + 1}^{*}$  and the second one follows from Equation (7). We also need to show  $d(\mathbf{y}_1,(1 - \delta)\mathcal{K})\leq d(\mathbf{y}_0,(1 - \delta)\mathcal{K})$  to finish the proof. We have
$$
d \left(\mathbf {y} _ {1}, \mathbf {x} _ {1} ^ {*}\right) \leq d \left(\mathbf {y} _ {1}, \mathbf {x} _ {0} ^ {*}\right) \leq d \left(\mathbf {y} _ {0}, \mathbf {x} _ {0} ^ {*}\right), \tag {8}
$$
where the first inequality is again due to the definition of  $\mathbf{x}_1^*$ , while the second one comes from  $\mathbf{y}_1$  is the metric projection of  $\mathbf{y}_0$  onto  $\mathbb{B}_{\mathbf{p}}(R)$  and  $\mathbf{x}_0^* \in (1 - \delta)\mathcal{K} \subseteq \mathbb{B}_{\mathbf{p}}(R)$ . Reminding  $\mathbf{y} = \mathbf{y}_k$  and unrolling the recurrence, we have
$$
d (\mathbf {y}, (1 - \delta) \mathcal {K}) ^ {2} \leq d (\mathbf {y} _ {0}, (1 - \delta) \mathcal {K}) ^ {2} - \frac {(k - 1) \delta^ {2} \bar {r} ^ {2}}{\zeta}.
$$
Thus, in the worst case, we need
$$
k = \frac {\zeta (d (\mathbf {y} _ {0} , (1 - \delta) \mathcal {K}) ^ {2} - d (\mathbf {y} , (1 - \delta) \mathcal {K}) ^ {2})}{\delta^ {2} \bar {r} ^ {2}} + 1
$$
iterations to stop.
Proof. Combining Algorithm 3 with Lemma 4, we know  $\tilde{\mathbf{y}}_t\in \mathcal{K}$  thus Algorithm 3 plays a feasible point at each round. For a fixed interval  $I = [s,e]\subseteq [T]$ , let  $\mathbf{x}_I = \mathrm{argmin}_{\mathbf{x}\in \mathcal{K}}\sum_{t = s}^{e}f_{t}(\mathbf{x})$  and  $\tilde{\mathbf{x}}_I = \mathrm{Exp}_{\mathbf{p}}((1 - \delta)\mathrm{Exp}_{\mathbf{p}}^{-1}\mathbf{x}_I)$ . Then
$$
d (\tilde {\mathbf {x}} _ {I}, \mathbf {x} _ {I}) = d (\mathrm {E x p} _ {\mathbf {p}} ((1 - \delta) \mathrm {E x p} _ {\mathbf {p}} ^ {- 1} \mathbf {x} _ {I}), \mathbf {x} _ {I}) = d (\mathrm {E x p} _ {\mathbf {x} _ {I}} (\delta \mathrm {E x p} _ {\mathbf {x} _ {I}} ^ {- 1} \mathbf {p}), \mathbf {x} _ {I}) = \delta d (\mathbf {p}, \mathbf {x} _ {I}) \leq \delta R.
$$
By Lemma 4,  $\tilde{\mathbf{y}}_t\in \mathcal{K}$  is an infeasible projection of  $\mathbf{y}_t$  over  $(1 - \delta)\mathcal{K}$ . Then by Lemma 1
$$
\sum_ {t = s} ^ {e} \left(f _ {t} (\tilde {\mathbf {y}} _ {t}) - f _ {t} (\tilde {\mathbf {x}} _ {I})\right) \leq \frac {d \left(\tilde {\mathbf {y}} _ {s} , \mathbf {x} _ {I}\right) ^ {2}}{2 \eta} + \frac {\eta \zeta \sum_ {t = s} ^ {e} \left\| \nabla f _ {t} \left(\tilde {\mathbf {y}} _ {t}\right) \right\| ^ {2}}{2} \leq \frac {2 R ^ {2}}{\eta} + \frac {\eta \zeta G ^ {2} T}{2}.
$$
On the other hand, by the gradient-Lipschitzness, we have
$$
f _ {t} \left(\tilde {\mathbf {x}} _ {I}\right) - f _ {t} \left(\mathbf {x} _ {I}\right) \leq G \cdot d \left(\tilde {\mathbf {x}} _ {I}, \mathbf {x} _ {I}\right) \leq G \delta R.
$$
Combining the above two equations, we have
$$
\sum_ {t = s} ^ {e} \left(f _ {t} (\tilde {\mathbf {y}} _ {t}) - f _ {t} (\mathbf {x} _ {I} ^ {*})\right) \leq \left(G R \delta + \frac {G ^ {2} \eta \zeta}{2}\right) T + \frac {2 R ^ {2}}{\eta}.
$$
Setting  $\eta = \frac{2R}{G\sqrt{\zeta T}}$  and  $\delta = \frac{1}{2\sqrt{T}} < 1$ , then we have
$$
\sum_ {t = s} ^ {e} \left(f _ {t} \left(\tilde {\mathbf {y}} _ {t}\right) - f _ {t} \left(\mathbf {x} _ {I} ^ {*}\right)\right) \leq \frac {5}{2} G R \sqrt {\zeta T},
$$
where we use  $\zeta \geq 1$ . It remains to bound the number of calls to the separation oracle. Denote  $\mathbf{x}_t^* \in \mathrm{argmin}_{\mathbf{x} \in (1 - \delta) \mathcal{K}} d(\mathbf{x}, \tilde{\mathbf{y}}_t)$ . Due to  $\mathbf{y}_{t+1} = \mathrm{Exp}_{\tilde{\mathbf{y}}_t}(-\eta \nabla f_t(\tilde{\mathbf{y}}_t))$ ,
$$
d \left(\mathbf {y} _ {t + 1}, (1 - \delta) \mathcal {K}\right) \leq d \left(\mathbf {x} _ {t} ^ {*}, \mathbf {y} _ {t + 1}\right) \leq d \left(\mathbf {x} _ {t} ^ {*}, \tilde {\mathbf {y}} _ {t}\right) + d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {y} _ {t + 1}\right) \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) + \eta \| \nabla f _ {t} \left(\tilde {\mathbf {y}} _ {t}\right) \|,
$$
where the first inequality is because  $\mathbf{x}_t^* \in (1 - \delta)\mathcal{K}$ , the second is due to the triangle inequality, while the third one follows from the definition of  $\mathbf{x}_t^*$  and  $\mathbf{y}_{t + 1} = \mathrm{Exp}_{\tilde{\mathbf{y}}_t}(-\eta \nabla f_t(\tilde{\mathbf{y}}_t))$ . Squaring both sides, we have
$$
\begin{array}{l} d \left(\mathbf {y} _ {t + 1}, (1 - \delta) \mathcal {K}\right) ^ {2} \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) ^ {2} + 2 d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) \eta G + \eta^ {2} G ^ {2} \tag {9} \\ \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) ^ {2} + 2 \delta R \eta G + \eta^ {2} G ^ {2}, \\ \end{array}
$$
where the second inequality is because