SlowGuess/NeurIPS_2023_OCR huggingface dataset:
for every  $\mathbf{x} \in \mathcal{K}$ . This gives us a separation oracle between  $\mathbf{y}$  and  $\mathcal{K}$ . The computation of this separation oracle requires no more than  $m$  function value evaluations and a single gradient evaluation. This is efficiently implementable when  $m$  is of moderate size.
Remark 5. Using Gauss's Lemma (Lee, 2006, Theorem 6.8), we can efficiently compute the projection onto a geodesic ball. Consider  $\mathbf{x} \notin \mathbb{B}_{\mathbf{p}}(r)$  for which we aim to compute its projection onto  $\mathbb{B}_{\mathbf{p}}(r)$ . First, we determine the geodesic segment, denoted as  $\gamma(t)$ , connecting  $\mathbf{x}$  and  $\mathbf{p}$ . This segment intersects the geodesic ball at point  $\mathbf{z}$ . Gauss's Lemma then tells us that the tangent vector of  $\gamma(t)$  at  $\mathbf{z}$  is orthogonal to the geodesic sphere  $\mathbb{S}_p(r)$ . This indicates that  $\mathbf{z}$  is the projection of  $\mathbf{x}$  onto  $\mathbb{B}_{\mathbf{p}}(r)$ . By performing  $O(\log \left(\frac{1}{\epsilon}\right))$  rounds of binary search on  $\gamma(t)$ , we can approximate  $\mathbf{z}$  to within an  $\epsilon$  precision. In each round, it suffices to verify if the condition  $d(\mathbf{x}_t, \mathbf{p}) \geq r$  is met.
Lemma 19. (Wang et al., 2023, Lemma 45) Suppose the sectional curvature of  $\mathcal{M}$  is in  $[\kappa_1,\kappa_2]$  where  $\kappa_{1}\leq 0$  and  $\kappa_{2}\geq 0$ . Assume a gsc-convex set  $\mathcal{K}$  satisfies  $\mathbb{B}_{\mathbf{p}}(r)\subseteq \mathcal{K}\subseteq \mathbb{B}_{\mathbf{p}}(R)$  where  $R\leq \frac{\pi}{2\sqrt{\kappa_2}}$  when  $\kappa_{2} > 0$ . Then for any  $\mathbf{y}\in (1 - \tau)\mathcal{K}$ , the geodesic ball  $\mathbb{B}_{\mathbf{y}}\left(\frac{s(\kappa_2,R + r)\cdot\tau r}{s(\kappa_1,R + r)}\right)\in \mathcal{K}$ , where  $s(\cdot ,\cdot)$  is defined in Definition 4.
Lemma 20. (Bridson & Haefliger, 2013, Chapter II.1) Suppose  $\Delta(\mathbf{p},\mathbf{q},\mathbf{r})$  is a triangle on a Hadamard manifold  $\mathcal{M}$  and  $\Delta(\bar{\mathbf{p}},\bar{\mathbf{q}},\bar{\mathbf{r}})$  is a comparison triangle in Euclidean space such corresponding side lengths are the same for both triangles. Then for any  $\mathbf{x} \in [\mathbf{p},\mathbf{q}], \mathbf{y} \in [\mathbf{p},\mathbf{r}], \bar{\mathbf{x}} \in [\bar{\mathbf{p}},\bar{\mathbf{q}}]$ ,  $\bar{\mathbf{y}} \in [\bar{\mathbf{p}},\bar{\mathbf{r}}]$  satisfying  $d(\mathbf{p},\mathbf{x}) = d_{\mathbb{E}}(\bar{\mathbf{p}},\bar{\mathbf{x}})$  and  $d(\mathbf{p},\mathbf{y}) = d_{\mathbb{E}}(\bar{\mathbf{p}},\bar{\mathbf{y}})$ , we have  $d(\mathbf{x},\mathbf{y}) \leq d_{\mathbb{E}}(\bar{\mathbf{x}},\bar{\mathbf{y}})$ .
Lemma 21. The function  $\frac{\sinh x}{x}$  is increasing for any  $x\in [0,\infty ]$
Proof. We denote  $g(x) = \frac{\sinh x}{x}$ , then
$$
g ^ {\prime} (x) = \frac {x \cosh x - \sinh x}{x ^ {2}}.
$$
It suffices to show  $h(x) \coloneqq x \cosh x - \sinh x \geq 0$  holds for any  $x \in [0, \infty)$ . This is obvious because  $h(0) = 0$  and  $h'(x) = x \sinh x \geq 0$  for any  $x \in [0, \infty)$ . It remains to compute  $g'(0)$ . By L'Hôpital's rule,
$$
g ^ {\prime} (0) = \lim _ {x \to 0} \frac {x \cosh x - \sinh x}{x ^ {2}} = \lim _ {x \to 0} \frac {x \sinh x}{2 x} = 0.
$$
Lemma 22. Let  $f: \mathcal{M} \to \mathbb{R}$  be gsc-convex and  $G$ -Lipschitz on a gsc-convex set  $\mathcal{K} \subseteq \mathcal{M}$ . Then
$$
\hat {f} (\mathbf {x}) := \frac {1}{V _ {\delta}} \int_ {\mathbb {B} _ {\mathbf {x}} (\delta)} f (\mathbf {u}) \omega
$$
$$
| \hat {f} (\mathbf {x}) - f (\mathbf {x}) | \leq G \delta .
$$
$$
\begin{array}{l} \left| \hat {f} (\mathbf {x}) - f (\mathbf {x}) \right| = \left| \mathbb {E} _ {\| \mathbf {v} \| = 1} \left[ f \left(\operatorname {E x p} _ {\mathbf {x}} (\delta \mathbf {v}) - f (\mathbf {x}) \right] \right| \right. \\ \leq \mathbb {E} _ {\| \mathbf {v} \| = 1} [ f (\operatorname {E x p} _ {\mathbf {x}} (\delta \mathbf {v})) - f (\mathbf {x}) ] \tag {55} \\ \leq \mathbb {E} _ {\| \mathbf {v} \| = 1} \left[ G \| \delta \mathbf {v} \| \right] \leq G \delta . \\ \end{array}
$$
where  $\mathbf{v} \in T_{\mathbf{x}}\mathcal{M}$ . The first inequality is due to Jensen's inequality, and the second one is by the gradient Lipschitzness of  $f$ .
Definition 5. Given a homogeneous Riemannian manifold  $\mathcal{M}$ , we use  $S_{\delta}$  and  $V_{\delta}$  to denote the surface area and the volume of a geodesic ball with radius  $\delta$ .
Remark 6. We note that on homogeneous Riemannian manifolds, the volume and the area of a geodesic ball only depend on its radius (Kowalski & Vanhecke, 1982), so  $S_{\delta}$  and  $V_{\delta}$  are well-defined.
Lemma 23. (Wang et al., 2023, Lemmas 11 and 13) For a homogeneous Hadamard manifold  $(\mathcal{M},g)$  with sectional curvature lower bounded by  $\kappa \leq 0$ ,  $f$  be a  $C^1$  and gsc-convex function defined on  $\mathcal{M}$  such that  $|f|\leq C$  for any  $x\in \mathcal{M}$ . We define
$$
\hat {f} (\mathbf {x}) = \frac {1}{V _ {\delta}} \int_ {\mathbb {B} _ {\mathbf {x}} (\delta)} f (\mathbf {u}) \omega
$$
where  $\omega$  is the volume element, and
$$
\mathbf {g} (\mathbf {u}) = f (\mathbf {u}) \frac {\operatorname {E x p} _ {\mathbf {x}} ^ {- 1} (\mathbf {u})}{\| \operatorname {E x p} _ {\mathbf {x}} ^ {- 1} (\mathbf {u}) \|}.
$$
is a corresponding gradient estimator. Then we have
1)  $\frac{S_{\delta}}{V_{\delta}}\mathbf{g}(\mathbf{u})$  is an unbiased estimator of  $\nabla \hat{f}(\mathbf{x})$ . More specifically, for any  $\mathbf{x} \in \mathcal{M}$ ,
$$
\mathbb {E} _ {\mathbf {u} \in \mathbb {S} _ {\mathbf {x}} (\delta)} \left[ \frac {S _ {\delta}}{V _ {\delta}} \mathbf {g} (\mathbf {u}) \Big | \mathbf {x} \right] = \nabla \hat {f} (\mathbf {x}).
$$
2) The expected norm of  $\frac{S_{\delta}}{V_{\delta}} \mathbf{g}(\mathbf{u})$  can be bounded by
$$
\mathbb {E} _ {\mathbf {u} \in \mathbb {S} _ {\mathbf {x}} (\delta)} \left[ \left\| \frac {S _ {\delta}}{V _ {\delta}} \mathbf {g} (\mathbf {u}) \right| \mathbf {x} \right\rVert ] \leq \frac {S _ {\delta}}{V _ {\delta}} C \leq \left(\frac {n}{\delta} + n | \kappa | \delta\right) C.
$$
3) Suppose  $\mathcal{K} \in \mathcal{M}$  is a gsc-convex set and  $\|\nabla f(\mathbf{x})\| \leq G$ , then  $\hat{f}$  satisfies
$$
\hat {f} (\mathbf {y}) - \hat {f} (\mathbf {x}) - \left\langle \nabla \hat {f} (\mathbf {x}), \operatorname {E x p} _ {\mathbf {x}} ^ {- 1} (\mathbf {y}) \right\rangle \geq - 2 \rho \delta G,
$$
where  $\rho$  solely depends on the set  $\mathcal{K}$ .
Remark 7. From the proof of Lemma 23, we can figure out the formal definition of  $\rho$  is
$$
\rho = \sup  _ {\mathbf {x}, \mathbf {y}, \mathbf {u} \in \mathcal {K}} \left\| \frac {1}{\sqrt {G}} \frac {\partial}{\partial x _ {i}} \left(\sqrt {G} \mathrm {E x p} _ {\mathbf {u}} ^ {- 1} \phi (\mathbf {u})\right) ^ {i} \right\| \quad s. t. \quad \phi (\mathbf {x}) = \mathbf {y}.
$$
By Wang et al. (2023), this quantity can be  $O(e^{D})$  even on a 2-dimensional Poincaré disk.
Algorithm 9: Riemannian Frank-Wolfe with line-search
Data: feasible gsc-convex set  $\mathcal{K}$ , initial point  $\mathbf{x}_0\in \mathcal{K}$ , gsc-convex objective  $f(\cdot)$ .
for  $i = 0,\ldots$  do
$$
\begin{array}{l l} & \mathbf {v} _ {i} = \operatorname * {a r g m i n} _ {\mathbf {x} \in \mathcal {K}} \{\left\langle \nabla f (\mathbf {x} _ {i}), \mathrm {E x p} _ {\mathbf {x} _ {i}} ^ {- 1} \mathbf {x} \right\rangle \} \\ & \sigma_ {i} = \operatorname * {a r g m i n} _ {\sigma \in [ 0, 1 ]} \{f (\mathrm {E x p} _ {\mathbf {x} _ {i}} (\sigma \mathrm {E x p} _ {\mathbf {x} _ {i}} ^ {- 1} \mathbf {v} _ {i})) \} \\ & \mathbf {x} _ {i + 1} = \mathrm {E x p} _ {\mathbf {x} _ {i}} (\sigma_ {i} \mathrm {E x p} _ {\mathbf {x} _ {i}} ^ {- 1} \mathbf {v} _ {i}) \end{array}
$$
Lemma 24. (Weber & Sra, 2022b, Theorem 1) Suppose the diameter of the gsc-convex set  $\mathcal{K} \subseteq \mathcal{M}$  is upper bounded by  $D$  and  $f$  is gsc-convex as well as  $L$  gsc-smooth on  $\mathcal{K}$ . With a linear optimization oracle in hand, the primal convergence rate of Algorithm 9 can be described by
$$
f (\mathbf {x} _ {k}) - f (\mathbf {x} ^ {*}) \leq \frac {2 L D ^ {2}}{k + 2}.
$$
Lemma 25. (Weber & Sra, 2022b, Lemma 2) Under the same assumptions as in Lemma 24, for a step  $\mathbf{x}_{k + 1} = \mathrm{Exp}_{\mathbf{x}_k}(\eta \mathrm{Exp}_{\mathbf{x}_k}^{-1}\mathbf{v})$  with  $\eta \in [0,1]$ ,
$$
f (\mathbf {x} _ {k + 1}) \leq f (\mathbf {x} _ {k}) + \eta \left\langle \nabla f (\mathbf {x} _ {k}), \operatorname {E x p} _ {\mathbf {x} _ {k}} ^ {- 1} \mathbf {v} \right\rangle + \frac {\eta^ {2} L D ^ {2}}{2}.
$$
Lemma 26. Under the same assumptions as in Lemma 24, denote
$$
g(\mathbf{x}):= \max_{\mathbf{v}\in \mathcal{K}}\left\langle -\operatorname{Exp}_{\mathbf{x}}^{-1}\mathbf{v},\nabla f(\mathbf{x})\right\rangle .
$$
If Algorithm 9 is run for  $K \geq 2$  rounds, then there exists  $k_*$  such that  $1 \leq k_* \leq K$  and
$$
g \left(\mathbf {x} _ {k _ {*}}\right) \leq \frac {2 7 L D ^ {2}}{4 (K + 2)}.
$$
Proof. The proof is an adaptation of Jaggi (2013, Theorem 2). We first assume the duality gap is large in the last one third of  $K$  iterations, then get a contradictory to prove conclusion. For simplicity, we define  $h_k \coloneqq f(\mathbf{x}_k) - f(\mathbf{x}^*)$  and  $g_k \coloneqq g(\mathbf{x}_k)$ .
To ease the elaboration, we denote  $C \coloneqq 2LD^2$ ,  $\alpha \coloneqq \frac{2}{3}$ ,  $b \coloneqq \frac{27}{8}$  and  $d \coloneqq K + 2$ . We assume  $g_k > \frac{bC}{K + 2} = \frac{bC}{d}$  holds for the last one third of the  $K$  iterations. More specifically,
$$
g _ {k} > \frac {b C}{d} \quad \text {f o r} \quad k \in \{\lceil \alpha d \rceil - 2, \dots , K \}.
$$
By Lemma 25 with  $\eta = \frac{1}{k + 2}$ , we have
$$
\begin{array}{l} h _ {k + 1} \leq h _ {k} - \frac {2}{k + 2} g _ {k} + \frac {2 L D ^ {2}}{(k + 2) ^ {2}} \tag {56} \\ = h _ {k} - \frac {2}{k + 2} g _ {k} + \frac {C}{(k + 2) ^ {2}} \\ \end{array}
$$
Plugging in the assumption that the duality gap is large, we arrive at
$$
\begin{array}{l} h _ {k + 1} <   h _ {k} - \frac {2}{k + 2} \frac {b C}{d} + \frac {C}{(k + 2) ^ {2}} \\ \leq h _ {k} - \frac {2 b C}{d ^ {2}} + \frac {C}{\alpha^ {2} d ^ {2}} \tag {57} \\ = h _ {k} - \frac {2 b C - C / \alpha^ {2}}{d ^ {2}} \\ \end{array}
$$
for all  $k = \lceil \alpha d \rceil - 2, \ldots, K$ , when the second inequality follows from  $\lceil \alpha d \rceil - 2 \leq k \leq K$  implies  $\alpha d \leq k + 2 \leq d$ .
Denote  $k_{min} \coloneqq \lceil \alpha d \rceil - 2$  and  $h_{min} = h_{k_{min}}$ , then there are  $K - k_{min} + 1 = K - (\lceil \alpha d \rceil - 2) + 1 \geq (1 - \alpha)d$  steps in the last one third iterations. Thus summing Equation (57) from  $k = \lceil \alpha d \rceil - 2$  to  $K$ , we have
$$
\begin{array}{l} h _ {K + 1} <   h _ {m i n} - (1 - \alpha) d \frac {2 b C - C / \alpha^ {2}}{d ^ {2}} \\ \leq \frac {C}{\alpha d} - (1 - \alpha) d \frac {2 \alpha b - 1 / \alpha}{d} \cdot \frac {C}{\alpha d} \tag {58} \\ = \frac {C}{\alpha d} (1 - (1 - \alpha) \cdot (2 \alpha b - 1 / \alpha)) \\ \leq 0, \\ \end{array}
$$
where  $h_{\min} \leq \frac{C}{\alpha d}$  is due to Lemma 24, while for the last step, we plug in the values  $\alpha = \frac{2}{3}$  and  $b = \frac{27}{8}$ . However,  $h_{K+1} < 0$  is impossible because  $h_{K+1} = f(\mathbf{x}_{K+1}) - f(\mathbf{x}^*)$ . Thus, there always exists  $k_*$  in the last one third iterations of  $K$  such that
$$
g \left(\mathbf {x} _ {k _ {*}}\right) \leq \frac {b C}{d} = \frac {\frac {2 7}{8} \cdot 2 L D ^ {2}}{K + 2} = \frac {2 7 L D ^ {2}}{4 (K + 2)}.
$$
Lemma 27. (Ahn & Sra, 2020, Prop. H.1) Let  $\mathcal{M}$  be a Riemannian manifold with sectional curvatures lower bounded by  $\kappa \leq 0$  and the distance function  $d(\mathbf{x}) = \frac{1}{2} d(\mathbf{x},\mathbf{p})^2$  where  $\mathbf{p} \in \mathcal{M}$ . For  $D \geq 0$ ,  $d(\cdot)$  is  $\zeta$  gsc-smooth within the domain  $\{\mathbf{u} \in \mathcal{M}: d(\mathbf{u},\mathbf{p}) \leq D\}$  where  $\zeta = \sqrt{-\kappa} D \coth (\sqrt{-\kappa} D)$  is the geometric constant defined in Definition 1.
Lemma 28. Suppose  $f_{i}(\mathbf{x})$  is  $\alpha$ -strongly gsc-convex for any  $i = 1, \ldots, N$ , then  $\sum_{i=1}^{N} f_{i}(\mathbf{x})$  is  $\alpha N$ -strongly gsc-convex.
Proof. By the  $\alpha$ -strongly gsc-convexity of each  $f_{i}(\mathbf{x})$
$$
f _ {i} (\mathbf {y}) \geq f _ {i} (\mathbf {x}) + \left\langle \nabla f _ {i} (\mathbf {x}), \operatorname {E x p} _ {\mathbf {x}} ^ {- 1} (\mathbf {y}) \right\rangle + \frac {\alpha}{2} d (\mathbf {x}, \mathbf {y}) ^ {2}.
$$
Summing from  $i = 1$  to  $N$ , we have
$$
\begin{array}{l} \sum_ {i = 1} ^ {N} f _ {i} (\mathbf {y}) \geq \sum_ {i = 1} ^ {N} f _ {i} (\mathbf {x}) + \sum_ {i = 1} ^ {N} \left\langle \nabla f _ {i} (\mathbf {x}), \operatorname {E x p} _ {\mathbf {x}} ^ {- 1} \mathbf {y} \right\rangle + \sum_ {i = 1} ^ {N} \frac {\alpha}{2} d (\mathbf {x}, \mathbf {y}) ^ {2} \tag {59} \\ = \sum_ {i = 1} ^ {N} f _ {i} (\mathbf {x}) + \left\langle \sum_ {i = 1} ^ {N} \nabla f _ {i} (\mathbf {x}), \operatorname {E x p} _ {\mathbf {x}} ^ {- 1} \mathbf {y} \right\rangle + \frac {1}{2} (\alpha N) d (\mathbf {x}, \mathbf {y}) ^ {2}. \\ \end{array}
$$
This indeed implies  $\sum_{i=1}^{N} f_i(\mathbf{y})$  is  $\alpha N$ -strongly gsc-convex.
Lemma 29. (Zhang & Sra, 2016, Lemma 5). Let  $\mathcal{M}$  be a Riemannian manifold with sectional curvature lower bounded by  $\kappa \leq 0$ . Consider a geodesic triangle fully lies within  $\mathcal{M}$  with side lengths  $a, b, c$ , we have
$$
a ^ {2} \leq \zeta (\kappa , c) b ^ {2} + c ^ {2} - 2 b c \cos A
$$
where  $\zeta (\kappa ,c)\coloneqq \sqrt{-\kappa} c\coth (\sqrt{-\kappa} c)$
Lemma 30. (Sakai, 1996, Prop. 4.5) Let  $\mathcal{M}$  be a Riemannian manifold with sectional curvature upper bounded by  $\kappa \leq 0$ . Consider  $\mathcal{N}$ , a gsc-convex subset of  $\mathcal{M}$  with diameter  $D$ . For a geodesic triangle fully lies within  $\mathcal{N}$  with side lengths  $a, b, c$ , we have
$$
a ^ {2} \geq b ^ {2} + c ^ {2} - 2 b c \cos A.
$$
E.3 Extension to  $\mathbf{CAT}(\kappa)$  Spaces
It is not too difficult to generalize the result in this work to  $\mathrm{CAT}(\kappa)$  spaces, which are simply connected manifolds with sectional curvature upper bounded by  $\kappa$ . To achieve this extension, the following adjustments would be necessary in our manuscript:
- We may assume that the sectional curvature lies in the range  $[\kappa, K]$ . In this context, we can replace Lemma 30 in our draft with Alimisis et al. (2020, Corollary 2.1). When  $K > 0$ , we must further assume that the diameter of the decision set is upper-bounded by  $\frac{\pi}{\sqrt{K}}$ .
- The separation theorem, specifically for Hadamard manifolds as outlined in Silva Louzeiro et al. (2022), would require generalization to  $\mathrm{CAT}(\kappa)$  spaces. This change ensures that the separation oracle remains well-defined.
- Lemma 3 requires modification to accommodate the recomputation of the Jacobi field, considering the manifold's sectional curvature. Lemmas 15 and 19 will be instrumental in achieving this.