SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
d (\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}) \leq d (\tilde {\mathbf {y}} _ {t}, \operatorname {E x p} _ {\mathbf {p}} ((1 - \delta) \operatorname {E x p} _ {\mathbf {p}} ^ {- 1} \tilde {\mathbf {y}} _ {t})) = d (\tilde {\mathbf {y}} _ {t}, \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} (\delta \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {p})) = \delta d (\tilde {\mathbf {y}} _ {t}, \mathbf {p}) \leq \delta R.
$$
Combining Equation (9) with Lemma 4, we can bound the number of separation oracle calls as:
$$
\begin{array}{l} N _ {c a l l s} \leq \sum_ {t = 1} ^ {T} \left(\zeta \frac {d (\mathbf {y} _ {t + 1} , (1 - \delta) \mathcal {K}) ^ {2} - d (\tilde {\mathbf {y}} _ {t + 1} , (1 - \delta) \mathcal {K}) ^ {2}}{\delta^ {2} \bar {r} ^ {2}} + 1\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\zeta \frac {d \left(\tilde {\mathbf {y}} _ {t} , (1 - \delta) \mathcal {K}\right) ^ {2} + 2 R \delta \eta G + \eta^ {2} G ^ {2} - d \left(\tilde {\mathbf {y}} _ {t + 1} , (1 - \delta) \mathcal {K}\right) ^ {2}}{\delta^ {2} \bar {r} ^ {2}} + 1\right) \tag {10} \\ \leq \zeta \left(\frac {2 R G \eta T}{\bar {r} ^ {2} \delta} + \frac {G ^ {2} \eta^ {2} T}{\bar {r} ^ {2} \delta^ {2}}\right) + T \\ = \left(\frac {8 R ^ {2} \sqrt {\zeta}}{\bar {r} ^ {2}} + \frac {1 6 R ^ {2}}{\bar {r} ^ {2}} + 1\right) T. \\ \end{array}
$$
where we use  $\tilde{\mathbf{y}}_1 = \mathbf{p} \in (1 - \delta)\mathcal{K}$  and thus  $d(\tilde{\mathbf{y}}_1, (1 - \delta)\mathcal{K}) = 0$  to derive the third inequality.
We first prove Lemma 7, which characterizes the relation between  $\delta^{\prime}$  and  $\tau$ .
Lemma 7. For a gsc-convex subset  $\mathcal{K} \subseteq \mathcal{M}$  where  $\mathcal{M}$  is Hadamard, any point  $\mathbf{y} \in \mathcal{K}$  and  $\mathbb{B}_{\mathbf{p}}(r) \subseteq \mathcal{K} \subseteq \mathbb{B}_{\mathbf{p}}(R)$ , we have
$$
\mathbb {B} _ {\mathbf {y}} \left(\frac {\sqrt {- \kappa} (R + r)}{\sinh (\sqrt {- \kappa} (R + r))}\right) \subset (1 + \tau) \mathcal {K}.
$$
Proof. The proof of this proposition takes inspiration from Lemma 19 (Wang et al., 2023), but with several significant adjustments. Notably, Lemma 19 hinges on the gsc-convexity of  $\mathcal{K}$  to bound the radius of the geodesic ball  $\mathbb{B}_{\mathbf{y}}(\tilde{r})$ , where  $\mathbf{y} \in (1 - \tau)\mathcal{K}$ , which resides within  $\mathcal{K}$ . However, the gsc-convexity of  $(1 + \tau)\mathcal{K}$  is unknown. Therefore, we must explore alternative strategies that leverage the gsc-convexity of  $\mathcal{K}$  to meet our objectives.
Let  $\mathbf{v} \in T_{\mathbf{y}}\mathcal{M}$  be a unit vector such that  $\| \mathbf{v} \| = 1$  and  $\mathbf{z} = \mathrm{Exp}_{\mathbf{y}}(\theta \tau r \mathbf{v})$ . The goal is to find the maximum  $\theta$  which ensures  $\mathbf{z} \in (1 + \tau)\mathcal{K}$  for any  $\mathbf{v}$ . Let  $\mathbf{y}' = \mathrm{Exp}_{\mathbf{p}}\left(\frac{\mathrm{Exp}_{\mathbf{p}}^{-1}\mathbf{y}}{1 + \tau}\right)$  and  $\mathbf{z}' = \mathrm{Exp}_{\mathbf{p}}\left(\frac{\mathrm{Exp}_{\mathbf{p}}^{-1}\mathbf{z}}{1 + \tau}\right)$ . It is immediate to see  $\mathbf{z}' \in \mathcal{K}$  iff  $\mathbf{z} \in (1 + \tau)\mathcal{K}$ . We denote  $\xi_{\tau}(s)$  as the geodesic satisfying  $\xi_{\tau}(0) = \mathbf{y}'$  and  $\xi_{\tau}(1) = \mathbf{z}'$ .
<sup>5</sup>We conjecture that on Hadamard manifolds,  $(1 + \tau)\mathcal{K}$  is gsc-convex for any  $\tau \geq 0$ .
We define an admissible family of curves:
$$
\Gamma : \left[ 0, \frac {1 + \tau}{\tau} \right] \times [ 0, 1 ] \rightarrow \mathcal {M} \qquad (t, s) \rightarrow \operatorname {E x p} _ {\mathbf {y}} (t (\operatorname {E x p} _ {\mathbf {y}} ^ {- 1} \xi_ {\tau} (s))).
$$
We can verify that  $\Gamma(0,1) = \mathbf{y}, \Gamma(1,0) = \mathbf{y}', \Gamma\left(\frac{1 + \tau}{\tau}, 0\right) = \mathrm{Exp}_{\mathbf{y}}\left(\frac{1 + \tau}{\tau} \cdot \mathrm{Exp}_{\mathbf{y}}^{-1}\mathbf{y}'\right) = \mathbf{p}$ , and  $\Gamma(1,1) = \mathbf{z}'$ . We also denote  $\mathbf{w} := \Gamma\left(\frac{1 + \tau}{\tau}, 1\right)$ . The idea of the proof is to show  $d(\mathbf{p}, \mathbf{w}) \leq r$  by Lemma 15, which implies  $\mathbf{w} \in \mathcal{K}$  since  $\mathbb{B}_{\mathbf{p}}(r) \subseteq \mathcal{K}$ . Combining with the fact that  $\mathbf{y} \in \mathcal{K}$  and  $\mathbf{z}'$  is on the geodesic connecting  $\mathbf{y}$  and  $\mathbf{w}$ , we have  $\mathbf{z}' \in \mathcal{K}$ . Thus  $\mathbf{z} \in (1 + \tau)\mathcal{K}$ .
We notice that  $v(t,s) = \frac{\partial\Gamma(s,t)}{\partial s}$  is a Jacobi field since it is a variation field of  $\gamma_s(t) = \mathrm{Exp}_{\mathbf{y}}(t\mathrm{Exp}_{\mathbf{y}}^{-1}\xi_\tau (s))$ . Let  $R(s) = d(\Gamma (0,s),\Gamma (1,s))$ . To apply Lemma 15, we need to normalize the geodesic  $\tilde{\gamma}_s(t) = \gamma_s\left(\frac{t}{R(s)}\right)$ . Since  $\mathcal{M}$  is Hadamard, by Lemma 15
$$
\| v (1, s) \| \geq R (s) \| \nabla_ {\dot {\gamma} _ {s}} v (0, s) \|. \tag {11}
$$
Remind that  $\| v(1,s)\| = \left\| \frac{\partial\xi_{\tau}(s)}{\partial s}\right\| = \|\dot{\xi}_{\tau}(s)\| = d(\mathbf{y}',\mathbf{z}')$ . Now we use Lemma 20 to bound  $d(\mathbf{y}',\mathbf{z}')$ . We construct  $\Delta (\bar{\mathbf{p}},\bar{\mathbf{y}},\bar{\mathbf{z}})$  in Euclidean space, a comparison triangle of  $\Delta (\mathbf{p},\mathbf{y},\mathbf{z})$  with comparison points  $\bar{\mathbf{y}}' \in [\bar{\mathbf{p}},\bar{\mathbf{y}}]$  and  $\bar{\mathbf{z}}' \in [\bar{\mathbf{p}},\bar{\mathbf{z}}]$ . We restrict  $d(\mathbf{p},\mathbf{y}') = d_{\mathbb{E}}(\bar{\mathbf{p}},\bar{\mathbf{y}}')$  and  $d(\mathbf{p},\mathbf{z}') = d_{\mathbb{E}}(\bar{\mathbf{p}},\bar{\mathbf{z}}')$ , then by Lemma 20, we have
$$
d \left(\mathbf {y} ^ {\prime}, \mathbf {z} ^ {\prime}\right) \leq d _ {\mathbb {E}} \left(\bar {\mathbf {y}} ^ {\prime}, \bar {\mathbf {z}} ^ {\prime}\right) \tag {12}
$$
On the other hand, it is immediate to verify  $\Delta (\bar{\mathbf{p}},\bar{\mathbf{y}},\bar{\mathbf{z}})$  is similar to  $\Delta (\bar{\mathbf{p}},\bar{\mathbf{y}}^{\prime},\bar{\mathbf{z}}^{\prime})$  by considering
$$
\frac {d _ {\mathbb {E}} \left(\bar {\mathbf {p}} ^ {\prime} , \bar {\mathbf {z}} ^ {\prime}\right)}{d _ {\mathbb {E}} \left(\bar {\mathbf {p}} ^ {\prime} , \bar {\mathbf {z}} ^ {\prime}\right)} = \frac {d (\mathbf {p} , \mathbf {z} ^ {\prime})}{d (\mathbf {p} , \mathbf {z})} = \frac {1}{1 + \tau} = \frac {d (\mathbf {p} , \mathbf {y} ^ {\prime})}{d (\mathbf {p} , \mathbf {y})} = \frac {d _ {\mathbb {E}} \left(\bar {\mathbf {p}} ^ {\prime} , \bar {\mathbf {y}} ^ {\prime}\right)}{d _ {\mathbb {E}} \left(\bar {\mathbf {p}} ^ {\prime} , \bar {\mathbf {y}} ^ {\prime}\right)}.
$$
$$
d _ {\mathbb {E}} \left(\bar {\mathbf {y}} ^ {\prime}, \bar {\mathbf {z}} ^ {\prime}\right) = \frac {1}{1 + \tau} d _ {\mathbb {E}} \left(\bar {\mathbf {y}}, \bar {\mathbf {z}}\right) = \frac {1}{1 + \tau} d (\mathbf {y}, \mathbf {z}) = \frac {\theta \tau r}{1 + \tau}, \tag {13}
$$
where the first equation is due to the property of similar triangles, the second one follows from the definition of the comparison triangle, and the last one is due to the definition of  $\mathbf{z}$ .
Combining Equations (11), (12) and (13), we have
$$
\left\| \nabla_ {\dot {\hat {\gamma}} _ {s}} v (0, s) \right\| \leq \frac {\| v (1 , s) \|}{R (s)} \leq \frac {\theta \tau r}{(1 + \tau) R (s)}. \tag {14}
$$
We also need to apply Lemma 15 at  $t = \frac{1 + \tau}{\tau}$ . Since the sectional curvature of  $\mathcal{M}$  is lower bounded by  $\kappa$ , we have
$$
\left\| v \left(\frac {1 + \tau}{\tau}, s\right) \right\| \leq \frac {1}{\sqrt {- \kappa}} \sinh \left(\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)\right) \cdot \| \nabla_ {\dot {\hat {\gamma}} _ {s}} v (0, s) \| \tag {15}
$$
Putting Equations (14) and (15) together, we have
$$
\begin{array}{l} \left\| v \left(\frac {1 + \tau}{\tau}, s\right) \right\| \leq \frac {\sinh \left(\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)\right)}{\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)} \cdot R (s) \frac {1 + \tau}{\tau} \cdot \frac {\theta \tau r}{(1 + \tau) R (s)} \tag {16} \\ = \frac {\sinh \left(\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)\right)}{\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)} \cdot \theta r. \\ \end{array}
$$
Now we can bound  $\theta$ . Note that
$$
\begin{array}{l} R (s) = d (\Gamma (0, s), \Gamma (1, s)) = d (\mathbf {y}, \xi_ {\tau} (s)) \\ \leq d (\mathbf {y}, \mathbf {y} ^ {\prime}) + d (\mathbf {y} ^ {\prime}, \mathbf {z} ^ {\prime}) \leq \frac {\tau}{1 + \tau} R + \frac {\theta \tau r}{1 + \tau} \leq \frac {\tau}{1 + \tau} (R + r). \\ \end{array}
$$
By Lemma 21, we have
$$
\frac {\sinh \left(\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)\right)}{\sqrt {- \kappa} R (s) \left(\frac {1 + \tau}{\tau}\right)} \leq \frac {\sinh \left(\sqrt {- \kappa} (R + r)\right)}{\sqrt {- \kappa} (R + r)}.
$$
Thus we can take
$$
\theta = \frac {\sqrt {- \kappa} (R + r)}{\sinh (\sqrt {- \kappa} (R + r))}
$$
to ensure  $\left\| v\left(\frac{1 + \tau}{\tau},s\right)\right\| \leq r$ . The length of the curve  $v\left(\frac{1 + \tau}{\tau},s\right)$  can be bounded by
$$
\int_ {0} ^ {1} \left\| v \left(\frac {1 + \tau}{\tau}\right) \right\| d s \leq r.
$$
This implies  $d(\mathbf{p},\mathbf{w})\leq r$  and thus  $\mathbf{w}\in \mathcal{K}$ . Then we know  $\mathbf{z}'\in \mathcal{K}$  because  $\mathbf{z}'$  lies on the geodesic connecting  $\mathbf{w}$  and  $\mathbf{y}$ , with both endpoints in  $\mathcal{K}$ . This finally leads to the fact that  $\mathbf{z}\in (1 + \tau)\mathcal{K}$ .
Now we give the proof of Theorem 2.
Proof of Theorem 2. Algorithm 4 always plays feasible points because by
$$
\delta^ {\prime} = \frac {\sqrt {- \kappa} (R + r)}{\sinh (\sqrt {- \kappa} (R + r))} \tau r
$$
and Lemma 7, we have  $\mathbf{z}_t\in (1 + \tau)\mathcal{K}$  and  $\tilde{\mathbf{z}}_t\in \mathcal{K}$ .
By the first item of Lemma 23,
$$
\mathbb {E} \left[ \left\langle \mathbf {g} _ {t}, \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] = \mathbb {E} \left[ \left\langle \mathbb {E} \left[ \mathbf {g} _ {t} | \tilde {\mathbf {y}} _ {t} \right], \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] = \frac {V _ {\delta^ {\prime}}}{S _ {\delta^ {\prime}}} \mathbb {E} \left[ \left\langle \nabla f _ {t} (\tilde {\mathbf {y}} _ {t}), \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] \tag {17}
$$
where  $\hat{f}_t(\mathbf{x})$  is a smoothed version of  $f_{t}(\mathbf{x})$ . More specifically,
$$
\hat {f} _ {t} (\mathbf {x}) := \frac {1}{V _ {\delta^ {\prime}}} \int_ {\mathbb {B} _ {\mathbf {x}} (\delta^ {\prime})} f _ {t} (\mathbf {u}) \omega
$$
where  $\omega$  is the volume form.
Applying Lemma 4 to Algorithm 4, we know  $\tilde{\mathbf{y}}_t$  is an infeasible projection of  $\mathbf{y}_t$  over  $(1 - \delta)\mathcal{K}$  which means
$$
d \left(\tilde {\mathbf {y}} _ {t + 1}, \mathbf {x}\right) ^ {2} \leq d \left(\mathbf {y} _ {t + 1}, \mathbf {x}\right) ^ {2} \leq d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {x}\right) ^ {2} + \zeta \eta^ {2} \left\| \mathbf {g} _ {t} \right\| ^ {2} - 2 \eta \left\langle \mathbf {g} _ {t}, - \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \tag {18}
$$
holds for any  $\mathbf{x} \in (1 - \delta)\mathcal{K}$ , where the second inequality is due to Lemma 29.
Equation (18) is equivalent to
$$
\left\langle \mathbf {g} _ {t}, - \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} _ {t} \right\rangle \leq \frac {d \left(\tilde {\mathbf {y}} _ {t} , \mathbf {x}\right) ^ {2} - d \left(\tilde {\mathbf {y}} _ {t + 1} , \mathbf {x}\right) ^ {2}}{2 \eta} + \frac {\eta \zeta}{2} \| \mathbf {g} _ {t} \| ^ {2}. \tag {19}
$$
By the third item of Lemma 23, we have
$$
\begin{array}{l} \hat {f} (\tilde {\mathbf {y}} _ {t}) - \hat {f} (\mathbf {x}) \leq \left\langle \nabla \hat {f} (\tilde {\mathbf {y}} _ {t}), - \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle + 2 \delta^ {\prime} \rho G \\ = \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \left\langle \mathbf {g} _ {t}, - \operatorname {E x p} _ {\tilde {\mathbf {y}} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] + 2 \delta^ {\prime} \rho G \tag {20} \\ \leq \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \cdot \frac {d (\tilde {\mathbf {y}} _ {t} , \mathbf {x}) ^ {2} - d (\tilde {\mathbf {y}} _ {t + 1} , \mathbf {x}) ^ {2}}{2 \eta} + \frac {\eta \zeta}{2} \cdot \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \| \mathbf {g} _ {t} \| ^ {2} \right] + 2 \delta^ {\prime} \rho G. \\ \end{array}
$$
where  $\rho$  is a constant solely depends on  $\kappa$ .
Combining Equations (17), (19), and (20), we have
$$
\mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(\hat {f} _ {t} (\tilde {\mathbf {y}} _ {t}) - \hat {f} _ {t} (\mathbf {x})\right) \right] \leq \frac {2 R ^ {2}}{\eta} \cdot \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} + \frac {\eta \zeta S _ {\delta^ {\prime}}}{2 V _ {\delta^ {\prime}}} \sum_ {t = s} ^ {e} \mathbb {E} \left[ \| \mathbf {g} _ {t} \| ^ {2} \right] + 2 \delta^ {\prime} \rho G T \tag {21}
$$
holds for any  $\mathbf{x} \in (1 - \delta) \mathcal{K}$ , where we use the fact that  $d(\tilde{\mathbf{y}}_s, \mathbf{x}) \leq 2R$ . We also need to bound  $\mathbb{E}\left[f_t(\widetilde{\mathbf{z}}_t) - \hat{f}_t(\tilde{\mathbf{y}}_t)\right]$  and  $\mathbb{E}\left[\hat{f}_t(\tilde{\mathbf{x}}_I) - f_t(\mathbf{x}_I)\right]$ . For the former term,
$$
\begin{array}{l} \mathbb {E} \left[ f _ {t} \left(\widetilde {\mathbf {z}} _ {t}\right) - \hat {f} _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) \right] = \mathbb {E} \left[ f _ {t} \left(\widetilde {\mathbf {z}} _ {t}\right) - f _ {t} \left(\mathbf {z} _ {t}\right) \right] + \mathbb {E} \left[ f _ {t} \left(\mathbf {z} _ {t}\right) - f _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) \right] + \mathbb {E} \left[ f _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) - \hat {f} _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) \right] \tag {22} \\ \leq G \cdot d \left(\tilde {\mathbf {z}} _ {t}, \mathbf {z} _ {t}\right) + G \delta^ {\prime} + G \delta^ {\prime} \leq G \tau R + 2 G \delta^ {\prime}, \\ \end{array}
$$
where we use Lemma 22 and the gradient Lipschitzness. Similarly, for the latter term,
$$
\begin{array}{l} \mathbb {E} \left[ \hat {f} _ {t} \left(\tilde {\mathbf {x}} _ {t}\right) - f _ {t} \left(\mathbf {x} _ {I}\right) \right] = \mathbb {E} \left[ \hat {f} _ {t} \left(\tilde {\mathbf {x}} _ {I}\right) - f _ {t} \left(\tilde {\mathbf {x}} _ {I}\right) \right] + \mathbb {E} \left[ f _ {t} \left(\tilde {\mathbf {x}} _ {I}\right) - f _ {t} \left(\mathbf {x} _ {I}\right) \right] \tag {23} \\ \leq G \delta^ {\prime} + G \cdot d (\tilde {\mathbf {x}} _ {I}, \mathbf {x} _ {I}) \leq G \delta^ {\prime} + G \delta R. \\ \end{array}
$$
By the second term of Lemma 23, we have
$$
\frac {S _ {\delta^ {\prime}} ^ {2}}{V _ {\delta^ {\prime}} ^ {2}} \| \mathbf {g} _ {t} \| ^ {2} \leq \left(\frac {n}{\delta^ {\prime}} + n | \kappa |\right) ^ {2} M ^ {2}. \tag {24}
$$
Combining Equations (21), (22), (23) and (24), summing from  $t = s$  to  $e$ , we get
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(f _ {t} \left(\widetilde {\mathbf {z}} _ {t}\right) - f _ {t} (\mathbf {x} _ {I})\right) \right] \\ = \mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(f _ {t} \left(\widetilde {\mathbf {z}} _ {t}\right) - \hat {f} _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right)\right) \right] + \mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(\hat {f} _ {t} \left(\widetilde {\mathbf {y}} _ {t}\right) - \hat {f} _ {t} \left(\mathbf {x} _ {I}\right)\right) \right] + \mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(\hat {f} _ {t} \left(\mathbf {x} _ {I}\right) - f _ {t} \left(\mathbf {x} _ {I}\right)\right) \right] \\ \leq \left(G \tau R + 2 G \delta^ {\prime}\right) T + \left(\frac {2 R ^ {2}}{\eta} \cdot \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} + \frac {\eta \zeta M ^ {2}}{2} \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} T + 2 \rho \delta^ {\prime} G T\right) + \left(G \delta^ {\prime} + G \delta R\right) T \\ \leq \left(G \tau R + 2 G \delta^ {\prime}\right) T + \left(\frac {2 R ^ {2}}{\eta} \cdot \left(\frac {n}{\delta^ {\prime}} + n | \kappa |\right) + \frac {\eta \zeta M ^ {2}}{2} \left(\frac {n}{\delta^ {\prime}} + n | \kappa |\right) T + 2 \rho \delta^ {\prime} G T\right) + \left(G \delta^ {\prime} + G \delta R\right) T. \tag {25} \\ \end{array}
$$
After plugging in  $\eta = T^{-\frac{1}{2}}$ ,  $\delta = T^{-\frac{1}{4}}$ ,  $\tau = T^{-\frac{1}{4}}$  and  $\delta' = \frac{\sqrt{-\kappa}(R + r)}{\sinh(\sqrt{-\kappa}(R + r))} T^{-\frac{1}{4}}r$ , we see  $\delta' \leq T^{-\frac{1}{4}}r$  and
$$
\frac {1}{\delta^ {\prime}} \leq \frac {\sinh \left(\sqrt {- \kappa} (R + r)\right)}{\sqrt {- \kappa} (R + r)} \cdot \frac {T ^ {\frac {1}{4}}}{r}.
$$
$$
\mathbb {E} \left[ \sum_ {t = s} ^ {e} \left(f _ {t} (\widetilde {\mathbf {z}} _ {t}) - f _ {t} (\mathbf {x} _ {I})\right) \right] = O \left(T ^ {\frac {3}{4}}\right)
$$
Denote  $\mathbf{x}_t^* \in \operatorname{argmin}_{\mathbf{x} \in (1 - \delta)K} d(\mathbf{x}, \tilde{\mathbf{y}}_t)$ . We now bound the number of calls to the separation oracle. Due to  $\mathbf{y}_{t+1} = \mathrm{Exp}_{\tilde{\mathbf{y}}_t}(-\eta \mathbf{g}_t)$ ,
$$
d \left(\mathbf {y} _ {t + 1}, (1 - \delta) \mathcal {K}\right) \leq d \left(\mathbf {x} _ {t} ^ {*}, \mathbf {y} _ {t + 1}\right) \leq d \left(\mathbf {x} _ {t} ^ {*}, \tilde {\mathbf {y}} _ {t}\right) + d \left(\tilde {\mathbf {y}} _ {t}, \mathbf {y} _ {t + 1}\right) \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) + \eta \| \mathbf {g} _ {t} \|.
$$
Squaring both sides, we have
$$
\begin{array}{l} d \left(\mathbf {y} _ {t + 1}, (1 - \delta) \mathcal {K}\right) ^ {2} \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) ^ {2} + 2 d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) \eta M + \eta^ {2} M ^ {2} \tag {26} \\ \leq d \left(\tilde {\mathbf {y}} _ {t}, (1 - \delta) \mathcal {K}\right) ^ {2} + 2 \delta R \eta M + \eta^ {2} M ^ {2}, \\ \end{array}
$$
$d(\tilde{\mathbf{y}}_t,(1 - \delta)\mathcal{K})\leq d(\tilde{\mathbf{y}}_t,\mathrm{Exp}_{\mathbf{p}}((1 - \delta)\mathrm{Exp}_{\mathbf{p}}^{-1}\tilde{\mathbf{y}}_t)) = d(\tilde{\mathbf{y}}_t,\mathrm{Exp}_{\tilde{\mathbf{y}}_t}(\delta \mathrm{Exp}_{\tilde{\mathbf{y}}_t}^{-1}\mathbf{p})) = \delta d(\tilde{\mathbf{y}}_t,\mathbf{p})\leq \delta R$  and  $\| \mathbf{g}_t\| = |f_t(\mathbf{z}_t)|\leq M$ . Combining Equation (26) with Lemma 4, we can bound the number of separation oracle calls as:
$$
\begin{array}{l} N _ {c a l l s} \leq \sum_ {t = 1} ^ {T} \left(\zeta \frac {d (\mathbf {y} _ {t + 1} , (1 - \delta) \mathcal {K}) ^ {2} - d (\tilde {\mathbf {y}} _ {t + 1} , (1 - \delta) \mathcal {K}) ^ {2}}{\delta^ {2} \bar {r} ^ {2}} + 1\right) \\ \leq \sum_ {t = 1} ^ {T} \left(\zeta \frac {d (\tilde {\mathbf {y}} _ {t} , (1 - \delta) \mathcal {K}) ^ {2} + 2 R \delta \eta M + \eta^ {2} M ^ {2} - d (\tilde {\mathbf {y}} _ {t + 1} , (1 - \delta) \mathcal {K}) ^ {2}}{\delta^ {2} \bar {r} ^ {2}} + 1\right) \\ \leq \frac {2 R M \zeta \eta T}{\bar {r} ^ {2} \delta} + \frac {M ^ {2} \zeta \eta^ {2} T}{\bar {r} ^ {2} \delta^ {2}} + T \tag {27} \\ = \frac {2 R M \zeta T}{\bar {r} ^ {2}} \cdot \frac {\eta}{\delta} + \frac {M ^ {2} \zeta T}{\bar {r} ^ {2}} \cdot \frac {\eta^ {2}}{\delta^ {2}} + T \\ = T + \frac {2 R M \zeta}{\bar {r} ^ {2}} \cdot T ^ {\frac {3}{4}} + \frac {M ^ {2} \zeta}{\bar {r} ^ {2}} \cdot T ^ {\frac {1}{2}} \\ \end{array}
$$
where the last inequality follows from  $\tilde{\mathbf{y}}_1 = \mathbf{p} \in (1 - \delta)\mathcal{K}$  and thus  $d(\tilde{\mathbf{y}}_1, (1 - \delta)\mathcal{K}) = 0$ .
Proof. We maintain the loop invariant  $\mathbf{x}_t \in \beta \mathcal{K}$ . So by Lemma 19,  $\mathbb{B}(\mathbf{x}_t, \delta') \subseteq \mathcal{K}$ , which means  $\mathbf{z}_t$  and  $\mathbf{z}_t'$  are feasible points.
For the two-point feedback setting, by Lemma 23,
$$
\begin{array}{l} \hat {f} _ {t} (\mathbf {x} _ {t}) - \hat {f} _ {t} (\mathbf {x}) \leq \left\langle \nabla \hat {f} _ {t} (\mathbf {x} _ {t}), - \operatorname {E x p} _ {\mathbf {x} _ {t}} ^ {- 1} \mathbf {x} \right\rangle + 2 \delta^ {\prime} \rho G \\ = \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \left\langle \mathbf {g} _ {t}, - \operatorname {E x p} _ {\mathbf {x} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] + 2 \delta^ {\prime} \rho G \tag {28} \\ = \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \left\langle \mathbf {g} _ {t}, - \Gamma_ {\mathbf {y} _ {t}} ^ {\mathbf {x} _ {t}} \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} \right\rangle + \left\langle \mathbf {g} _ {t}, \Gamma_ {\mathbf {y} _ {t}} ^ {\mathbf {x} _ {t}} \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} - \operatorname {E x p} _ {\mathbf {x} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] + 2 \delta^ {\prime} \rho G. \\ \end{array}
$$
By Lemma 17 of Wang et al. (2023),
$$
\frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \| \mathbf {g} _ {t} \| \right] \leq n G \left(1 + | \kappa | \delta^ {\prime 2}\right).
$$
$$
\begin{array}{l} \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \left\langle \mathbf {g} _ {t}, \Gamma_ {\mathbf {y} _ {t}} ^ {\mathbf {x} _ {t}} \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} - \operatorname {E x p} _ {\mathbf {x} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \right] \leq \frac {S _ {\delta^ {\prime}}}{V _ {\delta^ {\prime}}} \mathbb {E} \left[ \| \mathbf {g} _ {t} \| \right] \cdot \zeta d (\mathbf {x} _ {t}, \mathbf {y} _ {t}) \\ \leq n G \left(1 + | \kappa | \delta^ {\prime 2}\right) \cdot \zeta (1 - \beta) d \left(\mathbf {y} _ {t}, \mathbf {p}\right) \leq n G R \zeta (1 - \beta) \left(1 + | \kappa | \delta^ {\prime 2}\right) \tag {29} \\ = n G R \zeta \cdot \frac {\sinh (\sqrt {- \kappa} (R + r))}{\sqrt {- \kappa} (R + r)} \cdot \frac {\delta^ {\prime}}{r} \left(1 + \| \kappa | \delta^ {\prime 2}\right) = O (\delta^ {\prime}), \\ \end{array}
$$
where we use the  $\zeta$ -smoothness of  $\frac{1}{2} d(\mathbf{x},\mathbf{y})^2$  and the definition of  $\delta'$ .
Since  $\mathbf{y}_{t + 1}$  is an infeasible projection of  $\mathbf{y}_{t + 1}^{\prime}$  onto  $(1 - \delta)\mathcal{K}$ ,
$$
\begin{array}{l} d \left(\mathbf {y} _ {t + 1}, \mathbf {x}\right) ^ {2} \leq d \left(\mathbf {y} _ {t + 1} ^ {\prime}, \mathbf {x}\right) ^ {2} \leq d \left(\mathbf {y} _ {t}, \mathbf {x}\right) ^ {2} + \zeta d \left(\mathbf {y} _ {t}, \mathbf {y} _ {t + 1} ^ {\prime}\right) ^ {2} - 2 \left\langle \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {y} _ {t + 1} ^ {\prime}, \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \\ = d \left(\mathbf {y} _ {t}, \mathbf {x}\right) ^ {2} + \zeta \eta^ {2} \| \mathbf {g} _ {t} \| ^ {2} - 2 \left\langle - \eta \Gamma_ {\mathbf {x} _ {t}} ^ {\mathbf {y} _ {t}} \mathbf {g} _ {t}, \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \\ = d \left(\mathbf {y} _ {t}, \mathbf {x}\right) ^ {2} + \zeta \eta^ {2} \| \mathbf {g} _ {t} \| ^ {2} - 2 \eta \left\langle \mathbf {g} _ {t}, - \Gamma_ {\mathbf {y} _ {t}} ^ {\mathbf {x} _ {t}} \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \\ \end{array}
$$
holds for any  $\mathbf{x}\in (1 - \delta)\mathcal{K}$ , which means
$$
\left\langle \mathbf {g} _ {t}, - \Gamma_ {\mathbf {y} _ {t}} ^ {\mathbf {x} _ {t}} \operatorname {E x p} _ {\mathbf {y} _ {t}} ^ {- 1} \mathbf {x} \right\rangle \leq \frac {d \left(\mathbf {y} _ {t} , \mathbf {x}\right) ^ {2} - d \left(\mathbf {y} _ {t + 1} , \mathbf {x}\right) ^ {2}}{2 \eta} + \frac {\eta \zeta \| \mathbf {g} _ {t} \| ^ {2}}{2} \tag {30}
$$
Taking expectation and using  $\frac{S_{\delta'}}{V_{\delta'}} \leq \frac{n}{\delta'} + n|\kappa|$ , we have