SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Jacob Abernethy, Kevin A Lai, Kfir Y Levy, and Jun-Kun Wang. Faster rates for convex-concave games. In Conference On Learning Theory, pages 1595-1625. PMLR, 2018.
[2] Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. 2009.
[3] Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050-2057, 2004.
[4] Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM), 68(1):1-39, 2021.
[5] Thomas M Cover. Universal portfolios. Mathematical Finance, 1(1):1-29, 1991.
[6] Ashok Cutkosky. Anytime online-to-batch, optimism and acceleration. In International Conference on Machine Learning, pages 1446-1454. PMLR, 2019.
[7] Ashok Cutkosky, Harsh Mehta, and Francesco Orabona. Optimal stochastic nonsmooth non-convex optimization through online-to-non-convex conversion. arXiv preprint arXiv:2302.03775, 2023.
[8] Maryam Fazel, Yin Tat Lee, Swati Padmanabhan, and Aaron Sidford. Computing lewis weights to high precision. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 2723-2742. SIAM, 2022.
[9] Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms, pages 385-394, 2005.
[10] Dan Garber and Elad Hazan. A linearly convergent variant of the conditional gradient algorithm under strong convexity, with applications to online and stochastic optimization. SIAM Journal on Optimization, 26(3):1493-1528, 2016.
[11] Dan Garber and Ben Kretzu. New projection-free algorithms for online convex optimization with adaptive regret guarantees. In Conference on Learning Theory, pages 2326-2359. PMLR, 2022.
[12] JL Goffin. Affine and projective transformations in nondifferentiable optimization. Trends in Mathematical Optimization, pages 79-91, 1988.
[13] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2-3):169-192, 2007.
[14] Elad Hazan and Satyen Kale. Projection-free online learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1843-1850, 2012.
[15] Elad Hazan and Edgar Minasyan. Faster projection-free online learning. In Conference on Learning Theory, pages 1877-1893. PMLR, 2020.
[16] Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning, pages 427-435. PMLR, 2013.
[17] Rémi Jézquel, Dmitrii M Ostrovskii, and Pierre Gaillard. Efficient and near-optimal online portfolio selection. arXiv preprint arXiv:2209.13932, 2022.
[18] Ben Kretzu and Dan Garber. Revisiting projection-free online learning: the strongly convex case. In International Conference on Artificial Intelligence and Statistics, pages 3592-3600. PMLR, 2021.
[39] Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 259-278. SIAM, 2020.
[40] Jan van den Brand, Yin Tat Lee, Aaron Sidford, and Zhao Song. Solving tall dense linear programs in nearly linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 775-788, 2020.
[41] Julian Zimmert, Naman Agarwal, and Satyen Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory, pages 182-226. PMLR, 2022.
A Self-concordance properties
Throughout, for a twice-differentiable function  $f\colon \operatorname {int}\mathcal{K}\to \mathbb{R}$ , we let  $\lambda (x,f)\coloneqq \| \nabla f(x)\|_{\nabla^{-2}f(x)}$  denote the Newton decrement of  $f$  at  $x\in \operatorname {int}\mathcal{K}$ .
Lemma 1. Let  $f \colon \operatorname{int} \mathcal{K} \to \mathbb{R}$  be a self-concordant function with constant  $M_f \geq 1$ . Further, let  $x \in \operatorname{int} \mathcal{K}$  and  $x_f \in \operatorname{argmin}_{x \in \mathcal{K}} f(x)$ . Then, I) whenever  $\lambda(x, f) < 1 / M_f$ , we have
$$
\| x - x _ {f} \| _ {\nabla^ {2} f (x _ {f})} \vee \| x - x _ {f} \| _ {\nabla^ {2} f (x)} \leq \lambda (x, f) / (1 - M _ {f} \lambda (x, f));
$$
and II) for any  $M \geq M_f$ , the Newton step  $x^+ \coloneqq x - \nabla^{-2}f(x)\nabla f(x)$  satisfies  $x^+ \in \operatorname{int}\mathcal{K}$  and  $\lambda (x^{+},f) \leq M\lambda (x,f)^{2} / (1 - M\lambda (x,f))^{2}$ .
Lemma 2. Let  $f$ : int  $\mathcal{K} \to \mathbb{R}$  be a self-concordant function with constant  $M_f$  and  $x \in \operatorname{int} \mathcal{K}$ . Then, for any  $w$  such that  $r \coloneqq \| w - x \|_{\nabla^2 f(x)} < 1 / M_f$ , we have
$$
\left(1 - M _ {f} r\right) ^ {2} \nabla^ {2} f (w) \preceq \nabla^ {2} f (x) \preceq \left(1 - M _ {f} r\right) ^ {- 2} \nabla^ {2} f (x).
$$
The following result from [33, Theorem 5.1.5] will be useful to show that the iterates of algorithms are always in the feasible set.
Lemma 3. Let  $f\colon \operatorname {int}\mathcal{K}\to \mathbb{R}$  be a self-concordant function with constant  $M_{f}\geq 1$  and  $x\in \mathrm{int}\mathcal{K}$ . Then,  $\mathcal{E}_x\coloneqq \{w\in \mathbb{R}^d:\| w - x\| _x < 1 / M_f\} \subseteq \mathrm{int}\mathcal{K}$ . Furthermore, for all  $w\in \mathcal{E}_x$ , we have
$$
\left\| w - x \right\| _ {w} \leq \frac {\left\| w - x \right\| _ {x}}{1 - M _ {f} \left\| w - x \right\| _ {x}}.
$$
Finally, we will also make use of the following result due to [31]:
Lemma 4. Let  $f \colon \operatorname{int} \mathcal{K} \to \mathbb{R}$  be a self-concordant function with constant  $M_f > 0$ . Then, for any  $x, w \in \operatorname{int} \mathcal{K}$  such that  $r \coloneqq \|x - w\|_{\nabla^2 f(x)} < 1 / M_f$ , we have
$$
\left\| \nabla f (x) - \nabla f (w) \right\| _ {\nabla^ {- 2} f (x)} ^ {2} \leq \frac {1}{(1 - M _ {f} r) ^ {2}} \| w - x \| _ {\nabla^ {2} f (x)} ^ {2}.
$$
Our analysis relies on the crucial fact that the Newton decrement can be sufficiently decreased by taking a Newton step using only approximate gradients and Hessians. We state this fact next; the proof is in §D.1.
Lemma 5 (Decrease in the Newton decrement). Let  $\Phi$  be a self-concordant function over int  $\mathcal{K}$  with constant  $M_{\Phi} > 0$ , and let  $y \in \mathbb{R}^d$  be such that  $\lambda(y, \Phi) \leq 1/(40M_{\Phi})$ . Further, let  $H \in \mathbb{R}^{d \times d}$  and  $\widehat{\nabla}_y \in \mathbb{R}^d$  be such that
$$
\left\| \widehat {\nabla} _ {y} - \nabla \Phi (y) \right\| _ {\nabla^ {- 2} \Phi (y)} \leq \varepsilon <   \frac {1}{4 0 M _ {\Phi}}, \tag {20}
$$
$$
(1 - \alpha) \nabla^ {2} \Phi (y) \leqslant H \leqslant (1 + \alpha) \nabla^ {2} \Phi (y), \tag {21}
$$
for  $\alpha < 1 / 5$ . Then, for  $\tilde{y}^{+}\coloneqq y - H^{-1}\nabla \Phi (y)$  and  $y^{+}\coloneqq y - H^{-1}\widehat{\nabla}_{y}$ , we have
$$
\lambda \left(\tilde {y} ^ {+}, \Phi\right) \leq 9 M _ {\Phi} \lambda \left(y, \Phi\right) ^ {2} + 2. 5 \alpha \lambda (y, \Phi),
$$
$$
\lambda \left(y ^ {+}, \Phi\right) \leq 2 0 (1 + \alpha) \varepsilon + (1 + 2 0 (1 + \alpha) \varepsilon) \cdot \lambda \left(\tilde {y} ^ {+}, \Phi\right).
$$
Next, we show that as long as the Newton decrement is small enough at the current iterate  $w_{t-1}$ , the "intermediate" Newton iterates  $(w_t^m)$  remain close to the landmark point  $u_{t-1}$ ; this will be important for the proof of Theorem 5. The proof is in §D.2.
Lemma 6 (Invariance under Newton iterations). Let  $\Phi$  be a self-concordant function over int  $\mathcal{K}$  with constant  $M_{\Phi} > 0$ . Let  $b > 0$ ,  $m_{\mathrm{Newton}} \coloneqq \Theta \left( \log \frac{1}{\varepsilon M_{\Phi}} \right)$ ,  $\alpha \leq 1 / (1000M_{\Phi})$ ,  $\varepsilon < 1 / (20000M_{\Phi})$ ,  $\alpha = 0.001$ , and  $\eta \leq 1 / (1000M_{\Phi}b)$ . Further, let  $(w_t, w_t^m, u_t, g_t, H_t)$  be as in Algorithm 1 with input  $(\eta, \varepsilon, \alpha, m_{\mathrm{Newton}})$ . Suppose that at round  $t - 1$  of Algorithm 1, we have
$$
\lambda \left(w _ {t - 1}, \Phi_ {t - 1}\right) \leq \alpha \quad a n d \quad \left\| u _ {t - 1} - w _ {t - 1} \right\| _ {\nabla^ {2} \Phi \left(u _ {t - 1}\right)} \leq \frac {1}{4 0 M _ {\Phi}}. \tag {22}
$$
For  $t > 1$ , if the sub-gradient  $g_{t - 1}$  at round  $t - 1$  satisfies  $\| g_{t - 1}\|_{\nabla^{-2}\Phi (w_{t - 1})}\leq b$ , then
$$
\lambda \left(w _ {t} ^ {m}, \Phi_ {t}\right) \leq \left(\frac {1 5}{1 6}\right) ^ {m - 1} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) + 5 0 0 \varepsilon \leq \frac {1}{4 0 M _ {\Phi}}. \tag {23}
$$
Furthermore, we have for all  $m \in [m_{\mathrm{Newton}}]$ :
$$
\frac {1}{2} \nabla^ {2} \Phi \left(w _ {t} ^ {m}\right) \leqslant \nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right) \leqslant 2 \nabla^ {2} \Phi \left(w _ {t} ^ {m}\right), \tag {24}
$$
$$
\left\| w _ {t} ^ {m} - u _ {t - 1} \right\| _ {H _ {t - 1}} \leq \frac {1}{1 0 M _ {\Phi}}, \tag {25}
$$
$$
\left\| w _ {t} ^ {m} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \leq \frac {1}{4 9 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m - 1} + 2 4 0 \varepsilon , \tag {26}
$$
$$
\left| \left\| w _ {t} ^ {m} - u _ {t - 1} \right\| _ {H _ {t - 1}} - \left\| w _ {t - 1} - u _ {t - 1} \right\| _ {H _ {t - 1}} \right| \leq 2 \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {2} \Phi \left(w _ {t - 1}\right)} + \frac {1}{4 0 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m - 1} + 5 0 0 \varepsilon + 2 \alpha , \tag {27}
$$
where  $w_{t}^{\star}\in \mathrm{argmin}_{w\in \mathcal{K}}\Phi_{t}(w)$  is the optimum solution of  $\Phi_t$
We note that we have not made an attempt to optimize over the constants in Lemma 6.
Lemma 7. Let  $\Phi$  be a  $(M_{\Phi},\nu)$ -self-concordant barrier for  $\mathcal{K}$ , and let  $w^{\star} \in \mathrm{argmin}_{w \in \mathcal{K}} \Phi(w)$ . Further, suppose that the losses satisfy:
$$
\sup  _ {w \in \operatorname {i n t} \mathcal {K}, t \in [ T ]} \ell_ {t} \left(\left(1 - \frac {1}{T}\right) \cdot w + \frac {1}{T} \cdot w ^ {\star}\right) - \ell (w) \leq O \left(\frac {1}{\sqrt {T}}\right), \tag {28}
$$
Then, for any  $w \in \mathcal{K}$ , there exists  $\tilde{w} \in \mathcal{K}_{1 / T}$  (where  $\mathcal{K}_c$  is as in (4)) such that
$$
\sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (w)\right) \leq \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (\tilde {w}) + O (\sqrt {T}) \right. \tag {29}
$$
Proof. Fix  $w \in \mathcal{K}$  and define  $\tilde{w} = \frac{1}{T} w^{\star} + \left(1 - \frac{1}{T}\right) w \in \mathcal{K}_{1 / T}$ . Then, by (28), we have, for all  $t \in [T]$ ,
$$
\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (w) \leq \ell_ {t} (\tilde {w} _ {t}) - \ell_ {t} (\tilde {w}) + O \left(T ^ {- 1 / 2}\right). \tag {30}
$$
Summing this over  $t = 1,\dots ,T$  leads to the desired result.
C Proofs of the Main Results
Next, we present the proof of Theorem 5.
Proof. The proof consists of three parts: I) First, we show that BARONS keeps the Newton decrements  $\lambda(w_t, \Phi_t), t \geq 1$ , small—this is the main invariant of BARONS; II) Then, we bound the regret of BARONS using this invariant and the results of Lemma 6; III) Finally, we bound the runtime of BARONS.
Bounding the Newton decrements. We will show that the Newton decrements satisfy
$$
\lambda \left(w _ {s}, \Phi_ {s}\right) \leq \alpha := \min  \left\{\frac {1}{1 0 0 0 M _ {\Phi}}, 1 0 0 0 \varepsilon \right\}, \tag {31}
$$
for all  $s \geq 1$ . We will show (31) by induction over  $t \geq 1$ .
Base case. The base case follows by the facts that  $w_{1} \in \operatorname{argmin}_{w \in \mathcal{K}} \Phi(w)$ ,  $\Phi_{1} \equiv \Phi$  and that the Newton decrement is zero at the minimizer.
Induction step. Suppose that (31) holds with  $s = t - 1$  for some  $t \geq 1$ . We will show that it holds for  $s = t$ . First, note that by the update rule of landmark (see Lines 12 and 17 of Alg. 1), we have that
$$
\left\| w _ {t - 1} - u _ {t - 1} \right\| _ {H _ {t - 1}} \leq \frac {1}{4 1 M _ {\Phi}},
$$
where  $H_{t - 1} = \mathcal{O}_{\alpha}^{\mathrm{hess}}(u_{t - 1})$ . Thus, by the choice of  $\alpha$  in the theorem's statement, we have
$$
\left\| u _ {t - 1} - w _ {t - 1} \right\| _ {\nabla^ {2} \Phi \left(u _ {t - 1}\right)} \leq \frac {1}{4 0 M _ {\Phi}}. \tag {32}
$$
This, combined with the fact that (31) holds with  $s = t - 1$  (the induction hypothesis) implies that the conditions of Lemma 6 are satisfied. This in turn implies
$$
\lambda \big (w _ {t}, \Phi_ {t} \big) \stackrel {(a)} {=} \lambda \big (w _ {t} ^ {m _ {\mathrm {N e w t o n}}}, \Phi_ {t} \big) \leq \left(\frac {1 5}{1 6}\right) ^ {m _ {\mathrm {N e w t o n}}} \lambda \big (w _ {t} ^ {1}, \Phi_ {t} \big) + 5 0 0 \varepsilon \leq \frac {5 0}{M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m _ {\mathrm {N e w t o n}}} + 5 0 0 \varepsilon \stackrel {(b)} {\leq} \alpha ,
$$
where  $(a)$  follows by the fact that  $w_{t} = w_{t}^{m_{\mathrm{Newton}}}$  (by definition; see Algorithm 1) and  $(b)$  follows by the choice of  $m_{\mathrm{Newton}}$  in the theorem's statement. This shows that (31) holds for  $s = t$  and concludes the induction.
Bounding the regret. To bound the regret of BARONS, we make use of the FTRL iterates  $\{w_t^\star\}$ , which are given by  $w_t^\star \in \mathrm{argmin}_{w \in \mathcal{K}} \Phi_t(w)$ : By Lemma 6, we have that for all  $t \in [T]$
$$
\left\| w _ {t} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} = \left\| w _ {t} ^ {m _ {\text {N e w t o n}}} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \leq \frac {1}{4 9 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m _ {\text {N e w t o n}}} + 2 4 0 \varepsilon = O (\varepsilon), \tag {33}
$$
where the last inequality follows by the choice of  $m_{\mathrm{Newton}} = \Theta (\log (1 / (\varepsilon M_{\Phi})))$  in the theorem's statement. Using this and Hölder's inequality, we now bound the sum of linearized losses of the algorithm in terms of the sum of linearized losses with respect to  $\{w_t^*\}$ :
$$
\sum_ {t = 1} ^ {T} \left\langle w _ {t}, g _ {t} \right\rangle \leq \sum_ {t = 1} ^ {T} \left\langle w _ {t} ^ {\star}, g _ {t} \right\rangle + \sum_ {t = 1} ^ {T} \left\| w _ {t} ^ {\star} - w _ {t} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \cdot \left\| g _ {t} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right) ^ {- 1}}. \tag {34}
$$
Now, by (24) in Lemma 6 (which holds due to (32) and (31) with  $s = t - 1$  as we showed in the prequel), we have  $\frac{1}{2}\nabla^{2}\Phi (w_{t})\preccurlyeq \nabla^{2}\Phi (w_{t}^{\star})$ , which implies that  $\| g_t\|_{\nabla^{-2}\Phi (w_t^\star)}\leq 2\| g_t\|_{\nabla^{-2}\Phi (w_t)}$ . Combining this with (33) and (34), we get that
$$
\sum_ {t = 1} ^ {T} \left\langle w _ {t}, g _ {t} \right\rangle \leq \sum_ {t = 1} ^ {T} \left\langle w _ {t} ^ {\star}, g _ {t} \right\rangle + O (\varepsilon) \sum_ {t = 1} ^ {T} \left\| g _ {t} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t}\right)}. \tag {35}
$$
Now fix  $w \in \mathcal{K}$ . Subtracting  $\sum_{t=1}^{T} \langle w, g_t \rangle$  from both sides of (35) implies the following bound the regret of BARONS:
$$
\begin{array}{l} \operatorname {R e g} _ {T} ^ {\text {B A R O N S}} (w) \leq \sum_ {t = 1} ^ {T} \left\langle w _ {t} ^ {\star}, g _ {t} \right\rangle - \sum_ {t = 1} ^ {T} \left\langle w, g _ {t} \right\rangle + O (\varepsilon) \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}, \\ \leq \frac {1}{\eta} \Phi (w) + \eta \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})} ^ {2} + O (\varepsilon) \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}, \\ \end{array}
$$
where the last inequality follows by the regret bound of FTRL (see e.g. [30]).
Bounding the run-time. Note that BARONS updates the landmark points on the rounds where  $\| u_t - w_t\|_{H_t} > 1 / (41M_\Phi)$ . Now, by (27) in Lemma 6 (which holds due to (32) and (31) with  $s = t - 1$  as we showed in the prequel), we have
$$
\begin{array}{l} \left| \left\| u _ {t} - w _ {t} \right\| _ {H _ {t}} - \left\| u _ {t} - w _ {t - 1} \right\| _ {H _ {t}} \right| = \left| \left\| u _ {t} - w _ {t} ^ {m _ {\mathrm {N e w t o n}}} \right\| _ {H _ {t}} - \left\| u _ {t} - w _ {t - 1} \right\| _ {H _ {t}} \right|, \\ \leq 2 \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)} + \frac {1}{4 0 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m _ {\text {N e w t o n}}} + 5 0 0 \varepsilon + 2 \alpha \\ \leq 2 \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)} + O (\varepsilon), \\ \end{array}
$$
where the last inequality follows by the choice of  $m_{\mathrm{Newton}}$  in the theorem's statement. Hence, the quantity  $\| u_t - w_t\|_{H_t}$  increases each time by at most  $2\eta \| g_{t - 1}\|_{\nabla^{-2}\Phi (w_{t - 1})} + O(\varepsilon)$ . Therefore, the number of times that the landmark  $u_{t}$  changes is bounded by
$$
O \left(\frac {\sum_ {t = 1} ^ {T} \left(2 \eta \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})} + O (\varepsilon)\right)}{1 / \left(4 1 M _ {\Phi}\right)}\right) = O \left(M _ {\Phi} T \varepsilon + M _ {\Phi} \sum_ {t = 1} ^ {T} \eta \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}\right).
$$
Thus, the overall computational cost of recalculating the Hessians and their inverses at the landmark iterates is bounded by
$$
\mathcal {C} _ {\alpha} ^ {\text {h e s s}} \cdot \left(M _ {\Phi} T \varepsilon + M _ {\Phi} \sum_ {t = 1} ^ {T} \eta \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}\right).
$$
where the multiplicative cost  $\mathcal{C}_{\alpha}^{\mathrm{hess}}$  reflects the fact that the instance of BARONS in the theorem's statement needs  $1 \pm \alpha$  accurate approximations of the Hessians and their inverses (in the sense of Definition 4) at the landmark iterates. Moreover, BARONS needs to compute an  $\varepsilon$ -approximate gradient of  $\Phi$  at every point  $w_{t}^{m}$  for all  $t \in [T]$  and  $m \in [m_{\mathrm{Newton}}]$ . Thus, the cost of computing the gradients is  $\mathcal{C}_{\varepsilon}^{\mathrm{grad}} \cdot T \log \frac{1}{\varepsilon M_{\Phi}}$ . Finally, the matrix-vector product  $H_{t}^{-1} \nabla \Phi_{t}(w)$  in BARONS costs  $O(d^{2})$  work, and so overall the computational cost is
$$
O \left(\left(\mathcal {C} _ {\varepsilon} ^ {\text {g r a d}} + d ^ {2}\right) \cdot T \log \frac {1}{\varepsilon M _ {\Phi}} + \mathcal {C} _ {\alpha} ^ {\text {h e s s}} \cdot \left(M _ {\Phi} T \varepsilon + M _ {\Phi} \sum_ {t = 1} ^ {T} \eta \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}\right)\right).
$$
Proof. Note that without having any effect on the algorithm, we can add an arbitrary constant to the barrier  $\Phi$ . Thus, without loss of generality, we assume  $\Phi(w^{\star}) = 0$ , which implies  $\Phi(w) \geq 0$ , for all  $w \in \mathcal{K}$ . We define the restricted comparator class
$$
\widetilde {\mathcal {K}} := \left\{w \in \mathcal {K}: \Phi (w) \leq \Phi (w ^ {\star}) + \nu \log c \right\}.
$$
By [33, Corollary 5.3.3] and the fact that  $\Phi$  is an  $(M_{\Phi},\nu)$ -self-concordant barrier for  $\mathcal{K}$ , we have that
$$
\mathcal {K} _ {c} \subseteq \widetilde {\mathcal {K}}, \tag {36}
$$
and so it suffices to bound the regret against comparators in  $\widetilde{\mathcal{K}}$ . Fix  $\tilde{w} \in \widetilde{\mathcal{K}}$ . Under the assumptions of the theorem, the preconditions of Theorem 5 are satisfied and so we have,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (\tilde {w})\right) \lesssim \frac {1}{\eta} \Phi (\tilde {w}) + \eta \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})} ^ {2} + \varepsilon \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}, \\ = \frac {1}{\eta} \nu \log c + \eta b ^ {2} T + \varepsilon T b, \quad (\text {s i n c e} \tilde {w} \in \widetilde {\mathcal {K}} \text {a n d} \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})} \leq b) \\ = 2 b \sqrt {\nu T \log c} + b \sqrt {\nu T}, \\ \end{array}
$$
where in the last step we used the choices of  $\eta$  and  $\varepsilon$  in (5). Combining this with (36) implies the desired regret bound. The bound on the computational complexity follows immediately from Theorem 5, the fact that  $\| g_t\|_{\nabla^{-2}\Phi (\boldsymbol{w}_t)}\leq b$ , and the choices of  $\eta$  and  $\varepsilon$  in (5).
C.3 Proof of Theorem 7
Proof. Similar to the proof of Theorem 6, and without loss of generality, we assume that  $\Psi$  is zero at its minimum, i.e.  $\Psi (w^{\star}) = 0$ . We define the restricted comparator class
$$
\widetilde {\mathcal {K}} := \{w \in \mathcal {K}: \Psi (w) \leq \Psi (w ^ {\star}) + \nu \log T \}. \tag {37}
$$
By [33, Corollary 5.3.3] and the fact that  $\Psi$  is an  $(M_{\Psi},\nu)$ -self-concordant barrier for  $\mathcal{K}$ , we have that
$$
\mathcal {K} _ {1 / T} \subseteq \widetilde {\mathcal {K}}. \tag {38}
$$
On the other hand, by Lemma 7 we have that
$$
\sup  _ {w \in \operatorname {i n t} \mathcal {K}} \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (w)\right) \leq \sup  _ {\tilde {w} \in \operatorname {i n t} \mathcal {K}} \sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (\tilde {w}) + O (\sqrt {T}). \right.
$$
Combining this with (38) implies that it suffices to bound the regret against comparators in  $\widetilde{\mathcal{K}}$ . Fix  $\tilde{w} \in \widetilde{\mathcal{K}}$ . Note that since  $\Phi$  is equal to  $\Psi$  plus a quadratic,  $\Phi$  is also a self-concordant function with
constant  $M_{\Phi} = M_{\Psi}$  [33]. Thus, under the assumptions of the theorem the preconditions of Theorem 5 are satisfied and so we have,
$$
\sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (\tilde {w})\right) \lesssim \frac {1}{\eta} \Phi (\tilde {w}) + \eta \sum_ {t = 1} ^ {T} \left\| g _ {t} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t}\right)} ^ {2} + \varepsilon \sum_ {t = 1} ^ {T} \left\| g _ {t} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t}\right)}. \tag {39}
$$
Now, by the choice of  $\Phi$ , we have that
$$
\left\| g _ {t} \right\| _ {\nabla^ {- 2} \Phi (w _ {t})} \leq R \left\| g _ {t} \right\| / \sqrt {\nu} \leq R G / \sqrt {\nu}.
$$
Moreover, from the condition that  $\mathcal{K} \subseteq \mathcal{B}(R)$ , (37) and the fact that  $\Psi(w^{\star}) = 0$ , we have for all  $w \in \widetilde{\mathcal{K}}$ :
$$
\Phi (w) \leq \nu \log T + \frac {\nu}{2}.
$$
Plugging this into (39) and using that  $\tilde{w}\in \mathcal{K}$ , we get
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} (\ell_ {t} (w _ {t}) - \ell_ {t} (\tilde {w})) = \frac {1}{\eta} \nu \log T + \frac {1}{2 \eta} + \eta \frac {R ^ {2} G ^ {2}}{\nu} T + \varepsilon \frac {T R G}{\sqrt {\nu}}, \\ = 2 R G \sqrt {T \log T} + \frac {5}{2} R G \sqrt {T}, \\ \end{array}
$$
where in the last step we used the choices of  $\eta$  and  $\varepsilon$  in (8). Combining this with (38) implies the desired regret bound. The bound on the computational complexity follows from the computational complexity in Theorem 5 and the fact a gradient Oracle  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}(\Phi)$  [resp. Hessian Oracle  $\mathcal{O}_{\alpha}^{\mathrm{grad}}(\Psi)$ ] for  $\Phi (\cdot) = \Psi (\cdot) + \frac{\nu}{2R^2}\| \cdot \| ^2$  can be implemented with one call to  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}(\Psi)$  [resp.  $\mathcal{O}_{\alpha}^{\mathrm{hess}}(\Psi)$ ] plus  $d$  arithmetic operations.
D Proofs of the Technical Lemmas
D.1 Proof of Lemma 5
Proof. Throughout, we let  $h$  is the Newton step based on the exact gradient  $\nabla \Phi(y)$ :
$$
h = - H ^ {- 1} \nabla \Phi (y).
$$
Recall that  $\tilde{y}^{+}$  and  $y^{+}$  from the lemma's statement satisfy
$$
\tilde {y} ^ {+} = y + h \quad \text {a n d} \quad y ^ {+} = y - H ^ {- 1} \widehat {\nabla} _ {y}.
$$
Bounding the Newton decrement at  $\tilde{y}^{+}$ . First, we bound the Newton decrement at  $\tilde{y}^{+}$ . By definition, the square of the Newton decrement at  $\tilde{y}^{+} = y + h$  is