SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Khashayar Gatmiry  
MIT
gatmiry@mit.com
This paper presents new projection-free algorithms for Online Convex Optimization (OCO) over a convex domain  $\mathcal{K} \subset \mathbb{R}^d$ . Classical OCO algorithms (such as Online Gradient Descent) typically need to perform Euclidean projections onto the convex set  $\mathcal{K}$  to ensure feasibility of their iterates. Alternative algorithms, such as those based on the Frank-Wolfe method, swap potentially-expensive Euclidean projections onto  $\mathcal{K}$  for linear optimization over  $\mathcal{K}$ . However, such algorithms have a sub-optimal regret in OCO compared to projection-based algorithms. In this paper, we look at a third type of algorithms that output approximate Newton iterates using a self-concordant barrier for the set of interest. The use of a self-concordant barrier automatically ensures feasibility without the need of projections. However, the computation of the Newton iterates requires a matrix inverse, which can still be expensive. As our main contribution, we show how the stability of the Newton iterates can be leveraged to only compute the inverse Hessian a vanishing fractions of the rounds, leading to a new efficient projection-free OCO algorithm with a state-of-the-art regret bound.
We consider the Online Convex Optimization (OCO) problem over a convex set  $\mathcal{K} \subset \mathbb{R}^d$ , in which a learner (algorithm) plays a game against an adaptive adversary for  $T$  rounds. At each round  $t \in [T]$ , the learner picks  $w_t \in \mathcal{K}$  given knowledge of the history  $\mathcal{H}_{t-1} \coloneqq \{(\ell_s, w_s)\}_{s < t}$ . Then, the adversary picks a convex loss function  $\ell_t: \mathcal{K} \to \mathbb{R}$  with the knowledge of  $\mathcal{H}_{t-1}$  and the iterate  $w_t$ , and the learner suffers loss  $\ell_t(w_t)$  and proceeds to the next round. The goal of the learner is to minimize the regret after  $T$  rounds:
$$
\operatorname {R e g} _ {T} (w) = \sum_ {t = 1} ^ {T} \ell_ {t} (w _ {t}) - \sum_ {t = 1} ^ {T} \ell_ {t} (w),
$$
against any comparator  $w \in \mathcal{K}$ . The aim of this paper is to design computationally-efficient (projection-free) algorithms for OCO that enjoy the optimal (up to log-factor in  $T$ )  $\widetilde{O}(\sqrt{T})$  regret.
The OCO framework captures many optimization settings relevant to machine learning applications. For example, OCO algorithms can be used in offline convex optimization as more computationally- and memory-efficient alternatives to interior-point and cutting plane methods whenever the dimension  $d$  is large [14, 16]. OCO algorithms are also often used in stochastic convex optimization, where the standard  $O(\sqrt{T})$  regret (achieved by, e.g. Online Gradient Descent) translates into the optimal  $O(1 / \sqrt{T})$  rate via the classical online-to-batch conversion technique [3, 37]. It has been shown that OCO algorithms can also achieve state-of-the-art accelerated rates in both the offline and stochastic optimization settings despite being designed for the more general OCO framework [6, 28]. What is
This is the optimal rate when no further assumptions are made.
more, it has recently been shown that even non-convex (stochastic) optimization can be reduced to online linear optimization (a special case of OCO), where it is then possible to recover the best-known convergence rates for the setting [7].
Given the prevalent use of OCO algorithms in machine learning applications, it is important to have computationally-efficient algorithms that scale well with the dimension  $d$  of the ambient space. However, most OCO algorithms fall short of being efficient because of the need of performing (Euclidean) projections onto  $\mathcal{K}$  (potentially at each iteration) to ensure that the iterates are feasible. These projections are often inefficient, especially in high-dimensional settings with complex feasible sets. Existing projection-free OCO algorithms address this computational challenge by swapping potentially-expensive Euclidean projections for often much cheaper linear optimization or separation over the feasible set  $\mathcal{K}$ . However, existing projection-free algorithms have sub-optimal regret guarantees in terms of their dependence in  $T$ , or have potentially unbounded "condition numbers" for the feasible set multiplying their regret guarantee.
Contributions. In this paper, we address these computational and performance challenges by revisiting an existing (but somewhat overlooked) type of projection-free OCO algorithms. Unlike existing algorithms, our proposed method does not require linear optimization or separation over the feasible set  $\mathcal{K}$ . Instead, the algorithm, Barrier-Regularized Online Newton Step (BARONS), uses a self-concordant barrier  $\Phi$  for the set  $\mathcal{K}$  to always output iterates that are guaranteed to be within  $\mathcal{K}$ ; much like interior point methods for offline optimization. In particular, our algorithm outputs Newton iterates with respect to time-varying, translated versions of  $\Phi$ . The main novelty of our work is in devising a new efficient way of computing the Newton iterates without having to evaluate the inverse of the Hessian of the barrier at every iteration, which can be computationally expensive in high-dimensional settings. Our algorithm only needs to compute a full inverse of the Hessian a vanishing  $O(1/\sqrt{T})$  fraction of the rounds. For the rest of the rounds, the computational cost is dominated by that of evaluating the gradient of the barrier  $\Phi$ , which can be much cheaper than evaluating the inverse of its Hessian in many cases.
For the special case of a polytope with  $m$  constraints, we show that there is a choice of a barrier (e.g. the Lee-Sidford barrier) that when used within our algorithm, reduces the per-round computational cost to essentially  $\widetilde{O}(1)$  linear-system-solves of size  $m \times d$ . We show that this is often cheaper than performing linear optimization over  $\mathcal{K}$ , which other projection-free algorithms require. More importantly, our algorithm achieves a dimension-free  $\widetilde{O}(\sqrt{T})$  regret bound. This improves over the existing regret bounds of projection-free algorithms over polytopes. For example, among projection-free algorithms that achieve a  $O(\sqrt{T})$  regret, the algorithms by [28, 11, 26], which require a separation/membership Oracle for  $\mathcal{K}$ , have a multiplicative  $\kappa = R / r$  factor multiplying their regret bounds, where  $r, R > 0$  are such that  $\mathcal{B}(r) \subseteq \mathcal{K} \subseteq \mathcal{B}(R)$ . The constant  $\kappa$ , known as the asphericity [12], can in principle be arbitrarily large. Even after applying a potentially expensive pre-processing step, which would typically involve putting the set  $\mathcal{K}$  into (near-) isotropic position [9, 38],  $\kappa$  can still be as large as  $d$  in the worst-case, and so the regret bounds achieved by the algorithms of [28, 11, 26] can be of order  $O(d\sqrt{T})$ ; this is worse than ours by a  $d$  factor. Other projection-free algorithms based on the Frank-Wolfe method, e.g. those in [10, 35, 19], also have multiplicative condition numbers that are even less benign than the asphericity  $\kappa$ . In fact, the condition numbers in the regret bounds for polytopes appearing in, e.g. [10], can in principle be arbitrarily large regardless of any pre-processing.
Finally, another advantage of our algorithm is that it can guarantee a sublinear regret even for non-Lipschitz losses (i.e. where the norm of the sub-gradients may be unbounded). In particular, we show that the general guarantee of BARONS implies a  $\widetilde{O}(\sqrt{dT})$  regret bound for the portfolio selection problem [5] and a problem of linear prediction with log-loss [36], all while keeping the per-round computational cost under  $\widetilde{O}(d^2)$ , when  $T \geq d$ . The losses in both of these problems are neither bounded or Lipschitz.
Related works. In the past decade, many projection-free OCO algorithms have been developed to address the computational shortcoming of their projection-based counter parts [14, 16, 15, 18, 28, 11]. Most projection-free algorithms are based on the Frank-Wolfe method and perform linear optimization (typically once per round) over  $\kappa$  instead of Euclidean projection. Under no additional assumptions
2We credit the name BARONS to [27] who used barrier-regularized Newton steps for the portfolio selection problem.
other than convexity and Lipschitzness of the losses, the best-known regret bound for such algorithms scales as  $O(T^{3/4})$  [15]. While this bound is still sublinear in  $T$  and has no dependence in the dimension  $d$ , it is sub-optimal compared to the  $O(\sqrt{T})$  regret bound achievable with projection-based algorithms. In the recent years, there have been improvements to this bound under additional assumptions such as when the functions are smooth and/or strongly convex [15, 18], or when the convex set  $\mathcal{K}$  is smooth and/or strongly convex [1, 23, 29, 24]. For the case where  $\mathcal{K}$  is a polytope, [10] presented a linear-optimization-based algorithm that enjoys a  $O(\mu \sqrt{dT})$  regret bound, where  $\mu$  is a conditioning number for the set  $\mathcal{K}$ . Unfortunately,  $\mu$  can be large for many sets of interests as it essentially scales inversely with the minimum distance between the vertices of  $\mathcal{K}$ . In this work, we achieve a dimension-free  $\widetilde{O}(\sqrt{T})$  regret bound without the  $\mu$  factor.
More recently a new type of projection-free algorithms have emerged which use membership/separation oracle calls instead of linear optimization [28, 11, 24, 26]. From a computational perspective, separation-based and linear optimization-based algorithms are not really comparable, since there are sets over which separation is cheaper than linear optimization, and vice-versa. On the regret side, separation-based algorithms have been show to achieve a  $O(\kappa \sqrt{T})$  regret bound, where  $\kappa$  is the asphericity of the set  $\mathcal{K}$ . Separation-based algorithms are simple, often easy to analyze, and achieve the optimal-in-  $T$  regret bound, unlike linear optimization-based algorithms. However, the multiplicative factor  $\kappa$  in their regret bounds means that a pre-conditioning step may be required to ensure it is appropriately bounded. This precondition step would involve putting the set into (near-) isotropic position [9]; an operation, that can cost  $\widetilde{O}(d^4)$  arithmetic operations [38]; and even after such a pre-processing step,  $\kappa$  can still be as large as  $d$  in the worst-case. Our algorithm has the benefit of not requiring any pre-processing step.
A third type of algorithms avoid projections by outputting Newton iterates that are guaranteed to be feasible thanks to the use of a self-concordant barrier. The first such algorithm in the context of online learning was introduced by [2]. They presented a general recipe for using self-concordant barriers with Newton steps in online linear optimization. However, their approach falls short of being computationally-efficient as their algorithm needs to compute the inverse of the Hessian of the barrier at every iteration. Inspired by the work of [2], [31] used damped Newton steps with quadratic terms added to the barrier to design an efficient algorithm for the classical portfolio selection problem. Closer to our work is that of [30] who used a similar barrier for designing an algorithm for exp-concave optimization that can be viewed as a computationally-efficient version of the Online Newton Step [13]. Similar to our work, [30] also leverage the stability of the Newton iterates to avoid computing the inverse of the Hessian of the barrier at every step. However, their approach and analysis, which are tailored to the exp-concave setting do not necessarily lead to improved regret bounds in the general OCO setting we consider. In particular, their algorithm does not lead to a  $O(\sqrt{T})$  regret bound over polytopes.
Finally, for our application to polytopes, we make use of recent tools and techniques developed for solving linear programs efficiently. In particular, we make use of the Lee-Sidford barrier [20, 21, 22], which can be computed efficiently and, when used to compute Newton iterates, leads to the state-of-the-art  $\widetilde{O}(\sqrt{d})$  iteration upper-bound for solving a linear program. For the OCO setting, we show that using the Lee-Sidford barrier within our algorithm leads to a  $\widetilde{O}(\sqrt{T})$  regret bound. We also note that ideas similar to the ones we use to avoid computing the inverse of the Hessian of the barrier at every round were used to amortize computations in the context of solving linear programs (see e.g. [4, 39, 40]).
Outline. In section 2, we present our notation and relevant definitions. In Section 3, we present our algorithm and guarantees. In Section 4, we apply our results to the case of a polytope. All the proof are differed to the appendix.
Throughout the paper, we let  $\mathcal{K}$  be a closed convex subset of  $\mathbb{R}^d$ . We denote by  $\| \cdot \|$  the Euclidean norm and by  $\mathcal{B}(R)\subset \mathbb{R}^d$  the Euclidean ball of radius  $R > 0$ . We let int  $\mathcal{K}$  denote the interior of  $\mathcal{K}$ .
Our main algorithm, which can be viewed as an "online" counter-part to the Newton iterations [33], uses a self-concordant barrier over the set of interest to avoid the need of performing Euclidean projections onto  $\mathcal{K}$ . Next, we present the definition of a self-concordant barrier.
Self-concordant barriers. For the rest of this section, we let  $\mathcal{K}$  be a convex compact set with nonempty interior int  $\mathcal{K}$ . For a twice [resp. thrice] differentiable function, we let  $\nabla^2 f(\pmb{u})$  [resp.  $\nabla^3 f(\pmb{u})$ ] be the Hessian [resp. third derivative tensor] of  $f$  at  $\pmb{u}$ .
Definition 1 (Self-concordant function). A convex function  $f$ : int  $\mathcal{K} \to \mathbb{R}$  is called self-concordant with constant  $M_f \geq 0$ , if  $f$  is  $C^3$  and satisfies
-  $f(x_{k}) \to +\infty$  for  $x_{k} \to x \in \partial \mathcal{K}$ ; and
- For all  $x \in \operatorname{int} \mathcal{K}$  and  $u \in \mathbb{R}^d$ ,  $|\nabla^3 f(x)[u, u, u]| \leq 2M_f \|u\|_{\nabla^2 f(x)}^3$ .
Definition 2 (Self-concordant barrier). For  $M_f, \nu \geq 0$ , we say that  $f$ : int  $\mathcal{K} \to \mathbb{R}$  is a  $(M_f, \nu)$ -self-concordant barrier for  $\mathcal{K}$  if  $f$  is a self-concordant function over  $\mathcal{K}$  with constant  $M_f$  and
$$
\forall w \in \operatorname {i n t} \mathcal {K}, \quad \nabla f (w) ^ {\top} \nabla^ {- 2} f (w) \nabla f (w) \leq \nu .
$$
Computational Oracles. We will assume that our algorithm has access to a self-concordant function over the set  $\mathcal{K}$  through the following gradient and Hessian Oracles.
Definition 3 (Gradient Oracle). Given a point  $w \in \operatorname{int} \mathcal{K}$  and a tolerance  $\varepsilon > 0$ , the gradient Oracle  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}(\Phi)$  returns an  $\varepsilon$ -approximate vector  $\widehat{\nabla}_w$  of the gradient  $\nabla \Phi(w)$  in the dual local norm of the Hessian:
$$
\left\| \widehat {\nabla} _ {w} - \nabla \Phi (w) \right\| _ {\nabla^ {- 2} \Phi (w)} \leq \varepsilon .
$$
We denote by  $\mathcal{C}_{\varepsilon}^{\mathrm{grad}}(\Phi)$  the computational cost of one call to  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}(\Phi)$ .
When clear from the context, we will simply write  $\mathcal{C}_{\varepsilon}^{\mathrm{grad}}$  and  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}$  for  $\mathcal{C}_{\varepsilon}^{\mathrm{grad}}(\Phi)$  and  $\mathcal{O}_{\varepsilon}^{\mathrm{grad}}(\Phi)$ , respectively.
Definition 4 (Hessian Oracle). Given a point  $w \in \operatorname{int} \mathcal{K}$  and a tolerance  $\varepsilon > 0$ , the Hessian Oracle  $\mathcal{O}_{\varepsilon}^{\mathrm{hess}}(\Phi)$  returns a matrix  $H$  and its inverse  $H^{-1}$  which are  $1 \pm \varepsilon$  spectral approximations of the Hessian and inverse Hessian of  $\Phi$  at  $w$ :
$$
(1 - \varepsilon) \nabla^ {2} \Phi (w) \leqslant H \leqslant (1 + \varepsilon) \nabla^ {2} \Phi (w) \quad a n d \quad (1 - \varepsilon) \nabla^ {- 2} \Phi (w) \leqslant H ^ {- 1} \leqslant (1 + \varepsilon) \nabla^ {- 2} \Phi (w).
$$
We denote by  $\mathcal{C}_{\varepsilon}^{\mathrm{hess}}(\Phi)$  the computational cost of one call to  $\mathcal{O}_{\varepsilon}^{\mathrm{hess}}(\Phi)$ .
When clear from the context, we will simply write  $\mathcal{C}_{\varepsilon}^{\mathrm{hess}}$  and  $\mathcal{O}_{\varepsilon}^{\mathrm{hess}}$  for  $\mathcal{C}_{\varepsilon}^{\mathrm{hess}}(\Phi)$  and  $\mathcal{O}_{\varepsilon}^{\mathrm{hess}}(\Phi)$ , respectively.
Additional notation. We use the notation  $f \lesssim g$  to mean  $f \leq Cg$  for some universal constant  $C > 0$ . We also write  $f \leq \widetilde{O} g$  to mean  $f \leq \mathrm{polylog}(T, d) \cdot g$ . We let  $\nabla^{-2} \coloneqq (\nabla^2)^{-1}$  and  $\nabla^{-1/2}$  refer to the inverse of the Hessian and the inverse of the square root of the Hessian, respectively.
3 Algorithm and Regret Guarantees
In this section, we construct a projection-free algorithm for Online Convex Optimization. The algorithm in question (Alg. 1) outputs approximate Newton iterates with respect to "potential functions"  $(\Phi_t)$  that take the following form:
$$
\Phi_ {t} (w) := \Phi (w) + w ^ {\top} \sum_ {s = 1} ^ {t - 1} g _ {s},
$$
where  $(g_{s}\in \partial \ell_{s}(w_{s}))$  are the sub-gradients of the losses  $(\ell_s)$  at the iterates  $(w_{s})$  of Algorithm 1, and  $\Phi$  is a self-concordant function over  $\kappa$ . Algorithm 1 uses the approximate gradient and Hessian Oracles of  $\Phi$  (see 2) to output iterates  $(w_{t})$  approximate Newton iterates in the following sense:
$$
\forall t \in [ T ], \quad w _ {t + 1} \approx w _ {t} - \nabla^ {- 2} \Phi_ {t + 1} (w _ {t}) \nabla \Phi_ {t + 1} (w _ {t}). \tag {1}
$$
As is by now somewhat standard in the analyses of online Newton iterates of the form in (1), we will bound the regret of Algorithm 1 by showing that:
- The iterates  $(w_{t})$  are close (in the norm induced by the Hessian  $\nabla^2\Phi(w_t)$ ) to the FTRL iterates, which are given by
$$
w _ {t} ^ {\star} \in \operatorname * {a r g m i n} _ {w \in \mathcal {K}} \Phi_ {t} (w). \tag {2}
$$
- The regret of FTRL is bounded by  $O(\sqrt{T})$ .
Our main contribution is an algorithm that outputs iterates  $(w_{t})$  that satisfy the first bullet point (i.e. iterates that satisfy (1)) while only calling a Hessian Oracle (which is potentially computationally expensive) a  $O(1 / \sqrt{T})$  fraction of the rounds after  $T$  rounds. As we show in Section 4, for the case where  $\mathcal{K}$  is a polytope with  $m\in \mathbb{N}$  constraints, the algorithm achieves a  $\widetilde{O} (\sqrt{T})$  regret bound, where the per-iteration computational cost essentially reduces to a linear-system-solve involving a  $d\times m$  matrix. Among existing OCO algorithms that achieve a  $\widetilde{O} (\sqrt{T})$  regret bound, none can achieve this computational complexity for general polytopes with  $m$  constraints (see Section 4 for more details).
3.1 Efficient Computation of the Newton Iterates with BARONS
The key feature of BARONS (Algorithm 1) is that it uses an amortized computation of the Hessian. Namely, BARONS computes the inverse of the Hessian of the barrier  $\Phi$  only for a small fraction of the iterates  $(w_{t})$ . Henceforth, we refer to the iterates where the algorithm computes the full inverse of the Hessian as landmark iterates; these are the iterates  $(u_{t})$  in Lines 13 and 16 of Algorithm 1. The idea behind this is that for a sufficiently curved<sup>3</sup> barrier  $\Phi$ , the Newton iterates with respect to  $\Phi$  are stable enough that it suffices to compute the inverse of the Hessian of  $\Phi$  at the closest landmark iterate. For example, this is what was done in [30] to design an efficient algorithm for exp-concave optimization.
Unlike the setting of [30], where it is possible to add quadratic terms to the barrier for additional stability, in our setting we cannot do that without sacrificing performance in terms of regret. Without the quadratic terms, the Newton iterates are not stable enough for our desired guarantee. Instead of adding regularization terms, BARONS takes  $\widetilde{O}(1)$  Newton steps per round to get "closer" to the Newton iterate with the true Hessian matrix. This simple approach is key to the success of our approach.
In the next subsection, we give a generic guarantee for BARONS.
3.2 Generic Regret Guarantee of BARONS
In this subsection, we present a general regret and computational guarantee for BARONS under minimal assumptions on the sequence of losses and without turning the "step size"  $\eta$ . In the next subsection, we will instantiate the regret guarantee when additional assumptions on the sequence of losses are available. We now state the main guarantee of BARONS (the proof in Appendix C.1).
Theorem 5 (Master theorem). Let  $\Phi$  be a self-concordant function over  $\mathcal{K}$  with constant  $M_{\Phi} > 0$  and let  $b, \eta, \varepsilon, \alpha > 0$  and  $m_{\mathrm{Newton}} \in \mathbb{N}$  be such that  $\eta \leq \frac{1}{1000bM_{\Phi}}$ ,  $\varepsilon \leq \frac{1}{20000M_{\Phi}}$ ,  $\alpha = 0.001$ , and  $m_{\mathrm{Newton}} \coloneqq \Theta (\log \frac{1}{\varepsilon M_{\Phi}})$ . Further, let  $(w_{t})$  be the iterates of Algorithm 1 with input  $(\eta, \varepsilon, \alpha, m_{\mathrm{Newton}})$  and suppose that the corresponding sub-gradients  $(g_{t})$  satisfy  $\| g_t \|_{\nabla^{-2}\Phi(w_t)} \leq b$ , for all  $t \geq 1$ . Then, the regret of Algorithm 1 is bounded as:
$$
\sum_ {t = 1} ^ {T} \left(\ell_ {t} \left(w _ {t}\right) - \ell_ {t} (w)\right) \lesssim \frac {1}{\eta} \Phi (w) + \eta \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi \left(w _ {t}\right)} ^ {2} + \varepsilon \sum_ {t = 1} ^ {T} \| g _ {t} \| _ {\nabla^ {- 2} \Phi \left(w _ {t}\right)}, \quad \forall w \in \operatorname {i n t} \mathcal {K}. \tag {3}
$$
Furthermore, the computational cost of the algorithm is bounded by
$$
O \left(\left(\mathcal {C} _ {\bar {\varepsilon}} ^ {\text {g r a d}} + d ^ {2}\right) \cdot T \cdot \log \frac {1}{\varepsilon M _ {\Phi}} + \mathcal {C} _ {\alpha} ^ {\text {h e s s}} \cdot \left(M _ {\Phi} T \varepsilon + M _ {\Phi} \sum_ {t = 1} ^ {T} \eta \| g _ {t} \| _ {\nabla^ {- 2} \Phi (w _ {t})}\right)\right).
$$
Theorem 5 essentially shows that it is possible to achieve the same regret as FTRL, while only computing the inverse of the Hessian of  $\Phi$  at most  $\widetilde{O}(M_{\Phi}T\eta)$  number of times.
3 Informally, the "curvature" of a convex function is high when the rate of change of its gradients is high.