Title: Projection-Free Online Convex Optimization via Efficient Newton Iterations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online Learning

Online convex optimization

projection-free

Newton method
Abstract: This paper presents new projection-free algorithms for Online Convex Optimization (OCO) over a convex domain $\mathcal{K} \subset \mathbb{R}^d$. Classical OCO algorithms (such as Online Gradient Descent) typically need to perform Euclidean projections onto the convex set $\mathcal{K}$ to ensure feasibility of their iterates. Alternative algorithms, such as those based on the Frank-Wolfe method, swap potentially-expensive Euclidean projections onto $\mathcal{K}$ for linear optimization over $\mathcal{K}$. However, such algorithms have a sub-optimal regret in OCO compared to projection-based algorithms. In this paper, we look at a third type of algorithms that output approximate Newton iterates using a self-concordant barrier for the set of interest. The use of a self-concordant barrier automatically ensures feasibility without the need of projections. However, the computation of the Newton iterates requires a matrix inverse, which can still be expensive. As our main contribution, we show how the stability of the Newton iterates can be leveraged to only compute the inverse Hessian a vanishing fractions of the rounds, leading to a new efficient projection-free OCO algorithm with a state-of-the-art regret bound.

Projection-Free Online Convex Optimization via Efficient Newton Iterations

Abstract

This paper presents new projection-free algorithms for Online Convex Optimization (OCO) over a convex domain K ⊂ R d . Classical OCO algorithms (such as Online Gradient Descent) typically need to perform Euclidean projections onto the convex set K to ensure feasibility of their iterates. Alternative algorithms, such as those based on the Frank-Wolfe method, swap potentially-expensive Euclidean projections onto K for linear optimization over K . However, such algorithms have a sub-optimal regret in OCO compared to projection-based algorithms. In this paper, we look at a third type of algorithms that output approximate Newton iterates using a self-concordant barrier for the set of interest. The use of a self-concordant barrier automatically ensures feasibility without the need of projections. However, the computation of the Newton iterates requires a matrix inverse, which can still be expensive. As our main contribution, we show how the stability of the Newton iterates can be leveraged to only compute the inverse Hessian a vanishing fractions of the rounds, leading to a new efficient projection-free OCO algorithm with a state-of-the-art regret bound.

1 Introduction

We consider the Online Convex Optimization (OCO) problem over a convex set K ⊂ R d , in which a learner (algorithm) plays a game against an adaptive adversary for T rounds. At each round t ∈ [ T ] , the learner picks w t ∈ K given knowledge of the history H t -1 ∶= {( ℓ s , w s )} s < t . Then, the adversary picks a convex loss function ℓ t ∶ K → R with the knowledge of H t -1 and the iterate w t , and the learner suffers loss ℓ t ( w t ) and proceeds to the next round. The goal of the learner is to minimize the regret after T rounds:

against any comparator w ∈ K . The aim of this paper is to design computationally-efficient (projectionfree) algorithms for OCO that enjoy the optimal (up to log-factor in T ) ̃ O ( √ T ) regret.

The OCO framework captures many optimization settings relevant to machine learning applications. For example, OCO algorithms can be used in offline convex optimization as more computationallyand memory-efficient alternatives to interior-point and cutting plane methods whenever the dimension d is large [14, 16]. OCO algorithms are also often used in stochastic convex optimization, where the standard O ( √ T ) regret (achieved by, e.g. Online Gradient Descent) translates into the optimal O ( 1 / √ T ) rate 1 via the classical online-to-batch conversion technique [3, 37]. It has been shown that OCO algorithms can also achieve state-of-the-art accelerated rates in both the offline and stochastic optimization settings despite being designed for the more general OCO framework [6, 28]. What is

more, it has recently been shown that even non-convex (stochastic) optimization can be reduced to online linear optimization (a special case of OCO), where it is then possible to recover the best-known convergence rates for the setting [7].

Given the prevalent use of OCO algorithms in machine learning applications, it is important to have computationally-efficient algorithms that scale well with the dimension d of the ambient space. However, most OCO algorithms fall short of being efficient because of the need of performing (Euclidean) projections onto K (potentially at each iteration) to ensure that the iterates are feasible. These projections are often inefficient, especially in high-dimensional settings with complex feasible sets. Existing projection-free OCO algorithms address this computational challenge by swapping potentially-expensive Euclidean projections for often much cheaper linear optimization or separation over the feasible set K . However, existing projection-free algorithms have sub-optimal regret guarantees in terms of their dependence in T , or have potentially unbounded 'condition numbers' for the feasible set multiplying their regret guarantee.

Contributions. In this paper, we address these computational and performance challenges by revisiting an existing (but somewhat overlooked) type of projection-free OCO algorithms. Unlike existing algorithms, our proposed method does not require linear optimization or separation over the feasible set K . Instead, the algorithm, Barrier-Regularized Online Newton Step ( BARONS ), 2 uses a self-concordant barrier Φ for the set K to always output iterates that are guaranteed to be within K ; much like interior point methods for offline optimization. In particular, our algorithm outputs Newton iterates with respect to time-varying, translated versions of Φ . The main novelty of our work is in devising a new efficient way of computing the Newton iterates without having to evaluate the inverse of the Hessian of the barrier at every iteration, which can be computationally expensive in high-dimensional settings. Our algorithm only needs to compute a full inverse of the Hessian a vanishing O ( 1 / √ T ) fraction of the rounds. For the rest of the rounds, the computational cost is dominated by that of evaluating the gradient of the barrier Φ , which can be much cheaper than evaluating the inverse of its Hessian in many cases.

For the special case of a polytope with m constraints, we show that there is a choice of a barrier (e.g. the Lee-Sidford barrier) that when used within our algorithm, reduces the per-round computational cost to essentially ̃ O ( 1 ) linear-system-solves of size m × d . We show that this is often cheaper than performing linear optimization over K , which other projection-free algorithms require. More importantly, our algorithm achieves a dimension-free ̃ O ( √ T ) regret bound. This improves over the existing regret bounds of projection-free algorithms over polytopes. For example, among projection-free algorithms that achieve a O ( √ T ) regret, the algorithms by [28, 11, 26], which require a separation/membership Oracle for K , have a multiplicative κ = R / r factor multiplying their regret bounds, where r, R > 0 are such that B( r ) ⊆ K ⊆ B( R ) . The constant κ , known as the asphercity [12], can in principle be arbitrarily large. Even after applying a potentially expensive pre-processing step, which would typically involve putting the set K into (near-) isotropic position [9, 38], κ can still be as large as d in the worst-case, and so the regret bounds achieved by the algorithms of [28, 11, 26] can be of order O ( d √ T ) ; this is worse than ours by a d factor. Other projection-free algorithms based on the Frank-Wolfe method, e.g. those in [10, 35, 19], also have multiplicative condition numbers that are even less benign that the asphercity κ . In fact, the condition numbers in the regret bounds for polytopes appearing in, e.g. [10], can in principle be arbitrarily large regardless of any pre-processing.

Finally, another advantage of our algorithm is that it can guarantee a sublinear regret even for nonLipschitz losses (i.e. where the norm of the sub-gradients may be unbounded). In particular, we show that the general guarantee of BARONS implies a ̃ O ( √ dT ) regret bound for the portfolio selection problem [5] and a problem of linear prediction with log-loss [36], all while keeping the per-round computational cost under ̃ O ( d 2 ) , when T ≥ d . The losses in both of these problems are neither bounded or Lispchitz.

Related works. In the past decade, many projection-free OCO algorithms have been developed to address the computational shortcoming of their projection-based counter parts [14, 16, 15, 18, 28, 11]. Most projection-free algorithms are based on the Frank-Wolfe method and perform linear optimization (typically once per round) over K instead of Euclidean projection. Under no additional assumptions

other than convexity and lipschitzness of the losses, the best-known regret bound for such algorithms scales as O ( T 3 / 4 ) [15]. While this bound is still sublinear in T and has no dependence in the dimension d , it is sub-optimal compared to the O ( √ T ) regret bound achievable with projectionbased algorithms. In the recent years, there have been improvements to this bound under additional assumptions such as when the functions are smooth and/or strongly convex [15, 18], or when the convex set K is smooth and/or strongly convex [1, 23, 29, 24]. For the case where K is a polytope, [10] presented a linear-optimization-based algorithm that enjoys a O ( µ √ dT ) regret bound, where µ is a conditioning number for the set K . Unfortunately, µ can be large for many sets of interests as it essentially scales inversely with the minimum distance between the vertices of K . In this work, we achieve a dimension-free ̃ O ( √ T ) regret bound without the µ factor.

More recently a new type of projection-free algorithms have emerged which use membership/separation oracle calls instead of linear optimization [28, 11, 24, 26]. From a computational perspective, separation-based and linear optimization-based algorithms are not really comparable, since there are sets over which separation is cheaper than linear optimization, and vice-versa. On the regret side, separation-based algorithms have been show to achieve a O ( κ √ T ) regret bound, where κ is the asphercity of the set K . Separation-based algorithms are simple, often easy to analyze, and achieve the optimal-inT regret bound, unlike linear optimization-based algorithms. However, the multiplicative factor κ in their regret bounds means that a pre-conditioning step may be required to ensure it is appropriately bounded. This precondition step would involve putting the set into (near-) isotropic position [9]; an operation, that can cost ̃ O ( d 4 ) arithmetic operations [38]; and even after such a pre-processing step, κ can still be as large as d in the worst-case. Our algorithm has the benefit of not requiring any pre-processing step.

A third type of algorithms avoid projections by outputting Newton iterates that are guaranteed to be feasible thanks to the use of a self-concordant barrier. The first such algorithm in the context of online learning was introduced by [2]. They presented a general recipe for using self-concordant barriers with Newton steps in online linear optimization. However, their approach falls short of being computationally-efficient as their algorithm needs to compute the inverse of the Hessian of the barrier at every iteration. Inspired by the work of [2], [31] used damped Newton steps with quadratic terms added to the barrier to design an efficient algorithm for the classical portfolio selection problem. Closer to our work is that of [30] who used a similar barrier for designing an algorithm for exp-concave optimization that can be viewed as a computationally-efficient version of the Online Newton Step [13]. Similar to our work, [30] also leverage the stability of the Newton iterates to avoid computing the inverse of the Hessian of the barrier at every step. However, their approach and analysis, which are tailored to the exp-concave setting do not necessarily lead to improved regret bounds in the general OCO setting we consider. In particular, their algorithm does not lead to a O ( √ T ) regret bound over polytopes.

Finally, for our application to polytopes, we make use of recent tools and techniques developed for solving linear programs efficiently. In particular, we make use of the Lee-Sidford barrier [20, 21, 22], which can be computed efficiently and, when used to compute Newton iterates, leads to the state-ofthe-art ̃ O ( √ d ) iteration upper-bound for solving a linear program. For the OCO setting, we show that using the Lee-Sidford barrier within our algorithm leads to a ̃ O ( √ T ) regret bound. We also note that ideas similar to the ones we use to avoid computing the inverse of the Hessian of the barrier at every round were used to amortize computations in the context of solving linear programs (see e.g. [4, 39, 40]).

Outline. In section 2, we present our notation and relevant definitions. In Section 3, we present our algorithm and guarantees. In Section 4, we apply our results to the case of a polytope. All the proof are differed to the appendix.

2 Preliminaries

Throughout the paper, we let K be a closed convex subset of R d . We denote by ∥ ⋅ ∥ the Euclidean norm and by B( R ) ⊂ R d the Euclidean ball of radius R > 0 . We let int K denote the interior of K .

Our main algorithm, which can be viewed as an 'online' counter-part to the Newton iterations [33], uses a self-concordant barrier over the set of interest to avoid the need of performing Euclidean projections onto K . Next, we present the definition of a self-concordant barrier.

Self-concordant barriers. For the rest of this section, we let K be a convex compact set with nonempty interior int K . For a twice [resp. thrice] differentiable function, we let ∇ 2 f ( u ) [resp. ∇ 3 f ( u ) ] be the Hessian [resp. third derivative tensor] of f at u .

Definition 1 (Self-concordant function) . A convex function f ∶ int K → R is called self-concordant with constant M f ≥ 0 , if f is C 3 and satisfies

For all x ∈ int K and u ∈ R d , ∣∇ 3 f ( x )[ u, u, u ]∣ ≤ 2 M f ∥ u ∥ 3 ∇ 2 f ( x ) .

Definition 2 (Self-concordant barrier) . For M f , ν ≥ 0 , we say that f ∶ int K → R is a ( M f , ν ) -selfconcordant barrier for K if f is a self-concordant function over K with constant M f and

Computational Oracles. We will assume that our algorithm has access to a self-concordant function over the set K through the following gradient and Hessian Oracles.

Definition 3 (Gradient Oracle) . Given a point w ∈ int K and a tolerance ε > 0 , the gradient Oracle O grad ε ( Φ ) returns an ε -approximate vector ̂ ∇ w of the gradient ∇ Φ ( w ) in the dual local norm of the Hessian:

We denote by C grad ε ( Φ ) the computational cost of one call to O grad ε ( Φ ) .

When clear from the context, we will simply write C grad ε and O grad ε for C grad ε ( Φ ) and O grad ε ( Φ ) , respectively.

Definition 4 (Hessian Oracle) . Given a point w ∈ int K and a tolerance ε > 0 , the Hessian Oracle O hess ε ( Φ ) returns a matrix H and its inverse H -1 which are 1 ± ε spectral approximations of the Hessian and inverse Hessian of Φ at w :

We denote by C hess ε ( Φ ) the computational cost of one call to O hess ε ( Φ ) .

When clear from the context, we will simply write C hess ε and O hess ε for C hess ε ( Φ ) and O hess ε ( Φ ) , respectively.

Additional notation. We use the notation f ≲ g to mean f ≤ Cg for some universal constant C > 0 . We also write f ≤ ̃ Og to mean f ≤ polylog ( T,d ) ⋅ g . We let ∇ -2 ∶= (∇ 2 ) -1 and ∇ -1 / 2 refer to the inverse of the Hessian and the inverse of the square root of the Hessian, respectively.

3 Algorithm and Regret Guarantees

In this section, we construct a projection-free algorithm for Online Convex Optimization. The algorithm in question (Alg. 1) outputs approximate Newton iterates with respect to 'potential functions' ( Φ t ) that take the following form:

where ( g s ∈ ∂ℓ s ( w s )) are the sub-gradients of the losses ( ℓ s ) at the iterates ( w s ) of Algorithm 1, and Φ is a self-concordant function over K . Algorithm 1 uses the the approximate gradient and Hessian Oracles of Φ (see 2) to output iterates ( w t ) approximate Newton iterates in the following sense:

As is by now somewhat standard in the analyses of online Newton iterates of the form in (1), we will bound the regret of Algorithm 1 by showing that:

The iterates ( w t ) are close (in the norm induced by the Hessian ∇ 2 Φ ( w t ) ) to the FTRL iterates, which are given by

The regret of FTRL is bounded by O ( √ T ) .

Our main contribution is an algorithm that outputs iterates ( w t ) that satisfy the first bullet point (i.e. iterates that satisfy (1)) while only calling a Hessian Oracle (which is potentially computationally expensive) a O ( 1 / √ T ) fraction of the rounds after T rounds. As we show in Section 4, for the case where K is a polytope with m ∈ N constraints, the algorithm achieves a ̃ O ( √ T ) regret bound, where the per-iteration computational cost essentially reduces to a linear-system-solve involving a d × m matrix. Among existing OCO algorithms that achieve a ̃ O ( √ T ) regret bound, none can achieve this computational complexity for general polytopes with m constraints (see Section 4 for more details).

3.1 Efficient Computation of the Newton Iterates with BARONS

The key feature of BARONS (Algorithm 1) is that is uses an amortized computation of the Hessians. Namely, BARONS computes the inverse of the Hessian of the barrier Φ only for a small fractions of the iterates ( w t ) . Henceforth, we refer to the iterates where the algorithm computes the full inverse of the Hessian as landmark iterates ; these are the iterates ( u t ) in Lines 13 and 16 of Algorithm 1. The idea behind this is that for a sufficiently curved 3 barrier Φ , the Newton iterates with respect to Φ are stable enough that it suffices to compute the inverse of the Hessian of Φ at the closest landmark iterate. For example, this is what was done in [30] to design an efficient algorithm for exp-concave optimization.

Unlike the setting of [30], where it is possible to add quadratic terms to the barrier for additional stability, in our setting we cannot do that without sacrificing performance in terms of regret. Without the quadratic terms, the Newton iterates are not stable enough for our desired guarantee. Instead of adding regularization terms, BARONS takes ̃ O ( 1 ) Newton steps per round to get 'closer' to the Newton iterate with the true Hessian matrix. This simple approach is key to the success of our approach.

In the next subsection, we give a generic guarantee for BARONS .

3.2 Generic Regret Guarantee of BARONS

In this subsection, we present a general regret and computational guarantee for BARONS under minimal assumptions on the sequence of losses and without turning the 'step size' η . In the next subsection, we will instantiate the regret guarantee when additional assumptions on the sequence of losses are available. We now state the main guarantee of BARONS (the proof in Appendix C.1).

Theorem 5 (Master theorem) . Let Φ be a self-concordant function over K with constant M Φ > 0 , and let b, η, ε, α > 0 and m Newton ∈ N be such that η ≤ 1 1000 bM Φ , ε ≤ 1 20000 M Φ , α = 0 . 001 , and m Newton ∶= Θ ( log 1 εM Φ ) . Further, let ( w t ) be the iterates of Algorithm 1 with input ( η , ε , α , m Newton ) and suppose that the corresponding sub-gradients ( g t ) satisfy ∥ g t ∥ ∇ -2 Φ ( w t ) ≤ b , for all t ≥ 1 . Then, the regret of Algorithm 1 is bounded as:

Furthermore, the computational cost of the algorithm is bounded by

Theorem 5 essentially shows that it is possible to achieve the same regret as FTRL, while only computing the inverse of the Hessian of Φ at most ̃ O ( M Φ Tη ) number of times.

Algorithm 1 BARONS : Barrier-Regularized Online Newton Step

Require:

Parameters η, ε, α > 0 , and m Newton ≥ 1 . · Gradient/Hessian Oracles O grad ε /O hess α for self-concordant function Φ with constant M Φ > 0 . · w ⋆ ∈ argmin w ∈K Φ ( w ) . 1: Set u 1 = w 1 = w ⋆ , H 1 = O hess α ( w ⋆ ) , and s 0 = 0 . 2: for t = 1 , . . . , T do 3: Play w t and observe g t ∈ ∂ℓ t ( w t ) . 4: Set s t = s t -1 + ηg t . 5: Set w 1 t + 1 = w t . // Perform intermediate Newton steps to decrease the Newton decrement 6: for m = 1 , . . . , m Newton -1 do 7: Set ̂ ∇ t m + 1 = O grad ε ( w t m + 1 ) . 8: Set ̃ ∇ t m + 1 = ̂ ∇ t m + 1 + s t . // ̃ ∇ t m + 1 ≈ ∇ Φ t + 1 ( w t m + 1 ) 9: Set w m + 1 t + 1 = w t m + 1 -H -1 t ̃ ∇ t m + 1 . end for Set w t + 1 = w m + 1 t + 1 . // Check if the landmark needs updating if ∥ w t + 1 -u t ∥ H t ≤ 1 /( 41 M Φ ) then Set u t + 1 = u t . // Update landmark H t + 1 = H t and H -1 t + 1 = H -1 t . else Set u t + 1 = w t + 1 . // No landmark update Set H t + 1 = O hess α ( u t + 1 ) and compute H -1 t + 1 . end if end for

3.3 Regret Guarantee Under Local and Euclidean Norm Bounds on the Sub-Gradients

We now instantiate the guarantee in Theorem 5 with a ( M Φ , ν ) -self-concordant barrier Φ for the set K , with respect to which the local norms of the sub-gradients are bounded; that is, when ∥ g t ∥ ∇ -2 Φ ( w t ) ≤ b . We note that the regret bound in (36) has an additive Φ ( w ) which may be unbounded near the boundary of K . However, it is still possible to compete against comparators in int K by making additional assumptions on the range of the losses [27, 32]. We discuss some of these assumptions in the sequel. For the next theorem, we will state the regret bound of BARONS relative to comparators in the restricted set:

where ⊕ denotes the Minkowski sum, w ⋆ ∈ argmin w ∈K Φ ( w ) , and c ∈ ( 0 , 1 ) is a parameter.

With this, we now state a regret bound for BARONS when the sub-gradients of the losses have bounded local norms. The proof of the next theorem is in Appendix C.2.

Theorem 6 (Local norm bound) . Let Φ be an ( M Φ , ν ) -self-concordant barrier for K and let c ∈ ( 0 , 1 ) , b > 0 . Further, suppose that for all t ∈ [ T ] , ∥ g t ∥ ∇ -2 Φ ( w t ) ≤ b , where ( w t ) are the iterates of BARONS with input parameters ( η, ε, α, m Newton ) such that

For T ≥ 1 large enough such that η ≤ 1 1000 bM Φ , ε ≤ 1 20000 M Φ , the regret of BARONS is bounded as

where K c ⊂ K is as in (4) . Further, the computational complexity of BARONS in this case is bounded by

Remark 1. The regret bound in Theorem 6 is stated with respect to comparators in the restricted set K c defined in (4) . It is possible to extend this guarantee to all comparators in int K under an additional assumption on the range of the losses. For example, if for w ⋆ ∈ argmin w ∈K Φ ( w ) , we have

then the regret guarantee in (6) can be extended to all comparators in int K up to an additive O ( √ T ) term (see Lemma 7 in the appendix). In this case, the log c term in the computational complexity need be replaced by log T . We note that the condition in (7) does not require a uniform bound on the losses. Instead, it only restrict the rate of growth of the losses ( ℓ t ( w )) as w approaches the boundary of K . As we show in the sequel (§4.2), (7) is satisfied for some popular losses which are not Lipschitz.

We now instantiate the guarantee in Theorem 5 when the sub-gradients are bounded in Euclidean norm (instead of local norm); that is, we assume that for all t ∈ [ T ] , ∥ g t ∥ ≤ G for some G > 0 . We note that this assumption implies (7), and we will be able to bound the regret against all comparators in int K as alluded to in Remark 1. The proof of the next theorem is in Appendix C.3).

Theorem 7 (Euclidean norm bound) . Let Ψ be an ( M Ψ , ν ) self-concordant barrier for K and let Φ (⋅) ∶= Ψ (⋅)+ ν 2 R 2 ∥⋅∥ 2 . Further, let G,R > 0 and suppose that K ⊆ B( R ) and for all t ∈ [ T ] , ∥ g t ∥ ≤ G , where g t ∈ ∂ℓ t ( w t ) and ( w t ) are the iterates of BARONS with input parameters ( η, ε, α, m Newton ) such that

For T ≥ 1 large enough such that η ≤ 1 1000 GM Ψ , ε ≤ 1 20000 M Ψ , the regret of BARONS is bounded as

Further, the computational complexity of BARONS in this case is bounded by

4 Application to Polytopes Using the Lee-Sidford Barrier

In this section, we assume that the set K is a polytope in R d specified by m linear constraints:

and we construct efficient gradient and Hessian Oracles for a self-concordant barrier for K . This will then allow us to instantiate the guarantees of BARONS in Section 3 and provide explicit and state-of-the-art bounds on the regret of BARONS .

We will assume without loss of generality that ∥ a i ∥ = 1 , for all i ∈ [ m ] , and let A ∶= ( a 1 , . . . , a m ) ⊺ ∈ R m × d denote the constraint matrix of the set K . For the rest of this section, it will be convenient to define the 'slack' variables s w,i = a ⊺ i w -b ′ i , for i ∈ [ m ] . Here, s w,i essentially represents the distance of w to the i th facet of the polytope K . Further, we let S w ∶= diag ( s w ) be the diagonal matrix whose i th diagonal entry is s w,i .

The LS barrier. To perform Online Convex Optimization over K , we pick the regularizer Φ of BARONS to be the Lee-Sidford (LS) barrier Φ LS [22] with parameter p > 0 , which is defined as

where V = diag ( v ) . One way to think of the LS barrier is as a weighted log-barrier. As we will discuss in the sequel, this choice will confer computational and performance (in terms of regret) advantages over the standard log-barrier.

Self-concordance of the LS barrier. According to [8, Theorem 30], the LS barrier with the choice p = O ( log ( m )) is a self-concordant function with parameter M Φ LS satisfying

The other favorable property of this barrier is that its Newton decrement at any point w ∈ K is of order ̃ O ( √ d ) ; that is,

Therefore, Φ LS is a ( ̃ O ( 1 ) , ̃ O ( d )) -self-concordant barrier. For the log-barrier, the right-hand side of (11) would be √ m .

Cost of gradient and Hessian Oracles. We consider the computational complexities of gradient and Hessian Oracles for Φ LS . By [22], we have that for ε > 0 ,

where C sys is the computational cost of solving a linear system of the form A ⊺ diag ( v ) Ax = y , for vectors v ∈ R d ≥ 0 and y ∈ R d ; we recall that A = ( a 1 , . . . , a m ) ⊺ is the constraint matrix for K . In the worst-case, such a linear system can be solved with cost bounded as

where ω is the exponent of matrix multiplication, and m is the number of constraints of K . However, as we show in the sequel, C sys can be much smaller in many practical applications.

With this, we immediately obtain the following corollary for the regret and run-time of BARONS under local norm and Euclidean norm bounds on the sub-gradients.

Corollary 1 (OCO over a polytope with LS barrier) . Let c ∈ ( 0 , 1 ) , G, R, b > 0 , and suppose K is given by (10) and that Φ LS is the corresponding LS barrier. Further, let ( w t ) be the iterates of BARONS , and let K c be the restricted version of K defined in (4) . Then, the following holds:

Local norm bound: If ∥ g t ∥ ∇ -2 Φ ( w t ) ≤ b , for all t ≥ 1 , and the parameters ( η, ε, α, m Newton ) of BARONS are set as in Theorem 6 with Φ = Φ LS and ( M Φ , ν ) = ( ̃ O ( 1 ) , ̃ O ( d )) , then for T large enough (as specified in Theorem 6), the regret of BARONS is bounded by

Euclidean norm bound: If K ⊆ B( R ) and ∥ g t ∥ ≤ G , for all t ≥ 1 , and the parameters ( η, ε, α, m Newton ) of BARONS are set as in Theorem 7 with Φ (⋅) = Φ LS (⋅) + ν 2 R 2 ∥ ⋅ ∥ 2 and ( M Ψ , ν ) = ( ̃ O ( 1 ) , ̃ O ( d )) , then for T large enough (as in Theorem 7) BARONS has regret bounded as

In either case, the computational complexity is bounded by

where C sys is the computational cost of solving a linear system of the form A ⊺ diag ( v ) Ax = y , for vectors v ∈ R d ≥ 0 and y ∈ R d (recall that A is the constraint matrix for the polytope K ).

Using the log -barrier. We note that since K is a polytope, we could have used the standard log-barrier

This barrier is ( 1 , m ) -self-concordant, and so instantiating Theorem 5 with it would imply a ̃ O ( b √ mdT ) regret bound in the case of local sub-gradient norms bounded by b > 0 . Using the LS barrier replaces the √ m term in this bound by √ d regardless of the number of constraints-see

(13). However, this comes at a C sys computational cost, which can be as high as md ω -1 in the worst-case (see (12)). In the case of the log-barrier, this cost would be replaced by md (essentially because C grad ε ( Φ log ) ≤ O ( md ) ). Thus, when m is of the order of d , using the log-barrier may be more computational-efficient compared to using the LS barrier. In the next corollary, we bound the regret of BARONS when Φ = Φ log ; this result is an immediate consequence of Theorem 7.

Corollary 2 (OCO over a polytope with the log barrier) . Let G,b > 0 , and suppose K is given by (10) and that Φ log is the corresponding log -barrier. Further, let ( w t ) be the iterates of BARONS . If K ⊆ B( R ) and ∥ g t ∥ ≤ G , for all t ≥ 1 , and the parameters ( η, ε, α, m Newton ) of BARONS are set as in Theorem 7 with Φ (⋅) = Φ log (⋅) + ν 2 R 2 ∥ ⋅ ∥ 2 and ( M Ψ , ν ) = ( 1 , m ) , then for T large enough (as in Theorem 7) BARONS has regret bounded as

The computational complexity is bounded by

4.1 Implications for Lipschitz Losses

We now discuss implications of Corollary 1, and compare the bound of BARONS to those of existing algorithms for Lipschitz losses.

Dimension-free regret bound. We note when the Euclidean norms of the sub-gradients are bounded, BARONS achieves a dimension-free O ( √ T ) regret bound. In contrast, the best dimensionfree regret bound 4 achieved by existing projection-free algorithms is of order O ( T 3 / 4 ) (see e.g. [14, 11]). We also note that existing separation/membership-based algorithms that achieve a √ T regret; for examples those presented in [28, 11, 26], are not dimension-free. Their regret bounds are of order O ( κ √ T ) , where κ = R / r with r, R > 0 such that B( r ) ⊆ K ⊆ B( R ) . The asphercity parameter can depend on the dimension d [28], and even after a pre-conditioning step (which would involve putting the set K into near-isotropic position and can cost up to Ω ( d 4 ) [25]), κ can be as large as d in the worst-case. Of course, to make a fair comparison with existing projection-free algorithms, we also need to take computational complexity into account. This is what we do next.

Computational cost. The computational cost in (15) should be compared with that of existing projection-free algorithms. For linear optimization-based projection-free algorithms, the computational cost after T rounds is typically of order C lin ⋅ T , where C lin is the cost of performing linear optimization over K which, for a polytope K , reduces to solving a linear program. Using state-of-the-art interior point methods for solving such a linear program would cost C lin ≤ ̃ O ( √ d ⋅ C sys ) ; see e.g. [22]. Thus, linear optimization-based projection-free algorithms 5 can have a cost that is a factor √ d worse than that of BARONS in the setting of Corollary 1. On the other hand, separation/membership-based algorithms, the computational cost scales with O (C sep ⋅ T ) after T rounds, where C sep is the cost of performing separation for the set K . For a general polytope in R d with m constraints, we have C sep ≤ O ( md ) , which may be smaller than C sys (the latter can be as large as md ω -1 in the worse case;see (12)). Here, it may be more appropriate to compare against the computational guarantee of BARONS given in Corollary 2; by (18), we have that for T ≥ d ω -2 √ m , the computational cost of BARONS in the setting of the corollary is dominated by ( md + d 2 ) ⋅ T , which is comparable to that of existing separation-based algorithms.

4.2 Implications for Non-Lipschitz Losses

Another advantage BARONS has over projection-free, and even projection-based, algorithms is that it has a regret bound that scales with a bound on the local norms of the gradients-see (13). We now showcase two online learning settings where this leads to non-trivial performance and computational improvements over existing OCO algorithms.

Online Portfolio Selection [5]. The portfolio selection problem is a classical online learning problem where the gradients of the losses can be unbounded. In this paragraph, we demonstrate how the guarantee of BARONS in Corollary 1 leads to a non-trivial guarantee for this setting both in terms of regret and computational complexity. In the online portfolio setting, at each round t , a learner (algorithm) chooses a distribution w t ∈ ∆ d over a fixed set of d portfolios. Then, the environment reveals a return vector r t ∈ R d ≥ 0 , and the learner suffers a loss

The goal of the learner is to minimize the regret Reg T ( w ) ∶= ∑ T t = 1 ( ℓ t ( w t ) -ℓ t ( w )) after T ≥ 1 rounds. For this problem, it is known that a logarithmic regret is achievable, but the specialized algorithms that achieve this have a computational complexity that scales with min ( d 3 T,d 2 T 2 ) [5, 27, 32, 41, 17]. On the other hand, applying the generic Online Gradient Descent or the Online Newton Step to this problem leads to regret bounds that scale with the maximum norm of the gradient (which can be unbounded). Instantiating the guarantees of BARONS in Corollary 1 with Φ set to the standard log-barrier for the simplex 6 , in particular the bound in (13), to the online portfolio selection problem leads to an ̃ O ( √ dT ) regret bound, which does not depend on the norm of the observed gradients. Furthermore, we have C sys ≤ O ( d ) , and so by (15) the computational complexity is essentially O ( d 2 T ) after T rounds. Technically, the bound in (13) is only against comparators in the restricted set K c . However, by setting c = 1 / T , it possible to extend this guarantee to all comparators in int K as explained in Remark 1, since the losses in this case satisfy (6) [27, Lemma 10].

Linear prediction with the log-loss. Another classical online learning problem with unbounded gradients is that of linear prediction with the log-loss [36]. For this problem, at each round t , the learner receives a feature vector x t ∈ X ⊆ R d , outputs w t ∈ W ⊆ R d , then observes label y t ∈ Y ∶= {1 , 1 } and suffers loss

In the settings, where (X , W) = ( ∆ d , B ∞ ( 1 )) and (X , W) = (B ∞ ( 1 ) , ∆ d ) , we have that ∥∇ ℓ t ( w )∥ ∇ -2 Φ ( w ) ≤ O ( 1 ) for all w ∈ int W , where Φ is set to the corresponding log-barrier for W . Thus, instantiating Corollary 1 (in particular (13)) in this setting implies that BARONS achieves a regret bound of the form:

and has computational complexity bounded by ̃ O ( d 2 T ) , as long as T ≥ d . Again, we emphasize that the bound in (19) does not depend on the norm of the gradients, which may be unbounded.

Finally, we note that there exist a few specialized algorithms that provide sublinear regret bounds for non-lipschitz losses. This includes, for example, the Soft-Bayes algorithm [34]. However, this algorithm is specialized to the log-loss with a particular dependence on the predictions, and it is not clear, for example, what regret bound it would have in the linear prediction setting and other similar settings with non-Lipschitz losses.

Acknowledgments and Disclosure of Funding

ZMacknowledges support from the ONR through awards N00014-20-1-2336 and N00014-20-1-2394.

References

Jacob Abernethy, Kevin A Lai, Kfir Y Levy, and Jun-Kun Wang. Faster rates for convex-concave games. In Conference On Learning Theory , pages 1595-1625. PMLR, 2018.

Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. 2009.

Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

Michael B Cohen, Yin Tat Lee, and Zhao Song. Solving linear programs in the current matrix multiplication time. Journal of the ACM (JACM) , 68(1):1-39, 2021.

Thomas M Cover. Universal portfolios. Mathematical Finance , 1(1):1-29, 1991.

Ashok Cutkosky. Anytime online-to-batch, optimism and acceleration. In International Conference on Machine Learning , pages 1446-1454. PMLR, 2019.

Ashok Cutkosky, Harsh Mehta, and Francesco Orabona. Optimal stochastic nonsmooth non-convex optimization through online-to-non-convex conversion. arXiv preprint arXiv:2302.03775 , 2023.

Maryam Fazel, Yin Tat Lee, Swati Padmanabhan, and Aaron Sidford. Computing lewis weights to high precision. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 2723-2742. SIAM, 2022.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, 2005.

Dan Garber and Elad Hazan. A linearly convergent variant of the conditional gradient algorithm under strong convexity, with applications to online and stochastic optimization. SIAM Journal on Optimization , 26(3):1493-1528, 2016.

Dan Garber and Ben Kretzu. New projection-free algorithms for online convex optimization with adaptive regret guarantees. In Conference on Learning Theory , pages 2326-2359. PMLR, 2022.

JL Goffin. Affine and projective transformations in nondifferentiable optimization. Trends in Mathematical Optimization , pages 79-91, 1988.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007.

Elad Hazan and Satyen Kale. Projection-free online learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1843-1850, 2012.

Elad Hazan and Edgar Minasyan. Faster projection-free online learning. In Conference on Learning Theory , pages 1877-1893. PMLR, 2020.

Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning , pages 427-435. PMLR, 2013.

R´ emi J´ ez´ equel, Dmitrii M Ostrovskii, and Pierre Gaillard. Efficient and near-optimal online portfolio selection. arXiv preprint arXiv:2209.13932 , 2022.

Ben Kretzu and Dan Garber. Revisiting projection-free online learning: the strongly convex case. In International Conference on Artificial Intelligence and Statistics , pages 3592-3600. PMLR, 2021.

Simon Lacoste-Julien and Martin Jaggi. On the global linear convergence of frank-wolfe optimization variants. Advances in neural information processing systems , 28, 2015.

Yin Tat Lee and Aaron Sidford. Path finding methods for linear programming: Solving linear programs in o (vrank) iterations and faster algorithms for maximum flow. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 424-433. IEEE, 2014.

Yin Tat Lee and Aaron Sidford. Efficient inverse maintenance and faster algorithms for linear programming. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 230-249. IEEE, 2015.

Yin Tat Lee and Aaron Sidford. Solving linear programs with sqrt (rank) linear system solves. arXiv preprint arXiv:1910.08033 , 2019.

Kfir Levy and Andreas Krause. Projection free online learning over smooth sets. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1458-1466. PMLR, 2019.

Ning Liu and Benjamin Grimmer. Gauges and accelerated optimization over smooth and/or strongly convex sets. arXiv preprint arXiv:2303.05037 , 2023.

L´ aszl´ o Lov´ asz and Santosh Vempala. Simulated annealing in convex bodies and an o*(n4) volume algorithm. Journal of Computer and System Sciences , 72(2):392-417, 2006.

Zhou Lu, Nataly Brukhim, Paula Gradu, and Elad Hazan. Projection-free adaptive regret with membership oracles. In International Conference on Algorithmic Learning Theory , pages 1055-1073. PMLR, 2023.

Haipeng Luo, Chen-Yu Wei, and Kai Zheng. Efficient online portfolio with logarithmic regret. Advances in neural information processing systems , 31, 2018.

Zakaria Mhammedi. Efficient projection-free online convex optimization with membership oracle. In Conference on Learning Theory , pages 5314-5390. PMLR, 2022.

Zakaria Mhammedi. Exploiting the curvature of feasible sets for faster projection-free online learning. arXiv preprint arXiv:2205.11470 , 2022.

Zakaria Mhammedi and Khashayar Gatmiry. Quasi-newton steps for efficient online expconcave optimization. arXiv preprint arXiv:2211.01357 , 2022.

Zakaria Mhammedi and Alexander Rakhlin. Damped online newton step for portfolio selection. In Conference on Learning Theory, 2-5 July 2022, London, UK , volume 178, pages 5561-5595. PMLR, 2022.

Zakaria Mhammedi and Alexander Rakhlin. Damped online newton step for portfolio selection. In Conference on Learning Theory , pages 5561-5595. PMLR, 2022.

Yurii Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018.

Laurent Orseau, Tor Lattimore, and Shane Legg. Soft-bayes: Prod for mixtures of experts with log-loss. In International Conference on Algorithmic Learning Theory , pages 372-399. PMLR, 2017.

Fabian Pedregosa, Geoffrey Negiar, Armin Askari, and Martin Jaggi. Linearly convergent frank-wolfe with backtracking line-search. In International conference on artificial intelligence and statistics , pages 1-10. PMLR, 2020.

Alexander Rakhlin and Karthik Sridharan. Sequential probability assignment with binary alphabets and large classes of experts. arXiv preprint arXiv:1501.07340 , 2015.

Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and trends in Machine Learning , 4(2):107-194, 2011.

Tomasz Tkocz. Asymptotic convex geometry lecture notes. 2018.

Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 259-278. SIAM, 2020.

Jan van den Brand, Yin Tat Lee, Aaron Sidford, and Zhao Song. Solving tall dense linear programs in nearly linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 775-788, 2020.

Julian Zimmert, Naman Agarwal, and Satyen Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory , pages 182-226. PMLR, 2022.

A Self-concordance properties

Throughout, for a twice-differentiable function f ∶ int K → R , we let λ ( x, f ) ∶= ∥∇ f ( x )∥ ∇ -2 f ( x ) denote the Newton decrement of f at x ∈ int K .

Lemma 1. Let f ∶ int K → R be a self-concordant function with constant M f ≥ 1 . Further, let x ∈ int K and x f ∈ argmin x ∈K f ( x ) . Then, I) whenever λ ( x, f ) < 1 / M f , we have

and II) for any M ≥ M f , the Newton step x + ∶= x - ∇ -2 f ( x )∇ f ( x ) satisfies x + ∈ int K and λ ( x + , f ) ≤ Mλ ( x, f ) 2 /( 1 -Mλ ( x, f )) 2 .

Lemma 2. Let f ∶ int K → R be a self-concordant function with constant M f and x ∈ int K . Then, for any w such that r ∶= ∥ w -x ∥ ∇ 2 f ( x ) < 1 / M f , we have

The following result from [33, Theorem 5.1.5] will be useful to show that the iterates of algorithms are always in the feasible set.

Lemma 3. Let f ∶ int K → R be a self-concordant function with constant M f ≥ 1 and x ∈ int K . Then, E x ∶= { w ∈ R d ∶ ∥ w -x ∥ x < 1 / M f } ⊆ int K . Furthermore, for all w ∈ E x , we have

Finally, we will also make use of the following result due to [31]:

Lemma 4. Let f ∶ int K → R be a self-concordant function with constant M f > 0 . Then, for any x, w ∈ int K such that r ∶= ∥ x -w ∥ ∇ 2 f ( x ) < 1 / M f , we have

B Technical Lemmas

Our analysis relies on the crucial fact that the Newton decrement can be sufficiently decreased by taking a Newton step using only approximate gradients and Hessians. We state this fact next; the proof is in §D.1.

Lemma 5 (Decrease in the Newton decrement) . Let Φ be a self-concordant function over int K with constant M Φ > 0 , and let y ∈ R d be such that λ ( y, Φ ) ≤ 1 /( 40 M Φ ) . Further, let H ∈ R d × d and ̂ ∇ y ∈ R d be such that

for α < 1 / 5 . Then, for ˜ y + ∶= y -H -1 ∇ Φ ( y ) and y + ∶= y -H -1 ̂ ∇ y , we have

Next, we show that as long as the Newton decrement is small enough at the current iterate w t -1 , the 'intermediate' Newton iterates ( w t m ) remain close to the landmark point u t -1 ; this will be important for the proof of Theorem 5. The proof is in §D.2.

Lemma 6 (Invariance under Newton iterations) . Let Φ be a self-concordant function over int K with constant M Φ > 0 . Let b > 0 , m Newton ∶= Θ ( log 1 εM Φ ) , α ≤ 1 /( 1000 M Φ ) , ε < 1 /( 20000 M Φ ) , α = 0 . 001 , and η ≤ 1 /( 1000 M Φ b ) . Further, let ( w t , w t m , u t , g t , H t ) be as in Algorithm 1 with input ( η, ε, α, m Newton ) . Suppose that at round t -1 of Algorithm 1, we have

For t > 1 , if the sub-gradient g t -1 at round t -1 satisfies ∥ g t -1 ∥ ∇ -2 Φ ( w t -1 ) ≤ b , then

Furthermore, we have for all m ∈ [ m Newton ] :

where w ⋆ t ∈ argmin w ∈K Φ t ( w ) is the optimum solution of Φ t .

We note that we have not made an attempt to optimize over the constants in Lemma 6.

Lemma 7. Let Φ be a ( M Φ , ν ) -self-concordant barrier for K , and let w ⋆ ∈ argmin w ∈K Φ ( w ) . Further, suppose that the losses satisfy:

Then, for any w ∈ K , there exists ˜ w ∈ K 1 / T (where K c is as in (4) ) such that

Proof. Fix w ∈ K and define ˜ w = 1 T w ⋆ +( 1 -1 T ) w ∈ K 1 / T . Then, by (28), we have, for all t ∈ [ T ] ,

Summing this over t = 1 , . . . , T leads to the desired result.

C Proofs of the Main Results

Next, we present the proof of Theorem 5.

C.1 Proof of Theorem 5

Proof. The proof consists of three parts: I) First, we show that BARONS keeps the Newton decrements λ ( w t , Φ t ) , t ≥ 1 , small-this is the main invariant of BARONS ; II) Then, we bound the regret of BARONS using this invariant and the results of Lemma 6; III) Finally, we bound the runtime of BARONS .

Bounding the Newton decrements. We will show that the Newton decrements satisfy

for all s ≥ 1 . We will show (31) by induction over t ≥ 1 .

Base case. The base case follows by the facts that w 1 ∈ argmin w ∈K Φ ( w ) , Φ 1 ≡ Φ and that the Newton decrement is zero at the minimizer.

Induction step. Suppose that (31) holds with s = t -1 for some t ≥ 1 . We will show that it holds for s = t . First, note that by the update rule of landmark (see Lines 12 and 17 of Alg. 1), we have that

where H t -1 = O hess α ( u t -1 ) . Thus, by the choice of α in the theorem's statement, we have

This, combined with the fact that (31) holds with s = t -1 (the induction hypothesis) implies that the conditions of Lemma 6 are satisfied. This in turn implies

where ( a ) follows by the fact that w t = w m Newton t (by definition; see Algorithm 1) and ( b ) follows by the choice of m Newton in the theorem's statement. This shows that (31) holds for s = t and concludes the induction.

Bounding the regret. To bound the regret of BARONS , we make use of the FTRL iterates { w ⋆ t } , which are given by w ⋆ t ∈ argmin w ∈K Φ t ( w ) : By Lemma 6, we have that for all t ∈ [ T ] ,

where the last inequality follows by the choice of m Newton = Θ ( log ( 1 /( εM Φ ))) in the theorem's statement. Using this and H¨ older's inequality, we now bound the sum of linearized losses of the algorithm in terms of the sum of linearized losses with respect to { w ⋆ t } :

Now, by (24) in Lemma 6 (which holds due to (32) and (31) with s = t -1 as we showed in the prequel), we have 1 2 ∇ 2 Φ ( w t ) ≼ ∇ 2 Φ ( w ⋆ t ) , which implies that ∥ g t ∥ ∇ -2 Φ ( w ⋆ t ) ≤ 2 ∥ g t ∥ ∇ -2 Φ ( w t ) . Combining this with (33) and (34), we get that

Now fix w ∈ K . Subtracting ∑ T t = 1 ⟨ w,g t ⟩ from both sides of (35) implies the following bound the regret of BARONS :

where the last inequality follows by the regret bound of FTRL (see e.g. [30]).

Bounding the run-time. Note that BARONS updates the landmark points on the rounds where ∥ u t -w t ∥ H t > 1 /( 41 M Φ ) . Now, by (27) in Lemma 6 (which holds due to (32) and (31) with s = t -1 as we showed in the prequel), we have

where the last inequality follows by the choice of m Newton in the theorem's statement. Hence, the quantity ∥ u t -w t ∥ H t increases each time by at most 2 η ∥ g t -1 ∥ ∇ -2 Φ ( w t -1 ) + O ( ε ) . Therefore, the number of times that the landmark u t changes is bounded by

Thus, the overall computational cost of recalculating the Hessians and their inverses at the landmark iterates is bounded by

where the multiplicative cost C hess α reflects the fact that the instance of BARONS in the theorem's statement needs 1 ± α accurate approximations of the Hessians and their inverses (in the sense of Definition 4) at the landmark iterates. Moreover, BARONS needs to compute an ε -approximate gradient of Φ at every point w t m for all t ∈ [ T ] and m ∈ [ m Newton ] . Thus, the cost of computing the gradients is C grad ε ⋅ T log 1 εM Φ . Finally, the matrix-vector product H -1 t ∇ Φ t ( w ) in BARONS costs O ( d 2 ) work, and so overall the computational cost is

C.2 Proof of Theorem 6

Proof. Note that without having any effect on the algorithm, we can add an arbitrary constant to the barrier Φ . Thus, without loss of generality, we assume Φ ( w ⋆ ) = 0 , which implies Φ ( w ) ≥ 0 , for all w ∈ K . We define the restricted comparator class

By [33, Corollary 5.3.3] and the fact that Φ is an ( M Φ , ν ) -self-concordant barrier for K , we have that

and so it suffices to bound the regret against comparators in ̃ K . Fix ˜ w ∈ ̃ K . Under the assumptions of the theorem, the preconditions of Theorem 5 are satisfied and so we have,

where in the last step we used the choices of η and ε in (5). Combining this with (36) implies the desired regret bound. The bound on the computational complexity follows immediately from Theorem 5, the fact that ∥ g t ∥ ∇ -2 Φ ( w t ) ≤ b , and the choices of η and ε in (5).

C.3 Proof of Theorem 7

Proof. Similar to the proof of Theorem 6, and without loss of generality, we assume that Ψ is zero at its minimum, i.e. Ψ ( w ⋆ ) = 0 . We define the restricted comparator class

By [33, Corollary 5.3.3] and the fact that Ψ is an ( M Ψ , ν ) -self-concordant barrier for K , we have that

On the other hand, by Lemma 7 we have that

Combining this with (38) implies that it suffices to bound the regret against comparators in ̃ K . Fix ˜ w ∈ ̃ K . Note that since Φ is equal to Ψ plus a quadratic, Φ is also a self-concordant function with

constant M Φ = M Ψ [33]. Thus, under the assumptions of the theorem the preconditions of Theorem 5 are satisfied and so we have,

Now, by the choice of Φ , we have that

Moreover, from the condition that K ⊆ B( R ) , (37) and the fact that Ψ ( w ⋆ ) = 0 , we have for all w ∈ ̃ K :

Plugging this into (39) and using that ˜ w ∈ K , we get

where in the last step we used the choices of η and ε in (8). Combining this with (38) implies the desired regret bound. The bound on the computational complexity follows from the computational complexity in Theorem 5 and the fact a gradient Oracle O grad ε ( Φ ) [resp. Hessian Oracle O grad α ( Ψ ) ] for Φ (⋅) = Ψ (⋅) + ν 2 R 2 ∥ ⋅ ∥ 2 can be implemented with one call to O grad ε ( Ψ ) [resp. O hess α ( Ψ ) ] plus d arithmetic operations.

D Proofs of the Technical Lemmas

D.1 Proof of Lemma 5

Proof. Throughout, we let h is the Newton step based on the exact gradient ∇ Φ ( y ) :

Recall that ˜ y + and y + from the lemma's statement satisfy

Bounding the Newton decrement at ˜ y + . First, we bound the Newton decrement at ˜ y + . By definition, the square of the Newton decrement at ˜ y + = y + h is

Now for the vector z defined below, we define the function F as

The partial derivative of F in direction h is given by

Now, by (21), we have

Thus, the first term on the right-hand side of (41) can be bounded as

Plugging this into (41) and using the definition z in (40), we obtain

Now, let y ( s ) ∶= y + sh and F ○ y ( s ) ∶= F ( y ( s )) . With this, we have

On the other hand, by Lemma 8 and our assumption on λ ( y, Φ ) , we have

Thus, by Lemma 1, we have

This, together with (45) also implies that

After rearranging, this becomes

Combining this with (44) and the fact that c ≤ 1 4 gives

Finally, by Lemma 9 and (46), we obtain the following bound on the right-hand side of (43):

Integrating this over s gives

where the last inequality follows by (44). Now, note that from (46) (with s = 1 ) and the assumption that λ ( w, Φ ) ≤ 1 /( 40 M Φ ) , we have

This implies

Plugging this into (47) and using (42), we get

Now, from the definition of z , we have

Thus, since α < 1 / 4 , we finally get

This proves the first part of the claim, i.e. (43).

Bounding the Newton decrement at y + . We now bound the Newton decrement at y + = y -H -1 ̂ ∇ y in terms of that of ˜ y + = y + h . Note that

On the other hand, from (64) and (45), we have

which implies that

Therefore,

Combining this with (48) and our assumption on ̂ ∇ y from (20) implies

Thus, by Lemma 1, we have

Since ε < 1 /( 40 M Φ ) , we get

Now, by Lemma 4 instantiated with x = ˜ y + and w = y + , we have

where in the last inequality we used (49) and the fact that ε ≤ 1 /( 40 M Φ ) .

Using the triangle inequality, we can bound the Newton decrement at y + as

where the last inequality follows by (50) and (51). This completes the proof.

D.2 Proof of Lemma 6

Proof. By definition of ( w t m ) in Algorithm 1, we have w 1 t = w t -1 and w t = w m Newton t . We show properties (23), (25), (26), and 27 using induction over m = 1 , . . . , m Newton .

Base case. We start with the base case; m = 1 . Note that from the assumption in (22) and definition of w 1 t , we have

Now, by definition of the Oracle O hess α and the fact that H t -1 = O hess α ( u t -1 ) (see Algorithm 1) with α = 0 . 001 , we have

Combining this with (52) implies property (25) for the base case. Furthermore, since w 1 t = w t -1 (by definition), (27) follows trivially for the base case.

Now, using that Φ t ( w ) = Φ t -1 ( w ) + ηg ⊺ t -1 w, we have

where the last inequality follows by (22) and the fact that ∥ g t -1 ∥ ∇ -2 Φ ( w t -1 ) ≤ b . This shows property (23) for the base case. Thus, by Lemma 1, we have, for w ⋆ t ∈ argmin w ∈K Φ t ( w ) ,

Now, combining (53) with the fact that ∥ u t -1 -w 1 t ∥ ∇ 2 Φ ( u t -1 ) = ∥ u t -1 -w t -1 ∥ ∇ 2 Φ ( u t -1 ) ≤ 1 /( 40 M Φ ) (see (22)) and Lemma 2, we obtain

Plugging (57) into (56), we get

Now, by the triangle inequality

where in the last inequality we used (22) and (58). Combining (59) with (53) and Lemma 1, we get

Combining Equations (60) and (57) implies property (24) for the base of Induction. Furthermore, note that from Lemma 1:

which shows property (26) for the base case.

Induction step. Now, assume that properties (23), (25), (26), and (27) hold for m ≥ 1 . We will show that these properties holds for m + 1 . From the hypothesis of induction, we have

which combined with (53) and Lemma 1 implies

Thus, by Lemma 5 (instantiated with c = 1 / 5 ) and the fact that λ ( w t m , Φ t ) ≤ 1 /( 40 M Φ ) (by the induction hypothesis), we get that for ˜ w m + 1 t ∶= w t m -H -1 t -1 ∇ Φ t ( w t m ) :

Again, by Lemma 5 with c = 1 / 5 , we have

By the induction hypothesis, we also have that λ ( w t m , Φ t ) ≤ ( 15 16 ) m -1 λ ( w 1 t , Φ t ) + 500 ε . Combining this with (62), we get

This shows that (23) holds with m replaced by m + 1 .

Next, we show that (25) holds with m replaced by m + 1 . Combining (62) with Lemma 1 implies

This, together with (60) gives

Combining (59) with (64) gives:

which proves that (25) holds with m replaced by m + 1 .

Next, we show that (26) holds with m replaced by m + 1 . By (63) and Lemma 1, we have

where in the last inequality we used (55) and the bound on ε in the lemma's statement. This shows that (26) holds with m replaced by m + 1 .

Next, we show that (27) holds with m replaced by m + 1 . By plugging (60) into (65), we get

On the other hand, by (54), we have

Plugging this into (56) and using (55), we get

where the last inequality follows by (22). Combining (67) with (61) (instantiated with m = 1 ),

Now, by (66), (68), and the triangle inequality, we have

Via another triangle inequality, we get

This shows that (27) holds with m replaced by m + 1 . Finally, combining (60) with (61) implies

which completes the proof.

E Helper Lemmas

Lemma 8 (Bounding norm of the Newton step) . Let y ∈ K and H ∈ R d × d be such that ( 1 -c )∇ 2 Φ ( y ) ≼ H ≼ ( 1 + c )∇ 2 Φ ( y ) . Then, for h ∶= -H -1 ∇ Φ ( y ) , we have

Proof. We can write

For the middle matrix ∇ 2 Φ ( y ) 1 / 2 H -1 ∇ 2 Φ ( y ) 1 / 2 we have that

since ( 1 -c )∇ 2 Φ ( y ) ≼ H ≼ ( 1 + c )∇ 2 Φ ( y ) by assumption. Plugging this back into (69), we get

Lemma 9 (Cauchy-Schwarz) . If -B ≼ A ≼ B are symmetric matrices and B is PSD, then x ⊺ Ay ≤ ∥ x ∥ B ∥ y ∥ B .

Peer reviews:
comments: SUMMARY:
Building upon recent progress in projection-free OCO, this paper presents a new IPM-like algorithm attaining optimal regret with improved efficiency, which calculates Hessians only $O(\sqrt{T})$ number of times. The algorithm assumes access to a self-concordant function by gradient oracles and Hessian oracles.

STRENGTHS:
This paper makes a solid contribution on more efficient projection-free OCO, which admits optimal regret and better efficiency at the same time. Moreover, it also opens a new avenue of using IPM-like methods for projection-free OCO, besides classic LOO-based methods and SO/MO-based approaches developed recently.

WEAKNESSES:
There is an author's comment in line 160, which I suspect violates the double-blind rule. Though the contents are interesting, I regret to reject the paper for this reason.

I think the paper can benefit from a detailed comparison with the ONS algorithm in HAK07.

There are many typos, here are some that I caught:

Line 12: expansive - expensive (it appears multiple places)

Line 39: insure - ensure

Line 129: interior - interior of

Line 13 in the alg box: there should be another line of Set $H_{t+1}=H_t$?

---Update--- 

After a discussion with the AC, the comment is considered as a typo and I changed the score accordingly.

QUESTIONS:
Can you elaborate more on how to obtain Corollary 1?

LIMITATIONS:
There is an author's comment in line 160, which I suspect violates the double-blind rule.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-09

comments: SUMMARY:
This paper introduces new projection-free algorithms for online convex optimization, which utilize Newton iterates with a self-concordant barrier for the target set. The authors establish a state-of-the-art regret bound for this algorithm.

STRENGTHS:
Strengths And Weaknesses:
The paper presents a new approach for projection-free online convex optimization. The main advantage of the proposed algorithm is that it only requires computing a full inverse of the Hessian in a vanishing O(1/\sqrt{T}) fraction of each round. Additionally, for the case of a polytope with m constraints, their method exhibits lower per-iteration computational cost compared to linear optimization. Furthermore, their method achieves a better regret bound than existing works in this specific scenario.

However, I have some concerns that I would like the authors to address:

It would be beneficial to have a comparison of the gradient complexity between this method and other related works. Additionally, it would be valuable to provide further analysis of the computational complexity mentioned in line 208. Specifically, it would be helpful to clarify under which circumstances the gradient complexity dominates the Hessian complexity, and vice versa.

Regarding the computation complexity mentioned in line 208, there is a coefficient M_{\Phi}. In some cases, M_{\Phi} can be significantly larger than d. For instance, when considering a polytope with m constraints, M_{\Phi} is related to m, and m can be exponential in d.

In Theorem 6, the authors assume that the local norm of g_t is less than C for all w_t. This assumption is not trivial, as the Hessian of a self-concordant function is typically unbounded.

WEAKNESSES:
See above.

QUESTIONS:
See above.

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

comments: SUMMARY:
This paper proposes new projection-free algorithms for online convex optimization over a convex domaim $\mathcal{K}$. 
Specifically, this paper proposes efficient Newton iterations to obtain projection-free online convex optimization.

STRENGTHS:
This paper proposes an efficient online Newton method which is computation efficient.

WEAKNESSES:
This paper seems only using existing method to the online Newton. Though the author claims that unlike the setting of [26], where it is possible to add quadratic terms to the barrier for additional stability, in our setting, we cannot do that wihout sacrificing performance in terms of reget, we could not find any theoretical hardness in this paper to overcome this problem.

QUESTIONS:
no

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a \"projection-free\" algorithm for online convex optimization. The proposed algorithm adopts a self-concordant barrier of the constraint set as the regularizer, automatically ensuring the feasibility of the actions. The proposed algorithm only requires computing the inverse of an approximate Hessian at some rounds, instead of all rounds, achieving a small overall complexity when, e.g., the constraint set is a polytope. \n\n---\n\nThe authors have addressed my questions. I keep the original rating.\n\nSTRENGTHS:\n1. The algorithm is new and the analysis is non-trivial. \n2. The algorithm achieves the currently best regret performance for online convex optimization on a polytope.\n\nWEAKNESSES:\n1. **Computation of approximate gradient and Hessian.** I wonder how reasonable the error tolerances in the definitions of the approximate gradient and Hessian are, regarding that the Hessian inverse can have arbitrarily small eigenvalues in general. Can the approximate gradient and Hessian easily computed in general? \n2. **Explanations about complexity claims.** The claim that the per-round complexity of the algorithm is cheaper than that of the linear optimization oracle and the complexity bound in Corollary 1 need some explanations. \n3. **Inconsistent names.** It is claimed that the Lewis-Weights barrier will be used in Section 1 and the title of Section 4, but Ln. 226 says the barrier is called the Lee-Sidford barrier. \n4. **Presentation of Algorithm 1.** In Ln. 12--14, $H_{t + 1}$ is not specified. \n5. **Typos.** \n    - Ln. 160: \"[zm:cite]\"\n    - Ln. 188: an -> and\n    - Ln. 189: turning -> tuning\n    - Ln. 190: \"additional assumptions on the sequence of losses.\": incomplete sentence. \n    - Ln. 199: if -> of\n    - Ln. 232: $\\tilde{O} (1)$ is not an appropriate use of the notation. \n    - Ln. 299: \"in the sense of (3) *and (3)*\"\n\nQUESTIONS:\nPlease address the weaknesses.\n\nLIMITATIONS:\nThis is a theory paper. The assumptions are explicitly stated. Other possible issues I have noticed have been pointed out in the weaknesses block.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}