Title: Data-Dependent Bounds for Online Portfolio Selection Without Lipschitzness and Smoothness
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online portfolio selection

small-loss bound

gradual-variation bound

second-order bound

optimistic FTRL with self-concordant regularizers
Abstract: This work introduces the first small-loss and gradual-variation regret bounds for online portfolio selection, marking the first instances of data-dependent bounds for online convex optimization with non-Lipschitz, non-smooth losses. 
The algorithms we propose exhibit sublinear regret rates in the worst cases and achieve logarithmic regrets when the data is "easy," with per-round time almost linear in the number of investment alternatives. 
The regret bounds are derived using novel smoothness characterizations of the logarithmic loss, a local norm-based analysis of following the regularized leader (FTRL) with self-concordant regularizers, which are not necessarily barriers, and an implicit variant of optimistic FTRL with the log-barrier.

Data-Dependent Bounds for Online Portfolio Selection Without Lipschitzness and Smoothness

Abstract

This work introduces the first small-loss and gradual-variation regret bounds for online portfolio selection, marking the first instances of data-dependent bounds for online convex optimization with non-Lipschitz, non-smooth losses. The algorithms we propose exhibit sublinear regret rates in the worst cases and achieve logarithmic regrets when the data is 'easy,' with per-round time almost linear in the number of investment alternatives. The regret bounds are derived using novel smoothness characterizations of the logarithmic loss, a local norm-based analysis of following the regularized leader (FTRL) with self-concordant regularizers, which are not necessarily barriers, and an implicit variant of optimistic FTRL with the log-barrier.

1 Introduction

Designing an optimal algorithm for online portfolio selection (OPS), with respect to both regret and computational efficiency, has remained a significant open problem in online convex optimization for over three decades 1 . OPS models long-term investment as a multi-round game between two strategic players-the market and the INVESTOR-thereby avoiding the need for hard-to-verify probabilistic models for the market. In addition to its implications for robust long-term investment, OPS is also a generalization of probability forecasting and universal data compression [5].

The primary challenge in OPS stems from the absence of Lipschitzness and smoothness in the loss functions. Consequently, standard online convex optimization algorithms do not directly apply. For example, standard analyses of online mirror descent (OMD) and following the regularized leader (FTRL) bound the regret by the sum of the norms of the gradients (see, e.g., the lecture notes by

Orabona [28] and Hazan [12]). In OPS, the loss functions are not Lipschitz, so these analyses do not yield a sub-linear regret rate. Without Lipschitzness, the self-bounding property of smooth functions enables the derivation of a 'small-loss' regret bound for smooth loss functions [33]. However, the loss functions in OPS are not smooth either.

The optimal regret rate of OPS is known to be O ( d log T ) , where d and T denote the number of investment alternatives and number of rounds, respectively. This optimal rate is achieved by Universal Portfolio [9, 10]. Nevertheless, the current best implementation of Universal Portfolio requires O ( d 4 T 14 ) per-round time [20], too long for the algorithm to be practical. Subsequent OPS algorithms can be classified into two categories.

Algorithms in the first category exhibit near-optimal ˜ O ( d ) per-round time and moderate ˜ O ( √ dT ) regret rates. This category includes the barrier subgradient method [26], SoftBayes [30], and LB-OMD [34].

Algorithms in the second category exhibit much faster ˜ O (poly( d ) polylog( T )) regret rates with much longer ˜ O (poly( d ) poly( T )) per-round time, which is, however, significantly shorter than the per-round time of Universal Portfolio. This category includes ADABARRONS [22], PAE+DONS [24], BISONS [38], and VB-FTRL [17].

The aforementioned results are worst-case and do not reflect how 'easy' the data is. For instance, if the price relatives of the investment alternatives remain constant over rounds, then a small regret is expected. Data-dependent regret bounds refer to regret bounds that maintain acceptable rates in the worst case and much better rates when the data is easy. In this work, we consider three types of data-dependent bounds.

A small-loss bound bounds the regret by the cumulative loss of the best action in hindsight.

A gradual-variation bound bounds the regret by the gradual variation (6) of certain characteristics of the loss functions, such as the gradients and price relatives.

A second-order bound bounds the regret by the variance or other second-order statistics of certain characteristics of the loss functions, such as the gradients and price relatives.

Few studies have explored data-dependent bounds for OPS. These studies rely on the so-called no-junk-bonds assumption [2], requiring that the price relatives of all investment alternatives are bounded from below by a positive constant across all rounds. Given this assumption, it is easily verified that the losses in OPS become Lipschitz and smooth. Consequently, the result of Orabona et al. [29] implies a small-loss regret bound; Chiang et al. [8] established a gradual-variation bound in the price relatives; and Hazan and Kale [14] proved a second-order bound also in the price relatives. These bounds are logarithmic in the number of rounds T in the worst cases and can be constant when the data is easy.

The no-junk-bonds assumption may not always hold. Hazan and Kale [14] raised the question of whether it is possible to eliminate this assumption. In this work, we take the initial step towards addressing the question. Specifically, we prove Theorem 1.1.

Theorem 1.1. In the absence of the no-junk-bonds assumption, two algorithms exist that possess a gradual-variation bound and a small-loss bound, respectively. Both algorithms attain O ( d polylog( T )) regret rates in the best cases and ˜ O ( √ dT ) regret in the worst cases, with ˜ O ( d ) per-round time.

Theorem 1.1 represents the first data-dependent bounds for OPS that do not require the no-junk-bonds assumption. To the best of our knowledge, this also marks the first data-dependent bounds for online convex optimization with non-Lipschitz non-smooth losses. In the worst cases, Theorem 1.1 ensures that both algorithms can compete with the OPS algorithms of the first category mentioned above. In the best cases, both algorithms achieve a near-optimal regret with a near-optimal per-round time, surpassing the OPS algorithms of the second category in terms of the computational efficiency. Table 1 in Appendix A presents a detailed summary of existing OPS algorithms in terms of the worst-case regrets, best-case regrets, and per-round time.

We also derived a second-order regret bound by aggregating a variant of optimistic FTRL with different learning rates. The interpretation of the result is not immediately clear. Therefore, we detail the result in Appendix I.

Technical Contributions. The proof of Theorem 1.1 relies on several key technical breakthroughs:

Theorem 3.2 provides a general regret bound for optimistic FTRL with self-concordant regularizers, which may not be barriers, and time-varying learning rates. The bound generalizes those of Rakhlin and Sridharan [31] and Zimmert et al. [38, Appendix H] and is of independent interest.

Existing results on small-loss and gradual-variation bounds assume the loss functions are smooth, an assumption that does not hold in OPS. In Section 4, we present Lemma 4.3 and Lemma 4.7, which serve as local-norm counterparts to the Lipschitz gradient condition and the self-bounding property of convex smooth functions, respectively.

To apply Theorem 3.2, the gradient estimates and iterates in optimistic FTRL must be computed concurrently. Consequently, we introduce Algorithm 2, a variant of optimistic FTRL with the log-barrier and validate its definition and time complexity.

The gradual-variation and small-loss bounds in Theorem 1.1 are achieved by two novel algorithms, Algorithm 3 and Algorithm 4, respectively. Both are instances of Algorithm 2.

Notations. For any natural number N , we denote the set { 1 , . . . , N } by [ N ] . The sets of nonnegative and strictly positive numbers are denoted by R + and R ++ , respectively. The i -th entry of a vector v ∈ R d is denoted by v ( i ) . The probability simplex in R d , the set of entry-wise non-negative vectors of unit ℓ 1 -norm, is denoted by ∆ d . We often omit the subscript for convenience. The closure and relative interior of a set X is denoted by cl X and ri X , respectively. The ℓ p -norm is denoted by ∥·∥ p . The all-ones vector is denoted by e . For any two vectors u and v in R d , their entry-wise product and division are denoted by u ⊙ v and u ⊘ v , respectively. For time-indexed vectors a 1 , . . . , a t ∈ R d , we denote the sum a 1 + · · · + a t by a 1: t .

2 Related Works

2.1 Log-Barrier for Online Portfolio Selection

All algorithms we propose are instances of optimistic FTRL with the log-barrier regularizer. The first use of the log-barrier in OPS can be traced back to the barrier subgradient method proposed by Nesterov [26]. Later, Luo et al. [22] employed a hybrid regularizer, which incorporated the log-barrier, in the development of ADA-BARRONS. This marked the first OPS algorithm with a regret rate polylogarithmic in T and an acceptable per-round time complexity of O ( d 2 . 5 T ) . Van Erven et al. [35] conjectured that FTRL with the log-barrier (LB-FTRL) achieves the optimal regret. The conjecture was recently refuted by Zimmert et al. [38], who established a regret lower bound for LB-FTRL. Jézéquel et al. [17] combined the log-barrier and volumetric barrier to develop VB-FTRL, the first algorithm with near-optimal regret and an acceptable per-round time complexity of O ( d 2 T ) . These regret bounds are worst-case and do not directly imply our results.

2.2 FTRL with Self-Concordant Regularizer

Abernethy et al. [1] showed that when the regularizer is chosen as a self-concordant barrier of the constraint set, the regret of FTRL is bounded by the sum of dual local norms of the gradients. Rakhlin and Sridharan [31] generalized this result for optimistic FTRL.

The requirement for the regularizer to be a barrier is restrictive. For instance, while the log-barrier is self-concordant, it is not a barrier of the probability simplex. To address this issue, van Erven et al. [35], Mhammedi and Rakhlin [24], and Jézéquel et al. [17] introduced an affine transformation such that, after the transformation, the log-barrier becomes a self-concordant barrier of the constraint set. Nonetheless, this reparametrization complicates the proofs.

Theorem 3.2 in this paper shows that optimistic FTRL with a self-concordant regularizer, without the barrier requirement , still satisfies a regret bound similar to that by Rakhlin and Sridharan [31]. The proof of Theorem 3.2 aligns with the analyses by Mohri and Yang [25], McMahan [23], and Joulani et al. [19] of FTRL with optimism and adaptivity, as well as the local-norm based analysis by Zimmert et al. [38, Appendix H].

In comparison, Theorem 3.2 generalizes the analysis of Zimmert et al. [38, Appendix H] for optimistic algorithms and time-varying learning rates; Theorem 3.2 differs from the analyses of Mohri and Yang

[25], McMahan [23], and Joulani et al. [19] in that they require the regularizer to be strongly convex, whereas the log-barrier is not.

2.3 Data-Dependent Bounds

The following is a summary of relevant literature on the three types of data-dependent bounds. For a more comprehensive review, readers may refer to, e.g., the lecture notes of Orabona [28].

Small-loss bounds, also known as L ⋆ bounds, were first derived by Cesa-Bianchi et al. [6] for online gradient descent for quadratic losses. Exploiting the self-bounding property, Srebro et al. [33] proved a small-loss bound for convex smooth losses. Orabona et al. [29] proved a logarithmic small-loss bound when the loss functions are not only smooth but also Lipschitz and exp-concave.

Chiang et al. [8] derived the first gradual-variation bound, bounding the regret by the variation of the gradients over the rounds. Rakhlin and Sridharan [31, 32] interpreted the algorithm proposed by Chiang et al. [8] as optimistic online mirror descent and also proposed optimistic FTRL with self-concordant barrier regularizers. Joulani et al. [18] established a gradual-variation bound for optimistic FTRL.

Cesa-Bianchi et al. [7] initiated the study of second-order regret bounds. Hazan and Kale [13] derived a regret bound characterized by the empirical variance of loss vectors for online linear optimization. In the presence of the no-junk-bonds assumption, Hazan and Kale [14] proved a regret bound for OPS characterized by the empirical variance of price relatives.

Except for those for specific loss functions, these data-dependent bounds assume either smoothness or Lipschitzness of the loss functions. Nevertheless, both assumptions are violated in OPS.

Recently, Hu et al. [16] established small-loss and gradual-variation bounds in the context of Riemannian online convex optimization. We are unaware of any Riemannian structure on the probability simplex that would render the loss functions in OPS geodesically convex and geodesically smooth. For instance, Appendix B shows that the loss functions in OPS are not geodesically convex on the Hessian manifold induced by the log-barrier.

3 Analysis of Optimistic FTRL with Self-Concordant Regularizers

This section presents Theorem 3.2, a general regret bound for optimistic FTRL with regularizers that are self-concordant but not necessarily barriers . This regret bound forms the basis for the analyses in the remainder of the paper and, as detailed in Section 2.2, generalizes the results of Rakhlin and Sridharan [31] and Zimmert et al. [38, Appendix H].

Consider the following online linear optimization problem involving two players, LEARNER and REALITY. Let X ⊆ R d be a closed convex set. At the t -th round,

first, LEARNER announces x t ∈ X ;

then, REALITY announces a vector v t ∈ R d ;

finally, LEARNER suffers a loss given by ⟨ v t , x t ⟩ .

For any given time horizon T ∈ N , the regret R T ( x ) is defined as the difference between the cumulative loss of LEARNER and that yielded by the action x ∈ X ; that is,

The objective of LEARNER is to achieve a small regret against all x ∈ X . Algorithm 1 provides a strategy for LEARNER, called optimistic FTRL.

We focus on the case where the regularizer φ is a self-concordant function.

Definition 3.1 (Self-concordant functions) . A closed convex function φ : R d → ( -∞ , ∞ ] with an open domain dom φ is said to be M -self-concordant if it is three-times continuously differentiable on dom φ and

Algorithm 1 Optimistic FTRL for Online Linear Optimization

Input: A sequence of learning rates { η t } ⊂ R ++ .

1: ˆ v 1 ← 0 .

2: x 1 ∈ arg min x ∈X η -1 0 φ ( x ) .

3: for all t ∈ N do

4: Announce x t and receive v t ∈ R d .

5: Choose an estimate ˆ v t +1 for v t +1 .

6: x t +1 ← arg min x ∈X ⟨ v 1: t , x ⟩ + ⟨ ˆ v t +1 , x ⟩ + η -1 t φ ( x ) .

7: end for

Suppose that ∇ 2 φ is positive definite at a point x . The associated local and dual local norms are given by ∥ v ∥ x := √ ⟨ v, ∇ 2 φ ( x ) v ⟩ and ∥ v ∥ x, ∗ := √ ⟨ v, ∇ -2 φ ( x ) v ⟩ , respectively. Define ω ( t ) := t -log(1 + t ) .

Theorem 3.2. Let φ be an M -self-concordant function such that X is contained in the closure of dom φ and min x ∈X φ ( x ) = 0 . Suppose that ∇ 2 φ ( x ) is positive definite for all x ∈ X ∩ dom φ and the sequence of learning rates { η t } is non-increasing. Then, Algorithm 1 satisfies

If in addition, η t -1 ∥ v t -ˆ v t ∥ x t , ∗ ≤ 1 / (2 M ) for all t ∈ N , then Algorithm 1 satisfies

The proof of Theorem 3.2 is deferred to Appendix D. It is worth noting that the crux of the proof lies in Lemma D.1; the remaining steps follow standard procedure.

4 'Smoothness' in Online Portfolio Selection

4.1 Online Portfolio Selection

Online Portfolio Selection (OPS) is a multi-round game between two players, say INVESTOR and MARKET. Suppose there are d investment alternatives. A portfolio of INVESTOR is represented by a vector in the probability simplex in R d , which indicates the distribution of INVESTOR's wealth among the d investment alternatives. The price relatives of the investment alternatives at the t -th round are listed in a vector a t ∈ R d + .

The game has T rounds. At the t -th round,

first, INVESTOR announces a portfolio x t ∈ ∆ ⊂ R d ;

then, MARKET announces the price relatives a t ∈ R d + ;

finally, INVESTOR suffers a loss given by f t ( x t ) , where the loss function f t is defined as

The objective of INVESTOR is to achieve a small regret against all portfolios x ∈ ∆ , defined as 2

In the context of OPS, the regret corresponds to the logarithm of the ratio between the wealth growth rate of INVESTOR and that yielded by the constant rebalanced portfolio represented by x ∈ ∆ .

Assumption 1. The vector of price relatives a t is non-zero and satisfies ∥ a t ∥ ∞ = 1 for all t ∈ N .

The assumption on ∥ a t ∥ ∞ does not restrict the problem's applicability. If the assumption does not hold, then we can consider another OPS game with a t replaced by ˜ a t := a t / ∥ a t ∥ ∞ and develop algorithms and define the regret with respect to ˜ a t . It is obvious that the regret values defined with a t and ˜ a t are the same.

The following observation, readily verified by direct calculation, will be useful in the proofs.

Lemma 4.1. The vector x ⊙ ( -∇ f t ( x )) lies in ∆ for all x ∈ ri ∆ and t ∈ N .

4.2 Log-Barrier

Standard online convex optimization algorithms, such as those in the lecture notes by Orabona [28] and Hazan [12], assume that the loss functions are either Lipschitz or smooth.

Definition 4.2 (Lipschitzness and smoothness) . A function φ is said to be Lipschitz with respect to a norm ∥·∥ if

for some L > 0 . It is said to be smooth with respect to the norm ∥·∥ if

for some L ′ > 0 , where ∥·∥ ∗ denotes the dual norm.

Given that ⟨ a t , x ⟩ can be arbitrarily close to zero on ∆ ∩ dom f t in OPS, it is well known that there does not exist a Lipschitz parameter L nor a smoothness parameter L ′ for all loss functions f t . Therefore, standard online convex optimization algorithms do not directly apply.

We define the log-barrier as 3

It is easily checked that the local and dual local norms associated with the log-barrier are given by

In the remainder of the paper, we will only consider this pair of local and dual local norms.

Note that Lipschitzness implies boundedness of the gradient. The following observation motivates the use of the log-barrier in OPS, showing that the gradients in OPS are bounded with respect to the dual local norms defined by the log-barrier.

Lemma 4.3. It holds that ∥∇ f t ( x ) ∥ x, ∗ ≤ 1 for all x ∈ ri ∆ and t ∈ N .

A similar result was proved by van Erven et al. [35, (2)]. We provide a proof of Lemma 4.3 in Section E for completeness.

The following fact will be useful.

Lemma 4.4 (Nesterov [27, Example 5.3.1 and Theorem 5.3.2]) . The log-barrier h and loss functions f t in OPS are both 1 -self-concordant.

4.3 'Smoothness' in OPS

Existing results on small-loss and gradual-variation bounds require the loss functions to be smooth. For example, Chiang et al. [8] exploited the definition of smoothness (1) to derive gradual-variation bounds; Srebro et al. [33] and Orabona et al. [29] used the 'self-bounding property,' a consequence of smoothness, to derive small-loss bounds.

Lemma 4.5 (Self-bounding property [33, Lemma 2.1]) . Let f : R d → R be an L -smooth convex function with dom f = R d . Then, ∥∇ f ( x ) ∥ 2 ∗ ≤ 2 L ( f ( x ) -min y ∈ R d f ( y )) for all x ∈ R d .

While the loss functions in OPS are not smooth, we provide two smoothness characterizations of the loss functions in OPS. The first is analogous to the definition of smoothness (1).

Lemma 4.6. Let f ( x ) = -log ⟨ a, x ⟩ for some a ∈ R d + . Under Assumption 1 on the vector a ,

where ⊙ denotes the entrywise product and ∥·∥ x and ∥·∥ y are the local-norms defined by the logbarrier (3) .

The second smoothness characterization is analogous to the self-bounding property (Lemma 4.5). For any x ∈ ri ∆ and v ∈ R d , define

where x ( i ) and v ( i ) denote the i -th entries of x and v , respectively.

Lemma 4.7. Let f ( x ) = -log ⟨ a, x ⟩ for some a ∈ R d + . Then, under Assumption 1 on the vector a , it holds that

where the notation e denotes the all-ones vector and ∥·∥ x, ∗ denotes the dual local norm defined by the log-barrier (3) .

Remark 4.8. The value α x ( v ) is indeed chosen to minimize ∥ v + αe ∥ 2 x, ∗ over all α ∈ R .

The proofs of Lemma 4.6 and Lemma 4.7 are deferred to Appendix E.

5 LB-FTRL with Multiplicative-Gradient Optimism

Define g t := ∇ f t ( x t ) . By the convexity of the loss functions, OPS can be reduced to an online linear optimization problem described in Section 3 with v t = g t and X being the probability simplex ∆ . Set the regularizer φ as the log-barrier (2) in Optimistic FTRL (Algorithm 1). By Lemma 4.4, for the regret guarantee in Theorem 3.2 to be valid, it remains to ensure that ˆ g t , the estimate of g t , is selected to satisfy η t -1 ∥ g t -ˆ g t ∥ x t , ∗ ≤ 1 / 2 for all t . However, as x t , which defines the dual local norm, depends on ˆ g t in Algorithm 1, selecting such ˆ g t is non-trivial.

To address this issue, we introduce Algorithm 2. This algorithm simultaneously computes the next iterate x t +1 and the gradient estimate ˆ g t +1 by solving a system of nonlinear equations (5). As we estimate x t +1 ⊙ g t +1 instead of g t +1 , we call the algorithm LB-FTRL with Multiplicative-Gradient Optimism. Here, LB indicates that the algorithm adopts the log-barrier as the regularizer.

Algorithm 2 LB-FTRL with Multiplicative-Gradient Optimism for OPS

Input: A sequence of learning rates { η t } ⊆ R ++ .

1: h ( x ) := -d log d -∑ d i =1 log x ( i ) .

3: x 1 ← arg min x ∈ ∆ η -1 0 h ( x ) .

2: ˆ g 1 := 0 .

4: for all t ∈ N do

5: Announce x t and receive a t .

6: g t := ∇ f t ( x t ) .

7: Choose an estimate p t +1 ∈ R d for x t +1 ⊙ g t +1 .

8: Compute x t +1 and ˆ g t +1 such that

9: end for

By the definitions of the dual local norm (3) and p t (5), we write

It suffices to choose p t such that η t -1 ∥ x t ⊙ g t -p t ∥ 2 ≤ 1 / 2 . Indeed, Algorithm 3 and Algorithm 4 correspond to choosing p t = x t -1 ⊙ g t -1 and p t = 0 , respectively. Algorithm 5 in Appendix I corresponds to choosing p t = (1 /η 0: t -2 ) ∑ t -1 τ =1 η τ -1 x τ ⊙ g τ .

Theorem 5.1 guarantees that x t and ˆ g t are well-defined and can be efficiently computed. Its proof and the computational details can be found in Appendix F.1.

Theorem 5.1. If η t p t +1 ∈ [ -1 , 0] d , then the system of nonlinear equations (5) has a solution. The solution can be computed in ˜ O ( d ) time.

Algorithm 2 corresponds to Algorithm 1 with v t = g t , ˆ v t = ˆ g t = p t ⊘ x t , and φ ( x ) = h ( x ) . Corollary 5.2 then follows from Theorem 3.2. Its proof can be found in Appendix F.2.

Corollary 5.2. Assume that the sequence { η t } is non-increasing and p t ∈ ( -∞ , 0] d for all t ∈ N . Under Assumption 1, Algorithm 2 satisfies

In addition, for any sequence of vectors { u t } such that η t -1 ∥ ( g t + u t ) ⊙ x t -p t ∥ 2 ≤ 1 / 2 and ⟨ u t , x t -x t +1 ⟩ = 0 for all t ∈ N , Algorithm 2 satisfies

Remark 5.3. The vectors u t are deliberately introduced to derive a small-loss bound for OPS.

6 Data-Dependent Bounds for OPS

6.1 Gradual-Variation Bound

We define the gradual variation as

where ∥·∥ ∗ denotes the dual local norm associated with the log-barrier. The definition is a local-norm analog to the existing one [8, 18], defined as ∑ T t =2 max x ∈ ∆ ∥∇ f t ( x ) -∇ f t -1 ( x ) ∥ 2 for a fixed norm ∥·∥ . Regarding Lemma 4.3, our definition appears to be a natural extension.

In this sub-section, we introduce Algorithm 3, LB-FTRL with Last-Multiplicative-Gradient Optimism, and Theorem 6.1, which provides the first gradual-variation bound for OPS. Algorithm 3 is an instance of Algorithm 2 with p 1 = 0 and p t = x t -1 ⊙ g t -1 for t ≥ 2 . Note that the learning rates specified in Theorem 6.1 do not require the knowledge of V T in advance and can be computed on the fly.

The proof of Theorem 6.1 can be found in Appendix G.

Theorem 6.1. Let η 0 = η 1 = 1 / (16 √ 2) and η t = √ d/ (512 d +2+ V t ) for t ≥ 2 . Then, Algorithm 3 satisfies

By the definition of the dual local norm (3) and Lemma 4.1,

√

As a result, the worst-case regret of Algorithm 3 is O ( dT log T ) , comparable to the regret bounds of the barrier subgradient method [26], Soft-Bayes [30], and LB-OMD [34] up to logarithmic factors. On the other hand, if the price relatives remain constant over rounds, then V T = 0 and R T = O ( d log T ) .

Algorithm 3 LB-FTRL with Last-Multiplicative-Gradient Optimism for OPS

Input: A sequence of learning rates { η t } ⊆ R ++ .

1: h ( x ) := -d log d -∑ d i =1 log x ( i ) .

3: x 1 ← arg min x ∈X η -1 0 h ( x ) .

2: p 1 ← 0 .

4: for all t ∈ N do

5: Announce x t and receive a t .

6: g t ←∇ f t ( x t ) .

7: p t +1 ← x t ⊙ g t .

8: Compute x t +1 and ˆ g t +1 such that

9: end for

Time Complexity. The vectors g t and p t +1 , as well as the the quantity V t , can be computed using O ( d ) arithmetic operations. By Theorem 5.1, the iterate x t +1 can be computed in ˜ O ( d ) arithmetic operations. Therefore, the per-round time of Algorithm 3 is ˜ O ( d ) .

6.2 Small-Loss Bound

In this sub-section, we introduce Algorithm 4, Adaptive LB-FTRL, and Theorem 6.2, the first smallloss bound for OPS. The algorithm is an instance of Algorithm 2 with p t = 0 . Then, ˆ g t +1 = 0 and x t +1 is directly given by Line 8 of Algorithm 4. Note that Theorem 5.1 still applies.

Algorithm 4 Adaptive LB-FTRL for OPS

The proof of Theorem 6.2 is provided in Appendix H.

Under Assumption 1,

Assuming T > d , the worst-case regret bound is O ( √ dT log d log T ) , also comparable to the regret bounds of the barrier subgradient method [26], Soft-Bayes [30], and LB-OMD [34] up to logarithmic factors. On the other hand, suppose that there exists an i ⋆ ∈ [ d ] such that a t ( i ⋆ ) = 1 for all t ∈ [ T ] . That is, the i ⋆ -th investment alternative always outperforms all the other investment alternatives. Then, L ⋆ T = 0 and R T = O ( d log 2 T ) .

Assumption 1 does not restrict the applicability of Theorem 6.2, as mentioned earlier. If the assumption does not hold, Theorem 6.2 is applied with respect to the normalized price relatives ˜ a t = a t / ∥ a t ∥ ∞ . In this case, L ⋆ T is defined with respect to { ˜ a t } .

Time Complexity. Since ∥ g t + α t e ∥ 2 x t , ∗ = ∥ x t ⊙ g t + α t x t ∥ 2 2 , it is obvious that computing α t , η t , and g 1: t can be done in O ( d ) arithmetic operations. By Theorem 5.1, the iterate x t +1 can be computed in ˜ O ( d ) arithmetic operations. Hence, the per-round time of Algorithm 4 is ˜ O ( d ) .

7 Concluding Remarks

We have presented Theorem 6.1 and Theorem 6.2, the first gradual-variation and small-loss bounds for OPS that do not require the no-junk-bonds assumption, respectively. The algorithms exhibit sublinear regrets in the worst cases and achieve logarithmic regrets in the best cases, with per-round time almost linear in the dimension. They mark the first data-dependent bounds for non-Lipschitz non-smooth losses.

A potential direction for future research is to extend our analyses for a broader class of online convex optimization problems. In particular, it remains unclear how to extend the two smoothness characterizations (Lemma 4.6 and Lemma 4.7) for other loss functions.

Orabona et al. [29] showed that achieving a regret rate of O ( d 2 + log L ⋆ T ) is possible under the no-junk-bonds assumption, where L ⋆ T denotes the cumulative loss of the best constant rebalanced portfolio. This naturally raises the question: can a similar regret rate be attained without relying on the no-junk-bonds assumption? If so, then the regret rate will be constant in T in the best cases and logarithmic in T in the worst cases. However, considering existing results in probability forecasting with the logarithmic loss [5, Chapter 9]-a special case of OPS without the no-junk-bonds assumption-such a data-dependent regret rate seems improbable. Notably, classical rate-optimal algorithms for probability forecasting with the logarithmic loss, such as Shtarkov's minimax-optimal algorithm, the Laplace mixture, and the Krichevsky-Trofimov mixture, all achieve logarithmic regret rates for all possible data sequences [5, Chapter 9].

Zhao et al. [37] showed that for Lipschitz and smooth losses, an algorithm with a gradual-variation bound automatically achieves a small-loss bound. Generalizing their argument for non-Lipschitz non-smooth losses is a natural direction to consider.

Acknowledgments and Disclosure of Funding

The authors are supported by the Young Scholar Fellowship (Einstein Program) of the National Science and Technology Council of Taiwan under grant number NSTC 112-2636-E-002-003, by the 2030 Cross-Generation Young Scholars Program (Excellent Young Scholars) of the National Science and Technology Council of Taiwan under grant number NSTC 112-2628-E-002-019-MY3, and by the research project 'Pioneering Research in Forefront Quantum Computing, Learning and Engineering' of National Taiwan University under grant number NTU-CC-112L893406.

References

Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Interior-Point Methods for FullInformation and Bandit Online Learning. IEEE Trans. Inf. Theory , 58(7):4164-4175, 2012.

Amit Agarwal, Elad Hazan, Satyen Kale, and Robert E. Schapire. Algorithms for Portfolio Management Based on the Newton Method. In Proc. 23rd Int. Conf. Machine Learning , pages 9-16, 2006.

Heinz H. Bauschke, Jérôme Bolte, and Marc Teboulle. A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications. Math. Oper. Res. , 42(2): 330-348, 2017.

Nicolas Boumal. An introduction to optimization on smooth manifolds . Cambridge University Press, 2023.

Nicoló Cesa-Bianchi and Gabor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Nicoló Cesa-Bianchi, Philip M. Long, and Manfred K. Warmuth. Worst-Case Quadratic Loss Bounds for Prediction Using Linear Functions and Gradient Descent. IEEE Trans. Neural Netw. , 7(3):604-619, 1996.

Nicoló Cesa-Bianchi, Yishay Mansour, and Gilles Stoltz. Improved second-order bounds for prediction with expert advice. Mach. Learn. , 66:321-352, 2007.

Chao-Kai Chiang, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, and Shenghuo Zhu. Online Optimization with Gradual Variations. In Proc. 25th Annu. Conf. Learning Theory , volume 23, pages 6.1-6.20, 2012.

Thomas M Cover. Universal portfolios. Math. Financ. , 1991.

Thomas M. Cover and Erick Ordentlich. Universal portfolios with side information. IEEE Trans. Inf. Theory , 42(2):348-363, 1996.

H. Gzyl and F. Nielsen. Geometry of the probability simplex and its connection to the maximum entropy method. J. Appl. Math., Stat. Inf. , 16(1):25-35, 2020.

Elad Hazan. Introduction to Online Convex Optimization . The MIT Press, 2022.

Elad Hazan and Satyen Kale. Extracting certainty from uncertainty: Regret bounded by variation in costs. Mach. Learn. , 80:165-188, 2010.

Elad Hazan and Satyen Kale. An Online Portfolio Selection Algorithm with Regret Logarithmic in Price Variation. Math. Financ. , 25(2):288-310, 2015.

David P. Helmbold, Robert E. Schapire, Yoran Singer, and Manfred K. Warmuth. On-Line Portfolio Selection Using Multiplicative Updates. Math. Financ. , 8(4):325-347, 1998.

Zihao Hu, Guanghui Wang, and Jacob Abernethy. Minimizing Dynamic Regret on Geodesic Metric Spaces. arXiv preprint arXiv:2302.08652 , 2023.

Rémi Jézéquel, Dmitrii M. Ostrovskii, and Pierre Gaillard. Efficient and Near-Optimal Online Portfolio Selection. arXiv preprint arXiv:2209.13932 , 2022.

Pooria Joulani, András György, and Csaba Szepesvári. A Modular Analysis of Adaptive (Non-) Convex Optimization: Optimism, Composite Objectives, and Variational Bounds. In Proc. 28th Int. Conf. Algorithmic Learning Theory , volume 76, pages 681-720, 2017.

Pooria Joulani, András György, and Csaba Szepesvári. A modular analysis of adaptive (non-) convex optimization: Optimism, composite objectives, variance reduction, and variational bounds. Theor. Comput. Sci. , 808:108-138, 2020. Special Issue on Algorithmic Learning Theory.

A. Kalai and S. Vempala. Efficient Algorithms for Universal Portfolios. In Proc. 41st Annu. Symposium on Foundations of Computer Science , pages 486-491, 2000.

Haihao Lu, Robert M. Freund, and Yurii Nesterov. Relatively smooth convex optimization by first-order methods, and applications. SIAM J. Optim. , 28(1):333-354, 2018.

Haipeng Luo, Chen-Yu Wei, and Kai Zheng. Efficient Online Portfolio with Logarithmic Regret. In Adv. Neural Information Processing Systems , volume 31, 2018.

H. Brendan McMahan. A survey of Algorithms and Analysis for Adaptive Online Learning. J. Mach. Learn. Res. , 18(90):1-50, 2017.

Zakaria Mhammedi and Alexander Rakhlin. Damped Online Newton Step for Portfolio Selection. In Proc. 35th Annu. Conf. Learning Theory , 2022.

Mehryar Mohri and Scott Yang. Accelerating Online Convex Optimization via Adaptive Prediction. In Proc. 19th Int. Conf. Artificial Intelligence and Statistics , volume 51, pages 848-856, 2016.

Yurii Nesterov. Barrier subgradient method. Math. Program., Ser. B , 127:31-56, 2011.

Yurii Nesterov. Lectures on Convex Optimization . Springer, second edition, 2018.

Francesco Orabona. A Modern Introduction to Online Learning. arXiv preprint arXiv:1912.13213 , 2022.

Francesco Orabona, Nicoló Cesa-Bianchi, and Claudio Gentile. Beyond Logarithmic Bounds in Online Learning. In Proc. 15th Int. Conf. Artificial Intelligence and Statistics , volume 22, pages 823-831, 2012.

Laurent Orseau, Tor Lattimore, and Shane Legg. Soft-Bayes: Prod for Mixtures of Experts with Log-Loss. In Proc. 28th Int. Conf. Algorithmic Learning Theory , volume 76, pages 372-399, 2017.

Alexander Rakhlin and Karthik Sridharan. Online Learning with Predictable Sequences. In Proc. 26th Annu. Conf. Learning Theory , pages 993-1019. PMLR, 2013.

Alexander Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. In Adv. Neural Information Processing Systems 26 , 2013.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, Low Noise and Fast Rates. In Adv. Neural Information Processing Systems , volume 23, 2010.

Chung-En Tsai, Hao-Chung Cheng, and Yen-Huan Li. Online Self-Concordant and Relatively Smooth Minimization, With Applications to Online Portfolio Selection and Learning Quantum States. In Proc. 34th Int. Conf. Algorithmic Learning Theory , 2023.

Tim van Erven, Dirk Van der Hoeven, Wojciech Kotłowski, and Wouter M. Koolen. Open Problem: Fast and Optimal Online Portfolio Selection. In Proc. 33rd Annu. Conf. Learning Theory , volume 125, pages 3864-3869, 2020.

V. Vovk. A Game of Prediction with Expert Advice. J. Comput. Syst. Sci. , 56(2):153-173, 1998.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and nonstationarity: Problem-dependent dynamic regret for online convex optimization. arXiv preprint arXiv:2112.14368 , 2021.

Julian Zimmert, Naman Agarwal, and Satyen Kale. Pushing the Efficiency-Regret Pareto Frontier for Online Learning of Portfolios and Quantum States. In Proc. 35th Annu. Conf. Learning Theory , 2022.

A A Summary of OPS Algorithms

B Loss Functions in OPS are not Geodesically Convex

Let h ( x ) := -d log d -∑ d i =1 log x ( i ) be the log-barrier. Let ( M , g ) be the Hessian manifold induced by the log-barrier, where M = ri ∆ and g is the Riemannian metric defined by ⟨ u, v ⟩ x := ⟨ u, ∇ 2 h ( x ) v ⟩ .

Lemma B.1 (Gzyl and Nielsen [11, Theorem 3.1]) . Let x, y ∈ ri ∆ . The geodesic γ : [0 , 1] → ri ∆ connecting x, y is given by

where γ ( t )( i ) denotes the i -th entry of the vector γ ( t ) ∈ ri ∆ .

The proposition below shows that the loss functions in OPS are not geodesically convex on ( M , g ) in general.

Proposition B.2. Let a = (2 , 1) ∈ R 2 ++ . The function f ( x ) = -log ⟨ a, x ⟩ is not geodesically convex on ( M , g ) .

Proof. It suffices to show that f ◦ γ : [0 , 1] → R is not a convex function [4, Definition 11.3]. Let x = (1 / 2 , 1 / 2) and y = (1 / (1 + e) , e / (1 + e)) . By Lemma B.1, the geodesic connecting x, y is γ ( t ) = (1 / (1 + e t ) , e t / (1 + e t )) . We write

showing that f ◦ γ is not convex. The proposition follows.

C Properties of Self-Concordant Functions

This section reviews properties of self-concordant functions relevant to our proofs. Readers are referred to the book by Nesterov [27] for a complete treatment.

Throughout this section, let φ be a self-concordant function (Definition 3.1) and ∥·∥ x be the associated local norm, i.e.,

Self-concordant functions are neither smooth nor strongly convex in general. The following theorem indicates that nevertheless, they are locally smooth and strongly convex.

Theorem C.1 (Nesterov [27, Theorem 5.1.7]) . For any x, y ∈ dom φ such that r := ∥ y -x ∥ x < 1 /M ,

Define ω ( t ) := t -log(1 + t ) . Let ω ∗ be its Fenchel conjugate, i.e., ω ∗ ( t ) = -t -log(1 -t ) . Theorem C.2 (Nesterov [27, Theorem 5.1.8 and 5.1.9]) . For any x, y ∈ dom φ with r := ∥ y -x ∥

x ,

The following proposition bounds the growth of ω and ω ∗ .

Proposition C.3 (Nesterov [27, Lemma 5.1.5]) . For any t ≥ 0 ,

It is easily checked that both the loss functions in OPS and the log-barrier are not only self-concordant but also self-concordant barriers.

Definition C.4. A 1 -self-concordant function φ is said to be a ν -self-concordant barrier if

Lemma C.5 (Nesterov [27, (5.4.3)]) . The log-barrier (2) is a d -self-concordant barrier.

D Proof of Theorem 3.2 (Regret Analysis of Optimistic FTRL With Self-Concordant Regularizers)

D.1 Proof of Theorem 3.2

The proof of the following stability lemma is deferred to the next subsection.

Lemma D.1. Define F t +1 ( x ) := ⟨ v 1: t , x ⟩ + η -1 t φ ( x ) . Let { η t } be a non-increasing sequence of strictly positive numbers such that η t -1 ∥ v t -ˆ v t ∥ x t , ∗ ≤ 1 / (2 M ) . Then, Algorithm 1 satisfies

If r < 1 /M , then

For any t ∈ [0 , 1) ,

Define ˆ F t ( x ) := F t ( x ) + ⟨ ˆ v t , x ⟩ . Since ˆ v T +1 has no effect on the regret, we set ˆ v T +1 = 0 . By the strong FTRL lemma [28, Lemma 7.1],

The penultimate line above follows from the assumption that min x ∈X φ ( x ) = 0 and that x T +1 minimizes ˆ F T +1 on X ; the last line above follows from a telescopic sum and ˆ v 1 = ˆ v T +1 = 0 . The theorem then follows from Lemma D.1.

D.2 Proof of Lemma D.1

Since φ is M -self-concordant, the function η t -1 F t is M -self-concordant. By Theorem C.2,

Since x t minimizes F t ( x ) + ⟨ ˆ v t , x ⟩ on X , the optimality condition implies

Then, we obtain

Next, by the non-increasing of { η t } , φ ( x t +1 ) ≥ 0 , and the last inequality,

This proves the first inequality in the lemma.

By Hölder's inequality,

By the Fenchel-Young inequality,

Combining (9) and (10) yields

which proves the second inequality. The third inequality in the lemma then follows from the assumption that η t -1 ∥ v t -ˆ v t ∥ x t , ∗ ≤ 1 / (2 M ) and Proposition C.3.

E Proofs in Section 4 (Smoothness Characterizations)

E.1 Proof of Lemma 4.3

We write

E.2 Proof of Lemma 4.6

Define r := ∥ x -y ∥ y . Consider the following two cases.

(i) Suppose that r ≥ 1 / 2 . By Lemma 4.1,

(ii) Suppose that r < 1 / 2 . We write

showing that

Then [10, Lemma 1],

similarly,

We obtain

Since r < 1 / 2 ,

Therefore,

showing that ∥ x ⊙∇ f ( x ) -y ⊙∇ f ( y ) ∥ 2 < 4 ∥ x -y ∥ y .

Combining the two cases, we obtain ∥ x ⊙∇ f ( x ) -y ⊙∇ f ( y ) ∥ 2 ≤ 4 ∥ x -y ∥ y . Since x and y are symmetric, we also have ∥ x ⊙∇ f ( x ) -y ⊙∇ f ( y ) ∥ 2 ≤ 4 ∥ x -y ∥ x . This completes the proof.

E.3 Proof of Lemma 4.7

We will use the notion of relative smoothness.

Definition E.1 (Bauschke et al. [3], Lu et al. [21]) . A function f is said to be L -smooth relative to a function h if the function Lh -f is convex.

Lemma E.2 (Bauschke et al. [3, Lemma 7]) . Let f ( x ) = -log ⟨ a, x ⟩ for some non-zero a ∈ R d + . Then, f is 1 -smooth relative to the logarithmic barrier h .

Fix x ∈ ri ∆ . By Lemma E.2, the function h -f is convex and hence

where ri ∆ -x := { u -x | u ∈ ri ∆ } . By Lemma 4.4, Theorem C.2, and Proposition C.3, if ∥ v ∥ x ≤ 1 / 2 ,

Combining the two inequalities above yields

Since f ( x + v ) ≥ min y ∈ ∆ f ( y ) = 0 (Assumption 1) and ⟨ e, v ⟩ = 0 , we write

The left-hand side is continuous in v , so the condition that v ∈ ri ∆ -x can be relaxed to v ∈ ∆ -x := { u -x | u ∈ ∆ } . We get

We show that choosing

for any c ∈ (0 , 1 / 2] ensures that v ∈ ∆ -x and ∥ v ∥ ≤ 1 / 2

First, we check whether ∥ v ∥ x ≤ 1 / 2 . We write

where the last line follows from Remark 4.8 and Lemma 4.3.

Then, we check whether v ∈ ∆ -x or, equivalently, whether v + x ∈ ∆ . By the definition of α x ( ∇ f ( x )) (4), each entry of v is given by

where ∇ j f ( x ) is the j -th entry of ∇ f ( x ) . Obviously,

so ∑ i x ( i ) + v ( i ) = 1 . It remains to check whether x ( i ) + v ( i ) ≥ 0 for all i ∈ [ d ] . Notice that ∇ f ( x ) is entrywise negative. Then,

Then,

To show that the lower bound is non-negative, we write

If i = j , then max j x ( i ) x ( j ) ∑ k x ( k ) 2 ≤ 1 ; otherwise,

̸

Therefore,

̸

Plug the chosen v (12) into the inequality (11). We write

This completes the proof.

F Proof in Section 5 (LB-FTRL with Multiplicative-Gradient Optimism)

F.1 Proof of Theorem 5.1

We first check existence of a solution. For convenience, we omit the subscripts in p t +1 and η t and write g for g 1: t .

Proposition F.1. Assume that ηp ∈ [ -1 , 0] d . Define

Then, λ ⋆ exists and is unique. The pair ( x t +1 , ˆ g t +1 ) , defined by

solves the system of nonlinear equations (5) .

We will use the following observation [27, Theorem 5.1.1].

Lemma F.2. The function ψ is 1 -self-concordant.

Proof of Proposition F.1. By Lemma F.2 and the fact that dom ψ does not contain any line, the minimizer λ ⋆ exists and is unique [27, Theorem 5.1.13]; sine ηp ∈ [ -1 , 0] d and λ ∈ dom ψ , we have x t +1 ( i ) ≥ 0 ; moreover,

showing that x t +1 ∈ ∆ .

Then, we check whether ( x t +1 , ˆ g t +1 ) satisfies the two equalities (5). It is obvious that x t +1 ⊙ ˆ g t +1 = p . By the method of Lagrange multiplier, if there exists λ ∈ R such that

then x t +1 ∈ arg min x ∈ ∆ ⟨ g, x ⟩ + ⟨ ˆ g t +1 , x ⟩ + η -1 h ( x ) . It is easily checked that the above conditions are satisfied with λ = λ ⋆ and (14). The proposition follows.

We now analyze the time complexity. Nesterov [26, Section 7] [27, Appendix A.2] proved that when p = 0 , Newton's method for solving (13) reaches the region of quadratic convergence of the intermediate Newton's method [27, Section 5.2.1] after O (log d ) iterations. A generalization for possibly non-zero p is provided in Proposition F.3 below. The proof essentially follows the approach of Nesterov [27, Appendix A.2]. It is worth noting that while the function ψ is self-concordant, directly applying existing results on self-concordant minimization by Newton's method [27, Section 5.2.1] does not yield the desired O (log d ) iteration complexity bound.

Starting with an initial iterate λ 0 ∈ R , Newton's method for the optimization problem (13) iterates as

Proposition F.3. Assume that ηp ∈ [ -1 , 0] d . Define i ⋆ ∈ arg min i ∈ [ d ] ( -g ( i )) . Then, Newton's method with the initial iterate

enters the region of quadratic convergence of the intermediate Newton's method [27, Section 5.2.1] after O (log d ) iterations.

Firstly, we will establish Lemma F.4 and Lemma F.5. These provide characterizations of ψ ′ and λ t that are necessary for the proof of Proposition F.3.

Lemma F.4. The following hold.

(i) ψ ′ is a strictly increasing and strictly concave function.

(ii) ψ ′ ( λ ⋆ ) = 0 .

(iii) λ 0 ≤ λ ⋆ .

Proof of Lemma F.4. The first item follows by a direct calculation:

The second item has been verified in the proof of Proposition F.1. As for the third item, we write

The third item then follows from ψ ′ ( λ 0 ) ≤ ψ ′ ( λ ⋆ ) and the first item.

LemmaF.5. Let { λ t } be the sequence of iterates of Newton's method (15) . For any t ≥ 0 , ψ ′ ( λ t ) ≤ 0 and λ t ≤ λ ⋆ .

Proof of Lemma F.5. Recall that dom ψ = ( -g ( i ⋆ ) , ∞ ) . We proceed by induction. Obviously, λ 0 ∈ dom ψ . By Lemma F.4, the lemma holds for t = 0 . Assume that the lemma holds for some non-negative integer t . By the iteration rule of Newton's method (15) and the assumption that ψ ′ ( λ t ) ≤ 0 ,

which ensures that λ t +1 ∈ dom ψ . By Lemma F.4 (i) and the iteration rule of Newton's method (15), we have

By Lemma F.4 (i) and Lemma F.4 (ii), λ t +1 ≤ λ ⋆ . The lemma follows.

Proof of Proposition F.3. Define the Newton decrement as

By Lemma F.2, the region of quadratic convergence is [27, Theorem 5.2.2]

Define T := ⌈ (9 + 7 log 2 d ) / 4 ⌉ . By Lemma F.5, ψ ′ ( λ t ) ≤ 0 for all t ≥ 0 . If ψ ′ ( λ t ) = 0 for some 0 ≤ t ≤ T , then Newton's method reaches Q within T steps and the proposition follows. Now, let us assume that ψ ′ ( λ t ) < 0 for all 0 ≤ t ≤ T . By the concavity of ψ ′ (Lemma F.4 (i)),

Divide both sides by ψ ′ ( λ t -1 ) . By Lemma F.5 and the AM-GM inequality,

Then, for all 1 ≤ t ≤ T +1 , which further implies that

It remains to estimate ψ ′ ( λ 0 ) , ψ ′′ ( λ 0 ) , and ψ ′′ ( λ t ) . Given ηp ∈ [ -1 , 0] d and λ 0 + ηg ( i ) ≥ 1 , through direct calculation,

Since

by the Cauchy-Schwarz inequality,

Therefore, for 0 ≤ t ≤ T +1 ,

This implies that for all 0 ≤ t ≤ T +1 ,

In particular, δ ( λ T +1 ) < 1 / 2 , i.e., λ T +1 ∈ Q . This concludes the proof.

Proposition F.3 suggests that after O (log d ) iterations of Newton's method, we can switch to the intermediate Newton's method which exhibits quadratic convergence [27, Theorem 5.2.2]. Each iteration of both Newton's method and the intermediate Newton's method takes O ( d ) time. Therefore, it takes ˜ O ( d log d ) time to approximate λ ⋆ . Given λ ⋆ , both x t +1 and ˆ g t +1 can be computed in O ( d ) time, according to their respective definitions in Equation (14). Therefore, the overall time complexity of computing x t +1 and ˆ g t +1 is ˜ O ( d ) .

F.2 Proof of Corollary 5.2

By the definition of local norms (3), we write

By Lemma 4.4 and Theorem 3.2, if η t -1 ∥ ( g t + u t ) ⊙ x t -p t ∥ 2 ≤ 1 / 2 , then Algorithm 2 satisfies

For any x ∈ ∆ , define x ′ = (1 -1 /T ) x + e/ ( dT ) . Then,

and hence [22, Lemma 10]

G Proof of Theorem 6.1 (Gradual-Variation Bound)

G.1 Proof of Theorem 6.1

We will use the following lemma, whose proof is postponed to the end of the section.

Lemma G.1. Let { x t } be the iterates of Algorithm 2. If for all t ≥ 1 ,

We start with checking the conditions in Lemma G.1.

It is obvious that η t ≤ 1 / (16 √ 2) ≤ 1 / 6 .

By the definition of the dual local norm (3), the definition of ˆ g t , and Lemma 4.3,

Then, by the triangle inequality and Lemma 4.3,

It is obvious that the sequence { η t } is non-increasing. By the triangle inequality, Lemma 4.3 and the fact that √ a + b ≤ √ a + √ b for non-negative numbers a and b , we write

By Proposition C.3,

For t ≥ 2 , by Hölder's inequality and the definition of the dual local norm (3),

By the triangle inequality,

We bound the two terms separately. By the AM-GM inequality and Lemma 4.6, the first term is bounded by

By the AM-GM inequality, the second term is bounded by

Combine all inequalities above. We obtain

For t = 1 , by a similar argument,

By combining (17) and (18) and noticing that η t -1 η t ≤ 1 / 512 ,

Plug in the choice of learning rates into the regret bound. We obtain [28, Lemma 4.13]

G.2 Proof of Lemma G.1

Define

By the triangle inequality,

Define d 1 := ∥ x t -y t ∥ x t , d 2 := ∥ y t -z t ∥ y t , and d 3 := ∥ z t -x t +1 ∥ x t +1 . Suppose that d i ≤ 1 / 5 for all 1 ≤ i ≤ 3 , which we will verify later. By Theorem C.1,

We obtain

Then, the inequality (19) becomes

It is easily checked that the maximum of the right-hand side is attained at d 1 = d 2 = d 3 = 1 / 5 and the maximum value equals 269 / 320 . This proves the lemma.

Now, we prove that d i ≤ 1 / 5 for all 1 ≤ i ≤ 3 .

(Upper bound of d 1 .) By the optimality conditions of x t and y t ,

Sum up the two inequalities. We get

By Hölder's inequality, Lemma 4.4, and Theorem C.2, we obtain

Then, by the assumption (16), we write

(Upper bound of d 2 .) By the optimality conditions of y t and z t ,

Sum up the two inequalities. We get

By Definition C.4, Lemma C.5, and Theorem C.2,

Then, by the assumption (16), we write

(Upper bound of d 3 .) By the optimality conditions of z t and x t +1 ,

Sum up the two inequalities. We get

By Hölder's inequality, the assumption (16), and Theorem C.2, we obtain

Then, by the assumption (16), we write

H Proof of Theorem 6.2 (Small-Loss Bound)

Recall that α t minimizes ∥ g t + αe ∥ 2 x t , ∗ over all α ∈ R (Remark 4.8). We write

which, with the observation that η t ≤ 1 / 2 for all t ∈ N , implies that η t -1 ∥ g t + α t e ∥ x t , ∗ ≤ 1 / 2 . Then, since the all-ones vector e is orthogonal to the set ∆ -∆ , by Corollary 5.2 with p t = 0 and u t = α t e , we get

Plug in the explicit expressions of the learning rates η t . We write [28, Lemma 4.13]

The theorem then follows from Lemma 4.23 of Orabona [28].

I A Second-Order Regret Bound

In this section, we introduce AA + LB-FTRL with Average-Multiplicative-Gradient Optimism (Algorithm 6) that achieves a second-order regret bound. This bound is characterized by the variancelike quantity

For comparison, existing second-order bounds are characterized by the quantity

T times the empirical variance of the gradients (see, e.g., Section 7.12.1 in the lecture note of Orabona [28]). Given that every p ∈ R d can be expressed as p = x t ⊙ v for v = p ⊘ x t , and by the definition of the dual local norm (3), we obtain

showing that Q T is a local-norm analogue of V T .

The flexibility introduced by optimizing Q T over all v ∈ R d allows for a fast regret rate when the data is easy. In particular, Theorem I.2 in Section I.3 shows that when Q T ≤ 1 , then Algorithm 6 achieves a fast O ( d log T ) regret rate. However, while Q T looks reasonable as a mathematical extension of V T , the empirical variance interpretation for V T does not extend to Q T . Interpreting Q T or seeking a 'more natural' variant of Q T is left as a future research topic.

I.1 The Gradient Estimation Problem

To obtain a better regret bound by using Corollary 5.2, one has to design a good strategy of choosing p t . In Section 6.1, we have used p t = x t -1 ⊙ g t -1 . In this section, we consider a different approach.

Consider the following online learning problem, which we call the Gradient Estimation Problem . It is a multi-round game between LEARNER and REALITY. In the t -th round, LEARNER chooses a vector p t ∈ R d ; then, REALITY announces a convex loss function ℓ t ( p ) = η t -1 ∥ x t ⊙ g t -p ∥ 2 2 ; finally, LEARNER suffers a loss ℓ t ( p t ) .

I.2 LB-FTRL with Average-Multiplicative-Gradient Optimism

In the gradient estimation problem, the loss functions are strongly-convex. A natural strategy of choosing p t is then following the leader (FTL). In the t -th round, FTL suggests choosing

This strategy leads to Algorithm 5. Theorem I.1 provides a regret bound for Algorithm 5.

Algorithm 5 LB-FTRL with Average-Multiplicative-Gradient Optimism

Input: A sequence of learning rates { η t } t ≥ 0 ⊆ R ++ .

1: h ( x ) := -d log d -∑ d i =1 log x ( i ) .

2: x 1 ← arg min x ∈ ∆ η -1 0 h ( x ) .

3: for all t ∈ N do

4: Announce x t and receive a t .

5: g t := ∇ f t ( x t ) .

6: p t +1 ← (1 /η 0: t -1 ) ∑ t τ =1 η τ -1 x τ ⊙ g τ .

7: Solve x t +1 and ˆ g t +1 satisfying

8: end for

Theorem I.1. Assume that the sequence of learning rates { η t } is non-increasing and η 0 ≤ 1 / (2 √ 2) . Then, Algorithm 5 achieves

If η t = η ≤ 1 / (2 √ 2) is a constant, then

Proof. Since p t is a convex combination of x τ ⊙ g τ , by Lemma 4.1, p t ∈ -∆ . We restrict the action set of the gradient estimation problem to be -∆ . By the regret bound of FTL for strongly convex losses [28, Corollary 7.24],

Then, by Corollary 5.2,

The first bound in the theorem follows from

The second bound in the theorem follows from the inequality ∑ T t =1 η 2 / ( tη ) ≤ η (log T +1) .

Time Complexity. Suppose that a constant learning rate is used. Then, both g t and p t +1 can be computed in O ( d ) arithmetic operations. By Theorem 5.1, x t +1 can be computed in ˜ O ( d ) time. Hence, the per-round time of Algorithm 5 is ˜ O ( d ) .

I.3 AA + LB-FTRL with Average-Multiplicative-Gradient Optimism

The constant learning rate that minimizes the regret bound in Theorem I.1 is η = O ( √ d log T/Q T ) . However, the value of Q T is not unknown to INVESTOR initially.

In this section, we propose Algorithm 6, which satisfies a second-order regret bound without the need for knowing Q T in advance. Algorithm 6 uses multiple instances of Algorithm 5, which we call experts , with different learning rates. The outputs of the experts are aggregated by the Aggregating Algorithm (AA) due to Vovk [36].

Algorithm 6 AA + LB-FTRL with Average-Multiplicative-Gradient Optimism

Input: The time horizon T .

1: K := 1 + ⌈ log 2 T ⌉ .

2: for all k ∈ [ K ] do

3: w ( k ) 1 ← 1 .

4: q ( k ) ← 2 k .

6: Initialize Algorithm 5 with the constant learning rate η ( k ) as the k -th expert A k .

7: Obtain x ( k ) 1 from A k .

8: end for

10: for all t ∈ [ T ] do

11: Observe a t .

12: for all k ∈ [ K ] do

13: w ( k ) t +1 = w ( k ) t ⟨ a t , x ( k ) t ⟩ .

14: Feed a t into A k and obtain x ( k ) t +1 .

15: end for

16: Output x t +1 := 1 ∑ K k =1 w ( k ) t +1 ∑ K k =1 w ( k ) t +1 x ( k ) t +1 .

17: end for

The regret bound of Algorithm 6 is stated in Theorem I.2, which follows from the regret of AA and Theorem I.1.

Theorem I.2. Algorithm 6 with input T ∈ N satisfies

Remark I.3. Since Q T ≤ 2 T , the regret bound in Theorem I.2 is O ( √ dT log T ) in the worst case.

Proof. K = 1 + ⌈ log 2 T ⌉ is the number of experts in Algorithm 6. Let R ( k ) T be the regret of the k -th expert A k . Since the loss functions f t ( x ) = -log ⟨ a t , x ⟩ are 1 -mixable, by the regret bound of AA [36],

where R T and R ( k ) T denote the regret of Algorithm 6 and the regret of the k -th expert, respectively.

The quantity q ( k ) in Algorithm 6 is used as an estimate of Q T . Notice that Q T ≤ 2 T ≤ q ( K ) . If Q T ≥ 1 , then there exists k ⋆ ∈ [ K ] such that q ( k ⋆ -1) ≤ Q T ≤ q ( k ⋆ ) . By Theorem I.1 and some direct calculations,

If Q T < 1 , then by Theorem I.1,

The theorem follows by combining the inequalities above.

Time Complexity. In each round, Algorithm 6 requires implementing O (log log T ) copies of Algorithm 5. By Theorem 5.1, the per-round time is ˜ O ( d log log T ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the online portfolio selection problem. In this problem, one must allocate funds between d possible investment choices, with the goal of maximizing the total amount. In each round, the \"success\" of each choice is revealed, in the form of a ratio between new and old price, called price relative; these price relatives are adversarially chosen. The goal is to have low regret, in terms of the logarithm of total wealth, against any constant, fractional allocation of funds between the investment choices. \n\nIn general, where the number of rounds is T, existing work for this problem exhibits either polynomial running time in T and poly-logarithmic regret in T, or running time independent of T and regret polynomial in T (specifically, square root). \n \nThe paper considers a different approach, providing data-dependent bounds for the problem.\nThe paper presents three bounds w.r.t. the input: small-loss bound, w.r.t. the total loss of the optimal allocation; gradual-variation bound, w.r.t. the volatility of the gradients of the loss functions; and a second-order bound, w.r.t. some second-order statistics of the loss function. The first two appear in the body of the paper.\n\nThe paper presents an algorithm based on optimistic FTRL w.r.t. a log-barrier regularizer; the optimism refers to using an estimate for the upcoming loss function in choosing the allocation for the next time step. To my understanding, the algorithmic contribution of the paper is in the method for choosing the loss estimate. \n\nComments:\nLine 33: Maybe \"lower\" instead of \"faster\"\nLine 283: In the displayed equation, should be a_t instead of a\n\nSTRENGTHS:\nThe online portfolio selection is an interesting problem, and obtaining data-dependent bounds for this problem seems natural. \nIn that sense, the paper does a comprehensive job and provides bounds w.r.t. three parameters.\n\nWEAKNESSES:\nI have a concern regarding the small-loss bound, addressed in the questions for rebuttal.\n\nQUESTIONS:\nRegarding Assumption 1, the paper correctly states that this could be ensured by normalizing the price-relatives vector, without affecting regret. However, this does affect the total loss of the adversary, which comes into play in the small-loss bound in Theorem 6.2. If I understand correctly, the assumption is thus nontrivial to make, which weakens Theorem 6.2. Is this the case?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies online portfolio selection (OPS) problem and establishes regret bounds that is square-root dependent on some data-dependent quantities, namely, the cumulative loss of the best action or the variation of the gradients respectively, without lipschitzness or smoothness assumption. Although previous studies obtain regret bounds with logarithmic dependence on the same data-dependent factors, they require the lipschitzness assumptions, which this work circumvents by investigating the problem carefully and proposing alternative local-norm lemmas as substitutes for lipschitzness and smoothness conditions. Furthermore, this work designs a novel optimistic FTRL algorithm with a self-concordant function to cooperate with the local-norm techniques and validates that the prediction can be resolved in $\\tilde{O}(d)$ time pre round.\n\nSTRENGTHS:\n1.\tThis paper leverages the inherent structure of the problem and derives the first data-dependent regret bounds without lipschitzness or smoothness assumptions for the OPS setting.\n2.\tThe proposed optimistic FTRL algorithm is novel and interesting, which facilitates the application of local-norm techniques with optimism for this problem.\n3.\tThis paper is clearly written and presented well.\n\nWEAKNESSES:\nThe OPS problem is fundamentally challenging, and I am glad to witness advancements in this field even if the results have not yet reached an optimal level. Below are some questions that emerged when I am reviewing the paper, and I would be appreciated if the authors can provide some feedback.\n\n1.\tOrdinarily, one might anticipate that data-dependent bounds would demonstrate a marked advantage over minimax optimal results in the best-case scenarios, such as when $V_T = 0$ or $L_T^\\star = 0$. But, by Theorem 6.1, when $ V_T = 0$, the algorithm can only assure $O(d\\log T)$ regret bound, just matching the minimax optimal bound. By Theorem 6.2, $L_T^\\star = 0$ will imply an even less optimal bound than the minimax result. However, these results do attain Pareto frontier optimality when balancing the optimality and efficiency under the best cases. I advise the authors to elaborate further on the motivation and highlight the efficiency of the proposed algorithms to make the contributions clearer.\n2.\tAs the authors noted in line 242, the term \"implicit\" might potentially mislead readers since it is commonly associated with another class of algorithms [Kulis and Bartlett, 2010]. It may be beneficial to consider a different nomenclature for the algorithm. \n3.\tI’m interested in the relationship between two data-dependent bounds. Indeed, some earlier study (see Zhao et al., 2021, Theorem 6) posits that a gradual variation dynamic regret can imply a small-loss dynamic regret in the analysis, for general convex, non-negative, and smooth functions. This result is also true for static regret. A natural question that arises from this is whether it's feasible for one algorithm to concurrently secure guarantees for both the gradual-variation bound and small-loss bound.\n\nReferences:\n\nBrian Kulis and Peter L. Bartlett. Implicit Online Learning. ICML 2010.\nPeng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimization. ArXiv:2112.14368, 2021.\n\nQUESTIONS:\nSee comments above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents beyond-the-worst-case regret bounds for Online Portfolio Selection (OPS). In general online learning, beyond-the-worst-case bounds are established using structural assumptions on the loss functions, such as Lipschitzness and smoothness, but the loss functions in OPS are neither Lipschitz nor smooth, which is the main technical difficulty the paper addresses. To this end, local norm analogues of Lipschitzness and smoothness are established for OPS, and a generic optimistic FTRL algorithm with the log-barrier regularizer is proposed. Specializations of this algorithm achieve a gradient-variation-dependent bound and a $L^*$ bound, which are the first without the additional no junk bond assumption. In the worst case, these bounds match a type of classical regret-computation tradeoff. In better cases, these bounds are logarithmic in $T$, which is a substantial acceleration.\n\nSTRENGTHS:\n- Online portfolio selection is an iconic problem in online learning, with plenty of recent progresses in its worst case characterization. The paper introduces classical types of data dependent bounds to this problem, which has a very clear and natural motivation. \n\n- The challenge of non-Lipschitzness and non-smoothness is technically nontrivial. The solution relies on establishing local counterparts of these properties, which is interesting, and could be of broader applicability. \n\n- The specific data-dependent bounds are novel without the additional no junk bond assumption. \n\n- Related works are discussed in depth, which gives the new contributions a nice context.\n\nWEAKNESSES:\nOverall this is a good paper, and there isn't any major criticism I'd like to make. On less important issues, \n\n- The technical presentation of the paper could be improved. Currently there are quite a few of typos and unclear notations. For example, there's Algorithm 1 in the main paper, and an algorithm 1 in the appendix. $\\omega$ in Theorem 3.2 is not defined in the main paper. The notation $x(i)$ is used in line 201 but defined in line 227, ...\n\n- I think compared to data-dependent bounds in general online learning, the benefit of data-dependent bounds in OPS is a little less clear, particularly due to the existence of a computationally inefficient logarithmic regret algorithm. I would appreciate more discussions on computation, since the paper essentially considers a particular regret computation tradeoff. Is $\\tilde O(d)$ runtime and $\\tilde O (\\sqrt{dT})$ regret pareto-optimal for OPS in some sense? A numerical example would also be helpful, as it will make the computational savings over universal portfolio (and line 33 to 36) more clear.\n\nQUESTIONS:\n- Could the authors comment a bit more on the novelty of the smoothness characterizations (Lemma 4.6 and 4.7)? \n\n- I'm also generally wondering about the regret computation tradeoff in OPS. Are there lower bounds?\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies follow the regularized leader algorithm (FTRL) on the online portfolio selection problem without the assumption of no junk bonds. This makes the resulting loss function non-Lipschitz and non-smooth and makes analyzing the regularized follow the leader algorithm hard to analyze. The paper proposes using a self-concordant regularizer for the FTRL algorithm and proves regret bounds for the algorithm for a generic online convex optimization problem. They then use the result to propose two novel algorithms that have small-loss and gradual-variation regret bounds, respectively, for the online portfolio selection problem, which to the best of the author’s knowledge are the first regret bounds for non-Lipschitz and non-smooth losses.\n\nSTRENGTHS:\nThe paper seems significant as it provides a new result towards bounding regret for non-Lipschitz and non-smooth losses. The main contribution seems to originate from being able to prove that FTRL with a self-concordant regularizer without the barrier requirement obtains a regret bound similar to the setting with the barrier requirement. They then show how to apply the result to the online portfolio selection problem which is a canonical online convex optimization problem. The paper provides clear background for the problem to help motivate the goal and significance of the result.\n\nWEAKNESSES:\nThe paper lacks clarity in how exactly Theorem 3.2 is applied to the online portfolio selection problem. Specifically, it is unclear what conditions are needed to apply Theorem 3.2 as Algorithm 1 is applied to a very specific online convex optimization problem. I believe Section 4 tries to highlight these conditions via Lemma 4.4, 4.5, 4.6, and 4.7, but without looking at the appendix, it is unclear where the conditions factor into proving the regret bounds. \n\nMoreover, in Section 5 where Theorem 3.2 is applied to the online portfolio selection problem, it is very unclear why the implicit optimistic LB-FTRL Algorithm 2 solves it. The first challenge in understanding how Algorithm 2 maps the OPS problem to Algorithm 1 is in the limited explanation in the mapping. From my reading, it seems that the only explanation in the main body is in the second sentence of the first paragraph in Section 5, “By the convexity of the loss functions, OPS can be reduced to an online linear optimization problem described in Section 3 with $v_t = g_t$ and $\\mathcal{X}$ being the probability simplex $\\Delta$.” Writing the optimization problem explicitly would be more clear.\n\nQUESTIONS:\nQuestions\n1. Is it possible to state Theorem 3.2 for a generic loss that satisfies some set of specific conditions? This might streamline sections 3 and 4 by highlighting the necessary conditions ahead of time. Additionally, it might reduce the repetition in redefining the problem since fundamentally the only thing that changed was the loss function. \n2. Is there intuition how the newly proposed algorithms get around the non-Lipschitz loss? My intuition is that it depends on choosing the learning rate properly so that when the Lipschitz constant is large, the correct learning rate mitigates it. Is the intuition correct? This seems to imply the learning rate should change depending on where the location of the data point for FTL type algorithms.\n\nLIMITATIONS:\nThe paper is generally more theory-oriented and authors imply it is unclear how the analysis can be generalized to other settings, thus adequately addressing the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to achieve adaptive regret bound, including gradient-variation bound and small-loss bound, for the online portfolio management problem, without the classical no-junk bund assumption. The authors successfully achieve this goal by observing a new kind of smoothness for the function -log wx.\n\nSTRENGTHS:\nThe finding of a kind of smoothness of -log<wx> is the major contribution of the paper. I think this is a novel and significant observation, and lead to multiple new conclusions. Specially, for the small-loss bound, one can easily show a upper bound related to root{sum of gradients at x_t} by using a step size inversely proportional to root{sum of norm of gradients}. However, to get a small loss, bound, the next step is to assume smoothness to create a relationship between root{sum of gradients at x_t} and root{sum of function values at x_t} (using the classical Lemma 2.1 of [29]). In this paper, the authors show that, in the OPS problem, for the loss -log<wx>, we do not need smooth to construct this relationship. The key observation, to me, is Lemma 4.7, which shows a similar result to Lemma 2.1 of [29] with a “surrogate gradient”. In this way, the paper achieves the first small-loss bound for non-smooth functions, which is novel and interesting. For the gradient-variation bound, similar arguments also hold. \n\nNote that, with this observation, one can directly combine it with existing adaptive methods to achieve adaptive bounds. However, it does not mean the proposed methods are less novel. \n\nThis paper is generally well-written and easy to follow. The key finding, section 4.3, is easy to understand.\n\nWEAKNESSES:\nFor exp-concave functions with bounded gradients (e.g., with no-junk bund assumption),  algorithms such as variant of ONS () can get a O(log L^*) bound, while the algorithm in this paper yields a O(root{L^*} + log T) bound, which is sub-optimal. It would be great if the authors can consider how to get a best-of-both-world bound, which enjoys O(root{L^*} + log T) for non-smooth functions, and O(log L^*) for smooth functions. \n\nTo obtain he optimal gradient-variation bound, the proposed algorithms need to know V_T in advance. In the appendix, the authors propose an algorithm that can learn the step size adaptively. However, it somewhat lost the good property of the gradient-variation bound when V_T is small.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}