Title: Sequential Probability Assignment with Contexts: Minimax Regret, Contextual Shtarkov Sums, and Contextual Normalized Maximum Likelihood
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

log loss

probabilistic forecasting
Abstract: We study the fundamental problem of sequential probability assignment, also known as online learning with logarithmic loss, with respect to an arbitrary, possibly nonparametric hypothesis class. Our goal is to obtain a complexity measure for the hypothesis class that characterizes the minimax regret and to determine a general, minimax optimal algorithm. Notably, the sequential $\ell_{\infty}$ entropy, extensively studied in the literature (Rakhlin and Sridharan, 2015, Bilodeau et al., 2020, Wu et al., 2023), was shown to not characterize minimax regret in general. Inspired by the seminal work of Shtarkov (1987)
    and Rakhlin, Sridharan, and Tewari (2010), we introduce a novel complexity measure, the \emph{contextual Shtarkov sum}, corresponding to the Shtarkov sum after projection onto a multiary context tree, and show that the worst case log contextual Shtarkov sum equals the minimax regret. Using the contextual Shtarkov sum, we derive the minimax optimal strategy, dubbed \emph{contextual Normalized Maximum Likelihood} (cNML). Our results hold for sequential experts, beyond binary labels, which are settings rarely considered in prior work. 
    To illustrate the utility of this characterization, we provide a short proof of a new regret upper bound in terms of sequential $\ell_{\infty}$ entropy, unifying and sharpening state-of-the-art bounds by Bilodeau et al. (2020) and Wu et al. (2023).

Sequential Probability Assignment with Contexts: Minimax Regret, Contextual Shtarkov Sums, and Contextual Normalized Maximum Likelihood

Abstract

We study the fundamental problem of sequential probability assignment, also known as online learning with logarithmic loss, with respect to an arbitrary, possibly nonparametric hypothesis class. Our goal is to obtain a complexity measure for the hypothesis class that characterizes the minimax regret and to determine a general, minimax optimal algorithm. Notably, the sequential ℓ ∞ entropy, extensively studied in the literature (Rakhlin and Sridharan, 2015, Bilodeau et al., 2020, Wu et al., 2023), was shown to not characterize minimax regret in general. Inspired by the seminal work of Shtarkov (1987) and Rakhlin, Sridharan, and Tewari (2010), we introduce a novel complexity measure, the contextual Shtarkov sum , corresponding to the Shtarkov sum after projection onto a multiary context tree, and show that the worst case log contextual Shtarkov sum equals the minimax regret. Using the contextual Shtarkov sum, we derive the minimax optimal strategy, dubbed contextual Normalized Maximum Likelihood (cNML). Our results hold for sequential experts, beyond binary labels, which are settings rarely considered in prior work. To illustrate the utility of this characterization, we provide a short proof of a new regret upper bound in terms of sequential ℓ ∞ entropy, unifying and sharpening state-of-the-art bounds by Bilodeau et al. (2020) and Wu et al. (2023).

1 Introduction

Sequential probability assignment is a fundamental problem with connections to information theory [Ris84; MF98; XB00], machine learning [CL06; Vov95; RST15; FKLMS18; Sha20], and portfolio optimization [Kel56; Cov74; Cov91; CO96; Fed91]. In the original non-contextual setup, the learner aims to assign probabilities to a series of labels, which are revealed sequentially. The goal is to offer probabilistic forecasts over the label set such that the probability assigned to any observed sequence is comparable to that assigned by the best model in any fixed class of models.

The celebrated work of Shtarkov [Sht87] characterized minimax regret for context-free sequential probability assignment in terms of what is now known as the Shtarkov sum , and subsequently described the minimax algorithm, Normalized Maximum Likelihood (NML). NML represents the ideal probabilistic forecast in the sense of minimax regret, providing a benchmark for universal coding and prediction strategies. While often not used directly due to its computational complexity, NML has guided the design of practical algorithms and informed the development of efficient approxima-

tion methods. The principles underlying NML have inspired advances in both information theory and online learning, establishing fundamental limits and serving as a critical benchmark.

In this work, we study the problem of sequential probability assignment with contexts, which has been analyzed in recent works (e.g. [RS15; BFR20; WHGS23]) under the framework of online supervised learning formalized by Rakhlin, Sridharan, and Tewari [RST10]. In this setup, the problem is modeled as a T -round game between a learner and the nature : On each round t = 1 , . . . , T , the learner observes a context x t from nature and predicts a distribution ˆ p t over some finite label space Y . Then nature reveals a label y t ∈ Y and the learner incurs a logarithmic loss ℓ (ˆ p t , y t ) = -log(ˆ p t ( y t )) , where ˆ p t ( y t ) is the probability assigned to label y t by ˆ p t . The performance of the learner is measured by the regret with respect to a class F of experts , defined as the difference between the total loss of the learner and that of the best expert in F . The value of primary interest is the minimax regret , that is, the worst-case regret by the best learner over arbitrarily adaptive data sequences. The minimax regret serves as a benchmark for all algorithms and as a target for studies of adaptivity. Our goal is to address several fundamental questions:

Can we find a natural complexity measure of F that characterizes the minimax regret, enabling us to analyze the minimax regret in new ways? And can we identify, in view of this complexity measure, a general, minimax optimal algorithm?

Notably, the sequential covering number of F , a well studied measure of complexity, has been shown not to characterize the minimax regret on its own [RS15; BFR20; WHGS23]. This fact distinguishes sequential probability assignment and log loss: while sequential covering numbers enable a tight analysis in online learning problems with convex Lipschitz losses, like absolute loss [RST15] and square loss [RS14a], they do not yield minimax rates for log loss on some classes. Tackling such classes evidently requires new techniques.

Main contributions.

We introduce a new complexity measure, which we call the contextual Shtarkov sum , that serves as a natural generalization of the Shtarkov sum from the context-free setting. We show that the minimax regret is characterized by the worst-case contextual Shtarkov sum.

We derive the minimax optimal algorithm, dubbed contextual Normalized Maximum Likelihood (cNML), using a data-dependent variant of the contextual Shtarkov sum, thereby generalizing NML from the context-free setting.

We apply contextual Shtarkov sums to the study of sequential entropy bounds on minimax. Doing so, we provide a short proof of a new regret upper bound in terms of sequential entropy that unifies and even improves on state-of-the-arts bounds by [BFR20] and [WHGS23]. Our results extend beyond the binary label setting studied by recent work to arbitrary finite label sets.

Related work. Sequential probability assignment has been studied extensively. Various aspects of this problem have been investigated, including sequences with and without side information (contexts), parametric and nonparametric hypothesis classes, and stochastic or adversarial datagenerating mechanisms. This problem has a long history in the machine learning community, see [CL06, Ch. 9] and the references therein. In the information theory community, this problem is also known as universal prediction [MF98], where the regret is also referred to as redundancy with respect to a set of codes. This has been studied in both stochastic and adversarial settings [Fre96; Ris86; Ris96; Sht87; XB97; DS04; MF98; OS04; Sha06; Szp98], where the focus was primarily on the parametric classes of experts. Closely related topics include universal source coding [Kol65; Sol64; Fit66; Dav73], data compression with arithmetic coding [Ris76; RL81; ZL77; ZL78; FMG92], and the minimum description length (MDL) principle [Ris78; Ris84; Ris87; BRY98; Grü05; HY01]. Lastly, this topic is intimately tied with sequential gambling and portfolio optimization, as pointed out by [Kel56; Cov74; Cov91; CO96; Fed91].

A classical result in context-free sequential probability assignment is that the minimax regret is equal to the log contextual Shtarkov sum [Sht87], and the minimax algorithm is the well-known Normalized Maximum Likelihood. When the set of contexts is known in advance to the forecaster, namely, a fixed design setting, the minimax regret is equivalent to the log Shtarkov sum of the function class when projected onto the known set of contexts [JSS21; WHGS23].

A long line of work has focused on controlling minimax regret for rich classes in terms of covering numbers. [CL99; OH99] upper bounded the regret in terms of the (non-sequential) uniform covering number of the class. However, this complexity measure proved to be insufficient for obtaining optimal rates. [RS15] improved regret upper bounds by proposing a sequential covering measure. Thereafter, by utilizing the self-concordance property and the curvature of the log loss, [BFR20] further improved the upper bound in terms of the sequential covering number for nonparametric Lipschitz classes, through a non-constructive proof. [WHGS23] proposed a Bayesian algorithmic approach in order to upper bound the regret using a global notion of sequential covering. Notably, both the global and local sequential covering numbers do not fully characterize the regret, and the algorithm in [WHGS23] is not minimax optimal. By relaxing the worst-case analysis, [BHS23] studied this problem within the smoothed analysis framework, where nature is not fully adversarial but constrained.

Online learning with respect to arbitrary hypothesis classes and the zero-one loss, in the realizable case, is known to be characterized by the Littlestone dimension [Lit88]. The agnostic case was addressed by [BPS09; ABDMNY21]. Understanding sequential complexities in online learning with Lipschitz losses was extensively studied by [RST10; RS14a; RS14b; RST15]. Note that the logarithmic loss is neither Lipschitz nor bounded. Recently, [AHKKV23] characterized online regression in the realizable case, for any approximate pseudo-metric, such as the ℓ p loss.

2 Preliminaries

Notation. For a positive integer K , let [ K ] := { 1 , 2 , . . . , K } . For a finite set K with |K| = K , we use ∆( K ) to denote the set of all distributions on K . We may identify K with [ K ] (under arbitrary enumeration of elements in K ) and treat elements of ∆( K ) as vectors in R K . For a vector p ∈ R K and i ∈ [ K ] , let p ( i ) be the i -th coordinate of p . Let ∆ + ( K ) = { p ∈ ∆( K ) : p ( k ) > 0 , ∀ k ∈ K} . For a general finite sequence ( a i ) N i =1 , we will use a n : m to denote the sub-sequence ( a n , . . . , a m ) for any n ≤ m and the empty sequence for n > m . For any set A , let A ∗ = ∪ k ≥ 0 A k be the set of all finite length sequences over A .

Sequential probability assignment and minimax regret. Let X be the context space and Y be the finite label space. In each round t ∈ [ T ] during the game of sequential probability assignment, the learner receives a context x t ∈ X from nature and assigns a probability distribution ˆ p t ∈ ∆( Y ) to the possible labels. Then nature reveals the true label y t ∈ Y and the learner incurs a loss ℓ (ˆ p t , y t ) = -log(ˆ p t ( y t )) . Throughout, the learner is required to predict nearly as well as the best expert from an expert class, which is modeled as an arbitrary hypothesis class F ⊆ { ( X × Y ) ∗ ×X → ∆( Y ) } . More formally, the goal of the learner is make their regret with respect to F ,

as small as possible for all sequences x and y generated by nature, possibly in an adversarial manner. Here f ( x 1: t , y 1: t -1 ) ∈ ∆( Y ) can be understood as the prediction made by expert f at round t using past observations ( x 1: t -1 , y 1: t -1 ) as well as the fresh context x t . The main focus is to study the minimax regret R T ( F ) , which can be written as the following extensive form

where x t ∈ X , ˆ p t ∈ ∆( Y ) and y t ∈ Y , ∀ t ∈ [ T ] . In light of this formulation, we can see that the minimax regret concerns both the learner and the nature to be adaptive , meaning that their actions can rely on the revealed history so far.

Remark 2.1 (Sequential vs non-sequential experts) Experts f as mappings from ( X×Y ) ∗ ×X to ∆( Y ) are sometimes called fully sequential experts [WHGS23] due to their ability to predict based on the past history. However, the literature (e.g. [RS15; BFR20; WHGS23]) often considers the more limited notion of non-sequential experts, modeled as F ⊆ {X → ∆( Y ) } , reflecting the fact that prediction made by each expert f is simply f ( x t ) in each round t . In contrast, our results are more general as our novel techniques can be applied to the more flexible sequential experts.

Multiary trees. The complexity of online learning problems stems from the sequential and adaptive nature of the adversary, which we can capture with multiary trees . Formally, for a general space A and a finite set K , an A -valued K -ary tree v of depth d is a sequence of mappings v t : K t -1 → A for t ∈ [ d ] . A path in a depthd tree is a sequence ε = ( ε 1 , . . . , ε d ) ∈ K d . We use the notation v t ( ε ) to denote v t ( ε 1 , . . . , ε t -1 ) for t ∈ [ d ] and the boldface notation v ( ε ) to denote ( v 1 ( ε ) , . . . , v d ( ε )) ∈ A d . Throughout we will only consider Y -ary trees valued in either X or ∆( Y ) , where the paths are denoted by the boldface y . We refer to X -valued trees as context trees and ∆( Y ) -valued trees as probabilistic trees .

Time-varying context sets. So far we consider the context set X to be constant over time. But all of our results can be extended easily to allow for time-varying context spaces. Details of this generalization can be found in Appendix C.

2.1 Prior work: the Shtarkov sum in context-free and fixed designs

Before introducing our complexity measure that characterizes R T ( F ) , we review some prior settings where the minimax regret can be characterized by the well-studied Shtarkov sum . First we introduce the notion of likelihood of a hypothesis f with respect to a context and label sequence, which plays a key role in defining complexity measures and optimal algorithms.

Definition 2.2 (Likelihood) For f : ( X × Y ) ∗ × X → ∆( Y ) and length -d sequences x 1: d ∈ X d , y 1: d ∈ Y d , the likelihood P f ( y 1: d | x 1: d ) is defined as

where we use the compact notation f ( x 1: t , y 1: t -1 ) for f ( x 1 , y 1 , . . . , x t -1 , y t -1 , x t ) .

In the classical context-free setting where X can be thought of as a singleton, any sequential expert f degenerates to a joint distribution over label sequences. Indeed, given any label sequence y 1: t -1 , f ( y 1: t -1 ) ∈ ∆( Y ) can be interpreted as the conditional distribution f assigns to the next label y t . We use P f ( y 1: d ) = ∏ d t =1 f ( y 1: t -1 )( y t ) to denote this distribution. Similarly, the learner's strategy is also specified by a joint distribution that is decomposed to a sequence of conditional distributions ˆ p t = ˆ p t ( ·| y 1: t -1 ) ∈ ∆( Y ) . In this setup the minimax regret R T ( F ) is characterized by the Shtarkov sum [Sht87].

Proposition 2.3 ([Sht87]) In the context-free setting, for any hypothesis class F and horizon T , the Shtarkov sum S T ( F ) is defined as

Moreover, the minimax regret is given by R T ( F ) = log S T ( F ) , and the unique minimax optimal strategy is the normalized maximum likelihood (NML) distribution given by

To go beyond this classical context-agnostic setting and incorporate contextual information, prior work (e.g. [JSS21]) also considered an easier problem than the aforementioned sequential probability assignment, by forcing nature to reveal the context sequence x 1: T to the learner at the start of the game. This is known as the fixed design setting or transductive online learning [WHGS23], where the goal is to characterize the so-called fixed design maximal minimax regret

It is straightforward to see that after projecting on x 1: T , the hypothesis class F again collapses to a set of joint distributions over Y T specified by the likelihood function in Definition 2.2. Moreover, this set of distributions can be accessed by the learner from the start, so the fixed design setting can

be essentially reduced to the context-free setting. To be more specific, for any f ∈ F , it induces an expert in the context-free setting after being projected on x 1: T , which is denoted by f | x 1: T and

and let F| x 1: T := { f | x 1: T : f ∈ F} . Then given any predetermined x 1: T , the learner is equivalently competing with F| x 1: T in the context-free setting. With the following natural variant of the Shtarkov sum, we can easily characterize R FD T ( F ) .

Definition 2.4 (Conditional Shtarkov sum) Given a context sequence x 1: T ∈ X T , the Shtarkov sum of F conditioned on x 1: T is

In fact, S T ( F| x 1: T ) is just the Shtarkov sum of the projected class F| x 1: T in the context-free setting. The following result characterizes the fixed-design setting:

Proposition 2.5 (Minimax regret, fixed design [JSS21]) In the fixed design setting, for any hypothesis class F and horizon T , the fixed design maximal minimax regret is

and, given any context sequence x 1: T , the minimax optimal response is NML with respect to F| x 1: T .

3 Minimax regret via contextual Shtarkov sum

Now we state one of our main results about the characterization of the minimax regret of sequential probability assignment. First we introduce the key concept of contextual Shtarkov sum , which is a natural generalization of Shtarkov sum in the context-free setting.

Definition 3.1 (Contextual Shtarkov sum) The contextual Shtarkov sum S T ( F| x ) of a hypothesis class F on a given context tree x of depth T is defined as

Just like the conditional Shtarkov sum, the contextual Shtarkov sum S T ( F| x ) can be interpreted as the Shtarkov sum of the projected class F| x := { f | x : f ∈ F} where f | x is the induced context-free expert specified by

where we have slightly abused the notation to use x ( y 1: t -1 ) to denote the lengtht context sequence obtained by tracing tree x through the (partial) path y 1: t -1 . Next we show that the minimax regret R T ( F ) is characterized by the worst-case contextual Shtarkov sum:

Theorem 3.2 (Main result: minimax regret) For any hypothesis class F ⊆ { ( X × Y ) ∗ ×X → ∆( Y ) } and horizon T ,

where the supremum is taken over all X -valued context trees x of depth T .

Since any context sequence x 1: T can be thought as a special context tree x that is constant in each level t ∈ [ T ] (i.e., x t ( y ) = x t , ∀ y ), we can find that the supremum over context trees in Theorem 3.2 strictly subsumes the supremum over context sequences in Proposition 2.5. Thus we can see the separation between R T ( F ) and R FD T ( F ) is clearly exhibited.

The full proof of Theorem 3.2 as well as an overview are provided in Appendix A.

3.1 Applications: an improved regret upper bound in terms of sequential entropy

To illustrate the utility of our characterization in Theorem 3.2, we walk through some examples where we are able to recover and sharpen existing regret upper bounds with relatively short proofs via contextual Shtarkov sum. As a start, we provide a short proof in Appendix A.6 of the classical regret bound for a finite hypothesis class.

Proposition 3.3 (Finite classes) For any F ⊆ [0 , 1] X and horizon T , R T ( F ) ≤ log |F| .

Let us go back to the binary label setting with non-sequential experts, that is, Y = { 0 , 1 } and F ⊆ [0 , 1] X , and f ( x ) ∈ [0 , 1] is interpreted as the probability assigned to label 1 by this expert f . We will show a regret bound that outperforms the state-of-the-art ones in [BFR20; WHGS23] with a surprisingly simple proof. To proceed, we need the following notation. Given a context tree x of depth T , let F ◦ x = { f ◦ x : f ∈ F} , where f ◦ x is the [0 , 1] -valued tree such that

Next we introduce the definitions of sequential ℓ ∞ covers and entropy.

Definition 3.4 (Sequential ℓ ∞ cover and entropy) Given a hypothesis class F ⊆ [0 , 1] X and a context tree x of depth T , we say a collection of R -valued trees V x ,α is a sequential cover of F ◦ x at scale α > 0 if for any f ∈ F , y ∈ Y T , there exists some v ∈ V x ,α such that

Let the sequential ℓ ∞ covering number N ∞ ( F ◦ x , α, T ) be the size of the smallest such cover. The sequential ℓ ∞ entropy of F at scale α and depth T is defined as the logarithm of the worst-case sequential covering number: H ∞ ( F , α, T ) := sup x log N ∞ ( F ◦ x , α, T ) .

Definition 3.5 (Global sequential ℓ ∞ cover and entropy) Given a hypothesis class F ⊆ [0 , 1] X , we say a collection of mappings G α ⊆ [0 , 1] X∗ is a global sequential cover of F at scale α > 0 and depth T if for any f ∈ F , x 1: T ∈ X T , there exists some g ∈ G α such that

Let the global sequential ℓ ∞ covering number N G ( F , α, T ) be the size of the smallest such cover. The global sequential ℓ ∞ entropy of F at scale α and depth T is defined as

Proposition 3.6 ([BFR20; WHGS23]) For any F ⊆ [0 , 1] X and horizon T ,

It is easy to show that H ∞ ( F , α, T ) ≤ H G ( F , α, T ) , but, in general, the two bounds in Proposition 3.6 are incomparable due to constants and different dependence on α (more discussions on these bounds are deferred to Appendix C). Starting from the contextual Shtarkov sum, we are able to derive a bound that combines the best of these two bounds:

Theorem 3.7 (Main result: sequential entropy bound) For any F ⊆ [0 , 1] X and horizon T ,

Proof For any scale α > 0 and depthT context tree x , let V x ,α be a sequential cover of F ◦ x at scale α with size N ∞ ( F ◦ x , α, T ) . We can always assume V x ,α to be [0 , 1] -valued without loss of generality because otherwise we can just truncate it without violating its coverage guarantee. Define the smoothed covering set ˜ V x ,α = { ˜ v : ∀ t ∈ [ T ] , ˜ v t ( · ) = v t ( · )+ α 1+2 α , v ∈ V x ,α } , inspired by [BFR23; WHGS23]. Then for any f ∈ F , y ∈ Y T , there exists some v ∈ V x ,α such that | f ( x t ( y )) -v t ( y ) | ≤ α, ∀ t ∈ [ T ] and hence ˜ v satisfies

Hence

and

where the last equality follows from Lemma D.1, treating ˜ v as sequential experts. Finally,

Since our choice of α is arbitrary, the result follows.

3.2 The inadequacy of sequential ℓ ∞ covering number

Weconclude this section with a discussion on the suboptimality of regret bounds based on sequential covering numbers as in Proposition 3.6 and Theorem 3.7. Let us consider the binary label setting and the following hypothesis classes over the unit Hilbert ball X = B 2 :

We can see that the sequential ℓ ∞ covering numbers of F Lin and F AbsLin are of the same order for all scales, thus the aforementioned results will yield the same regret bound for these two classes. However, we have R T ( F Lin ) = ˜ O ( √ T ) while R T ( F AbsLin ) = ˜ Θ( T 2 / 3 ) [RS15; WHGS23], which implies that the sequential ℓ ∞ covering number, in its current form within the regret bound, cannot characterize the minimax regret.

It is worth mentioning that an ˜ Ω( √ T ) lower bound on R T ( F Lin ) is achievable, via an ˜ Ω(1 /α 2 ) lower bound on the sequential fat-shattering dimension sfat α ( F Lin ) combined with Proposition 2 in [WHGS23]. The same lower bound also holds in the finite-dimensional case, where B 2 is a unit d -dimensional Euclidean ball with d ≥ √ T [WHGS23, Footnote 6]. Our proof (Appendix A.7) of the next result works in both the infinite and finite dimensional (with d ≥ T ) cases.

Lemma 3.8 ( Ω( √ T ) lower bound for the linear class F Lin ) For F Lin defined as in Eq. (1) with B 2 being the unit Hilbert ball or the unit d -dimensional Euclidean ball with d ≥ T , then

The proof of Lemma 3.8 is based on lower bounding the conditional Shtarkov sums (and hence the contextual Shtarkov sums) of F Lin . From Theorem 3.2 we know that the ˜ O ( √ T ) upper bound holds for the log of contextual Shtarkov sums as well but we do not have a direct proof of this fact so far.

■

Algorithm 1 Contextual Normalized Maximum Likelihood (cNML)

Input: Hypothesis class F , horizon T

For t = 1 , 2 , ..., T do

Observe context x t ∈ X

If sup f ∈F P f ( y 1: t -1 | x 1: t -1 ) > 0 , predict ˆ p t ∈ ∆( Y ) with

and otherwise set ˆ p t to be an arbitrary member of ∆ + ( Y )

Receive label y t ∈ Y

End for

4 Contextual NML, the minimax optimal algorithm

So far we have settled the minimax regret of sequential probability assignment in a nonconstructive way. Now we switch to the algorithmic lens to study the optimal strategy that achieves the minimax regret. Remarkably, we show that the minimax optimal algorithm can be described by a data-dependent variant of the contextual Shtarkov sum, which is named contextual Shtarkov sum with prefix .

Definition 4.1 (Contextual Shtarkov sum with prefix) Given sequences x 1: t ∈ X t , y 1: t ∈ Y t , t ∈ [ T ] and a context tree x of depth T -t , the contextual Shtarkov sum S x 1: t ,y 1: t T ( F| x ) of F on x with prefix x 1: t , y 1: t is defined as

Now we present our prediction strategy, contextual normalized maximum likelihood (cNML), which is summarized in Algorithm 1. In each round t , with x 1: t , y 1: t -1 as past observations, the learner first checks whether sup f ∈F P f ( y 1: t -1 | x 1: t -1 ) > 0 since if that is not the case and sup f ∈F P f ( y 1: t -1 | x 1: t -1 ) = 0 , the cumulative losses of all experts in F have already blown up to + ∞ and the learner only needs to predict any ˆ p ∈ ∆ + ( Y ) in all remaining rounds. On the other hand, if sup f ∈F P f ( y 1: t -1 | x 1: t -1 ) > 0 , then

and the ˆ p t given by Eq. (2) is indeed a valid member of ∆( Y ) (shown in Appendix B) and is used as the learner's prediction. The following theorem shows that cNML is the minimax optimal algorithm, with proof deferred to Appendix B.

Theorem 4.2 (Main result: optimal algorithm) The contextual normalized maximum likelihood strategy (Algorithm 1) is minimax optimal.

To see that cNML is reduced to NML in the context-free setting, it suffices to consider the case where sup f ∈F P f ( y 1: T ) > 0 since otherwise NML will simply assign 0 probability on this sequence y 1: T and during the actual round-wise implementation of NML, it also predicts an arbitrary element from ∆ + ( Y ) in those rounds t where sup f P f ( y 1: t -1 ) = 0 . Now for any y 1: T such that sup f ∈F P f ( y 1: T ) > 0 , the prediction by cNML in each round t is

which can be summarized into a joint density over y 1: T by

Recall that this is exactly the NML prediction p nml ( y 1: T ) .

Remark 4.3 (Relaxations and efficient algorithms) One may wonder if more efficient algorithms are available when it is not easy to compute contextual Shtarkov sums with prefix. One solution is to apply the framework of admissible relaxation in [RSS12], which provides a systematic way of constructing efficient algorithms at the cost of worse regret guarantees. Notice that the worst-case log contextual Shtarkov sums with prefix constitute a trivially 'admissible relaxation' since they are the exact conditional game values.

5 Perspectives on contextual Shtarkov sums

In this section, we provide further insight into contextual Shtarkov sums, defined in Sections 3 and 4.

5.1 Contextual Shtarkov sums through martingales

We can relate our characterization of the minimax regret to the more extensively studied sequential Rademacher complexity , which arises in online learning problems with hypothesis class F ⊆ [0 , 1] X and bounded convex losses like absolute loss. Specifically, the (conditional) sequential Rademacher complexity [RST15] is defined by

where x is a depthT binary context tree and ε = ( ε 1 , . . . , ε T ) ∈ {± 1 } T is a sequence of i.i.d. Rademacher random variables. A notable feature of R T ( F ; x ) is that it is the expected supremum of the sum of a martingale differences, i.e., for any f, E [ ε t f ( x t ( ε )) | ε 1 , . . . , ε t -1 ] = 0 . Likewise, S T ( F| x ) also admits a martingale interpretation. To see this, let F ⊆ { ( X × Y ) ∗ ×X → ∆( Y ) } and rewrite S T ( F| x ) for any context tree x :

where y = ( y 1 , . . . , y T ) is a sequence of i.i.d. variables following the uniform distribution over Y . It is easy to check that E [ |Y| · f ( x 1: t ( y ) , y 1: t -1 )( y t ) | y 1 , . . . , y t -1 ] = 1 , and thus

is a martingale with respect to filtration F t = σ ( y 1 , . . . , y t ) , t ∈ [ T ] . It would be of independent interest to study the contextual Shtarkov sums more quantitatively by developing new tools for such product-type martingales.

5.2 General Shtarkov sums

We can also interpret contextual Shtarkov sums as an instance of general Shtarkov sums , which are defined over sub-probability measures.

Definition 5.1 (Sub-probability measure) Aset P = { p : K → [0 , 1] } is a class of sub-probability measures over a finite set K if

Due to Lemma D.1, it is easy to see that for any hypothesis class F ⊆ { ( X × Y ) ∗ ×X → ∆( Y ) } and depthT context tree x , they induce a class

that is a class of sub-probability measures over Y T . Moreover, for any F , depth-( T -t ) context tree x and sequences x 1: t ∈ X t , y 1: t ∈ Y t , the induced

is a class of sub-probability measure over Y T -t since

Next we introduce the notion of general Shtarkov sum over classes of sub-probability measures.

Definition 5.2 (General Shtarkov sum) Given any class P of sub-probability measures over K , the general Shtarkov sum of P is defined as

With the notion of general Shtarkov sum, it is not hard to verify that the contextual Shtarkov sums with & without prefix can be interpreted as instances of general Shtarkov sums:

Proposition 5.3 For any horizon T, t ∈ [ T ] , data sequence x 1: t ∈ X t , y 1: t ∈ Y t , and context trees x , x ′ of depth T, T -t respectively, we have

It would be interesting to find out other instances of general Shtarkov sums that capture the complexities of other online learning problems with log loss.

6 Discussions

In this paper, we characterize the minimax regret and the optimal prediction strategy for sequential probability assignment, generalizing the classical results in the context-free setting. Moreover, our results are general enough to subsume the setting of multiary labels and sequential hypothesis classes, which has not been sufficiently explored before. Remarkably, our characterization holds for arbitrary hypothesis classes that may not admit the regularity assumptions implicitly required by prior works (e.g. [RST15; BFR20]).

For future works, it would be interesting to study the minimax regret of specific classes more quantitatively using our contextual Shtarkov sums. It is also intriguing to consider the setting of infinite labels. Although most of our arguments would go through under sufficient regularity conditions, a more systematic study is needed. On the practical side, it is important to develop algorithms that are more computationally efficient than cNML and with provable guarantees.

References

[BPS09] S. Ben-David, D. Pál, and S. Shalev-Shwartz. 'Agnostic Online Learning.' In: COLT . Vol. 3. 2009, p. 1 (cit. on p. 3).

[BHS23] A. Bhatt, N. Haghtalab, and A. Shetty. 'Smoothed Analysis of Sequential Probability Assignment'. In: Thirty-seventh Conference on Neural Information Processing Systems . 2023 (cit. on p. 3).

[BFR20] B. Bilodeau, D. Foster, and D. Roy. 'Tight bounds on minimax regret under logarithmic loss via self-concordance'. In: International Conference on Machine Learning . PMLR. 2020, pp. 919-929 (cit. on pp. 2, 3, 6, 10, 14, 15, 25).

[BFR23] B. Bilodeau, D. J. Foster, and D. M. Roy. 'Minimax rates for conditional density estimation via empirical entropy'. In: The Annals of Statistics 51.2 (2023), pp. 762-790 (cit. on pp. 7, 15).

[CL06] N. Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge university press, 2006 (cit. on pp. 1, 2).

[CL99] N. Cesa-Bianchi and G. Lugosi. 'Minimax regret under log loss for general classes of experts'. In: Proceedings of the Twelfth annual conference on computational learning theory . 1999, pp. 12-18 (cit. on p. 3).

[Cov74] T. M. Cover. 'Universal gambling schemes and the complexity measures of Kolmogorov and Chaitin'. In: Technical Report, no. 12 (1974) (cit. on pp. 1, 2).

[Cov91] T. M. Cover. 'Universal portfolios'. In: Mathematical finance 1.1 (1991), pp. 129 (cit. on pp. 1, 2).

[CO96] T. M. Cover and E. Ordentlich. 'Universal portfolios with side information'. In: IEEE Transactions on Information Theory 42.2 (1996), pp. 348-363 (cit. on pp. 1, 2).

[Dav73] L. D. Davisson. 'Universal noiseless coding'. In: IEEE Trans. Inf. Theory 19 (1973), pp. 783-795 (cit. on p. 2).

[DS04] M. Drmota and W. Szpankowski. 'Precise minimax redundancy and regret'. In: IEEE Transactions on Information Theory 50.11 (2004), pp. 2686-2707 (cit. on p. 2).

[Fed91]

M. Feder. 'Gambling using a finite state machine'. In:

Information Theory

IEEE Transactions on

37.5 (1991), pp. 1459-1465 (cit. on pp. 1, 2).

[FMG92] M. Feder, N. Merhav, and M. Gutman. 'Universal prediction of individual sequences'. In: IEEE transactions on Information Theory 38.4 (1992), pp. 12581270 (cit. on p. 2).

[Fit66] B. Fitingof. 'Optimal encoding with unknown and variable message statistics'. In: Probl. Inform. Transm. 2 (1966), pp. 3-11 (cit. on p. 2).

[FKLMS18] D. J. Foster, S. Kale, H. Luo, M. Mohri, and K. Sridharan. 'Logistic regression: The importance of being improper'. In: Conference on learning theory . PMLR. 2018, pp. 167-208 (cit. on p. 1).

[Fre96] Y. Freund. 'Predicting a binary sequence almost as well as the optimal biased coin'. In: Proceedings of the ninth annual conference on Computational learning theory . 1996, pp. 89-98 (cit. on p. 2).

[Grü05] P. Grünwald. 'Minimum description length tutorial'. In: (2005) (cit. on p. 2).

[HY01] M. H. Hansen and B. Yu. 'Model selection and the principle of minimum description length'. In: Journal of the American Statistical Association 96.454 (2001), pp. 746-774 (cit. on p. 2).

[JSS21] P. Jacquet, G. Shamir, and W. Szpankowski. 'Precise minimax regret for logistic regression with categorical feature values'. In: Algorithmic Learning Theory . PMLR. 2021, pp. 755-771 (cit. on pp. 2, 4, 5).

[Kel56] J. L. Kelly. 'A new interpretation of information rate'. In: the bell system technical journal 35.4 (1956), pp. 917-926 (cit. on pp. 1, 2).

[Kol65] A. N. Kolmogorov. 'Three approaches to the definition of the concept 'quantity of information''. In: Problemy peredachi informatsii 1.1 (1965), pp. 3-11 (cit. on p. 2).

[Lit88] N. Littlestone. 'Learning quickly when irrelevant attributes abound: A new linearthreshold algorithm'. In: Machine learning 2 (1988), pp. 285-318 (cit. on p. 3).

A Proofs for Section 3

Notations. When the context and label sequences x 1: T , y 1: T are clear from the context, we may use f t to denote the probability vector f ( x 1: t , y 1: t -1 ) ∈ ∆( Y ) produced by hypothesis f at time t for notational convenience. We also adopt the notation for repeated operators in [RST15; BFR20], denoting Opt 1 · · · Opt T [ · · · ] by ⟪ Opt t ⟫ T t =1 [ · · · ] . For any discrete distribution P and discrete random variables X,Y , let H ( P ) be the entropy of P and H ( X | Y ) be the conditional entropy of X given Y .

A.1 Proof overview of Theorem 3.2

Before presenting the proof of Theorem 3.2 in full details in Appendices A.2 to A.4, we give a high-level overview here.

The proof starts from swapping the pairs of inf and sup (after randomizing the labels revealed by the nature) in the extensive formulation of R T ( F ) to move to the dual game , where the learner predicts after seeing the action of the nature. Trivially the value of this swapped game is a lower bound for R T ( F ) , and after rearranging we get that

where the supremum is taken over all context trees x and probabilistic trees p , of depth T . Also E y ∼ p means the nested conditional expectations E y 1 ∼ p 1 ( y ) E y 2 ∼ p 2 ( y ) · · · E y T ∼ p T ( y ) .

Similar to the proof of Lemma 6 in [BFR20] for the binary label setting, we apply the minimax theorem with a tweak that we devise to handle multiary labels to derive that

under some mild regularity condition for F . A key observation is that the supremum over depthT probabilistic trees p is equivalent to the supremum over joint distributions P over Y T . Based on this observation and some algebra, for a fixed context tree x , the supremum over p in Eq. (3) is

The value of this maximization problem can be easily computed to be

Thus,

However, Eq. (3) is not guaranteed when there is no assumed regularity condition for F . To get away from this, prior works have assumed a particular hypothesis is included in F such that the enlarged class allows for the minimax swap [RS15; BFR20]. Nevertheless, even adding a mere hypothesis may lead to suboptimal analysis for some classes F , say when R T ( F ) is of constant order. To completely get rid of any regularity assumption and obtain a unified characterization of the minimax regret for arbitrary class F , we provide a novel argument as follows. For an arbitrary class F , we study a smooth truncated version of it, denoted by F δ for any level δ ∈ (0 , 1 / 2) , such that F δ always validates the use of the minimax theorem and hence R T ( F δ ) = sup x log S T ( F δ | x ) . Then we give a series of refined analysis comparing the minimax regrets and contextual Shtarkov sums of F and F δ that yields

where C ( T, |Y| ) < ∞ is a positive constant that only depends on T and |Y| . Sending δ → 0 + will conclude that R T ( F ) ≤ sup x log S T ( F| x ) , which finishes the whole proof as we already have R T ( F ) ≥ sup x log S T ( F| x ) from the start.

A.2 Minimax swap

As standard in online learning literature, we will first move to a dual game after applying a minimax swap at each round of the game. Under mild assumptions, the value of the original game coincides with the that of the swapped game. More specifically, we have:

Lemma A.1 Whenever F satisfies that for every sequence x 1: T ∈ X T , y 1: T ∈ Y T ,

where the supremum is taken over all X -valued Y -ary trees x and ∆( Y ) -valued Y -ary trees p , of depth T . Also E y ∼ p means the nested conditional expectations E y 1 ∼ p 1 ( y ) E y 2 ∼ p 2 ( y ) · · · E y T ∼ p T ( y ) .

To deal with the unboundedness of log loss in the proof, we introduce the following truncation method inspired by [BFR23; WHGS23], generalizing the one in [BFR20] which was specific to binary labels.

Definition A.2 (Smooth truncation) The general smooth truncation map τ δ : ∆( Y ) → ∆( Y ) is defined such that for all p ∈ ∆( Y ) and y ∈ Y ,

we have that given threshold δ ∈ (0 , 1 / 2) .

It is easy to check that τ δ ( p ) is indeed a valid member in ∆( Y ) and τ δ ( p )( y ) ∈ [ δ/ (1 + |Y| δ ) , (1 + δ ) / (1 + |Y| δ )] . Moreover, it is not hard to verify that τ δ (∆( Y )) = { p ∈ ∆( Y ) : p ( y ) ∈ [ δ/ (1 + |Y| δ ) , (1 + δ ) / (1 + |Y| δ )] , ∀ y ∈ Y} . We will use ∆ δ ( Y ) to denote this image set τ δ (∆( Y )) .

Proof of Lemma A.1 Fix δ ∈ (0 , 1 / 2) . By restricting the learner's prediction ˆ p t to ∆ δ ( Y ) , we get an upper bound on R T ( F ) :

Now we can apply Sion's minimax theorem [Sio58] to the function

to derive that

This is because:

A (ˆ p T , p T ) is convex and continuous in ˆ p T over the compact ∆ δ ( Y ) and

A (ˆ p T , p T ) is concave and continuous in p T over the compact ∆( Y ) , which is further due to that A (ˆ p T , p T ) is linear in p T and is bounded given Eq. (4).

Hence

Again the order of inf ˆ p T -1 ∈ ∆ δ ( Y ) and sup p T -1 ∈ ∆( Y ) with respect to

can be swapped due to the same reason as above, leading to

Repeating this procedure through all T rounds yields

By Lemma A.7, we know that we do not lose too much by restricting learner's prediction to ∆ δ ( Y ) :

Sending δ → 0 + on the RHS of the above inequality, we get

It is easy to see that on the RHS of the above inequality, the inner infimum over ˆ p t ∈ ∆( Y ) is achieved at ˆ p t = p t due to the nature of log loss. So

where in the last equality we use the compact notation of trees to further simplify our expression and this concludes the proof. ■

Lemma A.3 For any hypothesis class F and horizon T ,

It is implied that whenever F satisfies Eq. (5) , we have

Proof of Lemma A.3 First we can see that the outcome sequence y 1: T generated under any tree p is the same thing as y 1: T generated by its associated joint distribution over Y T , and vice versa. So we can replace the supremum over trees p in the LHS of Eq. (6) by the supremum over joint distributions P over Y T . Hence,

where P t denotes the conditional distribution P t ( ·| y 1: t -1 ) ∈ ∆( Y ) of y t under P given y 1: t -1 . Now fix the context tree x and distribution P . Then we can see that E y ∼ P [ ℓ ( P t , y t )] = H ( y t | y 1: t -1 ) . So E y ∼ P [ ∑ T t =1 ℓ ( P t , y t )] = ∑ T t =1 H ( y t | y 1: t -1 ) = H ( P ) . Further notice that

So naturally we define the map F x : Y T → R ∪ {-∞} by

and then we see that

For any given tree x , notice that the optimization problem

is actually a maximization problem in the form of max P ∈ ∆( Y T ) H ( P ) + ⟨ P, v ⟩ , where v is some |Y| T -dimensional vector. According to the conjugacy between negative entropy function and logsum-exp function, the optimal P ∗ is given by

Note that the above formula for P ∗ is also valid when F x ( y ) = -∞ for some y , since P ∗ should be supported on { y ∈ Y T : F x ( y ) > -∞} , and F x ( y ) cannot be -∞ for all y due to Lemma D.1. The associated value of this maximization problem is

Therefore,

■

In the proof of Lemma A.1, if we do not restrict the learner's prediction and simply swap the order of inf and sup to produce an inequality at each time t , we will reach the following folklore result.

Lemma A.4 For any hypothesis class F and horizon T ,

Proof of Lemma A.4 To get Eq. (7), we simply need to reverse the order of sup and inf at each time in the extensive formulation of minimax regret and produce an inequality:

Iterating the argument and rearranging terms as above, we will get that

A.3 Smooth truncated hypothesis class

To remove the reliance on Eq. (4), we introduce a smooth truncated version of F that always satisfies Eq. (4) and study its minimax regret as well as contextual Shtarkov sums, compared to those of the untruncated class F . To be more specific, we will apply the smooth truncation map to hypotheses: for any δ ∈ (0 , 1 / 2) and f : ( X × Y ) ∗ × X → ∆( Y ) , we use f δ to denote its smooth truncated counterpart τ δ ◦ f ; for any hypothesis class F , we use F δ to denote the corresponding smooth truncated class τ δ ◦ F = { τ δ ◦ f : f ∈ F} . It is easy to verify that any smooth truncated class F δ satisfies Eq. (4) and hence

Next we control the effect of truncation on the minimax regret.

Lemma A.5 For any F , T and δ ∈ (0 , 1 / 2) ,

■

Proof of Lemma A.5 Fix threshold δ ∈ (0 , 1 / 2) and hypothesis f . By Lemma A.7, for any given sequences x 1: T , y 1: T , there is

Then, for any sequence of predictions ˆ p 1: T ,

which concludes the proof.

LemmaA.6 There exists a constant M ( T ) < ∞ that only depends on T such that for any f, x 1: T ∈ X T , y 1: T ∈ Y T and δ ∈ (0 , 1 / 2) ,

Proof of Lemma A.6 Fix threshold δ ∈ (0 , 1 / 2) , hypothesis f and sequences x 1: T , y 1: T . Then

̸

where we can set M ( T ) = T + 2 + 3 + · · · + T since f t ( y t ) 's are bounded by 1 . ■

A.4 Putting together

Now we are fully prepared to finish the proof of Theorem 3.2, our main result in Section 3. Proof of Theorem 3.2 By Lemma A.6, we have that for any context tree x of depth T ,

Thus

Together with Lemma A.5, we get that for any δ ∈ (0 , 1 / 2) ,

After sending δ → 0 + on the RHS of Eq. (9),

Recall that we have R T ( F ) ≥ sup x log S T ( F| x ) from Lemma A.4 and Lemma A.3. So finally,

A.5 Additional proofs

Lemma A.7 For any p ∈ ∆( Y ) and δ ∈ (0 , 1 / 2) ,

Proof of Lemma A.7 By direct computation, for any y ∈ Y ,

A.6 Proof of Proposition 3.3

Starting from Theorem 3.2 that R T ( F ) = sup x log S T ( F| x ) , we have

where the last equality is due to Lemma D.1.

A.7 Proof of Lemma 3.8

It suffices to show that R FD T ( F Lin ) ≥ √ T/ 4 . In particular, we only need to find some context sequence x 1: T such that log S T ( F Lin | x 1: T ) ≥ √ T/ 4 due to Proposition 2.5. Here we pick x 1: T such that x t are unit vectors and x t ⊥ x t ′ whenever t = t ′ . Such sequence exists because the

̸

■

■

dimension of B 2 is no smaller than T . In this way, for each possible label sequence y 1: T ∈ { 0 , 1 } T , we can see that the f w ∈ F Lin that is indexed by

achieves likelihood P f w ( y 1: T | x 1: T ) = ( 1+1 / √ T 2 ) T . Therefore,

which implies that R FD T ( F Lin ) = log S T ( F Lin | x 1: T ) ≥ T log(1 + 1 / T ) ≥ T/ 4 for all T ≥ 1 .

B Proofs for Section 4

Notations. Again we may use f t to denote the probability vector f ( x 1: t , y 1: t -1 ) ∈ ∆( Y ) produced by hypothesis f at time t when the context and label sequences x 1: T , y 1: T are clear from the context. For a context tree x of depth T -t and a path y ∈ Y T -t , we re-index x ( y ) as ( x t +1 ( y ) , . . . , x T ( y )) whenever it takes the last T -t entries of the entire context sequence. And we do the same for the probabilistic tree p as well. That is, whenever y = ( y t +1 , . . . , y T ) ∈ Y T -t takes the last T -t entries of the whole label sequence and y ∼ p , then we will denote this label generating process by y t +1 ∼ p t +1 ( y ) , . . . , y T ∼ p T ( y ) .

B.1 Proof of Theorem 4.2

Recall that the minimax regret is

Through this extensive form of the minimax regret, we know that given x 1: t , y 1: t -1 , the minimax prediction ˆ p ∗ t at round t is the one that minimizes the following expression over all ˆ p t ∈ ∆( Y ) :

Define

and now

The crux of the proof is to show the following:

Lemma B.1 For any hypothesis class F and sequences x 1: t ∈ X t , y 1: t ∈ Y t ,

The proof of Lemma B.1 is done by essentially following the same strategy in Appendix A since G ( F , x 1: t , y 1: t ) admits a similar extensive form with the minimax regret R T ( F ) . For completeness we provide its proof in Appendix B.2. Given Lemma B.1, we have

We apply the following result to solve the above program:

Lemma B.2 [MG22, Lemma 15] Let g : Y → [0 , + ∞ ] be a measurable function such that ∫ Y g ( y ) dµ ∈ (0 , + ∞ ) . Then,

where the infimum in Eq. (11) spans over all probability densities p : Y → [0 , + ∞ ) with respect to µ , and the infimum is reached at

Letting g ( y ) = sup x S x 1: t , ( y 1: t -1 ,y ) T ( F| x ) ∈ [0 , 1] and µ be the counting measure on the finite space Y , we can apply Lemma B.2 whenever not all g ( y ) 's are 0 . In this case, we solve that

On the other hand, if g ( y ) = 0 , ∀ y ∈ Y , then any ˆ p t such that ˆ p t ( y ) > 0 , ∀ y ∈ Y , is an minimax optimal prediction. Moreover, it implies that P f ( y 1: t -1 | x 1: t -1 ) = 0 , ∀ f ∈ F . This is because for arbitrary context tree x ,

So the cumulative loss for each expert f up to round t -1 already blows up to + ∞ and the learner only needs to predict an arbitrary ˆ p ∈ ∆ + ( Y ) in all remaining rounds to achieve R T ( F ; ˆ p 1: T , x 1: T , y 1: T ) = -∞ .

Overall, we can see that the minimax optimal prediction ˆ p ∗ t ∈ ∆( Y ) at round t given x 1: t , y 1: t -1 is

if there exists y ∈ Y such that sup x S x 1: t , ( y 1: t -1 ,y ) T ( F| x ) > 0 . Otherwise, select ˆ p ∗ t to be an arbitrary element in ∆ + ( Y ) (and so do all remaining rounds).

B.2 Auxiliary lemmas

Recall that for any hypothesis class F and sequences x 1: t ∈ X t , y 1: t ∈ Y t ,

To prove Lemma B.1, we need the following lemmas.

Lemma B.3 For any hypothesis class F and sequences x 1: t ∈ X t , y 1: t ∈ Y t ,

And whenever for every x t +1: T ∈ X T -t , y t +1: T ∈ Y T -t , it holds

then

Proof of Lemma B.3 First we see that similar to the proof of Lemma A.4, we can reverse every pair of sup over p s and inf over ˆ p s in the extensive formulation of G ( F , x 1: t , y 1: t ) and rearrange terms to obtain

and again due to the nature of log loss,

where in the last step we compress the expression using trees (of depth T -t ) and Eq. (12) is proved.

To show that the minimax swap is valid under Eq. (13), we follow the same strategy as in the proof of Lemma A.1 by restricting the learner's prediction ˆ p s to ∆ δ ( Y ) for any threshold δ ∈ (0 , 1 / 2) which yields

So Eq. (14) is proved by sending δ → 0 + on the RHS of the last inequality and the established Eq. (12). ■

Lemma B.4 For any hypothesis class F and sequences x 1: t ∈ X t , y 1: t ∈ Y t ,

Proof of Lemma B.4 The proof follows that of Lemma A.3. By replacing the probabilistic tree p by the joint distribution P ∈ ∆( Y T -t ) , we get

Similarly, for any fixed x , define the map F x 1: t ,y 1: t x : Y T -t → R ∪ {-∞} by

and now we solve

If there exists some y ∈ Y T -t such that F x 1: t ,y 1: t x ( y ) > -∞ , then the optimal P ∗ is given by

and then

However, if F x 1: t ,y 1: t x ( y ) = -∞ for all y , then it implies that for any context tree x , path y , and f ∈ F , P f ( y 1: t , y | x 1: t , x ( y )) = 0 and hence,

which finishes our proof.

Now we are able to prove the key result Lemma B.1.

Proof of Lemma B.1 Fix any hypothesis class F and sequences x 1: t ∈ X t , y 1: t ∈ Y t . First we know x ,y )

due to Eq. (12) and Lemma B.4. For the other direction, let us fix any threshold value δ ∈ (0 , 1 / 2) and then

where we have applied Lemma A.7, Lemma B.3, Lemma B.4, and Lemma A.6 accordingly. Similarly, we send δ → 0 + on the RHS of the last inequality and get

which concludes the proof.

■

C Additional discussions

C.1 On the time-variant context space

In this section we generalize our analysis to the setting where the context space can evolve over time. We model time-varying context sets by a sequence of maps X t : X t -1 ×Y t -1 → 2 X , t ∈ [ T ] as in [RS15; BFR20]. In each round t , instead of picking any context from X , the nature is now required to only choose x t from X t ( x 1: t -1 , y 1: t -1 ) ⊆ X . Then the minimax regret with respect to ( X t ) t ∈ [ T ] is rewritten as

A context tree x is consistent with respect to ( X t ) t ∈ [ T ] if for all t ∈ [ T ] and y ∈ Y T , x t ( y ) ∈ X t ( x 1: t -1 , y 1: t -1 ) . Then our results in Section 3 and Section 4 can be generalized simply by replacing the supremum over all context trees (of depth T ) by the supremum over all consistent context trees. For example, we will have

C.2 On the global and non-global sequential cover

Now we go back to consider the usual setting of binary label and constant experts, i.e., Y = { 0 , 1 } and F ⊆ [0 , 1] X . As mentioned in Section 3, previous works [BFR20; WHGS23] provided regret upper bounds based on ℓ ∞ sequential entropy. More specifically, both of their bounds are in the form of O (inf α> 0 { αT + H ( F , α, T ) } ) , with H ( F , α, T ) being either the non-global entropy H ∞ ( F , α, T ) or the global entropy H G ( F , α, T ) . It is then natural to ask which one of these two bounds is tighter. It is straightforward to prove that H ∞ ( F , α, T ) is no larger than H G ( F , α, T ) . In fact, the gap between them is at most a polylog factor, as we state and prove below. 1 The proof of H ∞ ( F , α, T ) ≤ H G ( F , α, T ) is also included for completeness. Before stating the results, we introduce the definition of sequential fat-shattering dimension.

Definition C.1 We say an X -valued binary tree x of depth d is α -shattered by a class F ⊆ [0 , 1] X for some α > 0 , if there exists a [0 , 1] -valued binary tree s of depth d such that

In this case, s is called the witness of the shattering. The sequential fat-shattering dimension of F at scale α , denoted by sfat α ( F ) , is the largest d such that some depthd context tree is α -shattered by F .

Proposition C.2 For any scale α > 0 , we have

Therefore, together with H ∞ ( F , α, T ) ≤ H G ( F , α, T ) and the folklore H G ( F , α, T ) ≤ O (sfat α ( F ) log( T/α )) , we conclude that the regret upper bounds O (inf α> 0 { αT + H ( F , α, T ) } ) , H ∈ {H ∞ , H G } , differ by at most a polylog factor.

Proof of Proposition C.2 Fix any α ′ > α > 0 and let d α ′ denote min { T, sfat 2 α ′ ( F ) } . Then there exists a context tree x and a witness tree s , both of depth d α ′ , such that for any path y ∈ { 0 , 1 } d α ′ , there exists an f ∈ F such that

Let V x ,α be an arbitrary sequential ℓ ∞ covering of F on x . Now we select a path y and a sequence of subsets V ( t ) x ,α ⊆ V x ,α , t ∈ [ d α ′ ] in the following recursive way. Define V (0) x ,α = V x ,α . For each t ∈ [ d α ′ ] , choose y t ∈ { 0 , 1 } such that 2 y t -1 ∈ {-1 , +1 } is the minority among all sgn( v t ( y 1: t -1 ) -s t ( y 1: t -1 )) , v ∈ V ( t -1) x ,α (ignoring those of 0 's). Finally update V ( t ) x ,α = { v ∈ V ( t -1) x ,α : sgn( v t ( y 1: t -1 ) -s t ( y 1: t -1 )) = 2 y t -1 } .

̸

First we argue that, if there is any time t ′ ∈ [ d α ′ ] such that V ( t ′ -1) x ,α = ∅ , V ( t ′ ) x ,α = ∅ , then V x ,α is not a valid cover of F on x . Otherwise, recall we have selected y 1 , . . . , y t ′ -1 . Now pick an arbitrary y t ′ ∈ { 0 , 1 } . By Eq. (15) we can find some f ∈ F such that (2 y t -1) · ( f ( x t ( y 1: t -1 )) -s t ( y 1: t -1 )) > α, ∀ t ∈ [ t ′ ] . Since V x ,α is a covering at scale α , there is v ∈ V x ,α such that | v t ( y ) -f ( x t ( y )) | ≤ α, ∀ t ∈ [ t ′ ] . This implies that sgn( f ( x t ( y )) -s t ( y )) = sgn( v t ( y ) -s t ( y )) = 2 y t -1 , ∀ t ∈ [ t ′ ] . So we can always find some member of V ( t ′ -1) x ,α to match the minority sign of v t ′ ( y 1: t ′ -1 ) -s t ′ ( y 1: t ′ -1 ) , v ∈ V ( t ′ -1) x ,α , which means that V ( t ′ ) x ,α = ∅ and yields a contradiction.

̸

Now we know that | V ( t ) x ,α | ≥ 1 , ∀ t ∈ [ d α ′ ] . By design | V ( t ) x ,α | ≤ | V ( t -1) x ,α | / 2 , ∀ t ∈ [ d α ′ ] , so we must have | V x ,α | = | V (0) x ,α | ≥ 2 d α ′ . As the choice of covering is arbitrary, the covering number N ∞ ( F ◦ x , α, d α ′ ) is also lower bounded by 2 d α ′ and hence H ∞ ( F , α, d α ′ ) ≥ d α ′ · log(2) . If sup α ′ >α sfat 2 α ′ ( F ) ≤ T , then we get that

If there is some α ′ > α such that sfat 2 α ′ ( F ) ≥ T , then

Combining these two cases together, we have

Proposition C.3 Let G α be a global sequential α -covering of F as defined in [WHGS23]. Then for any context tree x , there exists a sequential cover V x ,α of F ◦ x at scale α with | V x ,α | ≤ |G α | . This implies that H ∞ ( F , α, T ) ≤ log |G α | .

Proof of Proposition C.3 Fix arbitrary context tree x . For any g ∈ G α , define the [0 , 1] -valued tree v g by v g t ( y ) = g ( x 1: t ( y )) , ∀ t ∈ [ T ] , y ∈ Y T . Now let V x ,α = { v g : g ∈ G α } and we will show that V x ,α is indeed a sequential cover of F ◦ x at scale α .

For any f ∈ F and y ∈ Y T , tree x yields a length -T sequence x 1: T ( y ) and by definition of the global sequential covering, there exists g ∈ G α such that

So by our construction of V x ,α , v g ∈ V x ,α holds

which yields our claim after observing | V x ,α | ≤ |G α | .

D Additional proofs

Lemma D.1 For any X -valued Y -ary context tree x of depth T , and f : ( X × Y ) ∗ ×X → ∆( Y ) , we have

■

where we recall that x ( y ) denotes the context sequence ( x 1 ( y ) , . . . , x T ( y )) .

Proof of Lemma D.1 This is done by induction on the depth T . The key observation is that for any label sequence y , x t ( y ) = x t ( y 1 , . . . , y t -1 ) only depends on the first t -1 labels. For T = 1 , any context tree x is represented by its root node x 1 ( · ) = x 1 ∈ X and hence

Suppose Eq. (16) holds for all context trees x of depth T ≤ d and all sequential functions f . Now given any context tree x = ( x 1 , . . . , x d +1 ) of depth T = d + 1 , we denote its depth d subtree ( x 1 , . . . , x d ) by x [ d ] . Then

where the last step is due to induction. We are done.

■

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA]

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract summarizes theorems we have proven.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in the Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We don't see how to justify this without machine checkable proofs, which we have not provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: There is no data or code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: There are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the code and do not see any violation. Our work relates to the mathematical foundations of a basic task in ML.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper presents a mathematical characterization of the limits of probabilistic forecasting, and a (meta)algorithm that achieves these limits. For any class of interest, there remains significant work to realize that algorithm in an efficient way. As such, our impact is most directly on the theoretical community, who might then have direct societal impact by producing an minimax optimal algorithm. As such, our societal impact may be great, but it will always be quite indirect.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing models or data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We use no such assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No such experiments were performed.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of sequential probability assignment under logarithmic loss with context and assumes that both the contexts and labels are generated adversarially. The main contribution is a complete characterization of the minimax regret via a new complexity measure named \"contextual Shartkov sum.\" In particular, this allows the authors to recover and improve several previously known bounds with simpler proofs. The authors further demonstrate that such minimax regret can be achieved via a contextual normalized maximum likelihood predictor.\n\nSTRENGTHS:\nSequential probability assignment is a fundamental problem that finds many applications, such as universal compression, portfolio optimization, interactive decision making, and even the famous next-token prediction in LLMs. This is a clear significant result, which provides the first precise minimax regret of adversarial sequential probability assignment under log-loss and provides an explicit algorithm achieving such regrets. Given the importance and foundational nature of the problem, I would expect the paper to have a broad impact on the community.\n\nAlthough most of the proof ideas, such as the switching order of inf and sup, follow from prior literature, the truly novel technique that bypassed the difficulties faced by prior results is Lemma A.3. This allows the authors to characterize the minimax value directly (leveraging properties of log-loss) without resorting to the offset Rademacher complexity as in [RS15]. I also find the optimization problem in line 206 to be of independent interest. Although the paper is fairly preliminary (see point 4 in the Weakness section), it would serve as a stepping stone for important future research.\n\nOverall, this is a significant result that would inspire follow-up research. Therefore, I recommend a \"Strong Accept.\"\n\nWEAKNESSES:\nI do not see any significant weaknesses in the paper. However, I would like to outline a few minor remarks as follows:\n\n1. I believe the characterization in the current paper is similar in spirit to that of [BHS23], who provides a general reduction from the smoothed adversarial to the fixed design regret that also leverages the minimax switches. (Although I believe the techniques developed in the current paper could also recover this result.)\n\n2. I think Algorithm 1 can be fit into the relaxation-based algorithmic framework introduced by [RSS12]. (This does not mean the current algorithm is not novel.)\n\n3. It would be good if the authors could provide a short argument to explain why the P^* in line 507 attains the optimal of line 506. (This is clear to experts but is not immediately obvious for non-expert readers.)\n\n4. Section 3.2 could have been developed in much more detail, for example, by providing some (simple) examples to demonstrate the non-triviality. See also the Question section for more concrete examples.\n\n[BHS23] Bhatt, A., Haghtalab, N., and Shetty, A. (2023). Smoothed analysis of sequential probability assignment. NeurIPS 2023.\n\n[RSS12] Rakhlin, S., Shamir, O., & Sridharan, K. (2012). Relax and randomize: From value to algorithms. NeurIPS 2012.\n\nQUESTIONS:\nAlthough the contextual Shtarkov sum representation can be used to recover the upper bounds of [BFR20] and [WHGS23], potentially with improved constants, it would be beneficial to see some explicit examples demonstrating the benefits. For example, consider the expert class $\\mathcal{F}=\\left\\\\{\\frac{\\langle w,x\\rangle+1}{2} : w \\in B_2\\right\\\\}$, where $B_2$ is the unit $L_2$ ball. Assume the contexts are also selected from $B_2$. We know that the bounds in [BFR20] and [WHGS23] can only provide an $\\tilde{O}(T^{2/3})$ upper bound. However, it is known from [RS15] that a Follow-the-Regularized-Leader predictor achieves regret upper bounded by $\\tilde{O}(\\sqrt{T})$.\n\nCan the characterization introduced in the current paper provide an alternative (perhaps simpler) derivation of this $\\tilde{O}(\\sqrt{T})$ regret?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-12"}

comments: SUMMARY:
The authors explore the fundamental problem of sequential probability assignments, specifically focusing on log-loss online learning with an arbitrary, possibly non-parametric hypothesis class. They introduce a new complexity measure and demonstrate that the worst-case contextual Shtarkov sum equals minimal regret. With this, they derive a minimal optimal strategy, called cNML, which extends beyond binary labels.

STRENGTHS:
The paper successfully generalizes classical results in sequential probability assignment to contexts that involve multiary labels and sequential hypothesis classes. 
Another strength of this paper is its robust characterization of minimax regret and optimal prediction strategies holds for arbitrary hypothesis classes. This is particularly good as it relaxes the assumptions often required in earlier works.

WEAKNESSES:
The paper does not include experimental results to support the theoretical findings.

QUESTIONS:
Perhaps the authors could conduct some experiments to compare the performance of different algorithms, so as to visually demonstrate the effectiveness of the newly proposed cNML algorithm compared to other algorithms (e.g., traditional NML, Bayesian methods, etc.) on real datasets, especially when dealing with multi-label and complex sequence prediction problems.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors consider the problem of sequential probability assignment, also known as online learning with the log-loss.  While earlier results have shown that in the context-free or transductive settings, the Shtarkov complexity is minimax optimal (and attained with normalized maximum likelihood), the contextual analogue was bounded only in terms of sequential covering numbers.  The authors propose a new complexity measure, the contextual Shtarkov sum, that is precisely equal to the minimax regret.  The authors then bound this complexity in terms of the sequential covering numbers in a way that improves marginally on prior work.\n\nSTRENGTHS:\nThe paper discusses an important problem (that of sequential probability assignment) and provides a new bound on the minimax regret.  This new bound depends on a new notion of complexity that acts as the analogue of the sequential Rademacher complexity for log loss.  The paper is presented well and explains its technical details clearly.\n\nWEAKNESSES:\nThe paper's contributions are somewhat incremental.  The Shtarkov complexity is well-known for case of transductive learning and the introduction of contexts uses techniques introduced and used in many earlier works on sequential Rademacher complexity.  This is not helped by the fact that the Shtarkov complexity itself is almost tautalogically equal to the minimax regret.  This weakness would be somewhat mitigated if the new complexity measure led to improved bounds on the minimax regret, but the claimed improvement over BFR20 is only at the level of a constant as, asymptotically as $\\alpha \\downarrow 0$, the upper bound still scales like $\\inf_\\alpha \\{ 2 T \\alpha + \\mathcal H_\\infty(\\mathcal F, \\alpha, T) \\}$.  Furthermore, on the question of depth of technical contribution, it is not clear to me what is really novel.\n\nQUESTIONS:\n1. Could the authors more clearly explain the novelty in technical contributions in their work?\n\n2.  When invoking the Sion's minimax theorem in the proof of Theorem 3.2, the compactness of the space should be mentioned as it is a necessary condition.  The proof is correct, however.\n\n3. I assume that the $H(P)$ in line 206 is entropy, but I have a hard time seeing where this notation is introduced in the paper.\n\n4. While section 3.2 seems like an interesting interpretation of the Shtarkov sum, it seems like a bit of a non sequitur.  How does it relate to the rest of the paper?\n\n5. Much is made of the fact that the approaches in this paper address the multi-label setting, beyond binary $\\mathcal Y$, but it is not clear why this is technically more challenging.  Indeed, do the earlier works of BFR20 not immediately generalize beyond binary $\\mathcal Y$?\n\nLIMITATIONS:\nThe discussion of limitations is adequate, although mention of the computational challenges of their algorithm even in the case of convex classes (due to the necessity of optimizing over the space of trees, which is exponentially large) would be good to include.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies minimax regret in the online learning setup. The paper extends prior works to nonbinary labels and proves improved bounds on the regret. The analysis is through studying the Shtarkov sum that characterizes the minimax regret.  The authors develop a data-dependent variant of the contextual Shtarkov sum and introduce a variant of the Normalized Maximum Likelihood. Moreover, the paper takes an information-theoretic approach and studies sequential $\\ell_\\infty$ entropy terms to give upper bounds on regret.\n\nSTRENGTHS:\nThe paper is solid and presents a nice mathematical framework to prove bounds on the regret in online learning. The frameworks allows one to study a broader class of online learning problems including non binary labels. In addition it makes the proof a bit nicer with slightly improved bounds on the regret. Overall I liked the technical contributions of the paper as a framework to study various online learning problems.\n\nWEAKNESSES:\nOn the negative, side it seems that the novelty of the work is somewhat limited, given the cited prior works. It looks like that the authors extend the Sharkov sum and its analysis to the contextual case.\n\nThe presentation of the paper could be better. It starts with elementary definitions and explanations of the problem but jumps to the technical parts without a smooth transition. \n\nMoreover, the seemed that the paper is dry and lacks giving enough motivation and explanation.  For instance, the authors introduce multiary trees without explaining the high-level picture. It is not clear why and how this concept is used in the proofs. As another example, it is not clear at the beginning why the proposed algorithm (cNML) is useful.  Perhaps a discussion on this matter helps the reader kept interested in the paper.\n\nQUESTIONS:\nHas a contextual algorithm similar cNML been introduced before ? If so what are the benefits ? \n\nAny specific example of the multiary tree would be appreciated.\n\nLIMITATIONS:\nNothing significant.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}