Title: Online Estimation via Offline Estimation: An Information-Theoretic Framework
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

interactive decision making

oracle-efficient

regression

classification

conditional density estimation
Abstract: The classical theory of statistical estimation aims to estimate a parameter of interest under data generated from a fixed design (''offline estimation''), while the contemporary theory of online learning provides algorithms for estimation under adaptively chosen covariates (''online estimation''). Motivated by connections between estimation and interactive decision making, we ask: is it possible to convert offline estimation algorithms into online estimation algorithms in a black-box fashion? We investigate this question from an information-theoretic perspective by introducing a new framework, Oracle-Efficient Online Estimation (OEOE), where the learner can only interact with the data stream indirectly through a sequence of offline estimators produced by a black-box algorithm operating on the stream. Our main results settle the statistical and computational complexity of online estimation in this framework.

   $\bullet$ Statistical complexity. We show that information-theoretically, there exist algorithms that achieve near-optimal online estimation error via black-box offline estimation oracles, and give a nearly-tight characterization for minimax rates in the OEOE framework.

  $\bullet$ Computational complexity. We show that the guarantees above cannot be achieved in a computationally efficient fashion in general, but give a refined characterization for the special case of conditional density estimation: computationally efficient online estimation via black-box offline estimation is possible whenever it is possible via unrestricted algorithms.

Finally, we apply our results to give offline oracle-efficient algorithms for interactive decision making.

Online Estimation via Offline Estimation: An Information-Theoretic Framework

Abstract

The classical theory of statistical estimation aims to estimate a parameter of interest under data generated from a fixed design ('offline estimation'), while the contemporary theory of online learning provides algorithms for estimation under adaptively chosen covariates ('online estimation'). Motivated by connections between estimation and interactive decision making, we ask: is it possible to convert offline estimation algorithms into online estimation algorithms in a black-box fashion? We investigate this question from an information-theoretic perspective by introducing a new framework, Oracle-Efficient Online Estimation (OEOE), where the learner can only interact with the data stream indirectly through a sequence of offline estimators produced by a black-box algorithm operating on the stream. Our main results settle the statistical and computational complexity of online estimation in this framework.

Statistical complexity. We show that information-theoretically, there exist algorithms that achieve near-optimal online estimation error via black-box offline estimation oracles, and give a nearly-tight characterization for minimax rates in the OEOE framework.

Computational complexity. We show that the guarantees above cannot be achieved in a computationally efficient fashion in general, but give a refined characterization for the special case of conditional density estimation: computationally efficient online estimation via black-box offline estimation is possible whenever it is possible via unrestricted algorithms.

Finally, we apply our results to give offline oracle-efficient algorithms for interactive decision making.

1 Introduction

Consider a general framework for statistical estimation specified by a tuple ( X , Y , Z , K , F ) , which we will show encompasses classification, regression, and conditional density estimation. The learner is given a parameter space F (typically a function class), where each parameter f ∈ F is a map from the space of covariates X to the space of values Z . For an integer T ≥ 1 , the learner is given a dataset ( x 1 , y 1 ) , . . . , ( x T , y T ) , where x 1 , . . . , x T are covariates and y 1 , . . . , y T are outcomes generated via y t ∼ K ( · | f ⋆ ( x t )) , where f ⋆ ∈ F is an unknown target parameter that the learner wishes to estimate; here K is a probability kernel that assigns to each value z ∈ Z a distribution K ( · | z ) on the space of outcomes Y . We adopt the shorthand K ( z ) = K ( · | z ) throughout.

The classical theory of statistical estimation typically assumes that the covariates x 1 , . . . , x T are an arbitrary fixed design, and is concerned with estimating the target parameter f ⋆ ∈ F well in-distribution [64, 40, 62]. Formally, for a loss function D : Z×Z → R ≥ 0 on the space of values Z , the goal of the learner is to output an estimator ̂ f based on ( x 1 , y 1 ) , . . . , ( x T , y T ) such that the in-distribution error

is small; we refer to this as an offline estimation guarantee. Canonical examples include:

̸

Classification (i.e., distribution-free PAC learning [38, 63]), where Z = Y = { 0 , 1 } , K ( f ⋆ ( x )) = f ⋆ ( x ) , 1 and D 0 / 1 ( ̂ f ( x ) , f ⋆ ( x ) ) = { ̂ f ( x ) = f ⋆ ( x ) } is the 0 / 1 -loss.

✶ ✶ · Regression with a well-specified model [62, 66], where Z = Y = R , K ( f ⋆ ( x )) = N ( f ⋆ ( x ) , σ 2 ) , and D sq ( ̂ f ( x ) , f ⋆ ( x ) ) = ( ̂ f ( x ) -f ⋆ ( x )) 2 is the square loss.

Conditional density estimation [12], where Y is an arbitrary alphabet, Z = ∆( Y ) , K ( f ⋆ ( x )) = f ⋆ ( x ) , and D 2 H ( · , · ) is squared Hellinger distance; see Appendix C for details.

In parallel to statistical estimation, the contemporary theory of online learning [17, 49] provides estimation error algorithms that support adaptively chosen sequences of covariates, a meaningful form of out-of-distribution guarantee. Here, the examples ( x t , y t ) arrive one at a time. For each step t ∈ [ T ] , the learner produces an estimator ̂ f t : X → Z based on the data ( x 1 , y 1 ) , . . . , ( x t -1 , y t -1 ) observed so far. The covariate x t is then chosen in an arbitrary fashion, and the outcome is generated via y t ∼ K ( f ⋆ ( x t )) and revealed to the learner. The quality of the estimators is measured via 2

We refer to this as an online estimation guarantee; classical examples include online classification in the mistake-bound model [41], online regression [51], and online conditional density estimation [11]. Online estimation provides a non-trivial out-of-distribution guarantee, as it requires (on average) that the learner achieves non-trivial estimation performance on covariates x t that can be arbitrarily far from the previous examples x 1 , . . . , x t -1 . This property has many applications in algorithm design, notably in the context of interactive decision making , where it has recently found extensive use for problems including contextual bandits [25, 58, 24], reinforcement learning [27, 28], and imitation learning [56, 55].

In this paper, we investigate the relative power of online and offline estimation through a new information-theoretic perspective. It is well known that any algorithm for online estimation can be used as-is to solve offline estimation through online-to-batch conversion , a standard technique in learning theory and statistics [3, 9, 16, 61, 36, 8]. The converse is less apparent, as online estimation requires non-trivial algorithm design techniques that go well beyond classical estimators like least-squares or maximum likelihood [17]. In the case of regression with a finite class F , least squares achieves optimal offline estimation error Est Off D ( T ) ≤ O (log |F| ) , 3 and while it is possible to achieve a similar rate Est On D ( T ) ≤ O (log |F| ) for online estimation, this requires V ovk's aggregating algorithm or exponential weights [65]; directly applying least squares or other standard offline estimators leads to vacuous guarantees. This leads us to ask: Is it possible to convert offline estimation algorithms into online estimation algorithms in a black-box fashion?

Computationally speaking, this question has practical significance, since online estimation algorithms are typically far less efficient than their offline counterparts (the classical exponential weights algorithm maintains a separate weight for every f ∈ F , which is exponentially less memory-efficient than empirical risk minimization). In fact, at first glance this seems like a purely computational question: if the learner has access to an offline estimator, nothing is stopping them (information-theoretically) from throwing the estimator away and using the data to run an online estimation algorithm. 4 Yet, for aforementioned applications in interactive decision making [25, 58, 24, 27, 28, 56, 55], estimation algorithms-particularly online estimators-play a deeper information-theoretic role, and can be viewed as compressing the data stream into a succinct, operational representation that directly informs downstream decision making. With these applications in mind, the first contribution of this paper is to introduce a new protocol, Oracle-Efficient Online Estimation , which provides an information-theoretic abstraction of the role of online versus offline estimation, analogous to the framework of information-based complexity in optimization [43, 60, 48, 2] and statistical query complexity in theoretical computer science [13, 37, 21, 22].

Protocol 1 Oracle-Efficient Online Estimation (OEOE)

1: for t = 1 , . . . , T do

2: Learner receives estimator ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) from offline est. oracle.

3: Based on x 1 , . . . , x t -1 and ̂ f 1 , . . . , ̂ f t , learner produces an estimator ¯ f t ∈ Z X , which may be randomized according to a distribution µ t .

4: Based on µ t , nature selects covariate x t ∈ X and outcome y t ∼ K ( f ⋆ ( x t )) , but does not directly reveal them to the learner.

1.1 Our protocol: oracle-efficient online estimation

In the Oracle-Efficient Online Estimation (OEOE) framework, the aim is to perform online estimation in the sense of Eq. (2), with the twist that the learner does not directly observe the outcomes y 1 , . . . , y T ; rather, they interact with the environment indirectly through a sequence of offline estimators produced by a black-box algorithm operating on the historical data. We formalize this black-box algorithm as an estimation oracle Alg Off = { Alg t Off } T t =1 (e.g., Foster and Rakhlin [26]), which is a mapping from histories to estimators that enjoy bounded offline estimation error.

Definition 1.1 (Offline estimation oracle) . An offline estimation oracle Alg Off = { Alg t Off } T t =1 for a statistical estimation instance ( X , Y , Z , K , F ) and loss D is a mapping Alg t Off : ( X × Y ) t -1 → ( X → Z ) such that for any sequence ( x 1 , y 1 ) , . . . , ( x T , y T ) with y t ∼ K ( f ⋆ ( x t )) , the sequence of estimators ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) satisfies Est Off D ( t ) := ∑ t -1 s =1 D ( ̂ f t ( x s ) , f ⋆ ( x s ) ) ≤ β Off for all t ∈ [ T ] almost surely; we allow x t to be selected adaptively based on y 1 , . . . , y t -1 and ̂ f 1 , . . . , ̂ f t -1 . We refer to β Off ≥ 0 as the offline estimation parameter .

This definition simply asserts that the estimators ̂ f t produced by the offline estimation oracle satisfy the guarantee in Eq. (1), even when the covariates are selected adaptively. Examples include standard algorithms like least-squares for regression and maximum likelihood for conditional density estimation, which guarantee β Off ≤ O (log |F| ) with high probability whenever F is a finite class; see Appendix C.1 for further background. 5 Throughout the paper, we assume for simplicity that β Off > 0 is known in advance.

With this definition, we present the Oracle-Efficient Online Estimation protocol in Protocol 1. In the protocol, a learner aims to perform online estimation, but at each step t , the only information available is the covariates x 1 , . . . , x t -1 and the estimators ̂ f 1 , . . . , ̂ f t generated by an offline estimation oracle satisfying Definition 1.1; the outcomes y 1 , . . . , y T are not directly observed. Based on this information, the learner produces a new estimator ¯ f t such that the online estimation error Est On D ( T ) = ∑ T t =1 E ¯ f ∼ µ t [ D ( ¯ f ( x t ) , f ⋆ ( x t ) )] in Eq. (2) is minimized. 6 An algorithm is termed oracle-efficient if it attains low online estimation error (2) in the OEOE framework. Note that while the learner cannot directly observe the outcomes y 1 , . . . , y T , the covariates x 1 , . . . , x T are observed; we prove that without this ability, it is impossible to achieve non-trivial estimation performance (Section 3) .

The OEOE framework abstracts away the property that oracle-efficient algorithms implicitly interact with the environment through a compressed, potentially lossy channel (the estimation oracle Alg Est ). We believe this property merits deeper investigation: it is shared by essentially all algorithms from recent research that reduces interactive decision making and reinforcement learning to estimation oracles [25, 58, 27, 28, 56, 55], yet the relative power of offline oracles and analogously defined online oracles is poorly understood in this context. By providing an information-theoretic abstraction to study oracle-efficiency, the OEOE framework plays a role similar to information-based complexity in optimization [43, 60, 48, 2] and statistical query complexity in theoretical computer science [13, 37, 21, 22], both of which provide rich frameworks for designing and evaluating iterative algorithms that interact with the environment in a structured fashion. We expect that this abstraction will find broader use for more complex domains (e.g., decision making and active learning) as a means to guide algorithm design and prove lower bounds against natural classes of algorithms.

̸

Let us first build some intuition. Familiar readers may recognize that the classical halving algorithm for binary classification (e.g., Cesa-Bianchi and Lugosi [17]) can be viewed as oracle-efficient in our framework. Specifically, for binary classification with Y = Z = { 0 , 1 } and loss function D 0 / 1 ( ̂ f ( x ) , f ⋆ ( x ) ) = ✶ { ̂ f ( x ) = f ⋆ ( x ) } , the halving algorithm can use any offline oracle with β Off = 0 to achieve Est On D ( T ) = O (log |F| ) , which is optimal. However, little is known for noisy oracles with β Off > 0 , or more general outcome spaces and loss functions (e.g., regression or density estimation). In addition, the halving algorithm-while oracle-efficient-is computationally inefficient, as it requires maintaining an explicit version space. This leads us to restate our central question formally, in two parts:

Can we design oracle-efficient algorithms with near-optimal online estimation error (2) , up to polynomial factors (for general instances ( X , Y , Z , K , F ) and β Off > 0 )?

Can we do so in a computationally efficient fashion?

1.2 Contributions

For a general class of losses D , referred to as metric-like , we settle the statistical and computational complexity of performing online estimation via black-box offline estimation oracles up to mild gaps, answering questions (1) and (2) above.

Statistical complexity. Our first result concerning statistical complexity focuses on finite classes F , where the optimal rates for offline and online estimation with standard losses D ( · , · ) both scale as Θ(log |F| ) . For this setting, we show (Theorem 3.1) that there exists an oracle-efficient online estimation algorithm that achieves Est On D ( T ) = O (( β Off + 1) min { log |F| , |X|} ) in the OEOE framework, and that this is optimal (Theorem 3.2). This provides an affirmative answer to question (1), and characterizes the statistical complexity of oracle-efficient online estimation with finite classes F .

In the general OEOE framework, the learner can use the entire history of offline estimators ̂ f 1 , . . . , ̂ f t and covariates x 1 , . . . , x t -1 to produce the online estimator ¯ f t for step t . As a secondary result, we study a restricted class of memoryless oracle-efficient algorithms that choose ¯ f t only based on the most recent offline estimator ̂ f t , and show (Theorem 3.3) that it is impossible for such algorithms to achieve low online estimation error.

Lastly, we give a more general approach to deriving oracle-efficient reductions (Theorem D.1) that is based on delayed online learning [67, 42, 35, 47]. Using this result, we give a characterization of learnability with infinite classes for binary classification in the OEOE framework (Theorem D.2), proving that finite Littlestone dimension is necessary and sufficient for oracle-efficient learnability.

Computational complexity. On the computational side, we provide a negative answer to question (2), showing (Theorem 4.1) that under standard conjectures in computational complexity, there do not exist polynomial-time algorithms with non-trivial online estimation error in OEOE framework. In spite of this negative result, we provide a fine-grained perspective for the statistical problem of conditional density estimation , a general task that subsumes classification and regression and has immediate applications to reinforcement learning and interactive decision making [27, 28]. Here we show, perhaps surprisingly (Theorem 4.2), that online estimation in the OEOE framework is no harder computationally than online estimation with arbitrary, unrestricted algorithms. This result is salient in light of the applications we discuss below.

Implications for interactive decision making. As the preceding discussion has alluded to, our interest in studying oracle-efficient online estimation is largely motivated by a connection to the problem of interactive decision making . Foster et al. [27, 28], Foster and Rakhlin [26] propose a general framework for interactive decision making called Decision Making with Structured Observations (DMSO), which subsumes contextual bandits, bandit problems with structured rewards, and reinforcement learning with general function approximation. They show that for any decision making problem in the DMSO framework, there exists an algorithm that, given access to an online estimation algorithm (or, 'oracle') for conditional density estimation for an appropriate class F , it is possible to achieve near-optimal regret . The results above critically make use of online estimation oracles, as they require achieving low estimation error for adaptively chosen sequences of covariates, and it is natural to ask whether similar guarantees can be achieved using only offline estimation oracles. However, positive results are only known for certain special cases [18-20, 58], with scant results for reinforcement learning

in particular. In this context, our results have the following implication (Corollary E.1): Informationtheoretically, it is possible to achieve near-optimal regret for any interactive decision making problem using an algorithm that accesses the data stream only through offline estimation oracles .

Additional results. Due to space constraints, the following results are deferred to the appendix: (i) detailed examples for our statistical estimation framework (Appendix C); (ii) additional results concerning statistical complexity of the OEOE framework (Appendix D); and (iii) detailed results for our application to interactive decision making (Appendix E).

2 Preliminaries

Unless otherwise stated, our results assume the loss function D has metric-like structure.

Definition 2.1 (Metric-like loss) . A loss function D : Z × Z → [0 , 1] is metric-like on the set Z if it is symmetric and satisfies (i) D ( z 1 , z 2 ) ≥ 0 for any z 1 , z 2 ∈ Z and D ( z, z ) = 0 for all z ∈ Z ; and (ii) D ( z 1 , z 2 ) ≤ C D · ( D ( z 1 , z 3 ) + D ( z 3 , z 2 )) for all z 1 , z 2 , z 3 ∈ Z , for an absolute constant C D ≥ 1 .

Throughout the paper, we focus on three canonical applications, outlined in the introduction: Classification with the indicator loss D 0 / 1 ( C D = 1 ), regression with the square loss D sq ( C D = 2 ), and conditional density estimation with squared Hellinger distance D 2 H ( C D = 2 ). See Appendix C for detailed examples and discussion. (omitted for space).

Finite versus infinite classes. The majority of our results focus on finite classes F . We believe this captures the essential difficulty of the problem, but we expect that most of our sample complexity results (which typically scale with log |F| ) can be extended to infinite classes by combining our techniques with appropriate notions of complexity for the function class (Littlestone dimension for classification, sequential Rademacher complexity, and sequential covering numbers [50, 41, 52]). For the canonical settings of classification, regression, and conditional density estimation, there exist algorithms that achieve Est Off D ( T ) = O (log |F| ) and Est On D ( T ) = O (log |F| ) for arbitrary finite classes; see Appendix C for details.

We defer additional notation and related work to Appendices A and B

3 Statistical complexity of oracle-efficient online estimation

This section presents our main results concerning the statistical complexity of oracle-efficient online estimation. In Section 3.1, we focus on finite classes F and present an oracle-efficient algorithm that achieves near-optimal online estimation error (Theorem 3.1). We then provide a lower bound that shows that our reduction is near optimal (Theorem 3.2). In Section 3.2, we turn our attention to memoryless oracle-efficient algorithms, proving strong impossibility results (Theorem 3.3).

3.1 Minimax sample complexity for oracle-efficient algorithms

In this section, we present our main statistical conclusion for the OEOE framework: For any finite class F , it is possible to transform any black-box offline estimation algorithm into an online estimation algorithm with near-optimal error (up to a logarithmic factor that we show is unavoidable).

Algorithm and minimax upper bound. Our results are achieved through a new algorithm, Version Space Averaging , described in Algorithm 1. At each round t , the algorithm uses estimators ̂ f 1 , . . . , ̂ f t produced by an offline estimation oracle Alg Off , along with the previous covariates x 1 , . . . , x t -1 , to construct a version space F t ⊆ F in Eq. (3). Informally, F t consists of all f ∈ F that are consistent with the estimators ̂ f 1 , . . . , ̂ f t in the sense that for all s ∈ [ t ] , the offline estimation error relative to ̂ f s is small; as long as the offline estimation oracle Alg Off has offline estimation error β Off (Definition 1.1), it follows immediately that the construction in Eq. (3) satisfies f ⋆ ∈ F t . Given the version space F t , Algorithm 1 predicts by uniformly sampling: ¯ f t ∼ µ t := Unif( F t ) , then proceeds to the next round. 7 The main guarantee for Algorithm 1 is stated in Theorem 3.1.

̸

Algorithm 1 Version Space Averaging

1: input: parameter space F , offline estimation oracle Alg Off with parameter β Off ≥ 0 .

2: for t = 1 , 2 , . . . , T do

3: Receive ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) .

4: Calculate version space:

5: Predict ¯ f t ∼ µ t := Unif( F t ) and receive x t . // Nature draws y t ∼ K ( f ⋆ ( x t )) and passes to Alg Off .

Theorem 3.1 (Main upper bound for OEOE) . For any instance ( X , Y , Z , K , F ) , any metric-like loss D , and any offline estimator Alg Off with parameter β Off ≥ 0 , Algorithm 1 is oracle-efficient and achieves

Most notably, Algorithm 1 achieves Est On D ( T ) ≤ O ( C D · ( β Off +1) · log |F| ) ; that is, up to a O (log |F| ) factor, the reduction achieves online estimation rates in the OEOE framework that are no worse than the minimax rate for offline estimation. For classification, regression, and density estimation with generic finite classes F (Appendix C), the best possible offline estimation error rate is β Off = O (log |F| ) , so this shows that price of oracle-efficiency is at most quadratic.

Minimax lower bound. Next, we show that the upper bound in Theorem 3.1 is nearly tight, giving a lower bound that matches up to logarithmic factors.

Theorem 3.2 (Main lower bound for OEOE) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off > 0 , there exists an instance ( X , Y , Z , K , F ) with log |F| = |X| = N such that for any oracle-efficient algorithm, there is a sequence of covariates ( x 1 , . . . , x T ) and offline oracle with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { ( β Off +1) N,T } ) .

This result states that for a generic finite class F and offline estimation oracle Alg Off , any oracleefficient online estimator must have

in the worst case. This implies that the log |F| factor we pay for offline to online conversion is unavoidable, and that Theorem 3.1 is optimal up to a log T factor, giving a near-optimal characterization for the minimax rate for online estimation in the OEOE framework. We conclude with two remarks: (i) The ( β Off +1) scaling (as opposed to say, β Off ) in Theorem 3.1 is unavoidable, as witnessed by the optimality of the halving algorithm for noiseless binary classification [17]; (ii) if the space Z and the loss D are convex, then we can change Algorithm 1 to output a deterministic prediction by using the average of all parameters in F t rather than the uniform distribution on F t . See Lemma G.1 for details.

General reductions and infinite classes. Algorithm 1 is somewhat specialized to finite classes. In Appendix D (deferred to the appendix for space), we provide a more general approach to designing oracle-efficient algorithms based on delayed online learning (Theorem D.1), and use it to derive a characterization of oracle-efficient learnability for classification with infinite classes F (Theorem D.2).

Full memory vs. finite memory. Algorithm 1 requires full memory of all past offline estimators. The more general approach proposed in Appendix D can use N most recent offline estimators to obtain an estimation error bound of O ( C D ( N + β Off T/N ) + N · log |F| ) (Corollary D.1) for any integer N > 0 .

3.2 Impossibility of memoryless oracle-efficient algorithms

In the general OEOE framework, the learner can use the entire history of estimators ̂ f 1 , . . . , ̂ f t and covariates x 1 , . . . , x t -1 to produce the online estimator ¯ f t for step t ; notably the Version Space Averaging algorithm with which our upper bounds in the prequel are derived uses the entire history. In this section, we show that for memoryless oracle-efficient algorithms (Definition 3.1) that select the estimator ¯ f t only as a function of the most recent offline estimator ̂ f t , similar guarantees are impossible.

Definition 3.1 (Memoryless algorithm) . An online estimation algorithm is memoryless if there exists a map F t ( · ) such that we can write µ t = F t ( ̂ f t ) , where ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) and µ t is the randomization distribution for the online estimator ¯ f t . 8

Memoryless algorithms are more practical than arbitrary algorithms, since they do not require storing past estimators or covariates in memory. Our motivation for studying memoryless algorithms arises from recent work in interactive decision making [25, 27], which shows that there exist near-optimal algorithms for contextual bandits and reinforcement learning that use estimation oracles in memoryless fashion. We show that unfortunately, it is not possible to convert offline estimators into memoryless online estimation algorithms with non-trivial error.

Theorem 3.3 (Impossibility of memoryless algorithms for OEOE) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off ≥ 0 , there exists an instance ( X , Y , Z , K , F ) with |F| = |X| = N such that for any memoryless oracle-efficient algorithm, there exists a sequence of covariates ( x 1 , . . . , x T ) and a (potentially improper) offline oracle Alg Off with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { N ( β Off +1) , T } ) . This conclusion still holds when the online estimation algorithm remembers ̂ f 1 , . . . , ̂ f t -1 but not x 1 , . . . , x t -1 .

This result shows that in the worst case, any memoryless oracle-efficient algorithm must have

This precludes an online estimation error bound scaling with ( β Off +1)log |F| as in Theorem 3.1, and shows that the gap between general and memoryless oracle-efficient algorithms can be exponential.

Interestingly, the lower bound in Theorem 3.3 holds even if the online estimation algorithm is allowed to remember ̂ f 1 , . . . , ̂ f t -1 , but not x 1 , . . . , x t -1 . The intuition here is that without covariate information, it is not possible to aggregate the predictions of previous estimators or otherwise use them to reduce uncertainty. This provides post-hoc motivation for our decision to incorporate covariate memory into the OEOE protocol in Section 1.1.

The proof of Theorem 3.3 uses that the estimators ̂ f t produced by the offline estimation oracle may be improper (i.e., ̂ f t / ∈ F ). We defer a variant of the result that holds even if the estimation oracle is proper under additional assumptions as well as the complementary upper bound to Appendix D.2

4 Computational complexity of oracle-efficient online estimation

In this section, we turn our attention to the computational complexity of oracle-efficient online estimation in the OEOE framework. In Section 4.1, we show (Theorem 4.1) that in general, it is not possible to transform black-box offline estimation algorithms into online estimation algorithms in a computationally efficient fashion. Then, in Section 4.2, we provide a more fine-grained perspective, showing (Theorem 4.2) that for conditional density estimation, online estimation in the OEOE framework is no harder computationally than online estimation with unrestricted algorithms.

4.1 Computational hardness of oracle-efficient estimation

Our main upper bounds (Section 3.1) show that online estimation error Est On D ( T ) ≤ O (( β Off + 1) log |F| ) can be achieved in an oracle-efficient fashion for any finite class F , but the algorithm (Algorithm 1) is not computationally efficient. We now show that this is fundamental: There exist classes F for which offline estimation can be performed in polynomial time, yet no oracle-efficient algorithm running in polynomial-time algorithm can achieve sublinear online estimation error.

Computational model. To present our results, we must formalize a computational model for oracle-efficient online estimation, and in particular, define a notion of input length for oracle-efficient online algorithms. To do so, we restrict our attention to noiseless binary classification, and consider a sequence of classification instances indexed by n ∈ N , with Z = Y = { 0 , 1 } , X n := { 0 , 1 } n , K ( z ) = ✶ z , and indicator loss D 0 / 1 ( · , · ) . We consider a sequence of classes F n that have polynomial description lengthi.e. log |F n | is polynomial in n , so that f ∈ F n can be described in poly( n ) bits. In particular, we assume that f ∈ F n is represented as a Boolean circuit of size poly( n ) so that f ( x ) can be computed in poly( n ) time for x ∈ X n ; we refer to such sequences as polynomially computable .

To allow for offline estimators that are improper, we assume that for all t and all sequences ( x 1 , y 1 ) , . . . , ( x T , y T ) , the output ̂ f t : { 0 , 1 } n → { 0 , 1 } returned by Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) is a Boolean circuit of size poly( n ) ; we refer to such oracles as having poly( n ) -output description length . 9 Likewise, to allow the online estimation algorithm itself to be improper and randomized, we restrict to algorithms for which computing ¯ f t ( x ) for ¯ f t ∼ µ t can be implemented as ¯ f t ( x, r ) for a random bit string r ∼ Unif( { 0 , 1 } B ) , where B = poly( n ) ; we refer to the online estimator as having poly( n ) -output description length if ¯ f t ( · , · ) is itself a Boolean circuit of size poly( n ) . We refer to the online estimation algorithm polynomial time if it runs in time poly( n ) for any sequence of inputs t , x 1 , . . . , x t -1 , and ̂ f 1 , . . . , ̂ f t -1 , and has poly( n ) -output description length. 10

Main lower bound. Our main computational lower bound is as follows.

Theorem 4.1 (Computational lower bound for OEOE) . Assume the existence of one-way functions. 11 There exists a sequence of polynomially computable classes ( F 1 , F 2 , . . . , F n , . . . ) , along with a sequence of poly( n ) -output description length offline oracles with β Off = 0 associated with each F n , such that for any fixed polynomials p, q : N → N and all n ∈ N sufficiently large, any oracle-efficient online estimation algorithm with runtime bounded by p ( n ) must have E [ Est On D ( T )] ≥ T/ 4 for all 1 ≤ T ≤ q ( n ) . At the same time, there exists an inefficient algorithm that achieves E [ Est On D ( T )] ≤ O ( √ n ) for all T ∈ N .

Informally, Theorem 4.1 shows that there exist a class F and offline estimation oracles Alg Off for which no oracle-efficient online estimation algorithm that runs in time

can achieve sublinear estimation error, where len ( X ) and len ( F ) denote the number of bits required to describe x ∈ X and f ∈ F , and len ( ̂ f t ) denotes the size of the circuit required to compute ̂ f t ( x ) . Yet, low online estimation error can be achieved by an inefficient algorithm, and there exist efficient offline estimators with β Off = 0 as well. The result is essentially a corollary of Blum [14]; we refer to Appendix H.1 for the detailed proof. We mention in passing that Hazan and Koren [33] also give lower bounds against reducing online learning to offline oracles, but in a somewhat different computational model; see Appendix B for detailed discussion.

Theorem 4.1 is slightly disappointing, since one of the main motivations for studying oracle-efficiency is to leverage offline estimators as a computational primitive. Combined with our results in Section 3, Theorem 4.1 shows that even though it is possible to be oracle-efficient information-theoretically, it is not possible to achieve this computationally. Nonetheless, we are optimistic that our abstraction can (i) aid in designing computationally efficient algorithms for learning settings beyond online estimation, and (ii) continue to serve as a tool to formalize lower bounds against natural classes of algorithms, as we have done here; see Section 5 for further discussion.

Remark 4.1. Theorem 4.1 relies on the fact that the offline estimator may be improper (i.e., ̂ f t / ∈ F ). An interesting open problem is whether one can attain poly(log |F n | ) · o ( T ) online estimation error with runtime poly(log |F n | ) given a proper offline estimation oracle with parameter β Off = 0 . ◁

4.2 Conditional density estimation: computationally efficient algorithms

In spite of the negative result in the prequel, which shows that efficient computation in the OEOE framework is not possible in general, we can provide a more fine-grained perspective on the computational complexity of oracle-efficient estimation for the problem conditional density estimation , a general task which subsumes classification and regression, and has immediate applications to reinforcement learning and interactive decision making [27, 28].

Conditional density estimation. Recall that conditional density estimation [71, 12] is the special case of the online estimation framework in Section 1 in which X and Y are arbitrary, Z = ∆( Y ) , and the kernel is K ( z ) = z ; that is sampling y ∼ K ( f ⋆ ( x )) is equivalent to sampling y ∼ f ⋆ ( x ) . For

the loss, we use squared Hellinger distance: D 2 H ( f ( x ) , f ⋆ ( x ) ) = 1 2 ∫ (√ f ( y | x ) -√ f ⋆ ( y | x ) ) 2 ; with online estimation error given by Est On H ( T ) = ∑ T t =1 D 2 H ( ̂ f t ( x t ) , f ⋆ ( x t ) ) .

Base algorithm. Our result is based on a reduction. We assume access to a base algorithm A CDE for online estimation in the Conditional Density Estimation (CDE) framework, which is unrestricted in the sense that it is not necessarily oracle-efficient. That is, A CDE can directly use the full data stream ( x 1 , y 1 ) , . . . , ( x t -1 , y t -1 ) at step t . For parameters R CDE ( T ) and C F ≥ 1 , we assume that for any f ⋆ ∈ F , the base algorithm A CDE ensures that for all δ ∈ (0 , e -1 ) , with probability at least 1 -δ ,

We define the total runtime for A CDE across all rounds as Time ( F , T ) .

Main result. Our main result shows that any algorithm A CDE satisfying the guarantee above can be transformed into an oracle-efficient algorithm with only polynomial blowup in runtime. For technical reasons, we assume that V := e ∨ sup f,f ′ ∈F ,x ∈X ,y ∈Y f ( y | x ) f ′ ( y | x ) is bounded; our sample complexity bounds scale only logarithmically with respect to this parameter. In addition, we assume that all f ∈ F and x ∈ X have O (1) description length, and that one can sample y ∼ f ( x ) in O (1) time.

Theorem 4.2. Let A CDE be an arbitrary (unrestricted) online estimation algorithm that satisfies Eq. (4) and has runtime Time ( F , T ) . Then for any N ∈ N , there exists an oracle-efficient online estimation algorithm that achieves estimation error

with runtime poly( Time ( F , T ) , log |F| , log |X| , T ) , where β Off ≥ 0 is the offline estimation parameter. The distributions µ 1 , . . . , µ T produced by the algorithm have support size poly(log |F| , log |X| , T ) . As a special case, if the online estimation guarantee for the base algorithm holds with R CDE ( T ) ≤ C ′ F log T for some problem-dependent constant C ′ F ≥ 1 , then by choosing N appropriately, we achieve E [ Est On H ( T ) ] ≤ ˜ O ( ( C F ( C F + C ′ F ) β Off ) 1 / 2 log V · T 1 / 2 +( C F + C ′ F ) log V ) .

Note that the estimation error bound in Theorem 4.2 is sublinear whenever the rate R CDE ( T ) is. This implies that for squared Hellinger distance, online estimation in the OEOE framework is no harder computationally than online estimation with arbitrary, unrestricted algorithms.

The proof of Theorem 4.2 is algorithmic, and is based on several layers of reductions. The main reason why the result is specialized to conditional density estimation is as follows: If we have an estimator ̂ f for which D 2 H ( ̂ f ( x ) , f ⋆ ( x ) ) is small for some x , we can simulate y ∼ f ⋆ ( x ) up to low statistical error by sampling y ∼ ̂ f ( x ) instead, as ̂ f and f ⋆ are close in distribution. This allows us to implement a scheme based on simulating outcomes and feeding them to the base algorithm.

5 Discussion

Our work introduces the Oracle-Efficient Online Estimation protocol as an information-theoretic framework to study the relative power of online and offline estimators and gives a nearly complete characterization of the statistical and computational complexity of learning in this framework. In what follows, we discuss broader implications for our information-abstraction of oracle-efficiency.

Oracle-efficient learning as a general framework for analysis of algorithms. One of the most important contributions of this work is to formalize oracle-efficient algorithms as mappings that act upon a sequence of estimators but do not directly act on historical outcomes. While the computational lower bounds we provide for oracle-efficient learning are somewhat disappointing, we are optimistic that-similar to statistical query complexity in TCS and information-based complexity in optimization-our abstraction can (i) aid in designing computationally efficient algorithms for learning settings beyond online estimation, and (ii) continue to serve as a tool to formalize lower bounds against natural classes of algorithms, for estimation and beyond. That is, we envision oracleefficient learning as a more general framework to study oracle-based algorithms in any type of interactive learning problem. We remark that one need not restrict to offline oracles; it is natural to study oracle-efficient algorithms based on online estimation oracles or other types of oracles through our information-theoretic abstraction as well. For concreteness, let us mention a couple of natural settings where our information-theoretic abstraction can be applied.

Oracle-efficient interactive decision making. For interactive decision making problems like bandits and reinforcement learning (more broadly, the DMSO framework described in Appendix E.1), it is natural to formalize oracle-effiicent algorithms as algorithms that do not directly observe rewards (bandits) or trajectories (reinforcement learning), and instead must select their decision based on an (online or offline) estimator (e.g., regression for bandits or conditional density estimation for RL). Foster and Rakhlin [25] and Foster et al. [27] et seq. provide algorithms with this property for contextual bandits and RL, respectively, but the power of offline oracles in this context is not well understood.

Oracle-efficient active learning. For active learning, it is natural to consider algorithms that decide whether to query the label for a point in an oracle-efficient fashion (e.g., Krishnamurthy et al. [39]). For concreteness, consider pool-based active learning [32]. Suppose the learner is given a pool P = { x 1 , ..., x n } of covariates and a parameter space F . The learner can repeatedly choose x t ∈ P and call the offline oracle to obtain an estimator f t such Est Off ( t ) := ∑ D ( f t ( x i ) , f ⋆ ( x i ) ) ≤

β Off (in contrast to an unrestricted algorithm that observes y t = f ( x t ) ). The aim is to learn a hypothesis with low classification error using the smallest number of queries possible. Can we design oracle-efficient algorithms that do so with near-optimal label complexity?

5.1 Further Directions

We close with some additional directions for future research.

Refined notions of estimation oracles. This work considers generic offline estimation algorithms that satisfy the statistical guarantee in Definition 1.1 but can otherwise be arbitrary. Understanding the power of offline estimators that satisfy more refined (e.g., problem-dependent) guarantees is an interesting direction for future research.

Open questions for proper versus improper learning. Our results leave some interesting gaps in the power of proper versus improper oracles. First, the computational lower bounds in Section 4.1, leave open the possibility of attaining poly(log |M n | ) · o ( T ) online estimation error with runtime poly(log |M n | ) given access to a proper offline estimation oracle with parameter β Off = 0 . Second, our results in Section 3.2 leave open the possibility of bypassing the Ω( |X| ( β Off +1)) lower bound for memoryless algorithms under the assumption that the offline oracle is proper.

Acknowledgments and Disclosure of Funding

We acknowledge support from the ARO through award W911NF-21-1-0328 and from the Simons Foundation and the NSF through award DMS-2031883.

References

Anders Aamand, Justin Y Chen, Huy Lê Nguyen, and Sandeep Silwal. Improved space bounds for learning with experts. arXiv preprint arXiv:2303.01453 , 2023.

Alekh Agarwal, Peter L Bartlett, Pradeep Ravikumar, and Martin J Wainwright. Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory , 5(58):3235-3249, 2012.

Aizerman, Braverman, and Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning. Automation and remote control , 25:821-837, 1964.

Philip Amortila, Dylan J Foster, Nan Jiang, Ayush Sekhari, and Tengyang Xie. Harnessing density ratios for online reinforcement learning. International Conference on Learning Representations (ICLR) , 2024.

Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199 (1-2):165-214, 2023.

Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach . Cambridge University Press, 2009.

Idan Attias, Steve Hanneke, Alkis Kalavasis, Amin Karbasi, and Grigoris Velegkas. Optimal learners for realizable regression: Pac learning and online learning. arXiv preprint arXiv:2307.03848 , 2023.

̂ D i<t ̂ ⋆

Jean-Yves Audibert. Fast learning rates in statistical inference through aggregation. The Annals of Statistics , 37(4):1591-1646, 2009.

Andrew R Barron. Are bayes rules consistent in information? In Open problems in communication and computation , pages 85-91. Springer, 1987.

Shai Ben-David, Dávid Pál, and Shai Shalev-Shwartz. Agnostic online learning. In COLT , volume 3, page 1, 2009.

Blair Bilodeau, Dylan J Foster, and Daniel Roy. Tight bounds on minimax regret under logarithmic loss via self-concordance. In International Conference on Machine Learning , 2020.

Blair Bilodeau, Dylan J Foster, and Daniel M Roy. Minimax rates for conditional density estimation via empirical entropy. The Annals of Statistics , 51(2):762-790, 2023.

Avrim Blum, Merrick Furst, Jeffrey Jackson, Michael Kearns, Yishay Mansour, and Steven Rudich. Weakly learning dnf and characterizing statistical query learning using fourier analysis. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing , pages 253-262, 1994.

Avrim L Blum. Separating distribution-free and mistake-bound learning models over the boolean domain. SIAM Journal on Computing , 23(5):990-1000, 1994.

Mark Bun. A computational separation between private learning and online learning. Advances in Neural Information Processing Systems , 33:20732-20743, 2020.

Olivier Catoni. The mixture approach to universal model selection. 1997.

Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory , 50(9):2050-2057, 2004.

Constantinos Daskalakis and Vasilis Syrgkanis. Learning in auctions: Regret is hard, envy is easy. In 2016 ieee 57th annual symposium on foundations of computer science (focs) , pages 219-228. IEEE, 2016.

Miroslav Dudík, Nika Haghtalab, Haipeng Luo, Robert E Schapire, Vasilis Syrgkanis, and Jennifer Wortman Vaughan. Oracle-efficient online learning and auction design. Journal of the ACM (JACM) , 67(5):1-57, 2020.

Vitaly Feldman. A complete characterization of statistical query learning with applications to evolvability. Journal of Computer and System Sciences , 78(5):1444-1459, 2012.

Vitaly Feldman. A general characterization of the statistical query complexity. In Conference on Learning Theory , pages 785-830. PMLR, 2017.

Vitaly Feldman, Cristobal Guzman, and Santosh Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1265-1277. SIAM, 2017.

Dylan J Foster and Akshay Krishnamurthy. Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. Neural Information Processing Systems (NeurIPS) , 2021.

Dylan J Foster and Alexander Rakhlin. Beyond UCB: Optimal and efficient contextual bandits with regression oracles. International Conference on Machine Learning (ICML) , 2020.

Dylan J Foster and Alexander Rakhlin. Foundations of reinforcement learning and interactive decision making. arXiv preprint arXiv:2312.16730 , 2023.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. Conference on Learning Theory (COLT) , 2023.

Dylan J Foster, Noah Golowich, Jian Qian, Alexander Rakhlin, and Ayush Sekhari. Model-free reinforcement learning with the decision-estimation coefficient. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Oded Goldreich et al. Foundations of cryptography-a primer. Foundations and Trends® in Theoretical Computer Science , 1(1):1-116, 2005.

Alon Gonen, Elad Hazan, and Shay Moran. Private learning implies online learning: An efficient reduction. Advances in Neural Information Processing Systems , 32, 2019.

Steve Hanneke. Theory of disagreement-based active learning. Foundations and Trends® in Machine Learning , 7(2-3):131-309, 2014.

Elad Hazan and Tomer Koren. The computational power of optimization in online learning. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 128-141, 2016.

TS Jayram. Hellinger strikes back: A note on the multi-party information complexity of and. In International Workshop on Approximation Algorithms for Combinatorial Optimization , pages 562-573. Springer, 2009.

Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In International Conference on Machine Learning , pages 1453-1461. PMLR, 2013.

Anatoli Juditsky, Philippe Rigollet, and Alexandre B Tsybakov. Learning by mirror averaging. The Annals of Statistics , 36(5):2183-2206, 2008.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

Michael Kearns, Ming Li, Leonard Pitt, and Leslie Valiant. On the learnability of boolean formulae. In Proceedings of the nineteenth annual ACM symposium on Theory of computing , pages 285-295, 1987.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning , pages 1915-1924, 2017.

Erich L Lehmann and George Casella. Theory of point estimation . Springer Science & Business Media, 2006.

Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine learning , 2(4):285-318, 1988.

Jon Christian Mesterharm. Improving on-line learning . Rutgers The State University of New Jersey, School of Graduate Studies, 2007.

Arkadii Nemirovski, David Borisovich Yudin, and Edgar Ronald Dawson. Problem complexity and method efficiency in optimization. 1983.

Binghui Peng and Aviad Rubinstein. Near optimal memory-regret tradeoff for online learning. arXiv preprint arXiv:2303.01673 , 2023.

Binghui Peng and Fred Zhang. Online prediction in sub-linear space. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 1611-1634. SIAM, 2023.

Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. 2014.

Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. Advances in neural information processing systems , 28, 2015.

Maxim Raginsky and Alexander Rakhlin. Information-based complexity, feedback and dynamics in convex programming. IEEE Transactions on Information Theory , 57(10):7036-7056, 2011.

Alexander Rakhlin and Karthik Sridharan. Statistical learning and sequential prediction, 2012. Available at http://www.mit.edu/~rakhlin/courses/stat928/stat928_notes.pdf .

Alexander Rakhlin and Karthik Sridharan. Statistical learning theory and sequential prediction. Lecture Notes in University of Pennsyvania , 44, 2012.

Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression. In Conference on Learning Theory , 2014.

Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression with general loss functions. arXiv preprint arXiv:1501.06598 , 2015.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Random averages, combinatorial parameters, and learnability. Advances in Neural Information Processing Systems 23 , pages 1984-1992, 2010.

Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning via sequential complexities. Journal of Machine Learning Research , 2014.

Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5979 , 2014.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 2021.

Vaidehi Srinivas, David P Woodruff, Ziyu Xu, and Samson Zhou. Memory bounds for the experts problem. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1158-1171, 2022.

Joseph F Traub, Grzegorz W Wasilkowski, and Henryk Wo´ zniakowski. Information-based complexity. 1988.

Alexandre B Tsybakov. Optimal rates of aggregation. In Learning Theory and Kernel Machines , pages 303-313. Springer, 2003.

Alexandre B Tsybakov. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 2008.

Leslie G Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

S. A. van de Geer. Empirical Processes in M-Estimation. Cambridge University Press, 2000.

Vladimir Vovk. Competitive on-line linear regression. In NIPS '97: Proceedings of the 1997 conference on Advances in neural information processing systems 10 , pages 364-370, Cambridge, MA, USA, 1998. MIT Press.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Marcelo J Weinberger and Erik Ordentlich. On delayed prediction of individual sequences. IEEE Transactions on Information Theory , 48(7):1959-1976, 2002.

David P Woodruff, Fred Zhang, and Samson Zhou. Streaming algorithms for learning with experts: Deterministic versus robust. arXiv preprint arXiv:2303.01709 , 2023.

Tengyang Xie, Dylan J Foster, Yu Bai, Nan Jiang, and Sham M Kakade. The role of coverage in online reinforcement learning. International Conference on Learning Representations (ICLR) , 2023.

Yunbei Xu and Assaf Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876 , 2020.

Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Annals of Statistics , pages 1564-1599, 1999.

Contents of Appendix

Part I

Additional discussion and examples

A Additional notation

We denote R ≥ 0 = [0 , ∞ ) . For any a, b ∈ R , a ∧ b := min { a, b } and a ∨ b := max { a, b } . For any integer N > 0 , [ N ] = { 1 , . . . , N } . For any set X , ∆( X ) is the space of all distributions on X . For any integer T , the notation x 1: T will be the shorthand notation for the sequence x 1 , . . . , x T . For any real number x ∈ R , denote by ⌊ x ⌋ the largest integer that is smaller than or equal to x and by ⌈ x ⌉ the smallest integer that is greater than or equal to x . The indicator function is denoted by ✶ ( · ) . We define O ( · ) , Ω( · ) , o ( · ) , Θ( · ) , ˜ O ( · ) , ˜ Ω( · ) , ˜ Θ( · ) following standard non-asymptotic big-oh notation. We use the binary relation x ≲ y to indicate that x ≤ O ( y ) .

B Additional related work

In this section we discuss related work not already covered in detail.

Computational lower bounds for online learning. Beyond Blum [14], another work that considers computational lower bounds for online learning is Hazan and Koren [33]. This work proves lower bounds for online learning in a model where the learner has access to an ERM oracle that can minimize the training loss for an arbitrary dataset ( x 1 , y 1 ) , . . . , ( x T , y T ) . Their lower bound does not fit in our computational model due to details around the way description length is formalized. In particular, the main focus of [33] is to obtain a lower bound on the number of oracle calls any online learning algorithm must make to an ERM oracle.

Similar to the setup for Theorem 4.1, Hazan and Koren [33] consider a sequence of classification instances with X n = { 0 , 1 } n and classes F n of the size of Ω(2 √ |X n | ) , and show that any online learning algorithm requires Ω( √ |X n | ) oracle calls to achieve low regret for this class. However, the estimators f ∈ F n returned by the oracle in their construction have Ω( √ |X n | ) description length themselves, meaning that they do not satisfy the poly( n ) -description length required by the model described in Section 4.1 (in other words, the result is not meaningful as a lower bound on runtime, because simply reading in the output of the offline oracle takes exponential time). For completeness, we restate the example proposed by Hazan and Koren [33, Theorem 22] in our framework below.

Hard case from [33]. For any integer n ≥ 1 , consider a binary classification problem with X 2 n = { 0 , 1 } 2 n , Z = Y = { 0 , 1 } , D = D 0 / 1 and K ( z ) = ✶ z . Let N := 2 2 n , and let S be the collection of all sets S = { s 1 , . . . , s 2 n } ⊂ { 0 , 1 } 2 n where { 0 , 1 } 2 n is also treated as the integer set of { 0 , . . . , 2 2 n -1 } in left-to-right order and s i ∈ { 2 n ( i -1) , . . . , 2 n i -1 } for each i = 1 , . . . , 2 n . We define a class F 2 n = { f S,τ : S ∈ S , τ ≤ 2 n } , where

For this class, we reduce back to the Theorem 22 of Hazan and Koren [33] which states that any algorithm with runtime o ( √ N ) has to suffer online estimation error at least t/ 2 for all 1 ≤ t ≤ 2 n . The issue with this example for our computation model is that |F 2 n | = Ω(2 √ |X 2 n | ) . Any sufficient description for this parameter space in bit strings (or, e.g., boolean circuits) will scale with Ω( √ |X 2 n | ) . Thus, the description length required to return ̂ f t is too large (already larger than the lower bound obtained).

Online learning with memory constraints. A number of recent works focus on memory-regret tradeoffs in online learning [59, 45, 44, 1, 68]. Here, the learner can observe the full data stream ( x 1 , y 1 ) , . . . , ( x T , y T ) , but is constrained to B bits of memory. This framework is incomparable to the OEOE framework, but it would be interesting to explore whether there are deeper connections (e.g., any memoryless OEOE algorithm inherently has memory no larger than that of the offline oracle).

Gaps between offline and online. A long line of work aims to characterize the optimal regret for online learning, developing complexity measures (Littlestone dimension, sequential Rademacher complexity) that parallel classical complexity measures like VC dimension and Rademacher complexity for offline learning and estimation [10, 53, 54, 7]. It is well known that in general, the optimal rates for online learning can be significantly worse than those for offline learning. Our work primarily focuses on finite classes F , where there is no gap, but for infinite classes, any conversion from offline to online estimation will inevitably lead to a loss in the estimation error rate that scales with appropriate complexity measures for online learning (cf. Appendix D.1).

Other restricted computational models. Our information-theoretic formulation of oracleefficiency is inspired by statistical query complexity in theoretical computer science and information complexity in optimization, both of which can be viewed as restricted computational models with an information-theoretic flavor. The statistical query model is a framework in which the learner can only access the environment through an oracle that outputs noise estimates ('statistical queries') for a target parameter of interest [13, 37, 21-23]. Information complexity in optimization is a model in which algorithms can only access the parameter of interest through (potentially noisy) local queries to gradients or other information [43, 60, 48, 2, 5].

C Examples of estimation problems and loss functions

In what follows, we give detailed background on three canonical examples of the general estimation framework discussed in Section 1: Binary classification, square loss regression, and conditional density estimation.

̸

For offline estimation, observe that with covariates x 1 , . . . , x T and outcomes y t = f ⋆ ( x t ) for all t ∈ { 1 , . . . , T } , any empirical risk minimizer ̂ f that sets ̂ f ( x t ) = y t obtains

Classification [38, 63]. For binary classification, we take Z = Y = { 0 , 1 } with the binary loss D 0 / 1 ( z 1 , z 2 ) = ✶ ( z 1 = z 2 ) for z 1 , z 2 ∈ Z and kernel K ( z ) = ✶ z , which is noiseless. The binary loss is metric-like with C D = 1 .

For online estimation, the halving algorithm [17] achieves

We mention in passing that another natural classification setting we do not explore in detail in this paper is noisy classification , where the setting is as above, except that we set Z = [0 , 1] , K ( f ⋆ ( x )) = Ber( f ⋆ ( x )) , and take D abs ( z 1 , z 2 ) = | z 1 -z 2 | as the absolute loss for all z 1 , z 2 ∈ Z .

Square loss regression [62, 66]. For real-valued regression, we take Z = Y = R with the square loss D sq ( z 1 , z 2 ) = ( z 1 -z 2 ) 2 for z 1 , z 2 ∈ Z and the kernel K ( f ⋆ ( x )) = N ( f ⋆ ( x ) , 1) or another subGaussian distribution. Note that the square loss is a metric-like loss with C D = 2 .

For offline estimation, with covariates x 1 , . . . , x T and outcomes y t ∼ f ⋆ ( x t ) + ε t for all t ∈ { 1 , . . . , T } , the classical Empirical Risk Minimization (ERM) ̂ f := arg min f ∈F ∑ T t =1 ( f ( x t ) -y t ) 2 gives

with probability at least 1 -δ (cf. Lemma C.1below).

For online estimation, the exponential weights algorithm [17], with decision space F and the loss at each round chosen to be ℓ t ( f ) = ( f ( x t ) -y t ) 2 , achieves

with probability at least 1 -δ (cf. Foster and Rakhlin [25] for a proof).

Conditional density estimation [12]. For conditional density estimation, we consider an arbitrary outcome space Y and take Z = ∆( Y ) with squared Hellinger distance D 2 H given by 12

and K ( z ) = z for all z ∈ Z . Note that squared Hellinger distance is a metric-like loss with C D = 2 .

For offline estimation, with covariates x 1 , . . . , x T and outcomes y t ∼ f ⋆ ( x t ) for all t ∈ { 1 , . . . , T } , the classical Maximum Likelihood Estimator (MLE) ̂ f := arg max f ∈F ∑ T t =1 log f ( y t | x t ) gives

with probability at least 1 -δ (cf. Lemma C.2below).

For online estimation, the exponential weights algorithm [17], with decision space F and the loss at each round chosen to be ℓ t ( f ) = -log f ( y t | x t ) , achieves

with probability at least 1 -δ (cf. Foster et al. [27] for a proof).

C.1 Examples of offline oracles

For completeness, below we prove the offline estimation (fixed design) guarantees for square loss empirical risk minimization and maximum likelihood estimation mentioned in the prequel.

Empirical risk minimization for square loss regression. For any square loss regression instance ( X , Y , Z , K , F ) defined as in Appendix C. The Empirical Risk Minimizer ( ERM ) estimator ̂ f is defined as

We have the following bounds on the offline estimation error for the squared loss.

Lemma C.1. For any target parameter f ⋆ ∈ F , with probability at least 1 -δ , we have,

Proof of Lemma C.1. Observe that the ERM estimator satisfies

where the last inequality is by the definition of ̂ f , and ε t := y t -f ⋆ ( x t ) is a standard normal distribution for all t ∈ [ T ] . Then by the Gaussian tail bound, we have with probability at least 1 -δ ,

Plug the above inequality back into Eq. (12) and reorganize, we obtain the desired bound of

Maximum likelihood estimation for conditional density estimation. For any conditional density estimation instance ( X , Y , Z , K , F ) defined as in Appendix C. The Maximum Likelihood Estimator ( MLE ) ̂ f is defined as

We have the following bounds on the offline estimation error for squared Hellinger distance.

Lemma C.2. For any target parameter f ⋆ ∈ F , with probability at least 1 -δ , we have,

Proof of Lemma C.2. For any parameter f ∈ F , define

Then for any f ∈ F , by Lemma A.4 of Foster et al. [27], with probability at least 1 -δ/ |F| , we have,

We further have by the inequality of log(1 + x ) ≤ x that

Since y t ∼ f ⋆ ( x t ) , we have by standard calculus and the definition of the squared Hellinger distance that

Altogether, we have obtained for any f ∈ F , with probability at least 1 -δ/ |F|

Thus, by union bound, the above inequality holds for all f ∈ F with probability at least 1 -δ . Thus the MLE ̂ f satisfies

where the second inequality is by the defintion of MLE.

Algorithm 2 Reduction from OEOE to Online Learning with Delayed Feedback

1: input: Offline estimation oracle Alg Off with parameter β Off ≥ 0 , delay parameter N ∈ N , delayed online learning algorithm A DOL for class F .

2: for t = 1 , 2 , . . . , T do

3: Receive ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) .

4: if t > N then

5: Let ˜ f t -N := 1 N t ∑ i = t -N +1 ̂ f i .

6: Let ℓ t -N ( f ) := D ( ˜ f t -N ( x t -N ) , f ( x t -N ) ) and pass ℓ t -N ( · ) to A DOL as the delayed feedback.

7: Let µ t = A t DOL ( ℓ 1 , .., ℓ t -N ) be the delayed online learner's prediction distribution.

8: Predict with ¯ f t ∼ µ t and receive x t .

Part II

Omitted results

D General reductions for oracle-efficient online estimation

The oracle-efficient online estimation algorithm in Section 3.1, Algorithm 1, is somewhat specialized to finite classes. In this section, we provide a more general approach to designing oracle-efficient algorithms based on delayed online learning , and use it to derive a characterization of oracle-efficient learnability for classification with infinite classes F .

For the results in this section, we assume that Z and D are convex, which covers regression and conditional density estimation; variants of our result for classification are given in Appendix G.3.

Online learning with delayed feedback. Before introducing our algorithm, we first introduce an abstract delayed online learning framework [67, 42, 35, 47]. In our framework, the learner is given a class F ⊆ Z X . Their goal is to choose a sequence of parameters ¯ f 1 , . . . , ¯ f T that minimizes regret against the class F for an adversarially chosen sequence of loss functions ℓ 1 , . . . , ℓ T , with the twist being that the loss ℓ t is not revealed immediately at step t , and instead becomes available at step t + N for a delay parameter N ∈ N .

In more detail, the interaction between the learner and the environment proceeds as follows:

For t = 1 , . . . , T :

The learner picks ¯ f t ∼ µ t ∈ ∆( Z X ) .

Learner incurs loss ℓ t ( ¯ f t ) and adversary reveals loss function ℓ t -N : Z X → [0 , 1] for round t -N (if t < N +1 , nothing is revealed).

The goal of the learner is to minimize regret in the sense that

is small, where γ ≥ 1 is a parameter. For γ = 1 , this definition coincides with the standard notion of regret in online learning (e.g., Cesa-Bianchi and Lugosi [17]), but allowing for γ > 1 will prove useful for our technical results.

Algorithm and online estimation error bound. Algorithm 2 describes our reduction from oracleefficient online estimation to delayed online learning. In addition to an offline oracle Alg Off , the algorithm takes as input a delay parameter N ∈ N and a delayed online learning algorithm A DOL for the class F (Algorithm 2 does not explicitly take the class F as an argument, as the algorithm only implicitly makes use of F through A DOL ).

The basic premise behind Algorithm 2 is that for any sequence of consistent offline estimators, we can average to improve the predictions. Consider a step t ∈ [ T ] . Suppose ̂ f 1 , . . . , ̂ f T are produced by an

offline oracle Alg Off with parameter β Off ≥ 0 , where we augment the sequence by setting ̂ f T + s = ̂ f T for all s ∈ N . Then we can use an argument based on convexity (cf. proof of Theorem D.1) to show that for any N ∈ N , the averaged parameters

satisfies

In particular, as we increase N , the quality of the predictions increases, and we achieve sublinear estimation error as soon as N = ω ( T ) . Of course, the catch here is that ˜ f t depends on the predictions of future estimators, and cannot be computed at step t . However, ˜ f t can be computed at step t + N +1 , with a delay of N . This leads us to appeal to delayed online learning. In particular, at each step t ≥ N +1 , Algorithm 2 proceeds as follows. Using the new offline estimator ̂ f t from Alg Off , the algorithm computes the averaged estimator ˜ f t -N := 1 N t ∑ i = t -N +1 ̂ f i corresponding to the estimator in

Eq. (13) for step t -N . The algorithm then defines a loss function

and feeds it into the delayed online learning algorithm A DOL as the feedback for step t -N . Finally, Algorithm 2 uses the prediction distribution µ t produced by A DOL to sample the final estimator ¯ f t , then proceeds to the next step. Our main theorem shows that as long A DOL achieves low regret for delayed online learning, this strategy leads to low online estimation error.

Theorem D.1 (Reduction from oracle-efficient online estimation to delayed online learning) . Let D be any convex, metric-like loss. Suppose we run Algorithm 2 with delay parameter N ∈ N and a delayed online learning algorithm A DOL for the class F . Then for all γ ≥ 1 , Algorithm 2 ensures that

with any offline oracle Alg Off with parameter β Off ≥ 0 , where

is the regret of A DOL for the sequence of losses constructed in Algorithm 2.

The parameter N controls a sort of bias-variance tradeoff in Theorem D.1. The first term in Eq. (14) (corresponding to the bias of the averaged estimators) decreases with the delay N , while the second term (corresponding to the regret of A DOL ) increases; the optimal choice for N will balance these terms. To make this concrete, we revisit finite classes as a warmup.

Example: Finite classes. Delayed online learning is well-studied, and optimal algorithms are known for many classes of interest [67, 42, 35, 47]. The following standard result (a proof is given in Appendix G.3 for completeness) gives a delayed regret bound for arbitrary finite classes.

Lemma D.1. Consider the delayed online learning setting with a delay parameter N . There exists an algorithm that achieves

for any sequences of losses ℓ 1 , . . . , ℓ T ∈ [0 , 1] .

Combining Theorem D.1 with Lemma D.1, we can obtain the following upper bound for oracleefficient online estimation.

Corollary D.1 (Oracle-efficient online estimation for finite classes via delayed online learning) . Consider an arbitrary instance ( X , Y , Z , K , F ) and metric-like loss D , and assume Z is convex. By choosing A DOL as in Lemma D.1, Algorithm 2 ensures that for any N ≥ 1 ,

with any offline oracle Alg Off with parameter β Off .

By choosing N = √ C D β Off · T C D +log |F| ∨ 1 , Corollary D.1 gives an upper bound of O ( √ C D β Off ( C D +log |F| ) · T +log |F| ) . While the rate in Corollary D.1, is worse than Theorem 3.1 in terms of dependence on T , the reduction has two advantages: (1) It does require a-priori knowledge of the offline estimation parameter β Off : If we choose N = √ C D · T C D +log |F| ∨ 1 , Corollary D.1 obtains O (( β Off + 1)( √ C D ( C D +log |F| ) · T + C D + log |F| )) ; (2) The reduction by Algorithm 2 is more flexible, and allows for guarantees beyond finite classes, as we now illustrate.

D.1 Characterization of oracle-efficient learnability for classification

As an application of Algorithm 2 and Theorem D.1, we give a characterization for oracle-efficient learnability in the OEOE framework. To state the result, we define Ldim( F ) as the Littlestone dimension for a binary function class F (e.g., Ben-David et al. [10]).

Theorem D.2 (Characterization of oracle-efficient learnability for binary classification) . Consider a binary classification instance ( X , Y , Z , K , F ) with Z = Y = { 0 , 1 } , D = D 0 / 1 and K ( z ) = ✶ z . For any class F and β Off ≥ 0 , there exists an oracle-efficient algorithm that achieves online estimation error O ( √ β Off Ldim( F ) · T log T + Ldim( F ) log T ) . On the other hand, in the worst-case any algorithm must suffer at least Ω(Ldim( F )) online estimation error.

The main idea behind this result is to show that we can create a delayed online learner for the reduction in Algorithm 2 that achieves low regret for Littlestone classes. 13

D.2 Additional lower and upper bounds for memoryless oracle-efficient algorithms

The proof of Theorem 3.3 uses that the estimators ̂ f t produced by the offline estimation oracle may be improper (i.e., ̂ f t / ∈ F ). We next provide a variant of the result that holds even if the estimation oracle is proper , under the additional assumptions that (i) the learner is itself proper in the sense that µ t ∈ ∆( F ) , and (ii) the learner is time-invariant (i.e., the learner sets µ t = F ( ̂ f t ) for all t ).

Theorem 3.3 ′ (Impossibility of memoryless algorithms for OEOE; proper variant) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off ≥ 0 , there exists an instance ( X , Y , Z , K , F ) with |F| = |X| = N such that for any memoryless oracleefficient algorithm that is (i) proper , and (ii) time-invariant , there exists a sequence of covariates ( x 1 , . . . , x T ) and a proper offline oracle Alg Off with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { N ( β Off +1) , T } ) .

A complementary upper bound. For completeness, we conclude by showing that the (large) lower bound in Theorem 3.3 can be achieved with a memoryless oracle-efficient algorithm. We consider the 'trivial' algorithm that outputs the estimators produced by the offline oracle as-is.

Proposition D.1 (Upper bound for memoryless OEOE) . For any instance ( X , Y , Z , K , F ) , metric-like loss D , and offline oracle Alg Off with parameter β Off , the algorithm that returns ¯ f t = ̂ f t := Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) has online estimation error Est On D ( T ) ≤ O (( β Off +1) |X| log T ) .

E Application to interactive decision making

In this section, we apply our techniques for oracle-efficient online estimation to the Decision Making with Structured Observations (DMSO) framework for interactive decision making introduced by [27]. First, in Appendix E.1, we use our reductions to provide offline oracle-efficient algorithms for interactive decision making. Then, in Appendix E.2, we focus on reinforcement learning and show that it is possible to bypass the impossibility results for memoryless oracle-efficient algorithms (Theorem 3.3) for instances ( X , Y , Z , K , F ) corresponding to Markov decision processes that satisfy a structural property known as coverability.

E.1 Offline oracle-efficient algorithms for interactive decision making

In this section, we introduce the setting of Decision Making with Structured Observations (DMSO) and the applications of our results to this setting.

Decision Making with Structured Observations (DMSO). The DMSO framework [27] captures a large class of interactive decision making problems (e.g. contextual bandits and reinforcement learning). In this framework, the learner is given access to a model class M that contains an unknown true model M ⋆ : Π → ∆( R×O ) , where Π is the decision space , R⊆ R is the reward space and O is the observation space . Then the interaction between the learner and the environment proceeds in T rounds, where for each round t = 1 , . . . , T :

The learner selects a decision π t ∈ Π .

Nature selects a reward r t ∈ R and observation o t ∈ O based on the decision, where the pair ( r t , o t ) is drawn independently from the unknown distribution M ⋆ ( π t ) . The reward and observation is then observed by the learner.

Let g M ( π ) := E M ,π [ r ] denote the mean reward function and π M := arg max π ∈ Π g M ( π ) denote the decision with the greatest expected reward for M . The learner's performance is evaluated in terms of regret to the optimal decision for M ⋆ :

where p t ∈ ∆(Π) is the learner's distribution over decisions at round t .

Background: Reducing DMSO to online estimation. Any DMSO class ( M , Π , O ) induces an instance ( X , Y , Z , K , F ) of the estimation framework in Section 1 as follows. We associate F = M , X = Π , Y = O×R , Z = ∆( O×R ) , and K ( M ⋆ ( π )) = M ⋆ ( π ) . That is, we have a conditional density estimation problem in which the covariates are decisions π ∈ Π and the outcomes are observation-reward pairs drawn from the underlying model M ⋆ ( π ) . In particular for a sequence of decisions π 1 , . . . , π T and a sequence of estimators ̂ M 1 , . . . , ̂ M T , we define the online estimation error for a loss D as

We refer to any algorithm Alg On that ensures that Est On D ( T ) ≤ β On almost surely given access to { ( π t , o t , r t ) } T t =1 with ( o t , r t ) ∼ M ⋆ ( π t ) as an online estimation oracle with parameter β On .

Foster et al. [27, 28] give an algorithm, Estimation-to-Decisions ( E2D ), that provides bounds on the regret in Eq. (16) given access to an online estimation oracle Alg On . The algorithm is (online) oracleefficient and memoryless, in the sense that the decision π t at each step t is a measurable function of the oracle's output ̂ M t . To restate their result, we define the Decision-Estimation Coeficient for the class M as

for a reference model M and any losses D . We further define dec D γ ( M ) = sup M ∈M dec D γ ( M , M ) . With this notation, the main regret bound for E2D is as follows.

Algorithm 3 Estimation to Decisions Meta-Algorithm with Offline Oracles ( E2D.Off )

1: parameters :

Offline estimation oracle Alg Off with parameter β Off .

Oracle-efficient online estimation algorithm Alg OEOE .

Exploration parameter γ > 0 .

2: for t = 1 , 2 , · · · , T do

4: Feed ̂ M t to OEOE algorithm Alg OEOE and obtain µ t .

5: Let

6: Sample decision π t ∼ p t and o t ∼ M ⋆ ( π t ) and feed π t to OEOE algorithm Alg OEOE .

Proposition E.1 (Theorem 4.3 of [27]) . For any model class ( M , Π , O ) and metric-like loss D , any γ > 0 , and any online estimation oracle Alg On with parameter β On > 0 , the E2D algorithm ensures that

For the case of squared Hellinger distance where D = D 2 H , the Decision-Estimation Coefficient dec D γ ( M ) was shown to be a lower bound on the minimax optimal regret for any class M . Hence, Proposition E.1 shows that it is possible to achieve near-optimal regret for any interactive decision making problems whenever an online estimation oracle is available. However, it was unclear whether similar results could be achieved based on offline estimation oracles.

Making E2D offline oracle-efficient. Algorithm 3 ( E2D.Off ) invokes the E2D algorithm of Foster et al. [27] with any oracle-efficient online estimation algorithm Alg OEOE (which can be any of the algorithms we provide, e.g. Theorems 3.1 and 4.2), along with an offline estimation oracle Alg Off , to provide offline oracle-efficient guarantees for interactive decision making. Invoking the algorithm with Version Space Averaging (via Theorem 3.1) leads to the following corollary.

Corollary E.1. Consider any DMSO class ( M , Π , O ) and metric-like loss D . Algorithm 3, with exploration parameter γ > 0 and Alg OEOE chosen to be Algorithm 1, ensures that

for any offline estimation oracle Alg Off with parameter β Off .

This result shows that information-theoretically, it is possible to achieve low regret in the DMSO framework with offline oracles, though the result is not computationally efficient.

As an example, in the case of square loss regression (Appendix C) which is used for contextual bandits, an offline guarantee of β Off = O (log |M| ) is achievable. Meanwhile, it is known that dec Dsq γ ( M ) ≲ |A| /γ [27]. Thus Corollary E.1 achieves a bound of ˜ O ( √ |A| T · log |M| ) with an appropriate choice of γ . The best know regret guarantee for contextual bandit is ˜ O ( √ |A| T log |M| ) [58, 70]. The bound from Corollary E.1 matches the state-of-the-art result up to a factor ˜ O ( √ log |M| ) . How to remove this suboptimality is an interesting direction for future work.

Naturally, the other reductions for oracle-efficient online estimation developed in this paper can be combined with Algorithm 3 as well. In particular, by combining with Theorem 4.2 we derive the following corollary for squared Hellinger distance.

Corollary (Informal). Whenever online conditional density estimation can be performed efficiently with access to the full history, and whenever the minimax problem in Eq. (18) can be solved efficiently, there exists a computationally efficient and offline oracle-efficient algorithm with near-optimal regret in the DMSO framework.

E.2 Bypassing impossibility of memoryless algorithms via coverability

Recall that our results in Section 3.2 show that in general, it is impossible to obtain low online estimation error through memoryless oracle-efficient algorithms. In this section, we revisit memoryless algorithms for the Markov decision processes a particular type of class ( M , Π , O ) (or equivalently, ( X , Y , Z , K , F ) ). We prove that for any class of Markov decision processes for which a structural parameter called coverability [69] is small, any offline estimator can be directly converted into an online estimator.

Markov decision processes. We consider classes ( M , Π , O ) that correspond to an episodic finite-horizon reinforcement learning setting, following Foster et al. [27]. With H ∈ N denoting the horizon, each model M ∈ M specifies a non-stationary Markov decision process as a tuple M = { {S h } H h =1 , A , { P M h } H h =1 , { R M h } H h =1 , d 1 } , where S h is the state space for layer h , A is the action space, P M h : S h × A → ∆( S h +1 ) is the probability transition kernel for layer h , R M h : S h ×A → ∆( R ) is the reward distribution for layer h , and d 1 ∈ ∆( S 1 ) is the initial state distribution. We allow the reward distribution and transition kernel to vary across models in M and assume that the initial state distribution is fixed.

We set Π ⊂ Π RNS , which denotes the set of all randomized, non-stationary policies π = ( π 1 , . . . , π H ) ∈ Π RNS , where π h : S h → ∆( A ) . For a fixed MDP M ∈ M and π ∈ Π , the observation o ∼ M ( π ) is a trajectory ( s 1 , a 1 , r 1 ) , . . . , ( s H , a H , r H ) that is generated through the following process, beginning from s 1 ∼ d 1 . For h = 1 , . . . , H :

a h ∼ π h ( s h ) .

r h ∼ R M h ( s h , a h ) and s h +1 ∼ P M h ( · | s h , a h ) .

So the obseravtion space O = S 1 ×A× R ×··· × S H ×A× R . For notational convenience, we take s H +1 to be a deterministic terminal state. We use P M ,π and E M ,π [ · ] to denote the probability law and expectation over trajectories induced by M ( π ) . In addition, we define P M h ( · | s h , a h ) as the conditional distribution on s h +1 , r h given s h , a h under M for h ∈ [ H ] .

The guarantees we provide apply to any loss that has a particular layer-wise structure tailored to reinforcement learning.

Definition E.1 (Layer-wise loss) . For any sequence of losses { D h } h ∈ [ H ] bounded by [0 , 1] , where D h : ∆( S h × R ) × ∆( S h × R ) → [0 , 1] for all h ∈ [ H ] , we define the layer-wise loss D RL on ∆( O ) as 14

for any pair of MDPs M,M ′ ∈ M and policy π ∈ Π . 15

Examples of the layer-wise loss are scaled reverse KL-divergence (which is bounded by [0 , 1] whenever the density ratios under consideration are upper and lower bounded with an appropriate scaling) [27] and the squared Bellman error [29]. Another useful example is the the sum of layer-wise squared Hellinger distances given by

This loss coincides with the global squared Hellinger distance D 2 H ( M ( π ) , M ′ ( π )) up to an O ( H ) factor.

Coverability. We provide memoryless oracle-efficient algorithms for online estimation for any layer-wise loss D RL when the underlying MDP M ⋆ has bounded coverability [69].

Definition E.2 (Coverability) . For an MDP M ⋆ and a policy π , we define d π h ( s, a ) ≡ E M⋆ ,π [ ✶ ( s h , a h = s, a )] . The coverability coefficient C cov for a policy class Π for the MDP M ⋆ is given by

It is immediate to see that C cov ≤ | Π | , but in general it can be much smaller. Examples of MDP classes with low coverability include Block MDPs, Low-Rank MDPs, and exogenous block MDPs [69, 4].

Offline-to-online conversion under coverability. Our main result shows that under coverability, the outputs of any offline estimation oracle Alg Off satisfy an online estimation guarantee as-is .

Theorem E.1 (Offline-to-online conversion under coverability) . For any layer-wise loss D RL and MDP class ( M , Π , O ) and M ⋆ ∈ M , the sequence of estimators ( ̂ M 1 , . . . , ̂ M T ) produced by any offline estimation oracle Alg Off for D RL with parameter β Off satisfy

This result is based on a variant of the proof technique in Theorem 1 of Xie et al. [69]. An important application of the result, which can be applied in tandem with the guarantees in Foster et al. [27], concerns squared Hellinger distance.

Corollary E.2. For any MDP class ( M , Π , O ) and M ⋆ ∈ M , the sequence of estimators ( ̂ M 1 , . . . , ̂ M T ) produced by any offline estimation oracle Alg Off for squared Hellinger distance D 2 H with parameter β Off satisfy

This result follows by using that the layer-wise squared Hellinger distance in Eq. (21) is equivalent to D 2 H ( M ( π ) , M ′ ( π )) up to O ( H ) factors.

Application to interactive decision making. We apply Theorem E.1 to decision making via Algorithm 3.

Corollary E.3. Consider any layer-wise loss D = D RL and MDP class ( M , Π , O ) , and let C cov := sup M ∈M C cov ( M ) . Algorithm 3 with exploration parameter γ > 0 and Alg OEOE chosen to be the identity map ensures that

for any offline estimation oracle Alg Off with parameter β Off .

Contextual bandits and optimality of offline-to-online conversion. Another implication for Theorem E.1 concerns the special case of contextual bandits (that is, MDPs with horizon one). For the contextual bandit setting we abbreviate S = S 1 , and refer to d 1 ∈ ∆( S ) as the context distribution . We define g M ( s, a ) = E r ∼ R M 1 ( s,a ) [ r ] as the expected reward function under a model M , and following Foster and Rakhlin [25], use the squared error between mean reward functions as our divergence:

For this setting, the coverability coefficient C cov is always bounded by the number of actions |A| , which leads to the following corollary.

Corollary E.4. For any contextual bandit class ( M , Π , O ) and M ⋆ ∈ M , the sequence of estimators ( ̂ M 1 , . . . , ̂ M T ) produced by any offline estimation oracle Alg Off for D CB with parameter β Off satisfy

Recall that Foster and Rakhlin [25] show that any algorithm for online estimation with the divergence in Eq. (22) with Est On D ( T ) ≤ β On can be lifted to a contextual bandit algorithm with regret O (√ |A| T · β On ) via the inverse gap weighting strategy , even if contexts are chosen adversarially. Subsequent work of Simchi-Levi and Xu [58] shows that for stochastic contexts, the inverse gap weighting strategy also yields regret O (√ |A| T · β Off ) given access to an offline oracle with parameter β Off . On the other hand, combining Corollary E.4 with the guarantee from Foster and Rakhlin [25] gives regret O ( |A| 1 / 4 T 3 / 4 β Off 1 / 4 ) . This does not recover the result from Simchi-Levi and Xu [58], but nonetheless gives an alternative proof that sublinear offline estimation error suffices for sublinear regret.

The guarantee in Theorem E.1 leads to a degradation in rate from β Off to √ Tβ Off (suppressing problem-dependent parameters). Our next result shows that this is tight in general.

Proposition E.2 (Tightness of offline-to-online conversion) . For any integer T ≥ 1 and β Off > 0 , there exists a contextual bandit class ( M , Π = A S , O ) with |A| = 2 , a distribution d 1 ∈ ∆( S ) , a sequence ( π 1 , . . . , π T ) and an offline oracle Alg Off for D CB with parameter β Off such that the oracle's outputs ( ̂ M 1 , . . . , ̂ M T ) satisfy

Part III Proofs

F Technical tools

Lemma F.1. For any non-increasing sequence x 1 ≥ x 2 ≥ · · · ≥ x T +1 ≥ 1 ,

Proof of Lemma F.1. Since log(1 + a ) ≤ a for all a > -1 , for any t ∈ [ T ] , we have

Summing up over t ∈ [ T ] , we obtain

The following lemma gives an improvement to Lemma A.13 of Foster et al. [27] that removes a logarithmic factor. This shows that up to an absolute constant, squared Hellinger distance obeys a one-sided version of the chain rule for KL divergence.

Lemma F.2 (Subadditivity for squared Hellinger distance) . Let ( X 1 , F 1 ) , . . . , ( X n , F n ) be a sequence of measurable spaces, and let X i = ∏ i i = t X t and F i = ⊗ i t =1 F t . For each i , let P i ( · | · ) and Q i ( · | · ) be probability kernels from ( X i -1 , F i -1 ) to ( X i , F i ) . Let P and Q be the laws of X 1 , . . . , X n under X i ∼ P i ( · | X 1: i -1 ) and X i ∼ Q i ( · | X 1: i -1 ) respectively. Then it holds that

Proof of Lemma F.2. We appeal to the cut-and-paste property of [34], defining a collection of distributions indexed by a hypercube { 0 , 1 } n with the property that the vertices (0 , . . . , 0) and (1 , . . . , 1) correspond to the distribution P and Q . Concretely, for any vertex v ∈ { 0 , 1 } n of the hypercube, we define a probability distribution

Observe that P (0 ,..., 0) = P and P (1 ,..., 1) = Q . Now, consider any four vertices a, b, c, d ∈ { 0 , 1 } n with the property that { a i , b i } = { c i , d i } for each i ∈ [ n ] (with {·} interpreted as a multi-set). Then for any measure ν := ∏ n i =1 ν i ( ·| X 1: i -1 ) , where ν i ( ·| X 1: i -1 ) is any common dominating conditional measure 16 for P i ( ·| X 1: i -1 ) and Q i ( ·| X 1: i -1 ) , by the definition of squared Hellinger distance we have

Let k be the maximum integer such that 2 k ≤ n . Then since Eq. (23) holds, by Theorem 7 of Jayram [34] applied with the pairwise disjoint collection A j = { i ≤ n | i mod 2 k = j } for all j ∈ [2 k ] where | A j | ≤ 2 for all j , we have

To conclude, we note that ∏ n i =1 (1 -1 / 2 i ) > 2 / 7 .

G Proofs from Section 3

G.1 Proofs from Section 3.1

Theorem 3.1 (Main upper bound for OEOE) . For any instance ( X , Y , Z , K , F ) , any metric-like loss D , and any offline estimator Alg Off with parameter β Off ≥ 0 , Algorithm 1 is oracle-efficient and achieves

Proof of Theorem 3.1. Our main technical result is the following lemma, which is proven in the sequel.

Lemma G.1. Consider any instance ( X , Y , Z , K , F ) and a metric-like loss 17 D on Z . Let f ⋆ ∈ F be the target parameter, and consider a sequence of sets F = F 1 ⊇ F 2 ⊇ · · · ⊇ F T ⊇ { f ⋆ } and sequence of covariates x 1 , . . . , x T ∈ X with the property that for all t ∈ [ T ] , all f ∈ F t satisfy the following offline estimation guarantee:

Then, by defining µ t = Unif( F t ) , we have that

To invoke Lemma G.1, we observe that the version space construction in Algorithm 1 ensures that for all t ∈ [ T ] , all f ∈ F t satisfy

In addition, it is immediate to see that F = F 1 ⊇ F 2 ⊇ · · · ⊇ F T . Thus, by invoking Lemma G.1 with parameter β Off ′ = 2 C D β Off , we have that

To simplify, we note that ( C D β Off +1) ≤ C D ( β Off +1) , since C D ≥ 1 .

Proof of Lemma G.1. We begin by proving that Est On D ( T ) ≤ O ( C D · ( β Off +1) · log |F| ) . Let us adopt the convention that F T +1 = { f ⋆ } , so that F T +1 ⊆ F T ⊆ · · · ⊆ F 1 . For any parameter f ∈ F \ { f ⋆ } , define t f := min { t : f / ∈ F t +1 } . It is immediate to see that for all f ∈ F \ { f ⋆ } , 1 ≤ t f ≤ T and for all t ∈ [ T ] , |{ f : t f = t }| = |F t \ F t +1 | . Using that D ( f ⋆ ( x ) , f ⋆ ( x )) = 0 for all x ∈ X , we re-write the online estimation error as

̸

Using that F T ⊆ F T -1 ⊆ · · · ⊆ F 1 , we can upper bound this quantity by

To proceed, observe that for any function f ∈ F , if t f = t , then f ∈ F t . It follows from the assumed bound in Eq. (24) that if t f = t , then

where we have used the fact that the loss D is bounded by 1 . Using this fact, and recalling that for all t ∈ [ T ] , |{ f : t f = t }| = |F t \ F t +1 | , we bound

Finally, by Lemma F.1, we have that

We conclude that

We now prove the bound Est On ( T ) ≤ O ( C D · ( β Off +1) · |X | log T ) . For each x ∈ X

D , define N t -1 ( x ) = t -1 ∑ s =1 ✶ ( x s = x ) . Then we can write the online estimation error as

From the definition of µ t , we have that

Now, from the offline guarantee assumed in Eq. (24), we have for all x ∈ X and f ∈ F t ,

Combining these observations, we have

Now, for each x ∈ X , define t x = min { t ≤ T : x t = x } if this set is not empty, and set t x = T otherwise. From this definition and the fact that 1 + 1 / 2 + · · · +1 /T ≤ 1 + log T , we have that

We conclude that

Theorem 3.2 (Main lower bound for OEOE) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off > 0 , there exists an instance ( X , Y , Z , K , F ) with log |F| = |X| = N such that for any oracle-efficient algorithm, there is a sequence of covariates ( x 1 , . . . , x T ) and offline oracle with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { ( β Off +1) N,T } ) .

̸

✶ We first specify the offline estimation oracle, then specify an adversarially chosen covariate sequence. Fix T ∈ N , and for any 1 ≤ t ≤ T and sequence of covariates x 1 , . . . , x t define N t ( x ) := ∑ t s =1 ✶ ( x s = x ) . For any target parameter f ⋆ ∈ F and offline estimation parameter β Off > 0 , we consider the oracle Alg t Off ( · ; f ⋆ ) for the sequence x 1 , . . . , x t that returns

Proof of Theorem 3.2. Let N ≥ 1 be given, a consider the model class where X = { x 1 , . . . , x N } is an arbitrary discrete set, Z = Y = { 0 , 1 } , F = { 0 , 1 } X , D ( z 1 , z 2 ) = D 0 / 1 ( z 1 , z 2 ) = ✶ { z 1 = z 2 } , and K ( z ) = z .

To complete the construction, we consider sequence ( x 1 , . . . , x T ) in which

Equivalently, and perhaps more intuitively, we set

stopping earlier if T ≤ N ⌈ β Off ⌉ .

For any f ⋆ , we now show that Alg t Off ( · ; f ⋆ ) is an offline oracle with parameter β Off on the sequence x 1 , . . . , x T . This is because for any i < min {⌈ t/ ⌈ β Off ⌉⌉ , N } , the covariate x i is repeated ⌈ β Off ⌉ ≥ β Off times. Thus ̂ f t ( x i ) = f ⋆ ( x i ) . This implies for t ≤ N · ⌈ β Off ⌉ that

If t > N · ⌈ β Off ⌉ , then all the covariates are repeated for more than ⌈ β Off ⌉ times, thus ̂ f t = f ⋆ . Overall, we have shown that Alg t Off ( · ; f ⋆ ) is an offline oracle with parameter β Off .

Now, fix any oracle-efficient online estimation algorithm, and consider the expected regret under f ⋆ ∼ Unif( F ) (with Alg t Off ( · ; f ⋆ ) as the oracle). If T ≥ N ⌈ β Off ⌉ , then regardless of how the algorithm chooses µ t , since for any block of x i , f ⋆ ( x i ) is independent of µ t ( x i ) , the expected regret is lower bounded as

G.2 Proofs from Section 3.2

Theorem 3.3 (Impossibility of memoryless algorithms for OEOE) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off ≥ 0 , there exists an instance ( X , Y , Z , K , F ) with |F| = |X| = N such that for any memoryless oracle-efficient algorithm, there exists a sequence of covariates ( x 1 , . . . , x T ) and a (potentially improper) offline oracle Alg Off with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { N ( β Off +1) , T } ) . This conclusion still holds when the online estimation algorithm remembers ̂ f 1 , . . . , ̂ f t -1 but not x 1 , . . . , x t -1 .

Proof of Theorem 3.3. Given a parameter β Off > 0 and an integer N , assume without loss of generality that K := T/ ( ⌊ β Off ⌋ + 1) is an integer. Consider the instance ( X , Y , Z , K , F ) with X = [ N ] , Z = Y = { 0 , 1 } , D = D 0 / 1 , K ( z ) = ✶ z , and parameter space F = { f i } i ∈ [ N ] is defined as f i ( x ) = ✶ { x = i } . We consider a sequence of covariates ( x 1 , . . . , x T ) divided into K blocks, each with length ⌊ β Off ⌋ +1 . In each block, the covariates will be chosen to be the same, i.e., x 1 = · · · = x ⌊ β Off ⌋ +1 , x ⌊ β Off ⌋ +2 = · · · = x 2 ⌊ β Off ⌋ +2 . We define τ t = ⌈ t/ ( ⌊ β Off ⌋ +1) ⌉ as the index of the block the step t belongs to, and we adopt the convention that x τ ∈ X is value of the covariates for block τ , i.e., x t = x τ t for all t . We leave the precise choice for x 1 , . . . , x K as a free parameter for now.

Fix any memoryless oracle-efficient online estimation algorithm defined by a sequence of maps { F t } t ∈ [ T ] (cf. Definition 3.1). We set the true target parameter to be f ⋆ = f i ⋆ , where the index i ⋆ ∈ [ N ] will be chosen later in the proof (in an adversarial fashion based on the algorithm under consideration); for now, we leave i ⋆ ∈ [ N ] as a free parameter.

We first specify the offline estimation oracle under f i ⋆ . For each block index τ = 1 , . . . , K , define

as the set of covariates in X\{ i ⋆ } that have not been observed before block τ , and let X τ := X τ ∪{ i ⋆ } . We define

̂ f t ( x ) = ✶ { x ∈ X τ t } as the estimator returned by the oracle at round t . It is immediate from this construction that regardless of how x 1 , . . . , x T are chosen, the offline estimation error is bounded by

since the value of ̂ f t differs from f i ⋆ only for covariates that have not been observed before block τ t .

It remains to lower bound the algorithm's online estimation error. To start, note that D 0 / 1 coincides with D sq on the set { 0 , 1 } and D sq is convex. Hence, by Jensen's inequality, it suffices to lower bound

where f t ( x t ) is the mean under ¯ f t ( x t ) where ¯ f t ∼ µ t = F t ( ̂ f 1 , . . . , ̂ f t -1 , ̂ f t ) .

̸

To proceed, we specify the sequence x 1 , . . . , x T , choosing x t as a measurable function of X t and i ⋆ (recall that i ⋆ itself has yet to be chosen). Fix a round t , and suppose that X t = ∅ . We choose x t to lower bound the estimation error by considering two cases. In the process, we will also define a function ̂ j τ : (2 [ N ] ) ⊗ τ → [ N ] ∪ {⊥} for each τ ∈ [ K ] , where (2 [ N ] ) ⊗ τ = 2 [ N ] ×··· × 2 [ N ] ︸ ︷︷ ︸ τ copies . Let

τ ∈ [ K ] be fixed.

In this case, we define ̂ j τ ( X 1 , . . . , X τ ) = ⊥ .

If there exists j ∈ X τ such that ⌊ β Off ⌋ +1 ∑ p =1 ✶ ( f τ ( ⌊ β Off ⌋ +1)+ p ( j ) ≤ 1 / 2) < ⌊ β Off ⌋ +1 2 , we set x τ = j (equivalently, x τ ( ⌊ β Off ⌋ +1)+ p = j for p = 1 , . . . , ⌊ β Off ⌋ +1 ) for the least such j , so that

̸

In this case, we define ̂ j τ ( X 1 , . . . , X τ ) = j as well.

Note that since f t is a measurable function of X 1 , ̂ j 1 ,. . . , X τ t , ̂ j τ t is well-defined.

Combining these cases, it follows that for any choice of i ⋆ , choosing x 1 , . . . , x K in the fashion described above ensures that

̸

We now state and prove the following technical lemma, which asserts that there exists a choice of i ⋆ for which the right-hand side above is large.

Lemma G.2. For any algorithm, there exists a choice for i ⋆ ∈ [ N ] such that

Proof of Lemma G.2. Consider a more abstract process, which we claim captures the evolution of X τ . Let i ⋆ ∈ [ N ] , and let A 1 = [ N ] . We consider a sequence of sets { A τ } τ ≥ 1 evolving according to the following process, parameterized by a sequence of functions { g τ : (2 [ N ] ) ⊗ τ → [ N ] ∪ {⊥} } τ ≥ 1 and index i ⋆ ∈ [ N ] .

For τ ≥ 1 :

If g τ ( A 1 , . . . , A τ ) = ⊥ , A τ +1 ← A τ .

̸

If g τ ( A 1 , . . . , A τ ) = ⊥ , let A τ +1 = A τ \ { g τ ( A 1 , . . . , A τ ) } if g τ ( A 1 , . . . , A τ ) = i ⋆ , and let A τ +1 ← A τ otherwise.

̸

̸

We claim that there exists i ⋆ ∈ [ N ] such that g s ( A 1 , . . . , A s ) = i ⋆ for all s < N under this process. To see this define a set X τ inductively: Starting from X 1 = [ N ] , set X τ +1 ← X τ \ g τ ( X 1 , . . . , X τ ) if g τ ( X 1 , . . . , X τ ) = ⊥ , and set X τ +1 ← X τ otherwise; note that this process does not depend on the choice i ⋆ , since g τ itself does not depend on i ⋆ .

̸

For any τ , observe that for any i ∈ X τ , if we set i ⋆ = i , then g s ( A 1 , . . . , A s ) = i ⋆ for all s < τ , and so A τ = X τ . It follows that as long as X τ = ∅ , we can choose i ⋆ such that g s ( A 1 , . . . , A s ) = i ⋆ for all s < τ . Since X τ shrinks by at most one element per iteration, it follows that this is possible for all τ < N .

̸

̸

It follows immediately from Lemma G.2 that by choosing i ⋆ as guaranteed by the lemma, we have

Theorem 3.3 ′ (Impossibility of memoryless algorithms for OEOE; proper variant) . Consider the binary classification setting with Z = Y = { 0 , 1 } and loss D 0 / 1 ( · , · ) . For any N ∈ N and β Off ≥ 0 , there exists an instance ( X , Y , Z , K , F ) with |F| = |X| = N such that for any memoryless oracleefficient algorithm that is (i) proper , and (ii) time-invariant , there exists a sequence of covariates ( x 1 , . . . , x T ) and a proper offline oracle Alg Off with parameter β Off such that E [ Est On D ( T ) ] ≥ Ω(min { N ( β Off +1) , T } ) .

Proof of Theorem 3.3 ′ . Given a parameter N ∈ N , we consider the instance ( X , Y , Z , K , F ) given by X = { x i } i ∈ [ N ] , Z = Y = { 0 , 1 } , D = D 0 / 1 and K ( z ) = ✶ z , with parameter space F := { f i } i ∈ [ N ] given by f i ( x j ) = ✶ ( j ≥ i ) . Let any memoryless oracle-efficient algorithm defined by prediction map F 1 = · · · = F T = F be given. We lower bound the algorithm's online estimation error by considering two cases.

Case 1: There exists a parameter f i such that the distribution µ i = F ( f i ) satisfies µ i ( f i ) < 1 / 2 . We consider two sub-cases of Case 1. The first subcase is where µ i ( { f j : j > i } ) > 1 / 4 . In this case, we choose the sequence of covariates as x 1 = · · · = x T = x i , set f ⋆ = f i , and choose Alg Off to be the offline estimation oracle that sets ̂ f 1 = · · · = ̂ f T = f i . With this choice, the offline estimation error for the oracle satisfies

However, the online estimation error satisfies

̸

The second sub-case of Case 1 is where µ i ( { f j : j > i } ) ≤ 1 / 4 . This, combined with the fact that µ i ( f i ) < 1 / 2 , gives µ i ( { f j : j < i } ) > 1 / 4 . In this sub-case, we choose the sequence of covariates as x 1 = · · · = x T = x i -1 , set f ⋆ = f i , and choose Alg Off to be the offline estimation oracle that sets f 1 = · · · = f T = f i . In this case, the offline estimation error is zero:

In addition, the online estimation error is lower bounded by

Case 2: For all parameter f i ∈ F , the distribution µ i = F ( f i ) has µ i ( f i ) ≥ 1 / 2 .

In this case, we choose x 1 , . . . , x T by repeating each of the covariates ⌊ β Off ⌋ +1 number of times in increasing order by their index, and choose the offline estimation oracle Alg Off to return f 1 , . . . , f N in the same block-wise but with the index offset by 1 .

Formally, let τ t = ⌈ t/ ( ⌊ β Off ⌋ +1) ⌉ denote the index of the block that round t belongs to, so that τ 1 = · · · = τ ⌊ β Off ⌋ +1 = 1 , τ ⌊ β Off ⌋ +2 = · · · = τ 2 ⌊ β Off ⌋ +2 = 2 and so on. We choose x t = x min { τ t ,N } , and choose the offline oracle Alg Off to set ̂ f t = f min { τ t ,N } . Finally, we set f ⋆ = f N .

We have that for all t , the offline estimation error of the oracle is bounded as.

However, the online estimation error is lower bounded by

Proposition D.1 (Upper bound for memoryless OEOE) . For any instance ( X , Y , Z , K , F ) , metric-like loss D , and offline oracle Alg Off with parameter β Off , the algorithm that returns ¯ f t = ̂ f t := Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) has online estimation error Est On D ( T ) ≤ O (( β Off +1) |X| log T ) .

Proof of Proposition D.1. The proof is very similar to the second part of the proof of Lemma G.1. For each x ∈ X , define N t -1 ( x ) = t -1 ∑ s =1 ✶ ( x s = x ) . Then we can write the online estimation error as

As a consequence of the offline estimation guarantee for ̂ f t , we have that

Combining this with the preceding inequality gives

Now, for any x ∈ X , define t x := min { t ≤ T : x t = x } if this set is not empty, and let t x = T otherwise. From this definition and the fact that 1 + 1 / 2 + · · · +1 /T ≤ 1 + log T , we have that

We conclude that

G.3 Proofs from Appendix D

Theorem D.1 (Reduction from oracle-efficient online estimation to delayed online learning) . Let D be any convex, metric-like loss. Suppose we run Algorithm 2 with delay parameter N ∈ N and a delayed online learning algorithm A DOL for the class F . Then for all γ ≥ 1 , Algorithm 2 ensures that

with any offline oracle Alg Off with parameter β Off ≥ 0 , where

is the regret of A DOL for the sequence of losses constructed in Algorithm 2.

Proof of Theorem D.1. Using the metric-like loss property, we can bound the online estimation error of Algorithm 2 by

By the regret guarantee for the delayed online learning algorithm A DOL , we have

since f ⋆ ∈ F . Combining these observations, we have that

Finally, from the definition of ˜ f t and the convexity of the loss D , we have

where the final line uses the offline estimation guarantee for Alg Off . This completes the proof.

Lemma D.1. Consider the delayed online learning setting with a delay parameter N . There exists an algorithm that achieves

for any sequences of losses ℓ 1 , . . . , ℓ T ∈ [0 , 1] .

Proof of Lemma D.1. This result follows using Lemma G.3 with γ = 2 , choosing A OL to be the exponential weights algorithm described in Corollary 2.3 of Cesa-Bianchi and Lugosi [17], which has

for all T ∈ N and γ ≥ 1 .

Theorem D.2 (Characterization of oracle-efficient learnability for binary classification) . Consider a binary classification instance ( X , Y , Z , K , F ) with Z = Y = { 0 , 1 } , D = D 0 / 1 and K ( z ) = ✶ z . For any class F and β Off ≥ 0 , there exists an oracle-efficient algorithm that achieves online estimation error O ( √ β Off Ldim( F ) · T log T + Ldim( F ) log T ) . On the other hand, in the worst-case any algorithm must suffer at least Ω(Ldim( F )) online estimation error.

Proof of Theorem D.2. For the lower bound we recall that for β Off = 0 , Lemma 21.6 of [57] states that any algorithm (oracle-efficient or not) has to suffer Ω(Ldim( F )) online estimation error in the worst case.

For the remainder of the proof, we focus on establishing the upper bound. For any set of parameters F : X → ∆( { 0 , 1 } ) , define the majority vote function Majority( F ) for a class F via

for all x ∈ X . We will show that Algorithm 4 (a variant of Algorithm 2 that replaces averaging with a majority vote), with a properly chosen delayed online learning algorithm A DOL , can obtain

online estimation error.

Algorithm 4 Reduction to delayed online learning for binary loss

1: input: Offline estimation oracle Alg Off with parameter β Off ≥ 0 , delay parameter N ∈ N , delayed online learning algorithm A DOL for class F .

2: for t = 1 , 2 , . . . , T do

3: Receive ̂ f t from offline estimation algorithm.

4: if t > N then

5: Let ˜ f t -N = Majority ( { ̂ f i } t i = t -N +1 ) . // This is the only different step compared to Algorithm 2.

6: Let ℓ t -N ( f ) = D 0 / 1 ( ˜ f t -N ( x t -N ) , f ( x t -N ) ) and pass ℓ t -N ( · ) to A DOL as the delayed feedback.

7: Let µ t = A t DOL ( ℓ 1 , .., ℓ t -N ) be the delayed online learner's prediction distribution.

8: Predict with ¯ f t ∼ µ t and receive x t .

Let γ ≥ 1 be fixed, and consider any delayed online learning algorithm A DOL that achieves

for any sequence of losses in the delayed online learning setting with delay N (i.e., where we receive loss ℓ t at time t + N for some N ≥ 0 ).

We proceed to bound the regret of Algorithm 4. Since the loss D 0 / 1 is metric-like, the online estimation error is upper bounded by

Next, the guarantee of A DOL ensures that

since f ⋆ ∈ F . Combining these observations, we have that

Finally, we observe that for each step t , if D 0 / 1 ( ˜ f t ( x t ) , f ⋆ ( x t ) ) = 1 , it means that least N/ 2 of the predictors ̂ f t +1 , . . . , ̂ f t + N must have predicted f ⋆ ( x t ) incorrectly. This implies that

But since the offline estimation assumption states that

this implies that

We conclude that

To complete the proof, we set γ = 2 and choose A OL to be the algorithm described in Theorem 21.10 of Shalev-Shwartz and Ben-David [57], which (by incorporating the same technique 18 as Corollary 2.3 of Cesa-Bianchi and Lugosi [17]), ensures that

Then by Lemma G.3 with γ = 2 , we have

Setting N = √ β Off T/ (Ldim( F ) log T ) ∨ 1 , this yields

G.3.1 Supporting Lemmas

Algorithm 5 Reduction from delayed online learning to non-delayed online learning

The following lemma is a standard result [67, 42, 35, 47] which shows that the delayed online learning problem setting in Appendix D can be generically reduced to non-delayed online learning. The idea behind the reduction, which is displayed in Algorithm 5, is as follows. Given a delay parameter N ∈ N , we run N copies A 1 OL , . . . , A N OL of a given 'base' online learning algorithm A OL for a class F over disjoint subsequences of rounds. The following lemma gives a guarantee for this reduction

Lemma G.3 (Delayed online learning reduction) . Let A OL be a base online learning algorithm for the class F with the property that for any sequence of losses ℓ 1 , . . . , ℓ T in the non-delayed online learning setting and any γ ≥ 1 ,

If we run Algorithm 5 with delay parameter N ∈ N , then for all γ ≥ 1 , the algorithm ensures that

N

·

R

OL

(

T/N,γ

for online learning with delay N .

)

Proof of Lemma G.3. By the guarantee of A OL , we have that for all i ∈ [ N ] ,

Summing up over all i ∈ [ N ] , we obtain

H Proofs from Section 4

H.1 Proofs from Section 4.1

Theorem 4.1 (Computational lower bound for OEOE) . Assume the existence of one-way functions. 19 There exists a sequence of polynomially computable classes ( F 1 , F 2 , . . . , F n , . . . ) , along with a sequence of poly( n ) -output description length offline oracles with β Off = 0 associated with each F n , such that for any fixed polynomials p, q : N → N and all n ∈ N sufficiently large, any oracle-efficient online estimation algorithm with runtime bounded by p ( n ) must have E [ Est On D ( T )] ≥ T/ 4 for all 1 ≤ T ≤ q ( n ) . At the same time, there exists an inefficient algorithm that achieves E [ Est On D ( T )] ≤ O ( √ n ) for all T ∈ N .

Proof of Theorem 4.1. We frame the example proposed by Blum [14] in their Theorem 3.2 (see also Bun [15]) in our setting. For any integer n ≥ 1 , let the covariate space X n be X n = { 0 , 1 } n , and set Z = Y = { 0 , 1 } and K ( z ) = ✶ z . We define a class F n = { f s : s ∈ { 0 , 1 } √ n } with

for a certain collection of subsets { c s ∈ X n } s ∈{ 0 , 1 } √ n defined in Definition 2 of Blum [14], which is constructed based on cryptographic functions using the assumption of existence of one-way functions. The precise definition will not be important. The properties we will use are:

The value f s ( x ) can be computed in poly( n ) time for any x ∈ X n .

For any polynomials p ( n ) , q ( n ) , any (possibly randomized) online estimation algorithm (oracle-efficient or not) which runs in time p ( n ) , and any time step T ≤ q ( n ) , for sufficiently large n where q ( n ) ≪ 2 √ n , 20 there exists s ∈ { 0 , 1 } √ n and a sequence x 1 s , . . . , x 2 √ n -1 s , x 2 √ n s (the specific definition of this sequence can be found in Blum [14]) such that the online estimation error under this sequence when f ⋆ = f s is at least T/ 4 in expectation. Our lower bound construction for any oracle efficient online estimation algorithm with runtime bounded by p ( n ) in time step bounded by 1 ≤ T ≤ q ( n ) will choose the aforementioned covariate sequence as the covariates revealed with the aforementioned function as the true parameter, i.e., x τ = x τ s for τ ∈ [ T ] and f ⋆ = f s .

It is straightforward to see that the sequence ( F 1 , F 2 ..., F n , ... ) admits polynomial description length as claimed, since log |F n | = √ n . We are left to verify that there is an offline oracle that achieves β Off = 0 with poly( n ) -output description length, yet does not provide any information not already available to the learner in the setting of Blum [14] (recall that in the protocol of Blum [14], the learner gets to see the covariates x 1 ,. . . , x t and the true labels y 1 ,. . . , y t -1 at time step t before making their prediction, but does not receive any other feedback).

Consider the following offlines oracle Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) . The oracle output ̂ f t is a circuit that, on input x , compares x sequentially with x 1 , . . . , x t . If x is ever equal to x τ for some τ ∈ [ t -1] , the circuit will output y τ . If x is not equal to any x τ for τ ∈ [ t -1] , the circuit outputs 0 . Such a Boolean circuit can be constructed with polynomial size in n because each x τ has length n for τ ∈ [ t -1] and t ≤ T ≤ q ( n ) by assumption. It is easy to see that this oracle achieves β Off = 0 , yet does not provide any additional information about target parameter f ⋆ beyond what is available in the model of Blum [14]. Combining all the above, we complete our lower bound proof.

Lastly, we observe that since the setting we consider is an instance of noiseless binary classification, the classical halving algorithm achieves an online estimation error bound of O (log |F n | ) = O ( √ n ) [17].

H.2 Proofs from Section 4.2

In this section, we prove Theorem 4.2 through four layers of reductions through different variants of the online estimation setting. In Appendix H.2.1, we first introduce the relevant settings and the describe reductions through them. We then combine these reductions to prove Theorem 4.2. Finally, in Appendix H.2.2, we prove each of the four reduction results.

H.2.1 Proof of Theorem 4.2

Theorem 4.2. Let A CDE be an arbitrary (unrestricted) online estimation algorithm that satisfies Eq. (4) and has runtime Time ( F , T ) . Then for any N ∈ N , there exists an oracle-efficient online estimation algorithm that achieves estimation error

with runtime poly( Time ( F , T ) , log |F| , log |X| , T ) , where β Off ≥ 0 is the offline estimation parameter. The distributions µ 1 , . . . , µ T produced by the algorithm have support size poly(log |F| , log |X| , T ) . As a special case, if the online estimation guarantee for the base algorithm holds with R CDE ( T ) ≤ C ′ F log T for some problem-dependent constant C ′ F ≥ 1 , then by choosing N appropriately, we achieve E [ Est On H ( T ) ] ≤ ˜ O ( ( C F ( C F + C ′ F ) β Off ) 1 / 2 log V · T 1 / 2 +( C F + C ′ F ) log V ) .

Proof of Theorem 4.2. The proof of Theorem 4.2 is algorithmic, and is based on several layers of reductions.

First, using the scheme in Appendix D, we reduce the problem of oracle-efficient online estimation to delayed online learning with the loss function ℓ t -N ( f ) = D ( ˜ f t -N ( x t -N ) , f ( x t -N ) ) defined in Algorithm 2, where ˜ f t -N = 1 N ∑ t i = t -N +1 ̂ f i is an average of offline estimators and N ∈ N is a delay parameter.

Then, using a standard reduction [67, 42, 35, 47], we reduce the delayed online learning problem above to a sequence of N non-delayed online learning problems, with the same sequence of loss functions; both this and the preceding step are computationally efficient.

To complete the reduction, we argue that the base algorithm can be used to solve the online learning problem above in an oracle-efficient fashion. To do this, we simulate interaction with the environment by sampling fictitious outcomes y t ∼ ˜ f t ( x t ) from the averaged offline estimators and passing them into the base algorithm. We argue that the fictitious outcomes approximate the true outcomes well through a change-of-measure argument.

Combining the above, we conclude that given any base algorithm that efficiently performs online estimation with outcomes sampled from the target parameter f ⋆ , we can efficiently construct a computationally efficient and oracle-efficient algorithm. In more detail, we introduce four layers of reduction in reverse order from CDE to OEOE.

Conditional Density Estimation with Reference Outcomes (CDEwRO). The bottom-most reduction we consider is from a setting we refer to as Conditional Density Estimation with Reference Outcomes (CDEwRO) to the (realizable) CDE setting. CDEwRO is similar to CDE, but with the following difference. Instead of receiving outcomes y 1 , . . . , y T sampled from the true model f ⋆ ( x t ) directly, in CDEwRO, the outcome is sampled from a reference parameter ˜ f t ( x t ) which is guaranteed to be close to f ⋆ in a certain sense. Moreover, the covariates and the reference parameters ˜ f 1 ,. . . , ˜ f T are selected obliviously (i.e. the entire sequence is chosen by the adversary before the online learning protocol begins).

Algorithm 6 Reduction from CDEwRO to CDE

1: input: Time T ∈ N , base algorithm A CDE .

2: Nature selects T covariates x 1 , . . . , x T along with the reference parameters ˜ f 1 , . . . , ˜ f T .

3: for t = 1 , . . . , T do

4: Learner predicts ¯ f t ∼ µ t = A CDE ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) .

5: Outcome y t ∼ ˜ f t ( x t ) is sampled and revealed to the learner with the covariate x t .

Our reduction from CDEwRO to CDE is given in Algorithm 6. The main guarantee for this reduction is as follows.

where R CDE ( T ) is defined as in Eq. (4) by the assumption on A CDE . Then Algorithm 6 achieves an expected online estimation error upper bound of

in the CDEwRO setting, and has runtime poly( Time ( F , T ) , T ) .

The key technique in the proof of this lemma is a change of measure argument based on DonskerVaradhan [46].

Conditional Density Estimation with Reference Parameters (CDEwRP). The next reduction in our stack is from a setting we refer to as Conditional Density Estimation with Reference Parameters (CDEwRP) to the CDEwRO setting above. CDEwRP is identical to CDEwRO, except that in the former setting, the learner directly observes the reference parameter ˜ f t instead of observing y t ∼ ˜ f t ( x t ) as in CDEwRO.

A second difference is that we allow the adversary in the CDEwRP setting to be adaptive , while our definition of the CDEwRO setting only allows for oblivious adversaries. Thus, the reduction we consider serves two purposes:

Simulating the CDEwRO feedback model through sampling.

Reducing the adaptive adversary to an oblivious one.

The reduction from adaptive adversaries to oblivious follows and improves upon the result from [31], and may be of independent interest.

Our reduction from CDEwRP to CDEwRO is displayed in Algorithm 6, and takes as input an algorithm A CDEwRO ( · ; · ) for the CDEwRO setting, where A t CDEwRO ( T ; · ) denotes the algorithm's output at round t ≤ T as a function of the history. The main guarantee for the algorithm is as follows.

Algorithm 7 Reduction from CDEwRP to CDEwRO

1: input: Time T ∈ N , accuracy parameter ε > 0 , algorithm A CDEwRO . 2: Let L = O ( T log( T |F||X| ) /ε ) . 3: for t = 1 , . . . , T do 4: for i = 1 , . . . , L do 5: for s = 1 , . . . , t -1 do 6: Learner samples y s,t i ∼ ˜ f s ( x s ) . 7: Learner computes ¯ f t i = A CDEwRO ( T ; x 1 ,t i , . . . , x t -1 ,t i , y 1 ,t i , . . . , y t -1 ,t i ) . ¯ { ¯ } .

8: Learner predicts via f t ∼ µ t = Unif( f t i i ∈ [ L ] )

9: Nature selects and reveals the covariate x t and the reference parameter ˜ f t based on µ t .

where R CDEwRO ( T, ζ ) is defined as in Eq. (25). Then Algorithm 7 with parameter ε > 0 achieves an expected online estimation error upper bound of

in the CDEwRP setting, and runs in time poly( Time ( F , T ) , T, log |F| , log |X| , 1 /ε ) . The distributions µ 1 , . . . , µ T produced by the algorithm have support size poly(log |F| , log |X| , T, 1 /ε ) .

Conditional Density Estimation with Delayed Reference Parameters (CDEwDRP). Our next reduction is from a setting we refer to as Conditional Density Estimation with Delayed Reference Parameters (CDEwDRP) to the CDEwRP setting. CDEwDRP is identical to CDEwRP, except that the reference function ˜ f t is revealed only at round t + N instead of at round t , for a delay parameter N ∈ N .

Algorithm 8 Reduction from CDEwDRP to CDEwRP

1: input: Time T ∈ N , delay time N ∈ N , algorithm A CDEwRP .

3: for t = 1 , . . . , T do

2: Initialize N copies of the algorithm A CDEwRP as A 1 CDEwRP , . . . , A N CDEwRP .

4: Learner predicts ¯ f t ∼ µ t = A i CDEwRP ( T/N, 1 /N ; x i , x i + N , . . . , x t -N , ˜ f i , ˜ f i + N , . . . , ˜ f t -N ) where i ≡ t mod N .

5: Nature selects and reveals the covariate x t and the reference parameter ˜ f t -N based on µ t .

Our reduction from CDEwDRP to CDEwRP is displayed in Algorithm 7, and takes as input an algorithm A CDEwRP ( · ; · ) for the CDEwRP setting, where A t CDEwRP ( T, ε ; · ) denotes the algorithm's output at round t ≤ T with accuracy parameter ε > 0 (cf. Algorithm 7), as a function of the history. The main guarantee for the algorithm is as follows.

where R CDEwRP ( T/N,ζ i , 1 /N ) is defined as in Eq. (26). Then Algorithm 8 achieves an expected online estimation error upper bound of

in the CDEwDRP setting, and has runtime poly( Time ( F , T ) , T, log |F| , log |X| ) . The distributions µ 1 , . . . , µ T produced by the algorithm have support size poly(log |F| , log |X| , T ) .

Algorithm 9 Reduction from OEOE to CDEwDRP

1: input: Time T ∈ N , offline estimation oracle Alg Off with parameter β Off ≥ 0 , delay parameter N ∈ N , CDEwDRP algorithm A ODEwDRP .

2: for t = 1 , . . . , T do

3: Receive ̂ f t = Alg t Off ( x 1 , . . . , x t -1 , y 1 , . . . , y t -1 ) .

4: Learner computes reference parameter ˜ f t -N = 1 N t ∑ i = t -N +1 ̂ f i .

5: Learner predicts ¯ f t ∼ µ t = A ODEwDRP ( T, N ; x 1 , x 2 , . . . , x t , ˜ f 1 , ˜ f 2 , . . . , ˜ f t -N ) .

6: Nature selects and reveals the covariate x t based on µ t .

Oracle-Efficient Online Estimation (OEOE). Our final reduction reduces the Oracle-Efficient Online Estimation setting (OEOE) to the CDEwDRP setting described above. This reduction, which is displayed in Algorithm 8, is a variant of the approach used in Theorem D.1. The reduction takes as input a CDEwDRP algorithm A ODEwDRP ( · ; · ) , where A t ODEwDRP ( T, N ; · ) denotes the algorithm's output at round t ≤ T with delay parameter N , as a function of the history.

Lemma H.4. Algorithm 9 achieves an expected online estimation error upper bound of

in the OEOE setting, and has runtime poly( Time ( F , T ) , T, log |F| , log |X| ) . The distributions µ 1 , . . . , µ T produced by the algorithm have support size poly(log |F| , log |X| , T ) .

Completing the proof of Theorem 4.2. To prove Theorem 4.2, we compose all of the preceding reductions, with N left as a free parameter temporarily. We first apply Lemma H.4 to reduce from the OEOE setting to the CDEwDRP setting, with the guarantee that

Then by Lemma H.3, we can reduce the CDEwDRP setting to the CDEwRP setting, with the guarantee by Eq. (27) that

Then apply Lemma H.2 N times with T , ζ , and ε in the Lemma chosen to be T/N , ζ i , and 1 /N respectively for each i ∈ [ N ] , we can reduce the CDEwRP setting to the CDEwRO setting with guarantee by Eq. (26) that

Consequently, apply Lemma H.1 N times with T and ζ in the Lemma chosen to be T/N and ζ i for each i ∈ [ N ] , we can reduce the CDEwRO setting to the CDE setting with guarantee by Eq. (25) that

By choosing N = √ C F β Off T · log V R CDE ( T )+ C F · log( V C F T ) ∨ 1 , we can bound the expression in (28) as

Finally, under the assumption that R CDE ( T ) ≤ C ′ F log T , the bound above can be further simplified as

H.2.2 Proofs for supporting lemmas

Proof of Lemma H.1. For Algorithm 6, denote the randomness of the sequence ( y 1: T , µ 1: T ) under f ⋆ ∈ F and ( ˜ f 1: T ) by P f ⋆ and P ˜ f 1: T respectively. The data generating process for ( x 1: T , y 1: T ) in the CDEwRO setting implies that

By Donsker-Varadhan [46], we have that for all η > 0

For any random variables X,Y,Z , we denote by D KL ( P X ∥ P Y | Z ) = E Z [ D KL ( P X | Z ∥ P Y | Z ) ] . We further note that by the chain rule for KL divergence,

where the second equality holds because y t follows ˜ f t ( x t ) and f ⋆ ( x t ) respectively, and because the conditional distribution of x t is identical under both laws due to the oblivious assumption of the

covariates in the setting of CDEwRO. Then by the relation between KL and Hellinger (Lemma A.10 of [27]) and that 1 ≤ log V , we have

where the last inequality is by V ≥ e . Combining all of the results so far and using (29), we have

To proceed, using that for any postive random variable X , E [ X ] = ∫ ∞ 0 P ( X ≥ t ) d t , we have

Recall the assumption T ∑ t =1 D 2 H ( ˜ f t ( x t ) , f ⋆ ( x t ) ) ≤ ζ and let η = 1 C F . We have

where the second inequality uses the assumption Eq. (4) on the algorithm.

Proof of Lemma H.2. Our result improves uses the proof technique from the adversarial-to-oblivious reduction in Lemma 11 of Gonen et al. [31], but improves the result by a O (log T ) factor.

Consider the CDEwRP setting. Let E t [ · ] := E [ · | x 1: t -1 , ˜ f 1: t -1 ] . Let µ s,t := E t [ A CDEwRO ( y 1 ,t 1 , . . . , y s -1 ,t 1 , x 1 ,t 1 , . . . , x s -1 ,t 1 )] for all 1 ≤ s ≤ t ≤ T where the expectation is taken over all the random variables y 1 ,t 1 , . . . , y s -1 ,t 1 , x 1 ,t 1 , . . . , x s -1 ,t 1 .

Then by Bernstein's concentration inequality applied to µ t (interpreted as an empirical approximation to µ t,t ), conditioned on x 1: t -1 , ˜ f 1: t -1 , we have with probability at least 1 -ε 2 T , for all x ′ ∈ X and f ′ ∈ F ,

For any fixed t ∈ [ T ] and all t ′ such that t ≤ t ′ ≤ T , the different trajectories x 1 ,t ′ 1 , . . . , x t -1 ,t ′ 1 , y 1 ,t ′ 1 , . . . , y t -1 ,t ′ 1 are i.i.d. conditioned on x 1: t -1 , ˜ f 1: t -1 . Thus, we have

Finally, for Algorithm 7, we have by the guarantee of A CDEwRO ,

Combining the three results above, we have

where the first equality is by Eq. (30), the second equality is from Eq. (31), and the final inequality is by the guarantee of A CDEwRO .

Proof of Lemma H.3. Note that Algorithm 8 is a variant of the reduction in Lemma G.3, specialized to squared Hellinger distance, and the proof here will use the same idea as Lemma G.3.

For each i ∈ [ N ] , let ζ i = ∑ T/N j =1 D 2 H ( ˜ f i + N · j ( x i + N · j ) , f ⋆ ( x i + N · j ) ) . Then by the guarantee of A CDEwRP , we that for all i ∈ [ N ] ,

Summing up over all i ∈ [ N ] , we obtain

By the assumption on T ∑ t =1 D 2 H ( ˜ f t ( x t ) , f ⋆ ( x t ) ) , we have

Finally, we conclude

Proof of Lemma H.4. This reduction is arguably the most interesting one. This reduction is by noticing that averaging across the outputs of the offline oracle will generate reference parameters (although delayed) with small online estimation errors as shown later in Eq. (34). By the guarantee of A ODEwDRP , we have

where ζ can be chosen to be any upper bound of T ∑ t =1 D 2 H ( ˜ f t ( x t ) , f ⋆ ( x t ) ) since it is unknown to the learner in the setup where we augment the sequence of ̂ f 1 ,. . . , ̂ f T by setting ̂ f T + s = ̂ f T for all s ∈ N and define ˜ f t := 1 N ∑ t + N i = t +1 ̂ f i for t = T -N,T -N +1 , . . . , T . Furthermore, by the definition of ˜ f t , we can obtain

Thus, we conclude Algorithm 9 obtains

I Proofs from Appendix E

Theorem E.1 (Offline-to-online conversion under coverability) . For any layer-wise loss D RL and MDP class ( M , Π , O ) and M ⋆ ∈ M , the sequence of estimators ( ̂ M 1 , . . . , ̂ M T ) produced by any offline estimation oracle Alg Off for D RL with parameter β Off satisfy

Proof of Theorem E.1. This proof closely follows the proof of Theorem 1 in Xie et al. [69]. Define the shorthand d t h ( s, a ) ≡ d π t h ( s, a ) and C cov = C cov ( M ⋆ ) , and define

From the definition of the layer-wise loss, we have

We define a 'burn-in' phase for each state-action pair ( s, a ) ∈ S × A by defining

Let h ∈ [ H ] be With this definition, for we can write

For the first term above, we have

where the last inequality holds because

The remaining term is

Following the derivation in Theorem 1 of Xie et al. [69], we have

where the last inequality follows from Lemmas 3 and 4 of Xie et al. [69]. For the second term, as a consequence of the offline estimation assumption, we have that for all t ∈ [ T ] ,

Altogether, we conclude that

Corollary E.2. For any MDP class ( M , Π , O ) and M ⋆ ∈ M , the sequence of estimators ( ̂ M 1 , . . . , ̂ M T ) produced by any offline estimation oracle Alg Off for squared Hellinger distance D 2 H with parameter β Off satisfy

Proof of Corollary E.2. By Lemma F.2, for any two MDP models M and M ′ and any π ∈ ∆(Π RNS ) , we have

On the other hand, for any h ∈ [ H ] , we have from Lemma A.9 of Foster et al. [27] that

by choosing X = ( s h , a h ) and Y = ( r h , s h +1 ) in the aforementioned lemma. Summing up over h , we conclude that

Consider any sequence of policies π 1 , . . . , π T and outputs ̂ M 1 , . . . , ̂ M T from an offline oracle with parameter β Off for squared Hellinger distance. By Eq. (36), we have that for all t ∈ [ T ] ,

Theorem E.1 thus implies that

Finally, using Eq. (35), we can convert the inequality above into a bound on the square Hellinger distance:

Proposition E.2 (Tightness of offline-to-online conversion) . For any integer T ≥ 1 and β Off > 0 , there exists a contextual bandit class ( M , Π = A S , O ) with |A| = 2 , a distribution d 1 ∈ ∆( S ) , a sequence ( π 1 , . . . , π T ) and an offline oracle Alg Off for D CB with parameter β Off such that the oracle's outputs ( ̂ M 1 , . . . , ̂ M T ) satisfy

Proof of Proposition E.2. Let β Off > 0 and T be given, and let N ∈ N be chosen such that T = N ·⌊ Nβ Off ⌋ ; we assume without loss of generality that N is large enough such that Nβ Off > 1 , which implies that ⌊ Nβ Off ⌋ ≥ Nβ Off / 2 ). Define A = { a 0 , a 1 } , S = { s 0 , . . . , s N -1 } , and g M⋆ ( s, a ) ≡ 0

for all s ∈ S , a ∈ A . Let d 1 = Unif( S ) be the context distribution. For any t ∈ { 1 , . . . , T } , consider the deterministic policy π t : S → A and the offline estimator ̂ M t defined via

We first verify that ̂ M 1 , . . . , ̂ M T satisfy the offline oracle requirement. For any t ∈ { 1 , . . . , T } , we have

However, the online error is

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: See Section 1.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper provides detailed assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work. There is no societal impact on the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the possibility of converting offline estimation algorithms into online estimation algorithms using an information-theoretic approach. It introduces the Oracle-Efficient Online Estimation (OEOE) framework, where the learner interacts with a data stream indirectly through a sequence of offline estimators produced by a black-box algorithm. The main contributions are the characterization of the statistical and computational complexity of online estimation within this framework. It demonstrates that near-optimal online estimation error can be achieved via offline estimation oracles, though computational efficiency is generally not feasible except in specific cases like conditional density estimation. This framework is also applied to interactive decision-making problems, providing insights into the relative power of online and offline estimators.\n\nSTRENGTHS:\nHere are some strengths of the paper:\n\n1. This work introduces a novel framework, Oracle-Efficient Online Estimation (OEOE), which creatively combines existing ideas from offline and online estimation and therefore close the gap between classical statistical estimation and contemporary online learning.\n\n2. Paper provides theoretical analysis and proof to support its claims. The results show a deep understanding of the complexities involved in converting offline estimation algorithms to online contexts.\n\n3. By addressing the relative power of online and offline estimation methods, the paper studies a fundamental question in the field of statistical learning. The implications of this work extend to practical applications in interactive decision-making, such as contextual bandits and reinforcement learning.\n\nWEAKNESSES:\nHere are some weaknesses of the paper:\n\n1. While the paper provides a comprehensive theoretical analysis, it acknowledges the computational inefficiency of the proposed Oracle-Efficient Online Estimation (OEOE) framework in general cases. The authors could improve this aspect by exploring potential heuristic approaches or approximations that could offer practical computational advantages, even if they come with some trade-offs in theoretical guarantees.\n\n2.  The paper lacks empirical validation of the proposed framework. Including a few experimental results or simulations demonstrating the practical performance of the OEOE framework could significantly strengthen the paper. Even a small set of experiments showing the feasibility and comparative performance against existing online estimation methods would provide valuable insights.\n\n3. Certain sections of the paper, particularly the detailed proofs and technical discussions, are dense and may be difficult for readers to follow. Improving the clarity of these sections by adding more intuitive explanations, step-by-step breakdowns of complex proofs, or illustrative examples could make the paper more accessible. This would help readers better understand the contributions .\n\nQUESTIONS:\n1. Have you considered any specific real-world applications or practical scenarios where this framework could be directly applied or tested? How do you envision the implementation challenges being addressed in such cases?\n\n2. The theoretical results are derived under certain assumptions, such as the metric-like structure of the loss function and the bounded offline estimation error. How sensitive are your results to these assumptions? Could you provide insights or potential extensions to your framework that could handle more relaxed or different sets of assumptions, thereby increasing the generalizability of your findings?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm that can convert offline estimation oracles into online estimation algorithms in a black-box fashion. The conversion is built within the OEOE framework, which manipulates the learner, offline oracle, and environment simultaneously. The authors also propose certain upper bounds and lower bounds for their conversion algorithm, which shows that it achieves both statistical and computational efficiencies. This paper also presents several additional results like the hardness of memoryless algorithms, enhancing the soundness of this paper.\n\nSTRENGTHS:\n1. The framework of OEOE is general enough to include many real-world problems.\n\n2. The statistical complexity has its lower and upper bounds nearly matched.\n\n3. The author shows that although there is no computational algorithm for OEOC, their algorithm is computationally efficient in some fine-grained special cases.\n\nWEAKNESSES:\n1. I understand that many works assume knowing the intrinsic of the data-generating, but assuming that $\\mathcal{K}$ is known would exclude some cases of statistical estimation, which might reduce the generality of OEOE.\n\n2. Can you provide a quantified definition of “oracle-efficient”? Despite it has been heavily used in the paper, I can only find some rough explanations after searching through the text.\n\n3. I have not checked the proof of theorem 3.2 and theorem 4.1 carefully, so maybe I am wrong, but the hardness construction in them seems a bit ill-conditioned. The worst case is searched through all offline oracles, the adversary seems overpowered.\n\n4. As stated in Theorem 4.1, there is no computational algorithm in OEOE. So could it be that the definition of this framework includes some unwanted cases that should be excluded? And does that mean that this framework needs some redefinition?\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper studies the methods to convert offline estimation algorithms into online estimation algorithms in a black-box fashion. This work introduce a new protocol, Oracle-Efficient Online Estimation, which provides an information-theoretic abstraction of the role of online versus offline estimation.

STRENGTHS:
The problem of online estimation via a sequence of offline estimators is of great importance to the online decision making community.

WEAKNESSES:
The organization of the paper requires significant improvement as it is currently hard to follow. The introduction section occupies almost half of the main text, leaving the main methodology and theoretical results insufficiently discussed and lacking in clear theoretical insights.

QUESTIONS:
This work discusses the impossibility of memoryless oracle-efficient algorithms, I wonder what if the learner can select the online estimator at time t as a function of the most recent "w" offline estimators, where w is the window size.

In addition, the authors present a fine-grained perspective on the computational complexity of oracle-efficient estimation for conditional density estimation. Are there any insights for the other two problems of binary classification and regression?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers an online estimation problem, in which the learner does not have a direct access to the past values (labels), but rather instead just an oracle access to an offline estimator based on past covariate-value pairs. The main assumption is that the loss of these offline estimators is bounded. The paper addresses the question whether this sequence of offline estimators can be converted into an online estimator, both in terms of statistical complexity and computational complexity. From the statistical aspect, it is shown that an efficient conversion is possible, yet only when the learner knows the entire sequence of past offline estimators (and just storing the latest one cannot guarantee this). The proposed protocol can be considered as a generalization of the halving algorithm. \n\nFrom the computational aspect, a negative result is shown stating that polynomial run-time algorithms cannot achieve the optimal online statistical complexity, but is somewhat relieved by the setting of density estimation, since in this setting the learner can artificially produce values $y$ from the estimated density.\n\nSTRENGTHS:\n1. The setting is general, and appears to be related to a multitude of contemporary problems. \n\n2. The paper addresses the problem from both statistical and computational perspectives. \n\n3. The results are detailed, including different algorithms for finite and infinite classes (the latter in the appendix). The results are explained in comparison and in light of classic algorithms (like the halving algorithm). The scaling of the bounds and the dependency on each parameter is justified in detail.\n\nWEAKNESSES:\n1. The main proposed algorithm is somewhat brute-force and just keeps the hypotheses that agree with the data – in this case it is covariance-estimator pairs. \n\n2. It is not obvious that the OEOE model, in which both all past covariates as well as all past offline estimators are saved, actually captures aspects such as limited past data availability and compression of past observations and decisions. This is because storing an estimator may be much more storage-consuming than the value itself (e.g., as in binary classification). Generalizations based on finite past data seems to be challenging.\n\nQUESTIONS:\n1. In line 33 it is claimed that statistical estimation is mainly about fixed design. Random design of the covariate-value is also commonly considered.\n\n2. Theorem 3.3 is an impossibility result, showing that even a memoryless estimator is not efficient, even if it is improper. Doesn't this trivially imply the same result for the subset of proper learners ? Why thus a different result is needed ?\n\nLIMITATIONS:\nThe authors properly discuss limitations of their results throughout the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}