Online Estimation via Offline Estimation: An Information-Theoretic Framework
Authors: Dylan J Foster
Yanjun Han
Jian Qian
Alexander Rakhlin
Venue: NeurIPS 2024 poster (2024)
Keywords: online learning
interactive decision making
oracle-efficient
regression
classification
conditional density estimation
Primary area: online_learning

Abstract: The classical theory of statistical estimation aims to estimate a parameter of interest under data generated from a fixed design (''offline estimation''), while the contemporary theory of online learning provides algorithms for estimation under adaptively chosen covariates (''online estimation''). Motivated by connections between estimation and interactive decision making, we ask: is it possible to convert offline estimation algorithms into online estimation algorithms in a black-box fashion? We investigate this question from an information-theoretic perspective by introducing a new framework, Oracle-Efficient Online Estimation (OEOE), where the learner can only interact with the data stream indirectly through a sequence of offline estimators produced by a black-box algorithm operating on the stream. Our main results settle the statistical and computational complexity of online estimation in this framework.

   $\bullet$ Statistical complexity. We show that information-theoretically, there exist algorithms that achieve near-optimal online estimation error via black-box offline estimation oracles, and give a nearly-tight characterization for minimax rates in the OEOE framework.

  $\bullet$ Computational complexity. We show that the guarantees above cannot be achieved in a computationally efficient fashion in general, but give a refined characterization for the special case of conditional density estimation: computationally efficient online estimation via black-box offline estimation is possible whenever it is possible via unrestricted algorithms.

Finally, we apply our results to give offline oracle-efficient algorithms for interactive decision making.

Online Estimation via Offline Estimation: An Inoormation-Theoretf Framework
Dylan Foster dylanfosterrmiccrosse wwwpssttterrriiiooooottooooo
Yanjun Han yanjunhanênyu.c .du
Jian Qian jianqian@mit t.ddu
Alexander Rakhlin
rakhlin@mit.edd
Abstract
The classical theory of statistical estimation aims estimate parameter of interest under data generated from a fixed design ("offline estimation"), ")), while the contempo- rary theory of online learning provides algorithms for estimation under adaptively chosen covariates ("online estimation"). Motivated by connections between esti- mation and interactive decision making, ask: is il possible to convert offline estimation algorithms into online estimation algoritiuss in black-box fashion? We investigate this question from an information thhcorettcc perspective by introducing a new framework, Oracle- Efffcieet Online Estimation (OEOE), where the learner can only interact with the data stream indirectly through sequence of offline estimators produced by a black-box algorithm operating on the stream. Our main results settle the statistical and computational complexity of online estimation this framework.
1. Statistical complexity. We show that information enformation-hheoretiaalll theoretically, there exist algorithms that achieve near-optimal online estimation error via black-box offline estimation oracles, and give nearly-tight characterization for minimax rates in the OEOE framework.
Computational complexity. We show that the guarantees above cannot be achieved in computationally efficient fashion in general, but give refined characterization for the special case of conditional density estimation: computationally efficient online estimation via black-box offline estimation possible whenever it is possible via unrestricted algorithms.
Finally, we apply our results to give offline oracle-efficient algorithms for interactive decision making.
Introduction
Consider general framework for statistical estimation specified by a tuple (X,), which we will show encompasses classification, regression, and conditional density estimation. The learner is given parameter space FF (typically function class), where each parameter f E F is a map the learner
from the space of covariates X to the space values Z. For an integer T given a dataset (x* v²) (.'),..... (x"", y²", where xx ..... xx" are covariates and y² ..., y" are outcomes generated via y' K(- f*")), where f Fis is an unknown target parameter that the learner wishes to estimate; here IC is probability kernel that assigns to each value E Z distribution KC z) on space outcomes y. We adopt the shorthand K(z) = KC-
throughout.
The classical theory of statistical estimation typically assumes that the covariates x. x²,...,,²² are an ar- bitrary fixed design, and concerned with estimating the target parameter f* E F well in-distribution 64 40 62) Formally, for a loss function D Z Z + R on the space of values Z, the goal of the learner is output an estimator í based (ar² yy)) ( ..... (xF, w²) such that the in-distribution error
= Σ T D(((x')
Est Off (T)
(1)
small; we refer to this as an offline estimation guarantee. Canonical examples include:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Classification (i.e., Distribution--ree PAC learning 38 63 where 2 = y = {0,1}.
(x)) = (-) and Doy1 (f(x), (x)) = ff
is the 0/1-loss.
Regression with well-specified model 62 66), where 2 = = R. and Daq (fx)) / (x)) - (f(x) I (z=)² 2 is the square loss.
Conditional density estimation (12). where Y is an arbitrary alphabet, 2 = A0%,
(x(( = (z)) and (--)) is squared Hellinger distance: see Appendix C foo details.
In parallel statistical estimation, the contemporary theory of online learning [17 49 provides
estimation error algorithms that support adaptively chosen sequences of covariates, a meaningful form of oot-of-distributtinn guarantee. Here, the examples (x', y²) arrive one at time. For each step (.'....,''"",...
t E [T]. the learner produces an estimator f* X + Z based on the data (x*
observed so far. The covariate z' is then chosen in arbitrary fashion, and the outcome is generated via y ~ [f (x")) and revealed to the learner. The quality of the estimators is measured via ?
T - D(F(2') Est On (T): == E
(2)
We refer to this as an online estimation guarantee; classical examples include online classification in the mistake bound model [41] online regression [51), and online conditional density estimation [[]] Online estimation provides non- non-triiiall
guarantee, it requires (on average) that the learner achieves non-trivial estimation performance on covariates x that can be arbitrarily far from the previous examples æt ,...,,"""'' x²' 1. This property has many applications in algorithm design, notably in the context of interactive decision making, where it has recently found extensive use for
x²
problems including contextual bandits 25 58 24), reinforcement learning 22 28
and imitation
learning [56] (55).
In this paper, we investigate the relative power of online and offline estimation through a new information-theerret perspective. It is well known that any algorithm for online estimation can be used as-is to solve offline estimation through online-to-batel conversion, a standard technique in learning theory and statistics [3] 9
8). The converse is less apparent, as online estimation requires non-trivial algorithm design techniques that go well beyond classical estimators like least-squares or maximum likelihood 117. In the case of regression with a finite class F, least squares achieves optimal offline estimation error Est (T) S O(log) FF) and while it is possible to achieve a similar rate Est, (T) S (log]FF) for online estimation, this requires Vovk's aggregating algorithm or exponential weights [65]; directly applying least squares or other standard offline estimators leads to vacuous guarantees. This leads us to ask: it possible to convert offline estimation algorithms into online estimation algorithms in a black-box fashion?
16 61 36
Computationally speaking, this question has practical significance, since online estimation algorithms are typically far less efficient than their offline counterparts (the classical exponential weights algo- rithm maintains separate weight for every f F, which is exponentially less memory-efficee than empirical risk minimization). In fact, at first glance this seems like purely computational question: if the learner has access to offline estimator, nothing is stopping them (information thhooeettthrrtaallyyy.a from throwing the estimator away and using the data to run an online estimation algorithm
El Yet, for 28 56 55), estimation role, and can be viewed compressing the data stream into succinet, operational representation that directly informs downstream decision making. With these applications in mind, the first contribution of this paper to introduce a new protocol, Oracle- Effficeen Online Estimation, which provides an cormmation-thooretee abstraction of the role of online versus offline estimation, analogous to the framework of diformation-base complexity in optimization
aforementioned applications in interactive decision making [25 58 24 27
online estimators- playy a deeper coormationn-hhoreee
43 60 48
and statistical query
complexity in theoretical computer science 13 37 22
'e use 1, to indicate the delta distribution that places probability mass 1 on y.
FFor technical reasons, it is also common to consider randomized estimators where f n², and measure error by Estp := Σ E u [D(f((z))
(x')) (1')))
'We consider unnormalized estimation error in Eq. (1) following the convention of online lcarning. For
normalized estimation error, we have 1. E Off (T)= x T following the convention of statistical estimation.
'Related computational questions have already been studied, with negative results 44 33).
2

---

Protocol Oracle-Efficient Online Estimatiin (OEOE)
1: for = 1,...,,T do
2: Learner receives estimator f = Algón (x' x² .......... yf from offline est. oracle.
3: Based on x',................. ........ x²= and f ..... f learner produces an estimator f 2 which may be randomized according to a distribution yr. 4: Based on n4,, nature selects covariate x' E X' and outcome y ~ (C((((())))) but does not
directly reveal them to the learner.
1.1 Our protocol: oracle-efficient online estimation
In the Oracle-Efficient Online Estimation (OFOE) framework, the aim is to perform online estima- tion in the sense of [.q. (2), with the twist that the learner does not directly observe the outcomes ,'....,y' rather, they interact with the environment indirectly through sequence of offfine estima- tors produced by black-box algorithm operating on the historical data. We formalize this black-box
algorithm as estimation oracle Algon (Al (Algón ) (e.g., Foster and Rakhlin (26)). which is mapping from histories to estimators that enjoy bounded offline estimation error.
Definition 1.1 (Offline estimation oracle). An offline estimation oracle Algon = [Algon) 1
for a statistical estimation instance (X.V, F), and loss D is a mapping Algor (X x 2)-² + (X + Z) such that for any sequence (x',y')..... (x" v²) with y¹ (x)) the sequence of estimators f = Algon (""....," "..,,x"'.y ..''....''") satisfies Est Est = (t)
E
S Bos for all for E [T) almost surely; we allow x¹ to be selected adaptively
based on y'................ ..... y² and f'.....-.... We refer to Bon 2 0 as the offline estimation parameter.
This definition simply asserts that the estimators f produced by the offline estimation oracle satisfy the guarantee in Eq. (I) even when the covariates are selected adaptively. Examples include standard algorithms like least-squares for regression and maximum likelihood for conditional density estimation, which guarantee Bor S O(log]F)) with high probability whenever F is finite class; Appendix C.1 for further background Throughout the paper, we assume for simplicity that 33om > 0
known in advance.
With this definition, we present the Oracle-Efficient Online Estimation protocol in Protocol I In the protocol, a learner aims to perform online estimation, but at each step t, the only information available is the covariates x² z'....... x² and the estimators .'.... f generated by an offline estimation oracle satisfying Definition 1.1, the outcomes y' ." are not directly observed. Based on this infor- mation, the learner produces new estimator f such that the online estimation error Est On (T) Ey [D(f(='), f( in Eq. (2) is m An algorithm is termed oracle-efficient if it attains low online estimation error in the OEOE Tramework. Note that while the learner cannot directly observe the outcomes y' .'.....y' y", the covariates x' ......'' .x" are observed; we prove that without this ability, it is impossible to achieve non-trio non-trivial estimation performance (Section 3).
The OEOE framework abstracts away the property that oracle-efficient algorithms implicitly interact with the environment through compressed, potentially lossy channel (the estimation oracle Algest) We believe this property merits deeper investigation: it is shared by essentially all algorithms from recent rescarch that reduces interactive decision making and reinforcement learning to estimation oracles 225 58 27 28, (6, (55. yet the relative power of offline oracles and analogously defined inoommttion-thoreee abstraction to study oracle-efficiency the OEOE framework plays a role similar to information- based complexity
online oracles is poorly understood in this context. By providing
in optimization 43 60 48 2 and statistical query complexity in theoretical computer science 13 37 21 22], both of which provide rich frameworks for designing and evaluating iterative algorithms that interact with the environment in a structured fashion. We expect that this abstraction will find broader use for more complex domains (e.g., decision making and active learning) as a means to guide algorithm design and prove lower bounds against natural classes of algorithms.
Most algorithms only ensure that the guarantee in Definition 1.1 holds with high probability. We assume an almost sure bound to simplify exposition, but our results trivially extend. Likewise, our results immediately extend to handle the case in which Bor is allowed to grow as a (sublincar) function For technical reasons, allow the learner to randomize the estimator fe via a distribution μ².
3

---

Let us first build some intuition. Familiar readers may recognize that the classical halving algorithum for binary classification (e.g,, Desa-Bianchi and Lugosi (17] can be viewed as oracle-efficient in our framework. Specifically, for binary classification with ) = 2 {0, } and loss function ((x()) + (x)), the halving algorithm can use any offline oracle with Oflog) which is optimal. However, little is known for noisy oracles with Pof > 0, or more general outcome spaces and loss functions (e.g., regression density estimation). In addition, the halving algorithm- while oracle-efficient- computationally inefficient, as it requires maintaining an explicit version space. This leads us to restate our central question formally, in two parts:
f
fow = 0 to achieve Est 1r
1. Can we design oracle-efficient algorithms with near-optimal online estimation error up to polynomial factors (for general instances (X,Y, F, and (oon >0}?
2. Can do in computationally fashion?
1.2
Contributions
For general class of losses D, referred to as metric-like, we settle the statistical and computational complexity of performing online estimation via black-box offline estimation oracles up mild gaps. answering questions (1) and (2) above.
Statistical complexity. Our first result conceming statistical complexity focuses on finite classes F, where the optimal rates for offline and online estimation with standard losses ((-.)) both scale
as ((og Fl). For this setting, we show (Theorem 3.1) that there exists an oracle-efficient online
estimation algorithm that achieves Est. (T) = ((SoF + 1) min (log) Fl. (x{)) in the OEOE framework, and that this is optimal (Theorem 3.2) This provides an affirmative answer to question (1) and characterizes the statistical complexity of oracle-efficient online estimation with finite classes F.
In the general OEOE framework, the learner can use the entire history of offline estimators ƒ ,.....??
and covariates x² x'..... x=~1 to produce the online estimator fr for step t. As a secondary result, we study a restricted class of memoryless oracle-efficient algorithms that choose f only based on the most recent offline estimator f*, and show Theorem 3.3) that it is impossible for such algorithms to achieve low online estimation error.
Lastly, we give a more general approach to deriving oracle-efficient reductions (Theorem D.I) that is based delayed online learning [67 42 35 47]. Using this result, give characterization of learnability with infinite classes for binary classification in the OEOE framework (Thcorem D.2). proving that finite Littlestone dimension is necessary and sufficient for oracle-efficient learnability.
Computational complexity. On the computational side, we provide a negative answer to question (2) showing (Theorem 4.1) that under standard conjectures in computational complexity, there do not exist polynomial-time algorithms with non-trivial online estimation error in OEOE framework. In spite of this negative result, we provide a fine-grained perspective for the statistical problem of conditional density estimation, a general task that subsumes classification and regression and has immediate applications to reinforcement learning and interactive decision making (27] 28 Here we show, perhaps surprisingly ( (Theorem 4.2), that online estimation in the OEOE framework is no harder computationally than online estimation with arbitrary, unrestricted algorithms. This result is salient in light of the applications we discuss below.
Implications for interactive decision making. As the preceding discussion has alluded to, our interest in studying oracle-efficient online estimation is largely motivated by a connection to the prob- lem of interactive decision making. Foster et al. (27 28 . Foster and Rakhlin 26) propose p general framework for interactive decision making called Decision Making with Structured Observations (DMSO), which subsumes contextual bandits, bandit problems with structured rewards, and reinforce- ment learning with general function approximation. They show that for any decision making problem in the DMSO framework, there exists an algorithm that, given access to an online estimation algorithm (or, "oracle") for conditional density estimation for an appropriate class F, it is possible to achieve near-optimal regret. The results above critically make use of online extimation oracles, as they require achieving low estimation error for adaptively chosen sequences of covariates, and it is natural to ask whether similar guarantees can be achieved using only offline estimation oracles. However, positive re- sults are only known for certain special cases [18 20 58), with scant results for reinforcement learning

---

in particular. In this context, our results have the following implication (Corollary E. Information- theoretically, is possible to achieve near-optimal regret any interactive decision making problem using an algorithm that accesses the data stream only through offline estimation oracles.
Additional results. Due to space constraints, the following results are deferred to the appendix: (i) detailed exumples for our statistical estimation frumework (Appendix C); (ū) additional results concerning statistical complexity of the OEOE framework Appendixx D and (iii) detailed results for our application to interactive decision making (Appendix E).
2 Preliminaries
Unless otherwise stated, our results assume the loss function D has metric-like structure.
Definition 2.1 (Metric-like loss). A loss function D Z x Z ++ [0. 1] is metric-like on the set 2 if is symmetric and satisfies (i) D(21, 22) 2 0 for any 21, 22 E Z and ((z, 2) for all E Z; and
(ii) ((21, 22) - CD (D(z), 2a) + (22,, 22)) for all 21, 22, 23 E 2. for an absolute constant Cp 2 1.
Throughout the paper, we focus on three canonical applications, outlined in the introduction: Classi- fication with the indicator loss Do/1 (CD- = 1), regression with the square loss Dsq (CD = 2). and conditional density estimation with squared Hellinger distance D (Co 2). See Appendix Cfoo detailed examples and discussion. (omitted for space).
Finite versus infinite classes. The majority of our results focus on finite classes F. We believe this captures the essential difficulty of the problem, but we expect that most of our sample complexity results (which typically scale with log (Fl) can be extended to infinite classes by combining our techniques with appropriate notions of complexity for the function class (Littlestone dimension for classification, sequential Rademacher complexity. and sequential covering numbers [50] 41 (2]). For the canonical settings of classification, regression, and conditional density estimation, there exist algorithms that achieve Est (I') O(log) FI) and Est (T) Ollog) for arbitrary finite classes; see Appendix C for details.
We defer additional notation and related work to Appendices A and B
3 Statistical complexity of oracle-efficient online estimation
This section presents our main results concerning the statistical complexity of oracle-efficient online estimation. In [Section 3.1] we focus on finite classes F and present an oracle-effiiintt algorithm that achieves near-optimal online estimation error (Theorem 3. We then provide lower bound that shows that our reduction is near optimal (Theorem 3.2p. In Section 3,2] we turn our attention to memoryless oracle-efficient algorithms, proving strong impossibility results (Theorem 3.3).
3.1 Minimax sample complexity for oracle-efticient algorithms
In this section, we present our main statistical conclusion for the OEOE framework: For any finite class F, it is possible to transform any black-box offline estimation algorithm into online estimation algorithm with near-optimal error (up to a logarithmic factor that we show is unavoidable).
Algorithm and minimax upper bound. Our results are achieved through a new algorithm, Version
Space Averaging. described in Algorithm 1 At each round f, the algorithm uses estimators f f.....f' produced by an offline estimation oracle Algor along with the previous covariates xx. r'......r to construct a version space F, C F in [q. (3) Informally, F. consists of all ʃ E F that are [t]. the offline estimation error relative to /" is small; as long as the offline estimation oracle A Algor has offline estimation error Bor DDeinition 1.1). it follows immediately that the construction in [iq. (3) satisfies f" E F, Given the version space Fe, Algorithm predicts by uniformly sampling: ] 1 == Un))(FF), then proceeds to the next round The main guarantee for Algorithm is stated Theorem 3.1
consistent with the estimators f ,...., f in the sense that for all
(((r))((()) fx))) = 1f(()) * (x)) and Sor 0, the version space construction in in/Eq. coinnides with that of the halving algorithm (e.g.. Cesa-Biancr and Lugosi (17), and its estimation error bound [Theorem 3.1) matches the halving algorithm up to absolute constants. As such, Algorithm I can be viewed as a moisy/error-toleran generalization of the halving algorithm, which may find broader use.
For realizable binary classification with Daya (f(x).
5

---

Algorithm 1 Version Space Averaging
1: input: parameter space F, offline estimation oracle A Algon with parameter Bor 2 0.
2: for t = 1,2,.... ..... do
3: Receive f = Algon (x' ....,,
,....,"''' x'' ,.......,""
4: Calculate version space:
F. JEF E4 E.co D( S f0or)
5: Predict fr μ² Unif(FF, and receive x'. Nature draws 3f (xF) and passes to Algor
Theorem 3.1 (Main upper bound for OEOE). For any instance (X. R....2.K. Z,K.F), any metric-like loss D, and any offtine estimator Algon with parameter Son 2 0. Algorithm 7 is oracle-efficient and uchieves
On Est (T) S O(Cb (Bon + 1) min (log [F), X log T)).
Most notably, Algorithm achieves EstD (T) O(CD (Son + 1) log [F]); that is, up to (log][F) factor, the reduction achieves online estimation rates the OEOE framework that are no worse than the minimax rate for offline estimation. For classification, regression, and density estimation with generic finite classes F (Appendix Cj, the best possible offline estimation error rate this shows that price of oracle-efficiency is at most quadratic.
is Soo (logg]F),
Minimax lower bound. Next, we show that the upper bound in Theorem 3.1 is nearly tight, giving a lower bound that matches up to logarithmic factors.
Theorem 3.2 (Main lower bound for OEOE). Consider the binary classification setting with Z = = = {0, 1} and loss (ty)(:.).. For any N E N and Bom > O, there exists an instance U. with log FF| |2| N such that for any oracle-efficient algorithm, there is a sequence of covariates (x' (a'....,2') x²") and offline oracle with parameter Son such that On (T)) 2 Q(min [(Sor 1)),
E EEst
This result states that for generic finite class F and offline estimation oracle Algon any oracle- efficient online estimator must have
E (T)) 2 22min ((Por + 1) log [F), (Bor + 1)}X),
in the worst case. This implies that the log factor we pay for offline online conversion un- avoidable, and that Theorem is optimal up to log T factor, giving a near-optimal characterization for the minimax rate for online estimation in the OEOE framework. We conclude with two remarks: (i) The (Bof + 1) scaling (as opposed to say, (oo) in Theorem 1 is unavoidable, witnessed by the optimality of the halving algorithm for noiseless binary classification [7 (ii) the space Z and the loss D are convex, then we can change Algorithm to output a deterministic prediction by using the average of all parameters in F, rather than the uniform distribution on F,, See SeeLeemma G. for details.
General reductions and infinite classes. Algorithm is somewhat specialized to finite classes. In Appendix 1) (deferred to the appendix for space), we provide more general approach to designing oracle-efficient algorithms based on delayed online learning (Theorem D.T),, and use it to derive characterization of oracle-efficient learnability for classification with infinite classes JF (Theorem D.2).
Full memory vs. finite memory. Algorithm requires full memory of all past offline estimators. The more general approach proposed in Appendix D can use N most recent offline estimators to obtain an estimation error bound of O(CD(N + Bow T/N) + N log [Fl) (Corollary D.1) for any integer N >
3.2 Impossibility of memoryless oracle-efficient algorithms
In the general OBOE framework, the learner can use the entire history of estimators f...., f and covariates _=,,.- r to produce the online estimator f for step t; notably the Version Space Aver- aging algorithm with which our upper bounds in the prequel are derived uses the entire history. In this section, we show that for memoryless oracle-efficient a algorithms (Definition 3.T) that select the esti- mator f only a function of the most recent offline estimator f¹, similar guarantees are impossible.

---

Definition 3.1 (Memoryless algorithm). An online estimation algorithm is memoryless if there exists map (-) such that and μ¹ is the randomization distribution for the online estimator f
write μi' = (f*)). where f All ..... x'-1 ; y ..... y²*
Memoryless algorithms are more practical than arbitrary algorithms, since they do not require storing past estimators or covariates in memory. Our motivation for studying memoryless algorithms arises from recent work in interactive decision making [25] 27]. which shows that there exist near-optimal algorithms for contextual bandits and reinforcement learning that use estimation oracles in memoryless fashion. We show that unfortunately, it is not possible to convert offline estimators into memoryless online estimation algorithms with non-trivial error.
Theorem 3.3 (Impossibility of memoryless algorithms for ОЕOE), Consider the binary classification
[0,1] and loss (o For any N E N and 3on 2 0, there exists an instance (X, v 2, K,, with |F| |X| = N such that for any memoryless oracle-efficient algo- rithm, there exists a sequence of covariates "c",...,x") and (potentially improper) offline oracle Algoh with parameter Bon such that E EE 2 S2(min {N (Bom + +), This conclusion still holds when the online estimation algorithm remembers fr , but not x² r',
setting with Z =
This result shows that in the worst case, any memoryless oracle-efficient algorithm must have
E [E 2 2(( + 1) min (1*))
This precludes an online estimation error bound scaling with (Sor + 1) log| as in Theorem 3.1 and shows that the gap between general and memoryless oracle-efficient algorithms can be exponential.
Interestingly, the lower bound in Theorem 3.3 holds even if the online estimation algorithm is allowed to remember f ........................ f but not ,....,'"!! The intuition here is that without covariate information, it is not possible to aggregate the predictions of previous estimators otherwise use them to reduce uncertainty. This provides post-hoc motivation for our decision to incorporate covariate memory into the OEOE protocol in Section 1.1
The proof of Theorem 3.3 uses that the estimators f produced by the offline estimation oracle may be improper (i.e..) f F). We defer a variant of the result that holds even if the estimation oracle is proper under additional assumptions as well as the complementary upper bound to Appendix D.2
4 Computational complexity of oracle-efficient online estimation
In this section, we turn our attention to the computational complexity of oracle-efficient online estimation in the OEOE framework. In Section 4.1 we show (heorem 4.1) that in general, it is not possible to transform black-box offline estimation algorithms into online estimation algorithms in a computationally efficient fashion. Then, in /Section 4.2 we provide a more fine- mne-grained perspective, showing (Theorem 4.2) that for conditional density estimation, online estimation in the OEOE framework is no harder computationally than online estimation with unrestricted algorithms.
4.1 Computational hardness of oracle-efficient estimation
Our main upper bounds (Section 3.1 show that online estimation error Estp (T) S O((Bon + 1) log (F)) can be achieved in an oracle-efficiem fashion for any finite class F, but the algorithm (Algorithm T) is not computationally efficient. We now show that this is fundamental: There exist classes J for which offline estimation can be performed in polynomial time, yet oracle-efficient algorithm running in polynomial algorithm can achieve sublinear online estimation error.
Computational model. To present our results, we must formalize computational model for sralle-ffficient online estimation, and in particular, define a notion of input length for oracle-efficine online algorithms. To do so, we restrict our attention to noiseless binary classification, and consider sequence of classification instances indexed by n E N, with Z = y f0. 1}, X,, := {0. 1)", K(z) 1,. and indicator loss Doyi We consider sequence of classes F, that have polynomial description lengthi.e. log Fr is polynomial in n, so that f F,, can be described in poly(n) bits. In particular, we assume that f E F, is represented as Boolean circuit of size poly(1) so that f(x) can be computed in (nly(n) time for X,; we refer to such sequences as polynomially computable.
'There are many natural variants of this protocol. For example, algorithms could select μr based on f and B bits of auxiliary memory. We hope future work will explore these variants.
7

---

To allow for offline estimators that are improper, we assume that for all t and all sequences (x',y"), "'")..... (x" ). the output f 10 Algon (z'...., x=' x.'''"''' .........'"".......... .y is a Boolean circuit of size poly(n); we refer to such oracles as having voly(n)-ouupin) description length Likewise, to allow the online estimation algorithm it- self to be improper and randomized, we restrict to algorithms for which computing f())) for f ~ p can be implemented as rx, for random bit string Unif ((0, [0, 1))), where B poly(n); we refer to the online estimator as having poly (nly(y)--uppu description length if (';,,) is itself a Boolean circuit of size poly(n). We refer to the online estimation algorithm polynomial time if it runs in time poly(n) for any sequence of inputs t, x ,',..,, and description length 10
{0.1} returned by
and has (olyyn)-output
Main lower bound. Our main computational lower bound is as follows.
Theorem 4.1 (Computational lower bound for OЕOE). Assume the existence of one-way functions !T There exists a sequence of polynomially computable classes
Fn. along with a sequence of poly (n)ttutuu description length offline oracles with Bon 0 associated with each F...
such that for any fixed polynomials p. q N - N and all n E N sufficiently large, any oracle-effiiint online estimation algorithm with runtime bounded by p(n) must have EE EEst (T) 2 T/4 for all 1 1 T S q(n). At the same time, there exists an inefficient algorithm that achieves E Est O(yn) for all T E N.
On (T)) S
Informally, Theorem 4. shows that there exist a class F and offline estimation oracles Algon for which no oracle-efficient online estimation algorithm that runs in time
(Xen(x)), len(F), max, len (?),
can achieve sublinear estimation error, where len( and lFn denote the number of bits required to describe x E and f E F, and len()") denotes the size of the circuit required to compute (x). Yet, low online estimation error can be achieved by inefficient algorithm, and there exist efficient offline estimators with Bon = 0 as well. The result is essentially a corollary of Blum [14]; we refer to Appenddi H.I for the detailed proof. We mention in passing that Hazan and Koren [33 also give lower bounds against reducing online learning to offline oracles, but in a somewhat different computational model; see (Appendix for detailed discussion.
Theorem 4.T is slightly disappointing. since one of the main motivations for studying oracle-efficiency to leverage offline estimators as a computational primitive. Combined with our results in Section 3 informatiionthhoreticllll not possible to achieve this computationally, Nonetheless, we are optimistic that our abstraction can (i) aid in designing computationally efficient algorithms for learning settings beyond online estimation, and (ii) continue serve tool to formalize lower bounds against natural classes of algorithms, as we have done here; see [Section 3 for further discussion.
Theorem 4. shows that even though it is possible to be oracle-efficient
Remark 4.1. Theorem 4. relies on the fact that the offline estimator may be improper (i... f & F). An interesting open problem is whether can attain poly (log Fa))).of online estimation error with runtime poly (log| FFll) given a proper offline estimation oracle with parameter Bonl = O.
4.2 Conditional density estimation: computationally efficient algorithms
In spite of the negative result in the prequel, which shows that efficient computation in the OEOE framework is not possible in general, we can provide more fine-grained perspective on the com- putational complexity of oracle-efficient estimation for the problem conditional density estimation, general task which subsumes classification and regression, and has immediate applications to
reinforcement learning and interactive decision making (27 28).
Conditional density estimation. Recall that conditional density estimation [1] 12 is the special
case of the online estimation framework in SSection I in which X' and y are arbitrary, 2 = A()). and the kernel is K[2z) = z; that is sampling y ~ (x)) is equivalent to sampling y ~ f (x)) For
"See. e.g., Arora and Barak o. Definition 6.1
10e precise computational model for runtime under consideration (e.g.. Turing machines Boolean circuits)
does not change the nature of our results.
"Existence of nay functions is a standard and widely believed complexity thhorrttic assumptions, which forms the basis of modern cryptography (30).
8

---

the loss, we squared Hellinger distance: with online estimation eтror given by Est. (T) EI ((()))
-VF (f(())))
Base algorithm. Our result is based on a reduction. We assume access to base algorithm ACDE for online estimation in the Conditional Density Estimation (CDE) framework, which is unrestricted in the sense that it is not necessarily oracle-efficient. That is, ACDE can directly use the full data stream (x', y²) j')..... "e"-".""' at step t. For parameters ACDE (T) and CF 2 1, we assume that for any f E F, the base algorithm ACDE ensures that for all ó E (0, e ") with probability at least 1 - 8.
(T) REDE (T) Cr (1)6)
We define the total runtime for ACDE across all rounds as (ime(F.1)
Main result. Our main result shows that any algorithm ACDE satisfying the guarantee above can be transformed into an oracle-efficient algorithm with only polynomial blowup in runtime. For technical reasons, we assume that V := e V SUPS,E"F, bounds scale only logarithmically with respect this parameter. In addition, we assume that all I F and x E X' have 0(1) description length, and that one can sample y ~ f((x) in O(1) time.
f(ulz) is bounded; our sample complexity
(l(g
Theorem 4.2. Let ACDE be an arbitrary (unrestricted)) online estimation algorithm that satisfies Eg. (4) and has runtime Time(F. T).. Then for any N E N, there exists an oracle- efficeen online estimation algorithm that achieves estimation error
IE Est (T)] s LO(Cr log VogV BONTIN + N. (RCDE(T) log
the offline estima- tion parameter. The distributions ,²....,,'* ......" produced by the algorithm have support size poly(log [FI. X1,
with runtime poly (oly(TTmeeFF, (F,T), log log (x,,T), where 3on >
As a special case, if the online estimation guarantee for the base algorithum holds with RCDE(T) I log for some problem-depenue constant C²z 2 1, then by choosing N appropriately, we achieve E EstH log Ō(Cr((T + -(²)) (o2g TT/2 (Cx + Cz)
On (T)) s
Note that the estimation error bound in Theorem 4.2 is sublinear whenever the rate REDE (T') is. This implies that for squared Hellinger distance, online estimation in the OEOE framework is no harder computationally than online estimation with arbitrary, unrestricted algorithms.
The proof of Theorem 4.2 is algorithmic. and is based on several layers of reductions. The main reason why the result is specialized to conditional density estimation is follows: If we have an estimator f for which D (f(x), f*x)) is small for some x, we can simulate y (xx up low statistical error by sampling y ~ f(x) instead, as f and f are close in distribution. This allows us to implement a scheme based on simulating outcomes and feeding them to the base algorithm.
5 Discussion
Our work introduces the Oracle-Ffficient Online Estimation protocol as an information-theorees framework to study the relative power of online and offline estimators and gives a nearly complete characterization of the statistical and computational complexity of learning in this framework. In of oracle-efficiency.
what follows, we discuss broader implications for our information-abstrann
Oracle-efficient learning as general framework for analysis of algorithms. One of the most important contributions of this work is to formalize oracle-efficient algorithms mappings that act upon a sequence of estimators but do not directly act on historical outcomes. While the computa- tional lower bounds we provide for oracle-efficient learning are somewhat disappointing, we are optimistic that-simill to statistical query complexity in TCS and information- based complexity in optimization-our abstraction can (i) aid in designing computationally efficient algorithms for learning settings beyond online estimation, and (ii) continue to serve tool to formalize lower bounds against natural classes of algorithms, for estimation and beyond. That is, we envision oracle- efficient learning a more general framework to study oracle-based algorithms in any type of interactive learning problem. We remark that one need not restrict to offline oracles; is natural to study oracle-efficient algorithms based on online estimation oracles or other types of oracles through our information -thooretcc abstraction as well. For concreteness, us mention a couple of natural settings where our cetemaattoo-thooreeee abstraction can be applied.

---

Oracle-efficieed interactive decision making. For interactive decision making problems like bandits and reinforcement learning (more broadly, the DMSO framework described in Amppeediix E.T). I. is natural to formalize oracle-efficent algorithms as algorithms that do not directly observe rewards (handits) or trajectories (reinforcement learning), and instead must select their decision based on an (online or offline) estimator (e.g., regression for bandits or conditional density estimation for RL.. Fos- ter and Rakhlin [25] and Foster et al. [27] et seq. provide algorithms with this property for contextual bandits and RL, respectively, but the power of offline oracles in this context is not well understood.
Oracle-efficiett active learning. For active learning, it natural consider algorithms that decide whether to query the label for point in an oracle-efficient fashion (e.g., Krishnamurthy et al. [39].. For concreteness, consider pool- active learning [3].. Suppose the learner given a pool P = { .... J } of covariates and a parameter space F. The learner can repeatedly choose x' E P
and call the offline oracle to obtain an estimator / such E Off (1) == Eist (FF(), S Sor (in contrast to an unrestricted algorithm that observes y' = (x*)). The aim to learn hypothesis with low classification error using the smallest number of queries possible. Can design oracle-efficient algorithms that do so with optr-oltimal label complexity?
5.1 Further Directions
We close with some additional directions for future research.
Refined notions of estimation oracles. This work considers generic offline estimation algorithms that satisfy the statistical guarantee in DnDennnitin 1.1 but can otherwise be arbitrary. Understanding the power of offline estimators that satisty more refined (e.g.. problem-dependeee guarantees is an interesting direction for future research.
Open questions for proper versus improper learning. Our results leave some interesting gaps in the power proper versus improper oracles. First, the computational lower bounds in o[Section 4. leave open the possibility of attaining poly(log [M,,l) o(T") online estimation error with runtime poly [M,]]) given access to a proper offline estimation oracle with parameter forr = O. Second, results in Section 3.2 leave open the possibility of bypassing the D (Som + 1)) lower bound for memoryless algorithms under the assumption that the offline oracle proper.
Acknowledgments and Disclosure of F Funding
We acknowledge support from the ARO through award W911NF-21-1-0325 and from the Simons Foundation and the NSF through award DMS-203 201S-2038883.
References
[1] Anders Aamand, Justin Y Chen, Huy L Nguyen, and Sandeep Silwal. Improved space bounds for learning with experts. arXiv preprint arXiv:2303.01455 2023.
[2] Alekh Agarwal, Peter L Bartlett, Pradeep Ravikumar, and Martin Wainwright. Information- theoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory. 5(58):3235 3249,,3329,4- 2012.
[3] Aizerman, Braverman, and Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning. Automation and remote control, 25:821-837, 1964.
[4] Philip Amortila, Dylan J Foster, Nan Jiang, Ayush Sekhari, and Tengyang Xie. Harness- ing density ratios for online reinforcement learning. International Conference on Learning Representations (ICLR), 2024.
[5] Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Wood- worth. Lower bounds for non-conver stochastic optimization. Mathematical Programming, 199 (1-2):165-214, 2023.
[6] Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cambridge University Press, 2009.
[7] Idan Attias, Steve Hanneke, Alkis Kalavasis, Amin Karbasi, and Grigoris Velegkas. Op- timal learners for realizable regression: Pac learning and online learning. arXiv preprint arXiv:2307.03848 2023.
10

---

[8] Jean- Yves Audibert. Fast learning rates in statistical inference through aggregation. The Annals of Statistics, 159-1165, 2009.
[9] Andrew R Barron. Are bayes rules consistent in information? In Open problems in communica- fion and computation, pages 85-91. Springer, 1987.
[10] Shai Ben-David, Dávid Pál, and Shai Chalev-Shwarke Agnostic online learning. COLT, volume 3, page 1, 2009.
[11] Blair Bilodeau, Dylan J Foster, and Daniel Roy. Tight bounds minimax regret under logarithmic loss via self-concordance. In International Conference on Machine Learning, 2020.
[12] Blair Bilodcau, Dylan Foster, and Daniel M Roy. Minimax rates for conditional density estimation via empirical entropy. The Annals of Statistics, 51(2):62-790, 2023.
[13] Avrim Blum, Merrick Furst, Jeffrey Jackson, Michael Kearns, Yishay Mansour, and Steven Rudich. Weakly learning duf and characterizing statistical query learning using fourier analysis. In Proceedings of the twenty-sixth annual ACM symposium on Theory of computing, pages 253-262, 1994.
[14] Avrim L Blum. Separating distribution-fiee and mistake-bound learning models over the boolean domain. SIAM Journal on Computing, 23(5):990-1000 1000, 1994.
[15] Mark Bun. A computational separation between private learning and online learning. Advances in Neural Information Processing Systems, 1:20032-207433 2020.
[16] Olivier Catoni. The mixture approach to universal model selection. 1997.
[17] Nicolo Cesa- Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university press, 2006.
[18] Nicolo Cesa-Bianchi, Alex Conconi, and Claudio Gentile. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory, 50(9):2050-2057, 2004.
[19] Constantinos Daskalakis and Vasilis Syrgkanis. Learning in auctions: Regret is hard, envy casy. In 2016 ieee 57th annual symposium on foundations of computer science (focs), pages 219-228, IEEE, 2016.
[20] Miroslav Dudík, Nika Haghtalab, Haipeng Luo, Robert Schapire, Vasilis Syrgkanis, and Jennifer Wortman Vaughan. Oracle-efficient online learning and auction design. Journal of the ACM (JACM), 67()):1-57,2000 2020.
[21] Vitaly Feldman. A complete characterization of statistical query learning with applications to evolvability, Journal of Computer and System Sciences, 78(5):144411459, 2012.
[22] Vitaly Feldman. A general characterization of the statistical query complexity. In Conference on Learning Theory, pages 785-830. PMLR, 2017.
[23] Vitaly Feldman, Cristobal Guzman, and Santosh Vempala. Statistical query algorithms for mean vector estimation and stochastic convex optimization. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms,, pages 1265-1277 SIAM, 2017.
[24] Dylan J Foster and Akshay Krishnamurthy. Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. Neural Information Processing Systems (NeurIPS), 2021.
[25] Dylan Foster and Alexander Rakhlin. Beyond UCB:
Optimal and efficient contextual bandits with regression oracles. International Conference on Machine Learning (ICML), 2020.
[26] Dylan Foster and Alexander Rakhlin. Foundations of reinforcement learning and interactive decision making. arXiv preprint arXiv:2312.16730, 2023.
[27] Dylan Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13448 2021.

---

[28] Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. Conference on Learning Theory (COLT), 2023.
[29] Dylan Foster, Noah Golowich, Jian Qian, Alexander Rakhlin, and Ayush Sekhari. Model-free reinforcement learning with the decision-estimatee coefficient. In Thirth- Conference on Neural Information Processing Systems, 2023.
[30] Oded Goldreich et al. Foundations of cryptography a primer. Foundations and Trends@ in Theoretical Computer Science, 1(1):1-116, 2005.
[31] Alon Gonen, Elad Hazan, and Shay Moran. Private learning implies online learning: An efficient reduction. Advances in Neural Information Processing Systems, 32, 2019.
[32] Steve Hanneke. Theory of disagreement-basee active learning. Foundations and Trends@ in Machine Learning, 7(2-3):131-33) 2014.
[33] Elad Hazan and Tomer Koren. The computational power of optimization in online learning. In Proceedings of the forty- eighth annual ACM symposium on Theory of Computing, pages 128 148-141,20 2016.
[34] TS Jayram. Hellinger strikes back: note on the multi-party information complexity of and. In International Workshop on Approximation Algorithms for Combinatorial Optimization, pages 562-573. Springer, 2009.
[35] Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback.
In International Conference Machine Learning, pages 1453-1461. PMLR, 2013.
[36] Anatoli Juditsky, Philippe Rigollet, and Alexandre B Tsybakov. Learning by mirror averaging. The Annals of Statistics, 36(5):2183-2200 2008.
[37] Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM), 45(6):983-1066, 1998.
[38] Michael Kearns, Ming Li, Leonard Pitt, and Leslie Valiant. On the learnability of boolean formulae. Proceedings of the nineteenth annual ACM symposium on Theory of computing, pages 285-295, 1987.
[39] Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning, pages 1924-1924, 2017.
[40] Erich L Lehmann and George Casella. Theory point estimation. Springer Science & Business Media, 2006.
[41] Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine learning, 2(4):285-318, 1988.
[42] Jon Christian Mesterharm. Improving on-line learning. Rutgers The State University of New Jersey, School of Graduate Studies, 2007.
[43] Arkadii Nemirovski, David Borisovich Yudin, and Edgar Ronald Dawson. Problem complexity and method efficiency in optimization. 1983.
[44] Binghui Peng and Aviad Rubinstein. Near optimal memory-regret tradeoff for online learning. arXiv preprint arXiv:2303.01673 2023.
[45] Binghui Peng and Fred Zhang, Online prediction in sub-linear space. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1611-1634, SIAM, 2023.
[46] Yury Polyanskiy and Yihong Wu. Lecture notes on information theory. 2014.
[47] Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. Advances in neural information processing systems, 28, 2015.
12

---

[48] Maxim Raginsky and Alexander Rakhlin. Information-basee complexity, feedback and dynam- ics in convex programming. IEEE Transactions on Information Theory, 57(10)::036-7055 2011.
[49] Alexander Rakhlin and Karthik Sridharan. Statistical learning and sequential prediction, 2012. Available at http:://www..... wwt.
[50] Alexander Rakhlin and Karthik Sridharan. Statistical learning theory and sequential prediction. Lecture Notes in University of Pennsyvania, 44, 2012.
[51] Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression. In Conference on Learning Theory, 2014.
[52] Alexander Rakhlin and Karthik Sridharan. Online nonparametric regression with general loss functions. arXiv preprint arXiv: 0w115551105590 2015.
[53] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Random averages, combinatorial parameters, and learnability. Advances Neural Information Processing Systems 23,, pages 1984-1992, 2010.
[54] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning via sequential complexities. Journal of Machine Learning Research, 2014.
[55] Stephane Ross and J Andrew Bagnell. Reinforcement and imitation learning via interactive no-regret learning. arXiv preprint arXiv:1406.5977 2014.
[56] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[57] Shai Shalev-Shwarzz and Shai Ben-David. Understanding machine learning: From theory to algorithmss Cambridge university press, 2014.
[58] David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 2021.
[59] Vaidehi Srinivas, David P Woodruff, Ziyu Xu, and Samson Zhou Memory bounds for the experts problem. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 1158 1171, 2022.
[60] Joseph F Traub, Grzegorz W Wasilkowski, and Henryk Wožniakowski. Information- complexity. 1988.
[61] Alexandre B Tsybakow. Optimal rates of aggregation. In Learning Theory and Kernel Machines, pages 303-313. Springer, 2003.
[62] Alexandre B Tsybakov. Introduction to Nonparametric Estimation. Springer Publishing Company, Incorporated, 2008.
[63] Leslie Valiant. theory of the learnable. Communications of the ACM, 134114113 1984.
[64] A. van de Geer. Empirical Processes M-Estimation. Cambridge University Press, 2000.
[65] Vladimir Vovk. Competitive on-line linear regression. In NIPS 197: Proceedings of the
1997 conference Advances in neural information processing systems 10, pages 364-370, Cambridge, MA, USA, 1998. MIT Press.
[66] Martin J Wainwright. dime-diies statistics: non-asymptotic viewpoint, volume 48.
Cambridge University Press, 2019.
[67] Marcelo J Weinberger and Erik Ordentlich. delayed prediction of individual sequences.
IEEE Transactions on Information Theory, 48(7):1959-1976 2002.
13

---

[68] David P Woodruff, Fred Zhang, and Samson Zhou. Streaming algorithms for learning with experts: Deterministic versus robust. arXiv preprint arXiv:2333.07009 2023.
[69] Tengyang Xie, Dylan Foster, Yu Bai, Nan Jiang, and Sham Kakade. The role of coverage in online reinforcement learning. International Conference Learning Representations (ICLR), 2023.
[70] Yunbei Xu and Assaf Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07886 2020.
[71] Yuhong Yang and Andrew Barron. Information-thhoretie determination of minimax rates of convergence. Annals of Statistics, pages 1564-1599, 1999.
14

---

Contents of Appendix
I
Additional discussion and examples
16
Additional notation
16
Additional related work
16
Examples of estimation problems and loss functions C.I Examples of offline oracles
17
18
II Omitted results
20
General reductions for oracle-efficient online estimation
20
D.I Characterization of oracle-efficient learnability for classification
22
D.2 Additional lower and upper bounds for memoryless oracle-efficient algorithms...
22
E Application to interactive decision making
23
E.1 Offline oracle-efficient algorithms for interactive decision making
23
E.2 Bypassing impossibility of memoryless algorithms via coverability
25
III Proofs
28
Technical tools
28
Proofs from Section 3
29
G.1 Proofs from Section 3.1
29
G.2 Proofs from Section 3.2
32
G.3 Proofs from Appendix D
36
H Proofs from Section 4
40
H.I Proofs from Section 4.1
40
H.2
Proofs from Section 4.2
41
I Proofs from Appendix
48
15

---

Computational lower bounds for online learning, Beyond Blum 14]. another work that considers computational lower bounds for online learning is Hazan and Koren [33. This work proves lower bounds for online learning in a model where the learner has access to an ERM oracle that can Their lower bound does not Part I
Additional discussion and examples
A
Additional notation
We denote Rzo = 00, 20). For any a, E R. LaAbb- : min (a, and vb:= max (a,b). b). For any For any set X, A(X) is the space of all distributions on X. For any integer T, the notation ~²' will be the shorthand notation for the sequence x' , For any real number x E IR, denote by [æ] the largest integer that is smaller than or equal to x and by [æ] the smallest integer that is greater than equal x. The indicator function is denoted by I(.). We define 0(.). ())). o(), (c), O(),, Q(.). (-) following standard non-asymptotic big-oh notation. We use the binary relation S y to indicate that S O(y).
integer N > 0, [N] = (I....,NN N... .
x²,
B Additional related work
In this section we discuss related work not already covered in detail.
minimize the training loss for an arbitrary dataset (z", ........ (x", y²"). fit in our computational model due to details around the way description length is formalized. In particular, the main focus of 3 is obtain a lower bound on the number of oracle calls any online learning algorithm must make to an ERM oracle.
33 consider a sequence of classification instances with X₂ = {0.1) 1)" and classes F, of the size of ((22/Xx.)) and show that any online learning algorithm requires 22( (2TT) oracle calls to achieve low regret for this class. However, the estimators f E F, returned by the oracle in their construction have Q( / (Xx,)) description length themselves, meaning that they do not satisfy the poly (n)-(denrriiptti length required by the model described in [Section (in other words, the result is not meaningful as a lower bound runtime, because simply reading in the output of the offline oracle takes exponential time). For completeness, restate the example proposed by Hazan and Koren 33 Theorem 22] in our framework below.
Similar to the setup for Theorem 4.1, Hazan and Koren
Hard case from (33). For any integer n 2 1, consider binary classification problem with X (0, 1) 2m 2 = y (0, D Do/1 and KC==) 1,, Let N 22n, and let be the collection of all sets S = {.,,..., 82n} C {0, 2m where {0. 1)a 2m is also treated as the integer set of {0. ..... 22n 1 in left-to-right order and s; E (2 - 1),... 2" 1} for each 1,...,2" 2".
We define a class Fen = {fs, S S SES, S 2"), where
fs,( = = {
if E and r 2 ST,
otherwise.
For this class, we reduce back to the Thcorem 22 of Hazan and Koren 33 which states that any algorithm with runtime o(vN) has to suffer online estimation error at least 1/2 for all S 1 S 2". The issue with this example for our computation model is that |F2a 222v/1x..)) Any sufficient description for this parameter space in bit strings (or, e.g., boolean circuits) will scale with 2 ((x())n)) Thus, the description length required to return f is too large (already larger than the lower bound obtained).
Online learning with memory constraints, A number of recent works focus on memory- regrett tradeoffs in online learning (59 45
1 68 Here, the learner can observe the full data stream (x+ y'),,.., (x", ")²", but is constrained to B bits of memory. This framework is incomparable to the OEOE framework, but it would be interesting to explore whether there are deeper connections (e.g.. any memoryless OEOE algorithm inherently has memory no larger than that of the offline oracle).
16

---

Gaps between offline and online. A long line of work aims to characterize the optimal regret for online learning, developing complexity measures (Littlestone dimension, sequential Rademacher complexity) that parallel classical complexity measures like VC dimension and Rademacher com- plexity for offline learning and estimation TO 53 54 7 is well known that in general, the optimal rates for online learning can be significantly worse than those for offline learning. Our work primarily focuses on finite classes F, where there is no gap, but for infinite classes, any conversion from offline to online estimation will inevitably lead to a loss in the estimation error rate that scales with appropriate complexity measures for online learning (cf. Appendix D.I.
Other restricted computational models. Our information iiformationn-hhoorttic formulation of oracle- efficiency is inspired by statistical query complexity in theoretical computer science and information complexity in optimization, both of which can be viewed restricted computational models with an information-theoreti flavor. The statistical query model a framework in which the learner can only access the environment through an oracle that outputs noise estimates ("statistical queries") for a target parameter of interest (13 37 21H233. Information complexity in optimization is model in which algorithms can only access the parameter of interest through (potentially noisy) local queries to gradients or other information
43
60
48
S
C Examples of estimation problems and loss functions
In what follows, we give detailed background on three canonical examples of the general estimation framework discussed in [Section I density estimation.
Binary classification, square loss regression, and conditional
Classification [38
63
For binary classification, we take y {0, 1} with the binary loss
(1,1 22) = (=1 * 22) for 21, 2 and kernel K(z) 1,, which is noiseless. The binary loss metric-like with Co 1.
For offline estimation, observe that with covariates z'....,,"" xx and outcomes y' = fx(x') for all I E (I,..., J) any empirical risk minimizer ʃ 'that sets f(x²) = y² obtains
T Do/1 (f(x", = 0.
(5)
[=1
For online estimation, the halving algorithm
[7 achieves
T Σ Dons (P())) s log((FF).
o (T)
(6)
We mention in passing that another natural classification setting we do not explore in detail in this paper is noisy classification, where the setting is as above, except that we set (x)( = () and take (2bs 22) = 21 žg| as the absolute loss for all æ1, 22 E Z.
[0. 1].
Square loss regression 62 (66) For real-valued regression, we take 2= = with the square loss (21, 22) = (=) )2 for 21, E 2 and the kernel (C){(x)) = =N(S((x) (x), another
subGaussian distribution. Note that the square loss a metric-like loss with CD
For offline estimation, with covariates r',...,,² and outcomes y² ~ f((x)) + e" for all t E T (f=((((')) y²)
{1....,T, T), the classical Empirical Risk Minimization (ERM) f := arg minjeF
gives
Est Off (T) = Σ Dag (Ao)) < log((F(5~1), sq
1=1
with probability at least - 8 (cf. Lemma C.lloo..).
For online estimation, the exponential weights algorithm [7. with decision space F and the loss at each round chosen to be er (f) (f(z*) - y²)² achieves
T
Est On (T) = Σ Dza (F(r)) < log((F((-1)).
(8)
with probability at least - 8 (cf. Foster and Rakhlin (25 for proof).
17

---

For conditional density estimation, we consider an arbitrary outcome space and take A)) with squared Hellinger distance D given by 12
Conditional density estimation [2]
1 (V -) x) dy.
D(((()) =
(9)
and K[()) = for all & 2. Note that squared Hellinger distance is metric-like loss with Cp 2. .."
For offline estimation, with covariates x² ;-"""""" and outcomes (x²' for all / (I.... T), the classical Maximum Likelihood Estimator (MLE) f == arg max Σ log fly x²) gives
EstH
Off (T)
Σ D (fa))) s log((F()~1), 1=1
(10)
with probability at least - 8 (cf. Lemma C.2pelow).
For online estimation, the exponential weights algorithm 17, with decision space F and the loss at each round chosen be (f) - llo f'vv æ²), achieves
T
Est.
(o(@n(T (T)
Σ D (P('), s log((F(5~1),
(11)
with probability at least 5 (cf. Foster et al. for proof).
C.1 Examples of offline oracles
For completeness, below prove the offline estimation (fixed design) guarantees for square loss empirical risk minimization and maximum likelihood estimation mentioned in the prequel.
Empirical risk minimization for square loss regression. For any square loss regression instance (X 2, F) defined as Appendix C defined as
The Empirical Risk Minimizer (ERM) estimator is
arg min fet
E (f(x')- y')2
We have the following bounds the offline estimation error for the squared loss. Lemma C.1. For any target parameter f* F, with probability least 1 6, we have,
[(())-)())) s 8log(1F1/0).
Proof Lemma C.1
Observe that the ERR estimator satisfies
(f 1=1
f(x)))
-)) fxx)))
-
1=1
< Σ ((x'))
(12)
where the last inequality is by the definition of f. and ;= distribution for all & [T]. Then by the Gaussian tail bound, we have with probability at least
: y---'*''"" f*(x') is a standard normal
T 1(W))-
s 2 -((x)~
1=1
More generally, if is common dominating measure, then D (2, = S dv, where
JF di du and de are the choice of v.
derivatives. The notation in Eq. (9) reflects that this quantity is invariant under
18

---

Plug the above inequality back into Eq. (12) and reorganize, we obtain the desired bound of
E = 88100(FF100).
Maximum likelihood estimation for conditional density estimation. For any conditional density estimation instance (X' 2, F) defined as Appendix C (MLE) is defined as
The Maximum Likelihood Estimator
T
f = arg max JEF
log f x²).
We have the following bounds on the offline estimation error for squared Hellinger distance. Lemma C.2. For any target parameter f E F, with probability at least 1 - 6, we have,
Σ
Di (f()), S log((F1/6).
1=1
Proof of Lemma C.2
For any parameter f E F, define
Z. 2 (log (1' x²) log f(y |x)))
Then for any f F, by Lemma A.4 of Foster et with probability at least -6/1F), we have,
T
f(y x²) (t
2 (log f* (v' x²) log f(y" xx)) log
E
V
xt)
+ log((F1/6).
1=1
We further have by the inequality of log(1 + x)) I x that
f(y- x²) (2' x4)
fly x²) fryy xx)
log
E
E
[V
Since 3' (*(x), have by standard calculus and the definition of the squared Hellinger distance that
fly x²) (1' xxt)
(R((((("))
Altogether, we have obtained for any f E F, with probability least - -5/1F]
T 1 マ (log I (y' xx') - logglo/v f(y xx))) < (D(("(")) log F1/8).
T
Σ
Thus, by union bound, the above inequality holds for all f E F with probability at least 1 - 5. Thus the MLE f satisfies
T
T
Σ Da (M()),)))))
1
(log (" xx²) log J(v' xx + log((Fl/6))
2
1=1
Fl18).
where the second inequality is by the defintion MLE.
19

---

Algorithm 2 Reduction from OEOE to Online Learning with Delayed Feedback
1: input: Offline estimation oracle Algon with parameter Bow 2 O, delay parameter N E N, delayed online learning algorithm ADOL for class F.
2: for t = 1, 2, T do
3: Receive f = Algon (z'...., ,"....,"""1.... ........
4: if then
5: Let fi-N == E f.
i=1-N+1
6: Let e (s) D (i-M(x* (xt-M) 8), f( m1) and pass 8 (-)x ADOL the delayed feed-
back.
7: Let μ = Abou (l', ..., ( be the delayed online learner's prediction distribution.
8: Predict with f je and receive x'.
Part II
Omitted results
D
General reductions for oracle-efficien online estimation
The oracle-efficient online estimation algorithm in Section 3.1 Algorithm 1, is somewhat specialized finite classes. In this section, provide a more general approach to designing oracle-efficient algorithms based on delayed online learning, and use it to derive characterization of oracle-efficient learnability for classification with infinite classes F.
For the results in this section, we assume that and are convex, which covers regression and conditional density estimation; variants of our result for classification given in Appendiix G.3)
Online learning with delayed feedback. Before introducing our algorithm, we first introduce an 47). In framework, the learner is given class F C 2 Their goal is to choose a sequence of parameters f') f....,f"' that minimizes regret against the class F for an adversarially chosen sequence loss functions Ct ......, LT, with the twist being that the loss ! is not revealed immediately at step 6, and instead becomes available at step ++ N for delay parameter N N.
abstract delayed online learning framework 67
42 35
In more detail, the interaction between the learner and the environment proceeds as follows:
For t = I....,::
The learner picks fr ~ ~ E A(22*).
Learner incurs loss (f()) and adversary reveals loss function (C-N 2
[0, 1] for round
t- N tif < N + nothing is revealed).
The goal of the learner minimize regret in the sense that
T
T min Σ (()) JEF t=1
Rpol ( (T,N,7) = Σ Fr [C(F)] -
t=1
is small, where 7 2 1 is parameter. For 1, this definition coincides with the standard notion of regret online learning (e.g., Cesa-Bianchi and Lugosi 17) but allowing for y will prove useful for our technical results.
Algorithm and online estimation error bound. Algorithm 2 describes our reduction from oracle- efficient online estimation to delayed online learning. In addition to offline oracle Algor the algorithm takes as input delay parameter N N and delayed online learning algorithm Apol for the class JF (Algorithm does not explicitly take the class JF as argument, the algorithm only implicitly makes use of F through Apou).
The basic premise behind Algorithm is that for any sequence of consistent offline estimators, we can average to improve the predictions. Consider step [T]. Suppose ft f.....,"' are produced by
20

---

offline oracle Algor with parameter Sorl 2 O, where we augment the sequence by setting f f for all N. Then we can use an argument based on convexity (cf. proof of Theorem D.I) to show that for any N E N, the averaged parameters
(13)
1=1+
satisfies
T-N L+N
Eo(F(s))((a))
S SN+
1
Σ Σ
((P(E))(()))
N
i=#+1
T
D (f()). SN + SoRT/N.
= N +
N
In particular, we increase N, the quality of the predictions increases, and we achieve sublinear estimation eиror soon as N = (T)). Of course, the catch here is that f depends the predictions of future estimators, and cannot be computed at step I. However, f can be computed at step (++ N + with a delay of N. This leads us to appeal to delayed online learning. In particular, at each step t t N + 1, Algorithm proceeds as follows. Using the new offline estimator f from Algon the algorithm computes the averaged estimator fー~M== Σ
corresppnndig to the estimator in imt N+1
Eq. (13) for step L NN The algorithm then defines loss function
C- (f) (F"""
and feeds it into the delayed online learning algorithm ADOL the feedback for step t N. Finally, Algorithm uses the prediction distribution μ produced by ADOL to sample the final estimator f*, then proceeds the next step. Our main theorem shows that long ADOL achieves low regret for delayed online learning, this strategy leads to low online estimation error.
Theorem D.1 (Reduction from oracle-efficient online estimation to delayed online learning). Let D be any convex, metric-like loss. Suppose we run Algorithm 2) with delay parameter N E N and delayed online learning algorithm for the class F. Then for all 1, Algorithm 2 ensures that
E (Esto S O(CDY(N B/NT/N) (TDLLT,,,,))
(14)
with any offline oracle Algon with parameter Bofl 2 0, where
RooL (T,
EB [e(st]] -7 tml
min FEF
Σ en)
(15)
is the regret of ADOL for the sequence losses constructed in Algorithm
The parameter N controls sort of birs-variance tradeoff in Theorem D.I The first term in Eq. (14) (corresponding to the bias of the averaged estimators) decreases with the delay N, while the second term (corresponding to the regret of ApoL) increases; the optimal choice for N will balance these terms. To make this concrete, we revisit finite classes as a warmup.
Example: Finite classes. Delayed online learning is well-studied, and optimal algorithms known for many classes of interest [67 42 35 47]. The following standard result (a proof is given in Appendix G.3 for completeness) gives delayed regret bound for arbitrary finite classes.
Lemma D.1. Consider the delayed online learning setting with a delay parameter N. There exists an algorithm that achieves
R (T,N2))
Σ E Bp [C())) - 2 min FEF t=
( < 2N log F
1=1
for any sequences of losses C. e....., le E [0.
21

---

Combining Theorem D.1 with Lemma D.I we can obtain the following upper bound for oracle- efficient online estimation.
Corollary D.1 (Oracle-efficient online estimation for finite classes via delayed online learning). Consider an arbilrary instance (X, Y/ F) and metric-like loss D, and assume 2 is convex. By choosing ADOL as in Lemma D. Algorithm ensures that for any N 2 1,
E Est [ S O(CD(N + Bo#T/N) + N log FF).
with any offline oracle Algon with parameter Son-
By choosing N =
V
Cp To Co+log LAT
1, Corollary
D.1
gives an upper bound of
o( (( CVDooorCCa + log(F)) T + log (FI). While the rate in Corollary D.1 is worse than The-
orem 3.1 in terms of dependence T, the reduction has two advantages: (1) does require a-priori
knowledge of the offline estimation parameter 3on: If we choose N = / 1o--ogg27 [2
Cp-T V 1, Corol
lary D.I obtains O( + 1)(CCb(Ca + log]FF) T + CD + log (FI); (2) The reduction by
Algorithm 2 is more flexible, and allows for guarantees beyond finite classes, as we now illustrate.
D.1 Characterization of oracle-efficient learnability for classification
As an application Algorithm 2 and Theorem D.I we give a characterization for oracle-efficient learnability in the OEOE framework. To state the result, we define Ldim(F) as the Littlestone dimension for a binary function class (e.g., Ben-David et al. [10]
Theorem D.2 (Characterization of oracle-efficient learnability for binary classification), Consider a For any class F and for 2 0, there exists an oracle-efficient algorithm that achieves online estimation error O( V 3on Ldim(F) T log T + Ldim(F) log T). On the other hand, in the worst-case any algorithm must suffer at least ((Ldiim(F) online estimation error.
hinary classification instance (X, Y' Z, F) with 2 = (0, 1}, D Do/1 and K(=)= =1 1-
The main idea behind this result is to show that we can create a delaved online learner for the reduction in Algorithm that achieves low regret for Littlestone classes [3]
D.2 Additional lower and upper bounds for memoryless oracle-efficient algorithms
The proof of Theorem 3.3 uses that the estimators f produced by the offline estimation oracle may be improper (i.e., J E F). We next provide variant of the result that holds even if the estimation oracle is proper, under the additional assumptions that (i) the learner is itself proper in the sense that μ²¹ E ((F), and (ii) the learner is time-invariant (i.e., the learner sets μ F(F*) for all t).
Theorem 3.3' (Impossibility memoryless algorithms for OEOE; proper variant). Consider the binary classification setting with Z = y = {0, }} and loss Da/1(t.)). ). For any N E N and Son 2 O, there exists an instance (X,Y, y, K,K.F) with [F] = X N such that for any memoryless oracle- efficient algorithm that is (i) proper, and (ii) time-invariant, there exists a sequence of covariates (x+ ......") and proper offline oracle Algon with parameter Boh such that T.. [ESt (T)) (T)) Z Q(min (Bon + 1),T}).
A complementary upper bound. For completeness, we conclude by showing that the (large) lower bound Theorem 3.3 can be achieved with a memoryless oracle-efficient algorithm. We consider the "trivial" algorithm that outputs the estimators produced by the offline oracle as-is.
Proposition D.1 (Upper bound for memoryless OEOE). For any instance (N,),2KK,T) metric-like loss D, and offline oracle Algon with parameter 3on. the algorithm that returns On y² has online estimation error Est (T) s
ft ft := Algon(x' x y" .'....."" .-"))""")) O((3on log
Formally, to handle the fact that z = {0, 1} is not convex, this result requires slight modification to Line 4 in Algorithm 2 that replaces the average with majority vote. See the proof for details.
22

---

E Application to interactive decision making
In this section, we apply our techniques for oracle-efficient online estimation to the Decision Making with Structured Observations (DMSO) framework for interactive decision making introduced by [27] First, in Appendix E.T we use our reductions to provide offline oracle-efficient algorithms for interactive decision making. Then, Anppeediix E.2 we focus on reinforcement learning and show that it is possible to bypass the impossibility results for memoryless oracle-efficient algorithms (Theorem 3.3) for instances (X, Z, K. FF) corresponding to Markov decision processes that satisfy a structural property known as coverability.
E.1 Offline oracle-efficient algorithms for interactive decision making
In this section, we introduce the setting of Decision Making with Structured Observations (DMSO) and the applications of our results to this setting.
Decision Making with Structured Observations (DMSO). The DMSO framework 27 captures large class of interactive decision making problems (e.g. contextual bandits and reinforcement learning). In this framework, the learner is given access to a model class M that contains an unknown true model M II AR O), where II the decision space, C R is the reward space and O is the observation space. Then the interaction between the learner and the environment proceeds in T rounds, where for each round 1,...,77
1. The learner selects decision T' E 1I.
2. Nature selects a reward r' R and observation o' O based on the decision, where the pair (r"; oo) is drawn independently from the unknown distribution M (x'). The reward and observation is then observed by the learner.
Let (x) == BM [r] denote the mean reward function and Tx == arg max rell gMd (r) denote the decision with the greatest expected reward for M. The learner's performance is evaluated in terms of regret to the optimal decision for M
T
RegDM (T)
Σ E [9 (g g" (x*)]-
(16)
where p' E (TII is the learner's 's distribution over decisions at round t.
Background: Reducing DMSO to online estimation. Any DMSO class (M, II, O) induces an instance (X, K., of the estimation framework in Section ass follows. We associate F M, X' II, y density estimation problem in which the covariates observation-rewarr pairs drawn from the underlying model (=))). In particular for sequence of
R. 2 = AOO x R.), and XCMM* (T)) = M'(x), (x). That is, we have a conditional
decisions π 11 and the outcomes are
decisions T', ..... x² and sequence of estimators M'..... M", define the online estimation emror for a loss D as
T Est (T) Σ ((M)
(17)
We refer to any algorithm Algon that ensures that Estt S 3on almost surely given access to {(T', 0r, orrrr)) r')) with (o',)") ~ M M*(*") as an online estimation oracle with parameter 8on-
Foster et al. (27 28 give an algorithm, Estimation-to to- toeccision (E2D), that provides bounds on the regret in Eq. (16) given access to an online estimation oracle Algon The algorithm is (online) oracle- efficient and memoryless, in the sense that the decision n' at each step t is measurable function of
the oracle's output M To restate their result, we define the Decision-Fstimation Coeficient for the class M as
dec, D (M, M) = inf sup Eneo gdr (TT( -7) (MM())))))
(18)
PEA(TT) MEM
for a reference model M and any losses We further define dec,, dec..(M) SUPTEM dec, (M, With this notation, the main regret bound for E2D is follows.
23

---

Algorithm 3 Estimation to Decisions Meta-Algorithm with Offline Oracles (E2D.Off)
1: parameters:
Offline estimation oracle Alo with parameter Son. Oracle-efficient online estimation algorithm Algoroe-
Exploration parameter > 0.
2:
for 2, , do
3: Compute estimate M = Algor (T', (T''...,, 00'...,0')))
4: Feed M to OEOE algorithm AlgoEoE and obtain μt:
5: Let
(grnggg" (g" gM - (
p = arg min sup E PEA(T) MEM
(20)
Sample decision x¹ ~ p' and of ~ MM(T") and feed x' to OEOE algorithm AlgoEoE 6:
Proposition E.1 (Theorem 4.3 of (27)). For any model class (M, I1, O) and metric-like loss D. any y > O, and any online estimation oracle Algon with parameter 3on > 0, the E2D algorithm ensures that
RegoM (T) <
sup dec,, D (M, (EA(M)
y 8on-
(19)
For the case of squared Hellinger distance where D = DA, the Decision-Estimation Coefficient (MM) was shown be lower bound the minimax optimal regret for any class M. Hence, Proposition shows that it is possible to achieve near-optimal regret for any interactive decision making problems whenever online estimation oracle is available. However, it was unclear whether similar results could be achieved based on offline estimation oracles.
Off).Off invokes the E2D algorithm of Foster et al. [27] with any gracle-efficien online estimation algorithm Al (which can be any of the algorithms provide, e.g. Theorems 3.1 and 4.2 along with offline estimation oracle Algon-
Making E2D offline oracle-efficient. Algorithm 3
provide offline oracle-efficient guarantees for interactive decision making. Invoking the algorithm with Version Space Averaging (via Theorem 3.1) leads to the following corollary.
Corollary E.1. Consider any DMSO class (M, O) and metric-like loss D. Algorithm 3 with exploration parameter 7 > 0 and AlgoEoE chosen to be bllooriit 1 ensures that
E Regggl < Ollog max dec (M) T, 7 (Bon + 1) 1) M
for any offline estimation oracle Algor with parameter Bon-
This result shows that cnormatiion-hhoretiiann framework with offline oracles, though the result is not computationally efficient.
it possible achieve low regret in the DMSO
As an example, in the case of square loss regression (Appendix C which is used for contextual bandits, an offline guarantee of 3on Olog [M) is achievable. Meanwhile, it is known that dec (MM) X
|Al/T [27] Thus Corollary E. achieves a bound of Ö(V AAT log [M)) with an appropriate choice of The best know regret guarantee for contextual bandit is ((VIA) log MM)
58 70). The
bound from Corollary E. matches the state-of ofate-of-the-are art result up to a factor ( M)). How to remove this suboptimality is interesting direction for future work.
Naturally, the other reductions for oracle-efficient online estimation developed in this paper can be combined with Algorithm as well. In particular, by combining Theorem 4.2 we derive the following corollary for squared Hellinger distance.
Corollary (Informal). Whenever online conditional density estimation can be performed efficiently with access to the full history, and whenever the minimax problem in Eq. (18) can be solved efficiently, there exists a computationally efficient and offline oracle-efficies efficient algorithm with near-optinn near-optimal regret in the DMSO framework.
24

---

E.2
Bypassing impossibility of memoryless algorithms via coverability
Recall that our results in Section 3.2 show that in general, it is impossible to obtain low online esti- mation error through memoryless oracle-efficient algorithms. this section, we revisit memoryless algorithms for the Markov decision processes particular type of class (M, II, Q) (or equivalently, (X,3, 2, IC, F)). We prove that for any class of Markov decision processes for which a structural parameter called coverability [69] is small, any offline estimator can be directly converted into an online estimator.
Markov decision processes. We consider classes (M, O1, that correspond
episodic finite-horizon reinforcement learning setting, following Foster et al. [27]. With H E N denoting the horizon, each model M .M specifies non-stationary Markov decision process as a tuple M { h=1: A, [Ph I h==1 ( [M
H
] 1h=1: d1}. where S₁ is the state space for layer h, A is
the action space, PA S₁ x A + (S,+1) is the probability transition kernel for layer h, R Sr. A A(R) is the reward distribution for layer and dj E ((Si) is the initial state distribution. We allow the reward distribution and transition kernel to vary across models in M and assume that the initial state distribution is fixed.
We set C. IRRSS. which denotes the set of all randomized, non-stationary policies π = (I.I,..., TH) E IRRNS, where Th Sh A(A). For fixed MDP M M and II, the observation ~ (T) is a trajectory (51, 01, T1)..... (SH, TH) that is generated through the following process, beginning from 81 d1. For h 1,..., H:
ah (sh).
7h ~ Ri (sh, an) and 88+1 (. 8h, un).
So the obseravtion space O = S x A R x ... x SH x A x R. For notational convenience, we take + to be deterministic terminal state. We [p and EM* denote the probability law and expectation over trajectories induced by (x). In addition, we define PA (. [h. as the conditional distribution on $1+1, Th given 3h; ab under M for h [H].
The guarantees provide apply any loss that has particular layer-wise structure tailored to reinforcement learning.
Definition E.1 (Layer-wise loss). For any sequence of losses (Dinn bounded by [0, 1], where D (S, x R) x A(S, R) + [0, for all h E [H],, we define the layer-wise loss DRL on ((O)
as
(M(( = Σ N'
(p
an) sh. (a))))))))
h=1
for any pair of MDPs M, M' E M and policy π E II
Examples of the layer-wise loss are scaled reverse KL-divergence (which is bounded by [0, 1] whenever the density ratios under consideration are upper and lower bounded with an appropriate scaling) (77 and the squared Bellman error 229]. Another useful example is the the sum of layer-wise squared Hellinger distances given by
Df (M(T)
= Σ A
an), pmp(("" (118h,
8h, )-
(21)
h=1
This loss coincides with the global squared Hellinger distance (T) up O(H) factor.
1h ordering of M and M' on the right- hand side of the definition is due to the following technical reason: Theorem E. only works when the expectation on the right-hand side be taken with respect M (m)). which shows up as the second argument in offline oracle guarantee (??). 15 For the results in this section. it will be useful to work with asymmetric losses, and in this case we use the notation D(- || -) instead of D...).
25

---

Coverability. We provide memoryless wracle-efficient algorithms for online estimation for any layer-wise loss D when the underlying MDP M has bounded coverability [69]
Definition E.2 (Coverability), For an MDP M and a policy n, we define d (8, a) = E"C, (Shh,, dn = 5, a)). The coverability coefficient Ceou for a policy class 11 for the MDP M is given by
inf Coov (M)) (AASAA)) (EIL)E[H]]
sup
Ph.
It is immediate to see that Ceo < III but in general it can be much smaller. Examples of MDP classes with low coverability include Block MDPs, Low-Rank MDPs, and exogenous block MDPs 669 4].
Offine-to-online conversion under coverability. Our main result shows that under coverability, the outputs of any offline estimation oracle Algon satisfy online estimation guarantee as-is.
Theorem E.1 (Ofitine-to-online conversion under coverability). For any layer-wise loss DRL and MDP class (M, II,O) O) and M E M. the sequence of estimators (M*. ,...,M²) produced by any offline estimation oracle Algon for DRL with parameter Son satisfy
T Σ RL M (T) s ((())))))((CCoo/ooo (CCCoo(M*) log + (Ccov(M
This result is based on variant of the proof technique Theorem of Xie et al. [69]. An important application of the result, which can be applied in tandem with the guarantees in Foster et al. (27). concerns squared Hellinger distance.
Corollary E.2. For any MDP class (M,II,) and M E M, the sequence of estimators (M', ...., Mr) produced by any offline estimation oracle Algor for squared Hellinger distance D with parameter Bon satisfy
Est. ( = Σ (TT) S so( ( CCo Bon logg T + (M))
1=1
This result follows by using that the layer-wise squared Hellinger distance in Eq. (21) is equivalent to ( up O(H) factors.
Application to interactive decision making. We apply Theorem E. to decision making via Algorithm
Corollary E.3. Consider any layer-wise loss = D and MDP class (M, IT, and let Ceov == SuPMEM (M). Algorithm with exploration parameter > 0 and AlgoEoE chosen be the identity map ensures that
dec (M, T, 7 (HHCos/Ponloog ( + HCcoww
E RegpMl & O(log T) max sup (M)(M)
for any offline estimation oracle Algon with parameter Bon-
Contextual bandits and optimality of offline-to-online conversion. Another implication for Theorem E. concerns the special case of contextual bandits (that is, MDPs with horizon one). For the contextual bandit setting
abbreviate S St, and refer to d1 A(S) the context distribution. We define g" (s, a) = E,.nnf((sa) (s.a) [r] as the expected reward function under a model
M, and following Foster and Rakhlin (25. use the squared error between mean reward functions as divergence:
(MT, M Eyndi (-))()) Dea (g" (s, a))a
(22)
For this setting, the coverability coefficient Coov is always bounded by the number of actions [A], which leads to the following corollary.
26

---

Corollary E.4. For any contextual bandit class (M,II, and M E M, the sequence of estimators
(M', y..., M²) produced by any offline estimation oracle Algon for Dee with parameter Bon satisfy
Est on (T) = Σ DcB M (*())))))))) s OVVIA(T Bos log + [Al),
Recall that Foster and Rakhlin [25] show that any algorithm for online estimation with the divergence in Eq. (22) with Est (T) fon be lifted to contextual bandit algorithm with regret A Bon) via the inverse gap weighting strategy, even if contexts chosen adversarially. Subsequent work of Simchi-Levi and Xu (58) shows that for stochastic contexts, the inverse gap weighting strategy also yields regret O ( A (oFr) given access to an offline oracle with parameter Son. On the other hand, combining Corollary E.4]with the guarantee from Foster and Rakhlin 22 gives regret ([AA|//733// 90f 1/4. This does not recover the result from Simchi-Levi and Xu 58 but nonetheless gives alternative proof that sublinear offline estimation error suffices for sublinear regret.
The guarantee in Theorem E.1 leads to a degradation in rate from forr to vT (suppressing -eppedeed parameters). Our next result shows that this is tight in general.
Proposition E.2 (Tightness of offline-to-online conversion). For any integer T 2 and Poir > there exists a contextual bandit class (I, A O) with |A| 2, a distribution di A(S), a sequence (x², (r',,.., )²) and an offline oracle A Algon for DcB with parameter Bont such that the oracle's 's oulputs (M*. M²) satisfy
On Est (T")
Σ DcB M M' (T+)) 00 (TTB))
27

---

Part III
Proofs
F
Technical tools
Lemma F.1. For any non-increasing sequence 01 2 22 2
xT+1 2
3t J++1
log(z1).
Ft
Proof Lemma F.1 Since log(1 + a) S for all 11, for any [T'. we have
Ft+1 -1)
Ft+1 1).
-log(a//<+11) = log 1+
Summing up over t E [T], we obtain
21
T++1
T++1
log(#t/0<41) log( S log(z1).
St
t=
1=1
The following lemma gives an improvement to Lemma A.13 of Foster et al. [27] that removes logarithmic factor. This shows that up an absolute constant, squared Hellinger distance obeys one-sided version of the chain rule for KL divergence.
Lemma F.2 (Subadditivity for squared Hellinger distance), Lel (1), ( be se- quence of measurable spaces, and let X' -II N, and F 0i-15 1. For each let P- -) and Q- () be probability kernels from (X'-1 ((--)) to (X, F.). Let P and Q be the laws of X1,..., Xn under X, (P X1--1) and X, Q'( X13-1) respectively. Then holds that
D (P, 7 Ep Σ (TP X1=-1), 0'(-1X15-1))
Proof of Lemma M2
We appeal to the cut-and-paste property of 34 defining collection
of distributions indexed by hypercube {0, 1) with the property that the vertices (0,.... 0) and
(l..... 1) correspond to the distribution IP and Concretely, for any vertex 1)" of the hypercube, we define probability distribution
where R,, (111-1) (P' X1a-1) 1Q' X14-1) if
if = 0,
T II R,,, (- X1a-1),, sE(m]
and (1.....) Q. Now, consider any four vertices a, e, d E {o, 13" with the property that {a,, b,} (c,, de} for each E [n] (with {-} interpreted as multi-set.. Then for any measure v = II 14 (-Ix (11111s11 where px(-)X1;a-1) is any common dominating conditional measure TT for PP (- (111 and (-1X1x=1), ( have
Observe that (
by the definition of squared Hellinger distance we
d.a. (. (-1 dRo, (. (11s-1) du dv:
D (Pa, Bb) =1
dv:
d.R. (. (110-1) dRa, ( X1:1-1) du
du;
dv
= (PB,, Pa).
(23)
F6For example, we can take vz v-(-)Xi.)) = (P (((1xia-1))/2 for P and Q' The result is independent of the choice of v.
28

---

Let k be the maximum integer such that 2 I Then since Eq. (23) holds. by Theorem 7 of Jayram
34 applied with the pairwise disjoint collection Aj S mod 2 = = for all j [24]
where |Ayl for all j. we have
(P, Q) II
1/2))
D -
IP, Q'( (1a-1) II ( Xaw-i)) M&E
j=1
lEA,
I Ep
((1p((-)11-11;,4000 1'(- X1=-1))
To conclude, we note that I[ (1 1/2") > 2/7.
G Proofs from Section 3
G.1 Proofs from Section 3.1
Theorem 3.1 (Main upper bound for OBOE). For any instance (X, V. F)K.F), any metric-like loss D, and any offline estimator Algon with parameter Bon 2 Algorithm / is oracle-efficient and achieves
Est (IT < O(Cb (for + 1) min (log [F) X log 73).
Proof of Theorem 3.1. Our main technical result is the following lemma, which is proven in the sequel.
Lemma G.1. Consider any instance (X, V. F) and metric-like loss be the target parameter, and consider sequence of sets F = 2 F2 2 .... 2 Fr 2 (f* and x² sequence of covariates x²,,. ,.," e X with the property that for all E (7), all f E F, satisfy the
D Z. Let f E F
following offline estimation guarantee:
D(f(=") s Bor.
(24)
Then, by defining Unif(FF), have that
T
E (D(("")) s O(fon + 1) min (log [F), |x| log T}).
To invoke Lemma G.I we observe that the version space construction in Algorithm Jnnuress that for all t E [T], all E Fr satisfy
Σ EDU()) [ +D))))))))+(())))))())))))))))
2Cp
In addition, is immediate to that F F F2 2 Fr. Thus, by invoking Lemma G. with parameter Bon' 20D we have that
Est Estor (
Σ F F (D((())) < O((DDoon + min (log [F], 12 12 log T}).
To simplify, note that (Cb3of + 1) < CD 1), since Co 2 1.
For this lemma, D need not be metric-like loss; it suffices that D is bounded and has D(2, 2) = 0 for all
Z.
29

---

Proof of Lemma G.1 We begin by proving that Est
On (T) < O (3om - 1) log [FI). Let
us adopt the convention that JT+1 { so that FT+1 F S S Fi. For any parameter
I F ]. define L == min {1: f It immediate to see that for all E F [
1 -St, f S and for all [[] : = F Using that (()))
= 0 for
all x: E X, we we-write the online estimation emror
T
Σ E Byuse
Σ
Σ SEF.
E
((f()))
Ft
Fil
SEFUJ*1
Σ EEΣ
f((()),
t= ftt
Using that FT S Fr-1 C F1 can upper bound this quantity by
ΣΣΣ (U()))))) & Σ
ID(()))M)))
Fr
1= fi
1=1 fty=
.5t
To proceed, observe that for any function f E F, if bound in Eq. (24) if t, = i, then
t. then f E F.. It follows from the assumed
Σ (f D(F(F)) + Σ (((E))
1
1 Boft
where we have used the fact that the loss D is bounded by 1. Using this fact, and recalling that for all t [T, [ff ty th| |Fo Fetil, we bound
ΣΣ
1
(**)) S (3oft + 1)
T
1
S (Boff +1)
Fg Finll
F
f
[=1 fity=t
Finally, by Lemma F.l
we have that
T
T
Fel
Fe+1l
< log FFil = log FFl.
Fil
We conclude that
T E E (D(((=") < (Bom + 1) log [Fl.
We now prove the bound Est On (T) - OC,, (Bon + 1) |og log T). For each x E X, define 1-1
N = Σ 1(x x). Then we can write the online estimation error as
T
T ΣΣ
=xx=x)) By TVw=((WV-(GVV11)(C))r.tt
E By (D(((=)) Σ
[=1
1=1 ZEN
From the definition of μt, we have that
Ey~g (N-1
(N--(5)
Fi
(N-1(z) + 1)
F
JEF.
Now, from the offline guarantee assumed in Eq. (24)
we have for all X and F.
(N-1(xx) + +)D(f()),
Σ
D((
s Bor
s5t-1
30

---

Combining these observations, we have
1(x=z)
Σ By (D((() (Bo# 1)
Σ E t=1 =Ex M-1(=)
Now, for each & E X, define 4 = min {1 < T
= x) if this set is not empty, and set l = T xt
otherwise. From this definition and the fact that + 1/2 + ... 1/T + log T, have that
1(==v) = mEX (1
1z (x=x²)
T
1 x²)
Σ
Σ
(=)( V1
ZEN
1=1
1=
(x)
Σ
1+ Σ
#E
1=1
$ 2| + | log T S 3/ T.
We conclude that
(Vnx) 2 3(3r + +1) log
Theorem 3.2 (Main lower bound for ОΕОE). Consider the binary classification setting with 2 = = {0, } and loss (o For any N E N and fon > 0, there exists an instance (X,,, with log [1 = [ = N such that for any oracle-efficient algorithm, there is a sequence of covariates (x' ('',...,'"" x²) and offline oracle with parameter Bont such that
E [ [E)) 2 Q(min {(Son 1)N,
Proof of Theorem 3.2 Let N 2 be given, a consider the model class where X' {x1, 2N
AN is an arbitrary discrete set, 2 y {0, 1). J (0, 1) D(21, 22) Do (21, 1(2) * 22). and KC=))= 1_
We first specify the offline estimation oracle, then specify an adversarially chosen covariate sequence. Fix T E N, and for any S t < T and sequence of covariates x' .'....,"" xx' define N((x) == E 1sx = x). For any target parameter
F and offline estimation parameter Bon > consider the oracle Algon(::f) (*) for the sequence x², x',.... r'....,I' that returns
( = 5o (x)
if N < Bon, otherwise.
To complete the construction, we consider sequence (x), (x',...,,²) x²) in which
x² = Tmint [t/ "Pon Bonl1.
Equivalently, and perhaps more intuitively, we set
(x',...,x²) [1 (21.... T2.
IN...., IN, IN....)
[Rom]
[Pon]]
[Ron]]
stopping earlier T S 3.
For any S*, we now show that A Algón (:; J*) is an offline oracle with parameter 3ormr on the sequence
,',...,."* x² This is because for any < min [[t/[Bon) 11. N). the covariate 2; is repeated [Bon]] >
3or times. Thus f((x)) = (xxz). This implies for / S N [Bor]] that
Do/1 (P(''))
Σ
Doy1 Doys ( S [Bon]] - S ,on.
s=t- [t/Tioo 11-r
31

---

If > N [ then all the covariates are repeated for more than [Bon'] times, thus fo f Overall, we have shown that Algo#(:: f*) is an offline oracle with parameter Son-
Now, fix any oracle-efficient online estimation algorithm, and consider the expected regret under f* (Fi (with A f*) the oracle). If T Z N[ then regardless of how the algorithm chooses j²,, since for any block of Ts. f (2:) is independent of (z;), the expected regret lower bounded
E (*-(ini((FF) E Do/1 (f(r),
1=1
N
[Rni]
2
By By--Unit(F)) E B
1=1
j=1
NTPOn 2 (SON + 1)) 52minn {Sof + 1) log [F). (Borr + 1))
2
G.2 Proofs from Section 3.2
Theorem 3.3 (Impossibility of memoryless algorithms for OEOE), Consider the binary classification
setting with Z = y {0, and loss (. For any N E N and 3or 2 0, there exists an instance (X, y, Z, K, with |F| = | N such that for any memoryless oracle-efficiens algo- rithm, there exists a sequence of covariates (x' ......""""" x") and (potentially improper) offline oracle
Algon with parameter Bon such that E [Est (T') 2 Immin {N (Bor + 1),T)) This conclusion still holds when the online estimation algorithm remembers f'...., f but not x x',...,
Given a parameter PoHt 0 and an integer N, assume without loss of generality that K := =T/(1Bort + 1) is an integer. Consider the instance (X.V, 2,K. with X N1, y [0,1). Doy1, KC()) 1,, and parameter space F [fi)eenN] is defined
Proof of Theorem 3.3
as
[((x) = ii:
We consider a sequence of covariates (......., ,,")) divided into K blocks, each with length [BoH] + In each block, the covariates will be chosen to be the same, i.e., x²
PP x: go
x 2l We define [t/ [t(([ + 1)] as the index of the block the step t belongs to, and we adopt the convention that 2, X' is value of the covariates for block 7, i.e., x² Ir, for all t. We leave the precise choice for .I,..., K as a free parameter for now.
Fix any memoryless oracle-efficient online estimation algorithm defined by sequence of maps [F' Peetry (cf. Definition 3.1). We set the true target parameter to be f" = fer, where the index i* E [N] will be chosen later in the proof (in adversarial fashion based on the algorithm under consideration); for now, we leave i" [N] as a free parameter.
We first specify the offline estimation oracle under fc. For each block index 1,..., K, define
X = X' ((xx ((==))))(rr))))))) 1X U (i+))
the set of covariates in X', (i*} that have not been observed before block and let X = Xr((t").
We define
(x) 1(x X
as the estimator returned by the oracle at round is immediate from this construction that regardless of how x² ...., x," are chosen, the offline estimation error is bounded by
Vi [T), (rmn()())) <
Σ
Days (M(r)), [9om] S Bon,
sm 1o + +1)+1
since the value of f differs from for only for covariates that have not been observed before block T.-
32

---

It remains to lower bound the algorithm's 's online estimation enror. To start, note that Do/1 coincides with Dsq on the set {0, 1} and Dsq is convex. Hence, by Jensen's 's inequality, it suffices to lower bound
T
E
where (a(x") is the mean under (xx) where N~ = F P-1
To proceed, specify the sequence x' ...... x", choosing x' as measurable function of X and i* (recall that i* itself has yet to be chosen). Fix a round t, and suppose that X * 2. We choose x' to lower bound the estimation error by considering two cases. In the process, will also define
function j (2 [M] U { for each [K], where (2(N))@T 2 2
2 Let
copies
E [K] be fixed.
18on
If
Σ 1((-(100)11110()))) S 1/2)
Bos +1 for x E X', we set x, i* (equiva-
2
p=1
lently, (r(( 11)1 = for 1,..., [8om] + 1), so that
(7+1)(1800111)
[ +1
Σ
max(f (
[ ++)+1
In this case, we define }=(
P.) =1.
[Son|
If there exists j X such that = (equivalently, (L(oonn++1+ p=1
Σ
1 1(fr(Bon)-()+)()) S 1/2)
1 [0or|+1 +1
we set
j for I,..., [BoH] + 1) for the least such j. so
that
(1on)[[fon++11
[ +
(xx fi- (i-)))
16 *
8
(o ++1++
In this case, we define } (X.,..., X*) j as well.
Note that since f measurable function of X². ......... Px, J is well-defined.
Combining these cases, it follows that for any choice of i" choosing II,..., in the fashion described above ensures that
(( [Bon] + Σ 1100000,..,x**4 +*+]
We now state and prove the following technical lemma, which asserts that there exists a choice of it for which the right-hand side above is large.
Lemma G.2. For any algorithum, there exists a choice for i" E [N] such that
min ( x²)
P(N).
Proof of Lemma G.2 Consider more abstract process, which we claim captures the evolution of
XT. Let it [V], and let A' = [N]. We consider a sequence of sets {A- evolving according to
the following process, parameterized by sequence of functions {g- (2INT)er (NJ
and index i* E [N].
For 2 1:
gf (A), (A',..., A²) =1.1 A Ar.
gIf (A', (A',..., A') 1e, 1 A (A',.... Ar) if (A' ,...... A²) + i", and let
11 + A" otherwise.
33

---

We claim that there exists i" E [N] such that g' (A'..... A+) # 1" for all s < N under this process.
To see this define a set X inductively: Starting from X [N]. set X + X \ gX''..... ..... ) if d' (X), ,'...., X the choice i", since g" itself does not depend on
* 1, and set X <- X otherwise; note that this process does not depend on
For any T, observe that for any i E X", if we set i" = i, then "' A²) o i' for all < T, and so A X", It follows that as long as X A Ø, can choose i such that (A', ...,, A,) + for all < т. Since X shrinks by at most one element per iteration, it follows that this is possible for all N.
Lemma G.2 that by choosing i as guaranteed by the lemma, we have follows immediately from
2 [Bom]
Σ -(.))))
M(min (N, K})
8
Mmin {Nv 1),
Theorem 3.3' (Impossibility of memoryless algorithms for OROE; proper variant). Consider the binary classification setfing with 2 = = 10, and loss Do/1(;;)). For any N E N and foh 2 there exists instance (X, y, F) with [F| = X = N such that for any memoryless oracle- efficient algorithm that (i) proper, and (ii) time-invariant, there exists a sequence of covariates and a proper offline oracle Algon with parameter Bon such that E Esto On (T)) 2
(x' ....,"") ) Qmmin (Non + 1),T}).
Proof of Theorem 3.3 Given a parameter N E N, we consider the instance ( 2,K, given by X [ai Z y {0, 1]. D Do/1 and K(z) 12, with parameter space
F [filie[ given by
M1x,)) ( 2 i).
Let any memoryless oracle-efficient algorithm defined by prediction map F given. We lower bound the algorithm's 's online estimation error by considering two cases.
F. = F be
There exists a parameter such that the distribution μi, = F(f,) satisfies (T(fi) 1/2. We consider two sub-cases of Case 1. The first subcase is where fu(fffy i > i}}) > 1/4. In this case, we choose the sequence of covariates as x
Case I:
x= Ií, set f fs. and choose Algos to be the offline estimation oracle that sets f = fr = S.. With this choice, the offline estimation error for the oracle satisfies
Vt E [T], Do/ (M(E)))())) Σ Do
SSt
However, the online estimation error satisfies
Σ E ya (Po/
E B ([at(
Σ ((())))=((((t)))(((..)))
j>i
j>i
T
(z:)) = T/4.
4
34

---

The second sub-case of Case is where μr ( :j> i)) 1/4 This, combined with the fact that (f...) 1/2, gives Hi(lff : j < x > 1/4. In this sub-case, we choose the sequence covariates as x² x² = 1;-1, set I* = St. and choose Algor to be the offline estimation oracle that sets 1 - f = Si. In this case, the offline estimation error is zero:
Vt E [T], Day Dani ((()),) = >
Do (f((()( 0.
e<t
s<t
In addition, the online estimation error is lower bounded by
Σ t=1
f- (Doy1 (F
Σ E jom. E (()
t=1
jzi
M t= jEl
T
Dof1 De T/44
Case 2: For all parameter E F, the distribution μ4 F(ft) has (4(f)) 2 1/2.
In this case, we choose x² ,..,,"" by repeating each of the covariates [BoR] + number of times in increasing order by their index, and choose the offline estimation oracle Algor to return f1....JJa in the same block-wise but with the index offset by 1.
Formally, let = [e/[[ + 1)] denote the index of the block that round / belongs to, so that T[8or|+2
[2 +2 and so We choose x' Ne) and choose the offline oracle Algor to set f = min Finally, we set f fN-
T[
We have that for all t, the offline estimation error of the oracle is bounded
Σ Do (f(x))
Σ Do/1 (fN (Fmin(T...)) Smin(r..N) (Fmin(s,.t)))
Σ
15
[BOm]] S Bon.
However, the online estimation error is lower bounded by
1
E i (f))))
2 Doy1 (Fmiini(rr. (oin (Pm)))))))))
2 M(min (T, ((of 1)))-
Proposition D.1 (Upper bound for memoryless OΕOE). For any instance (X,Y, 2, F),, metric-like loss D, and offline oracle Algon with parameter Bon, the algorithm that returns On
=== f == Algon(a x' "'.....''') has online estimation error Est (T) <
O((Son + 1)| T).
Proof of Proposition D.I. The proof is very similar to the second part of the proof of Lemma G.I t-1
For each x E X, define Ne-1 Σ 1(x = x). Then we can write the online estimation error as
Σ D (f(("'))
T
Σ Σ
T
1(x = x') N-1(o)) (
I=1 nex
35

---

As consequence of the offline estimation guarantee for fr we have that
v1 vI 3on
Combining this with the preceding inequality gives
T Σ Eo(P(a)))(())) & (Bom + 1)
T
1(==x) (1)
1=1 ZER
Now, for any x E X, define i min ft S /) if this set not empty, and let ix T
otherwise. From this definition and the fact that + 1/2 + ... 1/T S I + log T. we have that
ΣΣ [(-=z) REA N1-1(z)Vi
1 x²)
1(x 1(==x') x')
Σ
Σ
N (z) V1
1 (z)
ZEA'
Ny (x)
s Σ
1+
t=1
log T 3|28 s 2| + |
We conclude that
Σ [o(((t))P(()) s 3BBrr +1) 1 log
G.3 Proofs from Appendix D
Theorem D.1 (Reduction from oracle-efficient online estimation delayed online learning). Let D be any convex, metric-like loss. Suppose we run Algorithm with delay parameter N E N and delayed online learning algorithm fpr the class F. Then for all 7 2 I, Algorithm 2 ensures that
E EEtt (T) & O(CDY(N + B/N) + Roou (N,
(14)
with any offline oracle Algon with parameter Bofl 2 0, where
ROOL (T, = E [C(P']] -y min Σ JEF
e())
(15)
the regret of ADOL for the sequence of losses constructed in Algorithm
Proof of Theorem D.I
Using the metric-like loss property, bound the online estimation
error of Aloorithm 2|by
T
Σ E Jnpe (ff((*),
Σ E [(F(o)) + Σo(po))
T
S Co
By the regret guarantee for the delayed online learning algorithm ADOL, we have
E [ (((F(=))F(<)) sr.
ED(P(E)) + (T.
36

---

since f E F. Combining these observations, we have that
BB CDlr+ + + Eo(Pw)) + RooL (T, N.).
Finally, from the definition of fr and the convexity of the loss D, have
T-N ++N
D (f
N +
1
((P(E))((()))
N
Σ (P(at)
N N
Kt
BOFT N
where the final line uses the offline estimation guarantee for Algon This completes the proof.
Lemma D.1. Consider the delayed online learning setting with a delay parameter N. There exists an algorithm that achieves
T
T ( S 2N log Fl
ROOL R (T,N,2) = Σ
BB [f((
min JET
for any sequences of losses e.....,
l" E 10.
Proof of Lemma D.1. This result follows using Lemma G.3 with y = 2, choosing AoL to be the exponential weights algorithm described in Corollary 2.3 of Cesa-Bianchi and Lugosi [17]. which has
Rou (T, S Ollgg [FI)
for all T and 7 2 1.
Theorem D.2 (Characterization of oracle-efficient learnability for binary classification), Consider binary classification instance (X, y, 2, K.F)) with 2 = y (0. 1). == and ([()==1 For any class FF and Bon 2 0, there exists oracle-efficient algorithm that achieves online estimation error O( fon Ldim Tlog T Ldim(F) log T). On the other hand, in the worst-case any algorithm must suffer at least (((.dim(FF) online estimation error.
Proof of Theorem D.2, For the lower bound we recall that for 9oH Lemma 21.6 of [57 states that any algorithm efffceeffficiett or not) has to suffer ((Ldim(F)) online estimation eтror in the worst case.
For the remainder of the proof, focus establishing the upper bound. For any set of parameters
F: + (((0, 1)), define the majority vote function Majority(J) for a class F via
(F) = 1 Σ f(11=) FEF
E(( x)) SEF
for all x E &. We will show that Algorithm (a variant of Algorithmm 2 that replaces averaging with
a majority vote), with a properly chosen delayed online learning algorithm ApoL, can obtain
o Bon - T + -ddmmFF) log
online estimation error.
37

---

Algorithm 4 Reduction to delayed online learning for binary loss
1: input: Offline estimation oracle Algon with parameter Bowr 2 0, delay parameter N E N, delayed online learning algorithm ADOL for class F.
2: for t = 1, ..,... do
3: Receive f from offline estimation algorithm.
4: if then
5: Let f Majority (1 ((+)) // This is the only different step compared
to Algorithm 2
6: Let (f) Do/1 (i-N f feedback.
(( and pass (**(() to ADOL as the delayed 7: Let ju' = Abbou(e) ,f,, () be the delayed online learner's prediction distribution. Predict with f μí' and receive x'.
8:
Let y be fixed, and consider any delayed online learning algorithm ADOL that achieves
Σ Ei [et] 1=11
min (() RooL NT)
for any sequence of losses in the delayed online learning setting with delay N (i.e., where we receive loss E at time & + N for some N 2 0).
We proceed bound the regret of Algorithm 4 estimation error is upper bounded by
Since the loss Doy1 is metric-like, the online
Σ Ei (Pns(())))))
Σ E B Doys (f(T)) + Σ Doj1 ((()))))))))
Next, the guarantee of ADOL ensures that
E [Do/1 (f(T)) s Don Don (f(tt) (D,
1=1
since / F. Combining these observations, we have that
B
S (1+) Σ
Don Do/1 (f(4)) + Roou (I,N,7)
Finally, we observe that for each step t, if Do/1 (f()), = 1. it means that least N/2 of the predictors fitt, fut must have predicted (x') incorrectly. This implies that
++
Day1 (F()))
2
Σ Day1 (F(t)))))))
N
But since the offline estimation assumption states that
Do/ (f(z') < 3on.
this implies that
T-N INN
Do/1 (f')),
SN+
2
Σ Do/1 (Mat))
N
T
N +
2
Σ
Σ
Do/1 (f(1))
N
1=2 k
2
< N +
N
38

---

We conclude that
p (f)))) & O(((N + BOn + RooL (T,N,
To complete the proof, we set 2 and choose AoL to be the algorithm described in Theorem 21.10 of Shalev-Shwartz and Ben-David [57], which (by incorporating the same technique as Corollary
2.3 of Cesa-Bianchi and Lugosi [7],, ensures that
Rol (TL(T/N < O(Ldim(FF) log T).
Then by Lemma G.3 with 7 2, we have
RooL (T, N, S O(N -Ldim( log
Setting N /
(ooTT)(LdmmF))oog T) 1, this yields
E P [D(M(r)), - o Bo BosLdim((F) .T T + Ldim(F) log T).
G.3.1 Supporting Lemmas
Algorithm 5 Reduction from delayed online learning to non-delayed online learning
input: Delay parameter N N, base online learning algorithm AOL-
2: Initialize N copies A A of the base algorithm.
3: for 1,...,, T do
4: i N then
5: Let p² = (8))
6: else
7:
Leti = mod N where E [N]. Receive loss (C-N Feed Pr-N to Abu
8:
9:
10:
Let μ² = Abi(E, (fN ,..,,, N-N).
11: Play f ~ pt.
The following lemma is standard result 67 42 35
which shows that the delayed online learning problem setting in Appendix D can be generically reduced to non non-delayed online learning. The idea behind the reduction, which displayed /Algorithm 5] is as follows. Given delay parameter N N, we run N copies F over disjoint subsequences rounds. The following lemma gives a guarantee for this reduction
of a given "base" online learning algorithm AoL for a class
Lemma G.3 (Delayed online learning reduction). Let AoL he a base online learning algorithm for the class F with the property that for any sequence of losses Et ......e fe. in the non-delayed online learning setting and any 2 1,
T
T E [e(())] -y min Σ
Σ
E
S (T)
FET
t=1
If we run Algorithm with delay parameter N N. then for all 7 2 1, the algorithm ensures that
N
T/N
T/N
Rool (,, <
Σ
E Bp+Nos [e+N (aa.vs(jtt) [++N0)]]
7 min fEF
(+Ns(())
NN- ROL (T/N,
for online learning with delay N.
1he algorithm described in Theorem 21.10 Shalev-Shwartz Shwartz and Sen-David (7 applies exponential weights algorithm to a specialized class of experts, and the guarantee obtained is for Rou (1, 1). The analysis from Corollary 2.3 of Cesa- Cesa-Bianchi and Lugosi [17] shows that for > e/(e 1). the same algorithm obtains Hou (I, 2) scaling with (LLdim(F) log We omit the details here since is standard argument.
39

---

Proof of Lemma G.3
By the guarantee of AOL, we have that for all i E [M]
T/N
T/N
Σ f=1
E B ( (
min FEF
()) & Roi (T/N,>)
Summing up over all i [N], we obtain
T/N
T/N
N ROL (OL(T/N,))
Σ
Σ
BRetNs u [e+tV= [fittt::]]]]] -7
min FEF
(+)x=(f)
j=1
j=1
N T/N
N
T/N
E Gy+ ((+tN((((tN.3)))) -
min JEF
(*)
1= j=1
Σ
B B [E(s*]]
-7
min
(().
JEF
H Proofs from Section 4
H.1 Proofs from Section 4.1
Theorem 4.1 (Computational lower bound for OEOE). Assume the existence of one-way functions
19
There exists a sequence of polynomially computable classes f2. Fn....) along with sequence of poly(n)-outph) description length offline oracles with Bon 0 associated with each F...
such that for any fixed polynomials q N - N and all n E N sufficiently large. any eracce-efficiens online estimation algorithm with runtime bounded by p(n) must have EE EEst Est
(T) 2 T/4 for all 1 S T S q(n). At the same time, there exists an inefficient algorithm that achieves E Est (T)) S O O(yn) for all T E N.
Proof Theorem 4.1, We frame the example proposed by Blum 14] in their Theorem 3.2 (see also Bun [5] in our setting. For any integer 1, let the covariate space X, be X, (0, 1)". and set
[ with 2 = 1) 1} and K(2)) = L2. We define class Fr. { fa: {0, 1]
fxxx) = {1 if r E C.. otherwise,
{=(x)=
for a certain collection of subsets {c, E Xx) ae defined in Definition 2 of Blum [4, which is constructed based on cryptographic functions using the assumption of existence of one-way functions. The precise definition will not be important. The properties will use
1. The value (z) can be computed in poly(n) time for any x E X,,
2. For any polynomials p(n), q(n), any (possibly randomized) online estimation algorithm (oracle-efficient or not) which runs in time p(n), and any time step < q(n). for {0, 13 and sequence x² 2- x²3 ,.24 (the specific definition of this sequence can be found in Blum [4] such that the online estimation error under this sequence when f = fs at least T/4 in expectation. Our lower bound construction for any oracle efficient online estimation algorithm with runtime bounded by p(n) in time step bounded by T S g(n) will choose the aforementioned covariate sequence the covariates revealed with the aforementioned function as the true parameter, i.e., x² x for (7) and f fs-
sufficiently large where q(n) 2v there exists
vg....,,"
Existence of one- one-way functions standard and widely believed complexity theoretic assumptions, which forms the basis of modern cryptography (30)
The argument is essentially asymptotic, since the choice of is determined by the power of the one-way function.
40

---

It is straightforward to see that the sequence F2... admits polynomial description length as claimed, since log [F,l vn. We are left to verify that there is an offline oracle that achieves Sor = 0 with poly(n)-outpu)) description length, yet does not provide any information not already available to the learner in the setting of Blum [14] (recall that in the protocol of Blum [14), the learner gets to see the covariates ,I" and the true labels y .'.""""""" at time step before making their prediction, but does not receive any other feedback).
Consider the following offlines oracle Algór((', .....,,,"".. : .'....,,'' circuit that, on input v, compares x sequentially with x² ,...,,"', If x is ever equal to x for some ,,'
'). The oracle output f is a
T E [ I]. the circuit will output y" If æ is not equal to any x" for r E [t - 11. the circuit outputs O. Such Boolean circuit can be constructed with polynomial size in because each x' has length
for r E [ 1] and t I T S q(n) by assumption. It is easy to see that this oracle achieves fon = O, yet does not provide any additional information about targel parameter f beyond what is available in the model of Blum [14. Combining all the above, we complete our lower bound proof.
Lastly, we observe that since the setting we consider is an instance of noiseless binary classification, the classical halving algorithm achieves an online estimation error bound of Ollog [F,,)) = ((vn)
H.2 Proofs from Section 4.2
In this section, prove Theorem 4.2 through four layers of reductions through different variants of the online estimation setting. In Anpeppddx I we first introduce the relevant settings and the describe reductions through them. We then combine these reductions to prove Theorem 4.2) Finally, we prove each of the four reduction results.
in Apppeddix H.2.2
H.2.1 Proof of Theorem 4.2
Theorem 4.2. Let ACDE be an arbitrury (unrestricted) online estimation algorithm that satisfies Eq. and has runtime Time(F, T). Then for any N N, there exists an oracle-effcient online estimation algorithm that achieves estimation error
Est (T)) S OCZ log V Bon /N + N. (RRCEE((T) C=logVV)
with runtime poly(Time(2 (ee,,,TT,,aa |Fl, (2), where 3om 0 is the offline estima-
tion parameter. The distributions ......""" produced by the algorithm have support size poly(log] [F1, X),
As a special case, if the online estimation guarantee for the base algorithm holds with RCDE(T) I C" logT for some problem-dependent constant C' 2 1, then by choosing N appropriately, we achieve E [[tt Ō((Cx(CF + C)) log V. TT12 (Cr + "og))
On (T)) S
Proof of Theorem 4.2, The proof of Theorem 4.2 is algorithmic, and is based on several layers of reductions.
First, using the scheme in Appendix D we reduce the problem of oracle-efficient online estimation to delayed online learning with the loss function e (f)
D(f-~Mx'-X) (x)-)) defined in Algorithm 2. where ft-N = 1 Ei=t-Ne1 f is an average of offline estimators and N E N a delay parameter.
Then, using a standard reduction 67 42 35 47
we reduce the delayed online learning problem above to a sequence of N non-delayed online learning problems, with the same sequence of loss functions; both this and the preceding step are computationally efficient.
To complete the reduction, we argue that the base algorithm can be used to solve the online learning problem above in oralle-efficient fashion. To do this, simulate interaction with the environment by sampling fictitious outcomes yf f'xx') from the averaged offline estimators and passing them into the base algorithm. We argue that the fictitious outcomes approximate the true outcomes well through change-of oeaofeer argument.
Combining the above, we conclude that given any base algorithm that efficiently performs online estimation with outcomes sampled from the target parameter ʃ*, we can efficiently construct computationally efficient and oracle-efficient algorithm. In more detail, we introduce four layers of reduction in reverse order from CDE to OEOE.
41

---

Conditional Density Estimation with Reference Outcomes (CDEwRO). The bottom- most reduction we consider is from a setting we refer to Conditional Density Estimation with Reference Outcomes (CDEwRO) to the (realizable) CDE setting. CDEwRO is similar to CDE, but with the following difference. Instead of receiving outcomes ,'...., y² sampled from the true model (x') directly, in CDEwRO, the outcome is sampled from a reference parameter (x') which is guaranteed to be close to f in certain sense. Moreover, the covariates and the reference parameters f "'....-"" selected obliviously (i.e. the entire sequence is chosen by the adversary before the online learning protocol begins).
Algorithm 6 Reduction from CDEwRO to CDE
1: input: Time T' E N, base algorithm ACDE.
2: Nature selects T covariates .','..,,'' along with the reference parameters f
xx"
3: for = 1...: do
4: Learner predicts f ~ pt AcnE(='.......... x==1 .. "...."""))
5: Outcome (z") is sampled and revealed to the learner with the covariate x'.
Our reduction from CDEwRO to CDE is given Algorithm The main guarantee for this reduction
is as follows.
Lemma H.1. For any fixed Ç 2 0, suppose Σ D ((()) S S. Let
REDEWRO (ROEWRO(TTC) (T, = 3Cy log V ++ (T) 20, log(CyTT),
(25)
where RCDE(T) is defined as in Eq. by the assumption on ACDE. Then Algorithm 6 achieves an expected online estimation error upper bound of
T Σ E(DA((()) & (CDEWRO(T,
in the CDEwRO setting, and has runtime
The key technique in the proof of this lemma is a change of measure argument based on Donsker- Varadhan (46).
Conditional Density Estimation with Reference Parameters (CDEwRP). The next reduction in stack is from setting refer to Conditional Density Estimation with Reference Parameters (CDEwRP) to the CDEwRO setting above. CDEwRP is identical to CDEwRO, except that in the former setting, the learner directly observes the reference parameter f' instead of observing y' ~ (x** as in CDEwRO.
second difference is that we allow the adversary the CDEwRP setting to be adaptive while our definition of the CDEwRO setting only allows for oblivious adversaries. Thus, the reduction we consider serves two purposes:
Simulating the CDEwRO feedback model through sampling.
Reducing the adaptive adversary to oblivious one.
The reduction from adaptive adversaries to oblivious follows and improves upon the result from 31 and may be independent interest.
Our reduction from CDEwRP CDEwRO is displayed in Algorithm 6) and takes input algorithm ACDEWRO (+: for the CDEwRO setting, where ALDEWRO (7;)) denotes the algorithm's 's output round / < T function of the history. The main guarantee for the algorithm follows.
Lemma H.2. For any fixed Ç 2 O, suppose Σ D3 ((())), SC. Let
RCDEWRR
(T. CTNSWRP(T.CC.
== 2RCDEWRO (T. (T.C) +
(26)
42

---

Algorithm 7 Reduction from CDEwRP to CDEwRO
1: input: Time 7' E N. accuracy parameter > O, algorithm ACDEWRO-
2: Let L Oog (TT)
3: for 1= 1=-1,....
do
4: for 1......... do
5:
for s = 1....,t 1 do
6:
Learner samples yt ~ f (x*).
7: Learner computes Fj ACDEWRO (T; 3..., 2i 1. M/
Vi 1.)).
8: Learner predicts via f ~ μt = Unif( {; 1e(2))
9: Nature selects and reveals the covariate x' and the reference parameter I based on jt"
where RODEWRO (T, 5) is defined as in Eq. (25) Then Algorithum 7 with parameter e >> 0 achieves an expected online estimation error upper bound of
Σ (IDA(F(W))MM)))) - REDEWRP (T,
in the CDEwRP setting, and runs time poly(Time({) T), log log X 1/E). The distribu- tions u², ..... n²² produced by the algorithm have support size poly(log FF, log T,
Conditional Density Estimation with Delayed Reference Parameters (CDEwDRP). Our next reduction is from a setting we refer to as Conditional Density Estimation with Delayed Reference Parameters (CDEwDRP) to the CDEwRP setting. CDEwDRP is identical to CDEwRP, except that the reference function f' is revealed only at round & N instead of round for delay parameter N E N.
Algorithm 8 Reduction from CDEwDRP to CDBwRP
1: input: Time T E N, delay time N N, algorithm ACDEWRP.
N 2: Initialize copies of the algorithm ACDEWRP as ACDEWRP..... CDEWRP-
3: for /. 1,..., do
4: Learner predicts fr μ²
x fo, futN fc-N)
where i = mod N.
5: Nature selects and reveals the covariate x' and the reference parameter f based on μ².
Our reduction from CDEwDRP to CDEwRP is displayed in Algorithm and takes as input an algorithm ACDEWRP (:5-) for the CDEwRP setting, where ACDEWRP e; -) denotes the algorithm's output at round < 7" with accuracy parameter >> O (cf. Algorithm 7), as a function of the history. The main guarantee for the algorithm follows.
Lemma H.3. For any fixed Ç O, suppose Σ DJ (P(r), SC. Let
(T, =
sup
REDEWRP (T/N.G,1/N),
(27)
CN.....C2200
GSC 1=1
where (TDDWRP((/N) G., 1/N) is defined as in Eg. (26) Then Algorithm 8 achieves expected
online estimation error upper bound of
Σ EEPH((W))M()) REDEWDRP (T, NT,
in the CDEwDRP setting, and has runtime poly(Time(F.1)) T)), log Fi, log (2)). The distributions 1t "...,, He produced by the algorithm have support size poly poly(log) [F], log (),
43

---

Algorithm 9 Reduction from OEOE to CDEwDRP
1: input: Time T E N, offline estimation oracle Algon with parameter Bor 2 0, delay parameter N E N. CDEwDRP algorithm AODEWDRP.
2: for 1= 1..... T do
3: Receive f = Algon (x'
.....)''')).
4: Learner computes reference parameter fo-N
Σ f.
N+1
5: Learner predicts f μi' = AODEWDRP (T, N: ,I,...,,
x', f , fr~)).
6: Nature selects and reveals the covariate x' based on ji"
Oracle-Efficient Online Estimation (OEOE). Our final reduction reduces the Oracle-Efficient CDEwDRP setting described above. This reduction, which
Online Estimation setting (OEOE) is displayed in Algorithm 8 is variant of the approach used in Theorem D. The reduction takes as input a CDEwDRP algorithm AODEWDRP( ( :),, where AODEWpRe N:.) denotes the algorithm's output at round < T with delay parameter N, as function of the history.
Lemma H.4. Algorithm achieves an expected online estimation ervor upper bound of
T
(F()), < RCDEWDRP(T, (,, BoNT/N)
in the OEOE setting, and has runtime poly(Time(F, T). log [F log (.t)). The distributions p². ......"" produced by the algorithm have support size poly (log [F], log (),
Completing the proof of Theorem 4.2 To prove Theorem 4.2 we compose all of the preceding reductions, with N left free parameter temporarily. We first apply Lemma H.4 to reduce from the OEOE setting to the CDEwDRP setting, with the guarantee that
T
Σ E
(f))
RCdEWDRP NT,N, + B/NT/N).
Then by Lemma H.3
we can reduce the CDEwDRP setting to the CDEwRP setting, with the
guarantee by Eq. (27) that
N REDEWRP (T/N, 1/N).
REDEWDRP (T, NT, + B/NT/N)
sup GSN+PorT/N 1=1
Then apply Lemma H.2 N times with T. Ç, and in the Lemma chosen to be T/N, G, and 1/N respectively for each [N], we can reduce the CDEwRP setting to the CDEwRO setting with guarantee by Eq. (26) that
N
sup
REDEWRP CTOEWRP(T/N.G.... T/N.G,11N) = 1+
sup
> ReDEWRO (T/N, (.).
GSN++oorT/ 2ml
GSN+BoNT/N
44

---

Consequently, apply Lemma H.1 N times with T and Ç in the Lemma chosen to be T/N and C for each E [.] we can reduce the CDEwRO setting to the CDE setting with guarantee by Eq. th5)
N
sup
REDEWRO (T/N,
GEEN+BOFT/N 1=1
sup
(3Cr log V. C + RCDE(T) 2C, log(C>7))
USN+
N Pon'T)
$ CF log V.
+ N (RcDE(T) + CF log(CrT))
N
POnTT +N- (RcDE(T) + CF log( log(VC)T)),
CFlog V
(28)
By choosing N v Cr log Cog V 1, we can bound the expression (28 as
(T, x (C (REDE (T) + Cy log(VC+T)) + RCDE + Cy log(VC)
I RCDE log V + Cy v (ognToo((CCyT))Le + RCDE(T) + Cf (og(ooC(TV)
Finally, under the assumption that (CDE(T) & C'2 log T, the bound above can be further simplified as
E Estt (, & (Cr(Cr 0zH) log Vlog(V Vlog(VCr7)))222 (Cr Co)
H.2.2 Proofs for supporting lemmas
Proof Lemma H.1, For For/Allorithm 6 denote the randomness of the sequence (y" ())))) under f E F and (f=:²) by [Pf" and pf respectively. The data generating process for (x* y in the CDEwRO setting implies that
Σ R (A((()))
Σ R Pa [D(((())))()))
By Donsker- Varadhan
Donsker-Varadhan [66. have that for all 17 0
T
DKL fp
| 2 E Eppar
Σ
E E
(x()
T log Epr exp 2
E (f()
(29)
For any random variables X,Y, 7, we denote by DKL (Px IPy () = Ez (Pxiz |l Pr[z)]. We further note that by the chain rule for KI. divergence,
T
DKL f.
7. DKL
( || [IP(ir,gy |--10 ( )
E Epjer
T Σ DKL (f(2')
where the second equality holds because y follows (a and f*(x' respectively, and because the conditional distribution of z is identical under both laws due to the oblivious assumption of the
45

---

covariates in the setting of CDEwRO. Then by the relation between KL and Hellinger (Lemma A.10 of (27]) and that 1 < log we have
T (f(',,
I Σ
Σ DKL ((()))1 = (2 + log
Σ
DR
& log V
(P(T)))))))
D
1=1
where the last inequality is by V2 e. Combining all the results so far and using 29 we have
T
B pk (f()
3log V
Σ Di
T
(f("),
1
T
+ log Eep exp 1)
Σ
Biine (H(F()))
"
1=1
To proceed, using that for any postive random variable X, E{X] = Jo P(X 2 t)dt, we have
log Epp exp
E
(f'),
(x'))))
1]
1=1
-RDDE(T) log Epy. exp
1
E Bp [D((F(2), RCDE(T))
- RcDE(T)) + log Epr exp
Σ B Bi (D) - RCDE(T)
RCDE (T) log ( pr ((
T
1 log dl.
E B [ RCDE(T)
0
"
Recall the assumption
Di (xxx)
s and let
We have
CF
T
3log V
7 Σ Di (((())))
+ 1 17
log Epr exp 11
Σ
Bi (D(
3 log
< + (T)
1
log /-0 pr (()
T
E B (D(((()), - REDE(T))
- log )as
2
re Sx-
$ 3Cr log V + REDE (T) + Cog
1+
dt
< 3C log V + REDE (T) + 2Cy log(C)T))
where the second inequality uses the assumption Eq. (4) on the algorithm.
Proof of Lemma H.2 Our result improves uses the proof technique from the adversaaiil--o-oolvviouss reduction Lemma of Gonen al. (3I], but improves the result by O(log T' factor.
Consider the CDEwRP setting. E' ["[CCEEwRO (y1" ...., . - taken over all the random variables yi
Let E+ = E|-
x -i fit-t]. Let 1M
21
"...,"" 1.) for all < x
< T' where the expectation is 2
...... "...,
Then by Bernstein's concentration inequality applied to je' (interpreted an empirical approximation to (*)"), conditioned on x -i-- fuit-,, we have with probability at least 1 - 2 for all a' e X and f F.F.
E Et Jumee [D [D)((()), s E j [D(((()) +2(()
(30)
46

---

For any fixed t E [T] and all t' such that t - t' S T. the different trajectories Li Le
1. u. vi" ..... V/ 1.1 are ii.d. conditioned x² fu-- Thus, we have
E
LE ympue [D(((()))
E B (f(e))
(31)
(32)
E B f-uue
By (i(f("'), (x
(33)
Finally, for Algorithm 7 we have by the guarantee of ACDEWRO.
T
E B7 x
(DA(F("") - RCDEWRO(T.()
Combining the three results above, we have
T
T
Σ E E|
E B Sup ((())))) +e
t=1
E
( +e u
t=1
2RcDEWRO (T, () t
where the first equality is by Eq. (30) the second equality from Eq. (31) and the final inequality is by the guarantee of ACDEWRO-
Proof of Lemma H.3 Note that Algorithm 8 is variant of the reduction in Lemma G.3 to squared Hellinger distance, and the proof here will use the same idea as Lemma G.3
specialized
For each i E [N], let Sa =E D (F f( ACDEWRP, we that for all E [N].
(a
Then by the guarantee of
T/N
Σ
E EingtN . [D (f (*(x(X)))]] S ReDEWRR RCDEWRP(T/N, G,1N),,
Summing up over all i E [N], we obtain
N T/N
R Jm [D(([==),
D [D (f("'t0))) ((rtxx))))
REDEWRP (i,N,i,1/N)
T Σ D
By the assumption on
(x(x), f((t), we have
T/N
D
[ Σ D (M()))M())) Sc
-=1
1=1 =
1=1
Finally, we conclude
T
N
E Bi [D((F()),
< sup
Σ
RCDEWRP RcDEWRPTT/N, G, 1/N).
G iml
47

---

Proof Lemma H.4, This reduction is arguably the most interesting one. This reduction is by noticing that averaging across the outputs of the offline oracle will generate reference parameters (although delayed) with small online estimation errors as shown later in Eq. (34) By the guarantee of AODEWDRP, have
T
Σ
B By (PA(((")
< REDEWDRR (T, C)2
where { can be chosen be any upper bound of
Σ D (z'), f( since it is unknown to the
(=1
learner in the setup where we augment the sequence f fr by setting fr+ = fr for all E N and define f = R Eiit1 f for t - O.N,T N + 1. T. Furthermore, by the definition of f', we can obtain
T
T-N
Σ DA (f()t) = = 1=1
Σ ((((())
i=++1
T
N +
((((()))
N 1=2 K
$ N +
(34)
N
Thus, conclude Algorithm obtains
T
E 1~ (P(()) < REDEWDRP (T, N,N Σ
N + N,, Bo#T/NN.
I Proofs from Appendix E
Theorem E.1 (Offiinett-online conversion under coverability). For any layer-wise loss D and MDP class (M, II, O) and M E M, the sequence of estimators (M* .......") produced by any offline estimation oracle Algon for DRL with parameter Bon satisfy
T Σ RL (TT) & & (M log + HCcov(M))).
Proof of Theorem E.1 This proof closely follows the proof of Theorem in Xie et al. 69 Define the shorthand di, (s, dx. (s,a) and Coov Ccov (M)), and define
a)) = Σ (s,a))) and 1 = arg min sup
dp
(EE((SA)=EEI) =ENI
44h
From the definition of the layer-wise loss, we have
Σ RL (M(TT)))MM))
=I Σ
(s, (P (8, Ph (s.
h= t= (a,0)SSAA
We define "burn-in" phase for each state-action pair (s, a) E A by defining
(8 a) min { (8, 2 Ccon ())
48

---

Let E [H] be With this definition, for we can write
Σ
Σ
(s,
Ph (s, () (s.a))
(5.0)ESxA
E
Σ
d2(s,a)Dn PWW (s,a))1PT ((.)))
E Σ (i (s,a))lP (s.
(s,0)ESxAA (CTa))
(s.))5xA:12m(e)))
For the first term above, we have
Σ
Σ an (P" (s,a)llPh (s.a))
Σ
4 (s, 2Coo Σ
(ī(s,a) a) 2C
(9.0) (x.)))
(s,a)E5xA
(a.0)ESAA
where the last inequality holds because
Σ (hitaa) (s,a)=
Σ
( (3. + dx (s, S 2Covv 1 (s,a).
(A
(*0))SxA
The remaining term is
Σ
Σ
dh(s,a)Dn (P (s)1i)
(h(3,aa)
1/2
Σ (1
1f1 2 mnss,))Dn (P"" P (s,
(s,))
h= (=1 (s,a)ESxA
T
(1 2 (s)
T ΣΣ
ΣΣ
as (P (s, (s.a)).
d (8.
A=
(s0)ESXA
A= (x
Following the derivation in Theorem of Xie et al.
69
we have
(1 as)d
(s, (d,,))di(ss.
Σ
Σ
Σ
dz (s,a)
(s, + Coov 1 (s
(sa)ESxA
(sa))SxxA
ds
2
max di (8,a) ('E(T]
(8, + Coav (8,
(Ma))SxA
di (8, a) + Coou 1 ( )
(8,a))
<2 max 9,0
max (8,
(sSa)SSxA
EE(T)
S Ccov log T
where the last inequality follows from Lemmas 3 and 4 of Xie et al. 69 consequence of the offline estimation assumption, we have that for E [T).
For the second term, as
17
Σ
as (P"
(P" (s. (s.a)) = DRL M (TT) (MT S Bon.
DRL
1=1 (sa)E5xA
Altogether, conclude that
Σ RLL (M((')))NM(*') s s0( WVCoonSooTTooo Tlog + HC
49

---

Corollary E.2. For any MDP class (M,IL and M E M, the sequence of estimators (M',... ,..., M²) produced by any offline estimation oracle Algor for squared Hellinger distance D with parameter Son satisfy
Esth (T) = Σ Di (MT)),
On
s s0 ( ( oogg + Con
Proof of Corollary E.2 have
By Lemma F.2
for any two MDP models M and M' and any и E A(IRRN))
" Bdd at [
(T), ( M(T) S
Σ
Ph ar (sh, on), Ph (sh, an))
(35)
On the other hand, for any h E [H], we have from Lemma A.9 of Foster et al. 27 that
EM' [i PM (n), an), PK (sh: .n))] - 4D (MM(), M
==8a; by choosing X = (sn, an) and Y = (1+1)8+1) in the aforementioned lemma. Summing up over h, we conclude that
RAV' . [ PK (sh,an), an), PK (sh, an)) < 4H (((
(36)
1=1
M7 from an offline oracle with parameter for for squared Hellinger distance. By Eq. (36) we have that for all f [2].
Consider any sequence of policies x²,.... , n² and outputs M
1-1
H
1-1
E
[DA
Ph (s.a)) s
DH (T
(T M. S 4.
1=1h=1
Thhorrmm thus implies that
T H
Σ
D PP (s, Ph (s, a))] z H CCooononnTTon + H
1=1
Finally, using distance:
Eq. (35), we can convert the inequality above into a bound on the square Hellinger
T
T
H DR Ph (s, P (s,
D M (T'), (T'))
Σ EMr."
h=1
H. CconBon T'ogo'llogg + 1122
Proposition E.2 (Tightness of offline-to-online conversion). For any integer T 2 and Boн >> there exists a contextual bandit class (M, A O) with |Al 2, a distribution di A(S), a sequence (r',,..,,7)) T") and an offline oracle Algor for DcB with parameter Bont such that the oracle's 's outputs (M) M² satisfy
T
Est
On (T) Σ
DcB M (T)) 11 (T
Proof of Proposition E.2
Let foo > 0 and T be given, and let N E N be chosen such that T =
N. LN Borl] ; we assume without loss of generality that N large enough such that 00fft > 1, which implies that [N Bon] > NNoor/2) Define A {ao, a.). S {so..... SN-1), and g" gM (s, a) 0
50

---

for all E S, A. Let d1 = Unif(S) be the context distribution. For any E {I,..., T), consider the deterministic policy n¹: - A and the offline estimator M defined via
01; if = Lo and 9 Th (8, a)
if = LL/LN a 01; {1, 0,
(sx)
00 otherwise.
otherwise.
We first verify that M have
2 satisfy the offline oracle requirement. For any / E (1,..., T), we
Dea (8, (',, (s)), gdd (s, (+TT())))
Σ (s((N1))) = =11
N
1 L Bonl] (ow S Bon-
However, the online error is
T Σ [NDon]
Σ Egndi
[Daa (g"
(s, (s)
2 VTn..n
1
2
51

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provvdddd proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
52

---

Answer: [Yes].
Justification:
See See|Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
model well-specificatte
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ted system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
Justification:
The paper provides detailed assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
53

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [NA].
Justification:
This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification:
This paper does not include experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc/ hooollc/guuisss/ooeeelllmisssoo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
54

---

including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results, See the NeurIPS code and data submission guidelines tht //nips. httpps.cccpublicc deeegguueeeeeeeeebmiiii ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [NA].
Justification:
This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA].
Justification:
This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula. call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
55

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of emrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA].
Justification:
This paper does not include experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpps///eerrrs..cc//oal/tttttaa
Answer: [Yes].
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This is theoretical work. There is no socictal impact on the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
56

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA].
Justification:
This paper does not existing assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
57

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. www.co///aaooooo coomdatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
This paper does not involve crowdsourcing or research with human subjects.
58

---

Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
59


Reviews:
[{"review_id": "yPYOUzflgm", "reviewer": "Reviewer_do6a", "summary": "The paper considers an online estimation problem, in which the learner does not have a direct access to the past values (labels), but rather instead just an oracle access to an offline estimator based on past covariate-value pairs. The main assumption is that the loss of these offline estimators is bounded. The paper addresses the question whether this sequence of offline estimators can be converted into an online estimator, both in terms of statistical complexity and computational complexity. From the statistical aspect, it is shown that an efficient conversion is possible, yet only when the learner knows the entire sequence of past offline estimators (and just storing the latest one cannot guarantee this). The proposed protocol can be considered as a generalization of the halving algorithm. \n\nFrom the computational aspect, a negative result is shown stating that polynomial run-time algorithms cannot achieve the optimal online statistical complexity, but is somewhat relieved by the setting of density estimation, since in this setting the learner can artificially produce values $y$ from the estimated density.", "questions": "1. In line 33 it is claimed that statistical estimation is mainly about fixed design. Random design of the covariate-value is also commonly considered.\n\n2. Theorem 3.3 is an impossibility result, showing that even a memoryless estimator is not efficient, even if it is improper. Doesn't this trivially imply the same result for the subset of proper learners ? Why thus a different result is needed ?", "limitations": "The authors properly discuss limitations of their results throughout the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The setting is general, and appears to be related to a multitude of contemporary problems. \n\n2. The paper addresses the problem from both statistical and computational perspectives. \n\n3. The results are detailed, including different algorithms for finite and infinite classes (the latter in the appendix). The results are explained in comparison and in light of classic algorithms (like the halving algorithm). The scaling of the bounds and the dependency on each parameter is justified in detail.", "weaknesses": "1. The main proposed algorithm is somewhat brute-force and just keeps the hypotheses that agree with the data – in this case it is covariance-estimator pairs. \n\n2. It is not obvious that the OEOE model, in which both all past covariates as well as all past offline estimators are saved, actually captures aspects such as limited past data availability and compression of past observations and decisions. This is because storing an estimator may be much more storage-consuming than the value itself (e.g., as in binary classification). Generalizations based on finite past data seems to be challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness:\n\n>> The main proposed algorithm is somewhat brute-force and just keeps the hypotheses that agree with the data – in this case it is covariance-estimator pairs.\n\nThis is the optimal approach from an information-theoretical perspective, achieving the optimal minimax guarantee for our setup. We envision better algorithms under more assumptions on the model class in future works.\n\n>> It is not obvious that the OEOE model, in which both all past covariates as well as all past offline estimators are saved, actually captures aspects such as limited past data availability and compression of past observations and decisions. This is because storing an estimator may be much more storage-consuming than the value itself (e.g., as in binary classification). Generalizations based on finite past data seems to be challenging.\n\nIf only the estimator at the current round is remembered without covariates, Theorem 3.3’s lower bound applies, making the algorithm inefficient. However, finding efficient methods to remember the version space without previous estimators allows the use of Algorithm 1. Additionally, Algorithm 2 in Appendix D can mitigate this issue by storing only the $N$ most recent estimators, albeit with increased regret.\n\nQuestion:\n\n>> In line 33 it is claimed that statistical estimation is mainly about fixed design. Random design of the covariate-value is also commonly considered.\n\nWhile random design is common, the OEOE setup reveals covariates, making fixed design regression more appropriate.\n\n>> Theorem 3.3 is an impossibility result, showing that even a memoryless estimator is not efficient, even if it is improper. Doesn't this trivially imply the same result for the subset of proper learners? Why thus a different result is needed ?\n\nIn our OEOE setup, the adversary controls the offline oracle, and the learner only has the offline guarantee. If the oracle is improper, the adversary is stronger, so a lower bound with an improper oracle does not imply one with a proper oracle.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VEoVuVxwwS", "reviewer": "Reviewer_erpa", "summary": "This paper studies the methods to convert offline estimation algorithms into online estimation algorithms in a black-box fashion. This work introduce a new protocol, Oracle-Efficient Online Estimation, which provides an information-theoretic abstraction of the role of online versus offline estimation.", "questions": "This work discusses the impossibility of memoryless oracle-efficient algorithms, I wonder what if the learner can select the online estimator at time t as a function of the most recent \"w\" offline estimators, where w is the window size.\n\nIn addition, the authors present a fine-grained perspective on the computational complexity of oracle-efficient estimation for conditional density estimation. Are there any insights for the other two problems of binary classification and regression?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The problem of online estimation via a sequence of offline estimators is of great importance to the online decision making community.", "weaknesses": "The organization of the paper requires significant improvement as it is currently hard to follow. The introduction section occupies almost half of the main text, leaving the main methodology and theoretical results insufficiently discussed and lacking in clear theoretical insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness:\n\n>> The organization of the paper requires significant improvement as it is currently hard to follow. The introduction section occupies almost half of the main text, leaving the main methodology and theoretical results insufficiently discussed and lacking in clear theoretical insights.\n\nPlease see the general rebuttal for a detailed clarification of the theoretical insights.\n\n\nQuestion:\n\n>> This work discusses the impossibility of memoryless oracle-efficient algorithms, I wonder what if the learner can select the online estimator at time t as a function of the most recent \"w\" offline estimators, where w is the window size.\n\nYes, Algorithm 2 in Appendix D presents a general reduction from OEOE to online learning, which only requires the learner to remember the most recent $N$ offline estimators, with tradeoffs in regret depending on $N$.\n\n>> In addition, the authors present a fine-grained perspective on the computational complexity of oracle-efficient estimation for conditional density estimation. Are there any insights for the other two problems of binary classification and regression?\n\nYes, the technique for conditional density estimation extends from Algorithm 2 in Appendix D, which also applies to binary classification and regression. The conclusion remains similar: if efficient online algorithms exist for binary classification and regression, then efficient algorithms for the OEOE setting also exist.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kffq6eoQqW", "reviewer": "Reviewer_3FiW", "summary": "This paper proposes an algorithm that can convert offline estimation oracles into online estimation algorithms in a black-box fashion. The conversion is built within the OEOE framework, which manipulates the learner, offline oracle, and environment simultaneously. The authors also propose certain upper bounds and lower bounds for their conversion algorithm, which shows that it achieves both statistical and computational efficiencies. This paper also presents several additional results like the hardness of memoryless algorithms, enhancing the soundness of this paper.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The framework of OEOE is general enough to include many real-world problems.\n\n2. The statistical complexity has its lower and upper bounds nearly matched.\n\n3. The author shows that although there is no computational algorithm for OEOC, their algorithm is computationally efficient in some fine-grained special cases.", "weaknesses": "1. I understand that many works assume knowing the intrinsic of the data-generating, but assuming that $\\mathcal{K}$ is known would exclude some cases of statistical estimation, which might reduce the generality of OEOE.\n\n2. Can you provide a quantified definition of “oracle-efficient”? Despite it has been heavily used in the paper, I can only find some rough explanations after searching through the text.\n\n3. I have not checked the proof of theorem 3.2 and theorem 4.1 carefully, so maybe I am wrong, but the hardness construction in them seems a bit ill-conditioned. The worst case is searched through all offline oracles, the adversary seems overpowered.\n\n4. As stated in Theorem 4.1, there is no computational algorithm in OEOE. So could it be that the definition of this framework includes some unwanted cases that should be excluded? And does that mean that this framework needs some redefinition?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">> I understand that many works assume knowing the intrinsic of the data-generating, but assuming that is known would exclude some cases of statistical estimation, which might reduce the generality of OEOE.\n\nThe OEOE is concerned with transforming an offline guarantee into an online one. Even when the data-generating process is unknown, whenever the offline guarantee can be obtained, the OEOE can be applied to transfer it into an online guarantee.\n\n\n>> Can you provide a quantified definition of “oracle-efficient”? Despite it has been heavily used in the paper, I can only find some rough explanations after searching through the text.\n\nWe define the term 'oracle-efficient' in lines 106-107: 'An algorithm is termed oracle-efficient if it attains low online estimation error (2) in the OEOE framework.' In other words, the algorithm is statistically efficient, given access to an offline oracle without any extra labeling information.\n\n>> I have not checked the proof of theorem 3.2 and theorem 4.1 carefully, so maybe I am wrong, but the hardness construction in them seems a bit ill-conditioned. The worst case is searched through all offline oracles, the adversary seems overpowered.\n\nYes, the adversary can search through all offline oracles. However, it is not overpowered for two reasons: First, in the OEOE setup, the learner has no information on how the offline oracle output is generated. Second, the offline oracle must still satisfy the stringent offline guarantee.\n\n\n\n>> As stated in Theorem 4.1, there is no computational algorithm in OEOE. So could it be that the definition of this framework includes some unwanted cases that should be excluded? And does that mean that this framework needs some redefinition?\n\n\nIt is unfortunate that there is no computational algorithm in OEOE, which is interesting in its own right. To improve computational results, Section 4.2 considers cases for conditional density estimation where an efficient online density estimation algorithm is present. This result presents a tradeoff between computational efficiency and theoretical guarantees. The OEOE with conditional density estimation can be computationally efficient with an efficient online density estimation algorithm but is not optimal statistically in terms of minimax regret. This result extends the general reduction from OEOE to online learning in Appendix D. The tradeoff also applies to the classification and regression setup we consider. Refining the framework to more specific cases could ensure the presence of an efficient algorithm. We look forward to further developments in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrwsKOnKKt", "reviewer": "Reviewer_HaHn", "summary": "This work studies the possibility of converting offline estimation algorithms into online estimation algorithms using an information-theoretic approach. It introduces the Oracle-Efficient Online Estimation (OEOE) framework, where the learner interacts with a data stream indirectly through a sequence of offline estimators produced by a black-box algorithm. The main contributions are the characterization of the statistical and computational complexity of online estimation within this framework. It demonstrates that near-optimal online estimation error can be achieved via offline estimation oracles, though computational efficiency is generally not feasible except in specific cases like conditional density estimation. This framework is also applied to interactive decision-making problems, providing insights into the relative power of online and offline estimators.", "questions": "1. Have you considered any specific real-world applications or practical scenarios where this framework could be directly applied or tested? How do you envision the implementation challenges being addressed in such cases?\n\n2. The theoretical results are derived under certain assumptions, such as the metric-like structure of the loss function and the bounded offline estimation error. How sensitive are your results to these assumptions? Could you provide insights or potential extensions to your framework that could handle more relaxed or different sets of assumptions, thereby increasing the generalizability of your findings?", "limitations": "None", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Here are some strengths of the paper:\n\n1. This work introduces a novel framework, Oracle-Efficient Online Estimation (OEOE), which creatively combines existing ideas from offline and online estimation and therefore close the gap between classical statistical estimation and contemporary online learning.\n\n2. Paper provides theoretical analysis and proof to support its claims. The results show a deep understanding of the complexities involved in converting offline estimation algorithms to online contexts.\n\n3. By addressing the relative power of online and offline estimation methods, the paper studies a fundamental question in the field of statistical learning. The implications of this work extend to practical applications in interactive decision-making, such as contextual bandits and reinforcement learning.", "weaknesses": "Here are some weaknesses of the paper:\n\n1. While the paper provides a comprehensive theoretical analysis, it acknowledges the computational inefficiency of the proposed Oracle-Efficient Online Estimation (OEOE) framework in general cases. The authors could improve this aspect by exploring potential heuristic approaches or approximations that could offer practical computational advantages, even if they come with some trade-offs in theoretical guarantees.\n\n2.  The paper lacks empirical validation of the proposed framework. Including a few experimental results or simulations demonstrating the practical performance of the OEOE framework could significantly strengthen the paper. Even a small set of experiments showing the feasibility and comparative performance against existing online estimation methods would provide valuable insights.\n\n3. Certain sections of the paper, particularly the detailed proofs and technical discussions, are dense and may be difficult for readers to follow. Improving the clarity of these sections by adding more intuitive explanations, step-by-step breakdowns of complex proofs, or illustrative examples could make the paper more accessible. This would help readers better understand the contributions .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weakness:\n\n>> While the paper provides a comprehensive theoretical analysis, it acknowledges the computational inefficiency of the proposed Oracle-Efficient Online Estimation (OEOE) framework in general cases. The authors could improve this aspect by exploring potential heuristic approaches or approximations that could offer practical computational advantages, even if they come with some trade-offs in theoretical guarantees.\n\nIndeed, to improve results regarding computation, in Section 4.2, we consider the cases for conditional density estimation where an efficient online density estimation algorithm is present. This result presents a tradeoff between computational efficiency and theoretical guarantees.\nThe OEOE with conditional density estimation can be computationally efficient with an efficient online density estimation algorithm but is not optimal statistically in terms of minimax regret. \nThis result is an extension of the general reduction from OEOE to online learning in Appendix D. The tradeoff also extends to the classification and regression setup we consider. \n\n\n>> The paper lacks empirical validation of the proposed framework. Including a few experimental results or simulations demonstrating the practical performance of the OEOE framework could significantly strengthen the paper. Even a small set of experiments showing the feasibility and comparative performance against existing online estimation methods would provide valuable insights.\n\n\nThis paper presents a purely theoretical result. We look forward to seeing experiments in future works.\n\n>> Certain sections of the paper, particularly the detailed proofs and technical discussions, are dense and may be difficult for readers to follow. Improving the clarity of these sections by adding more intuitive explanations, step-by-step breakdowns of complex proofs, or illustrative examples could make the paper more accessible. This would help readers better understand the contributions.\n\nPlease see the general rebuttal for a detailed clarification of the theoretical insights.\n\n\nQuestion:\n\n>> Have you considered any specific real-world applications or practical scenarios where this framework could be directly applied or tested? How do you envision the implementation challenges being addressed in such cases?\n\nTransferring a theoretical algorithm to real-world applications requires domain specification and engineering. This work provides guidance in algorithm design principles for this new setup rather than improving existing online learning algorithms. Together with [1], it shows that using offline regression outputs suffices for interactive decision-making. For any reinforcement learning task, the proposed algorithm involves: (1) Performing offline regression to approximate the RL model, (2) Creating a version space (trust regions) of the RL model, (3) Averaging the possible RL models to create an online regression output, and (4) Using the E2D algorithm in [1] to balance exploration and exploitation. This work provides guidance in steps (2) and (3).\n\n[1] The Statistical Complexity of Interactive Decision Making\nDylan J. Foster, Sham M. Kakade, Jian Qian, Alexander Rakhlin, (2021)\n \n\n>> The theoretical results are derived under certain assumptions, such as the metric-like structure of the loss function and the bounded offline estimation error. How sensitive are your results to these assumptions? Could you provide insights or potential extensions to your framework that could handle more relaxed or different sets of assumptions, thereby increasing the generalizability of your findings?\n\nThe metric-like losses are quite general, covering cases such as square loss, absolute loss, and KL divergence with a bounded density ratio. Assuming the offline estimation error is bounded is reasonable. The only restrictive assumption is knowing an upper bound for the offline estimation error for Algorithm 1. When this assumption is not met, Algorithm 2 in Appendix D, which has worse but still sublinear statistical regret, can be used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General rebuttal:

### Theoretical insights for Theorem 3.1:

We would like to highlight the technical challenges and our contributions. Online learning has been extensively studied for decades. The most classical algorithm is the exponentially weighted aggregation, which weights all parameter functions exponentially with respect to their loss but never eliminates any functions. This algorithm is inapplicable to our setup as the label and loss are never revealed. The HALVING algorithm, which predicts based on the majority vote of the functions in the version space, is also not suitable due to its binary case specification. Our algorithm generalizes both algorithms, a surprising innovation given decades of research in online learning.

When $\beta = 0$, offline estimators must reveal the correct labels of all past covariates, recovering the classical online learning setup and allowing immediate application of our algorithm. Version space averaging is a soft aggregation compared to the majority vote, making it a generalization of the HALVING algorithm. Exponential weight aggregates the whole function class, while version space averaging constrains the averaging space, making it more rigid. Our algorithm interpolates between the exponentially weighted aggregation and the HALVING algorithm. Moreover, the theorem is proven through a novel potential argument, which is of independent interest.

### Theoretical insights for Corrollary D.1:

Nevertheless, we also have a general reduction from OEOE to online learning where the loss for the online learning is generated by averaging over the most recent $N$ estimators. This reduction relaxes the condition that we need to remember all the estimators and that we need to know an upper bound of the offline estimation error. By averaging over only the most recent estimators, our method simplifies the process and reduces memory requirements, making the approach more efficient. Additionally, this reduction allows for greater flexibility in handling the offline estimation error, as it no longer necessitates a strict upper bound. 

In conclusion, we believe our technical contributions are substantial, and our setup is likely to stimulate further research in this area.

Meta-review: The paper investigates whether offline estimation methods can be adapted for online use without needing to understand their internal workings. It proposes a new framework called Oracle-Efficient Online Estimation (OEOE) that uses a sequence of offline estimators. The authors provide theoretical results on both ends of the statistical and computational complexity of online estimation. They find that while it is possible to efficiently convert offline methods to online ones, doing so computationally can be challenging. However, they demonstrate that for a specific task (for example, estimating conditional density), computationally efficient online estimation is achievable. The paper contains impressive theoretical results. It would be useful to improve the initially submitted version by fully discussing the limitations and future developments at the end of the paper.
Decision: Accept (poster)