Title: Assouad, Fano, and Le Cam with Interaction: A Unifying Lower Bound Framework and Characterization for Bandit Learnability
Venue: NeurIPS 2024
Accepted: yes
Keywords: Information-theoretic lower bound

online learning

interactive decision making

statistical learning

minimax lower bound
Abstract: We develop a unifying framework for information-theoretic lower bound in statistical estimation and interactive decision making. Classical lower bound techniques---such as Fano's method, Le Cam's method, and Assouad's lemma---are central to the study of minimax risk in statistical estimation, yet are insufficient to provide tight lower bounds for \emph{interactive decision making} algorithms that collect data interactively (e.g., algorithms for bandits and reinforcement learning). Recent work of Foster et al. provides minimax lower bounds for interactive decision making using seemingly different analysis techniques from the classical methods. These results---which are proven using a complexity measure known as the \emph{Decision-Estimation Coefficient} (DEC)---capture difficulties unique to interactive learning, yet do not recover the tightest known lower bounds for passive estimation. We propose a unified view of these distinct methodologies through a new lower bound approach called \emph{interactive Fano method}. As an application, we introduce a novel complexity measure, the \emph{Fractional Covering Number}, which facilitates the new lower bounds for interactive decision making that extend the DEC methodology by incorporating the complexity of estimation. Using the fractional covering number, we (i) provide a unified characterization of learnability for \emph{any} stochastic bandit problem, (ii) close the remaining gap between the upper and lower bounds in Foster et al. (up to polynomial factors) for any interactive decision making problem in which the underlying model class is convex.

Assouad, Fano, and Le Cam with Interaction: A Unifying Lower Bound Framework and Characterization for Bandit Learnability

Abstract

We develop a unifying framework for information-theoretic lower bound in statistical estimation and interactive decision making. Classical lower bound techniques-such as Fano's method, Le Cam's method, and Assouad's lemmaare central to the study of minimax risk in statistical estimation, yet are insufficient to provide tight lower bounds for interactive decision making algorithms that collect data interactively (e.g., algorithms for bandits and reinforcement learning). Recent work of Foster et al. [40, 42] provides minimax lower bounds for interactive decision making using seemingly different analysis techniques from the classical methods. These results-which are proven using a complexity measure known as the Decision-Estimation Coefficient (DEC)-capture difficulties unique to interactive learning, yet do not recover the tightest known lower bounds for passive estimation. We propose a unified view of these distinct methodologies through a new lower bound approach called interactive Fano method . As an application, we introduce a novel complexity measure, the Fractional Covering Number , which facilitates the new lower bounds for interactive decision making that extend the DEC methodology by incorporating the complexity of estimation. Using the fractional covering number, we (i) provide a unified characterization of learnability for any stochastic bandit problem, (ii) close the remaining gap between the upper and lower bounds in Foster et al. [40, 42] (up to polynomial factors) for any interactive decision making problem in which the underlying model class is convex.

1 Introduction

The minimax criterion is a standard approach to studying the intrinsic difficulty of problems in statistics and machine learning. For an algorithm ALG that collects data (either passively or interactively) from the model M , the minimax criterion (stated somewhat informally here) is

The expression reflects the best cost that can be achieved by an algorithm ALG for a worst-case problem instance in a collection M , measured according to an appropriate cost function Cost . In statistics, the minimax approach was pioneered by A. Wald [90], who made the connection to von Neumann's theory of games [84] and unified statistical estimation and hypothesis testing under the umbrella of statistical decision theory . Minimax optimality and minimax rates of convergence of

estimators have since become a central object in the modern non-asymptotic statistics [82, 89]; here, for instance, ALG is an estimator of an unknown parameter based on noisy observations.

Upper bounds on the minimax value (1) are typically achieved by choosing a particular algorithm, while lower bounds often require specialized techniques. In statistics, three such techniques are widely used: Le Cam's two-point method, Fano's method, and Assouad's method. These techniques entail constructing 'difficult' subsets of the class M . Le Cam's method focuses on two hypotheses, while Assouad's method and Fano's method involve multiple hypotheses. The relationships between these methods are explored in Yu [96].

Classical statistical estimation is a purely passive task. A parallel line of research [57] considers the task of interactive decision making , where ALG is a multi-round procedure that directly interacts with the data generating process and iteratively makes decisions with the (often contradictory) aims of minimizing cost and collecting information. Proving minimax lower bounds for interactive decision making problems presents unique challenges. The aforementioned lower bound techniques for estimation require quantifying the amount of information that can be gained from passively acquired data from a hard problem instance, but the amount information acquired by an interactive algorithm is harder to quantify [4, 68, 69], since it depends on the decisions made by the algorithm itself over multiple rounds.

In spite of the challenges, recent work of Foster et al. [40, 42] shows that a complexity measure known as the Decision-Estimation Coefficient (DEC) leads to both lower and upper bounds on the minimax rates for a general class of interactive decision making problems. Interestingly, the lower bound techniques in Foster et al. [40] proceed in a seemingly different fashion from classical lower bounds for statistical estimation; most notably, their techniques involve an algorithm-dependent (as opposed to oblivious) choice of a hard-to-distinguish alternative problem instance.

Given the differences between the classical Assouad, Fano, and Le Cam methods, and the even larger disparity between these methods and the interactive decision making techniques of Foster et al. [40, 42], it is natural to ask whether there is a hope of unifying these lower bounds techniques. Beyond the fundamental nature of this question, there is hope that a unified understanding might lead to tighter lower bounds, or even inspire new algorithms and upper bounds; of particular interest is to close the remaining (estimation-based) gaps between the upper and lower bounds on the minimax rates for interactive decision making left open by Foster et al. [42].

Contributions. Wepresent a new framework for information-theoretic lower bounds which allows for a unifying presentation of classical lower bounds in statistical estimation (Assouad, Fano, and Le Cam) and recent DEC-based lower bounds for interactive decision making [40, 42].

Interactive lower bound framework (Section 3). Our main result is to introduce a new lower bound technique, the interactive Fano method . The interactive Fano method generalizes the stringent separation condition in the classical Fano inequality to a novel algorithm-dependent condition by introducing the concept of 'ghost data' generated from a reference distribution. This technique recovers the Le Cam two-point method (and convex hull method), Assouad method, and Fano method as special cases. By virtue of being algorithm-dependent in nature, the interactive Fano method seamlessly recovers DEC-based lower bounds for interactive decision making as a special case, and leads to refined quantile-based variants.

Fractional covering number and bandit learnability (Section 4). As an application of the interactive Fano method, we derive lower bounds for interactive decision making based on a new complexity measure, the fractional covering number , which quantifies the difficulty of estimating a near-optimal policy/decision, and complements the original DEC lower bounds (which reflect difficulty of exploration as opposed to difficulty of estimation). As an application, the fractional covering number provides both lower and upper bound for learning any structured bandit problem, up to an exponential gap. In particular, finiteness of the fractional covering number is the first necessary and sufficient condition for finite-time learnability of any structured bandit problem. As a secondary result, we use the fractional covering number to close the remaining gap between the upper and lower bounds in Foster et al. [40, 42] (up to polynomial factors) for any interactive decision making problem in which the underlying model class is convex.

Related work. Due to space limitations, we discuss the related work in Appendix A.

1.1 Preliminaries

Let P and Q be two distributions over a space Ω such that P is absolutely continuous with respect to Q . Then, for a convex function f : [0 , + ∞ ) → ( -∞ , + ∞ ] such that f ( x ) is finite for all x > 0 , f (1) = 0 , and f (0) = lim x → 0 + f ( x ) , the f -divergence of between P and Q is defined as

Concretely, we make use of three well-known f -divergences: the KL-divergence D KL , the squared Hellinger distance D 2 H , and the total variation distance D TV , for which the function f ( x ) is chosen to be x log x , 1 2 ( √ x -1) 2 , and 1 2 | x -1 | respectively. For a pair of random variables ( X,Y ) with joint distribution P X,Y , the mutual information is defined as

where P Y | X is the conditional distribution of Y | X and P Y is the marginal distribution of Y .

2 Statistical Estimation and Interactive Decision Making

We work in a general framework we refer to as Interactive Statistical Decision Making (ISDM). We adopt this framework as a convenient formalism which encompasses statistical estimation and interactive decision making in a unified fashion. We first introduce the framework and show how it subsumes statistical estimation (Section 2.1) and interactive decision making (Section 2.2), then give brief background on existing lower bound techniques and gaps in understanding (Appendix A).

Interactive Statistical Decision Making. An ISDM problem is specified by ( X , M , D , L ) , where X is the space of outcomes, M is a model class (parameter space), D is the space of algorithms, and L is a non-negative risk function. For an algorithm ALG ∈ D chosen by the learner and a model M ∈ M specified by the environment, an observation X is generated from a distribution induced by M and ALG : X ∼ P M, ALG . The performance of the algorithm ALG on the model M is then measured by the risk function L ( M,X ) . The learner's goal is to minimize the risk by choosing the algorithm ALG . As described in the Introduction, the best possible expected risk the learner may achieve is the following minimax risk :

While our main results concern the general problem formulation in Eq. (2), we focus on applications to statistical estimation and interactive decision making throughout. Below, we give additional background on these settings and show how to view them as special cases.

2.1 Statistical estimation

In statistical decision theory [90, 13, 12], the learner is given the parameter space Θ , observation space Y , decision space A , and a loss function L . For an underlying parameter θ ⋆ ∈ Θ , n i.i.d. samples Y 1 , ..., Y n ∼ P θ ⋆ are drawn and observed by the learner. The learner then chooses a decision A = A ( Y 1 , · · · , Y n ) ∈ A based on the observations, and incurs the loss L ( θ ⋆ , A ) .

Any general statistical estimation problem can be trivially viewed as a ISDM instance, by choosing the model class as M = { P θ : θ ∈ Θ } and the algorithm space as D = { ALG : Y ⊗ n → A} . For model M = P θ and algorithm ALG , the distribution of the whole observation X ∼ P M, ALG is given by

The loss under model M is then measured by the loss of the decision A , i.e., L ( M,X ) := L ( θ, A ) .

2.2 Interactive decision making

For interactive decision making, we consider the following variant of the Decision Making with Structured Observations (DMSO) framework [40], which subsumes bandits and reinforcement learning. The learner interacts with the environment (described by an underlying model M ⋆ : Π → ∆( O ) , unknown to the learner) for T rounds. For each round t = 1 , ..., T :

The learner selects a decision π t ∈ Π , where Π is the decision space.

The learner receives an observation o t ∈ O via o t ∼ M ⋆ ( π t ) , where O is the observation space.

The underlying model M ⋆ is formally a conditional distribution, and the learner is assumed to have access to a known model class M⊆ (Π → ∆( O )) with the following property.

Assumption 1 (Realizability) . The model class M contains M ⋆ .

The model class M represents the learner's prior knowledge of the structure of the underlying environment. For example, for structured bandit problems, the models specify the reward distributions and hence encode the structural assumptions on the mean reward function (e.g. linearity, smoothness, or concavity). For a more detailed discussion, see Appendix B.

To each model M ∈ M , we associate a risk function L ( M, · ) : Π → R ≥ 0 , which measures the performance of a decision in Π under M . We consider two types of learning goals under the DMSO framework:

Generalized no-regret learning: The goal of the agent is to minimize the cumulative sub-optimality during the course of the interaction, given by

where π t can be randomly drawn from a distribution p t ∈ ∆(Π) chosen by the learner at step t .

Generalized PAC (Probably Approximately Correct) learning: the goal of the agent is to minimize the sub-optimality of a final output decision ̂ π (possibly randomized), which is selected by the learner once all T rounds of interaction conclude. We measure performance via

With an appropriate choice for L , the setting captures reward maximization (regret minimization) [40, 42], model estimation and preference-based learning [23], multi-agent decision making and partial monitoring [37], and various other tasks. In the main text, we focus on reward maximization and defer the results for more general choices L to the appendices (cf. Appendix B).

Example 1 (Reward maximization) . In the reward-maximization task, R : O → [0 , 1] is a known reward function. 1 For a model M ∈ M , E M,π [ · ] denotes expectation under the process o ∼ M ( π ) , and f M ( π ) := E M,π [ R ( o )] denotes the expected value function. For any M ∈ M , we let π M ∈ arg max π ∈ Π f M ( π ) be an optimal decision under M , and the risk function is defined by L ( M,π ) = f M ( π M ) -f M ( π ) , measuring the sub-optimality of the decision π under model M .

DMSO as an instance of ISDM. Any DMSO class ( M , Π) induces an ISDM as follows. For any t ∈ [ T ] , denote the full history of decisions and observations up to time t by H t -1 = ( π s , o s ) t -1 s =1 . The space of observations X consists of all X of the form X = ( H T , ̂ π ) , where ̂ π is a final decision. An algorithm ALG = { q t } t ∈ [ T ] ∪{ p } is specified by a sequence of mappings, where the t -th mapping q t ( · | H t -1 ) specifies the distribution of π t based on H t -1 , and the final map p ( · | H T ) specifies the distribution of the output decision ̂ π based on H T . The algorithm space D consists of all such algorithms. The loss function is chosen to be L ( M ⋆ , X ) = L ( M ⋆ , ̂ π ) for PAC learning (4), and L ( M ⋆ , X ) = ∑ T t =1 L ( M ⋆ , π t ) for no-regret learning (3). For any algorithm ALG and model M , P M, ALG ( · ) is the distribution of X = ( H T , ̂ π ) generated by the algorithm ALG under the model M , and we let E M, ALG [ · ] to be the corresponding expectation.

3 A General Lower Bound

In this section, we introduce our general lower bound technique, the interactive Fano method, and use it to provide minimax lower bounds for the ISDM framework.

Theorem 1 (Interactive Fano method) . Fix a f -divergence D f . Let ALG be a given algorithm, µ ∈ ∆( M ) be a given prior distribution over models, and ∆ > 0 be a given risk level. For any reference distribution Q ∈ ∆( X ) , we define

Then, the following lower bound holds:

where we denote d f,δ ( p ) = D f (Bern(1 -δ ) , Bern( p )) if p ≤ 1 -δ , and d f,δ ( p ) = 0 otherwise.

This result generalizes the classical Fano method in the prequel (as well as more sophisticated variants [97, 36, 27]) in multiple ways:

It encompasses general interactive learning/estimation problems in the ISDM framework, as opposed to purely passive estimation. This is reflected in the fact that the distribution over the outcome X is allowed to depend on ALG itself.

The most important and novel change is that Theorem 1 generalizes the 'hard' separation condition required in the classical Fano method to a 'soft' notion of separation captured by the quantile ρ ∆ , Q in Eq. (5). The quantile ρ ∆ , Q reflects the average separation under 'ghost data' X generated from an arbitrary reference distribution Q , which is independent of the true model M ∼ µ .

In addition, instead of relying on mutual information, which is can difficult to quantify for interactive problems, we use divergence with respect to the reference distribution Q , generalizing a central idea in Foster et al. [40, 42].

In what follows, we will show that these generalizations allow the Interactive Fano method to achieve two important desiderata: (1) unifying the methods of Fano, Le Cam, and Assouad (Section 3.1), and (2) integrating these traditional lower bound techniques with the DEC approach [40, 42] to derive new lower bounds (see Section 3.2).

3.1 Recovering non-interactive lower bounds

We begin by applying Theorem 1 to recover classical non-interactive lower bounds for statistical estimation. Since a goal of our paper is to integrate the Fano and Assouad methods with the DEC framework, this serves as an important sanity check to demonstrate that our framework can recover the non-interactive versions of these methods.

Fano method. We specialize Theorem 1 to the KL divergence. Observe that for any reference distribution Q ,

By choosing Q = E M ∼ µ P M, ALG in Theorem 1, we obtain the following proposition, which encompasses prior generalizations of Fano's inequality [97, 36, 27] developed in statistical estimation.

Proposition 2 (Recovering the generalized Fano method) . Fix an algorithm ALG and prior distribution µ ∈ ∆( M ) , and let I µ, ALG ( M ; X ) be the mutual information between M and X under M ∼ µ and X ∼ P M, ALG . The following Bayes risk lower bound holds for all ∆ ≥ 0 :

̸

When applied to the statistical estimation setting (Section 2.1), the classical Fano method corresponds to the special case of Proposition 2 where Θ = A = { 1 , 2 , . . . , m } , L ( θ, a ) = 1 ( θ = a ) is the indicator loss, µ = Unif(Θ) is the uniform prior, and ∆ = 1 .

Note that in Proposition 2, the term log sup x µ ( M ∈ M : L ( M,x ) < ∆) in the denominator of Eq. (6) takes the supremum over the outcome x , resulting in a simplified expression that removes the role of the algorithm ALG . This simplification is often sufficient to derive tight guarantees for estimation, but is insufficient for interactive decision making in general. The DEC, which we define in Section 3.2, more precisely accounts for the role of decisions selected by the algorithm.

Le Cam's method and Assouad's method. To recover Le Cam's two-point method and Assouad's method from Theorem 1, we appeal to the following result, which recovers a lower bound known as the Le Cam convex hull method [61, 96] which generalizes both approaches.

Proposition 3 (Recovering Le Cam's convex hull method) . For a parameter space Θ and observation space Y , consider a class of distributions P = { P θ | θ ∈ Θ } ⊆ ∆( Y ) indexed by Θ . Let L : Θ ×A→ R + be a loss function. Suppose Θ 0 ⊆ Θ and Θ 1 ⊆ Θ satisfy the separation condition

Then

where the infimum is taken over all algorithms ALG : Y ⊗ n → A , and P ⊗ n ν i is the distribution on Y ⊗ n induced by θ ∼ ν i , Y = ( Y 1 , . . . , Y n ) i . i . d ∼ P θ for i ∈ { 0 , 1 } .

Le Cam's convex hull method is the most general formulation of the Le Cam two-point method, which-in its most basic form-corresponds to the case in which ν 0 and ν 1 are singletons. The convex hull method is also capable of recovering Assouad's method [96]. It is important to note that the classical Fano's method, e.g. in the form of Proposition 2, cannot recover Proposition 3. This is because of fundamental differences between the divergences (KL versus TV) used in the traditional Fano method and the convex hull method.

3.2 Recovering DEC-based lower bounds for interactive decision making

Within the DMSO framework (Section 2.2), Foster et al. [40, 42] introduced the Decision-Estimation Coefficient (DEC) as a complexity measure, providing both upper and lower bounds for any model class M . Wenowshow how to recover the lower bounds of Foster et al. [40, 42] through Theorem 1. Wefocus on the lower bounds from Foster et al. [42], which are based on a variant of the DEC called the constrained DEC , and provide the tightest guarantees from prior work.

Background on the Decision-Estimation Coefficient. Consider the reward maximization setting (Example 1) under DMSO. For a model class M and a reference model Ď M : Π → ∆( O ) (not necessarily in M ), we define the constrained regret-DEC via

and define the constrained PAC-DEC via

Here, the superscript ' c ' indicates 'constrained', and the superscript ' r ' (resp. ' p ') indicates 'regret' (resp. 'PAC'). We further define

where co( M ) denotes the convex hull of the model class M .

Based on these complexity measures, Foster et al. [42] (see also Glasgow and Rakhlin [44]) provide the following lower and upper bounds on optimal risk and regret, under mild growth conditions on the DECs.

Theorem 4 (Informal; Foster et al. [42]) . Consider the reward maximization variant of the DMSO setting (Example 1). For any model class M and T ∈ N , the following lower and upper bounds hold: (1) For PAC learning,

where ε ( T ) ≍ √ 1 /T and ¯ ε ( T ) ≍ √ log |M| /T (up to logarithmic factors).

(2) For no-regret learning,

Therefore, up to the log |M| -gap between the parameters ε ( T ) and ¯ ε ( T ) appearing in the lower and upper bounds, the constrained PAC-DEC tightly captures the minimax risk of PAC learning, and the constrained regret-DEC captures the minimax regret of no-regret learning.

A new complexity measure: The quantile Decision-Estimation Coefficient. We recover the DEC-based lower bounds from Foster et al. [42] through a new variant we refer to as the quantile DEC . To do so, we briefly recount the proof technique used by Foster et al. [42].

Given an algorithm ALG , the proof strategy is to first fix an arbitrary reference model Ď M , then adversarially choose a hard alternative model M ∈ M (in a way that is guided by the DEC and the algorithm ALG itself) such that D TV ( P M, ALG , P Ď M, ALG ) is small, yet ALG cannot achieve low risk on model M . This lower bound technique does not explicitly require a separation condition between

M and Ď M , which is a departure from the classical Fano and two-point methods. Thus to recover it, the lack of an explicit separation condition in Theorem 1 will be critical. More precisely, for any model M , we consider the following distributions over decisions:

That is, q M, ALG is the expected empirical distribution over the decisions ( π 1 , · · · , π T ) played by the algorithm under M , and p M, ALG is the expected distribution of the final decision ̂ π .

With these definitions, we instantiate Theorem 1 with the Hellinger distance. We will use the subadditivity of Hellinger distance (Lemma C.1), which allows us to bound

Theorem 1 then yields the following intermediate result.

Lemma 5 (Recovering interactive two-point method) . Let δ ∈ [0 , 1] be given, and consider an algorithm ALG . Define

Using Lemma 5, as a starting point, we derive a new quantile-based variant of the DEC, which we will show can be viewed as a slight generalization of the original PAC DEC of Foster et al. [42].

For any model M ∈ M and any parameter δ ∈ [0 , 1] , we define the δ -quantile risk as follows:

this serves as a measure of the sub-optimality of the distribution p ∈ ∆(Π) in terms of δ -quantile. We now define the quantile PAC DEC as follows:

and define p-dec q ε,δ ( M ) := sup Ď M ∈ co( M ) p-dec q ε,δ ( M , Ď M ) . Applying Lemma 5, it is immediate to see that the quantile PAC-DEC provides a lower bound on the PAC risk.

Theorem 6 (Quantile DEC lower bound) . Let any T ≥ 1 and δ ∈ [0 , 1) be given, and define ε δ ( T ) := 1 14 √ δ T . Then, for any algorithm ALG , there exists M ⋆ ∈ M such that

Unlike the original constrained DEC lower bounds (Theorem 4), which are restricted to the reward maximization variant of the DMSO setting (Example 1), the quantile DEC lower bound in Theorem 6 holds for any risk function L . As a result, the lower bound applies to a broader range of generalized PAC learning tasks, including model estimation [23] and multi-agent decision making [37], where DEC-based lower bounds from prior work are loose in general; as a concrete application, we derive a new lower bound for interactive estimation (Example 3) in Appendix E.2.

Recovering DEC-based lower bounds using the quantile DEC. At first glance, Theorem 6 might appear to be weaker than the constrained PAC-DEC lower bound in Theorem 4 due to the loose conversion from quantile risk to expected risk. However, by specializing to reward maximization (Example 3) and leveraging the structure of the risk function L , we show that quantile PAC-DEC is equivalent to its constrained counterpart for this setting, leading to a tight lower bound.

Proposition 7 (Recovering the PAC DEC lower bound) . Under the reward maximization setting (Example 1), for any ε > 0 and δ ∈ [0 , 1) it holds that

As a corollary, we may choose δ = 1 2 and ε ( T ) = 1 20 √ T in Theorem 6, so that

Thus, the quantile PAC-DEC lower bound indeed recovers the constrained PAC-DEC lower bound in Theorem 4.

Our quantile DEC lower bound extends to regret with minor modifications, allowing us to recover the regret lower bounds in Theorem 4. We defer the details to the Appendix E.1 (Theorem E.1).

3.3 Recovering mutual information-based lower bounds for interactive decision making

The following result uses Theorem 1 to extend classical Fano method to interactive decision making and achieves tight dependence on the problem dimension that is not recovered by the standard DEC lower bound in Foster et al. [40, 42].

Proposition 8 (Mutual information-based lower bound) . Consider the DMSO setting. For any T ≥ 1 and prior µ ∈ ∆( M ) , we define the maximum T -round mutual information as

where we recall that I µ, ALG ( M ; H T ) is the mutual information between M and H T under M ∼ µ and H T ∼ P M, ALG , and the supremum is taken over all possible DMSO algorithms ALG . Then for any algorithm ALG ,

Using Proposition 8, along with mutual information bounds from Rajaraman et al. [70], we recover a Ω( d/ √ T ) PAC lower bound for d -dimensional linear bandits, which in turn recovers the Ω( d √ T ) regret lower bound [30, 73, 57, etc.].

Corollary 9. For d ≥ 2 , consider the d -dimensional linear bandit problem with decision space Π = { π ∈ R d : ∥ π ∥ 2 ≤ 1 } , parameter space Θ = { θ ∈ R d : ∥ θ ∥ 2 ≤ 1 } , and Gaussian rewards. The model class is M = { M θ } θ ∈ Θ , where for each θ ∈ Θ , the model M θ is given by M θ ( π ) = N ( ⟨ π, θ ⟩ , 1) . Then Proposition 8 implies a minimax risk lower bound:

In Section 4, we also instantiate Proposition 8 to derive a new complexity measure for DMSO.

4 Application to Interactive Decision Making: Bandit Learnability and Beyond

In this section, we focus on the DMSO setting and apply our general results (Theorem 1) to derive new lower and upper bounds for interactive decision making that go beyond the previous results based on the Decision-Estimation Coefficient [40, 42] by incorporating hardness of estimation.

Background: Gaps between DEC-based and upper and lower bounds. A fundamental open question of the DEC framework is whether the log |M| -gap between DEC lower and upper bounds in Theorem 4 can be closed. To highlight this gap in a more interpretable fashion, we re-state Theorem 4 in terms of a quantity we refer to as the minimax sample complexity . Let us focus on regret. Recall that for a fixed model class M , the following notion of minimax regret (2) is the central objective of interest:

Given a parameter ∆ > 0 , we define the minimax sample complexity

as the least value T for which there exists an algorithm that achieves ∆ T regret. Clearly, characterizing T ⋆ ( M , ∆) is equivalent to characterizing the minimax regret Reg ⋆ ( M , T ) .

Consider the following quantity induced by DEC for a class M and parameter ∆ > 0 :

With this definition, Theorem 4 is equivalent to the following characterization of the minimax sample complexity T ⋆ ( M , ∆) :

That is, Theorem 4 characterizes the minimax sample complexity up to a multiplicative log |M| factor. Our main result in this section will be to use the fractional covering number and interactive Fano method (Theorem 1), to (i) tighten the above characterization (16) for various special cases of interest, and (ii) give a new characterization for T ⋆ ( M , ∆) in structured bandit problems which avoids spurious parameters such as log |M| altogether.

4.1 New upper and lower bounds through the fractional covering number

For the a model class M and parameter ∆ > 0 , we define the fractional covering number as follows:

Informally, the fractional covering number N frac ( M , ∆) represents the best possible coverage over ∆ -optimal decisions that can be achieved through a single exploratory distribution in the face of an unknown model M ∈ M . As we will now show, this quantity naturally arises as a lower bound on optimal risk through the interactive Fano method. We begin with the following assumption.

Assumption 2 (Regular model class) . There exists a constant C KL > 0 and a reference model Ď M such that D KL ( M ( π ) ∥ Ď M ( π )) ≤ C KL for all M ∈ M and π ∈ Π .

Assumption 2 is a mild assumption on the boundedness of KL divergence. As an example, for structured bandits with means in [0 , 1] and Gaussian rewards, Assumption 2 holds with C KL = 1 2 . Details and more examples are provided in Appendix B.2. Our main lower bound based on the fractional covering number follows by specializing Theorem 1 to KL divergence.

Theorem 10 (Fractional covering number lower bound) . Suppose that M satisfies Assumption 2 with parameter C KL > 0 . Then for any algorithm ALG and ∆ > 0 , unless T ≥ log N frac ( M , ∆) -2 2 C KL , there exists M ⋆ ∈ M such that P M⋆ , ALG [ L ( M ⋆ , ̂ π ) ≥ ∆] ≥ 1 2 .

In particular, for (generalized) no-regret learning, fractional covering number also implies a regret lower bound through Theorem 10. That is, for any algorithm to achieve ∆ T -regret, it is necessary to have T = Ω(log N frac ( M , 2∆)) . Combining this with Theorem 4, we conclude that boundedness of both the DEC and the fractional covering number is necessary for learning with any model class M .

Upper bounds based on the fractional covering number. We now complement Theorem 10 by showing that for any reward maximization instance of the DMSO setting (Example 1), boundedness of the fractional covering number alone is also sufficient to derive upper bounds on the sample complexity of learning. The caveat is that while the lower bound is logarithmic in N frac ( M , ∆) , the upper bound will be polynomial.

Theorem 11 (Fractional covering number upper bound) . Consider the reward maximization task (Example 1). There exists an algorithm that for any class M and ∆ > 0 , ensures that with probability at least 1 -δ ,

Combining Theorem 10 and Theorem 11 yields the following bounds on T ⋆ ( M , ∆) (omitting polylogarithmic factors):

The gap between the lower and upper bounds of (18) is exponential; However, for model classes with C KL = O (1) , (18) suffices to characterize finite-time learnability . As a special case, we now show that fractional covering number characterizes the learnability of any structured bandit problem.

4.2 Application: Bandit learnability

Weconsider a structured bandit setting given by a reward function class H ⊆ (Π → [0 , 1]) . The protocol is as follows: For each round t ∈ [ T ] , the learner chooses a decision π t ∈ Π , then receives a reward r t ∼ N ( h ⋆ ( π t ) , 1) in response, where the mean reward function h ⋆ ∈ H . This corresponds to an instance of the DMSO framework with induced model class M H = { π ↦→N ( h ( π ) , 1) | h ∈ H} . We define the fractional covering number for H via

where we denote π h := arg max π ∈ Π h ( π ) . This exactly coincides with the notion of maximin volume of Hanneke and Yang [45], which was shown to give a tight characterization of learnability for the special case of noiseless binary-valued structured bandits. We discuss the connection to Hanneke and Yang [45] in more detail in Appendix G.2.

It is straightforward to show that for any structured bandit problem, the induced class M H satisfies Assumption 2 with C KL = 1 2 (Example 7). This leads to the following lower bound.

Corollary 12 (Lower bound for stochastic bandits) . For the bandit model class M H defined as above, it holds that T ⋆ ( M H , ∆) ≥ 2 log N frac ( H , ∆) -2 .

Combining this result with the upper bound in Theorem 11, we obtain the following bounds on the minimax-optimal sample complexity for the structure bandit problem with class H :

This implies that N frac ( H , ∆) characterizes learnability for structured bandits.

Theorem 13 (Structured bandit learnability) . For a given reward function class H , the bandit problem class M H is learnable for finite T if and only if N frac ( H , ∆) < + ∞ for all ∆ > 0 .

We remark that the lower and upper bound in Eq. (20) cannot be improved in terms of the fractional covering number alone: (1) For K -armed bandits, we have N frac ( H , ∆) ≤ K , meaning the upper bound is tight. (2) For d -dimensional linear bandits, we have log N frac ( H , ∆) = Ω( d ) , meaning the lower bound is nearly tight. Nevertheless, the exponential gap in Eq. (20) can be partly mitigated by combining the fractional covering number with the DEC, as we will show in Section 4.3.

Our characterization bypasses the impossibility results of Hanneke and Yang [45]. Specifically, Hanneke and Yang [45] show that for noiseless structured bandit problems, there exist classes H for which bandit learnability is independent of the axioms of ZFC. Therefore, their results rule out the possibility of a characterization of noiseless bandit learnability through any combinatorial dimension [11] for the problem class. Our characterization is compatible with this result because the argument of Hanneke and Yang [45] relies on the noiseless nature of the bandit problem, and hence does not preclude a characterization for the noisy setting. Additional discussion is deferred to Appendix G.2.

4.3 Improved upper bounds for general decision making

To close this section, we derive tighter upper bounds that scale with log N frac ( M , ∆) by combining the fractional covering number with the Decision-Estimation Coefficient. For simplicity of presentation, we focus on regret minimization under the setting of Example 1, and we assume the following condition to simplify our bounds (the fully general upper bound is detailed in Appendix G.3).

Assumption 3 (Regularity of constrained DEC) . A function d : [0 , 1] → R is said to have moderate decay if d ( ε ) ≥ 10 ε ∀ ε ∈ [0 , 1] , and there exists a constant c ≥ 1 such that c d ( ε ) ε ≥ d ( ε ′ ) ε ′ for all ε ′ ≥ ε . We assume the function ε ↦→ r-dec c ε (co( M )) , as a function of ε , satisfies moderate decay for a constant c reg ≥ 1 .

This condition essentially requires that the DEC for co( M ) exhibits moderate growth, which means that learning with co( M ) is not 'too easy'.We now state our upper bound, which tightens Theorem 4 by replacing the log |M| dependence in the upper bound with log N frac ( M , ∆) (with the caveat that the upper bound scales with the DEC for the convexified model class co( M ) ).

Theorem 14 (Upper bound with DEC and fractional covering number) . Consider the reward maximization variant of the DMSO setting. Let M be any class for which Assumption 3 holds, and assume that Π is finite. Let ¯ ε ( T ) ≍ √ log N frac ( M , ∆) /T . Then for any ∆ > 0 , Algorithm 1 (see Appendix G.3) ensures that with high probability, √

Restating this upper bound in terms of minimax sample complexity and combining it with the preceding lower bounds yields the following result.

Theorem 15. For any class M that satisfies Assumption 2 and 3, we have

up to dependence on c reg and logarithmic factors.

In particular, when the model class M is convex (i.e. co( M ) = M ) and C KL = O (1) , Theorem 15 provides lower and upper bounds for learning with M that match up to a quadratic factor. Indeed, for convex model classes, the upper bound of (21) is always tighter than (16) (and also tighter than the result in Foster et al. [40, 41]), as by definition we have

As applications, we apply Theorem 15 to structured bandits and contextual bandits (Appendix G).

References

Jacob Abernethy, Kareem Amin, Michael Kearns, and Moez Draief. Large-scale bandit problems and kwik learning. In International Conference on Machine Learning , pages 588-596, 2013.

Jayadev Acharya, Ziteng Sun, and Huanyu Zhang. Differentially private assouad, fano, and le cam. In Algorithmic Learning Theory , pages 48-78. PMLR, 2021.

Alekh Agarwal, Martin J Wainwright, Peter Bartlett, and Pradeep Ravikumar. Informationtheoretic lower bounds on the oracle complexity of convex optimization. Advances in Neural Information Processing Systems , 22, 2009.

Alekh Agarwal, Peter L Bartlett, Pradeep Ravikumar, and Martin J Wainwright. Informationtheoretic lower bounds on the oracle complexity of stochastic convex optimization. IEEE Transactions on Information Theory , 58(5):3235-3249, 2012.

Noga Alon, Shai Ben-David, Nicolo Cesa-Bianchi, and David Haussler. Scale-sensitive dimensions, uniform convergence, and learnability. Journal of the ACM , 44:615-631, 1997.

Noga Alon, Mark Bun, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private and online learnability are equivalent. ACM Journal of the ACM (JACM) , 69(4):1-34, 2022.

Ery Arias-Castro, Emmanuel J Candes, and Mark A Davenport. On the fundamental limits of adaptive sensing. IEEE Transactions on Information Theory , 59(1):472-481, 2012.

Patrice Assouad. Deux remarques sur l'estimation. Comptes rendus des séances de l'Académie des sciences. Série 1, Mathématique , 296(23):1021-1024, 1983.

Peter L Bartlett, Philip M Long, and Robert C Williamson. Fat-shattering and the learnability of real-valued functions. In Proceedings of the seventh annual conference on Computational learning theory , pages 299-310, 1994.

Shai Ben-David, David Pal, and Shai Shalev-Shwartz. Agnostic online learning. In Proceedings of the 22th Annual Conference on Learning Theory , 2009.

Shai Ben-David, Pavel Hrubeš, Shay Moran, Amir Shpilka, and Amir Yehudayoff. Learnability can be undecidable. Nature Machine Intelligence , 1(1):44-48, 2019.

James O Berger. Statistical Decision Theory and Bayesian Analysis . Springer Science & Business Media, 1985.

P.J. Bickel and K.A. Doksum. Mathematical Statistics: Basic Ideas and Selected Topics . Number v. 1 in Mathematical Statistics: Basic Ideas and Selected Topics. Prentice Hall, 2001. ISBN 9780138503635.

Lucien Birgé. Approximation dans les espaces métriques et théorie de l'estimation. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete , 65(2):181-237, 1983.

Lucien Birgé. On estimating a density using hellinger distance and some other strange facts. Probability theory and related fields , 71(2):271-291, 1986.

Anselm Blumer, Andrzej Ehrenfeucht, David Haussler, and Manfred K Warmuth. Learnability and the vapnik-chervonenkis dimension. Journal of the ACM (JACM) , 36(4):929-965, 1989.

Jean Bretagnolle and Catherine Huber. Estimation des densités: risque minimax. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete , 47:119-137, 1979.

Sébastien Bubeck, Ofer Dekel, Tomer Koren, and Yuval Peres. Bandit convex optimization: √ T regret in one dimension. In Conference on Learning Theory , pages 266278. PMLR, 2015.

Sébastien Bubeck, Jian Ding, Ronen Eldan, and Miklós Z Rácz. Testing for high-dimensional geometry in random graphs. Random Structures & Algorithms , 49(3):503-532, 2016.

Mark Bun, Roi Livni, and Shay Moran. An equivalence between private classification and online prediction. In 2020 IEEE 61st Annual Symposium on Foundations of Computer Science (FOCS) , pages 389-402. IEEE, 2020.

Rui M Castro and Robert D Nowak. Minimax bounds for active learning. IEEE Transactions on Information Theory , 54(5):2339-2353, 2008.

Nicolò Cesa-Bianchi, Pierre Gaillard, Claudio Gentile, and Sébastien Gerchinovitz. Algorithmic chaining and the role of partial feedback in online nonparametric learning. In Conference on Learning Theory , 2017.

Fan Chen, Song Mei, and Yu Bai. Unified algorithms for rl with decision-estimation coefficients: pac, reward-free, preference-based learning, and beyond. arXiv preprint arXiv:2209.11745 , 2022.

Fan Chen, Junyu Zhang, and Zaiwen Wen. A near-optimal primal-dual method for off-policy learning in cmdp. Advances in Neural Information Processing Systems , 35:10521-10532, 2022.

Fan Chen, Huan Wang, Caiming Xiong, Song Mei, and Yu Bai. Lower bounds for learning in revealing pomdps. arXiv preprint arXiv:2302.01333 , 2023.

Fan Chen, Constantinos Daskalakis, Noah Golowich, and Alexander Rakhlin. Near-optimal learning and planning in separated latent mdps. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 995-1067. PMLR, 30 Jun-03 Jul 2024.

Xi Chen, Adityanand Guntuboyina, and Yuchen Zhang. On bayes risk lower bounds. The Journal of Machine Learning Research , 17(1):7687-7744, 2016.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Thomas M Cover and Joy A Thomas. Elements of information theory . John Wiley & Sons, 1999.

Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.

Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory , pages 578-598. PMLR, 2021.

David L Donoho and Richard C Liu. Geometrizing rates of convergence, I. Technical Report 137a, Dept. Statistics, Univ. California, Berkeley, 1987.

David L Donoho and Richard C Liu. Geometrizing rates of convergence, II. The Annals of Statistics , pages 633-667, 1991.

David L Donoho and Richard C Liu. Geometrizing rates of convergence, III. The Annals of Statistics , pages 668-701, 1991.

John C Duchi. Lecture notes on statistics and information theory. 2024.

John C Duchi and Martin J Wainwright. Distance-based and continuum fano inequalities with applications to statistical estimation. arXiv preprint arXiv:1311.2669 , 2013.

Dean Foster, Dylan J Foster, Noah Golowich, and Alexander Rakhlin. On the complexity of multi-agent decision making: From learning in games to partial monitoring. In The Thirty Sixth Annual Conference on Learning Theory , pages 2678-2792. PMLR, 2023.

Dylan J Foster and Alexander Rakhlin. Beyond UCB: Optimal and efficient contextual bandits with regression oracles. arXiv preprint arXiv:2002.04926 , pages 3199-3210, 2020.

Dylan J Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. arXiv preprint arXiv:2010.03104 , 2020.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Dylan J Foster, Alexander Rakhlin, Ayush Sekhari, and Karthik Sridharan. On the complexity of adversarial decision making. arXiv preprint arXiv:2206.13063 , 2022.

Dylan J Foster, Noah Golowich, and Yanjun Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. In The Thirty Sixth Annual Conference on Learning Theory , pages 3969-4043. PMLR, 2023.

Dylan J Foster, Yanjun Han, Jian Qian, and Alexander Rakhlin. Online estimation via offline estimation: An information-theoretic framework. arXiv preprint arXiv:2404.10122 , 2024.

Margalit Glasgow and Alexander Rakhlin. Tight bounds for γ -regret via the decisionestimation coefficient. arXiv preprint arXiv:2303.03327 , 2023.

Steve Hanneke and Liu Yang. Bandit learnability can be undecidable. In The Thirty Sixth Annual Conference on Learning Theory , pages 5813-5849. PMLR, 2023.

Tatsunori Hashimoto, Steve Yadlowsky, and John Duchi. Derivative free optimization via repeated classification. In International Conference on Artificial Intelligence and Statistics , pages 2027-2036. PMLR, 2018.

Rafail Z Hasminskii and Ildar A Ibragimov. On the nonparametric estimation of functionals. In Proceedings of the Second Prague Symposium on Asymptotic Statistics , volume 473, pages 474-482. North-Holland Amsterdam, 1979.

Ildar Abdulovich Ibragimov and Rafail Zalmanovich Has'Minskii. Statistical estimation: asymptotic theory . Springer Science & Business Media, 1981.

TS Jayram. Hellinger strikes back: A note on the multi-party information complexity of and. In International Workshop on Approximation Algorithms for Combinatorial Optimization , pages 562-573. Springer, 2009.

Yassir Jedra and Alexandre Proutiere. Sample complexity lower bounds for linear system identification. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 26762681. IEEE, 2019.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Bandits and experts in metric spaces. Journal of the ACM (JACM) , 66(4):1-77, 2019.

Akshay Krishnamurthy, Alekh Agarwal, and John Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems , pages 1840-1848, 2016.

Tor Lattimore. Improved regret for zeroth-order adversarial bandit convex optimisation. Mathematical Statistics and Learning , 2(3):311-334, 2020.

Tor Lattimore and Andras Gyorgy. Mirror descent and the information ratio. In Conference on Learning Theory , pages 2965-2992. PMLR, 2021.

Tor Lattimore and Csaba Szepesvári. An information-theoretic approach to minimax regret in partial monitoring. In Conference on Learning Theory , pages 2111-2139. PMLR, 2019.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Tor Lattimore and Csaba Szepesvári. Exploration by optimisation in partial monitoring. In Conference on Learning Theory , pages 2488-2515. PMLR, 2020.

Lucien Le Cam. Asymptotic methods in statistical decision theory . Springer New York, NY, 1986.

Lucien Le Cam and Grace Lo Yang. Asymptotics in statistics: some basic concepts . Springer Science & Business Media, 2000.

Lucien LeCam. Convergence of estimates under dimensionality restrictions. The Annals of Statistics , pages 38-53, 1973.

Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics , 52(1):233-260, 2024.

Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine learning , 2(4):285-318, 1988.

Qinghua Liu, Alan Chung, Csaba Szepesvári, and Chi Jin. When is partially observable reinforcement learning not scary? arXiv preprint arXiv:2204.08967 , 2022.

Guanyu Nie, Mridul Agarwal, Abhishek Kumar Umrawal, Vaneet Aggarwal, and Christopher John Quinn. An explore-then-commit algorithm for submodular maximization under full-bandit feedback. In Uncertainty in Artificial Intelligence , pages 1541-1551. PMLR, 2022.

Ian Osband and Benjamin Van Roy. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732 , 2016.

Yury Polyanskiy and Yihong Wu. Dualizing le cam's method for functional estimation, with applications to estimating the unseens. arXiv preprint arXiv:1902.05616 , 2019.

Maxim Raginsky and Alexander Rakhlin. Information-based complexity, feedback and dynamics in convex programming. IEEE Transactions on Information Theory , 57(10):70367056, 2011.

Maxim Raginsky and Alexander Rakhlin. Lower bounds for passive and active learning. In Advances in Neural Information Processing Systems , pages 1026-1034, 2011.

Nived Rajaraman, Yanjun Han, Jiantao Jiao, and Kannan Ramchandran. Statistical complexity and optimal algorithms for non-linear ridge bandits. arXiv preprint arXiv:2302.06025 , 2023.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Philippe Rigollet and Assaf Zeevi. Nonparametric bandits with covariates. arXiv preprint arXiv:1003.1630 , page 54, 2010.

Paat Rusmevichientong and John N Tsitsiklis. Linearly parameterized bandits. Mathematics of Operations Research , 35(2):395-411, 2010.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems , volume 26, pages 2256-2264, 2013.

Daniel Russo and Benjamin Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Daniel Russo and Benjamin Van Roy. Learning to optimize via information-directed sampling. Operations Research , 66(1):230-252, 2018.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. arXiv preprint arXiv:2003.12699 , 2020.

MaxSimchowitz and Dylan Foster. Naive exploration is optimal for online lqr. In International Conference on Machine Learning , pages 8937-8948. PMLR, 2020.

Max Simchowitz, Kevin Jamieson, and Benjamin Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Conference on Learning Theory , pages 1794-1834. PMLR, 2017.

Max Simchowitz, Horia Mania, Stephen Tu, Michael I Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. In Conference On Learning Theory , pages 439-473. PMLR, 2018.

Alexandre B Tsybakov. Introduction to Nonparametric Estimation . Springer Publishing Company, Incorporated, 2008.

Sara A van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge university press, 2000.

Vladimir N. Vapnik and Alexey A. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of Complexity , 16(2):11, 1971.

John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. 1944.

Andrew Wagenmaker and Kevin Jamieson. Active learning for identification of linear dynamical systems. In Conference on Learning Theory , pages 3487-3582. PMLR, 2020.

Andrew Wagenmaker, Guanya Shi, and Kevin G Jamieson. Optimal exploration for modelbased rl in nonlinear systems. Advances in Neural Information Processing Systems , 36, 2024.

Andrew J Wagenmaker and Dylan J Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. In The Thirty Sixth Annual Conference on Learning Theory , pages 1322-1472. PMLR, 2023.

Andrew J Wagenmaker, Max Simchowitz, and Kevin Jamieson. Task-optimal exploration in linear dynamical systems. In International Conference on Machine Learning , pages 1064110652. PMLR, 2021.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics , pages 265-280, 1945.

Yuanhao Wang, Ruosong Wang, and Sham M Kakade. An exponential lower bound for linearly-realizable MDPs with constant suboptimality gap. Neural Information Processing Systems (NeurIPS) , 2021.

Gellért Weisz, Philip Amortila, and Csaba Szepesvári. Exponential lower bounds for planning in MDPs with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory , pages 1237-1264. PMLR, 2021.

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

Yunbei Xu and Assaf Zeevi. Bayesian design principles for frequentist sequential learning. In International Conference on Machine Learning , pages 38768-38800. PMLR, 2023.

Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Annals of Statistics , pages 1564-1599, 1999.

Bin Yu. Assouad, fano, and le cam. In Festschrift for Lucien Le Cam: research papers in probability and statistics , pages 423-435. Springer, 1997.

Tong Zhang. Information-theoretic upper and lower bounds for statistical estimation. IEEE Transactions on Information Theory , 52(4):1307-1321, 2006.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Ingvar Ziemann and Henrik Sandberg. Regret lower bounds for learning linear quadratic gaussian systems. IEEE Transactions on Automatic Control , 2024.

Contents of Appendix

A Related Work

To motivate our results, we briefly survey the most relevant lower bound techniques for estimation and decision making.

Minimax bounds for statistical estimation. There is a vast body of literature on minimax risk bounds for statistical estimation, including Hasminskii and Ibragimov [47], Bretagnolle and Huber [17], Birgé [15], Donoho and Liu [33], Cover and Thomas [29], Ibragimov and Has'Minskii [48], Tsybakov [81] as well as references therein. For minimax lower bounds, the most widely applied techniques are Le Cam's two-point method and convex-hull method [61], Assouad's lemma [8], and Fano's method [29]. Variants and applications of these three methods abound [2, 27, 67, 36]; Fano's method in particular has perhaps the largest number of variants, of which the most general version

we are aware of is due to Chen et al. [27], which is recovered by our interactive Fano method (cf. Proposition 2). Another celebrated thread, starting from the seminal work of Donoho and Liu [32], provides upper and lower bounds for a large class of non-parametric estimation problems based on Le Cam's two-point method through the study of a complexity measure known as the modulus of continuity [33, 34, 60, 67].

Lower bounds for interactive learning. There is a long line of work studying the fundamental limits of online learning and reinforcement learning (RL), including lower bounds for structured bandits [30, 73, 79, 57, 52, etc.], contextual bandits [72, 39, etc.], Markov Decision Processes (MDPs) [66, 31, 98, 92, 91, etc.], partially observable RL [53, 64, 25, 26, etc.], dynamical systems and control [80, 50, 78, 85, 99, etc.], and offline RL [71, 93, 88, 51, 24, 62, 86, etc.]. Most of these lower bounds are proven in a case-by-case basis, as the constructions of hard instances are specialized to the specific settings.

Decision-Estimation Coefficient. Toward a unifying understanding of the minimax complexity for interactive decision making problems, Foster et al. [40, 42] introduce Decision-Estimation Coefficient (DEC) as a complexity measure and show that it characterizes the minimax-optimal regret up to a log |M| factor. The DEC can be viewed as an interactive counterpart of the modulus of continuity [32], and captures hardness of interactive decision making related to exploration, but not necessarily estimation. An active line of research has built on the DEC to encompass a variety of more general decision making settings [40, 41, 23, 42, 37, 44], including adversarial decision making [41], PAC decision making [23, 42], reward-free learning and preference-based learning [23], and multi-agent decision making and partial monitoring [37].

However, there is a remaining gap between the DEC lower and upper bounds [40, 42], which closely relates to the complexity of estimation . Specifically, through the DEC framework, the sample complexity (number of rounds required to achieve ε -risk) is characterized as

where T DEC ( M , ε ) is a quantity measuring the complexity of exploration , 2 and Est ( M ) is a measure of the complexity of online estimation over M . The dependency on Est ( M ) can be necessary: For example, in linear bandits, the optimal sample complexity scales as d 2 /ε 2 , while T DEC ( M , ε ) ≍ d/ε 2 , and Est ( M ) ≍ d . However, the complexity of estimation Est ( M ) is missing from the DEC lower bound. This gap remains one of the main open questions in the DEC approach.

One potential reason is that the DEC lower bound does not recover Fano's method or Assouad's lemma, as it essentially generalizes Le Cam's two-point method. More specifically, while the statistical estimation task is subsumed by the DMSO framework, the DEC lower bound specialized to this setting at best recovers Le Cam's two-point method. On the other hand, the Ω( d 2 /ε 2 ) lower bound for linear bandits is typically proven through Assouad's lemma [18] or Fano's method [70], similar to its statistical estimation analog. Therefore, to close the remaining gap in the DEC approach, it is necessary to have a deeper understanding of the latter two methods in the interactive setting.

Additional related work. A large portion of the aforementioned lower bounds for interactive learning are proven using (variants of) the two-point method and can be recovered by the DEC lower bound approach [40, 42]. Beyond the two-point method, comparatively fewer lower bounds for interactive learning have been established using Assouad's lemma or Fano's method [21, 72, 3, 69, 39, 78, 70, etc.]. The approaches in these papers are specialized to the specific settings under consideration, and there is not a general principle through which Fano's method or Assouad's lemma can be lifted to handle interactivity. Indeed, the challenge of applying Fano's method in interactive contexts has been highlighted in various prior works, e.g., Arias-Castro et al. [7, Section 1.3] and Rajaraman et al. [70, Section 1.5.4].

The DEC is also closely related to a Bayesian complexity measure known as the information ratio [75, 76, 56, 55, etc.],which was originally introduced to analyze Bayesian algorithms such as posterior sampling. It is also related to a more recent generalization known as the algorithmic information ratio (AIR) [94], developed for frequentist algorithms. Additionally, the DEC is connected to asymptotic instance-dependent complexity, as explored by [87].

B Additional Background on DMSO Framework

The DMSO framework (Section 1.1) encompasses a wide range of learning goals beyond the reward maximization setting [40, 42], including reward-free learning, model estimation, and preferencebased learning [23], and also multi-agent decision making and partial monitoring [37]. We provide two examples below for illustration.

Example 2 (Preference-based learning) . In preference-based learning, each model M ∈ M is assigned with a comparison function C M : Π × Π → R (where C M ( π 1 , π 2 ) typically the probability of τ 1 ≻ τ 2 for τ 1 ∼ ( M,π 1 ) , τ 2 ∼ ( M,π 2 ) ), and the risk function is specified by L ( M,π ) = max π ⋆ C M ( π ⋆ , π ) . Chen et al. [23] provide lower and upper bounds for this setting in terms of Preference-based DEC (PBDEC).

Example 3 (Interactive estimation) . In the setting of interactive estimation (a generalized PAC learning goal), each model M ∈ M is assigned with a parameter θ M ∈ Θ , which is the parameter that the agent aims to estimate. The decision space Π = Π 0 × Θ , where each decision π ∈ Π consists of π = ( π 0 , θ ) , where π 0 is the explorative policy to interact with the model 3 , and θ is the estimator of the model parameter. In this setting, we define L ( M,π ) = Dist( θ M , θ ) for certain distance Dist( · , · ) .

This setting is an interactive version of the statistical estimation task, and it is also a generalization of the model estimation task studied in Chen et al. [23]. Natural examples include estimating some coordinates of the parameter θ in linear bandits. We provide nearly tight guarantee for this setting in Appendix E.2.

Applicability of our results Our general interactive Fano method Lemma 5 applies to any generalized no-regret / PAC learning goal (Section 1.1). Therefore, our risk lower bound in terms of quantile PAC-DEC Theorem 6 and fractional covering number lower bound Theorem 10 both apply to any generalized learning goal. For a concrete example, see Appendix E.2 for the application to interactive estimation.

B.1 Examples for statistical estimation (Section 2.1)

Example 4 (Mean estimation) . For the mean estimation task, the parameter space Θ ⊆ R d , and for each θ ∈ Θ , P θ = N ( θ, I d ) . The goal is to estimate the ground truth parameter θ ⋆ , i.e., the decision space is A = R d , and the loss is given by L ( θ, A ) = ∥ θ -A ∥ , where ∥·∥ is a norm over R d .

Example 5 (Functional estimation) . In the functional estimation task, a function T : Θ → R is given, and the goal is to estimate the value of T ( θ ⋆ ) , i.e., the decision space is A = R , and the loss is L ( θ, A ) = | T ( θ ) -A | .

Example 6 (Density estimation) . In the density estimation task, the goal is to estimate P θ ⋆ , i.e., the decision space A ⊆ ∆( Y ) , and the loss is given by L ( θ, A ) = D ( P θ , A ) , where D is a certain divergence (e.g., TV distance or KL divergence).

B.2 Examples for Assumption 2

In this section, we provide three general types of model classes where Assumption 2 holds with mild C KL . It is worth noting that in Assumption 2, the reference model Ď M does not necessarily belong to co( M ) .

Example 7 (Gaussian bandits) . Suppose that H ⊆ ( A → [0 , 1]) is a class of mean value function, and M H , V is the class of the model M associated with a h M ∈ H :

Then, consider the reference model Ď M given by Ď M ( π ) = N (0 , 1) ∀ π ∈ A . It is clear that for any π , and model M ∈ M H , V ,

and hence Assumption 2 holds with C KL = 1 2 .

Example 8 (Problems with finite observations) . Suppose that the observation space O is finite. Then, consider the reference model Ď M given by Ď M ( π ) = Unif( O ) ∀ π ∈ Π . It holds that

and hence Assumption 2 holds with C KL = log |O| .

Example 8 can further be generalized to infinite observation space, as long as every model in M admits a bounded density function with respect to the same base measure.

Example 9 (Contextual bandits) . Suppose that H ⊆ ( C × A → [0 , 1]) is a class of mean value function, and M H , V is the class of the model M specified by a value function h M ∈ H and a context distribution ν M ∈ ∆( C ) . More specifically, for any π ∈ Π = ( C → A ) , M ( π ) is the distribution of ( c, a, r ) , generated by c ∼ ν M , a = π ( c ) , and r ∼ N ( h M ( c, a ) , 1) .

Then, consider the reference model Ď M specified by ν Ď M = Unif( C ) and h Ď M ≡ 0 . It is clear that for any π , and model M ∈ M H , V ,

and hence Assumption 2 holds with C KL = log |C| +1 .

The factor of log |C| in Example 9 is due to the definition (45) of log N frac ( H , ∆) , where we take supremum over all context distribution µ . This factor can be removed if we instead restrict the model class to have a common context distribution (i.e., the setting where context distribution is known or can be estimated from samples).

C Technical Tools

The following lemma is the 'chain rule' of Hellinger distance [49] (see also Duchi [35, Lemma 11.5.3] and Foster et al. [43, Lemma D.2]).

Lemma C.1 (Sub-additivity for squared Hellinger distance) . Let ( X 1 , F 1 ) , . . . , ( X T , F T ) be a sequence of measurable spaces, and let X t = ∏ t i =1 X i and F t = ⊗ t i =1 F i . For each t , let P t ( · | · ) and Q t ( · | · ) be probability kernels from ( X t -1 , F t -1 ) to ( X t , F t ) .

Let P and Q be the laws of X 1 , . . . , X T under X t ∼ P t ( · | X 1: t -1 ) and X t ∼ Q t ( · | X 1: t -1 ) respectively. Then it holds that

In particular, given a T -round algorithm ALG and a model M , we can consider random variables X 1 = ( π 1 , o 1 ) , · · · , X T = ( π T , o T ) . Then, P M, ALG ( X t = · | X 1: t -1 ) is the distribution of ( π t , o t ) , where π t ∼ p t ( · | π 1 , o 1 , · · · , π t -1 , o t -1 ) , and o t ∼ M ( π t ) . Therefore, applying Lemma C.1 to D 2 H ( P M, ALG , P Ď M, ALG ) gives the following corollary.

Corollary C.2. For any T -round algorithm ALG , it holds that

Lemma C.3 (Foster et al. [40, Lemma A.4]) . For any sequence of real-valued random variables ( X t ) t ≤ T adapted to a filtration ( F t ) t ≤ T , it holds that with probability at least 1 -δ , for all t ≤ T ,

Lemma C.4. For any pair of random variable ( X,Y ) , it holds that

Lemma C.5. Suppose that for a random variable X , its mean and variance under P is µ P and σ 2 P , and its mean and variance under Q is µ Q and σ 2 Q . Then it holds that

In particular, when µ P , µ Q , σ P , σ Q ∈ [0 , 1] , we have D 2 H ( P , Q ) ≥ 1 10 | µ P -µ Q | 2 .

On the other hand, when P = N ( µ P , 1) , Q = N ( µ Q , 1) , then D 2 H ( P , Q ) ≤ 1 8 | µ P -µ Q | 2 .

Proof. Let ν = P + Q 2 be the common base measure and set µ = µ P + µ Q 2 . Then

D Proofs from Section 3

In this section, we present proofs for the results in Section 3, except Section 3.2.

Before proceeding to proofs, we first discuss the classical Fano's method to motivate our approach.

D.1 Additional background on classical Fano's method

To motivate our approach, which can be viewed as a generalization of the classical Fano method [29], let us briefly recall the classical approach and highlight some shortcomings. The classical Fano method applies to the statistical estimation setting Section 2.1 (a special case of ISDM), and takes the following form.

Proposition D.1 (Classical Fano method) . Consider the statistical estimation setting (Section 2.1) with parameter space Θ . Suppose that there exist θ 1 , . . . , θ m ∈ Θ such that the following separation condition holds:

̸

Let µ be the uniform distribution over { θ 1 , · · · , θ m } , and let I µ ( θ ; Y ) denote the mutual information of ( θ, Y ) ∼ P µ generated by θ ∼ µ and Y = ( Y 1 , . . . , Y n ) i . i . d ∼ P θ . Then for any algorithm ALG , we have

where the binary KL divergence is defined as kl( p ∥ q ) = D KL (Bern( p ) ∥ Bern( q )) . This implies the minimax lower bound

The log m factor in Eq. (25) reflects the complexity of estimation in the parameter space, which is a key concept we aim to incorporate into interactive decision making. Looking deeper, the 'estimation complexity' term log m in Eq. (25) arises from the "quantile" parameter 1 /m appearing in the Eq. (24). This parameter reflects the fact that under the separation condition (23), the following quantile probability is at most 1 /m for any distribution Y ∼ Q :

Note that in this expression, θ is drawn from the uniform prior µ and Y is drawn independently of θ . To deduce Eq. (24), it suffices to choose Q = E θ ∼ µ P θ and apply data-processing inequality:

In particular, for any δ ∈ (0 , 1) such that I µ ( θ ; Y ) ≤ kl(1 -δ ∥ 1 /m ) , we have P µ ( L ( θ, ALG ( Y )) < ∆) ≤ 1 -δ , using the monotonicity of the KL divergence. This argument gives Eq. (24) immediately, and by choosing δ ⋆ = 1 -I µ ( θ ; Y )+log 2 log m in Eq. (24), we arrive in the canonical statement in Eq. (25).

To summarize, the structure of the classical Fano lower bound involves (i) a prior µ , (ii) a reference distribution Q = E θ ∼ µ P θ , and (iii) a quantile parameter δ determined by the (iv) separation condition (23) in the argument above. Crucially, we understand that the complexity of estimation log m arises from the quantile probability P θ ∼ µ,Y ∼ Q ( L ( θ, ALG ( Y )) < ∆) , and the only use of the traditional separation condition (23) is to further bound this quantile by 1 /m as in Eq. (26).

Having gained these insights into classical Fano method, we would like to point out several limitations:

First, in the form above, it is only applicable to statistical estimation rather than general interactive settings.

Second, it relies on mutual information due to the choice of the reference distribution, which depends on the evolution of the algorithm over all T rounds in interactive settings, making it difficult to analyze in many interactive problems.

Third, and perhaps most importantly, the separation condition (23) must hold for an arbitrary decision a . This 'hard' separation condition is unlikely to hold for general model classes, as noted throughout the DEC approach [42, Remark 2.3] line of work.

To address these shortcomings, we make use of core concepts (prior, reference distribution, quantile parameter, separation condition) above, but adopt a new perspective that emphasizes and generalizes the role of the quantile probability P θ ∼ µ,Y ∼ Q ( L ( θ, ALG ( Y )) < ∆) .

D.2 Proof of Theorem 1

In the following, we fix a prior µ ∈ ∆( M ) , parameter ∆ > 0 , f -divergence D f , and an algorithm ALG . For simplicity, we denote D f ( x, y ) = D f (Bern( x ) , Bern( y )) for x, y ∈ [0 , 1] .

We first prove the following quantile lower bound:

We only need to prove the following claim.

Claim. Suppose that there exists a reference distribution Q such that

then P M ∼ µ,X ∼ P M, ALG ( L ( M,X ) ≥ ∆) ≥ δ .

We denote ¯ ρ ∆ = P M ∼ µ,X ∼ P M, ALG ( L ( M,X ) < ∆) , and recall that we define ρ ∆ , Q = P M ∼ µ,X ∼ Q ( L ( M,X ) < ∆) . We then consider the following two distributions over M×X :

By the data processing inequality of f -divergence, we have

Therefore, using d f,δ ( ρ ∆ , Q ) > E M ∼ µ D f ( P M, ALG , Q ) , we know that d f,δ ( ρ ∆ , Q ) > D f (¯ ρ ∆ , ρ ∆ , Q ) . In particular, this implies ρ ∆ , Q < 1 -δ , and

Hence, we consider two cases: (1) ¯ ρ ∆ ≤ ρ ∆ , Q , and (2) ¯ ρ ∆ > ρ ∆ , Q . For case (1), we have ¯ ρ ∆ ≤ ρ ∆ , Q < 1 -δ . For case (2), we can use the monotone property of D f (Lemma D.2), which also implies ¯ ρ ∆ < 1 -δ .

Therefore, it holds that ¯ ρ ∆ < 1 -δ , and

Hence, the proof of Eq. (27) is completed. The in-expectation lower bounds then follows from the fact that

Lemma D.2. For x, y ∈ [0 , 1] , the quantity D f ( x, y ) is increasing with respect to x when x ≥ y .

Proof of Lemma D.2 Fix any 1 ≥ x > z ≥ y ≥ 0 , we define

Then, by definition, and by definition, we have

By Theorem 1, for any δ ∈ (0 , 1) such that I µ, ALG ( M ; X ) < kl(1 -δ ∥ ρ ∆ , Q ) , we have

and hence for the channel P from { 0 , 1 } to itself given by P ( ·| 0) = Bern( p ) , P ( ·| 1) = Bern( q ) , it holds that

Therefore, by data-processing inequality, we have

This is the desired result.

D.3 Proof of Proposition 2

Fix the parameter ∆ > 0 and let µ ∈ ∆( M ) be given. To apply Theorem 1, we consider KL divergence (corresponding to f ( x ) = x log x ) and choose the reference distribution

Then, by the choice of Q and definition of KL-divergence, we have

In particular, we may choose

As long as δ ⋆ > 0 , we have I µ, ALG ( M ; X ) < kl(1 -δ ⋆ ∥ ρ ∆ , Q ) , and hence E M ∼ µ E X ∼ P M, ALG [ L ( M,X )] ≥ δ ⋆ ∆ . This gives the desired lower bound (note that if δ ⋆ ≤ 0 , there is nothing to prove).

D.4 Proof of Proposition 3

We recover this result by applying Theorem 1 with TV distance. We first frame the problem in the ISDM framework. Consider the 'enlarged' model class M = { M ν : ν ∈ ∆(Θ) } , where for any algorithm ALG : Y ⊗ n →A , the distribution P M ν , ALG is given by

In other words, P M ν , ALG is the distribution induced by the prior ν and the algorithm ALG . We then extend the loss function to L : M×X → R + as

By the separation condition (7), we have L ( M ν 0 , X ) + L ( M ν 1 , X ) ≥ 2∆ for any ν 0 ∈ ∆(Θ 0 ) , ν 1 ∈ ∆(Θ 1 ) . Therefore, choosing the prior µ = Unif( { M ν 0 , M ν 1 } ) and the reference distribution Q = E M ∼ µ P M, ALG gives

and by the data-processing inequality,

Therefore, for any 0 ≤ δ < 1 2 -1 2 D TV ( P ⊗ n ν 0 , P ⊗ n ν 1 ) , we have

and hence applying Theorem 1 gives

Taking the supremum over ν ∈ ∆(Θ ) and ν ∈ ∆(Θ )

0 0 1 1 gives the desired result.

We can use the Le Cam convex hull method to recover the classic two-point method and Assouad's method, as follows.

Example 10 (Le Cam's two-point method) . In Proposition 3, we can take the distribution ν 0 ( ν 1 ) to be supported on a single point in Θ 0 ( Θ 1 ), to recover the classical two-point method. Concretely, under the setting and assumption of Proposition 3, we have the following two-point lower bound:

where P ⊗ n θ is the distribution of Y = ( Y 1 , · · · , Y n ) i . i . d ∼ P θ .

◁

Example 11 (Assouad's method) . Suppose that Θ = {-1 , 1 } d for some d ≥ 1 , and that the loss function has the following coordinate-wise structure:

We write θ ∼ j θ ′ if θ and θ ′ differ only in the j -th coordinate. Assume that the following separation condition holds for all j ∈ [ d ] :

To apply Proposition 3, we consider Θ j i = { θ : θ j = i } for i ∈ { 0 , 1 } and j ∈ [ d ] . Then, for each j ∈ [ d ] , the separation condition (7) holds for the loss L j and the subsets Θ j 0 , Θ j 1 . Therefore, applying Proposition 3 for each j ∈ [ d ] with ν j 0 = Unif(Θ j 0 ) and ν j 1 = Unif(Θ j 1 ) gives the following Assouad-type lower bound:

◁

D.5 Proof of Corollary 9

Consider the following setup of linear bandits: let θ ⋆ ∈ R d be an unknown parameter. At time t , the learner chooses an action π t ∈ { π ∈ R d : ∥ π ∥ 2 ≤ 1 } and receives a Gaussian reward r t ∼ N ( ⟨ θ ⋆ , π t ⟩ , 1) . For T ∈ N , let H T = ( π 1 , r 1 , · · · , π T , r T ) be the observed history up to time T . The central claim of this section is the following upper bound on the mutual information.

Theorem D.3. For any r > 0 , we define the prior µ r over B d ( r ) by

Then for any algorithm ALG , we have

Proof. Denote λ = r 2 4 . We first prove that if θ ⋆ ∼ µ = N (0 , λI d /d ) , then

By the Bayes rule, the posterior distribution of θ ⋆ conditioned on ( H t -1 , π t ) is

which is a Gaussian distribution with covariance (Σ t -1 ) -1 , where

Therefore, by the chain rule of mutual information, we have

which is exactly Eq. (29).

Next we deduce the claimed result from Eq. (29). Consider the random variable Z = 1 {∥ θ ⋆ ∥ 2 ≤ r } ∈ { 0 , 1 } , and then

Therefore,

With the upper bound (30) of ρ ∆ , we know that for ∆ = 1 2 , it holds

as long as c 0 is a sufficiently small constant. Therefore, Proposition 8 gives that

This completes the proof of the first inequality.

Finally, using the fact that P θ ⋆ ∼ µ ( ∥ θ ⋆ ∥ ≤ c 1 r ) ≤ 1 100 for a small absolute constant c 1 , we can conclude that

This is the desired result.

Here the first inequality is Eq. (29), the second inequality follows from I ( X ; Y ) -I ( X ; Y | f ( X )) = I ( f ( X ); Y ) -I ( f ( X ); Y | X ) ≥ 0 , the third identity follows from the definition of conditional mutual information. Finally, noticing that P µ ( ∥ θ ⋆ ∥ 2 ≤ r ) ≥ 1 2 by concentration of χ 2 d random variable, we arrive at the desired statement.

Next we show how to translate the mutual information upper bound in Theorem D.3 to lower bounds of estimation and regret.

Theorem D.4. Let T ≥ 1 , r = min { c 0 d √ T , 1 } for a small absolute constant c 0 , and consider the prior µ = µ r . For any T -round algorithm with output ̂ π , Proposition 8 implies that

Therefore, we may deduce that

where we denote normalize ( θ ) = θ ∥ θ ∥ ∈ B d (1) . Notice that for θ ∈ Θ , we have

For ∆ ∈ (0 , 1) , we first claim that

To see so, by symmetry of Gaussian distribution, we know for fixed any π ∈ R d ,

and hence we can instead consider the uniform distribution over the sphere S d -1 . By rotational invariance, we may assume that π = ( x, 0 , · · · , 0) , with x ≥ 0 . Then

By Bubeck et al. [19, Section 2], for θ ∼ Unif( S d -1 ) , the density of θ 1 ∈ [ -1 , 1] is given by

E Additional Results from Section 3.2

In addition to the reward-maximization setting (Example 1), we also introduce a slightly more general setting. In this setting, we assume that for each model M ∈ M , the risk function is L ( M,π ) = f M ( π M ) -f M ( π ) , but f M is not assumed to be the expected reward function (Example 1). Instead, we only require f M satisfying the following assumption, where M + ⊆ (Π → ∆( O )) is a prespecified model class of reference models that contains co( M ) (following Foster et al. [42]). The lower bound we prove can be stronger by allowing M + to be a larger model class.

Assumption 4. Let M + ⊆ (Π → ∆( O )) be a given class of reference models, such that co( M ) ⊆ M + . For any M ∈ M , the risk function takes form L ( M,π ) = f M ( π M ) -f M ( π ) for some functional f M : Π → R , so that f M can be extended to M + , such that for any model M ∈ M and reference model Ď M ∈ M + we have

In some cases, considering a larger reference model class M + can be convenient for proving lower bounds, see e.g., Appendix B.2 and Appendix H.7.

E.1 Recovering DEC-based regret lower bounds

In this section, we demonstrate how our general lower bound approach recovers the regret lower bounds of Foster et al. [42], Glasgow and Rakhlin [44]. We first state our lower bound in terms of constrained DEC in the following theorem.

Theorem E.1. Under the reward maximization setting (Example 1), for any T -round algorithm ALG , there exists M ⋆ ∈ M such that

with probability at least 0 . 01 under P M⋆ , ALG , where ε ( T ) = 1 100 √ T .

Theorem E.1 immediately yields an in-expectation regret lower bound in terms of constrained DEC. It also shaves off the unnecessary logarithmic factors in the lower bound of Foster et al. [42, Theorem 2.2].

For the remainder of this section, we sketch how we prove Theorem E.1 in a slightly more general setting (Assumption 4), following Appendix F.3. Before providing our regret lower bounds, we first present several important definitions.

Definition of quantile regret-DEC We note that it is possible to directly modify the definition of quantile PAC-DEC (12), and then apply Theorem 6 to obtain an analogous regret lower bound immediately. However, as Foster et al. [42] noted, the 'correct' notion of regret-DEC (cf. Eq. (8)) turns out to be more sophisticated. Therefore, we define the quantile version of regret-DEC similarly, as follows.

Throughout the remainder of this section, we fix the integer T . Define

i.e., Π T is the class of all T -round mixture decision. We introduce the mixture decision space Π T here to handle the average of T -round profile ( π 1 , · · · , π T ) of the algorithm. In particular, when Π is convex, we may regard Π T = Π .

Next, we define the quantile regret-DEC as

The following proposition relates our quantile regret-DEC to the constrained regret-DEC (proof in Appendix F.6).

Proposition E.2. Suppose that Assumption 4 holds for M . Then, for any Ď M ∈ M + , it holds that r-dec c ( M∪{ Ď M } , Ď M ) ≤ 2 · r-dec q ( M , Ď M ) + c δ C r ε,

where we denote c δ = max { δ 1 -δ , 1 } . In particular, it holds that

Lower bound with quantile regret-DEC Now, we prove the following lower bound for the regret of any T -round algorithm, via our general interactive Fano method (Lemma 5). The proof is presented in Appendix F.5.

Theorem E.3. Suppose that Assumption 4 holds for M . Then, for any T -round algorithm ALG , parameters ε, δ, C > 0 , there exists M ∈ M such that

As a corollary, there exists M ⋆ ∈ M such that

with probability at least 0 . 01 under P M⋆ , ALG , where ε ( T ) = 1 100 √ T .

Theorem E.1 is now an immediate corollary, because for reward-maximization setting, we always have C r = √ 2 in Assumption 4.

E.2 Results for interactive estimation

More generally, we show that for a fairly different task of interactive estimation (Example 3), we also have an equivalence between quantile PAC-DEC with constrained PAC-DEC.

Recall that in this setting, each model M ∈ M is assigned with a parameter θ M ∈ Θ , which is the parameter that the agent want to estimate. The decision space Π = Π 0 × Θ , where each decision π ∈ Π consists of π = ( π 0 , θ ) , where π 0 is the explorative decision to interact with the model, and θ is the estimator of the model parameter. The risk function is then defined as L ( M,π ) = ρ ( θ M , θ ) , for certain distance ρ ( · , · ) .

In interactive estimation, we can show that the quantile DEC is in fact lower bounded the constrained DEC, as follows (proof in Appendix F.7).

Proposition E.4. Consider the setting of Example 3. Then as long as δ < 1 2 , it holds that

In particular, for such a setting (which encompasses the model estimation task considered in Chen et al. [23]), Theorem 6 provides a lower bound of estimation error in terms of constrained PACDEC. This is significant because the constrained PAC-DEC upper bound in Theorem 4 is actually not restricted to Example 1, and we have hence shown that

where ε ( T ) ≍ √ 1 /T and ¯ ε ( T ) ≍ √ log |M| /T . Therefore, for interactive estimation, constrained PAC-DEC is also a nearly tight complexity measure.

Remark E.5. The log |M| -gap between the lower and upper bound can further be closed for convex model class, utilizing the upper bounds in Appendix G.3. More specifically, we consider a convex model class M , where M ↦→ θ M is a convex function on M . Then, a suitable instantiation of ExO + (Algorithm 1) achieves

where N (Θ , ∆) is the ∆ -covering number of Θ , because we have log N frac ( M , ∆) ≤ log N (Θ , ∆) by considering the prior q = Unif(Θ 0 ) for a minimal ∆ -covering of Θ . Similar to Theorem H.5, we can upper bound p-dec o γ ( M ) by p-dec c ε ( M ) . Taking these pieces together, we can show that under the assumption that p-dec c ε ( M ) is of moderate decay, ExO + achieves

In particular, for the (non-interactive) functional estimation problem (see e.g. Polyanskiy and Wu [67]), the parameter space Θ ⊂ R , and hence by considering covering number, we have log | Θ | = ˜ O (1) . Therefore, for convex M , under mild assumption that the DEC is of moderate decaying (Assumption 3), the minimax risk is then characterized by (up to logarithmic factors)

This result can be regarded as a generalization of Polyanskiy and Wu [67] to the interactive estimation setting.

F Proofs from Section 3.2 and Appendix E

Additional notations For notational simplicity, for any distribution q ∈ ∆(Π) and reference model Ď M , we denote the localized model class around Ď M as

F.1 Proof of Lemma 5

To apply Theorem 1, we consider the squared Hellinger distance (which we recall is a f -divergence corresponding to f ( x ) = 1 2 ( √ x -1) 2 ). Consider a fixed tuple ( Ď M,M, ∆) with M ∈ M , Ď M ∈ co( M ) , and ∆ ≥ 0 that satisfies

We choose the reference distribution to be Q = P Ď M, ALG , and take µ to be the singleton distribution supported on M . Recall that for the DMSO framework, the observation is X = ( H T , ̂ π ) , and the loss function is L ( M,X ) = L ( M, ̂ π ) (Section 2.2). Then, by definition, we have

Further, using the sub-additivity of Hellinger distance (11), we have

Therefore, using the condition (33), we have

Hence, it holds that D 2 H ( P M, ALG , Q ) < D 2 H (1 -δ, ρ ∆ , Q ) , and applying Theorem 1 gives P M, ALG ( L ( M, π ) ≥ ∆) ≥ δ.

Taking supremum over all pairs ( Ď M,M, ∆) satisfying Eq. (33) gives the desired lower bound.

F.2 Proof of Theorem 6

Fix any algorithm ALG and abbreviate ε = ε δ ( T ) . Take an arbitrary parameter ∆ 0 < p-dec q ε,δ ( M ) . Then there exists Ď M such that ∆ 0 < p-dec q ε,δ ( M , Ď M ) . Hence, by the definition (12), we know that

Therefore, there exists M ∈ M such that

This immediately implies

where √ δ 1 = √ δ -√ 14 Tε 2 . Notice that δ 1 > δ 2 by the choice of ε = 1 14 √ δ T , and hence applying Lemma 5 shows that there exists M ∈ M such that P M, ALG ( L ( M, ̂ π ) ≥ ∆ 0 ) ≥ δ 2 . Letting ∆ 0 → p-dec q ε,δ ( M ) completes the proof.

F.3 Proof of Proposition 7

In this section, we prove Proposition 7 under the slightly more general setting of Assumption 4.

Proposition F.1. Under Assumption 4, for any reference model Ď M ∈ M + and ε > 0 , δ ∈ [0 , 1) , it holds that

For Example 1, we always have C r ≤ √ 2 , and hence Proposition 7 follows immediately from Proposition F.1.

Proof of Proposition F.1. Fix a reference model Ď M and a ∆ 0 > p-dec q ε,δ ( M , Ď M ) . Then, we pick a pair (¯ p, ¯ q ) such that

whose existence is guaranteed by the definition of p-dec q ε,δ ( M , Ď M ) in (12). In other words, we have

Consider q = ¯ p +¯ q 2 and ε ′ = ε √ 2 . Also let

By definition, for π ∼ p we have f ˜ M ( π ) ≥ f ˜ M ( π ˜ M ) -∆ 0 , and hence

Notice that for any M ∈ M q,ε ′ ( Ď M ) , we have f M ( π M ) ≤ f ˜ M ( π ˜ M ) and also

Combining these inequalities gives

Letting ∆ 0 → p-dec q ε,δ ( M , Ď M ) completes the proof.

F.4 Proof of Proposition 8

Recall that we frame the DMSO setting as an instance of ISDM in Section 2.2, where the observation is given by X = ( H T , ̂ π ) , and the loss is L ( M,X ) = L ( M, ̂ π ) . In particular, for any prior µ ∈ ∆( M ) , we have

Now, consider p ∈ ∆(Π) given by

and by definition, I µ, ALG ( M ; X ) ≤ I µ ( T ) . In particular, for any pair (∆ , µ ) such that

we have I µ, ALG ( M ; X )+log 2 log sup x µ ( M : L ( M,x ) < ∆) ≥ -1 2 , and hence applying Proposition 2 gives sup M ∈M E M, ALG [ L ( M, ̂ π )] ≥ ∆ 2 . Taking supremum over all pairs (∆ , µ ) satisfying (34) gives the desired lower bound.

F.5 Proof of Theorem E.3

Our proof adopts the analysis strategy originally proposed by Glasgow and Rakhlin [44].

Fix a 0 < ∆ < r-dec q ε,δ ( M ) and a parameter c ∈ (0 , 1) . Then there exists Ď M ∈ M + such that r-dec q ε,δ ( M , Ď M ) > ∆ .

Fix a T -round algorithm ALG with rules p 1 , · · · , p T , we consider a modified algorithm ALG ′ : for t = 1 , · · · , T , and history H ( t -1) , we set p ′ t ( ·|H ( t -1) ) = p t ( ·|H ( t -1) ) if ∑ t -1 s =1 L ( Ď M,π s ) < T ∆ -1 , and set p ′ t ( ·|H ( t -1) ) = 1 π Ď M if otherwise. By our construction, it holds that under ALG ′ , we have ∑ T t =1 L ( Ď M,π t ) < T ∆ almost surely. Furthermore, we can define the stopping time

If τ ≤ T , then it holds that ∑ τ t =1 L ( Ď M,π t ) ≥ T ∆ -1 .

Now, we consider p = P Ď M, ALG ′ ( 1 T ∑ T t =1 π t = · ) ∈ ∆(Π T ) . Using our definition of r-dec q , we know that E π ∼ p L ( Ď M,π ) < ∆ by our construction, and hence there exists M ∈ M such that

By definition of p and Lemma C.1, we have

We also know

and hence by Markov inequality,

In the following, we consider events

and the random variable X := ∑ T t =1 | f M ( π t ) -f Ď M ( π t ) | . By definition, P Ď M, ALG ′ ( E 1 ) > δ , P Ď M, ALG ′ ( X ≥ CTC r ε ) ≤ 1 C 2 . We have the following claim.

Claim: Under the event E 1 ∩ { τ ≤ T } , we have

To prove the claim, we bound

where the first inequality follows from E 1 , and the second inequality follows from ∑ T t = τ +1 | f M ( π t ) -f Ď M ( π t ) | ≤ X . On the other hand, we can also bound

where the first inequality follows from ∑ τ t =1 | f M ( π t ) -f Ď M ( π t ) | ≤ X , and the second inequality is because ∑ τ t =1 L ( Ď M,π t ) ≥ T ∆ -1 given τ ≤ T , which follows from the definition of the stopping time τ . Therefore, taking maximum over the above two inequalities proves our claim.

Now, using the claim, we know

Notice that D 2 H ( P M, ALG ′ , P Ď M, ALG ′ ) ≤ 7 Tε 2 , and hence for any event E , it holds P M, ALG ′ ( E ) ≥ P Ď M, ALG ′ ( E ) -√ 14 Tε 2 . In particular, we have

Finally, we note that ALG and ALG ′ agree on the first τ ∧ T rounds (formally, ALG and ALG ′ induce the same distribution of ( π 1 , · · · , π τ ∧ T ) ), and hence

The proof is hence complete by noticing that ∑ τ ∧ T t =1 L ( M,π t ) ≤ ∑ T t =1 L ( M,π t ) = Reg DM ( T ) and taking ∆ → r-dec q ε,δ ( M ) .

F.6 Proof of Proposition E.2

Fix a Ď M ∈ M + , and ∆ > r-dec q ε,δ ( M , Ď M ) . Choose p ∈ ∆(Π T ) such that

The existence of p is guaranteed by the definition (32). In other words, we have E π ∼ p [ L ( Ď M,π )] ≤ ∆ and

We then has the following claim.

Claim. For any model M ∈ M , it holds that

Fix any M ∈ M , we prove (36) as follows. Consider the event E = { π : L ( M,π ) ≤ ∆ } . Then,

where the inequality uses L ( M,π ) ≤ ∆ for π ∈ E and Assumption 4. Therefore,

This completes the proof of our claim.

Therefore, using (36) with E π ∼ p [ L ( Ď M,π )] ≤ ∆ yields

This immediately implies

Finally, taking ∆ → r-dec q ε,δ ( M , Ď M ) completes the proof.

F.7 Proof of Proposition E.4

Fix a reference model Ď M and let ∆ 0 > p-dec q ε,δ ( M , Ď M ) . Then there exists p, q ∈ ∆(Π) such that

Therefore, it holds that

If the constrained set M q,ε ( Ď M ) is empty, then we immediately have p-dec c ε ( M , Ď M ) = 0 , and the proof is completed. Therefore, in the following we may assume M q,ε ( Ď M ) is non-empty, and ̂ M ∈ M q,ε ( Ď M ) .

Claim. Let ̂ θ = θ ̂ M and ̂ π = ( π 0 , ̂ θ ) for an arbitrary π 0 , it holds that

This is because for any M ∈ M q,ε ( Ď M ) , it holds that

Hence, there exists θ ∈ Θ such that ρ ( θ M , θ ) ≤ ∆ 0 and ρ ( θ ̂ M , θ ) ≤ ∆ 0 holds. Therefore, it must hold that ρ ( θ M , ̂ θ ) ≤ 2∆ 0 for any M ∈ M q,ε ( Ď M ) .

The above claim immediately implies that

Letting ∆ 0 → p-dec q ε,δ ( M , Ď M ) yields p-dec c ε ( M , Ď M ) ≤ 2 p-dec q ε,δ ( M , Ď M ) , which is the desired result.

G Additional Discussion and Results from Section 4

G.1 Properties of the fractional covering number

Before proceeding to applications, let us briefly discuss some connections between the fractional covering number and classical notions of covering number considered in the context of statistical estimation.

To start, we recall that for many standard statistical estimation tasks such as regression and nonparametric estimation, the risk function L is given by a (pseudo) metric (e.g., ℓ 2 distance between parameters or mean-squared error in predictions). The following lemma shows that in this case, the fractional covering number coincides with the classical covering number induced by the metric, e.g., in location estimation (Example 4), density estimation (Example 6), etc.

Lemma G.1 (Connection to classical covering numbers) . Suppose the decision space Π is equipped with a pseudo-metric ρ : Π × Π → R + and there is a map M ↦→ π M ∈ Π such that the risk function is given by

Let Π M := { π M : M ∈ M} ⊆ Π , and define N (Π M , ∆) to be the ∆ -covering number of Π M under ρ . Then

Duality between fractional covering and fractional packing For classical covering numbers with respect to a pseudo-metric (as in Lemma G.1), it is known that there is a duality between covering and packing. In spite of a lack of metric structure for the general setting we study, we can show that fractional covering number naturally admits a dual representation in terms of a fractional packing number . Specifically, it holds that

as long as the minimax theorem can be applied (e.g. when Π or M are finite or satisfy appropriate compactness conditions). Therefore, in this case, we have

which can be interpreted as a fractional packing number. We mention in passing that using this interpretation, it is possible to derive Theorem 10 directly from Proposition 8.

Recovering the Yang-Barron method. As a simple example of the fractional covering number, we recover and further generalize the well-known Yang-Barron method [95] for statistical estimation problems (see also [89, Section 15.3.5]).

Example 12 (Yang-Barron method) . For a statistical estimation problem with model class M , we define the KL covering number of M as

For a fixed parameter ε > 0 , we can pick k = N KL ( M , ε ) and M 1 , · · · , M k ∈ M such that min i ∈ [ k ] D KL ( M ∥ M i ) ≤ ε 2 for all M ∈ M . Then, let us consider the localized sub-class M i := { M ∈ M : D KL ( M ∥ M i ) ≤ ε 2 } for each i ∈ [ k ] . It is clear that Assumption 2 holds for each M i with C KL ≤ ε 2 . Further, using M = ⋃ n i =1 M i , we have

For details, see Appendix H.4. Therefore, applying Theorem 10 to M i and take supremum over i ∈ [ k ] and ε > 0 gives the following result: For any algorithm ALG to achieve sup θ ∈ Θ E θ, ALG L ( M,π ) ≤ ∆ , it is necessary that

When the risk function L is given by a metric (cf. Eq. (37)), this inequality coincides with the Yang-Barron method formulated in Wainwright [89, Section 15.3.5], as the fractional covering number N frac ( M , ∆) can be lower bounded by the covering number (Lemma G.1).

Recovering the local packing lower bound for statistical estimation. As a simple example of the fractional covering number, we recover the well-known local packing-based lower bound [15] for the classical problem of location estimation [89].

Example 13 (Local packing lower bound for location estimation) . In the location estimation task (Example 4), recall that the model class is given by M = { M θ : θ ∈ Θ } , where M θ = N ( θ, I d ) . Consider the local packing number of Θ around θ ⋆ ∈ Θ , which is given by

̸

Then, for the localized sub-class M ε,θ ⋆ := { M θ : ∥ θ -θ ⋆ ∥ ≤ ε } ⊆ M , Assumption 2 holds with C KL ≤ 1 2 ε 2 , and we also have N frac ( M ε,θ ⋆ , ε/ 4) ≥ N loc (Θ , ε ; θ ⋆ ) . Therefore, we can apply Theorem 10 to M ε,θ ⋆ and take supremum over all θ ⋆ ∈ Θ and ε ≥ 8∆ to show the following result: For any algorithm ALG to achieve sup θ ∈ Θ E θ, ALG ∥ ̂ π -θ ∥ ≤ ∆ , it is necessary that

where N loc (Θ , ε ) = sup θ ⋆ ∈ Θ N loc (Θ , ε ; θ ⋆ ) is the local packing number of Θ . This lower bound is known to be tight in general [14, 15, 59, etc.]. ◁

G.2 Additional discussion from Section 4.2

In the literature on statistical learning, there is a long line of work which characterizes learnability of hypothesis classes in terms of abstract complexity measures. Examples include the VapnikChervonenkis dimension for binary classification [83, 16], the Littlestone dimension [63] for online classification [10] and differentially private classification [20, 6], and their real-valued counterparts (e.g. scale-sensitive dimensions) for regression [9, 5].

Beyond the settings above-particularly for interactive settings-learnability is less well understood. The question of what complexity measure characterizes bandit learnability has been explored in Russo and Van Roy [74], Abernethy et al. [1], Simchowitz et al. [79], Hashimoto et al. [46, etc.], but a complete resolution has yet to be reached. Remarkably, Ben-David et al. [11] demonstrate that there exists a simple and natural learning task for which it is impossible to characterize learnability through any combinatorial dimension. More recently, Hanneke and Yang [45] provide similar impossibility results for characterizing the learnability of noiseless structured noiseless bandits with real-valued rewards.

Our characterization bypasses the impossibility results of Hanneke and Yang [45]. Specifically, Hanneke and Yang [45] show that for noiseless structured bandit problems, there exist classes H for which bandit learnability is independent of the axioms of ZFC. Therefore, their results rule out the possibility of a characterization of noiseless bandit learnability through any combinatorial dimension [11] for the problem class. Our characterization is compatible with this result because the argument of Hanneke and Yang [45] relies on the noiseless nature of the bandit problem, and hence does not preclude a characterization for the noisy setting.

Connection to the maximin volume. Hanneke and Yang [45] propose maximin volume , a complexity measure that tightly characterizes the complexity of learning noiseless binary-valued structured bandit problems. For such problem classes, the fractional covering number is exactly the inverse of the maximin volume. While the fractional covering number can be viewed as a generalization of the maximin volume in this sense, we emphasize that the fractional covering number directly arises from our general lower bound framework, and is applicable to general decision making problems in the DMSO framework.

Noise distribution. Wenote that the upper bound in (20) applies to any reward distribution with subGaussian noise (cf. Appendix H.2). Meanwhile, since the lower bound in Corollary 12 is specialized to Gaussian noise, it acts as a lower bound for the broader class of sub-Gaussian noise distributions as well. We expect the lower bound to extend to other 'reasonable' noise distributions.

G.3 Exploration-by-Optimization Algorithm

In this section, we present a slightly modified version of the Exploration-by-Optimization Algorithm ( ExO + ) developed by Foster et al. [41], built upon Lattimore and Szepesvári [58], Lattimore and

Algorithm 1 Exploration-by-Optimization ( ExO + )

Input: Problem ( M , Π) , prior q ∈ ∆(Π) , parameter T ≥ 1 , γ > 0 .

1: Set q 1 = q .

2: for t = 1 , · · · , T do

3: Solve the exploration-by-optimization objective

4: Sample π t ∼ p t , execute π t and observe o t

5: Update

Gyorgy [55]. The original ExO + algorithm has an adversarial regret guarantee for any model class M , scaling with r-dec o γ (co( M )) , the offset DEC of the mode class co( M ) , and log | Π | , the logcardinality of the decision space. For our purpose, we adapt the original ExO + algorithm by using a prior q ∈ ∆(Π) not necessarily the uniform prior, and with a suitably chosen prior q , ExO + then achieves a regret guarantee scaling with log N frac ( M , ∆) , instead of log | Π | (cf. Foster et al. [41]), which is always an upper bound of log N frac ( M , ∆) .

Offset DEC for regret. We first recall the following (original) definition of DEC [40]:

and r-dec o γ ( M ) := sup Ď M ∈ co( M ) r-dec o γ ( M , Ď M ) . Through the Estimation-to-Decision (E2D) algorithm [40], offset regret-DEC provides an upper bound of Reg DM for any learning problem, and it is also closely related to the complexity of adversarial decision making.

As discussed in Foster et al. [42], in the reward maximization setting (Example 1), the constrained regret-DEC r-dec c can always be upper bounded in terms of the offset DEC r-dec o . Conversely, in the same setting, we also show that the offset DEC can also be upper bounded in terms of the constrained DEC (Theorem H.5), and hence the two concepts can be regarded as equivalent under mild assumptions (e.g. moderate decaying, Assumption 3).

Exploration-by-Optimization algorithm. The algorithm, ExO + , is restated in Algorithm 1. At each round t , the algorithm maintains a reference distribution q t ∈ ∆(Π) , and use it to obtain a decision distribution p t ∈ ∆(Π) and an estimation function ℓ t ∈ L := (Π × Π × O → R ) , by solving a joint minimax optimization problem based on the exploration-by-optimization objective: Defining

and

the algorithm solve ( p t , ℓ t ) ← arg min p ∈ ∆(Π) ,ℓ ∈L Γ q t ,γ ( p, ℓ ) . The algorithm then samples π t ∼ p t , executes π t and observes o t from the environment. Finally, the algorithm updates the reference distribution by performing the exponential weight update with weight function ℓ t ( · ; π t , o t ) .

Guarantee of ExO + . Following Foster et al. [41], we define

and exo 1 /γ ( M ) = sup q ∈ ∆(Π) exo 1 /γ ( M , q ) . The following theorem is deduced from Foster et al. [41, Theorem 3.1 and 3.2].

Theorem G.2. Under the reward maximization setting 4 (Assumption 4), it holds that

Now, we present the main guarantee of Algorithm 1, which has the desired dependence on the prior q ∈ ∆(Π) .

Theorem G.3. It holds that with probability at least 1 -δ ,

Proof. Consider the set Π ⋆ := { π : f M⋆ ( π M⋆ ) -f M⋆ ( π ) ≤ ∆ } and the distribution q ⋆ = q ( ·| Π ⋆ ) . Following Proposition G.4, we consider

and Proposition G.4 implies that

Applying Lemma C.3, we have with probability at least 1 -δ ,

Notice that

E t -1 [ exp ( -X t ( π t , o t ) )] = E π ∼ p t E o ∼ M ⋆ ( π ) E π ′ ∼ q t [ exp ( ℓ t ( π ′ ; π, o ) -E π ⋆ ∼ q ⋆ ℓ t ( π ⋆ ; π, o ) )] .

Using the fact that 1 -x ≤ -log x and Jensen's inequality, we have

where we denote

Therefore, it holds that

Applying Theorem G.2 completes the proof.

Proposition G.4. For any q ′ ∈ ∆(Π) , it holds that

Proof. This is essentially the standard guarantee of exponential weight updates. For simplicity, we assume Π is discrete. Then, by definition,

and hence

Therefore, taking summation over t = 1 , · · · , T , we have

The proof is then completed by the following basic fact of KL divergence: for any function h : Π → R ,

G.4 Application: Structured bandits

We now instantiate our general results to give tighter guarantees for structured bandits, improving the upper bounds in Section 4.2.

DEC for structured bandits. We consider the same structured bandit protocol as in Section 4.2; recall that H denotes the reward function class and M H denotes the induced model class. In what follows, we simplify the results in Theorem 15 to be stated purely in terms of H . For a reference value function s h : C × A → [0 , 1] , we define

where we recall that π h := max π ∈ Π h ( π ) . We then define the DEC for H as

As a corollary of Theorem G.3, the r-dec c ε ( H ) and log N frac ( H , ∆) together provide an upper bound for structured bandits with H .

Theorem G.5. Let H be given. Suppose that Π is finite, and that ε ↦→ r-dec c ε (co( H )) satisfies moderate decay as a function of ε (Assumption 3) with constant c reg . Let ¯ ε ( T ) ≍ √ log N frac ( H , ∆) /T . The Algorithm 1 ensures that high probability,

As a corollary, the minimax sample complexity of structured bandit learning with H is bounded as

where we denote T DEC ( H , ∆) = inf ε ∈ (0 , 1) { ε -2 : r-dec c ε ( H ) ≤ ∆ } (following Eq. (15)) and omit logarithmic factors and dependence on the constant c reg .

There are many standard structured bandit problems where the value function class H is convex, including multi-armed bandits, linear bandits, and non-parametric bandits (with smoothness [72], or concavity [54], or sub-modularity [65], or etc.). For these problem classes, the complexity of no-regret learning is completely characterized by the DEC of H and the fractional covering number N frac ( H , ∆) (up to a quadratic factor).

We also note that the lower bound of Eq. (44) is proven for Gaussian noise, while our upper bound applies to a much more general class of reward distributions (with bounded variance).

G.5 Application: Contextual bandits with general function approximation

Next, we instantiate our general results for stochastic contextual bandits with general function approximation, generalizing the structured bandit problem. We consider the stochastic contextual bandit problem with context space C , action space A , and a reward function class H ⊆ ( C×A → [0 , 1]) . This problem is a special case of the DMSO setting with decision space Π = ( C → A ) , and the environment is specified by a tuple ( h ⋆ ∈ H , ν ⋆ ∈ ∆( C )) . The protocol is as follows: For each round t , the environment draws c t ∼ ν , and the learner takes action a t = π t ( c t ) based on the decision π t : C → A , and receives a reward r t ∼ N ( h ⋆ ( c t , a t ) , 1) .

We can formulate the model class as follows. For a reward function h ∈ H and context distribution ν ∈ ∆( C ) , the corresponding model M h,ν is specified as

Let M H = { M h,ν : h ∈ H , ν ∈ ∆( C ) } be the induced model class of contextual bandits. Following Appendix G.4, we instantiate Theorem 15 to provide characterization of learning M H .

DEC for contextual bandits. For any context c ∈ C , the value function class H induces a restricted value function class H| c = { h ( c, · ) : h ∈ H} , which corresponds to a (non-contextual) bandit function class. We define the following variant of the DEC

which corresponds to the maximum of the per-context DEC over all contexts. We also define T DEC ( H , ∆) = inf ε ∈ (0 , 1) { ε -2 : r-dec c ε ( H ) ≤ ∆ } , following Eq. (15).

Fractional covering number for contextual bandits. Specializing the fractional covering number to contextual bandits, we define

where π h ∈ Π is defined via π h ( c ) := arg max a ∈A h ( c, a ) for c ∈ C .

Intuitively, the value of the fractional covering number log N frac ( H , ∆) for contextual bandits captures the difficulty of estimating optimal actions, but also the difficulty of generalizing across contexts. For example, when we consider the unstructured contextual bandit problems (i.e., H = ( C × A → [0 , 1]) ), it holds that log N frac ( H , ∆) = |C| log |A| , but in general we can have log N frac ( H , ∆) ≪ log | Π | = |C| log |A| .

As a corollary of Theorem 15, we derive the following upper and lower bounds on the complexity of contextual bandit learning with H .

Theorem G.6. Let H be given. Suppose that both the context space C and the action space A are finite, and that ε ↦→ r-dec c ε (co( H )) satisfies moderate decay as a function of ε (Assumption 3) with constant c reg . Let ¯ ε ( T ) ≍ √ log N frac ( H , ∆) /T . Then Algorithm 1 ensures that with high probability, ;

As a corollary, the complexity of learning M H is bounded by

omitting dependence on c reg and logarithmic factors.

By definition, we have r-dec c ε (co( H )) = r-dec c ε ( H ) if the per-context value function class H| c is convex for every context c ∈ C . Natural settings in which H| c is convex include contextual linear bandits [28], contextual non-parametric bandits [22], contextual concave bandits [54], etc. For these problem classes, the complexity of no-regret learning is completely characterized by the DEC of H and the newly proposed N frac ( H , ∆) (up to a quadratic factor and a factor of log |C| ).

As a concrete example. we can derive upper bounds based on the fractional covering number for finite-action contextual bandits as follows.

Corollary G.7. For any value function class H , Algorithm 1 ensures the following regret bound with high probability.

Compared to the well-known regret bound of O ( √ T |A| · log |H| ) for learning any with any finite contextual bandit class H [38, 77], this result above always provides a tighter upper bound, as log N frac ( H , ∆) ≤ log |H| . For certain (very simple) function classes H , the quantity log N frac ( H , ∆) can be much smaller than log |H| (for details, see Example 14). More importantly, log N frac ( H , ∆) leads to lower bounds for any contextual bandit function class (Theorem G.6). By contrast, lower bounds for structured contextual bandits in prior work have been proven in a caseby-case fashion (for specific value function classes H ).

H Proofs from Section 4 and Appendix G

In this section, we mainly focus on no-regret learning, and we present the regret upper and lower bounds in terms of DEC and log N frac ( M , ∆) . The results can be generalized immediately to PAC learning.

H.1 Proof of Theorem 10

Fix an arbitrary reference model Ď M ∈ (Π → ∆( O )) such that Assumption 2 holds. We remark that Ď M is not necessarily in M or co( M ) .

We only need to prove the following fact.

Fact. If T < log N frac ( M , ∆) -2 2 C KL , then for any T -round algorithm ALG , there exists a model M ∈ M such that Risk DM ( T ) ≥ ∆ with probability at least 1 2 under P M, ALG .

Proof. By the definition (17) of N frac ( M , ∆) , we know

Therefore, we have

and hence there exists M ∈ M such that

Notice that by the chain rule of KL divergence, we have

Hence, using data-processing inequality,

This immediately implies p M, ALG ( π : L ( M,π ) ≤ ∆) < 1 2 by the monotonicity of KL divergence.

H.2 Proof of Theorem 11

In this section, we present an algorithm based on reduction to multi-arm bandits (Algorithm 2) that achieves the desired upper bound. For the application to bandits with Gaussian rewards, we relax the assumption R : O → [0 , 1] as follows.

Assumption 5. For any M ∈ M and π ∈ Π , the random variable R ( o ) is 1-sub-Gaussian under o ∼ M ( π ) .

Suppose that ∆ > 0 is given, and fix a distribution p ⋆ ∆ that attains the infimum of (17). Based on p ⋆ ∆ , we consider a reduced decision space Π sub ⊂ Π , generated as

where we set N = N frac ( M , ∆)log(1 /δ ) . Then the space Π sub is guaranteed to contain a nearoptimal decision, as follows.

Lemma H.1. With probability at least 1 -δ , there exists π ∈ Π sub such that L ( M ⋆ , π ) ≤ ∆ .

Therefore, we can then regard M ⋆ as a N -arm bandit instance with action space A = Π sub , and for each pull of an arm π ∈ A , the stochastic reward r is generated as r = R ( o ) , o ∼ M ⋆ ( π ) . Then, we pick a standard bandit algorithm BanditALG , e.g. the UCB algorithm (see e.g. Lattimore and Szepesvári [57]), and apply it to the multi-arm bandit instance M ⋆ Bandit , and the guarantee of BanditALG yields

with probability at least 1 -δ . Therefore, we have

with probability at least 1 -2 δ . This gives the desired upper bound, and we summarize the full algorithm in Algorithm 2.

Proof of Lemma H.1. By definition,

H.3 Proof of Lemma G.1

Proof of the upper bound. Take a minimal ∆ -covering of Π M , i.e., a set { π 1 , · · · , π n } ⊆ Π such that for all M ∈ M , there exists i ∈ [ n ] such that ρ ( π M , π i ) ≤ ∆ . Therefore, we may consider the distribution p = Unif( { π 1 , · · · , π n } ) , which guarantee

̸

Proof of the lower bound. Consider the maximal 2∆ -packing of Π M , i.e., let { π 1 , · · · , π m } ⊆ Π M be a maximal set such that ρ ( π i , π j ) > 2∆ for any i = j . Then, by the duality between packing and covering, the set { π 1 , · · · , π m } form a 2∆ -covering of Π M , and hence we have m ≥ N (Π M , 2∆) . On the other hand, the sets Π i := { π : ρ ( π, π j ) ≤ ∆ } are pairwise disjoint, and hence for any p ∈ ∆(Π) , we have

Therefore, it holds that N frac ( M , ∆) ≥ m ≥ N (Π M , 2∆) .

Algorithm 2 A reduction algorithm based on the fractional covering number

Input: Problem ( M , Π) , parameter ∆ , δ > 0 , T ≥ 1 , Algorithm BanditALG for multi-arm bandits.

1: Set

2: Set N = N frac ( M , ∆)log(1 /δ ) and sample the decision subspace Π sub = { π (1) , · · · , π ( N ) } ⊂ Π as

3: Run the bandit algorithm BanditALG on the instance M ⋆ Bandit for T rounds.

H.4 Proof of Example 12

It remains to prove Eq. (39). More generally, we prove the following lemma.

Lemma H.2. For model class M = ⋃ n i =1 M i , it holds that

Proof of Lemma H.2 For each i ∈ [ n ] , we define λ i = N frac ( M i , ∆) ∑ n j =1 N frac ( M j , ∆) .

Fix p 1 , · · · , p n ∈ ∆(Π) . Then, let us consider the distribution p = ∑ n i =1 λ i p i ∈ ∆(Π) . For any model M ∈ M , there exists i ∈ M i , and hence p ( π : L ( M,π ) ≤ ∆) ≥ λ i min M i ∈M i p i ( π : L ( M i , π ) ≤ ∆) . Therefore, it holds that

In other words,

Taking infimum over p 1 , · · · , p n ∈ ∆(Π) gives

where the last line follows from the definition of λ 1 , · · · , λ n . This is the desired result.

H.5 Proof of Theorem 14

We first state the following more general result, and Theorem 14 is then a direct corollary (under Assumption 3). Analoguous guarantees also hold for PAC learning.

Theorem H.3. Let T ≥ 1 , δ ∈ (0 , 1) . With suitably chosen prior q ∈ ∆(Π) , ExO + (Algorithm 1) achieves with probability at least 1 -δ :

In particular, when M is a reward-maximization problem class (Example 1), ExO + achieves (with a suitable parameter γ ) that with probability at least 1 -δ :

where C is an absolute constant, ¯ ε ( T ) = √ log N frac ( M , ∆)+log(1 /δ ) T , and the modified version of constrained regret-DEC is defined as

Remark H.4 (Upper bound without regularity condition) . In Theorem 14 (and Eq. (49)), we assume that (1) M is a reward-maximization problem, and (2) the constrained regret-DEC of co( M ) satisfies certain regularity condition (Assumption 3). We can relax these two assumptions and obtain a weaker upper bound. Specifically, we may only assume that f M : Π → [0 , 1] is affine with respect to M ∈ co( M ) (cf. Theorem G.2). In this case, we can still bound the regret of ExO + as

which follows from Eq. (48) and the fact that (by definition)

In particular, using Eq. (51) above, we can show that (omitting poly-logarithmic factors)

This is worse than the upper bound of Theorem 15 by (roughly) a factor of ∆ -1 .

Proof of Theorem H.3. By the definition (17) of N frac ( M , ∆) , we know

Therefore, there exists q ∈ ∆(Π) such that

We then instantiate Algorithm 1 with such a prior q , and Eq. (48) follows immediately from Theorem G.3. To prove Eq. (49), we invoke the following structural result that relates offset DEC to constrained DEC.

Theorem H.5. Suppose that Assumption 4 holds for the model class M . Then for any ε ∈ (0 , 1] , it holds that

Under the assumption that ε ↦→ r-dec c ε (co( M )) is of moderate decay with a constant c reg , we have

Hence, Eq. (49) follows from (48) as long as the parameter γ is chosen according to Eq. (H.5).

H.5.1 Proof of Theorem H.5

Fix a ε ∈ (0 , 1] and Ď M ∈ co( M ) . We only need to prove the following result:

Claim. Suppose that r-dec c ε ′ ( M , Ď M ) ≤ Dε ′ for all ε ′ ∈ [ ε, 1] . Then there exists γ = γ ( D,ε ) such that

Set K = ⌊ log 2 (1 /ε ) ⌋ +1 and fix a parameter c = c ( ε ) ∈ (0 , 1 2 ] to be specified later in proof. Define ε i := 2 -i for i = 0 , · · · , K -1 and ε K = ε . We also define λ i := cε · 2 i for i = 0 , · · · , K -1 , and λ K = 1 -∑ K -1 i =0 λ i ≥ c .

Define ∆ i = r-dec c ε i ( M∪{ Ď M } , Ď M ) , and let p i attains the inf p . In the following, we choose γ = γ ( D,ε ) = 9( D + C r ) 8 cε .

By definition of p i , it holds that

In particular, we may abbreviate M i := { M ∈ M : E π ∼ p i D 2 H ( M ( π ) , Ď M ( π ) ) ≤ ε 2 i } , and it holds

Next, we choose p = ∑ K i =0 λ i p i ∈ ∆(Π) , and we know

Fix a M ∈ M . Let j ∈ { 0 , · · · , K } be the maximum index such that M ∈ M j . Such a j must exists because M = M 0 . Now,

Case 1: j = K . Then, using AM-GM inequality, we have

Case 2: j < K . Then for each i > j , it holds that E π ∼ p j D 2 H ( M ( π ) , Ď M ( π ) ) > ε 2 j , and hence

Therefore, using AM-GM inequality,

By our choice of γ , we have γε ≥ 9 8 c ( ∆ j ε j + C r ) , and hence in both cases, we have

Note that by definition, we have ∆ ≤ ( cK +1) Dε and γ ( ε ) · ε = 9 8 c ( D + C r ) , and hence

Thus,

Balancing c and re-arranging yields the desired result.

H.6 Proof of Theorem 15

Note that the minimax-optimal sample complexity T ⋆ ( M , ∆) is just a way to better illustrate our minimax regret upper and lower bounds. By the definition of T ⋆ ( M , ∆) , we have

Under Assumption 3, the regret upper bound in Theorem 14 implies (up to c reg , C KL and logarithmic factors)

And the regret lower bound Theorem E.1 implies (up to c reg and logarithmic factors)

By the definition of T ⋆ ( M , ∆) and T DEC ( M , ∆) , we then have

Together with Theorem 10, we prove that

H.7 Proof of Theorem G.5

For the upper bound, we work with more general noise structure (beyond Gaussian noises). We define M H , V to be the class of all bandits models with mean reward function in H and variance bounded by 1. Specifically, for any M ∈ M H , V , it is associated with a value function h M ∈ H , such that for any decision π ∈ Π , the distribution M ( π ) of the random reward r has mean h M ( π ) and variance at most 1.

Wealso recall that the subclass M H ⊆ M H , V is the bandit problem class with the standard Gaussian noise.

Proof of Theorem G.5: lower bound of (44). The lower bound with log N frac ( H , ∆) is exactly Corollary 12.

To prove the lower bound with T DEC ( H , ∆) , we need to lower bound the DEC of M H in terms of the DEC of H , as follows.

Lemma H.6. Consider M + = M co( H ) , V as the class of all reference models (Appendix E). Then,

Notice that for M + , Assumption 4 holds with C r = √ 10 (by Lemma C.5). Therefore, as a corollary of Theorem E.3: for any T -round algorithm ALG , there exists M ⋆ ∈ M H such that

with probability at least 0 . 01 under P M⋆ , ALG , where ε ( T ) = 1 50 √ T . Therefore, the lower bound in terms of T DEC ( H , ∆) follows immediately (using regularity condition Assumption 3).

Combining both lower bounds completes the proof.

Proof of Theorem G.5: upper bound. Weapply Theorem H.3 similar to the proof of Theorem 15 (in Appendix H.2).

Using Theorem H.3, we know that ExO + can be suitably instantiated on the model class M H , V so that with probability at least 1 -δ ,

where C is an absolute constant, ¯ ε ( T ) = √ log N frac ( H , ∆)+log(1 /δ ) T . We only need to upper bound the r-dec c ε (co( M H , V )) (defined in (50)) in terms of the DEC of co( H ) .

Lemma H.7. For any ε ≥ 0 , it holds that

We also note that co( M H , V ) ⊆ M co( H ) , V because the model class M co( H ) , V is convex and it contains M H , V . Therefore, we know

Using the regularity of ε ↦→ r-dec c ε (co( H )) , we know

This gives the desired upper bound.

H.7.1 Proof of Lemma H.6

Fix a ε ∈ [0 , 1] , we denote ε 1 = 2 √ 2 ε and take any ∆ < r-dec c ε 1 ( H ) . We pick s h ∈ co( H ) such that r-dec c ε 1 ( H , s h ) > ∆ . Then, it holds that

Suppose that s h ∈ co( H ) is given by s h = E h ∼ µ [ h ] with µ ∈ ∆( H ) . Then, consider the reference model Ď M ∈ M + with mean reward function s h and Gaussian noise, i.e. Ď M ( π ) = N ( s h ( π ) , 1 ) . Then, we know that for M = M H ,

where the last line follows from Lemma C.5. Taking ∆ → r-dec c ε 1 ( H ) completes the proof of (52).

H.7.2 Proof of Lemma H.7

Fix a reference model Ď M ∈ co( M H , V ) . By definition, we know the mean reward function h Ď M of Ď M belongs to co( H ) , i.e. Ď M ∈ M co( H ) , V . Therefore, for any model M ∈ M H , V and decision π ∈ Π , by Lemma C.5,

Therefore, for M = M H , V ,

where the second equality follows from the fact that when = h , we have L ( M,π ) = h ( π h ) -h ( π ) . Taking supremum over Ď M completes the proof.

H.8 Proof of Theorem G.6

Similar to Appendix H.7, we consider a larger model class M H , V of models with general noise structure. A model M ∈ M H , V is specified by a context distribution ν M ∈ ∆( C ) , a reward function h M ∈ H , and a reward distribution R M ( ·|· , · ) , such that for any c ∈ C , a ∈ A , r ∼ R M ( ·| c, a ) has mean h M ( c, a ) and variance at most 1. The model M is then given by

The model class M H , V is defined to be the set of all possible models described above.

Proof of Theorem G.6: lower bound. The lower bound with log N frac ( H , ∆) follows immediately by applying Theorem 10 to the class M H , which admits C KL = O (log |C| ) in Assumption 2 (as shown in Example 9).

On the other hand, the lower bound with T DEC ( H , ∆) follows from the reduction to the per-context bandits problem. Specifically, for a fixed context c ∈ C , H| c corresponds to a structure bandits class M H| c . Notice that we can naturally regard M H| c ⊂ M H by viewing M H| c as a contextual bandits class with the fixed context c . Therefore, by Theorem G.5 (specifically (53)):

Taking maximum over c ∈ C yields

This gives the desired lower bound with T DEC ( H , ∆) .

Combining both lower bounds completes the proof.

Proof of Theorem G.6: upper bound. We follow the proof strategy of Appendix H.7. By Theorem H.3, ExO + can be suitably instantiated on the problem class M H , V so that with probability at least 1 -δ :

We also note that co( M H , V ) ⊆ M co( H ) , V . Therefore, it remains to upper bound the offset DEC of M co( H ) , V .

Lemma H.8. For γ > 0 , it holds that

Then, we can apply the result of Theorem H.5. From the proof of Theorem H.5, it is not hard to see that: for any ε > 0 , there exists γ = γ ( ε ) such that for any c ∈ C ,

where we also use the regularity condition of ε ↦→ r-dec c ε (co( H )) . This immediately gives

H.8.1 Proof of Lemma H.8

Fix a reference model Ď M ∈ co( M H , V ) , and then Ď M ∈ M co( H ) , V by definition. In particular, Ď M has mean value function h Ď M ∈ H and context distribution ¯ ν ∈ ∆( C ) . We also know that for each c ∈ C , h Ď M ( x, · ) ∈ co( H| c ) .

Then, by Lemma C.4, we also have

Thus, we adopt the following notations: For each c ∈ C and model M ∈ M H , V , we define M c ∈ M H| c , V to be a bandit model such that for every action a ∈ A , M c ( a ) = R M ( r = ·| c, a ) . Then by definition, it holds that

Now, combining the inequalities above, we have

where the equality (1) is because for a sequence ( p c ∈ ∆( A )) c ∈C , there is a corresponding p ∈ ∆(Π) such that for π ∼ p , we have π ( c ) ∼ p c independently; in inequality (2) we bound the expectation over c ∼ ν M by the supremum sup c ∈C ; the equality (3) follows from the fact that M c ∈ M H| c , V is a bandit model with mean reward function h Mc ( · ) = h M ( c, · ) ; and the equality (4) is because we can choose p c separately for every c ∈ C . By the arbitrariness of Ď M ∈ co( M ) , we now have

H.9 Proof of Corollary G.7

We follow the notations of Appendix H.8. By Lemma H.8, we have

Notice that for each c ∈ C , M H| c , V is a class of |A| -arm bandits, and hence by Foster et al. [40, Proposition 5.1] and Lemma C.5, we have

Therefore, Theorem H.3 implies that ExO + achieves with probability at least 1 -δ :

Balancing γ > 0 gives the desired upper bound.

As a remark, we provide an example of function class H with log N frac ( H , ∆) ≪ log |H| .

Example 14. Suppose that A = { 0 , 1 } , and the function class H = { h x } x ∈C , where

̸

Clearly, we have log |H| = log |C| .

On the other hand, we consider a distribution p over policies, such that π ∼ p is generated as π ( c ) ∼ Bern( ε ) , independently over all c ∼ C . Then, for any h = h x ∈ H and ν ∈ ∆( C ) , we have

̸

Notice that π ( x ) = 1 with probability ∆ , and conditional on the event { π ( x ) = 1 } ,

̸

which implies log N frac ( H , ∆) ≤ log(2 / ∆) .

Therefore, for unbounded context space C , we have log N frac ( H , ∆) ≪ log |H| for the function class H defined above.

Hence,

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are validated by detailed proofs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: The paper discusses the limitations of the work performed.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The paper provides detailed assumptions and proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA] .

Justification: This paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical work. There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: This paper does not involve crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to provide a unified framework for deriving lower bounds of interactive decision-making problems using an extended technique adapted from Chen et al., 2016 [11], which was established only for non-interactive estimation problems. The paper offers a quite general formulation of the minimax value of interactive decision-making that encompasses both previously known and novel formulations. In particular, the paper demonstrates that such a lower bounding technique can be used to rederive the lower bounds from Foster et al., 2023 [25] and provide tighter lower and upper bounds via a new complexity named \"decision coverage.\"\n\nSTRENGTHS:\nI find the general formulation and the technique based on percentiles (instead of the expected value used in prior literature such as [25]) for deriving lower bounds to be quite appealing. Although it appears to me that the lower bounding techniques are essentially \"abstracted out\" from prior literature, such as [11] and [23, 25], the paper presents it in a quite clean manner. The paper also provides several new results, such as interactive parameter estimation and tighter bounds for interactive reward maximization. I generally enjoyed reading this paper and find it worthy of publishing at NeurIPS. I expect the paper to have a broad impact on the community.\n\nWEAKNESSES:\nMy main complaint about this paper is that it contains many hand-waving arguments. I had to fill in many gaps in the proofs myself to verify and understand the results.\n\nI outline some specific comments below (some are minor, while others need additional justifications):\n\n1. I'm not sure if it is appropriate to call Corollary 2 a \"Generalized Fano’s inequality\", as it is not clear to me that it implies the original Fano's inequality (though they are \"qualitatively\" equivalent, i.e., both require $I(M,X) \\ll \\log |\\mathcal{M}|$ to arrive at a constant lower bound).\n2. You claim in line 186 \"Fano’s inequality, e.g., in the form of Corollary 2, cannot be used to prove Lemma 3.\" Is there an argument why this is true?\n3. Can you explain how Theorem 5 is instantiated from Theorem 1? It appears to me that its proof is completely self-contained without invoking any parts of Theorem 1.\n4. Can you comment on how prior technique for deriving Corollary 9 differs from your Fano's inequality-based argument?\n5. Can you explain how $T^*(\\mathcal{M},\\Delta)$ characterizes the regret? Specifically, it does not appear to me that there is a simple way to express the regret in terms of $T^*(\\mathcal{M},\\Delta)$. Moreover, why does Theorem 4 imply the statement in line 282?\n6. The proof of Theorem 10 is too sketchy. It is not clear to me that (12) shares the structure as Proposition 8. Are you applying the minimax theorem here?\n7. Theorem 10 looks quite trivial to me, as it does not depend on any information-theoretical structure of the model class (except Assumption 2).\n8. Given point 5, I'm not sure how to interpret your Theorem 12 and how to compare it with [25]. It appears you are claiming this as a main result; can you provide any specific examples that demonstrate the improvement (i.e., explicitly computing the \"decision coverage\" for some classes)?\n9. You claim that \"Theorem 12 provides (polynomially) matching lower and upper bounds for learning M.\" Isn't $\\max\\\\{a,b\\\\}$ can be arbitrarily smaller than $a \\cdot b$? How should the \"matching\" be interpreted?\n10. It appears to me the upper bound of Theorem 11 essentially converts the dependency on the size of the model class to that of the decision space. Given the similarity of the proof with that in [24], can you comment on if the results from [24] recover your upper bound (with possible replacement of decision coverage with $\\log|\\Pi|$)?\n11. Typos (there are many, but I only include what I still remember):\n    - $D_f(a,b)$ was never properly defined, though one can guess it is for $D_f(\\text{Bern}(a),\\text{Bern}(b))$.\n    - What is the $\\alpha$ in line 873?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops the notion of Interactive Statistical Decision Making (ISDM), and a generic lower bound (Theorem 1) which can be instantiated to capture the standard Le Cam, Assaoud, Fano methods as well as recent lower bound results in interactive decision making. The authors further use Theorem 1 to derive new sample complexity bounds (Theorem 12, 13) on interactive decision making contexts (under some regularity conditions on the model class).\n\nSTRENGTHS:\nI think it's interesting to try to unify different lower bound approaches in order to gain new insights, so the premise of the paper is very intriguing to me. The submission further uses the general theorem to derive new bounds, based on the new notion of Decision Coverage, that tightens existing results in interaction decision making.\n\nWEAKNESSES:\nIt may be due to my lack of expertise in interaction decision making literature, but I'm having trouble understanding and evaluating the new contributions, with intuition missing that I hoped the paper would give. Some of the contributions also seem a bit overclaimed. I hope the following constructive criticisms would help the authors improve the paper.\n\n- \"Addressing remaining gap\" and \"complete characterization\": the authors say that the new results (Theorems 12/13) \"completely characterize\" the sample complexity of interactive decision making for convex model classes, but the left and right hand sides differ quadratically (and ignores log factors). Am I misinterpreting the results? They aren't even tight up to constants.\n\n- Line 56: I don't quite understand why \"unifying two-point vs mixture-vs-mixture methods\" is a new contribution. Mixture-vs-mixture is clearly a generalization of singleton two-point methods, so why is the unification sold as a new contribution?\n\n- Generally I find the bound in Theorem 1 challenging to interpret, as opposed to two-point or mixture-vs-mixture methods that make intuitive sense. I'm struggling to understand the insight gained by the Theorem 1 formulation through unifying two-point methods and Fano with interaction decision making. To me, ISDM reads like a very generic minimax game formulation, and then Theorem 1 tags on the reference distribution/ghost data $\\mathbb{Q}$ in order to encompass existing techniques for interaction decision making. Lemma 3 then removes this extra component by declaring $\\mathbb{Q}$ simply as the transcript of the algorithm, so what have we learned about standard statistical estimation through Theorem 1? In other words, my question is, why is Le Cam/Assaoud/Fano even part of the paper, instead of focusing only on the new bounds in interactive decision making? What am I missing?\n\n- Theorem 12 is a bit too informal, hiding the log factors (especially without specifying log factors in what). It also wasn't actually proven -- I think the calculations using Assumptions 2 and 3 (and applied to Theorem 11) should be explicitly shown in the appendix.\n\n- Relatedly, again on the topic of needing more interpretation, I wish the submission explained how DC is better than the $\\log |\\mathcal{M}|$ factor in Line 282.\n\nMisc typos I spotted and other small comments:\n\n- First page, should define Perf as cost, so that minimization is the correct direction.\n- End of Line 123, should the asterisk in $M^\\ast$ be removed?\n- Line 177, $L:\\Theta \\times \\mathcal{A} \\rightarrow \\mathbb{R}_+$ right? Instead of domain being $\\Theta \\times \\Theta$? Just a consistency issue with the rest of the lemma.\n- Line 594, I presume \"ISDM\" instead of \"ASDM\"?\n\nQUESTIONS:\nPlease see weaknesses above.\n\n- Could the authors also comment on how the techniques apply/adapt if we want to show lower bounds on high-probability loss instead of expected loss? Two-point methods are straightforward (and even simpler than Le Cam) to use in a high-probability setting.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified framework for lower-bound methods in statistical estimation and interactive decision-making. The authors integrate classical lower bound techniques (Fano's inequality, Le Cam's method, Assouad's lemma) with recent minimax lower bounds for interactive decision-making (Decision-Estimation Coefficient). The framework is based on a general algorithmic lower bound method and introduces a novel complexity measure, decision coverage. The paper also has a lot of other results, including the unification of classical and interactive methods, the generalization of Fano's separation condition, and the derivation of new lower bounds for interactive decision-making.\n\nSTRENGTHS:\n* The framework developed in the paper is valid for a very general class of problems and unifies classical and classical lower bound techniques, Fano's inequality, Le Cam's method, and Assouad's lemma, and provides a comprehensive framework for statistical estimation and decision making.\n* Lots of other contributions: incorporating the previous work on DEC into the framework of this paper introduces a new complexity measure called decision coverage, which quantifies the  complements of the previous DEC lower bounds.\n* Except for a few typos, the paper is well-written.\n* Overall, it is a solid paper, and contributions are significant.\n\nWEAKNESSES:\n* I dont find any major technical weakness in this work. The work is good and a bit hard to follow for someone who is a non-expert in this area. Most of the points I highlight below are bunch of typos that I found and clarification that I think its good to include. \n* Line 123, I think it should be $L(M,X)$ instead of $L(M^*,X)$\n* The proof of corollary 2 is skipped. It involves 3-4 steps, and therefore I don't think it is trivial, especially for those who are not experts in dealing with these inequalities.\n\n* There seem to be a lot of (minor) typos in Lemma 3, and it's proof.\n1. Firstly, it's better to clarify whether sets $\\theta_0, \\theta_1$ are required to be distinct or not in the Lemma statement.\n2. ASDM is not defined in the first line of proof.\n3. I think the second equality in Line 602 will contain a factor of $1/2$. Can authors also clarify if the next step follows by data processing? \n4. In line 603, $ d_{3/4}(\\cdot,\\cdot)$ is not defined. May be authors mean $ d_{f, 3/4}(\\cdot,\\cdot)$. Please also explain why that inequality follows, is it by choice of $\\Delta$ and so that Theorem 1 can be applied?  \n5. There seems to be a typo in the subscripts of expectation in line 604, third inequality.\n\n* I guess in Line 218 Eq. 9, $p_{out}$ is not defined.\n* I don't think $\\pi_{out}$  in Line 226 is defined before (in the statement of theorem 5).\n* I would appreciate if the authors mention where the realizability assumption (Assumption 1) is needed and the issues that arise in the agnostic setting, i.e., when realizability does not hold. Also, please provide examples (or references) of well-posed model classes in Assumption 2.\n* It is not clear to me why there exists $M$ in the model class, i.e., why is the supremum achieved here by some $M$ in proof of Theorem 5.\n* Line 329, I don't think $\\mathbb{V}$ is defined.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nEssentially no broader societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides a unified perspective of existing techniques for deriving lower bounds. This viewpoint covers techniques that are useful for traditional statistical estimation (e.g., Fano's inequality, Le Cam's method, and Assouad's approach) as well as the recently proposed approach using decision-estimation coefficients that concerns interactive decision-making. In addition, this work proposes a novel complexity measure called decision coverage. Using this novel measure, this work derives lower and a polynomially matching upper bound for learning convex model classes.\n\nSTRENGTHS:\nS1. The paper discusses a unified perspective of techniques to derive lower bounds. It integrates classical techniques (Fano’s inequality, Le Cam’s method, and Assouad’s lemma) with contemporary methods for interactive decision-making, based on the Decision-Estimation Coefficient (DEC). \n\nS2. This work introduces a novel complexity measure called decision coverage. This measure facilitates the derivation of new lower bounds specifically tailored for interactive decision-making.\n\nWEAKNESSES:\nSee questions\n\nQUESTIONS:\nQ1. How realistic is Assumption 2? It seems to require that we have a model $\\bar{M}$ that is close to all models $M \\in \\mathcal{M}.$ It is unclear why should such an assumption be true for a finite $C_{KL}?$ Similarly, how realistic is Assumption 3? Can you give some application examples where Assumptions 2 and 3 hold?\n\nQ2. What is $\\mathbb{V}$ in line 329? Is it variance?\n\nQ3. Why is Assumption 2 not needed in Theorem 13, even though the line above says that this result is a corollary to Theorem 12?\n\nQ4. Can the ideas in this work be extended to the RL setup? \n\nQ5. Can the ideas in this work be extended to the case of interactive decision-making with multiple agents? \n\nI am happy to increase my score based on your response.\n\nLIMITATIONS:\nThe work does not explicitly discuss the limitations of their work; at least, I could not find it anywhere.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}