Title: Regret Minimization via Saddle Point Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: sequential decision-making

decision-estimation coefficient

regret minimization

bandits

reinforcement learning

partial monitoring
Abstract: A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs. 
In the corresponding saddle-point game, the min-player optimizes the sampling distribution against an adversarial max-player that chooses confusing models leading to large regret. The most recent instantiation of this idea is the decision-estimation coefficient (DEC), which was shown to provide nearly tight lower and upper bounds on the worst-case expected regret in structured bandits and reinforcement learning. By re-parametrizing the offset DEC with the confidence radius and solving the corresponding min-max program, we derive an anytime variant of the Estimation-To-Decisions algorithm (Anytime-E2D). Importantly, the algorithm optimizes the exploration-exploitation trade-off online instead of via the analysis. Our formulation leads to a practical algorithm for finite model classes and linear feedback models. We further point out connections to the information ratio, decoupling coefficient and PAC-DEC, and numerically evaluate the performance of E2D on simple examples.

Regret Minimization via Saddle Point Optimization

Abstract

A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs. In the corresponding saddle-point game, the min-player optimizes the sampling distribution against an adversarial max-player that chooses confusing models leading to large regret. The most recent instantiation of this idea is the decision-estimation coefficient (DEC), which was shown to provide nearly tight lower and upper bounds on the worst-case expected regret in structured bandits and reinforcement learning. By reparametrizing the offset DEC with the confidence radius and solving the corresponding min-max program, we derive an anytime variant of the Estimation-To-Decisions algorithm (ANYTIME-E2D). Importantly, the algorithm optimizes the explorationexploitation trade-off online instead of via the analysis. Our formulation leads to a practical algorithm for finite model classes and linear feedback models. We further point out connections to the information ratio, decoupling coefficient and PAC-DEC, and numerically evaluate the performance of E2D on simple examples.

1 Introduction

Regret minimization is a widely studied objective in bandits and reinforcement learning theory [Lattimore and Szepesvári, 2020a] that has inspired practical algorithms, for example, in noisy zero-order optimization[e.g., Srinivas et al., 2010] and deep reinforcement learning [e.g., Osband et al., 2016]. Cumulative regret measures the online performance of the algorithm by the total loss suffered due to choosing suboptimal decisions. Regret is unavoidable to a certain extent as the learner collects information to reduce uncertainty about the environment. In other words, a learner will inevitably face the exploration-exploitation trade-off where it must balance collecting rewards and collecting information. Finding the right balance is the central challenge of sequential decision-making under uncertainty.

More formally, denote by Π a decision space and O an observation space. Let M be a class of models, where f = ( r f , M f ) ∈ M associated with a reward function r f : Π → R and observation

map M f : Π → P ( O ) , where P ( O ) is the set of all probability distributions over O . 1 The learner's objective is to collect as much reward as possible in n steps when facing a model f ∗ ∈ M . The learner's prior information is M and the associated reward and observation maps, but does not know the true instance f ∗ ∈ M . The learner constructs a stochastic sequence π 1 , . . . , π n of decisions taking values in Π and adapted to the history of observations y t ∼ M f ∗ ( π t ) . The policy of the learner is the sequence of probability kernels µ 1: n = ( µ t ) n t =1 that are used to take decisions. The expected regret of a policy µ 1: n and model f ∗ after n ∈ N steps is

The literature studies regret minimization for various objectives, including worst-case and instancedependent frequentist regret [Lattimore and Szepesvári, 2020a], Bayesian regret [Russo and Van Roy, 2014] and robust variants [Garcelon et al., 2020, Kirschner et al., 2020a]. For the frequentist analysis, all prior knowledge is encoded in the model class M . The worst-case regret of policy µ 1: n on M is sup f ∈M R n ( µ 1: n , f ) , and therefore the optimal minimax regret inf µ sup f ∈M R n ( µ 1: n , f ) only depends on M and the horizon n . The Bayesian, in addition, assumes access to a prior ν ∈ P ( M ) , which leads to the Bayesian regret E f ∼ ν [ R n ( µ 1: n , f )] . Interestingly, the worst-case frequentist regret and Bayesian regret are dual in the following sense [Lattimore and Szepesvári, 2019]: 2

Unfortunately, directly solving for the minimax policy (or the worst-case prior) is intractable, except in superficially simple problems. Ths is because the optimization is over the exponentially large space of adaptive policies. However, the relationship in Eq. (1) has been directly exploited in prior works, for example, to derive non-constructive upper bounds on the worst-case regret via a Bayesian analysis [Bubeck et al., 2015]. Moreover, it can be seen as inspiration underlying 'optimization-based' algorithms for regret minimization: The crucial step is to carefully relax the saddle point problem in a way that preserves the statistical complexity, but can be analyzed and computed more easily. This idea manifests in several closely related algorithms, including information-directed sampling [Russo and Van Roy, 2014, Kirschner and Krause, 2018], ExpByOpt [Lattimore and Szepesvári, 2020b, Lattimore and Gyorgy, 2021], and most recently, the Estimation-To-Decisions (E2D) framework [Foster et al., 2021, 2023]. These algorithms have in common that they optimize the information tradeoff directly, which in structured settings leads to large improvements compared to standard optimistic exploration approaches and Thompson sampling. On the other hand, algorithms that directly optimize the information trade-off can be computationally more demanding and, consequently, are often not the first choice of practitioners. This is partly due to the literature primarily focusing on statistical aspects, leaving computational and practical considerations underexplored.

Contributions Building on the results by Foster et al. [2021], we introduce the average-constrained decision-estimation coefficient (dec ac ϵ ), a saddle-point objective that characterizes the frequentist worst-case regret in sequential decision-making with structured observations. Compared to the decision-estimation coefficient of [Foster et al., 2021], the dec ac ϵ is parametrized via the confidence radius ϵ , instead of the Lagrangian offset multiplier. This allows optimization of the information tradeoff online by the algorithm, instead of via the derived regret upper bound. Moreover, optimizing the dec ac ϵ leads to an anytime version of the E2D algorithm (ANYTIME-E2D) with a straightforward analysis. We also point out relations between the dec ac ϵ , the information ratio [Russo and Van Roy, 2016], the decoupling coefficient [Zhang, 2022] and a PAC version of the DEC [Foster et al., 2023]. We further detail how to implement the algorithm for finite model classes and linear feedback models, and demonstrate the advantage of the approach by providing improved bounds for linear bandits with sideobservations. Lastly, we report the first empirical results of the E2D algorithm on simple examples.

1.1 Related Work

There is a broad literature on regret minimization in bandits [Lattimore and Szepesvári, 2020a] and reinforcement learning [Jin et al., 2018, Azar et al., 2017, Zhou et al., 2021, Du et al., 2021,

Zanette et al., 2020]. Arguably the most popular approaches are based on optimism, leading to the widely analysed upper confidence bound (UCB) algorithms [Lattimore and Szepesvári, 2020a], and Thompson sampling (TS) [Thompson, 1933, Russo and Van Roy, 2016].

A long line of work approaches regret minimization as a saddle point problem. Degenne et al. [2020b] showed that in the structured bandit setting, an algorithm based on solving a saddle point equation achieves asymptotically optimal regret bounds, while explicitly controlling the finite-order terms. Lattimore and Szepesvári [2020b] propose an algorithm based on exponential weights in the partial monitoring setting [Rustichini, 1999] that finds a distribution for exploration by solving a saddle-point problem. The saddle-point problem balances the trade-off between the exponential weights distribution and an information or stability term. The same approach was further refined by Lattimore and Gyorgy [2021]. In stochastic linear bandits, Kirschner et al. [2021] demonstrated that information-directed sampling can be understood as a primal-dual method solving the asymptotic lower bound, which leads to an algorithm that is both worst-case and asymptotically optimal. The saddle-point approach has been further explored in the PAC setting [e.g., Degenne et al., 2020b,a].

Our work is closely related to recent work by Foster et al. [2021, 2023]. They consider decision making with structured observations (DMSO), which generalizes the bandit and RL setting. They introduce a complexity measure, the offset decision-estimation coefficient (offset DEC), defined as a min-max game between a learner and an environment, and provide lower bounds in terms of the offset DEC. Further, they provide an algorithm, Estimation-to-Decisions (E2D) with corresponding worst-case upper bounds in terms of the offset DEC. Notably, the lower and upper bound nearly match and recover many known results in bandits and RL. More recently, Foster et al. [2023] refined the previous bounds by introducing the constrained DEC and a corresponding algorithm E2D + .

There are various other results related to the DEC and the E2D algorithm. Foster et al. [2022a] show that the E2D achieves improved bounds in model-free RL when combined with optimistic estimation (as introduced by Zhang [2022]). Chen et al. [2022] introduced two new complexity measures based on the DEC that are necessary and sufficient for reward-free learning and PAC learning. They also introduced new algorithms based on the E2D algorithm for the above two settings and various other improvements. Foster et al. [2022b] have shown that the DEC is necessary and sufficient to obtain low regret for adversarial decision-making. An asymptotically instance-optimal algorithm for DMSO has been proposed by Dong and Ma [2022], extending a similar approach for the linear bandit setting [Lattimore and Szepesvari, 2017].

The decision-estimation coefficient is also related to the information ratio [Russo and Van Roy, 2014] and the decoupling coefficient [Zhang, 2022]. The information ratio has been studied under both the Bayesian [Russo and Van Roy, 2014] and the frequentist regret [Kirschner and Krause, 2018, Kirschner et al., 2020b, 2021, 2023] in various settings including bandits, reinforcement learning, and partial monitoring. The decoupling coefficient was studied for the Thompson sampling algorithm in contextual bandits [Zhang, 2022], and RL [Dann et al., 2021, Agarwal and Zhang, 2022].

2 Setting

We consider the sequential decision-making problem already introduced in the preface. Recall that Π is a compact decision space and O is an observation space. The model class M is a set of tuples f = ( r f , M f ) containing a reward function r f : Π → R and an observation distribution M f : Π → P ( O ) . We define the gap function

where π ∗ g = argmax π ∈ Π r g ( π ) is an optimal decision for model g , chosen arbitrarily if not unique. A randomized policy is a sequence of kernels µ 1: n = ( µ t ) n t =1 from histories h t -1 = ( π 1 , y 1 , . . . , π t -1 , y t -1 ) ∈ (Π ×O ) t -1 to sampling distributions P (Π) . The filtration generated by the history h t is F t . The learner's decisions π 1 , . . . , π n are sampled from the policy π t ∼ µ t and observations y t ∼ M f ∗ ( π t ) are generated by an unknown true model f ∗ ∈ M . The expected regret under model f ∗ is formally defined as follows:

For now, we do not make any assumption about the reward being observed. This provides additional flexibility to model a wide range of scenarios, including for example, duelling and ranking feedback [Yue and Joachims, 2009, Radlinski et al., 2008, Combes et al., 2015, Lattimore et al., 2018, Kirschner and Krause, 2021] (e.g. used in reinforcement learning with human feedback, RLHF) or dynamic pricing [den Boer, 2015]. The setting is more widely known as partial monitoring Rustichini [1999]. The special case where the reward is part of the observation distribution is called decision-making with structured observations [DMSO, Foster et al., 2021]. Earlier work studies the closely related structured bandit setting [Combes et al., 2017].

A variety of examples across bandit models and reinforcement learning are discussed in [Combes et al., 2017, Foster et al., 2021, 2023, Kirschner et al., 2023]. For the purpose of this paper, we focus on simple cases for which we can provide tractable implementations. Besides the finite setting where M can be enumerated, these are the following linearly parametrized feedback models.

Example 2.1 (Linear Bandits, Abe and Long [1999]) . The model class is identified with a subset of R d and features ϕ π ∈ R d for each π ∈ Π . The reward function is r f ( π ) = ⟨ ϕ ( π ) , f ⟩ and the observation distribution is M f ( π ) = N ( ⟨ ϕ π , f ⟩ , 1) .

The linear bandit setting can be generalized by separating reward and feedback maps, which leads to the linear partial monitoring framework [Lin et al., 2014, Kirschner et al., 2020b]. Here we restrict our attention to the special case of linear bandits with side-observations [c.f. Kirschner et al., 2023], which, for example, generalizes the classical semi-bandit setting Mannor and Shamir [2011]

Example 2.2 (Linear Bandits with Side-Observations) . As in the linear bandit setting, we have M⊂ R d , and features ϕ π ∈ R d that define the reward functions r f ( π ) = ⟨ ϕ π , f ⟩ . Observation matrices M π ∈ R m π × d for each π ∈ Π define m π -dimensional observation distributions M f ( π ) = N ( M π f, σ 2 1 m π ) . In addition, we assume that ϕ π ϕ ⊤ π ⪯ M ⊤ π M π , which is automatically satisfied if ϕ ⊤ π is included in the rows of M π , i.e. when the reward is part of the observations.

3 Regret Minimization via Saddle-Point Optimization

̸

The goal of the learner is to choose decisions π ∈ Π that achieve a small gap ∆( π, f ∗ ) under the true model f ∗ ∈ M . Since the true model is unknown, the learner has to collect data that provides statistical evidence to reject models g = f ∗ for which the regret ∆( π, g ) is large. To quantify the information-regret trade-off, we use a divergence D ( ·∥· ) defined for distributions in P ( O ) . For a reference model f , the information (or divergence) function is defined by:

where D KL ( ·∥· ) is the KL divergence. Intuitively, I f ( π, g ) is the rate at which the learner collects statistical information to reject g ∈ M when choosing π ∈ Π and data is generated under the reference model f . Note that I f ( π, f ) = 0 for all f ∈ M and π ∈ Π . As we will see shortly, the regret-information trade-off can be written precisely as a combination of the gap function, ∆ , and the information function, I f . We remark in passing that other choices such as the Hellinger distance are also possible, and the KL divergence is mostly for concreteness and practical reasons.

To simplify the notation and emphasize the bilinear nature of the saddle point problem that we study, we will view ∆ , I f ∈ R Π ×M + as | Π | × |M| matrices (by fixing a canonical ordering on Π and M ). For vectors µ ∈ R Π and ν ∈ R M , we will frequently write bilinear forms µ ∆ f ν and µI f ν . This also means that by convention, µ will always denote a row vector, while ν will always denote a column vector. The standard basis for R Π and R M is ( e π ) π ∈ Π and ( e g ) g ∈M .

3.1 The Decision-Estimation Coefficient

To motivate our approach, we recall the decision-estimation coefficient (DEC) introduced by Foster et al. [2021, 2023], before introducing the main quantity of interest, the average-constrained DEC . First, the offset decision-estimation coefficient (without localization) [Foster et al., 2021] is

The tuning parameter λ > 0 controls the weight of the information matrix relative to the gaps: Viewing the above as a two-player zero-sum game, we see that increasing λ forces the max-player

to avoid models that differ significantly from f under the min-player's sampling distribution. The advantage of this formulation is that the information term µI f e g can be telescoped in the analysis, which directly leads to regret bounds in terms of the estimation error (introduced below in Eq. (8)). The disadvantage of the λ -parametrization is that the trade-off parameter is chosen by optimizing the final regret upper bound. This is inconvenient because the optimal choice requires knowledge of the horizon and a bound on max f ∈M dec o λ ( f ) . Moreover, any choice informed by the upper bound may be conservative, leading to sub-optimal performance.

The constrained decision-estimation coefficient [Foster et al., 2023] is

In this formulation, the max player is restricted to choose models g that differ from f at most by ϵ 2 in terms of the observed divergence under the min-player's sampling distribution. Note that because e π I f e f = 0 for all e π ∈ Π , there always exists a feasible solution. For horizon n , the radius can be set to ϵ 2 ≈ β M n , where β M is a model estimation complexity parameter, thereby essentially eliminating the trade-off parameter from the algorithm. However, because of the hard constraint, strong duality of the Lagrangian saddle point problem (for fixed µ ) fails, and consequently, telescoping the information gain in the analysis is no longer easily possible (or at least, with the existing analysis). To achieve sample complexity dec c ϵ ( f ) , Foster et al. [2023] propose a sophisticated scheme that combines phased exploration with a refinement procedure (E2D + ).

As the main quantity of interest in the current work, we now introduce the average-constrained decision-estimation coefficient , defined as follows:

Similar to the dec c ϵ , the parameterization of the dec ac ϵ is via the confidence radius ϵ 2 , making the choice of the hyperparameter straightforward in many cases (more details in Section 3.2). By convexifying the domain P ( M ) of the max-player, we recover strong duality of the Lagrangian (for fixed µ ). Thereby, the formulation inherits the ease of choosing the ϵ -parameter from the dec c ϵ , while, at the same time, admitting a telescoping argument in the analysis and a much simpler algorithm.

Specifically, Sion's theorem implies three equivalent Lagrangian representations for Eq. (3):

When fixing the outer problem, strong duality holds for the inner saddle-point problem in each line, however, the joint program in Eq. (5) is not convex-concave. An immediate consequence of relaxing the domain of the max player and Eq. (5) is that

The dec ac ϵ can therefore be understood as setting the λ parameter of the dec o λ optimally for the given confidence radius ϵ 2 . On the other hand, the cost paid for relaxing the program is that there exist model classes M where the inequality in Eq. (7) is strict, and dec ac ϵ does not lead to a tight characterization of the regret [Foster et al., 2023, Proposition 4.4]. The remedy is that under a stronger regularity condition and localization, the two notions are essentially equivalent [Foster et al., 2023, Proposition 4.8].

3.2 Anytime Estimation-To-Decisions (Anytime-E2D)

Estimations-To-Decisions (E2D) is an algorithmic framework that directly leverages the decisionestimation coefficient for choosing a decision in each round. The key idea is to compute a sampling distribution µ t ∈ P (Π) attaining the minimal DEC for an estimate ˆ f t of the underlying model, and then define the policy to sample π t ∼ µ t . The E2D approach, using the dec ac ϵ formulation, is summarized in Algorithm 1. To compute the estimate ˆ f t , the E2D algorithm takes an abstract estimation oracle EST as input, that, given the collected data, returns ˆ f t ∈ M . The final guarantee

depends on the estimation error (or estimation regret), defined as the sum over divergences of the observation distributions under the estimate ˆ f t and the true model f ∗ :

Intuitively, the estimation error is well-behaved if ˆ f t ≈ f ∗ , since µ t I f ∗ e f ∗ = 0 . Equation (8) is closely related to the total information gain used in the literature on information-directed sampling [Russo and Van Roy, 2014] and kernel bandits [Srinivas et al., 2010].

To bound the estimation error, Foster et al. [2021] rely on online density estimation (also, online regression or online aggregation ) [Cesa-Bianchi and Lugosi, 2006, Chapter 9]. For finite M , the default approach is the exponential weights algorithm (EWA), which we provide for reference in Appendix A. When using this algorithm, the estimation error always satisfies Est n ≤ log( |M| ) , see [Cesa-Bianchi and Lugosi, 2006, Proposition 3.1]. While these bounds extend to continuous model classes via standard covering arguments, the resulting algorithm is often not tractable without additional assumptions. For linear feedback models (Examples 2.1 and 2.2), one can rely on the more familiar ridge regression estimator, which, we show, achieves bounded estimation regret Est n ≤ O ( d log( n )) . For further discussion, see Appendix A.1.

With this in mind, we state our main result.

Theorem 1. Let λ t ≥ 0 be any sequence adapted to the filtration F t . Then the regret of ANYTIMEE2D (Algorithm 1) with input sequence λ t satisfies for all n ≥ 1 :

where we defined dec ac ϵ,λ ( f ) = min µ ∈ P (Π) max ν ∈ P ( M ) µ ∆ ν -λ ( µI f ν -ϵ 2 ) .

As an immediate corollary, we obtain a regret bound for Algorithm 1 where the sampling distribution µ t is chosen to optimize dec ac ϵ t for any sequence ϵ t .

Corollary 1. The regret of ANYTIME-E2D (Algorithm 1) with input ϵ t ≥ 0 satisfies for all n ≥ 1 :

Importantly, the regret of Algorithm 1 is directly controlled by the worst-case DEC, max f ∈M dec ac ϵ ( f ) , and the estimation error Est n . It remains to set ϵ 2 t (respectively λ t ) appropriately. For a fixed horizon n , we let ϵ 2 t = Est n n . With the reasonable assumption that max f ∈M { ϵ -2 dec ac ϵ ( f ) } is non-decreasing in ϵ , Corollary 1 reads

This almost matches the lower bound R n ≥ Ω( n dec c 1 / √ n ( F )) 3 [Foster et al., 2023, Theorem 2.2], up to the estimation error and the beforehand mentioned gap between dec c ϵ and dec ac ϵ .

To get an anytime algorithm with essentially the same scaling as in Eq. (9), we set ϵ 2 t = log( |M| ) /t for finite model classes, and ϵ 2 t = β M t if Est t ≤ β M log( t ) for β M > 0 . For linear bandits, dec ac ϵ ≤ ϵ √ d (see Section 3.3), and Est n ≤ d log( n ) . Choosing ϵ 2 t = d/t recovers the optimal regret bound R n ≤ ˜ O ( d √ n ) [Lattimore and Szepesvári, 2020a]. Alternatively, one can also choose λ t by minimizing an upper bound on max t ∈ [ n ] ,f ∈M { dec ac ϵ t ,λ t ( f ) /ϵ 2 t } . For example, in linear bandits, dec ac ϵ t ,λ ≤ d 4 λ + λϵ 2 t (see Table 1); hence, for ϵ 2 t = d/t , we can set λ t = t/ 4 . Further discussion and refined upper bound for linear feedback models are in Section 3.3.

Proof of Theorem 1. Let µ ∗ t and ν ∗ t be a saddle-point solution to the offset dec,

Note that µ ∗ t ∆ ν ∗ t -λ t µ ∗ t I f ν ∗ t ≥ µ ∗ t ∆ e f -λ t µ ∗ t I f e f ≥ 0 , which implies that λ t ϵ 2 t ≤ dec ac ϵ t ,λ t . Next,

So far, we only introduced the saddle point problem by maximizing over f ∗ . The last equality is by our choice of λ t and µ t , and noting that ν ∈ P ( M ) can always be realized as a Dirac. Continuing,

We first drop the negative term in ( i ) and use the beforehand stated fact that λ t ϵ 2 t ≤ dec ac ϵ t ,λ t ( ˆ f t ) . The last step, ( ii ) , is taking the maximum out of the sum.

3.3 Certifying Upper Bounds

As shown by Corollary 1, the regret of Algorithm 1 scales directly with the dec ac ϵ . For analysis purposes, it is however useful to compute upper bounds on the dec ac ϵ to verify the scaling w.r.t. parameters of interest. Via the equivalence Eq. (7), bounds on the dec o λ directly translate to the dec ac ϵ (see Table 1). For a detailed discussion of upper bounds in various models, we refer to Foster et al. [2021]. Below, we highlight three connections that are directly facilitated by the dec ac ϵ .

To this end, we first introduce a variant of the dec ac ϵ where the gap function depends on f :

where ∆ f ( π, g ) = r g ( π ∗ g ) -r f ( π ) . We remark that for distributions ν ∈ P ( M ) and µ ∈ P (Π) , the gap ∆ f can be decoupled, µ ∆ f ν = δ f ν + µ ∆ f e f , where we defined δ f ( g ) = r g ( π ∗ g ) -r f ( π ∗ f ) . The following assumption implies that the observations for a decision π are at least as informative as observing the rewards.

Assumption 1 (Reward Data Processing) . The rewards and information matrices are related via the following data-processing inequality that holds for any µ ∈ P (Π) :

The next lemma shows that under Assumption 1, dec ac ϵ ( f ) and dec ac,f ϵ ( f ) are essentially equivalent, at least for the typical worst-case bounds where max f ∈M dec ac ϵ ( f ) ≥ Ω( ϵ ) .

Lemma 1. If Assumption 1 holds, then

The proof is in Appendix C.1. We remark that Algorithm 1 where the sampling distribution is computed for dec ac,f ϵ ( ˆ f t ) and ∆ f achieves a bound analogous to Theorem 1, as long as Assumption 1 holds. For details see Lemma 8 in Appendix C.

Upper Bounds via Decoupling First, we introduce the information ratio ,

The definition is closely related to the Bayesian information ratio [Russo and Van Roy, 2016], where ν takes the role of a prior over M . The Thompson sampling distribution is µ TS ν = ∑ h ∈M ν h e π ∗ h . The decoupling coefficient, dc ( f ) , [Zhang, 2022, Definition 1] is defined as the smallest number K ≥ 0 , such that for all distributions ν ∈ P ( M ) ,

The next lemma provides upper bounds on the dec ac ϵ ( f ) in terms of the information ratio, which is further upper-bounded by the decoupling coefficient.

Lemma 2. With Ψ( f ) = max ν ∈M min µ ∈ P (Π) Ψ f ( µ, ν ) and Assumption 1 satisfied, we have

The proof follows directly using the AM-GM inequality, see Appendix C.2. By [Zhang, 2022, Lemma 2], this further implies dec ac,f ϵ ≤ ϵ √ d . An analogous result for the generalized information ratio [Lattimore and Gyorgy, 2021] that recovers rates ϵ ρ for ρ ≤ 1 is given in Appendix C.4.

PAC to Regret Another useful way to upper bound the dec ac,f ϵ is via an analogous definition for the PAC setting [c.f. Eq. (10), Foster et al., 2023]:

Lemma 3. Under Assumption 1,

The proof is given in Appendix B.1. Lemma 3 combined with Theorem 1 leads to O ( n 2 / 3 ) upper bounds on the regret that are reminiscent of so-called globally observable games in linear partial monitoring [Kirschner et al., 2023].

Application to Linear Feedback Models To illustrate the techniques introduced, we compute a regret bound for Algorithm 1 for linear bandits with side-observations (Examples 2.1 and 2.2).

Lemma 4. For linear bandits with side-observations and divergence I f ( π, g ) = ∥ M π ( g -f ) ∥ 2 ,

where V ( µ ) = ∑ π ∈ Π µ π M π M ⊤ π . Moreover, denoting Ω = min µ ∈ P (Π) max b ∈ Π ∥ ϕ b ∥ V ( µ ) -1 ,

The proof is given in Appendix B.2. While in the worst-case for linear bandits, there is no improvement over the standard O ( d √ n ) without further refinement or specification of the upper bounds, in the case of linear side-observations there is an improvement whenever Ω ≤ max f ∈M Ψ( f ) . To exemplify the improvement, consider a semi-bandit with a 'revealing' action ˆ π , e.g. M ˆ π = 1 d . Here, the regret bound improves to R n ≤ min { d √ n, d 1 / 3 n 2 / 3 } , since then pac-dec ac,f ϵ ( f ) ≤ ϵ . The corresponding improvement in the regime n ≤ d 4 might seem modest, but is relevant in high-dimensional and non-parametric models. Moreover, in (deep) reinforcement learning, high-dimensional models are commonly used and the learner obtains side information in the form of state observations. Therefore, it is plausible that the n 2 / 3 rate is dominant even for a moderate horizon. Exploring this effect in reinforcement learning is therefore an important direction for future work.

Notably, this improvement is not observed by upper confidence bound algorithms and Thompson sampling, because both approaches discard informative but suboptimal actions early on [c.f. Lattimore and Szepesvari, 2017], including the action ˆ π in the example above. E2D for a constant offset parameter λ > 0 , in principle, attains the better rate, but only if one pre-commits to a fixed horizon. Lastly, we note that a similar effect was observed for information-directed sampling in sparse high-dimensional linear bandits [Hao et al., 2020].

3.4 Computational Aspects

For finite model classes, Algorithm 1 can be readily implemented. Since almost no structure is imposed on the gap and information matrices of size | Π | × |M| , avoiding scaling with | Π | · |M| seems hardly possible without introducing additional assumptions. Even in the finite case, solving Eq. (3) is not immediate because the corresponding Lagrangian is not convex-concave. A practical approach is to solve the inner saddle point for Eq. (5) as a function of λ . Strong duality holds for the inner problem, and one can obtain a solution efficiently by solving the corresponding linear program using standard solvers. It then remains to optimize over λ ≥ 0 . This can be done, for example, via a grid search over the range [0 , max f ∈M ϵ -2 dec ac ϵ ( f )] .

In the linear setting, the above is not satisfactory because most commonly M is identified with parameters in R d . As noted before, ridge regression can be used instead of online aggregation while preserving the optimal scaling of the estimation error (see Appendix A.1). The next lemma further shows that the saddle point problem Eq. (3) can be rewritten to only scale with the size of the decision set | Π | .

Lemma 5. Consider linear bandits with side observations, M = R d and quadratic divergence, I f ( π, g ) = ∥ M π ( g -f ) ∥ 2 , and denote ϕ µ = ∑ π ∈ Π µ π ϕ π and V ( µ ) = ∑ π ∈ Π µ π M ⊤ π M π . Then

Moreover, the objective is convex in µ ∈ P (Π) .

The proof is a straightforward calculation provided in Appendix C.5. Note that the saddle point expression is analogous to Eq. (5), and in fact, one can linearize the inner maximization over P (Π) , such that the inner saddle point becomes convex-concave. This leads to expressions equivalent to Eqs. (4) and (6), albeit the objective is no longer linear in µ ∈ P (Π) . We use Lemma 5 to employ the same strategy as before: As a function of λ ≥ 0 , solve the inner problem of the expression in Lemma 5, for example, as a convex program with | Π | variables and | Π | constraints (Appendix D). Then all that remains is to solve a one-dimensional optimization problem over λ ∈ [0 , max f ∈M ϵ -2 dec ac ϵ ( f )] . We demonstrate this approach in Appendix E to showcase the performance of E2D on simple examples.

4 Conclusion

We introduced ANYTIME-E2D, an algorithm based on the estimation-to-decisions framework for sequential decision-making with structured observations. The algorithm optimizes the averageconstrained decision-making coefficient, which can be understood as a reparametrization of the corresponding offset version. The reparametrization facilitates an elegant anytime analysis and makes setting all remaining hyperparameters immediate. We demonstrate the improvement with a novel bound for linear bandits with side-observations, that is not attained by previous approaches. Lastly, we discuss how the algorithm can be implemented for finite and linear model classes. Nevertheless, much remains to be done. For example, one can expect the reference model to change very little from round to round, and therefore, it seems wasteful to solve Eq. (3) from scratch repetitively. Preferable instead would be an incremental scheme that iteratively computes updates to the sampling distribution.

Acknowledgments and Disclosure of Funding

Johannes Kirschner gratefully acknowledges funding from the SNSF Early Postdoc.Mobility fellowship P2EZP2_199781.

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Abe and P. M. Long. Associative reinforcement learning using linear probabilistic concepts. In Proceedings of the Sixteenth International Conference on Machine Learning , ICML '99, pages 3-11, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2.

Agarwal and T. Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659 , 2022.

G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Bubeck, O. Dekel, T. Koren, and Y. Peres. Bandit convex optimization: \ sqrtt regret in one dimension. In Conference on Learning Theory , pages 266-278. PMLR, 2015.

Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

Chen, S. Mei, and Y. Bai. Unified algorithms for rl with decision-estimation coefficients: No-regret, pac, and reward-free learning. arXiv preprint arXiv:2209.11745 , 2022.

Combes, S. Magureanu, A. Proutiere, and C. Laroche. Learning to rank: Regret lower bounds and efficient algorithms. In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems , pages 231-244. ACM, 2015. ISBN 978-14503-3486-0.

Combes, S. Magureanu, and A. Proutiere. Minimal exploration in structured stochastic bandits. Advances in Neural Information Processing Systems , 30, 2017.

Dann, M. Mohri, T. Zhang, and J. Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems , 34:12040-12051, 2021.

Degenne, P. Ménard, X. Shang, and M. Valko. Gamification of pure exploration for linear bandits. In International Conference on Machine Learning , pages 2432-2442. PMLR, 2020a.

Degenne, H. Shao, and W. Koolen. Structure adaptive algorithms for stochastic bandits. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2443-2452. PMLR, 13-18 Jul 2020b. URL https://proceedings.mlr.press/v119/degenne20b.html .

V. den Boer. Dynamic pricing and learning: Historical origins, current research, and new directions. Surveys in Operations Research and Management Science , 20(1):1-18, 2015.

Dong and T. Ma. Asymptotic instance-optimal algorithms for interactive decision making. arXiv preprint arXiv:2206.02326 , 2022.

Du, S. Kakade, J. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

C. Dunn and S. Harshbarger. Conditional gradient algorithms with open loop step size rules. Journal of Mathematical Analysis and Applications , 62(2):432-444, 1978.

J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

J. Foster, N. Golowich, J. Qian, A. Rakhlin, and A. Sekhari. A note on model-free reinforcement learning with the decision-estimation coefficient. arXiv preprint arXiv:2211.14250 , 2022a.

J. Foster, A. Rakhlin, A. Sekhari, and K. Sridharan. On the complexity of adversarial decision making. arXiv preprint arXiv:2206.13063 , 2022b.

J. Foster, N. Golowich, and Y. Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. arXiv preprint arXiv:2301.08215 , 2023.

Frank and P. Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly , 3(1-2):95-110, 1956.

Garcelon, B. Roziere, L. Meunier, J. Tarbouriech, O. Teytaud, A. Lazaric, and M. Pirotta. Adversarial attacks on linear contextual bandits. Advances in Neural Information Processing Systems , 33:14362-14373, 2020.

György, D. Pál, and C. Szepesvári. Online learning: Algorithms for big data. Lecture Notes , 2013.

Hao, T. Lattimore, and M. Wang. High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems , 33:10753-10763, 2020.

Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning , pages 427-435. PMLR, 2013.

Jin, Z. Allen-Zhu, S. Bubeck, and M. I. Jordan. Is q-learning provably efficient? Advances in neural information processing systems , 31, 2018.

Kirschner and A. Krause. Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory , pages 358-384. PMLR, 2018.

Kirschner and A. Krause. Bias-robust bayesian optimization via dueling bandits. In International Conference on Machine Learning , pages 5595-5605. PMLR, 2021.

Kirschner, I. Bogunovic, S. Jegelka, and A. Krause. Distributionally Robust Bayesian Optimization. In Proc. International Conference on Artificial Intelligence and Statistics (AISTATS) , August 2020a. URL http://proceedings.mlr.press/v108/kirschner20a/kirschner20a.pdf .

Kirschner, T. Lattimore, and A. Krause. Information directed sampling for linear partial monitoring. In Conference on Learning Theory , pages 2328-2369. PMLR, 2020b.

Kirschner, T. Lattimore, C. Vernade, and C. Szepesvári. Asymptotically optimal informationdirected sampling. In Conference on Learning Theory , pages 2777-2821. PMLR, 2021.

Kirschner, T. Lattimore, and A. Krause. Linear partial monitoring for sequential decision making: Algorithms, regret bounds and applications. Journal of Machine Learning Research , August 2023.

Lattimore and A. Gyorgy. Mirror descent and the information ratio. In Conference on Learning Theory , pages 2965-2992. PMLR, 2021.

Lattimore and C. Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728-737. PMLR, 2017.

Lattimore and C. Szepesvári. An information-theoretic approach to minimax regret in partial monitoring. In Conference on Learning Theory , pages 2111-2139. PMLR, 2019.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020a.

Lattimore and C. Szepesvári. Exploration by optimisation in partial monitoring. In Conference on Learning Theory , pages 2488-2515. PMLR, 2020b.

Lattimore, B. Kveton, S. Li, and C. Szepesvári. Toprank: A practical algorithm for online stochastic ranking. In Advances in Neural Information Processing Systems , pages 3949-3958. Curran Associates, Inc., 2018.

Lin, B. Abrahao, R. Kleinberg, J. Lui, and W. Chen. Combinatorial partial monitoring game with linear feedback and its applications. In International Conference on Machine Learning , pages 901-909, 2014.

Mannor and O. Shamir. From bandits to experts: On the value of side-observations. Advances in Neural Information Processing Systems , 24, 2011.

Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems , 29, 2016.

Radlinski, R. Kleinberg, and T. Joachims. Learning diverse rankings with multi-armed bandits. In Proceedings of the 25th International Conference on Machine Learning , pages 784-791. ACM, 2008.

Russo and B. Van Roy. Learning to optimize via information-directed sampling. Advances in Neural Information Processing Systems , 27, 2014.

Russo and B. Van Roy. An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research , 17(1):2442-2471, 2016.

Rustichini. Minimizing regret: The general case. Games and Economic Behavior , 29(1-2): 224-243, 1999.

Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.

Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. International Conference on Machine Learning (ICML) , 2010.

R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Yue and T. Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th International Conference on Machine Learning , pages 1201-1208. ACM, 2009.

Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Zhou, Q. Gu, and C. Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Algorithm 2: Exponential Weights Algorithm (EWA) for Density Estimation

Input : Finite model class M , data D t = { ( y 1 , π 1 ) , . . . , ( y t , π t ) } , Learning rate η > 0

1 Define L ( f ) = -∑ t s =1 log M f ( y s | π s )

3 For convex M : Return ∑ f ∈M p ( f ) f

2 Let p ( f ) ∝ exp( -ηL ( f ))

4 Else: Return f ∼ p ( · ) .

A Online Density Estimation

For any f ∈ M and π ∈ Π , we denote by p ( ·| π, f ) the the density function of the observation distribution M f ( π ) w.r.t. a reference measure over the observation space O . Consider a finite model class M and the KL divergence,

In this case, the estimation error can be written as follows:

The last line can be understood as the estimation regret of the estimates ˆ f 1 , . . . , ˆ f n under the logarithmic loss. A classical approach to control this term is the exponential weights algorithm (EWA) given in Algorithm 2. For the EWA algorithm, we have the following bound.

Lemma 6 (EWA for Online Density Estimation) . For any data stream { y 1 , π 1 , . . . , y n , π n } the predictions ˆ f 1 , . . . ˆ f n obtained via Algorithm 2 with η = 1 satisfy

For a proof, see [Cesa-Bianchi and Lugosi, 2006, Proposition 3.1].

A.1 Bounding the Estimation Error of Projected Regularized Least-Squares

In this section, we consider the linear model from Example 2.2. We denote by ∥ · ∥ the Euclidean norm. For simplicity, the observation maps M π ∈ R m × d are assumed to have the same output dimension m ∈ N . The observation distribution is such that y t = M π t f ∗ + ξ t , where ξ ∈ R m is random noise such that E t [ ξ ] = 0 and E t [ ∥ ξ ∥ 2 ] ≤ σ 2 . Here, E t [ · ] = E [ ·| π 1 , y 1 , . . . , π t -1 , y t -1 , π t ] is the conditional observation in round t including the decision π t chosen in round t .

We will use the quadratic divergence 4 , e π I f e g = 1 2 ∥ M π ( g -f ) ∥ 2 This choice corresponds to the Gaussian KL, but we do not require that the noise distribution is Gaussian is the following. In the linear bandit model, this choice reduces to e π I f e g = 1 2 ⟨ ϕ π , g -f ⟩ 2 .

Let K ⊂ R d be a closed convex set. Our goal is to control the estimation regret for the projected regularized least-squares estimator,

where V 0 is a positive definite matrix, V t = ∑ t -1 s =1 M ⊤ π s M π s + V 0 and Proj V t ( · ) is the orthogonal projection w.r.t. the ∥ · ∥ V t norm. For K = R d and V 0 = η 1 d , this recovers the standard ridge

regression. The projection is necessary to bound the magnitude of the squared loss, and the result will depend on an almost-surely bound on the 'observed' diameter,

Recall that our goal is to bound the estimation error,

We remark that one can get the following naive bound by applying Cauchy-Schwarz:

The last inequality follows from the elliptic potential lemma and standard concentration inequalities [Lattimore and Szepesvári, 2020a, Lemma 19.4 and Theorem 20.5]. However, this will lead to an additional d -factor in the regret that can be avoided, as we see next.

For K = R d , one-dimensional observations and noise bounded in the range [ -¯ B, ¯ B ] , one can also directly apply [Cesa-Bianchi and Lugosi, 2006, Theorem 11.7] to get Est n ≤ O ( ¯ B 2 d log( n )) , thereby improving the naive bound by a factor d log( n ) . This result is obtained in a more general setting, where no assumptions, other than boundedness, are placed on the observation sequence y 1 , . . . , y n . Here we refine and generalize this result in two directions: First, we allow for the more general feedback model in with multi-dimensional observations (Example 2.2). Second, we directly exploit the stochastic observation model to obtain a stronger result that does not require the observation noise to be bounded.

Theorem 2. Consider the linear observation setting with additive noise and quadratic divergence e π I f e g = 1 2 ∥ M π ( g -f ) ∥ 2 , as described at the beginning of this section. Assume that max f,g ∈M ,π ∈ Π ∥ M π ( f -g ) ∥ ≤ B and E [ ∥ ξ t ∥ 2 ] ≤ σ 2 . Then

If in addition ∥ M π ∥ ≤ L and V 0 = η 1 d , then Est n ≤ ( σ 2 + B 2 ) log ( 1 + nL 2 ηd ) .

Remark 1. Note that by the [Lattimore and Szepesvári, 2020a, Theorem 19.4], log ( det V n det V 0 ) can further be upper bounded by d log ( trace V 0 + nL 2 d det( V 0 ) 1 /d ) , which effectively results the desired bound.

Proof. The proof adapts [György et al., 2013, Theorem 19.8] to multi-dimensional observations and takes advantage of the stochastic loss function by taking the expectation.

First, define l t ( f ) = 1 2 ∥ M π t f -y t ∥ 2 . Then, using that E t [ y t ] = M π t f ∗ ,

Further, by directly generalizing [György et al., 2013, Lemma 19.7], we have that

We now start upper bounding the estimation error,

The inequality ( i ) follows from [Shalev-Shwartz et al., 2012, Lemma 2.3]. For ( ii ) we used Eq. (18). For ( iii ) we used that E t [ ξ t ] = 0 . In ( iv ) , we introduce the maximum eigenvalue λ max ( A ) for A ∈ R m × m and denote ¯ x t = M π t ( f ∗ -f t -1 ) . Lastly, in ( v ) we used that ∥ ¯ x t ∥ 2 ≤ B and E t [ ∥ ξ t ∥ 2 ] ≤ σ 2 .

We conclude the proof with basic linear algebra. Denote by λ i ( A ) the i -th eigenvalue of a matrix M ∈ R m × m . Using the generalized matrix determinant lemma, we get

Note that λ i ( M ⊤ π t V -1 t M π t ) ∈ (0 , 1] . Next, using that log(1 -x ) ≤ -x for all x < 1 , we get that

Rearranging the last display, and bounding the sum by its maximum element, we get

The proof is concluded by combining Eqs. (19) and (20).

Remark 2 (Expected Regret) . The beauty of Theorem 1 is that the proof uses only in-expectation arguments. This is unlike most previous analysis, that controls the regret via controlling tail-events, and bounds on the expected regret are then derived a-posteriori from high-probability bounds. In the context of linear bandits, Theorem 2 leads to bound on the expected regret that only requires the noise variance to be bounded, whereas most previous work relies on the stronger sub-Gaussian noise assumption [e.g. Abbasi-Yadkori et al., 2011].

Remark 3 (Kernel Bandits / Bayesian Optimization) . Using the standard 'kernel-trick', the analysis can further be extended to the non-parametric setting where M is an infinite-dimensional reproducing kernel Hilbert space (RKHS).

B PAC to Regret Bounds

B.1 Proof of Lemma 3

Proof. The Lagrangian for Eq. (12) is

Reparametrize any µ ∈ P (Π) as ¯ µ ( p ) = (1 -p ) e π ∗ f + pµ 2 . We bound dec ac,f ϵ by a function of pac-dec ac,f ϵ . Starting from Eq. (5), we have

B.2 Proof of Lemma 4

Proof of Lemma 4. For the first part, note that

Equation ( i ) follows by computing the maximizer attaining the quadratic form over M = R d . The inequality ( ii ) is by definition of π ∗ f and the last inequality ( iii ) by the assumption that the reward is observed, respectively, ϕ π ϕ ⊤ π ⪯ M ⊤ π M π , and the Kiefer-Wolfowitz theorem.

The second part of the statement follows by combining Lemmas 2 and 3.

C Coefficient Relations Results and Proofs

Lemma 7. Assume Assumption 1 holds, i.e.

Then

Proof. First Jensen's inequality, then Eq. (21).

C.1 Proof of Lemma 1

Proof of Lemma 1. Note that

where the first inequality is by Lemma 7. Also, by lower bounding the sum in the second inequality by -√ µI f ν we get left inequality.

C.2 Proof of Lemma 2

Proof of Lemma 2. For the first inequality, using the definition of dec ac ϵ ( f, ∆ f ) and the AM-GM inequality:

Further, by Eq. (11) and Assumption 1 we have µ TS ν ∆ f ν ≤ √ K ∑ g,h ∈M ν g ν h e π ∗ h ( r g -r f ) 2 ≤ √ Kµ TS f I f ν , which gives Ψ( f ) ≤ K . Plugging this into Eq. (23) gives the second inequality.

C.3 Regret bound for Algorithm 1 defined for ∆ f and dec ac,f ϵ

Lemma 8. If Assumption 1 holds, then the regret of ANYTIME-E2D (Algorithm 1) with ∆ replaced with ∆ f is bounded as follows:

Proof. The proof follows along the lines of the proof of Theorem 1. The main difference is that when introducing ∆ f , we get a term that captures the reward estimation error:

For the last inequality, we used Cauchy-Schwarz and Assumption 1 to bound the error term,

C.4 Generalized Information Ratio

The generalized information ratio [Lattimore and Gyorgy, 2021] for µ ∈ P (Π) , ν ∈ P ( M ) , and α > 1 is defined as

For α = 2 , we get the standard information ratio introduced by Russo and Van Roy [2014] with ν as a prior over the model class M . Define Ψ α ( f ) = max ν ∈M min µ ∈ P (Π) Ψ α,f ( µ, ν ) . To upper bound dec ac ϵ , we have the following lemma.

Lemma 9. For the reference model f , the ac-dec can be upper bounded as

for α > 1 .

Proof. Westart by noting that for x 1 , . . . , x α ≥ 0 , from AM-GM we have that α ( x 1 · x 2 · · · x α ) 1 /α ≤ x 1 + · · · + x α . Substituting x 2 = x 3 = · · · x α , we get α · x 1 α 1 · x α -1 α 2 -x 1 ≤ ( α -1) x 2 . Writing x 1 = λµI f ν and x 2 = α α 1 -α ( ( µ ∆ ν ) α λµI f ν ) 1 α -1 and using the previous inequality with the dec ac ϵ program gives the result.

The information ratio Ψ α,f ( µ, ν ) can be thought of as the Bayesian information ratio in [Russo and Van Roy, 2014] where the expectation is taken over the distribution ν of possible environments. However, for information gain I f , [Russo and Van Roy, 2014] use entropy difference in the posterior distribution of π ∗ f before and after the observation is revealed.

C.5 Proof of Lemma 5

Proof of Lemma 5.

The first equality is by definition, and the second equality follows from solving the quadratic maximization over g ∈ M = R d . To show that the problem is convex in µ , note that taking inverses of positive semi-definite matrices X,Y is a convex function, i.e. ((1 -η ) X + ηY ) -1 ⪯ (1 -η ) X -1 + ηY -1 . In particular, V ((1 -η ) µ 1 + ηµ 2 ) -1 ⪯ (1 -η ) V ( µ 1 ) -1 + ηV ( µ 2 ) -1 . With this the claim follows.

D Convex Program for Fixed λ

Take Eq. (28) and fix λ > 0 . Then we have the following saddle-point problem:

Up to the constant additive term, this saddle point problem is equivalent to the following convex program

E Experiments

All experiments below were run on a semi-bandit problem with a "revealing action", as alluded to in the paragraph below Lemma 4. Specifically, we assume a semi-bandit model where M = R d and the features are ϕ π ∈ R d . For an instance f ∗ ∈ M , the reward function is r f ∗ = ⟨ ϕ π , f ∗ ⟩ for all π ∈ Π . There is one revealing (sub-optimal) action ˆ π = π ∗ f ∗ . The observation for any action π = ˆ π is

̸

̸

Define M ˆ π = [ ϕ π 1 , . . . , ϕ π | Π | ] ⊤ . Then the observation for action ˆ π is

̸

Thus, the information for any action π = ˆ π is

while the information for action ˆ π is

For this setting Est n ≤ O ( d log( n )) (see Appendix A.1).

E.1 Experimental Setup

Our main objective is to compare our algorithm ANYTIME-E2D to the fixed-horizon E2D algorithm by Foster et al. [2021]. ANYTIME-E2D and E2D were implemented by using the procedure described in Section 3.4. Both ANYTIME-E2D and E2D need to solve the inner convex problem in Lemma 5. To do so we use Frank-Wolfe [Frank and Wolfe, 1956, Dunn and Harshbarger, 1978, Jaggi, 2013] for 100 steps and warm-starting the optimization at the solution from the previous round, µ t -1 . For ANYTIME-E2D we further perform a grid search over λ ∈ [0 , max g ∈M ϵ -2 dec ac,f ( g )] (with a discretization of 50 points) to optimize over lambda within each iteration of Frank-Wolfe. For both the E2D and ANYTIME-E2D algorithm we used the version with the gaps ∆ replaced with ∆ f , since we noticed that both algorithms performed better with ∆ f . For E2D, the scale hyperparameter λ was set using λ = √ n 4 log( n ) as mentioned in Foster et al. [2021, Section 6.1.1]. While for ANYTIMEE2D we set the hyper-parameter ϵ 2 t = d/t . Further, we compare to standard bandit algorithms: Upper

Confidence Bound (UCB) and Thompson Sampling (TS) [Lattimore and Szepesvári, 2020a].

E.2 Experiment 1

In this experiment, we aim to demonstrate the advantage of having an anytime algorithm. Specifically, we tune λ in the E2D algorithm for different horizons n = 200 , 500 , 1000 , 2000 , but run it for a fixed horizon of n = 2000 . As such, we expect our algorithm ANYTIME-E2D to perform better than E2D when λ was tuned for the incorrect horizons (i.e. n = 200 , 500 , 1000 ). The feature dimension is d = 3 . The number of decisions is | Π | = 10 . We generated the features ϕ π for each π ∈ Π and parameter f ∗ ∈ R d randomly at the beginning and then kept them fixed throughout the experimentation. 100 independent runs were performed for each algorithm.

The results of the experiment can be seen as the left plot in Fig. 1. As expected, our algorithm ANYTIME-E2D performs better than E2D (for n = 200 , 500 , 1000 ). This indicates that the E2D algorithm is sensitive to different settings of λ , which is problematic when the horizon is not known beforehand. Whereas our ANYTIME-E2D algorithm performs well even when the horizon is not known.

E.3 Experiment 2

In this experiment, we investigate the case when n < d 4 . As pointed out below Lemma 3, we expect improvement in this regime as the regret bound of our algorithm is R n ≤ min { d √ n, d 1 / 3 n 2 / 3 } ,

0

500

1000

1500

2000

step

0

10

20

30

regret

0

200

400

600

800

1000

step

0

5

10

15

20

25

regret

TS

UCB

E2D-200

E2D-500

E2D-1000

E2D-2000

AnytimeE2D

while the default, fixed-horizon E2D algorithm cannot achieve these bounds simultaneously and one has to pick one of d √ n or d 1 / 3 n 2 / 3 beforehand for setting the scale hyperparameter λ . It is standard that the choice of λ is made according to the d √ n regret bound for E2D Foster et al. [2021](which is not optimal when n < < d 4 ), especially, if the horizon is not known beforehand. Thus, we set the horizon to n = 1000 and the dimension of the feature space to d = 30 , which gives us that n = 1000 < < 810000 = d 4 . The rest of the setup and parameters are the same as in the previous experiment except for the features ϕ π and f ∗ which are again chosen randomly in the beginning and then kept fixed throughout the experiment.

The results of the experiment can be seen as the right plot in Fig. 1. As expected, our algorithm ANYTIME-E2D performs better than E2D, UCB, and TS. This indicates that indeed, ANYTIMEE2D is likely setting λ appropriately to achieve the preferred d 1 / 3 n 2 / 3 regret rate for small horizons. The poor performance of the other algorithms can be justified, since E2D is optimized based on the worse d √ n regret rate (for small horizons), while the UCB and TS algorithms are not known to get regret better than d √ n .

Peer reviews:
comments: SUMMARY:
This paper focuses on regret minimization in sequential decision-making under uncertainty. It introduces the average-constrained decision-estimation coefficient, a saddle-point objective that characterizes the worst-case regret, enabling optimization of the information trade-off directly by the algorithm. Moreover, the paper presents a version of the Estimation-To-Decisions (E2D) algorithm (ANYTIME-E2D), with practical implementation details, improved bounds for high-dimensional linear bandits, and the first empirical results of the E2D algorithm.

STRENGTHS:
The paper offers several results. The regret results are new since they are based on (3).

WEAKNESSES:
It is unclear how innovative the paper is. The brunt of the mathematics in the paper is drawn from existing prior results. Many mathematical derivations rely on standard Lagrangian (equations (3)-(7) and the proof of Lemma 1).

QUESTIONS:
What is the importance and impact of the new algorithm? Why is it better, more effective?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a framework to solve the bandit problem by means of the minimax problem, which has been rapidly developed in recent years.\nAmong them, the paper focuses on the decision-estimation coefficient (DEC).\nThe DEC was developed in a series of studies by Foster+ and is known to characterize the upper and lower bound of the worst-case regret in a general class of bandit problems.\nThe Estimation-to-Decisions (E2D) framework, an algorithm using DEC, determines action selection probabilities based on DEC.\nHowever, the algorithm based on offset DEC, which is an existing DEC, is expected to have conservative performance because it needs to determine the tradeoff parameter based on the regret upper bound.\nFurthermore, the recently developed constrained DEC does not require a tradeoff parameter, but has the problem that the strongly duality of a Lagrangian saddle point fails.\nTo solve these problems, the authors propose average-constrained DEC, a formulation in which the feasible region of the max player in the constrained DEC formulation is relaxed.\nThis eliminates the need to determine the tradeoff parameter based on the regret upper bound while maintaining the good properties of the optimization problem, resulting in an anytime algorithm.\nFurthermore, by considering a variant of average-constrained DEC, the authors show that an improved rate can be obtained for high-dimensional linear bandits, and this is confirmed through actual numerical experiments.\n\nSTRENGTHS:\n- Overall, the paper is written in a very clear manner. In particular, it clearly summarizes the characteristics of the DECs proposed so far.\n- It clearly points out the problems of existing algorithms and presents simple solutions to solve them.\n- The authors devise a specific procedure for obtaining a regret upper bound not only when the model is discrete, but also when linear bandits ( possibly with side observations) are used.\n- While no numerical experimental evaluation of the DEC has been done so far, the effectiveness of the DEC is demonstrated through numerical experiments in the high dimensional linear bandits.\n\nWEAKNESSES:\n- Abstract and introduction are slightly misleading, as it appears that the average-constrained DEC results can be used directly to obtain results for high-dimensional linear bandits.\n- Although partial monitoring is mentioned in l120-125, the authors are actually solving the problem in a setting where rewards are also observed (or can be assumed to be included), as described in Example 1.2. The authors do not explain to what extent there is a gap with partial monitoring, and it is not clear to what extent the proposed algorithm is applicable to partial monitoring.\n- As mentioned by the authors, there are other approaches to characterizing sequential decision problems by minimax problems, such as IDS and ExpByOpt. A brief explanation of why the focus is on DEC would be desirable.\n\nMinor issues and typos:\n\n- l31: $M_f$ should be $M_{f^*}$\n- the meaning of \"localization\" was unclear and it would be better to explain the meaning in short.\n- In footnote 3, $\\max_{f \\in co(\\mathcal{M})}$ is $\\max_{f \\in \\mathcal{F}}$?\n\nQUESTIONS:\n- While the original DEC is formulated based on the Hellinger distance, this paper uses KL divergence for simplicity. What are the advantages and disadvantages that arise from these?\n- (e.g.,l169) The meaning of telescope was unclear. Can you explain the meaning of this?\n- (after l244) For the part of inequality (i), shouldn't the second term of max contains $\\mathrm{dec}^{ac}_{\\epsilon_t}(\\hat{f}_t)$? (The order and the last equation are correct.)\n- (l226) Why is the assumption \"$\\max_f \\{ \\epsilon^2 \\mathrm{dec}^{ac}_{\\epsilon} (f)\\}$ is non-decreasing in $\\epsilon$\" reasonable? Is there any exception?\n- Foster+2022 also devised DEC in the adversarial setting. Is the same kind of reparameterization possible in adversarial setting as in the paper? Is there any exception?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the estimation-to-decisions framework for sequential decision-making problems with structured observations. They propose the ANYTIME-E2D algorithm, improving precious approaches with a novel bound for linear bandits. Numerical simulations are presented to show the performance of their algorithm against the baselines.

STRENGTHS:
- This paper is in general well-written and the overall structure is nicely organized. The proofs seem to be theoretically rigorous. 
- The significance of the paper is relatively high, since the proposed algorithm can be applied to linear bandit settings and is achieving improved rates in the high-dimensional regime, which is getting increasing attention in the literature.

WEAKNESSES:
- My first concern is the originality and significance of this work, over the existing E2D papers. It is claimed that they introduce the new objective $\operatorname{dec}_\epsilon^{a c}$, which enables the optimization of the information trade-off directly instead of the regret bound. However, no clear evidence is presented to validate the superiority of the new objective. I wonder if there are any special cases where the original objective fails to balance the trade-off, while the new one could. 

- Secondly, the authors could provide more compelling motivation for the notion of 'anytime' algorithms. As a new reader of the E2D literature, I don't get the advantage of an anytime algorithm over its original version.

QUESTIONS:
As is discussed in the Weakness part, I hope the authors could provide more explanations and examples about the advantage of the proposed new objective and the anytime variant algorithm.

LIMITATIONS:
I don't see any limitations or potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper focuses on regret minimization in sequential decision-making through min-max optimization. The authors introduce an anytime variant of the estimation-to-decisions algorithm that utilizes the average-constrained decision-estimation coefficient. The proposed algorithm is shown to effectively balance exploration and exploitation. The algorithm achieve a slightly improved regret performance in the context of high-dimensional linear bandits.

STRENGTHS:
The paper proposed anytime variant of the estimation-to-decisions algorithm and show the algorithm could potentially improve the linear bandits for a given fixed period.

WEAKNESSES:
1) A major concern is the contribution of the paper compared to the existing literature, particularly the work by Foster et al. in 2021 and 2023. The concept of average constrained DEC appears to be very similar to the (constrained) DEC introduced in Foster et al.'s papers. It is unclear what distinguishes the introduction of the average constrained DEC in this paper and what advantages it brings. 

2) The min-max problem in E2D algorithm is inefficient to solve in general, where the paper needs to find the dual variable via grid search and the choice of stepsize could be an issue.

3) While the paper states that the formulation of the average DEC leads to a practical algorithm, it would be more convincing to provide numerical results to support this claim. Comparisons with Foster et al.'s work from 2021 and 2023, as well as classical methods like UCB, Thompson sampling, and Information Direct Sampling, would justify the practicality and effectiveness of the proposed algorithm.

QUESTIONS:
Please see the weakness.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07