SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Johannes Kirschner
jkirschn@ualberta.ca
Kushagra Chandak
kchandak@ualberta.ca
Seyed Alireza Bakhtiari
sbakhtia@ualberta.ca
Volodymyr Tkachuk
vtkachuk@ualberta.ca
A long line of works characterizes the sample complexity of regret minimization in sequential decision-making by min-max programs. In the corresponding saddle-point game, the min-player optimizes the sampling distribution against an adversarial max-player that chooses confusing models leading to large regret. The most recent instantiation of this idea is the decision-estimation coefficient (DEC), which was shown to provide nearly tight lower and upper bounds on the worst-case expected regret in structured bandits and reinforcement learning. By reparametrizing the offset DEC with the confidence radius and solving the corresponding min-max program, we derive an anytime variant of the Estimation-To-Decisions algorithm (ANYTIME-E2D). Importantly, the algorithm optimizes the exploration-exploitation trade-off online instead of via the analysis. Our formulation leads to a practical algorithm for finite model classes and linear feedback models. We further point out connections to the information ratio, decoupling coefficient and PAC-DEC, and numerically evaluate the performance of E2D on simple examples.
Regret minimization is a widely studied objective in bandits and reinforcement learning theory [Lattimore and Szepesvári, 2020a] that has inspired practical algorithms, for example, in noisy zero-order optimization[e.g., Srinivas et al., 2010] and deep reinforcement learning [e.g., Osband et al., 2016]. Cumulative regret measures the online performance of the algorithm by the total loss suffered due to choosing suboptimal decisions. Regret is unavoidable to a certain extent as the learner collects information to reduce uncertainty about the environment. In other words, a learner will inevitably face the exploration-exploitation trade-off where it must balance collecting rewards and collecting information. Finding the right balance is the central challenge of sequential decision-making under uncertainty.
More formally, denote by  $\Pi$  a decision space and  $\mathcal{O}$  an observation space. Let  $\mathcal{M}$  be a class of models, where  $f = (r_f,M_f)\in \mathcal{M}$  associated with a reward function  $r_f:\Pi \to \mathbb{R}$  and observation
map  $M_f: \Pi \to \mathcal{P}(\mathcal{O})$ , where  $\mathcal{P}(\mathcal{O})$  is the set of all probability distributions over  $\mathcal{O}$ . The learner's objective is to collect as much reward as possible in  $n$  steps when facing a model  $f^* \in \mathcal{M}$ . The learner's prior information is  $\mathcal{M}$  and the associated reward and observation maps, but does not know the true instance  $f^* \in \mathcal{M}$ . The learner constructs a stochastic sequence  $\pi_1, \ldots, \pi_n$  of decisions taking values in  $\Pi$  and adapted to the history of observations  $y_t \sim M_{f^*}(\pi_t)$ . The policy of the learner is the sequence of probability kernels  $\mu_{1:n} = (\mu_t)_{t=1}^n$  that are used to take decisions. The expected regret of a policy  $\mu_{1:n}$  and model  $f^*$  after  $n \in \mathbb{N}$  steps is
$$
R _ {n} (\mu_ {1: n}, f ^ {*}) = \max _ {\pi \in \Pi} \mathbb {E} \left[ \sum_ {t = 1} ^ {n} r _ {f ^ {*}} (\pi) - r _ {f ^ {*}} (\pi_ {t}) \right]
$$
The literature studies regret minimization for various objectives, including worst-case and instance-dependent frequentist regret [Lattimore and Szepesvári, 2020a], Bayesian regret [Russo and Van Roy, 2014] and robust variants [Garcelon et al., 2020, Kirschner et al., 2020a]. For the frequentist analysis, all prior knowledge is encoded in the model class  $\mathcal{M}$ . The worst-case regret of policy  $\mu_{1:n}$  on  $\mathcal{M}$  is  $\sup_{f\in \mathcal{M}}R_n(\mu_{1:n},f)$ , and therefore the optimal minimax regret  $\inf_{\mu}\sup_{f\in \mathcal{M}}R_n(\mu_{1:n},f)$  only depends on  $\mathcal{M}$  and the horizon  $n$ . The Bayesian, in addition, assumes access to a prior  $\nu \in \mathcal{P}(\mathcal{M})$  which leads to the Bayesian regret  $\mathbb{E}_{f\sim \nu}[R_n(\mu_{1:n},f)]$ . Interestingly, the worst-case frequentist regret and Bayesian regret are dual in the following sense [Lattimore and Szepesvári, 2019]:
$$
\inf  _ {\mu_ {1: n}} \sup  _ {f \in \mathcal {M}} R _ {n} \left(\mu_ {1: n}, f\right) = \sup  _ {\nu \in \mathscr {P} (\mathcal {M})} \inf  _ {\mu_ {1: n}} \mathbb {E} _ {f \sim \nu} \left[ R _ {n} \left(\mu_ {1: n}, f\right) \right] \tag {1}
$$
Unfortunately, directly solving for the minimax policy (or the worst-case prior) is intractable, except in superficially simple problems. This is because the optimization is over the exponentially large space of adaptive policies. However, the relationship in Eq. (1) has been directly exploited in prior works, for example, to derive non-constructive upper bounds on the worst-case regret via a Bayesian analysis [Bubeck et al., 2015]. Moreover, it can be seen as inspiration underlying "optimization-based" algorithms for regret minimization: The crucial step is to carefully relax the saddle point problem in a way that preserves the statistical complexity, but can be analyzed and computed more easily. This idea manifests in several closely related algorithms, including information-directed sampling [Russo and Van Roy, 2014, Kirschner and Krause, 2018], ExpByOpt [Lattimore and Szepesvári, 2020b, Lattimore and Gyorgy, 2021], and most recently, the Estimation-To-Decisions (E2D) framework [Foster et al., 2021, 2023]. These algorithms have in common that they optimize the information trade-off directly, which in structured settings leads to large improvements compared to standard optimistic exploration approaches and Thompson sampling. On the other hand, algorithms that directly optimize the information trade-off can be computationally more demanding and, consequently, are often not the first choice of practitioners. This is partly due to the literature primarily focusing on statistical aspects, leaving computational and practical considerations underexplored.
Contributions Building on the results by Foster et al. [2021], we introduce the average-constrained decision-estimation coefficient  $(\mathrm{dec}_{\epsilon}^{ac})$ , a saddle-point objective that characterizes the frequentist worst-case regret in sequential decision-making with structured observations. Compared to the decision-estimation coefficient of [Foster et al., 2021], the  $\mathrm{dec}_{\epsilon}^{ac}$  is parametrized via the confidence radius  $\epsilon$ , instead of the Lagrangian offset multiplier. This allows optimization of the information trade-off online by the algorithm, instead of via the derived regret upper bound. Moreover, optimizing the  $\mathrm{dec}_{\epsilon}^{ac}$  leads to an anytime version of the E2D algorithm (ANYTIME-E2D) with a straightforward analysis. We also point out relations between the  $\mathrm{dec}_{\epsilon}^{ac}$ , the information ratio [Russo and Van Roy, 2016], the decoupling coefficient [Zhang, 2022] and a PAC version of the DEC [Foster et al., 2023]. We further detail how to implement the algorithm for finite model classes and linear feedback models, and demonstrate the advantage of the approach by providing improved bounds for linear bandits with sideobservations. Lastly, we report the first empirical results of the E2D algorithm on simple examples.
There is a broad literature on regret minimization in bandits [Lattimore and Szepesvári, 2020a] and reinforcement learning [Jin et al., 2018, Azar et al., 2017, Zhou et al., 2021, Du et al., 2021,
To simplify the presentation, we ignore tedious measure-theoretic details in this paper. The reader could either fill out the missing details, or just assume that all sets, unless otherwise stated, are discrete.
2The result by Lattimore and Szepesvári [2019] was only shown for finite action, reward and observation spaces, but can likely be extended to the infinite case under suitable continuity assumptions.
Zanette et al., 2020]. Arguably the most popular approaches are based on optimism, leading to the widely analysed upper confidence bound (UCB) algorithms [Lattimore and Szepesvári, 2020a], and Thompson sampling (TS) [Thompson, 1933, Russo and Van Roy, 2016].
A long line of work approaches regret minimization as a saddle point problem. Degenne et al. [2020b] showed that in the structured bandit setting, an algorithm based on solving a saddle point equation achieves asymptotically optimal regret bounds, while explicitly controlling the finite-order terms. Lattimore and Szepesváři [2020b] propose an algorithm based on exponential weights in the partial monitoring setting [Rustichini, 1999] that finds a distribution for exploration by solving a saddle-point problem. The saddle-point problem balances the trade-off between the exponential weights distribution and an information or stability term. The same approach was further refined by Lattimore and Gyorgy [2021]. In stochastic linear bandits, Kirschner et al. [2021] demonstrated that information-directed sampling can be understood as a primal-dual method solving the asymptotic lower bound, which leads to an algorithm that is both worst-case and asymptotically optimal. The saddle-point approach has been further explored in the PAC setting [e.g., Degenne et al., 2020b,a].
Our work is closely related to recent work by Foster et al. [2021, 2023]. They consider decision making with structured observations (DMSO), which generalizes the bandit and RL setting. They introduce a complexity measure, the offset decision-estimation coefficient (offset DEC), defined as a min-max game between a learner and an environment, and provide lower bounds in terms of the offset DEC. Further, they provide an algorithm, Estimation-to-Decisions (E2D) with corresponding worst-case upper bounds in terms of the offset DEC. Notably, the lower and upper bound nearly match and recover many known results in bandits and RL. More recently, Foster et al. [2023] refined the previous bounds by introducing the constrained DEC and a corresponding algorithm  $\mathrm{E2D^{+}}$ .
There are various other results related to the DEC and the E2D algorithm. Foster et al. [2022a] show that the E2D achieves improved bounds in model-free RL when combined with optimistic estimation (as introduced by Zhang [2022]). Chen et al. [2022] introduced two new complexity measures based on the DEC that are necessary and sufficient for reward-free learning and PAC learning. They also introduced new algorithms based on the E2D algorithm for the above two settings and various other improvements. Foster et al. [2022b] have shown that the DEC is necessary and sufficient to obtain low regret for adversarial decision-making. An asymptotically instance-optimal algorithm for DMSO has been proposed by Dong and Ma [2022], extending a similar approach for the linear bandit setting [Lattimore and Szepesvari, 2017].
The decision-estimation coefficient is also related to the information ratio [Russo and Van Roy, 2014] and the decoupling coefficient [Zhang, 2022]. The information ratio has been studied under both the Bayesian [Russo and Van Roy, 2014] and the frequentist regret [Kirschner and Krause, 2018, Kirschner et al., 2020b, 2021, 2023] in various settings including bandits, reinforcement learning, and partial monitoring. The decoupling coefficient was studied for the Thompson sampling algorithm in contextual bandits [Zhang, 2022], and RL [Dann et al., 2021, Agarwal and Zhang, 2022].
We consider the sequential decision-making problem already introduced in the preface. Recall that  $\Pi$  is a compact decision space and  $\mathcal{O}$  is an observation space. The model class  $\mathcal{M}$  is a set of tuples  $f = (r_f, M_f)$  containing a reward function  $r_f: \Pi \to \mathbb{R}$  and an observation distribution  $M_f: \Pi \to \mathcal{P}(\mathcal{O})$ . We define the gap function
$$
\Delta (\pi , g) = r _ {g} (\pi_ {g} ^ {*}) - r _ {g} (\pi),
$$
where  $\pi_g^* = \arg \max_{\pi \in \Pi} r_g(\pi)$  is an optimal decision for model  $g$ , chosen arbitrarily if not unique. A randomized policy is a sequence of kernels  $\mu_{1:n} = (\mu_t)_{t=1}^n$  from histories  $h_{t-1} = (\pi_1, y_1, \ldots, \pi_{t-1}, y_{t-1}) \in (\Pi \times \mathcal{O})^{t-1}$  to sampling distributions  $\mathcal{P}(\Pi)$ . The filtration generated by the history  $h_t$  is  $\mathcal{F}_t$ . The learner's decisions  $\pi_1, \ldots, \pi_n$  are sampled from the policy  $\pi_t \sim \mu_t$  and observations  $y_t \sim M_{f^*}(\pi_t)$  are generated by an unknown true model  $f^* \in \mathcal{M}$ . The expected regret under model  $f^*$  is formally defined as follows:
$$
R _ {n} (\mu_ {1: n}, f ^ {*}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {n} \mathbb {E} _ {\pi_ {t} \sim \mu_ {t} (h _ {t})} [ \Delta (\pi_ {t}, f ^ {*}) ] \right]
$$
For now, we do not make any assumption about the reward being observed. This provides additional flexibility to model a wide range of scenarios, including for example, duelling and ranking feedback [Yue and Joachims, 2009, Radlinski et al., 2008, Combes et al., 2015, Lattimore et al., 2018, Kirschner and Krause, 2021] (e.g. used in reinforcement learning with human feedback, RLHF) or dynamic pricing [den Boer, 2015]. The setting is more widely known as partial monitoring Rustichini [1999]. The special case where the reward is part of the observation distribution is called decision-making with structured observations [DMSO, Foster et al., 2021]. Earlier work studies the closely related structured bandit setting [Combes et al., 2017].
A variety of examples across bandit models and reinforcement learning are discussed in [Combes et al., 2017, Foster et al., 2021, 2023, Kirschner et al., 2023]. For the purpose of this paper, we focus on simple cases for which we can provide tractable implementations. Besides the finite setting where  $\mathcal{M}$  can be enumerated, these are the following linearly parametrized feedback models.
Example 2.1 (Linear Bandits, Abe and Long [1999]). The model class is identified with a subset of  $\mathbb{R}^d$  and features  $\phi_{\pi}\in \mathbb{R}^{d}$  for each  $\pi \in \Pi$ . The reward function is  $r_f(\pi) = \langle \phi (\pi),f\rangle$  and the observation distribution is  $M_{f}(\pi) = \mathcal{N}(\langle \phi_{\pi},f\rangle ,1)$ .
The linear bandit setting can be generalized by separating reward and feedback maps, which leads to the linear partial monitoring framework [Lin et al., 2014, Kirschner et al., 2020b]. Here we restrict our attention to the special case of linear bandits with side-observations [c.f. Kirschner et al., 2023], which, for example, generalizes the classical semi-bandit setting Mannor and Shamir [2011]
Example 2.2 (Linear Bandits with Side-Observations). As in the linear bandit setting, we have  $\mathcal{M} \subset \mathbb{R}^d$ , and features  $\phi_{\pi} \in \mathbb{R}^d$  that define the reward functions  $r_f(\pi) = \langle \phi_{\pi}, f \rangle$ . Observation matrices  $M_{\pi} \in \mathbb{R}^{m_{\pi} \times d}$  for each  $\pi \in \Pi$  define  $m_{\pi}$ -dimensional observation distributions  $M_f(\pi) = \mathcal{N}(M_{\pi}f, \sigma^2 \mathbf{1}_{m_{\pi}})$ . In addition, we assume that  $\phi_{\pi} \phi_{\pi}^\top \preceq M_{\pi}^\top M_{\pi}$ , which is automatically satisfied if  $\phi_{\pi}^\top$  is included in the rows of  $M_{\pi}$ , i.e. when the reward is part of the observations.
3 Regret Minimization via Saddle-Point Optimization
The goal of the learner is to choose decisions  $\pi \in \Pi$  that achieve a small gap  $\Delta (\pi ,f^{*})$  under the true model  $f^{*}\in \mathcal{M}$ . Since the true model is unknown, the learner has to collect data that provides statistical evidence to reject models  $g\neq f^{*}$  for which the regret  $\Delta (\pi ,g)$  is large. To quantify the information-regret trade-off, we use a divergence  $D(\cdot \| \cdot)$  defined for distributions in  $\mathcal{P}(\mathcal{O})$ . For a reference model  $f$ , the information (or divergence) function is defined by:
$$
I _ {f} (\pi , g) = D _ {\mathrm {K L}} (M _ {g} (\pi) \| M _ {f} (\pi)),
$$
where  $D_{\mathrm{KL}}(\cdot \| \cdot)$  is the KL divergence. Intuitively,  $I_{f}(\pi ,g)$  is the rate at which the learner collects statistical information to reject  $g\in \mathcal{M}$  when choosing  $\pi \in \Pi$  and data is generated under the reference model  $f$ . Note that  $I_{f}(\pi ,f) = 0$  for all  $f\in \mathcal{M}$  and  $\pi \in \Pi$ . As we will see shortly, the regret-information trade-off can be written precisely as a combination of the gap function,  $\Delta$ , and the information function,  $I_{f}$ . We remark in passing that other choices such as the Hellinger distance are also possible, and the KL divergence is mostly for concreteness and practical reasons.
To simplify the notation and emphasize the bilinear nature of the saddle point problem that we study, we will view  $\Delta, I_f \in \mathbb{R}_+^{\Pi \times \mathcal{M}}$  as  $|\Pi| \times |\mathcal{M}|$  matrices (by fixing a canonical ordering on  $\Pi$  and  $\mathcal{M}$ ). For vectors  $\mu \in \mathbb{R}^{\Pi}$  and  $\nu \in \mathbb{R}^{\mathcal{M}}$ , we will frequently write bilinear forms  $\mu \Delta_f \nu$  and  $\mu I_f \nu$ . This also means that by convention,  $\mu$  will always denote a row vector, while  $\nu$  will always denote a column vector. The standard basis for  $\mathbb{R}^{\Pi}$  and  $\mathbb{R}^{\mathcal{M}}$  is  $(e_{\pi})_{\pi \in \Pi}$  and  $(e_g)_{g \in \mathcal{M}}$ .
3.1 The Decision-Estimation Coefficient
To motivate our approach, we recall the decision-estimation coefficient (DEC) introduced by Foster et al. [2021, 2023], before introducing the main quantity of interest, the average-constrained DEC. First, the offset decision-estimation coefficient (without localization) [Foster et al., 2021] is
$$
\operatorname{dec}_{\lambda}^{o}(f) = \min_{\mu \in \mathscr{P}(\Pi)}\max_{g\in \mathcal{M}}\mu \Delta e_{g} - \lambda \mu I_{f}e_{g}
$$
The tuning parameter  $\lambda > 0$  controls the weight of the information matrix relative to the gaps: Viewing the above as a two-player zero-sum game, we see that increasing  $\lambda$  forces the max-player
to avoid models that differ significantly from  $f$  under the min-player's sampling distribution. The advantage of this formulation is that the information term  $\mu I_{f}e_{g}$  can be telescoped in the analysis, which directly leads to regret bounds in terms of the estimation error (introduced below in Eq. (8)). The disadvantage of the  $\lambda$ -parametrization is that the trade-off parameter is chosen by optimizing the final regret upper bound. This is inconvenient because the optimal choice requires knowledge of the horizon and a bound on  $\max_{f\in \mathcal{M}}\mathrm{dec}_{\lambda}^{\circ}(f)$ . Moreover, any choice informed by the upper bound may be conservative, leading to sub-optimal performance.
The constrained decision-estimation coefficient [Foster et al., 2023] is
$$
\operatorname {d e c} _ {\epsilon} ^ {c} (f) = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {g \in \mathcal {M}} \mu \Delta e _ {g} \quad \text {s . t .} \quad \mu I _ {f} e _ {g} \leq \epsilon^ {2} \tag {2}
$$
In this formulation, the max player is restricted to choose models  $g$  that differ from  $f$  at most by  $\epsilon^2$  in terms of the observed divergence under the min-player's sampling distribution. Note that because  $e_{\pi}I_{f}e_{f} = 0$  for all  $e_{\pi}\in \Pi$ , there always exists a feasible solution. For horizon  $n$ , the radius can be set to  $\epsilon^2\approx \frac{\beta_{\mathcal{M}}}{n}$ , where  $\beta_{\mathcal{M}}$  is a model estimation complexity parameter, thereby essentially eliminating the trade-off parameter from the algorithm. However, because of the hard constraint, strong duality of the Lagrangian saddle point problem (for fixed  $\mu$ ) fails, and consequently, telescoping the information gain in the analysis is no longer easily possible (or at least, with the existing analysis). To achieve sample complexity  $\mathrm{dec}_{\epsilon}^{c}(f)$ , Foster et al. [2023] propose a sophisticated scheme that combines phased exploration with a refinement procedure  $(\mathrm{E2D}^{+})$ .
As the main quantity of interest in the current work, we now introduce the average-constrained decision-estimation coefficient, defined as follows:
$$
\sec_ {\epsilon} ^ {a c} (f) = \min  _ {\mu \in \mathscr {P} (\Pi)} \max  _ {\nu \in \mathscr {P} (\mathcal {M})} \mu \Delta \nu \quad \text {s . t .} \quad \mu I _ {f} \nu \leq \epsilon^ {2} \tag {3}
$$
Similar to the  $\mathrm{dec}_{\epsilon}^{c}$ , the parameterization of the  $\mathrm{dec}_{\epsilon}^{ac}$  is via the confidence radius  $\epsilon^2$ , making the choice of the hyperparameter straightforward in many cases (more details in Section 3.2). By convexifying the domain  $\mathcal{P}(\mathcal{M})$  of the max-player, we recover strong duality of the Lagrangian (for fixed  $\mu$ ). Thereby, the formulation inherits the ease of choosing the  $\epsilon$ -parameter from the  $\mathrm{dec}_{\epsilon}^{c}$ , while, at the same time, admitting a telescoping argument in the analysis and a much simpler algorithm.
Specifically, Sion's theorem implies three equivalent Lagrangian representations for Eq. (3):
$$
\begin{array}{l} \operatorname {d e c} _ {\epsilon} ^ {a c} (f) = \min  _ {\mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \min  _ {\lambda \geq 0} \mu \Delta \nu - \lambda \left(\mu I _ {f} \nu - \epsilon^ {2}\right) (4) \\ = \min  _ {\lambda \geq 0, \mu \in \mathcal {P} (\Pi)} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \mu \Delta \nu - \lambda \left(\mu I _ {f} \nu - \epsilon^ {2}\right) (5) \\ = \min  _ {\lambda \geq 0} \max  _ {\nu \in \mathcal {P} (\mathcal {M})} \min  _ {\mu \in \mathcal {P} (\Pi)} \mu \Delta \nu - \lambda \left(\mu I _ {f} \nu - \epsilon^ {2}\right) (6) \\ \end{array}
$$
When fixing the outer problem, strong duality holds for the inner saddle-point problem in each line, however, the joint program in Eq. (5) is not convex-concave. An immediate consequence of relaxing the domain of the max player and Eq. (5) is that
$$
\operatorname {d e c} _ {\epsilon} ^ {c} (f) \leq \operatorname {d e c} _ {\epsilon} ^ {a c} (f) = \min  _ {\lambda \geq 0} \left\{\operatorname {d e c} _ {\lambda} ^ {o} (f) + \lambda \epsilon^ {2} \right\} \tag {7}
$$
The  $\mathrm{dec}_{\epsilon}^{ac}$  can therefore be understood as setting the  $\lambda$  parameter of the  $\mathrm{dec}_{\lambda}^{o}$  optimally for the given confidence radius  $\epsilon^2$ . On the other hand, the cost paid for relaxing the program is that there exist model classes  $\mathcal{M}$  where the inequality in Eq. (7) is strict, and  $\mathrm{dec}_{\epsilon}^{ac}$  does not lead to a tight characterization of the regret [Foster et al., 2023, Proposition 4.4]. The remedy is that under a stronger regularity condition and localization, the two notions are essentially equivalent [Foster et al., 2023, Proposition 4.8].
3.2 Anytime Estimation-To-Decisions (Anytime-E2D)
Estimations-To-Decisions (E2D) is an algorithmic framework that directly leverages the decision-estimation coefficient for choosing a decision in each round. The key idea is to compute a sampling distribution  $\mu_t \in \mathcal{P}(\Pi)$  attaining the minimal DEC for an estimate  $\hat{f}_t$  of the underlying model, and then define the policy to sample  $\pi_t \sim \mu_t$ . The E2D approach, using the  $\mathrm{dec}_{\epsilon}^{ac}$  formulation, is summarized in Algorithm 1. To compute the estimate  $\hat{f}_t$ , the E2D algorithm takes an abstract estimation oracle EST as input, that, given the collected data, returns  $\hat{f}_t \in \mathcal{M}$ . The final guarantee
Algorithm 1: ANYTIME-E2D