SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evgenii Chzhen Christophe Giraud
Université Paris-Saclay, CNRS, Laboratoire de mathématiques d'Orsay, 91405, Orsay, France {evgenii.chzhen, christophe.giraud}@universite-paris-saclay.fr
Zhen Li
BNP Paribas Corporate and Institutional Banking, 20 boulevard des Italiens, 75009 Paris, France zhen.li@bnpparibas.com
Gilles Stoltz
Université Paris-Saclay, CNRS, Laboratoire de mathématiques d'Orsay, 91405, Orsay, France  
HEC Paris, 78351 Jouy-en-Josas, France
gilles.stoltz@universite-paris-saclay.fr, stoltz@hec.fr
We consider contextual bandit problems with knapsacks [CBwK], a problem where at each round, a scalar reward is obtained and vector-valued costs are suffered. The learner aims to maximize the cumulative rewards while ensuring that the cumulative costs are lower than some predetermined cost constraints. We assume that contexts come from a continuous set, that costs can be signed, and that the expected reward and cost functions, while unknown, may be uniformly estimated—a typical assumption in the literature. In this setting, total cost constraints had so far to be at least of order  $T^{3/4}$ , where  $T$  is the number of rounds, and were even typically assumed to depend linearly on  $T$ . We are however motivated to use CBwK to impose a fairness constraint of equalized average costs between groups: the budget associated with the corresponding cost constraints should be as close as possible to the natural deviations, of order  $\sqrt{T}$ . To that end, we introduce a dual strategy based on projected-gradient-descent updates, that is able to deal with total-cost constraints of the order of  $\sqrt{T}$  up to poly-logarithmic terms. This strategy is more direct and simpler than existing strategies in the literature. It relies on a careful, adaptive, tuning of the step size.
1 Setting, literature review, and main contributions
We consider contextual bandits with knapsacks [CBwK], a setting where at each round  $t \geqslant 1$ , the learner, after observing some context  $\pmb{x}_t \in \mathcal{X}$ , where  $\mathcal{X} \subseteq \mathbb{R}^n$ , picks an action  $a_t \in \mathcal{A}$  in a finite set  $\mathcal{A}$ . We do not impose the existence of a null-cost action. Contexts are independently drawn according to a distribution  $\nu$ . The learner may pick  $a_t$  at random according to a probability distribution, denoted by  $\pi_t(\pmb{x}_t) = (\pi_{t,a}(\pmb{x}_t))_{a \in \mathcal{A}}$  for consistency with the notion of policy defined later in Section 2. The action  $a_t$  played leads to some scalar reward  $r_t \in [0,1]$  and some signed vector-valued cost  $c_t \in [-1,1]^d$ . Actually,  $r_t$  and  $c_t$  are generated independently at random in a way such that the conditional expectations of  $r_t$  and  $c_t$  given the past,  $\pmb{x}_t$ , and  $a_t$ , equal  $r(\pmb{x}_t, a_t)$  and  $c(\pmb{x}_t, a_t)$ , respectively. We denoted here by  $r: \mathcal{X} \times \mathcal{A} \to [0,1]$  and  $c = (c_1, \dots, c_d): \mathcal{X} \times \mathcal{A} \to [-1,1]^d$  the unknown expected-reward and expected-cost functions. The modeling and the estimation of  $r$
BOX A: CONTEXTUAL BANDITS WITH KNAPSACKS [CBWK]
Known parameters: finite action set  $\mathcal{A}$ ; context set  $\mathcal{X} \subseteq \mathbb{R}^n$ ; number  $T$  of rounds; vector of average cost constraints  $\pmb{B} \in [0,1]^d$
Unknown parameters: context distribution  $\nu$  on  $\mathcal{X}$ ; scalar expected-reward function  $r: \mathcal{X} \times \mathcal{A} \to [0,1]$  and vector-valued expected-cost function  $c: \mathcal{X} \times \mathcal{A} \to [-1,1]^d$ , both enjoying some modeling allowing for estimation, see Section 2.2
For rounds  $t = 1,2,3,\ldots ,T$
1. Context  $\pmb{x}_t \sim \nu$  is drawn independently of the past;
2. Learner observes  $\pmb{x}_t$  and picks an action  $a_t \in \mathcal{A}$ , possibly at random according to a distribution  $\pi_t(\pmb{x}_t) = (\pi_{t,a}(\pmb{x}_t))_{a \in \mathcal{A}}$ ;
3. Learner gets a reward  $r_t \in [0,1]$  with conditional expectation  $r(\pmb{x}_t, a_t)$  and suffers constraints  $c_t$  with conditional expectations  $c(\pmb{x}_t, a_t)$ .
Goal: Maximize  $\sum_{t\leqslant T}r_t$  while ensuring  $\sum_{t\leqslant T}c_t\leqslant TB$
and  $c$  are discussed in Section 2.2. The aim of the learner is to maximize the sum of rewards (or equivalently, to minimize the regret defined in Section 2) while controlling cumulative costs. More precisely, we denote by  $B = (B_{1},\ldots ,B_{d})\in [0,1]^{d}$  the normalized (i.e., average per-instance) cost constraints, which may depend on the coordinates. The number  $T$  of rounds is known, and the learner must play in a way such that  $c_{1} + \dots +c_{T}\leqslant TB$ . The example of Section 2.1 illustrates how this setting allows for controlling costs in absolute values. The setting described is summarized in Box A.
Overview of the literature review. Contextual bandits with knapsacks [CBwK] is a setting that is a combination of the problems of contextual bandits (where only rewards are obtained, and no costs: see, among others, Lattimore and Szepesvári, 2020, Chapter 18 for a survey of this rich area) and of bandits with knapsacks (without contexts, as initially introduced by Badanidiyuru et al., 2013, 2018). The first approaches to CBwK (Badanidiyuru et al., 2014, Agrawal et al., 2016) relied on no specific modeling of the rewards and costs, and made the problem tractable by using as a benchmark a finite set of so-called static policies (but picking this set is uneasy, as noted by Agrawal and Devanur, 2016). Virtually all subsequent approaches to CBwK thus introduced structural assumptions in one way or the other. The simplest modelings are linear dependencies of expected rewards and costs on the contexts (Agrawal and Devanur, 2016) and a logistic conversion model (Li and Stoltz, 2022). The problem of CBwK attracted attention in recent past: we discuss and contrast the contributions by Chohlas-Wood et al. [2021], Slivkins et al. [2022], Han et al. [2022], Ai et al. [2022], Li and Stoltz [2022] after stating our main contributions.
Overview of our main contributions. Each contribution calls or allows for the next one.
1. This article revisits the CBwK approach by Chohlas-Wood et al. [2021] to a problem of fair spending of resources between groups while maximizing some total utility. Fairness constraints require to deal with signed costs and with possibly small total-cost constraints (typically close to  $T^{1/2}$ ), while having no null-cost action at disposal.
2. We therefore provide a new CBwK strategy dealing with these three issues, the most significant being handling cost constraint  $TB$  as small as  $T^{1/2 + \varepsilon}$ , breaking the state-of-the-art  $T^{3/4}$  barrier for CBwK with continuous contexts, while preserving attractive computational efficiency.
3. This new strategy is a direct, simple, explicit, dual strategy, relying on projected-gradient-descent updates (instead of using existing general algorithms as subroutines). We may perform an ad hoc analysis. The latter leads to refined regret bounds, in terms of the norms of some optimal dual variables. We also discuss the optimality of the results achieved by offering a proof scheme for problem-dependent (not only minimax) regret lower bounds.
1.1 Detailed literature review
We now contrast our main contributions to the existing literature.
CBwK and fairness: missing tools. In the setting of Chohlas-Wood et al. [2021], there is a budget constraint on the total spending on top of the wish of sharing out these spendings among groups (possibly favoring to some maximal extent some groups that are more in need). However,
Chohlas-Wood et al. [2021] incorporated the fairness constraints in the reward function (through some Lagrangian penalty) instead of doing so in the vector-cost function, which does not seem the most natural way to proceed. One reason is that (see a discussion below) the total cost constraints  $TB$  were so far typically assumed to be linear, which is undesirable in fairness applications, where one wishes that  $TB$  is sublinear—and actually, is sometimes as close as possible to the natural  $\sqrt{T}$  deviations suffered even in a perfectly fair random allocation scheme. Also, there is no null-cost action and costs are signed—two twists to the typical settings of CBwK that only Slivkins et al. [2022] and our approach deal with, to the best of our knowledge. We describe our alternative modeling of this fairness problem in Section 2.1.
Orders of magnitude for total-cost constraints  $TB$ . The literature so far mostly considered linear total-cost constraints  $TB$  (see, e.g., Han et al., 2022 or Ai et al., 2022 among recent references), with two series of exceptions: (i) the primal strategy by Li and Stoltz [2022] handling total-cost constraints of the order of  $\sqrt{T}$  up to poly-logarithmic terms but critically assuming the finiteness of the context set  $\mathcal{X}$ ; and (ii) the two-stage dual strategies of Agrawal and Devanur [2016], Han et al. [2022] handling  $T^{3/4}$  total-cost constraints. These two-stage strategies use  $\sqrt{T}$  preliminary rounds to learn some key hyperparameter  $Z$  and then transform strategies that work with linear total-cost constraints into strategies that may handle total-cost constraints larger than  $T^{3/4}$ ; see discussions after Lemma 2. In contrast, we provide a general theory of small cost constraints for continuous context sets, while obtaining similar regret bounds as the literature does: for some components  $j$ , we may have  $TB_{j} \ll T^{3/4}$  (actually, any rate  $TB_{j} \gg T^{1/2 + \varepsilon}$  is suitable for a sublinear regret), while for other components  $k$ , the total-cost constraints  $TB_{k}$  may be linearly large.
Typical CBwK strategies: primal approaches suffer from severe limitations. CBwK is typically solved through dual approaches, as the latter basically rely on learning a finite-dimensional parameter given by the optimal dual variables  $\lambda^{\star}$  (see the beginning of Section 3), while primal approaches rely on learning the distribution  $\nu$  of the contexts in  $\mathbb{L}^{1}$ -distance. This may be achieved in some cases, like finiteness of the context set  $\mathcal{X}$  (Li and Stoltz, 2022, Ai et al., 2022) or, at the cost of degraded regret bounds and under additional assumptions on  $\nu$  when resorting to  $\mathbb{L}^{1}$ -density-estimation techniques (Ai et al., 2022). On top of these strong limitations, such primal approaches are also typically computationally inefficient as they require the computation of expectations over estimates of  $\nu$ .
Typical CBwK strategies: dual approaches are modular and somewhat indirect. Typical dual approaches in CBwK take two general forms. One, illustrated in Slivkins et al. [2022] (extending the non-contextual approach by Immorlica et al., 2019), takes a game-theoretic approach by identifying the primal-dual formulation (4) as some minimax equilibrium, which may be learned by separate regret minimization of a learner picking actions  $a_{t}$  and an opponent picking dual variables  $\lambda_{t}$ . The second approach (Agrawal and Devanur, 2016, Han et al., 2022) learns more directly the  $\lambda_{t}$  via some online convex optimization algorithm fed with the suffered costs  $c_{t}$ ; this second approach however requires, at the stage of picking  $a_{t}$ , a suitable bound  $Z$  on the norms of the  $\lambda_{t}$ .
The dual strategies discussed above are modular and indirect as they all consist of using as building blocks some general-purpose strategies. We rather propose a more direct dual approach, tailored to our needs. (We also believe that it is simpler and more elegant.) Our strategy picks the arm  $a_{t}$  that maximizes the Lagrangian penalization of rewards by costs through the current  $\lambda_{t-1}$  (see Step 2 in Box B), as also proposed by Agrawal and Devanur [2016] (while Han et al., 2022 add some randomization to this step), and then, performs some direct, explicit, projected-gradient-descent update on  $\lambda_{t-1}$  to obtain  $\lambda_{t}$ , with step size  $\gamma$ . We carefully and explicitly tune  $\gamma$  (in a sequential fashion, via regimes) to achieve our goals. Such fine-tuning is more difficult to perform with approaches relying on the black-box use of subroutines given by existing general-purpose strategies.
Regret (upper and lower) bounds typically proposed in the CBwK literature. A final advantage of our more explicit strategy is that we master each piece of its analysis: while not needing a null-cost action, we provide refined regret bounds that go beyond the typical  $(\mathrm{OPT}(r,c,B) / \min B)\sqrt{T}$  bound offered by the literature so far (see, among others, Agrawal and Devanur, 2016, Li and Stoltz, 2022, Han et al., 2022), where  $\mathrm{OPT}(r,c,B)$  denotes the expected reward achieved by the optimal static policy (see Section 2 for definitions). Namely, our bounds are of the order of  $\| \pmb{\lambda}^{\star}\| \sqrt{T}$ , where  $\pmb{\lambda}^{\star}$  is the optimal dual variable; we relate the norm of the latter to quantities of the form  $\big(\mathrm{OPT}(r,c,B) - \mathrm{OPT}(r,c,\alpha B)\big) / \big((1 - \alpha)\min B\big)$ , for some  $\alpha < 1$ . Again, we may do so without the existence of a null-cost action, but when one such action is available, we may take  $\alpha = 0$  in the interpretation of the bound. We also offer a proof scheme for lower bounds in Section 4 and
explain why  $\| \pmb{\lambda}^{\star}\| \sqrt{T}$  appears as the correct target. We compare therein our problem-dependent lower bound-approach to the minimax ones of Han et al. [2022] and Slivkins et al. [2022].
1.2 Outline
In Section 2, we further describe the problem of CBwK (by defining the regret and recalling how  $r$  and  $c$  may be estimated) and state our motivating example of fairness. Section 3 is devoted to our new dual strategy, which we analyze first with a fixed step size  $\gamma$ , for which we move next to an adaptive version, and whose bounds we finally discuss. Section 4 offers a proof scheme for regret lower bounds and lists some limitations of our approach, mostly relative to optimality.
2 Further description of the setting, and statement of our motivating example
We define a static policy as a mapping  $\pi : \mathcal{X} \to \mathcal{P}(\mathcal{A})$ , where  $\mathcal{P}(\mathcal{A})$  denotes the set of probability distributions over  $\mathcal{A}$ . We denote by  $\pi = (\pi_{a})_{a \in \mathcal{A}}$  the components of  $\pi$ . We let  $\mathbb{E}_{\boldsymbol{X} \sim \nu}$  indicate that the expectation is taken over the random variable  $\boldsymbol{X}$  with distribution  $\nu$ . In the sequel, the inequalities  $\leqslant$  or  $\geqslant$  between vectors will mean pointwise satisfaction of the corresponding inequalities.
Assumption 1. The contextual bandit problem with knapsacks  $(r, c, B', \nu)$  is feasible if there exists a stationary policy  $\pi$  such that
$$
\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right] \leqslant \boldsymbol {B} ^ {\prime}.
$$
We denote the average cost constraints by  $B'$  in the assumption above as in Section 3, we will actually require feasibility for average cost constraints  $B' < B$ , not just for  $B$ .
In a feasible problem  $(r, c, B', \nu)$ , the optimal static policy  $\pi^{\star}$  is defined as the policy  $\pi: \mathcal{X} \to \mathcal{P}(\mathcal{A})$  achieving
$$
\operatorname {O P T} (r, \boldsymbol {c}, \boldsymbol {B} ^ {\prime}) = \sup  _ {\boldsymbol {\pi}} \left\{\mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} r (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right]: \mathbb {E} _ {\boldsymbol {X} \sim \nu} \left[ \sum_ {a \in \mathcal {A}} \boldsymbol {c} (\boldsymbol {X}, a) \pi_ {a} (\boldsymbol {X}) \right] \leqslant \boldsymbol {B} ^ {\prime} \right\}. \tag {1}
$$
Maximizing the sum of the  $r_t$  amounts to minimizing the regret  $R_T = T \mathrm{OPT}(r, \mathbf{c}, \mathbf{B}') - \sum_{t=1}^{T} r_t$ .
2.1 Motivating example: fairness—equalized average costs between groups
This example is inspired from Chohlas-Wood et al. [2021] (who did not provide a strategy to minimize regret), and features a total budget constraint  $TB_{\mathrm{total}}$  together with fairness constraints on how the budget is spent among finitely many subgroups, whose set is denoted by  $\mathcal{G}$ . We assume that the contexts  $\boldsymbol{x}_t$  include the group index  $\mathrm{gr}(\boldsymbol{x}_t) \in \mathcal{G}$ , which means, in our setting, that the learner accesses this possibly sensitive attribute before making the predictions (the so-called awareness framework in the fairness literature). Each context-action pair  $(\boldsymbol{x}, a) \in \mathcal{X} \times \mathcal{A}$ , on top of leading to some expected reward  $r(\boldsymbol{x}, a)$ , also corresponds to some spending  $c_{\mathrm{spd}}(\boldsymbol{x}, a)$ . We recall that we denoted that  $r_t$  and  $c_t$  (here, this is a scalar for now) the individual payoffs and costs obtained at round  $t$ ; they have conditional expectations  $r(\boldsymbol{x}_t, a_t)$  and  $c_{\mathrm{spd}}(\boldsymbol{x}_t, a_t)$ . We want to trade off the differences in average spending among groups with the total reward achieved: larger differences lead to larger total rewards but generate feelings of inequity or of public money not optimally used. (Chohlas-Wood et al. [2021] consider a situation where unfavored groups should get slightly more spending than favored groups.)
Formally, we denote by  $\operatorname{gr}(\pmb{x}) \in \mathcal{G}$  the group to which a given context  $\pmb{x}$  belongs and introduce a tolerance factor  $\tau$ , which may depend on  $T$ . We issue a simplifying assumption: while the distribution  $\nu$  of the contexts is complex and is unknown, the respective proportions  $\gamma_g = \nu \{\pmb{x} : \operatorname{gr}(\pmb{x}) = g\}$  of the sub-groups may be known. The total-budget and fairness constraints then read:
$$
\sum_ {t = 1} ^ {T} c _ {t} \leqslant T B _ {\text {t o t a l}} \quad \text {a n d} \quad \forall g \in \mathcal {G}, \quad \left| \frac {1}{T \gamma_ {g}} \sum_ {t = 1} ^ {T} c _ {t} \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x} _ {t}) = g \}} - \frac {1}{T} \sum_ {t = 1} ^ {T} c _ {t} \right| \leqslant \tau ,
$$
This simplification is not unheard of in the fairness literature; see, for instance, Chzhen et al. [2021]. It amounts to having a reasonable knowledge of the breakdown of the population into subgroups. We use this assumption to have an easy rewriting of the fairness constraints in the setting of Box A. The knowledge of the  $\gamma_{g}$  is key for determining the average-constraint vector  $B$ .
which corresponds, in the setting of Box A, to the following vector-valued expected-cost function  $\pmb{c}$ , with  $d = 1 + 2|\mathcal{G}|$  components and with values in  $[-1,1]^{1 + 2|\mathcal{G}|}$ :
$$
\left. \right.\left(c _ {\operatorname {s p d}} (\boldsymbol {x}, a), \left(c _ {\operatorname {s p d}} (\boldsymbol {x}, a) \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = g \}} - \gamma_ {g} c _ {\operatorname {s p d}} (\boldsymbol {x}, a), \gamma_ {g} c _ {\operatorname {s p d}} (\boldsymbol {x}, a) - c _ {\operatorname {s p d}} (\boldsymbol {x}, a) \mathbb {1} _ {\{\operatorname {g r} (\boldsymbol {x}) = g \}}\right) _ {g \in \mathcal {G}}\right)
$$
as well as to the average-constraint vector  $\pmb{B} = \left(B_{\mathrm{total}}, \left(\gamma_g \tau, \gamma_g \tau\right)_{g \in \mathcal{G}}\right)$ .
The regimes we have in mind, and that correspond to the public-policy issues reported by Chohlas-Wood et al. [2021], are that the per-instance budget  $B_{\mathrm{total}}$  is larger than a positive constant, i.e., a constant fraction of the  $T$  individuals may benefit from some costly action(s), while the fairness threshold  $\tau$  must be small, and even, as small as possible. Because of central-limit-theorem fluctuations, a minimal value of the order of  $1 / \sqrt{T}$ , or slightly larger (up to logarithmic terms, say), has to be considered for  $\tau$ . The salient point is that the components of  $B$  may therefore be of different orders of magnitude, some of them being as small as  $1 / \sqrt{T}$ , up to logarithmic terms. More details on this example and numerical simulations about it are provided in Section 5 and Appendix G.
2.2 Modelings for, and estimation of, expected-reward and expected-cost functions
As is common in the literature (see Han et al., 2022 and Slivkins et al., 2022, who use the terminology of regression oracles), we sequentially estimate the functions  $r$  and  $c$  and assume that we may do so in some uniform way, e.g., because of the underlying structure assumed. Note that the estimation procedure of Assumption 2 below does not force any specific choice of actions, it is able to exploit actions  $a_{t}$  picked by the main strategy (this is what the "otherwise" is related to therein). We denote by  $\widehat{r}_t: \mathcal{X} \times \mathcal{A} \to [0,1]$  and  $\widehat{c}_t: \mathcal{X} \times \mathcal{A} \to [-1,1]^d$  the estimations built based on  $(\boldsymbol{x}_s, a_s, r_s, c_s)_{s \leqslant t}$  for  $t \geqslant 1$ . We also assume that initial estimations  $\widehat{r}_0$  and  $\widehat{c}_0$ , based on no data, are available. We denote by 1 the column-vector  $(1, \ldots, 1)$ .
Assumption 2. There exists an estimation procedure such that for all individual sequences of contexts  $\pmb{x}_1, \pmb{x}_2, \dots$  and of actions  $a_1, a_2, \dots$  played otherwise, there exist known error functions  $\varepsilon_t: \mathcal{X} \times \mathcal{A} \times (0,1] \to [0,1]$ , relying each on the pairs  $(\pmb{x}_s, a_s)_{s \leqslant t}$ , where  $t = 0,1,2,\ldots$ , ensuring that for all  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ ,
$$
\forall 0 \leqslant t \leqslant T, \forall \boldsymbol {x} \in \mathcal {X}, \forall a \in \mathcal {A},
$$
$$
\left| \widehat {r} _ {t} (\boldsymbol {x}, a) - r (\boldsymbol {x}, a) \right| \leqslant \varepsilon_ {t} (\boldsymbol {x}, a, \delta)
$$
$$
\left| \widehat {\boldsymbol {c}} _ {t} (\boldsymbol {x}, a) - \boldsymbol {c} (\boldsymbol {x}, a) \right| \leqslant \varepsilon_ {t} (\boldsymbol {x}, a, \delta) \mathbf {1},
$$
where we assume in addition that  $\beta_{T,\delta} \stackrel{\mathrm{def}}{=} \sum_{t=1}^{T} \varepsilon_{t-1}(\pmb{x}_t, a_t, \delta) = O\left(\sqrt{T} \ln \frac{T}{\delta}\right)$ .
This assumption is satisfied at least for the two modelings discussed below, which are both of the form: for known transfer functions  $\varphi$ , link functions  $\Phi$ , and normalization function  $\eta$ , for some unknown finite-dimensional parameters  $\mu_{\star}, \theta_{\star,1}, \ldots, \theta_{\star,d}$ ,
$$
\forall \boldsymbol {x} \in \mathcal {X}, \forall a \in \mathcal {A},
$$
$$
r (\boldsymbol {x}, a) = \eta_ {r} (a, \boldsymbol {x}) \Phi_ {r} \left(\varphi_ {r} (\boldsymbol {x}, a) ^ {\mathrm {T}} \boldsymbol {\mu} _ {\star}\right),
$$
$$
\text {a n d} 1 \leqslant i \leqslant d,
$$
$$
c _ {i} (\boldsymbol {x}, a) = \eta_ {c, i} (a, \boldsymbol {x}) \Phi_ {c, i} \left(\varphi_ {c, i} (\boldsymbol {x}, a) ^ {\mathrm {T}} \boldsymbol {\theta} _ {\star , i}\right).
$$
See also the exposition by Han et al. [2022, Section 3.3].
Based on Assumption 2 and given the ranges  $[0,1]$  for rewards and  $[-1,1]^d$  for costs, we may now define the following (clipped) upper- and lower-confidence bounds: given a confidence level  $1 - \delta$  where  $\delta \in (0,1]$ , for all  $t \leq T$ , for all  $\boldsymbol{x} \in \mathcal{X}$  and  $a \in \mathcal{A}$ ,