Contextual Bandits with Knapsacks for a Conversion Model
Authors: Zhen LI
Gilles Stoltz
Venue: NeurIPS 2022 Accept (2022)
Keywords: mutli-armed bandits
bandits with knapsacks
logistic bandits
primal-dual approaches

Abstract: We consider contextual bandits with knapsacks, with an underlying structure between rewards generated and cost vectors suffered. We do so motivated by sales with commercial discounts. At each round, given the stochastic i.i.d.\ context $\mathbf{x}_t$ and the arm picked $a_t$ (corresponding, e.g., to a discount level), a customer conversion may be obtained, in which case a reward $r(a,\mathbf{x}_t)$ is gained and vector costs $\mathbf{c}(a_t,\mathbf{x}_t)$ are suffered (corresponding, e.g., to losses of earnings). Otherwise, in the absence of a conversion, the reward and costs are null. The reward and costs achieved are thus coupled through the binary variable measuring conversion or the absence thereof. This underlying structure between rewards and costs is different from the linear structures considered by Agrawal and Devanur [2016] (but we show that the techniques introduced in the present article may also be applied to the case of these linear structures). The adaptive policies exhibited in this article solve at each round a linear program based on upper-confidence estimates of the probabilities of conversion given $a$ and $\mathbf{x}$. This kind of policy is most natural and achieves a regret bound of the typical order $(\mathrm{OPT}/B) \smash{\sqrt{T}}$, where $B$ is the total budget allowed, $\mathrm{OPT}$ is the optimal expected reward achievable by a static policy, and $T$ is the number of rounds. 

Contextual Bandits with Knapsacks for a Conversion Model
Zhen Li
BNP Paribas, 16 boulevard des Italiens, 75009 Paris, France zhen www.llipppaabbaa...... com
Gilles Stoltz
Université Paris-Saclay, CNRS, Laboratoire de mathématiques d' O'Osay, 91405, Orsay, France gilles. stoltz@unt versi -e-paris-saclly. HEC Paris, rue de la Libération, 78350 Jouy-en-Josas, France stoltzthec. fr
Abstract
We consider contextual bandits with knapsacks, with an underlying structure between rewards generated and cost vectors suffered. We do so motivated by sales with commercial discounts. At each round, given the stochastic i.i.d. context H, and the arm picked a₃ (corresponding, e.g., to a discount level), a customer conversion may be obtained, in which case reward r(a, xx) is gained and vector costs c(a,, 2t) are suffered (corresponding, e.g., to losses of earnings). Otherwise, in the absence of conversion, the reward and costs null. The reward and costs achieved are thus coupled through the binary variable measuring conversion the absence thereof. This underlying structure between rewards and costs different from the linear structures considered by AAgawal and Devanur (2066) (but we show that the techniques introduced in the present article may also be applied to the case of these linear structures). The adaptive policies exhibited solve at each round a linear program based on upper-confidence estimates of the probabilities of conversion given a and This kind of policy is most natural and achieves a regret bound of the typical order (OPT/ B) VT, where is the optimal expected reward achievable by static policy, and 7' is the number of rounds.
total budget allowed, OPT
Introduction and Literature Review
We consider the framework of stochastic multi-armed bandits, which has been extensively studied since the early works by [Tommson [9333 and Robbins [1952]. Two recent (and complementary surveys summarizing the latest research in the field were written by Lattimore and Szepesvári (2200] and [Slivkins] [2019]. On the one hand, we are particularly interested in the setting of contexiual stochastic multi-armed bandits, preferably with some structural assumptions
the dependency between rewards and contexts: linear models (again, a rich literature, see, among many others, Chu et [201]] and Abbasi-Yaakoon Yadkori et [201]] whose work marked a turning point), and, for [0, []-vlledd rewards, logistic models (Filippi et al. [2010) and Faury et al. [2020). On the other hand, we are also particularly interested stochastic multi-armed bandits with knapsacks, i.c. with cumulative vector-cost constraints to be abided by on top of maximizing the accumulated rewards.
The setting was introduced by Badanidiyuru
[2033 2018 and comprehensive summary the results achieved since then may be found in Slivkins 2019 Chapter 10]. The intersection of these two frameworks of interest is called contextual bandits with knapsacks [CBwK] and is the focus of the present article.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Literature review on CBwK. The first approach to CBwK, by Badanidiyuru et al. [2144] and [Agrwall et al. [016]) assumes a joint stochastic generation of triplets of contexts- costs, with no specific underlying structure, and makes the problem tractable by using as a benchmark a finite set of static policies. As noted by Agrawal and Devanur (2016), picking this finite set may be uneasy, which is why they introduce instead a structural assumption of linear modeling: the (unknown) expected rewards and cost vectors depend linearly on the contexts.
We consider a different modeling assumption, motivated by sales with commercial discounts (see Appendix A): general (known) reward and cost functions are considered but they are coupled via a 0/1-valued factor, called (customer) conversion, obtained as the realization of a Bernoulli variable with parameter P(a. æ)) depending on the context z observed (customer's characteristics) and the action a taken (discount level offered). The probabilities P(a, x) are themselves modeled by a logistic regression, whose parameters may be learned through an adaptation of the techniques by Filippi et al.) (2010) Faury et al. [200)) We do so in the first phase of the adaptive policy introduced in this article. More details on the comparison of the new setting considered to known settings of CBwK may be found in Section 2.2
Primal-dual approach. The second phase of the adaptive policy exhibited uses the primal-dual approach to a convex optimization problem-actually, a simple optimization problem given by a linear program. This approach was already used various ways for bandits with knapsacks, including CBwK, to define policies based on the dual problem: this is explicit in the LagrangeB w policy Immorlica al. [2019] and is implicit in the seward-minus-weipg
approach of grawwal and Devanur [2016] and Agrawal et al. (016). we underline in the proof sketch of Section as well as in the discussion of Section 6 However, we only use the primal-dual approach in
analysis and state our adaptive policy directly in terms of the primal problem, where we substituted upper-confidence estimates of the probabilities P(a, æ). We therefore end up with a most natural adaptive policy, which mimics the optimal static policy used as a benchmark. This direct primal statement of the policy actually also works for the setting of linear CBwK studied by Agrawal and Devanur 2016]. we show in Section 6 Policies based such direct primal statements were already considered for bandits with knapsacks (see Li et al. 2021 and references therein) but do not seem easily extendable to CBwK.
The first contribution of this article a new structured setting of CBwK, based on coupling between general rewards and cost vectors through conversions modeled based on a logistic regression; we present and discuss in Section 2.1 (and explain its origins in Appendix of the supplementary material). The adaptive policy introduced is described in Section 3 Its first phase consists of learning the parameter of logistic regression and is adapted from Faury et al. [2020) Its second phase-and and this is the second contribution of this article-diredd problem with optimistic conversion probabilities. The analysis, which we believe is concise, elegant, and natural, is provided in Sections (when the context distribution // is known) and (when is unknown). As mentioned above, Section draws the consequences of our second contribution for linear CBwK.
Outline and main contributions.
solves primal
Notation. Throughout the article, vectors are denoted with bold symbols. In particular, and denote the vectors with all components equal to 0 and 1, respectively. With no additional subscript, ||v] denotes the Euclidean norm of a vector v, while subseript given by a non-negative symmetric matrix M refers to ||vl|as = vVT Mv.
2 Learning Protocol and Motivation
We describe the learning protocol and objectives considered (Section 2.1 and explain why it is not covered by earlier works (Section 2.2). We also detail (Appendix A in the supplementary material) how this learning protocol was defined based on industrial motivation in the banking sector: market share expansion for loans by granting discounts, under commercial budget constraints.
2.1 Learning Protocol and Modeling Assumptions
We consider a finite action set A. including a special action amil called no-op, and finite context set C RR,. (We discuss and mitigate finiteness of X' in Section 2.2 A scalar reward function
A x X [0, 1] and vector- vector-valued cost function e A x X + 0. d evaluate the performance

---

of actions given the contexts. There are several sources of costs to control: each corresponds to component of c. We assume that these function are known, and (with no loss of generality) that their ranges are [0, 1] and [0, 1j. The no-op action induces null reward and costs: r(A 2x) O and cl ((0mm; 2) for all a E X.
Contexts- correspond, for instance, to customers' characteristics, see Appendix -are are drawn sequentially according some distribution is which may be known or unknown (we will deal with both cases). At each round & > 1, upon observing the context 2; E X' drawn, the learner picks an action a E A. which corresponds, for instance, to an offer made to the customer t. If the latter accepts the offer, an event which we denote y/ 1, then the learner obtains a reward r r(a,, 24) and suffers some costs c(a,, Zt). When the customer declines the offer, we set ye 0, and null reward and costs are obtained. Thus, in both cases, the reward and costs may be written as r r(a,, ze) and c(at, F:)) yt- We call y/t the conversion and now explain how it modeled.
Modeling conversions. We model each conversion as an independent Bernoulli random drawn, with parameter P(at, æx)) depending on the context o, and action a, # Omm- We further assume that these probabilities may be written as logistic regression model, i.e., there exists a known transfer
function yp: A {Amill J x X' + I and some unknown parameter O, E I such that
Væ E X, Va E Aaaamn)
P(a, x) ((<(a,,)) x))(,,))..)) 0.), where n/xx) 1/(1+e (I)
We assume that yp is normalized in way that its Euclidean norm satisfies ||0]|| < 1 and that bounded convex set θ containing 0, is known. Such a modeling is natural and opens the toolbox of logistic bandits; see Faury et al. [2020) and references cited therein. We however note (and discuss this fact in Appendix CC that the logistic regression model above is slightly different from the one by /Farry et al. 2020
The concept of a conversion y for a round when the no-op action Gma is played is void, and thus, we leave the probabilities P(amut: æ) undefined, though by abuse of notation, these quantities might appear but always multiplied by O, given, e.g., by indicator functions like I {m null rewards r(anat, 2), or null costs c(amatt x).
Policies: static vs. adaptive. The learner is given a number of rounds T and maximal budget B (the same for all cost components, with no loss of generality: up to some normalization). static X + ((A), where P(A) is the set of probability distributions over A. As is
policy is a function traditional in the literature of CBwK (we recall below why this the case), we take as benchmark the static policy n' with largest expected cumulative rewards under the condition that its cumulative costs abide by the budget constraints in expectation. More formally, n achieves the maximum defining
O,
=
max TEX~ RiX+P((A
Σ r(a, X) X) (X)
aaE
(2)
under
TEX
Σ c(a, X) P(a, X) (X)
< B1,
LEEA
where E Ex~r denotes an expectation solely over random contexts X following distribution v, where A, and where & is understood component-
Ta (X) denotes the probability mass put by (X) wise. Of course, the sums in the two expectations above are taken indifferently over A or A {amii) }
The learner uses an adaptive policy, i.e., a sequence of measurable functions P; H'-1 x X - P(A) indexed by > 1, where H = X' x A (0. 1). Indeed, the history available to the learner at the beginning of the round > 2 is summarized by h = (æ.,, Ox Vs)sst-1, and we define hg as the empty vector. Such a policy draws the action a, for round t > independently at random according to p, (1; Rz). We impose hard budget constraints adaptive policies: they must satisfy
c(as, Dt) < B1
a.s.
Such adaptive policies are called feasible in the literature. To abide by these hard constraints, we may restrict our attention to adaptive policies that pick Dirac masses on amml whenever one component of the cumulative costs is larger than B 1. At the same time, an adaptive policy should maximize the cumulative rewards obtained or, equivalently, minimize its regret:
RT P,
Σ GT
r(a,, Zt)yt.
3

---

BOX A: CONTEXTUAL BANDITS WITH KNAPSACKS [CBWK] FOR CONVERSION MODEL.
Known parameters: finite action set A including no-op action Quili' finite context set X C R"; scalar reward function r : A x - [0, 1]; vector-valued cost function Ax > [0, 1j4; number T of rounds; total budget constraint B >
Possibly unknown parameters: context distribution // on X'; probability of conversion given action and context P: A {mmin) x [0. 1], modeled as P(a. x) ()p((a, x)' 0.) for some known transfer function yp AA {a } X IRm with ||s/|| < 1, and some unknown parameter 0. E IRm" lying in known bounded convex set θ
For rounds 1,.,,,...,,TT
1. Context at, ~ is drawn independently of the past;
2. Learner observes 2 and picks an action at A:
3. Conversion y/ E {0, 1) is drawn according Ber (((ot, æi)):
4. Learner observes Yt. gets reward r(az, xz) yr. and suffers costs c(a:; wt) yr-
Goals: Maximize
Σ EST
r(as, xt) ye
while controlling
c(a,, xz)) ye E B
may be proved (along the same lines as Agrawal and Devanur (0066] Appendix B] do for a different model) that the optimal static policy n² obtains, on average and in expectation, a cumulative reward least good as the best feasible adaptive policy.
Summary. summary of the learning protocol and of the goals is provided Box A. We note here that rewards gained and vector costs suffered round in the V/ 1 of a conversion could be stochastic with expectations r(a,, zz) and c(a,, 2z): our analysis and the regret bounds would be unchanged, as long as the expectation functions and are known.
2.2 Discussion and Comparison to Existing Learning Protocols
The setting described above may be reduced to the general setting of CBwK, introduced by Badanidiyuru et al. (2014) and AArrwaa et al. [2016]. Indeed, introduce independent Bernoulli variables Vt,a with parameters P(a, æt), for all E A {ama)) and set Vt, 0. The vectors
(m2, (ra(a)) (ca(a))nea)
where (a( r(a, ax.) Vt.a and ca(a) c(a, mx) Vt,a
are i.i.d., and upon picking action a, E A, the obtained and observed rewards and cost vectors equal ra(e)) and e (az). When X' discrete, we may consider the set II of base policies that map X' to (6a E A). the of Dirac masses policies X - P(A), against which we would like our policy to compete; but the adaptive policies by Badanidiyuru al. (2014) and Agrawal et (2016) only compete with respect the best single element in II, not the best convex combination of elements of 11.
A. The convex hull of IT is the set of all static
The setting of linear CBwK (Agrawal and Devanur 2016)) provides structural link between contexts and expected rewards and cost vectors, but in a linear way that is incomparable to the setting of CBwK for conversion model introduced above. More details mention that linear and logistic structural links between contexts (prices) and rewards or costs were studied in non-contextual setting (i.e., not in CBwK) by Miao al. [2021) Their strategy bears some resemblance to the one by Agrawal and Devanur (006)) in particular, both consider an online convex optimization strategy subroutine.
given in Section 6 We also
All mentioned references consider a no-op action amoll- (It could be replaced by the existence of a standard action ago-cost always achieving null costs and possibly some positive rewards.)
On the contrary. none of the mentioned references assumes that the context technical necessity for a part of the adaptive policy introduced; see the discussion of computational complexity the end of Section
is finite. This is
But somehow, considering a finite set of policies, as in Badanidiyuru et al. 2014) and Agrawal et al. [016), is counterpart to assuming finiteness of X. Also, Appendix Factually mitigates this restriction that X' is finite: learning the logistic parameter 0. may be achieved with continuous contexts (see Phase 1 in Section 3); only the subsequent

---

optimization part (Phase 2 in Section 3 requires finiteness of X. We may thus well discretize only X for this Phase 2, which is exactly what Appendix/Fjperforms. This mitigation comes with possible theoretical guarantees as Sections 4 and | reveal that the errors Ex(a, .)) for learning O, and P. obtained as outcomes of the first step of the analyses, are carried over in the subsequent steps, where the optimization part is evaluated.
3 Description of the Adaptive Policy Considered
At each stage ł > 1, the policy first updates estimator 0,-1 of 0, based the history / available so far, based on an adaptation of the Logistic-UCBI I algorithm by Faury et al. [2020]. and deduces estimators P (a, a) and upper confidence bounds U-- (a, æ)) of the probabilities P(a, a). The policy then solves the corresponding estimated version of the optimization problem (2). We now describe the corresponding two steps. In the description below, quantities that depend on information available at round & (respectively, t) are indexed by t (respectively, t).
Phase 0: case the cost constraints are about to be violated. To make sure cost constraints are never violated, whenever at least one of the components of the current cumulative costs is larger than B 1 and could possibly be larger than B at the end of round t. we play anuli (and we actually do so for the rest of the rounds). This corresponds to defining Pe (ht-1. x) Semun for all æ E X, where dسmeg denotes the Dirac mass on aguli- Otherwise, we proceed as described below in Phase 1 and Phase 2.
Phase Learning 0, via an adapted Logistic-UCBI. This first phase depends on a regularization parameter X > O and on upper-confidence bonuses E, (a, æ)) > O, both to be specified by the analysis.
At rounds > we first maximize a regularized likelood of the history hr-1:
In 101 va).mmmm1////)1 (n(I-((p(a,,,))) x. 0)) (y. (o In x.)" x)
A
0:-15 argmax 2E
Σ
1 (ami)
1111.
(3)
In the expression above, we read that we only gather information about I, those rounds s when ag F Guull. When 1 does not belong θ, an ad hoc projection step corrects for this, needed:
i 1 argmin AEN
(0) V ( W [
1-1 ,r-1
where
Σ
1 Hamtamm) (<sp(as; x. 0) sp(as; xt)) 10
1=1
and
W,-1(0) = A Am 1 (o. ( (a,, xx 0) (000))(aa00) 2x)p(a (5)
We recall that the function ij denotes the derivative of n7, i.., nfxx) e (1 We have v 711 7))
By plug-in, we finally define estimators and upper-confidence bounds the probabilities P(a, x) for + amml and all E X::
and U,,i(a, a) = (a ax) St-1(a, x); 1} For ammn, no estimators or upper-confidence bounds need to be defined, the quantities P(amu, x) are actually undefined.
(a, x) ()(((,, x) 0.-1)
Sampling, via solving optimization problem with expected constraints. This phase relics on a conservative-budee parameter denoted by Br, which is only slightly smaller than B and whose exact value is to be specified by the analysis.
Phase 2:
We start with the case known context distribution v. At round = 1, we play an arbitrary action in A \ {ammi }. At rounds > 2, if at least one component of the cumulative vector costs suffered so far is larger than B 1, we pick ag = Onm- Otherwise, we pick for p, (ht-1,,) .) the solution of the optimization problem OPT(P, (2, U, Br) defined in (2). and draw 0; according to P, (1,=1,
In the definition 2 of (P, U1- Br). expectations are only over X ~ and not over the random variable U--1; more comments and explanations on this fact may be found in Appendix B.3
5

---

Box B: LOGISTIC-UCB FOR DIRECT SOLUTIONS TO OPT PROBLEMS
regularization parameter > 0; conservative-budget parameter By; upper- confidence bonuses E (a, x) > for > and (a,x) (AA) {amil) }) x X.
Parameters:
Round Ê = 1: play an arbitrary action A {Awm)
At rounds >
Phase
as) < (B- 1)1 violated, then Pr (, Samall for all a +
Σ
Phase
Otherwise, compute a maximun-likelihodd estimator 0.-1 of 0, according to 33. compute its projection Q onto according to 4 and define, for a * Oull
Pr-i(a, x)) n(p(a,, x) 0.-1) and Uz-1(a, x) - min P:: P7-1(0, (a, x) +) (a)
Phase 2 Compute the solution P, (h of
UB,1; ((A)P(A)
T TEx
Σ ((1.a)) Ta)
LEEA
under
TEx~
Σ (i- ma
< Br1,
LAEA
where denotes either v (when is known) or empirical estimate 17 in Draw 0₂ P, (
When the context distribution is unknown, we rather pick for p. (ht--1, -) the solution of the opti- mization problem OPT(D, U,.1, where
D.
6R.
(6)
with &n denoting the Dirac mass X. Since æ, is revealed at the beginning of round f, before we pick an action, we may indeed use D. at round t.
Summary and discussion of the computational complexity. We summarize the considered adaptive policy Box B and now discuss its computational complexity.
As In to and In(1 - :)) are strictly concave and smooth, the maximuu--likelibne step B of Phase 1 consists of maximizing strictly concave and smooth function over TR. which may performed efficiently. The projection step (4) of Phase 1 is however an issue, both with the version of Logistic- UCB1 discussed here and with earlier approach by Filippi et al. [2000, Section 3). The latter and
Faury et al. [020) Section 4.1] both underline that the projection step (4) is complex optimization problem that however does not often need to be solved in practice, they usually observe 0,-1 e. Our numerical experiments concur with this statement (but admittedly, they rely on choosing rather large value of O).
On the contrary, Phase of the adaptive policy consists of solving linear program with XX x |A] constraints, where where |X' and A denote the cardinality of X' and A. respectively- see the detailed rewriting (13) in the supplementary material. Therefore, the computational complexity Phase is polynomial (of weak order) in | [A]. To achieve this acceptable complexity we had however to restrict our attention finite sets of contexts X. which requires in practice segmenting countable or continuous context sets into finitely many clusters, for instance. We do so in our experiments.
Simulation study. simulation study on partially simulated but realistic data may be found in Appendix F The underlying dataset is the standard "default of credit card clients" dataset of UCT (2016). initially provided by Yeh and Lien [2009] (It may be used under a Creative Commons Attribution 4.0 International [CC BY 4.0] license.)

---

Analysis for a Known Context Distribution //
Since θ is bounded. the following quantity, standardly introduced in the context of logistic bandits (see Faury et al. 2020] and references therein), is finite, though possibly large:
sup (i( xF 0) = E X, a a famm): 0 E @ <
We denote by ||o]| = max max(110) E θ the maximal Euclidean norm of an element in O.
By construction, given that individual cost vectors lie in [0, 1] and due to its "Phase 0", the adaptive policy considered always satisfies the budget constraints. The bound on rewards reads as follows.
Theorem 1. In the setting of Box of Section 2.7 we consider the adaptive policy of Box B of Section 3 assuming that the distribution the contexts is known, ie., with i We set confidence level 1 E (0, 1) and use parameters = m (n + T/m),
By = B 2
(2TTI(44//)
and Ee(a, x) stated in (9) of the supplementary material. Then, provided that T > 2m and > + 10/2T1((Ad//)) we have, with probability at least 26,
4d
OPT(U, P) + Er + 1 2T in 4 +1,
OPT(V, P,B)
Er(a,, x4) < 4 2 2T
B
5
EET
where the closed-form expression of Ex O (m (mVVTInT) is in 35 of the supplementary material.
We will rather discuss the bound of the more general Theorem 2 tto be stated and proved in Section 5) than the one of Theorem T We provide proof sketch in Section 4.1 and discuss the main technical
novelty in Section 4.2
4.1 Proof Sketch for Theorem
The detailed proof of Theorem may be found in Appendix B We provide here overview thereof, highlighting the four main ingredients. The third and fourth steps benefited from some inspiration drawn from the proof techniques of Agrawal and Devanur 2016). The first step is an adaptation of Lemmas and 2 by Faury et al.] [2020).
First, the mentioned adaptation provides values of the parameters ca æ)) such that, with probability
least 1 - 8,
Vt > 1, Va A {amith Vo e X,
Pi((a x) P(a, x) < x)) U2(a, ar) (a, m) P(a, v) & (,)a,)),
hence
while
Σ T
E xz) (aut) is of order /T up to voly-logarithmic terms.
Second, the Phase 2 formulation of the strategy, in a primal form, is equivalently restated in a dual form. For each round t > 2, strong duality holds and entails the existence of a vector 3!ude E Rd such that p. (hr-1, -) may be identified the argmax over क X - P(A) of
Ex ~-
T Σ - aEA
- (Bhhaus c(a, xi) (a, TT
+ Σ
Σ
pos,
TT (x)
ΦE aEA1
By exploiting the KKT conditions, we are able to get rid of the double sum above and finally get X' -'pointwise characterization of (14-11 -): for all E X,
Pr (1 x:) E aargmax Σ (r(a, ar) (g c(a, x)) 1 x:) ga
qEP(A) aEA
Σ (r(a, x)) - (spuue:+)) (e(alt,))cc(az000 x)) Un x)) qu
arggaax qEP(A) aEA
7

---

Non-negative parts (+)+ may be introduced thanks to the existence of the no-op action Caall The distributions (, x) may therefore interpreted as maximizing some upper-confidence bound on penalized gains (rewards minus some scalarized costs); the dual variables Br play role similar to the Z parameter of Agrawal and Devanur 2016, Section 3.3] in terms of weighing gains versus costs. In passing, we also prove
OP,(U, - Br) > (r
based on the KKT conditions. The latter inequality is comparable in spirit to the bound of Agrawal and Devanur 2016 Corollary 31, relating Z to OPT(V, P. B)
Third, for > 2, whenever the policy p. (1 +) is obtained by solving the optimization problem OPT(U, U Br) of Phase 2 and by independence of æ, and h have
OPT(U, UL-1, BT)
= Exav
Σ r(a, X) U2-1(a, X) (h x) LaEA
T
E [r(a 2x) Ut-1 (a,, 3t){ 14-1]=
Therefore, repeated applications of the Hoeffding-Azumi inequality and the inequalities of the first step entail that, up to quantities of the order of VT,
T
OPT(V, U1-1, Br)
T
r(a. (1))(-1(a4,)))
T
1=2
T
EL-1 (as, x4)1
r(as, x) P(a x2)
Σ
r(a x ye-
(ast
t=2
OPT(V, Br) which may be assumed >> 0.
We thus only need to control OPT B)
T
The value By B 2 / 27 In (ndd/)) and similar Sareiiin-Azuaa-basssee arguments show that with high probability, the budget limit B 1 is indeed never reached and that we always compute (h -) in the way indicated by Phase 2.
Fourth, we collect all bounds together. We start with
Σ BT
hode 1 OPT(V, UT-1,
OPT(U, P,B).
T
We the exploit the dual characterization of p.(h-11 :) and the control P P& U-- to get that with high probability, for all E X,
Σ (r(a, .)) (Br c(a, U- 1- x) Pr.o (r.0(1111::0 aEAl
x)) (1thatt) cla, x)) P(a, x)) n (x))
After integration over X and substituting of the definitions of x² and Pt,a (he-1, +), as well as the equality stemming from the KKT conditions, we have
=OPT(v.U,-1.B1))
Ex
Σ
r(a, X), X) Pr,a (ht-11,1,
LEd
Exxu
Σ E(Brutee (Bhute::)** "ela,,)UU-i U,ii(a,, X) Pr.a (he X)
LaEA
(B7/T)((82ule. hoieee
> Ex
Σ r(a, X) P(a, xX)
-(3))0le.)) Exar
Σ
c(a, x)a,X) X) x) xx
LaEa
LaEd
=OPT(,,P.B)/T
<(B/T)1

---

Rearranging and summing over 2 t T.
we obtain
T
OPT(U, P. OPT(u,U,-1, Br)
T
Br
B Br -1)
OPT(V, P. P)
T
T
1=2
where we substituted the first incquality stated in this fourth step. This concludes the proof.
4.2 Discussion on the Main Technical Novelties
As should be clear from the comments at the beginning of Section 4.1 the technical novelties lies in the second step of the proof of Theorem [ On the one hand, we are able directly analyze a strategy stated in primal form, which is a more natural formulation. On the other hand, doing so, we are also able to avoid the issues that come with dual formulations, relying, e.g., on some critical parameter Z, as in Agrawal and Devanur (2016) Theorem 3]. to trade off rewards and costs. This parameter Z should be of order OPT/ B and has to be learned, e.g., through 'T initial exploration rounds. (More details are to be found in Section E.3 In our analysis, this parameter Z is superseded by dual optimal variables B 2 O, that are only used in the analysis and not to state the policy, unlike in Agrawal and Devanur 2016]_ Put differently, the clever use in this context of KKT conditions is the main technical novelty. On side note, we are also able to take care in an explicit and detailed fashion of the no-op action Omall. whose specific treatment is often unaddressed in the literature.
Analysis for an Unknown Context Distribution V/
When the context distribution v is unknown, we simply estimate it through its empirical fre- quencies T The regret bound is almost unchanged: an additional mild factor of, e.g., B)BBB. This term comes
2 2T In (1/8) appears the vT term multiplying from some uniform deviation argument stated in 7 and
Theorem 2. the setting of Box of Section 2.7 we consider the adaptive policy of Box B of Section with i- = D. at rounds t > 2. We set a confidence level 1 - E (0, 1) and use parameters A m T/m),, a working budget of
B-br.
where
br= 2 2T 1 2T 2n (2T x1/1/)
and ea x) stated in (9) the the supplementary material. Then, provided that T > 2m and B >> 2br, have, with probability at least 1 35,
OPT(V, P) - Σ r(a,, m.) De & 2br (1+ OPT(U, + Er,
B
1ET
where the expression of E = 10m0mmmTT may be found in (35 of the supplementary material.
The order of magnitude of the regret bound is (m + |X|OPT(L, P. B which is remi- niscent of all known regret upper bounds for CBwK (e.g.. the ones by Badanidiyuru al. [014 and Agrawal et al. 2016). for general CBwK, and and Agrawal and Devanur [2016] for linear CBwK, Section 6). The factor |.]] may be improved, see below, but this is detail. A discussion on exhibiting corresponding lower bounds is to be found at the end of Section [
A detailed proof of Theorem 2 is provided in Appendix of the supplementary material. It follows closely the proof of Theorem with modifications mostly consisting of relating quantities of the form
Ex~ø, [f)) vs.
Ex~r [X()] where, e.g., f(X) = Σ r(a, ((aa Ua, Pr,a (h-1,
To do so, we use that for all functions f : X' - [0, 1).
aEA
T, Ex [x() - Ex [(x)
Σ [1((z)) - v(z) def |00 v 111, (7)
ZEX
where |/ - // |1, is the total variation distance between D2 and In Appendix D upper bound
the latter, for the sake of simplicity, in crude way by applying T|x times the Hoeffding-Azuma inequality (once for each 1 & t & T and a E X¹) and obtain that with probability at least 1 - 5,
In x
14 < T,
| - u/1 < 1181
(8)

---

The / In (2)) term in the regret bound of Theorem appears as the sum over t < T of the deviation bounds 8 The bounds (8) may actually be improved into bounds of the order of
/t, via some Cauchy-Schwand bound and a deviation argument in Banach spaces by Pinelis [1994]. or by more direct techniques described by Devrroee [1983] Lemma 3] and [Berend and Kontorovich [012]. In any case, the regret bound of Theorem 2 automatically benefits from such improvements, by replacing the 2 2 term in the bound therein by the (sum over t & T of the) better uniform deviation bounds.
6 Extension to Linear Contextual Bandits with Knapsacks
This section a brief summary Appendix We explain therein how the adaptive policy of Box may be adapted to the setting of linear CBwK, introduced by (Arawall and Devanur (2016), where the bounded rewards ra and vector costs er are independently generated at each round according to bounded distributions with respective expectations F(a,, Øx) and a(a, æz). depending linearly (a
transfer function y of) the contexts: for all a + amull and x E X, for all components i of c,
F(a, x:) = (p(a, p. and Ca x:) = (( x 0.,5-
We consider the same benchmark OPT(U, F,C, B) as Agrawal and Devanur (0166) and are able to exhibit a similar (OPT( c. B)/B) mn In 7 regret bound, with however slight relaxation on the order of magnitude required for B. We do so with a strategy that we deem more direct and natural, inspired from the one of Box B, where in Phase a LinUCB-type (Abbasi- Yadkori et al. (2011)) estimation of the parameters is performed, and where in Phase 2, a direct solution to an OPT problem with estimated parameters is performed. The parameters are upper-confidence functions U,-1 and lower-confidence vector functions -t-1 on T.
The main advantage of our approach in the case of linear contextual bandits is exactly as described in Section 4.2] avoiding the critical parameter Z of Afrawwl and Devanur [2016] Theorem 31, which is used to trade off rewards and costs. The main limitation of our approach is the assumption of a finite context set X, which is required to make the Phase-2 linear program tractable.
Future Work
We conclude this article with list of issues to be further investigated.
First, discussed in Section 2.2 the restrictions of finiteness should be alleviated: finiteness of the context set in the setting of this article, or finiteness of the set of benchmark policies in other settings (see Badanidiyuru et al. 2014 and Agrawal et al.) 2016).
Second, we only dealt with < budget constraints (and do does the literature so far). Direct approaches to constraints of the form > remain to be further investigated.
A third series of questions to be clarifies concerns regret lower bounds, and more generally, the tightness of the results-ip in particular, the required conditions on budget sizes. Earlier references for contextual bandits with knapsacks did also not provide lower bounds statements that were simultancously optimal (i.e., matching the obtained upper bounds) and general (i.c., valid for all problems with a given number of rounds T, given budget B, and given value OPT for the optimal expected reward achievable by static policy). [Badanidiyuru et al. [2014, comments after Theorem 1] merely indicates that the obtained regret upper bound optimal budget B grows linearly with the number of rounds T. Agrawal and Devanur 2016, comments after Theorem I] only compares the obtained regret upper bound to the case of no budget constraints. In particular, as far as the orders of magnitude rates achieved (up to pogy-loga factors) in the present article and in the two mentioned references are optimal. These rates do not match the optimal rates in the case of no contexts, which were stated and proved by BBadnnidiyuru et al. (0013).
some regimes, e.g., when the
are concerned, is unclear whether the (OPT/ B))
Acknowledgments and Disclosure of Funding
Zhen Li and Gilles Stoltz have no direct funding to acknowledge other than the salaries paid by their employers, BNP Paribas, CNRS, and HEC Paris. They have competing interests to declare.
10

---

References
Default of credit eard clients. UCT Machine Learning Repository, 2016. UtL.htttps:/aarchive
iis.ucci http://www..aaae..eee.//o.aeoat.
Y. Ybbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits, In Advances in Neural Information Processing Systems (NeurIPS' 11), volume 24, 2011.
S. Agrawal and N. Devanur. Linear contextual bandits with knapsacks. Advances in Neural Information Processing Systems (NeurIPS' /6), volume 29, 2016.
S. Agrawal, N.R. Devanur, and L. Li. An efficient algorithm for contextual bandits with knapsacks, and an extension concave objectives. In Proceedings of the 29th Annual Conference on Learning Theory (COLT" volume PMLR:49, pages 4-18, 2016.
A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with knapsacks. In IEEE 54th Annual Symposium on Foundations of Computer Science (FOCS'13), pages 207-216, 2013. Latest extended version available at arXiv: 1305,2545, dated September 2017.
A. Badanidiyuru, J. Langford, and A. Slivkins. Resourceful contextual bandits. In Proceedings of the 27th Conference on Learning Theory (COLT" volume PMLR:35, pages 1109-1134, 2014.
A. Badanidiyuru, R. Kleinberg, and A. Slivkins. Bandits with global convex constraints and objective. Journal of the ACM, 65(3):155,, 2018.
D. Berend and A. Kontorovich. On the convergence the empirical distribution, 2012. Preprint, arXiv: 205.6711.
S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
W. Chu, L. Li, L.. Reyzin, and R. Schapire. Contextual bandits with linear payoff functions. In Pro- ceedings of the 14th International Conference on Artificial Intelligence and Statistics (AlStats" 11), volume 15, pages 208-214, 2011.
L. Devroye. The equivalence of weak, strong and complete convergence in l for kernel density estimates. Annals of Statisticy, 1163)896-904, 1983.
L. Faury, M. Abeille, C. Calauzenes, and O. Fercoq. Improved optimistic algorithms for logistic bandits. In Proceedings of the 37th International Conference on Machine Learning (CCML'20), volume PMLR:1199 pages
2020.
Filippi, O. Cappe, A. Garivier, and C. Szepesvári. Parametric bandits: The generalized linear case. In Advances in Neural Information Processing Systems (NeurIPS' 10), volume 23, 2010.
J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3): 90-95, 2007.
N. Immorlica, K.A. Sankararaman, R. Schapire, and A. Slivkins. Adversarial bandits with knapsacks. In 2019 IEEE 60th Annual Symposium on Foundations of Computer Science (FOCS' 19), pages 202-219, 2019.
T. Lattimore and C. Szepesvári. Bandit Algorithmss. Cambridge University Press, 2020.
X. Li, C. Sun, and Y. Ye. The symmetry between arms and knapsacks: A ppimal-duul approach for bandits with knapsacks. In Proceedings of the 38th International Conference on Machine Learning (ICML'21), pages
2021.
S. Miao, Y. Wang, and J. Zhang. A general framework for resource constrained revenue management with demand learning and large action space. Available at SSRN 3841273, 2021.
I. Pinelis. Optimum bounds for the distributions of martingales in Banach spaces. Annals of Probability, 22(4):1679-1700 1994.
11

---

H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Sociery. 58(5):527-535, 1952.
A. Slivkins. Introduction to multi-armed bandits. Foundations and Trends@ in Machine Learning, 12 (1-2):1-286, 2019.
W.R. Thompson. On the likelihood that unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294 1933.
Z. Xu and V.-A. Truong, Reoptimization algorithms for contextual bandits with knapsack constraints, 2019. URL httl:ttt///wwt http::/wwww.....o..eueeeeooooaaa
pdf
I.C. Yeh and C.H. Lien. The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients. Expert Systems with Applications, 36(2):2473-2480 2009.
M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML'03), pages 928 936, 2003.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The main ones are (i) that we only consider finite context sets X' (because of Phase 2 of the adaptive policy) and (ii) the computational complexity of the projection step (Phase 1 of the adaptive policy); detailed discussion the end of Section See also Section 6 as far as (i) is concemed, and Section 2.2 and Appendix for a mitigation of this limitation.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] To the best we could see the cover page of the supplementary material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We do at various places in Appendix [F] note however that simulations are mostly included for the sake of illustration only.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] To the best we could as the computation a single run was taking 8h
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the final section of Appendix
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Existing assets consist (only) UCI Machine Learning Repository dataset; we provide the citations required by this repository.
(b) Did you mention the license of the assets? [Yes] namely, CC BY 4.0 see the beginning of Appendix F
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We classic dataset that has been around for more than decade.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use classic dataset that has been around for more than decade.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "wxDpAZmonR", "reviewer": "Reviewer_7uoE", "summary": "This paper studied the problem of contextual bandits with knapsack constraints, where, in each round, given the stochastic i.i.d. context and the selected arm, a conversion rate is obtained. The reward and costs are coupled through the binary variable measuring conversion. The authors proposed a new adaptive policy that solves a linear program based on upper-confidence estimates of the probabilities of conversion in each round, and they show that the proposed policy achieves $(\\mathrm{OPT}/B)\\sqrt{T}$, where $B$ is the total budget allowed, $\\mathrm{OPT}$ is the optimal expected reward, and $T$ is the number of rounds. The authors also showed that their proposed techniques can also be applied to the linear structures considered in [Agrawal and Devanur 2016].", "questions": "- For the projection step in Phase 1 of the proposed adaptive policy, could the authors elaborate on how the projection problem should be formulated and solved?\n\n- The authors stated in quite a few places that their regret bounds can likely be improved. Could they provide a tighter analysis?\n\n- The proposed adaptive policy and its analysis are based on or inspired by integrating several existing techniques (e.g., Logistic-UCB1 by [Faury et al. 2020], [Agrawal and Devanur 2016]). The key analysis steps are standard and similar to these existing works. Could the authors highlight the key differences, challenges, novelty, and contributions in their proofs?", "limitations": "This paper is theoretical and irrelevant to negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This paper studied a new MAB model that integrates contextual stochastic bandits and knapsack constraints, which is well motivated in practice.\n+ The authors proposed an adaptive policy that achieves $O(\\sqrt{T})$ regret for the proposed contextual MAB model with knapsack constraints.\n+ The proposed adaptive policy can also be applied to traditional linear contextual bandits with knapsacks.\n\nWeaknesses:\n\n- The proposed adaptive policy requires a finite context set.\n- The computational complexity of the proposed adaptive policy is high: Phase 1 of the proposed policy requires a non-trivial projection step and Phase 2 assumes a finite context set. \n- Theoretical analysis techniques are standard and some theoretical regret bounds are not sharp.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jctrIYyz5aF", "reviewer": "Reviewer_xxh2", "summary": "This paper proposes a new contextual bandit model with knapsacks. In this new model, the decision-maker sequentially offers actions to arriving users after observing their contexts. Rewards and costs are only realized if the users accept the offer (a conversion happens). Conversion probabilities follow a logistic model that depends on action-context features and an unknown parameter vector. The goal is to design an adaptive policy that performs almost as well as the best static policy that knows the context distribution and conversion probabilities. The authors propose logistic-UCB1 algorithm to solve this problem. The proposed algorithm at each round computes the MLE estimate of the unknown parameter vector based on history. Then, it computes action selection probabilities by solving an optimization problem with expected constraints. This is a linear program that can be computationally manageable for finite context spaces. Sublinear regret upper bounds are shown both when the learner knows the context distribution and when the learner uses an empirical estimate of the context distribution. Supplemental material provides a motivating example from the banking sector, where the actions correspond to discounts on offered loans. They also investigate how to extend the current work to linear contextual bandits with knapsacks and come up with an algorithm with better budget requirements and parameter tuning capabilities, albeit requiring the finiteness of contexts.", "questions": "Why negative regret appears in simulations? \n\nWhen justifying comparing with the optimal static policy in regret, the work of Agrawal and Devanur (2016) is mentioned. However, the setup here is not the same due to the conversion model. How is it ensured that the relation between the optimal static policy and the best feasible adaptive policy remains the same as in this prior work under the conversion model? ", "limitations": "This work does not have any societal impact. Limitations are well discussed. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper is well written. It proposes a new structural model called conversion for contextual bandits with knapsacks inspired by work on logistic bandits. The model has interesting real-world applications. The regret metric and the proposed algorithm are intuitive. I think the main paper does a good job positioning the current work within the literature on contextual bandits with knapsacks. Comparison with existing works is sufficient. \n\n\nWeaknesses: \n\nThe paper can do a better job of highlighting the technical novelty in the analysis. In the main paper, attention is paid to describing steps that bear similarities with the earlier works. Please elaborate if any non-trivial new technical machinery was introduced in the regret analysis apart from the KKT analysis. \n\nA discussion on lower bounds and tightness of the results will help readers understand the merits of the proposed work better. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PWDuWGz1rDi", "reviewer": "Reviewer_fCTz", "summary": "In this paper, the authors studied the contextual bandits with knapsack problem, where the rewards and costs are coupled through a conversion term. The authors proposed an adaptive policy for the proposed setting, and showed that it achieves a regret bound of order $\\tilde{O}(\\frac{\\text{OPT}}{B} \\sqrt{T})$, regardless of whether the context distribution is known or unknown. Finally, the authors also briefly discussed how their approach also applies to the linear contextual bandits with knapsack setting studied in [Agrawal and Devanur 2016].\n", "questions": "1. The authors mentioned that the regret bound in Theorem 2 can be further improved if the uniform deviation can be improved. Could you provide more details related to this statement? Why is the bound in equation (7) likely to be improved?\n2. The authors state that the projection step in Phase 1 could be an issue. I wonder how this issue is dealt with? Do you do something different in numerical studies? Can you make certain changes to this step to improve the theoretical complexity? \n", "limitations": "None. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors considered a novel setting that couples reward and cost using a conversion term, which corresponds to whether the customer accepts the offer. This is a realistic setting (though it will be ideal if the authors can move some of the discussion on real-world examples to the main body of the paper). \n2. The authors proposed an adaptive policy for their setting and provided detailed discussion about each stage. They also provided theoretical results for the settings of both known and unknown context distributions and the proof appears sound. \n\nWeaknesses:\n1. The finiteness of the context set does appear limited. The authors mentioned that this is required only for Phase 2 of the algorithm and I wonder what are the theoretical results when one discretizes a continuous context set $\\mathcal{X}$ for Phase 2? \n2. Some numerical studies should be included in the main body of the paper. In particular, an illustration of the regret bound can be helpful in illustrating the dependency of the algorithm on $T$.\n3. Including the lower bound results in the paper could be helpful, even though the authors referred to other works. Currently it is unclear how good the regret upper bounds are. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept