Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewards
Authors: Ashwinkumar Badanidiyuru
Zhe Feng
Tianxi Li
Haifeng Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Learning to Bid
Incrementality Bidding
Reinforcement Learning

Abstract: Incrementality, which  measures the causal effect of showing an ad to a potential customer (e.g. a user in an internet platform) versus not, is a central object for advertisers in online advertising platforms. This paper  investigates the problem of how an advertiser can learn to optimize the bidding sequence in an online manner \emph{without} knowing the incrementality parameters in advance. We formulate the offline version of this problem as a specially structured episodic Markov Decision Process (MDP) and then, for its online learning counterpart,  propose a novel reinforcement learning (RL) algorithm with regret at most $\widetilde{O}(H^2\sqrt{T})$, which depends on the number of rounds $H$ and number of episodes $T$, but does not depend on the number of actions (i.e., possible bids). A fundamental difference between our learning problem from standard RL problems is that the realized reward feedback from conversion incrementality is \emph{mixed} and \emph{delayed}. To handle this difficulty we propose and analyze a novel pairwise moment-matching algorithm to learn the conversion incrementality, which we believe is of independent  interest.

Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewards
Ashwinkumar Badanidiyuru Google astwinkcumarbbv@@oogll. com
Zhe Feng Google Research googgg com
Tianxi Li
University of Virginia tianxili@vizginia edu
Haifeng Xu
University of Chicago
haiffengxu@uchicago.cc
Abstract
Incrementality, which measures the causal effect of showing an ad to a potential customer (e.g. a user in an internet platform) versus not, is central object for advertisers in online advertising platforms. This paper investigates the problem of how an advertiser can learn to optimize the bidding sequence in an online manner without knowing the incrementality parameters advance. We formulate the offline specially structured episodic Markov Decision Process
version of this problem (MDP) and then, for its online learning counterpart, propose a novel reinforcement learning (RI.) algorithm with regret at most VTH2VT), which depends on the number of rounds H and number of episodes T, but does not depend the number of actions (i.e., possible bids). A fundamental difference between our learning problem from standard RL problems is that the realized reward feedback from conversion incrementality is mixed and delayed. To handle this difficulty we propose and analyze a novel pairwise moment-matching algorithm to learn the conversion incrementality, which we believe is of independent interest.
Introduction
Nowadays, online advertising systems (e.g. Google, Facebook, and Amazon) have demonstrated their significant power for connecting advertisers and potential customers. Moreover, automated auctions are widely used in online ad platforms for matching advertisers and users, and for price discovery 223 g. Many advertisers are bidding repeatedly online to show ads to users. Therefore, an important problem for advertisers is to design efficient bidding algorithms to maximize their accumulated utility. Moreover, this also an important problem for online platforms, because auto-bidding - i.e., the advertisers authorize ad platforms to bid on behalf of them has become prevalent in online advertising
A growing body of work is investigating the problem of learning to bid in repeated auctions 25 10
1 12
[9]. However, existing work assumes that the value of showing ads to user (after winning the auction) comes from either an oblivious adversary or is sampled from an underlying fixed distribution. Unfortunately, this ignored the crucial causal effect of showing an to a potential customer and cannot capture how much an ad will change the user's conversion rate, which should have been what the advertiser bidding for. For example, a user sees the same ad multiple times in very short time period, then the conversion of this user may not be much better in fact, may
A conversion represents desired interaction between the user and advertiser, e.g., download the App buy the products from the advertiser.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

even be worse - than that of showing ads to this user only once within this period. Such multiple redundant ad placements are commonly known in reality be cost-inefficient for the advertisers, but cannot be captured by previous models of learning to bid with either random or completely adversarial ad values. To overcome the drawbacks of previous approaches in measuring the value of an advertising opportunity, a recent visionary whitepaper by Lewis and Wong (18) introduced the important notion of incrementality, which properly quantifies this causal effect of showing an ad to the user validated by massive real data from leading advertising platforms. However, [8] focuses on the offline learning problem of estimating incrementality from past data. The online learning of incrementality parameters, well as how to utilize it to optimize bidding sequences, has not been studied; this is what we embark on in this paper.
In Figure ! we visualize the conversion incrementality caused by three ad placements and used the colored areas to capture the incrementality caused by each ad placement. There are two main challenges in learning incrementality, i.e., delayed and mixed conversion feedback. First, the conver- sion may not happen immediately when the ad is shown to the user, even though the conversation rate may quickly peak after showing the ad (see Ad 2 and Ad 3 in Figure I. Such delayed conversion feedback in online advertising is studied only recently in offline setups, 66 21 22 31. Second, when a conversion happens, it is mixed effect from multiple previous ad placements before the conversion. In this work, we formulate incre- mentality bidding as episodic Markov Decision Process (MDP), where each episode represents an interaction with single user drawn from population of the same charac- teristics (and thus assumed to have the same incrementality parameters). The advertiser wants to learn the incremental- ity and optimize bidding for this user population. This is hardly possible in general given the two difficulties above as well as the potentially complex dependence of incre- mentality on the entire history. We thus make a Markovian assumption that the conversion incrementality of an ad placement at each round only depends on its last ad impression shown before this round. In addition, we in- troduce the heterogeneous Poisson process to formally capture the mixed and delayed rewards feedback in the conversion process (Poisson process is also mentioned by Lewis and Wong [8], but only qualitatively).
12
AF
Ad
A13
3.0-
e.g..
02-
0.0
Time
Figure I: Toy example: visualization of the conversion incrementality triggered by three ad impressions. The rate of conversion in- crementality of each ad follows Gamma 2). Exp(0.5) (shifted right by 3) and Exp(I) (shifted right by 8) distributions, respectively.
To optimize the sequential strategy for incrementality bidding in online learning, there is an intrinsic exploration versus exploitation tradeoff. On one hand, the advertiser needs bid high in order to win the ad placement opportunity and also to learn the conversion incrementality. On the other hand, she also doesn't 't want to bid too high such that her conversion reward may be less than her cost, i.c., the payment due to winning the auction. This observation naturally motivates our design of a model-based RL algorithm that adopts a UCB-style approach to balance exploration and exploitation
Our Results and Contributions. The main contributions of this paper are two folds: (1) we intro- duce an episodic MDP, coupled with a heterogeneous Poisson process, to capture the incrementality bidding problem; (2) we design a novel reinforcement learning algorithm for the advertiser to optimize her sequential bidding while learning incrementality simultaneously. Technically, our main result is the design of an RL algorithm for incrementality bidding that provably has regret at most O(22V)) and is near-optimal in terms of the dependence on T. Conceptually, our result demonstrates the possibility of designing highly-efficient RL algorithms for incrementality bidding despite its various challenges. To the best of our knowledge, this is the first work that analyzes incrementality bidding in an online setting theoretically.
Notably, our regret bound is independent of the number of possible bids (equiv. actions in standard RL). This is due to the special structures of the incrementality bidding problem, which has a small effective action space. Specifically, there are only two effective outcomes for an advertiser-winning
We remark that designing model-free no-regret RL algorithms is an intriguing open problem. Due to delayed and mixed reward feedback, it appears challenging to use algorithms like Q-learning in the incrementality bidding problem.
2

---

or losing- that eventually affect the state transitions. The conversion incrementality only happens if the learner wins. Finally, we remark that our results are independent of auction formats and all our results hold for second price auctions, first price auctions, and other standard single-item auction formats. To achieve this regret bound, we provide a novel regret decomposition to incorporate with the convergence rate analysis of our novel parameter estimation method.
One key technical novelty of this paper is a new parameter estimation method, Pairwise Moment- Matching (P PMMM) algorithm, that can estimate reward parameters under mixed and delayed conversion feedback. In this type of models with mixture effects, standard likelihood- beashoodd-bsedd model fitting procedure would lead to solving non-convex optimization problems resulting in difficulties for computation and analysis, as well as unpleasant gap between theory and algorithm [15]. However, P AMM estimates incrementality parameters in an online and computationally efficient manner without the need of solving nonconvex optimization problems. The estimators are provably consistent with a nearly-optimal convergence rate. We believe this novel technique will be of independent interest for other online learning problems with delayed and mixed reward feedback.
Related Work. As mentioned in the introduction, our work is highly inspired by the visionary work of Lewis and Wong [88, who firstly proposed incrementality bidding (and attribution). However, the method proposed in [18] lacks theoretical guarantee and cannot be adapted to online learning
settings. Our work is generally related to the learning to bid literature [25] 10 13 12 4 However, all these existing papers usually model this problem (contextual) bandits and don't 't consider the
causal effect of showing an ad to the user captured by incrementality (i.e., the additional value or conversion given the ad placements in previous rounds), which causes the main difference from this paper. Specifically, in all these previous works [25 10 S
L 12 4 the reward for an ad impression realized immediately at the current round whereas, in our model, any ad impression only increases the rate of conversions, which however may be realized in any later time following a Poisson process and, more importantly, is mixed with the incrementality of other ad impressions. We believe our model captures the effect of advertising more realistically. Some other less related works include delayed feedback in other different RL problems, which have only been studied recently [I7 [4]. We handle delayed conversion feedback through a parametric inhomogeneous Poisson process, which is different from these works.
2 The Sequential Optimization Problem of Incrementality Bidding (IB)
This paper adopts the perspective of an online advertiser, and studies the advertiser's learning problem of optimizing her sequential bidding policy. Next, we will first describe the background and motivation of this problem and then introduce the formal model.
We consider the sequential bidding problem for single advertiser i... the decision maker or learner- - in online advertising systems. The advertiser participates online advertising auctions in order to win opportunities for showing ads Internet users and, ultimately, gain conversions (i.e., purchases of products or services) from Internet users. The interaction between the advertiser and Internet users happens in an episodic manner; each episode corresponds to one Internet user. Specifically, each episode has H rounds, during which the advertiser (learner) will interact with the user of that episode for at most H times. Notably, H is finite number since (1) an interaction happens only when a user visits a particular webpage and (2) if an advertiser already bids for user for sufficiently many times and still was not able to get conversion, the advertiser typically will cease to advertise to this user (at least for some time). We use T to denote the total number of episodes To focus on the fundamentals of the problem, this work studies the basic setup with i.i.d. episodes (i.e., Internet users) that occur sequentially and evenly distributed If rounds at integer time point 1, 2. ... H within each episode. Interesting future directions include examining the more general settings in which each Internet user in the corresponding episode may depend on a context feature, the time points of the H rounds are different for different episodes, and the episodes may happen in batch (i.c., multiple similar users arrive at the same time).
Conversion incrementality. Central to our bidding optimization problem is the notion of "comvr-- sion incrementality" [8]. Intuitively, incrementality captures how much increase the advertiser's ads can boost the user's conversion rates. Notably, the reward of showing an ad is the incrementality
In reality, H is usually at the scale of around 50 (the number of times an advertiser would like to interact with a user). whereas the number of similar Internet users (i.e., episodes) 1' is of the scale of millions.
3

---

while not the conversion rate itself. This is very different from the previous learning to bid literature since they assume the conversion (or value) comes independently with the previous ad placements (i.e., from oblivious adversary or an underlying distribution). This is why previous studies almost all use the (contextual) bandits setup, while not MDP.
We now formally describe the model of incrementality bidding, following (18). Within any episode t E [2]. each round h E [#]] has an incrementality rate function (T; An, 0n) for any continuous time r E [h, 0o) which is the "density" of the conservation incrementality at any time E [h, oo). Intuitively, di (T: A,, 0/)) models how much an ad shown round boosts the conversion rate density at any future time T > h. Notably, the integral of this rate function di (T; AA. 0x) does not need to be as there may be many conversions per user impression. We assume that di, (T; Ar. 9%) depends on an to-be-learnt unknown parameter 9, and additionally a subset A,, C [h]] containing all previous rounds, at which our learner's s ads were shown to the corresponding user of this episode. Let 0 (01, 02 (zz) denote all the to-be-learnt parameters.
is generally intractable to learn an arbitrary rate function di An: Next, we introduce a natural parameterized family of functions for da (T; AN, 0n), with to-be-learnt parameters, in order to capture advertising applications. First, we have the following assumption for dn,
Assumption (Markovian Incrementality). da is only affected by the learner's last ad impression before h, rather than the entire history Ad, of the learner's 's advertising performance. Formally, at current round h, suppose the last round at which the learner's ad was shown h then the incrementality function (T; An, 8r) = (T; (.))).
The above assumption is built upon the sense that the users normally weighs more on the recently visited ads and are generally memory-less. Moreover, we assume each incrementality rate function dn(T; e, 0%) for any h E [H]] has the following parametric form:
Bn (()())ee- dn e, 0n) = {0
Vr E (0),
(I)
otherwise,
in which 0, = ((x(e)) u } contain all the parameters at round h of the given IB instance. In other words, the incrementality rate is a re-scaled exponential distribution and the re-scaling is due to fact that each user impression may lead to more (or less) than one user conversions. In our RL formulation of the problem, these are all the to-be-learnt parameters. The Markovian property is reflected in 8 (f). Note that fT da (T; E, )dr= B% (f). So B, (f) can be viewed as the expected number of conversions triggered by the ad impression at round h. given the last time for showing the
learner's ad is l rounds before. In this paper, we assume Vh, l, cp < 8p (e) I C3 for some positive
constants C3 and Cg. Without loss of generality, we assume C3 1
Remark 1. The assumption of expomential decaying rate in Equation TT is not necessary for our approach to work, and primarily for the exposition. Our techniques can be applied to most parametric family of rate functions such as truncated exponential, logistic, Beta, and Gamma distributions [181, as long as the CDF function is invertible.
From conversion incrementality to realized conversions. While incrementality captures how the rate of conversions increases, it did not model how conversions are realized in reality. This would be important when we introduce our reinforcement learning problem - after all, in real applications, the only observations our learner (an advertiser) can see are the realized conversions at different time, rather than the continuous conversion rate function. To capture this, we adopt the standard inhomogeneous Poisson process [8] More specifically, let the ordered subset W [W1, u2.... ,,, S [H] denote the rounds at which the learning agent wins within the episode (for convenience, let we = 0). Then the conversions arrive according a Poisson process with a time-varying rate r(7). defined as follows
assumption that conversions are realized based on a continuous-timm
'It is possible to have more features of the last ad impressions (instead of just the time gap ?) in the state and our analysis will still go through as long as the number of states is still finite. For theoretical simplicity, in this paper, we focus on the case that the state is just l.
"I this paper, we assume (e) to strictly positive, however, the conversion incrementality can be negative in practice, e.g., keep seeing the same ads very often may decrease the conversion from the users. Indeed, our results still hold when allowing negative G (f). as long as Bi and the time-varying rate r(r) are bounded from zero. See the discussion in the proof of Theorem Ii Appendix B

---

r(r: w. 0)
Σ dr (T; w,-1, 0w;);
Vr E [0, H].
(2)
More details of inhomogeneous Poisson process is deferred to Appendix A.
The optimization problem bidding for incrementality. While the above incrementality mod- cling of user conversions originates from the influential idea of [18), our following formulation of the optimal bidding problem under their user modeling new to the best of our knowledge, and so its reinforcement learning solutions presented in this paper for addressing the situations with unknown environment parameters.
Following current practice, we assume online ad opportunities are sold to advertisers by auctions, and our learner is one of these advertisers. For the ease of presentation, we restrict our descriptions to the widely adopted second price auctions; however, all our results can be easily adapted to any single-item auction format without entry fees (e.g.. the first price auction). Let B denote the bidding space, which may be continuous, Notably, 0 E I, meaning the bidder/learner can choose to not participate this round's s auction. Moreover, denote the distribution of the highest bid among other bidders (HOB) at round h as Fj (the cumulative distribution function or CDF). For any episode t (corresponding to an Internet user), at each round h, the learner submits bid b B and wins the opportunity to show its ad to the user if her bid is the highest. Let Ph : B + Rzo represents the average (conditional) payment function of the learner she wins. Let e denote the learner's 's average value per conversion, which is known to the learner. We assume is bounded, w.l.o.g., within [0, 1]. Moreover, we can assume B S [0, 1]. Then for any bid & B, given the current state E, the immediate expected utility of the learner at round (for any episode) is.
un (b: 0,) = Fb((b) ST. udu(T; 2,0%)dr ph pb(b) (b) (3x()) (b))
(3)
The astute reader may already find the expected utility function of the learner doesn't 't depend on parameter λn- This raises the question why we still need know the function class of d, (i.e., exponential function) and why we still need to learn parameter A. As we will see later in Section 4 due the complexity of inhomogeneous Poisson process, it is crucial to know the function class of d, and we have to learn parameters À first in order to get a good estimator to 3 afterwards. In second price auctions, pa (b) is the expected second highest bid conditioning on winning, which can be computed as follows
(b) /
1
(b. J. Fu(l)dd.
Ph (b)
vdF =
As mentioned above, our techniques apply to other auction formats well. For example, in first price auctions, Ph (b) has an even simpler format.
The Optimal Policy of IB and its Reinforcement Learning
While it may appear obvious the first glance, explain how IB can be reduced finite- horizon Markov Decision Process (MDP) in this section and then introduce the corresponding reinforcement learning problem for TB when the environment parameters namely, the incremen- tality parameters 0, (1) U (Ax) } and HOB distribution F, unknown.
3.1 The Offline Optimal Policy of IB
now formulate the optimal planning problem for the IB problem in offline version, i.e., when all environment parameters are known. We start by defining what is a policy for the IB problem.
Definition (Policy). A deterministic (dynamic) policy is sequence of mappings т = (TI..... TH) in which mapping Fi [h] + maps any & E [h] to bid. Here, é chosen such that the last winning of the learner happens at round h
Note that it suffice to consider policies that depend only on the number of consecutive loses & due to the Markovian property of incrementality as mentioned above.
5

---

The MDP re-formulation of IB. The MDP will have H states, with state E [H] denoting that the last winning round rounds ago from current round. The action set is B, containing all possible bids. Both the state and action space are the same for each round h, however the transition and rewards are different for different rounds. Specifically, at round h. the transition probability from state I and action b is to enter state 1 at round h + 1 with probability Fn, (b), i.e., the probability of winning the auction, and to enter state & + with probability 1 (b). The immediate reward from transitioning is TΛ (4, b, 1) Jn (T;) E, 0h )dT (b( for tuple (e, 6, 1) (the winning case) and r (,, b, + 1) = 0 for tuple (e, b, É + 1) (the losing case). This last statement depends crucially on the independence of the incrementality across different winning rounds. The initial state of this MDP is always the lowest state, i..., 6 I.
Consequently, the optimal policy in IB can be computed via standard dynamic programming. Specifically, when the action set B has finite support (i.e., the advertiser has finite bids to choose from), the following proposition follows from the fact that MDPs with finite states, actions and horizons can be solved efficiently by dynamic programming (I)
The offline optimal policy for any IB instance can be computed by dynamic programming in (Bly H) time.
Proposition 1.
We remark that even when B C R. is continuous with upper bound B, an e-optimal policy can be computed in poly( _ H) time since the reward function is continuous in b E B and an e-optimal action can be found at each round during the backward induction by discretizing the entire line of bid into e segments. This e-optimal action choices lead to an approximately optimal policy [T].
3.2 Reinforcement Learning of IB with Unknown Parameters
We now turn to the much more interesting and realistic situation of the online learning, particularly reinforcement learning, of the optimal IB solution. While we have shown that the offline version of Te problem can be viewed as an MDP, its online counterpart cannot be similarly tackled by standard reinforcement learning techniques due to the following novel challenges specific to the TB problem:
Challenge I: mixed reward feedback. Any realized learner reward, arising from a user conver- sion, is naturally mixture effect of all the previous winning rounds, as modeled by the accumulated incrementality rate in Equation (2). This is in contrast to standard MDP, in which the reward of this round is realized independently, conditioned on its state. It is not difficult to see that the loglikelihood of our mixed rewards model is non-convex, which renders the standard RL techniques based on maximum likelihood estimation for reward parameter estimation not applicable.
Challenge 2: delayed reward realizations. Moreover, the conversions in the IB problem follow a intricate Poisson process with heterogeneous rate and thus has delays. That is, showing an ad may not immediately lead to conversions. Such lack of immediate reward feedback renders standard Q-learning style of algorithms not easily applicable to our setup since we cannot use immediate reward feedback to update Q-functions any more.
To overcome the first challenges above, we design novel parameter estimation methods based on moment matching to estimate 0, 's from mixed reward feedback, and prove its convergence rate in Section To address the second challenge above, we design a madel-based no-regret RL algorithm that integrate the parameter estimation above, and analyze its regret in Section 5
Learner's Observations. We now formalize our learning setup. We consider the episodic online learning setting, where the learner does not know the IB parameters 0; = {B, (0)2 U {A, for each h E [H] neither the distribution of the highest other bid (HOB) F,, Following the typical practice, we assume that the learner can always observe the realized HOB at any round h in each episode, denoted as m/, regardless whether the learner wins or loses. This is usually termed as "fill information feedback" in the literature of learning to bid Moreover, the learner also knows the set of its winning rounds denoted by an ordered set W C [H] as well as the set of the time stamps of all conversions Ct (a subset of R) from each episode.
'Thi assumption is practical. First, online ad systems, the manual bidder can always get this feedback (also known as "minimum bid to win") no matter she wins or not, e.g., in Google Ad exchange platform 7 Second, for the auto bidding algorithms, they are designed by the platform and it does know the realized bids at each round.

---

In the online learning setting, the learner aims to design a bidding algorithm A to minimize the expected regret defined in the following,
T R(T'; 8 F)
Regret(T) = T OPT(O, F)
Σ
(5)
(
where OPT(0, F) represents the optimal expected utility achieved by the learner at each episode if the ground-truie truth parameters 0 and F are given (recall that the MDP always starts from state = 1). E represents the expectation over the randomness of algorithm A, and n' is the policy in the tt episode generated by the algorithm A. Here, we slightly abuse the notation denote R(T; 0, F) as the expected utility achieved by policy x for estimated parameters 1 and F.
H Σ
R(T; OTz;,FF) (6 Ln Eni
(Bu(ew) (.(())---(((m((()))) Fa(Talle))
(6)
where P' represents the joint distribution of states (fi, E..) induced by bidding policy a in the MDP formulated in Section 3.1 with estimated parameters @ and F. Note, for second price auctions, pr. (b)
(b) Jo Fu(U)do (replacing F by F in Eq. (4)). In other words, R(z; 0, F) captures the expected total reward in the MDP when the reward function and transition probability parameterized by 0 and F.
Remark on non-degeneracy assumption on the learner. We assume that the learner's bidding space B is upper bounded away from extreme bids. More formally, there exists small constant co such that F(b) S Co for any E B (i.e., no feasible bids guarantee sure winning). We call such learner no-bounded. There are two reasons for making this assumption. The first is a technical reason: makes sure that the learner always has at least cp probability to lose, which turns out to be crucial to estimate the incrementality parameters, as we do in the next section. This is because incrementality captures the difference between winning and losing. If the probability of losing is extremely small, there is generally afford small winning probabilities). The second reason for assuming co co-bounded learners is realistic motivations since in auto bidding practice, bidders typically explicitly specify upper bounds of their bids, under which the bidder's s winning probability almost always capped strictly below 1.
way to accurately estimate their differences (though it turns out that we
4 Incrementality Parameter Estimation via Pairwise Comnent-Matching
Though maximum likelihood estimation (MLE) is a natural and straightforward parameter estimation strategy, it comes with many drawbacks we want avoid. First, the MLE criterion under our model of mixed rewards is not a convex problem and optimizing the likelihood would suffer from multiple local optimal or saddle point issues. The theoretical claims may not match the true algorithm performance. Secondly, the computational efficiency of the MLE problem is worse than what we will introduce next, which makes less attractive in large-scale problems. Our algorithm carefully tailored for estimating parameters of our model, given the observed trajectories of first t rounds. Specifically, the trajectory of each episode i [T] can be described by two sequences: (1) the time stamps C of realized conversions within time [0, co]: and (2) the ordered subset W = (wf,
w } S [H] which contains all the winning steps within this episode (for convenience, we extend the above notations and by letting w/- = 0, w H + 1), where ni denotes the number of winning events (showing the ad) in the episode 1. Equivalently, we can also represent W by a H-length binary sequence Ç² E {0, 1 such that <t = and only if episode t wins at time h. Notably, both C' and W may be empty if there is conversion nor winning. Our estimation is based online algorithm: from a high-level perspective, it progresses by time intervals [h, h + 1) sequentially to produce estimates of parameters On, by pairwisely coupling episodes and matching statistical moments. Therefore, we call it the pairwise moment- moment-matching (PAMM) algorithm, described in Algorithm [
The PAM algorithm is specifically designed for estimating parameters from mixed reward signals, each drawn from a Poisson process. Its primary advantage lies in its online nature, guaranteed convergence and efficient computation. This achieved by the following observations. First, all information about reward scale parameter Bh and reward variance parameter X is intrinsically
7

---

Algorithm 1 Pairwise Moment-Matching (PAMM) Algorithm
1: Input: A sample of episodes ""), E [T] up to time h 1).
2: Output: the estimates of parameter 8, denoted by (O), ve E [m]
3: For any i, define N% and N to be the number of conversions of episode within (hr, h + 2) and
[h + 2.2 h + 1), respectively.
4: for For each given h of interest, do
5: Define n, to be the total number of episodes with winning bid at time h. Define v {0, (0 be the set of all possible winning sequences hefore time h. For each winning sequence Φi E V/. define
ELET: Sh 0,5; 1, و Moreover, define no (0t) n²p = |Abl. and 7s (~= ((n')))))))) of na and n'z.
22 - LLET ch 1,, - 1o,1ijshh-1
1-1 as the harmonic mean
6: For each & E (0,1)--1
X ns
Σ No.
Σ No and
Yp
1
Σ
N
Σ
N
(7)
n tEAs
Tue tE
tEQ
Φ tEAs
7: For each @ E 10-1 define
ns
ar = Σ Lo nay
(8)
and set
Th = Σ
a X and 7m = Σ asy
(9)
VEV&
Estimate An by
An 2(log p. fj.)
(10)
8:
for 1 1 h do
9:
Define the fth slice of v Vn = ( E Vn on-e 1, db 0, 1<
< h 1).
For each winning sequence & E Vn,,, still define N₂ and Y following 7
10: Define
The =
Σ Xs and The = E ase DEVE
(11)
SEVAN
where and is defined following (8) after replacing vn by Vnne-- 11: Estimate B% (f) by
ph.t
(e) = Jine iiht
(12)
end for 13: end for
reflected only in the "differential behaviors" for those episodes which had conversions vs. those episodes which did not have conversions at time step h, conditioning their matched history before time h as defined in the sample matching step. This is due to the Markovian property for the Poisson process, and our moment matching is hinged into this property. Second, for the Poisson distribution (as a member of the exponential family), the method of matching the canonical parameter (mean) gives an equivalent estimator as the MLE in a single Poisson case (but not necessary in our more complicated model). These two ideas motivate our procedure, by matching the episodes, we precisely locate the needed Poisson signals and solve the estimators by the mean value structure. Because of this, the P PAMM can recover the parameters with essentially optimal rate (in the online sense), even though we do not use the full likelihood that would result in non-convexity and an unpleasant gap between the theory and the practical algorithm.
8

---

In practice, termss in (10) and (12) may result in =oo, but such an event only happens with probability tending zero (which is why Theorem [ has high probability guarantees). Standard numerical protection can be introduced in practice which will not change our theoretical claims. Moreover, from the implementation perspective, one start from the first interval [0, 1) and move forward by running P PAMM for each [h, h + 1). In each new interval, the matching step can be directly carried over by further refining the matching of the previous interval. As a result, the whole estimation procedure only involves one sweep of the data without the commuting back and forth between different time intervals. We have introduced the PAMM algorithm in the current form as a self-contained algorithm for estimating the inhomogeneous Poisson process, which can be of independent interest. However, the steps of PAMM can be easily embedded in our online learning algorithm presented in Section without losing its validity.
Theorem (Estimation Errors with Finite Samples), Suppose there exist positive constants Cx and Ca, such that Cx & X $1/0, and 3n (f) 2 cs for any h and f, and the learner is co-bounded, Also assume that the expected number of total conversions for each episode within each interval [h. h + 1) is always bounded above by constant Cr. Let nh be the total number of winning episodes at time h and n4 (2) be the total number of winning episodes time h with state f. Let Xn. and Br be the estimators obtained from Algorithm 7 Then, for any 0 S h < H and any S < h. we have the following guarantees for PAMM;
log 1/8 1. For any 4e- Comb < e < 1 1/2, with probability at least 1 - (An. Anl S C1
ns
2. For any 4e 4c-Co))( < 5 < 1/2, with probability at least 1 - 5, (3x(() (n) s
C2v
Ing 1/8
[e)
In the above results, Co = -gg Ci = 4
C7 and C2 = 8V
2C
where e' is the
oz (1--0- 5 /2pee
c'ca
Bernstein constant (see Lemma 7.
5 The Full RL Algorithm and its Regret Analysis
We propose reinforcement learning algorithm by incorporating our parameter estimation method PAMM and the optimal offline planning for the MDP mentioned in Section 3 to learn parameters 0, as well as the conversion incrementality and decide the bid at each round.
Our algorithm adopts UCB-style procedure to handle the exploration-explotttion tradcoff in incrementality bidding, which is summarized in Algorithm 2 The algorithm adopts pure explorations in several beginning episodes so that we can get enough ad impressions for each state in every round. These pure explorations are achieved by setting bids appropriately such that we can enforce to explore one (h, E) pair in each exploration (Line in Algorithm 2 In particular, we need O (5g( explorations to make sure n, (f) 2 with high probability, where Co is a positive constant explicitly defined in Theorem Recall (f) the total number of winnings at round h given the state is l.
(T/(4 so that we can provide a regret bound Co
After pure explorations, at each episode f, we use the PAMM algorithm to get an estimator Ø (resp. 3') given the observed ad impression time W and the conversion time C up to episode t. In can get
addition, we can estimate F casily using empirical distribution function and this implies uniform convergence for the transition probabilities for any action E B given any state f. Then we apply UCB-style RL algorithm with estimated parameters Ø² to decide the policy in next episode. Specifically, given the estimated 3" Vh E [H], we construct confidence region centered at & and we do offline planning for the optimism in the confidence region to get the policy in next episode.
Remark (Computational Efficiency of Alg. 2p. An important observation is that R(;;; O, F) is non-decreasing in (e) (see Eq. ( for any fixed π and F. Therefore, the optimal choice of B in
'hhen f - h. setting 66 = means "fake" winning at round 0 and it is consistent with our assumption that initial state 4 1.

---

Algorithm 2 Online Bidding algorithm
1: Input: Given 6, H, T. and Co: C2 defined in Theorem
Set ru (f) = 0, vh E [M] M E [M]
2: for Episode 1,
3: if 3h, e, (e) < 1og(4T/6)
then
Co
4: Set bids, s.t. bj. b h-t
1, and b/. 0, wh' * h, L.
[Pure Exploration]
5: else
6: For [H] observe state e and set the bid bj, म. (4))
7:
end if
8:
Observe the set of the ad impression time W the set of the conversion time C*, and the vector
HOBs m' = {m] A/ E [H]]
9:
Update ny (1) given W
Update Ø = (3% (f), Ve E [b]] through PAMM method (Algorithm) D given (W*, C²).
10:
11:
Update (b) = E. Im < <b..bb. vb.
if vh [H] E [h), ( then
Construct confidence region CR for parameters 0, s.t.
CR = - (3,) : vh (3a(1) - (a) = SCy (E)
log(T/6)
14:
Compute the bidding policy nh for (t + 1)th episode, s.t.,
1+1 argmax max R(T; 0, Fi) ecR
(13)
end if
16: end for
Eq. 139 in Algorithm is set Bu(() ()(() / Voee and x'+1 will be the optimal policy (E
1og(118)
for the MDP parameterized by Fsr and B, and thus can be computed efficiently (Proposition) 7
Regret Analysis. Combining the convergence rate analysis from Theorem and novel regret decomposition technique (necessary for our particular setup), we are able to prove the regret guarantee of our online bidding algorithm. Our regret bound is summarized in Theorem 2 the proof of which is deferred to Appendix C.
Theorem 2. Under Assumption 7 for any fixed 6, we have with probability least 8, the regret achieved by our Algorithm is bounded by
Regret(T (T) S O(H (8T + H² log(T/8)).
6 Discussions and Future Work
Our RL algorithm lies in the model-based RL algorithm literature. The best known regret bound for model-based RL is Ō SAT) (2), where S is the number of states and A is the number of actions. Since S H in our case, our regret bound is slightly worse from the best possible by an factor of VH, but is independent of the number of actions. This may be due to the complexity of our model. Whether we can close this gap is interesting open question. As mentioned earlier, our algorithm can be extended handle continuous bidding space and our regret analysis still works. In this paper, we assume the conversion rate function d, doesn't depend on the shown ads of other bidders but only on the learner's last ad impression. Relaxing these assumptions may result very interesting future work.
Acknowledgements
Li is supported part by the NSF grant DMS-2015298 and 3Caverliers award from the University of Virginia. H. Xu is supported in part by an ARO award 19112N223-100000.
10

---

References
[1] A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun. Reinforcement learning: Theory and algorithms. Working Draft, 2019.
[2] M. G. Azar, Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 263-272. PMLR, 06-11 Aug 2017.
[3] A. Badanidiyuru, A. Evdokimov, V. Krishnan, P. Li, W. Vonnegut, and J. Wang. Handling many conversions per click in modeling delayed feedback. AdKDD, 2021.
[4] A. Badanidiyuru, Z. Feng, and G. Guruganesh. Learning to bid in contextual first price auctions. CoRR, abs/2109.03 2021.
[5] S. Balseiro, N. Golrezaci, M. Mahdian, V. Mirrokni, and J. Schneider. Contextual bandits with cross-learning. In Advances in Neural Information Processing Systems 32, pages 9679-9688. 2019.
[6] O. Chapelle. Modeling delayed feedback in display advertising. In Proceedings of the 20th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, page 1097-1105, 2014.
[7] G. Developers. Real-time bidding, http:///wwwwerross......oooooooe guide, 2022.
[8] B. Edelman, M. Ostrovsky, and M. Schwarz. Internet advertising and the generalized second- price auction: Selling billions of dollars worth of keywords. American Economic Review, 97(1): 242-259, March 2007.
[9] Facebook. About bid strategies. (6955177444211000222999555666662
http://wwwffeccooooo.cc///ssssss
2022.
[10] Z. Feng, C. Podimata, and V. Syrgkanis. Learning to bid without knowing your value. In Proceedings of the 2018 ACM Conference on Economics and Computation, page 505-522, 2018.
[11] Google. Autobidding support page. swer/2979071, 2022.
http://sspporreooog....o///ooogp an-
[12] Y. Han, Z. Zhou, A. Flores, E. Ordentlich, and T. Weissman, Learning to bid optimally and efficiently in adversarial first-price auctions. CoRR, abs/2007.04568, 2020.
[13] Y. Han, Z. Zhou, and T. Weissman. Optimal no-regret learning in repeated first-price auctions. CoRR, abs/2003.09795, 2020.
[14] B. Howson, C. Pike-Burke, and S. Filippi. Delayed feedback in episodic reinforcement learning. arXiv preprint arXiv:2111.07615 2021.
[15] Jain and P. Kar. Non-convex optimization for machine learnin.. arXiv: 171111722777777 2017.
arXiv preprint
[16] M. Kearns and S. Singh. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49:209-225 232, 2002.
[17] Lancewicki, A. Rosenberg, and Y. Mansour, Learning adversarial markov decision processes with delayed feedback. In Proceedings of the AAAI Conference Artificial Intelligence, 2022.
[18] R. A. Lewis and J. Wong. Incrementality bidding & attribution. Available on SSRN, 2018.
[19] T. Nedelec, C. Calauzênes, N. E. Karoui, and V. Perchet. Learning in repeated auctions. Founclations and Trendso in Machine Learning, 15(3):176-34,, 2022, ISSN 1935-8237. doi: 10.1561/220000077.

---

[20] G. Noti and V. Syrgkanis. Bid prediction in repeated auctions with learning. In Proceedings of the Web Conference 2021, wWW '21, page 3953-3964, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127.
[21] Y. Saito, Morisihta, and S. Yasui. Dual Learning Algorithm for Delayed Conversions, page 1849-1852. 2020.
[22] Y. Su, L. Zhang, Q. Dai, B. Zhang, J. Yan, D. Wang, Y. Bao, S. Xu, Y. He, and W. Yan. An attention- based model for conversion rate prediction with delayed feedback via post-click calibration. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IICAI'20, 2021.
[23] H. R. Varian, Position auction. International Journal of Industrial Organization, 2006.
[24] R. Vershynin. High-dimenssionll probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[25] J. Weed, V. Perchet, and P. Rigollet. Online learning in repeated auctions. In 29th Annual Conference on Learning Theory, pages 1562-1583, 2016.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review
Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "kUL5buDIRqp", "reviewer": "Reviewer_Bj5w", "summary": "The authors study a practically-motivated learning-to-bid problem in the online setting, motivated by the importance of incrementality bidding. To capture the causal effects of incrementality, the authors pose and analyze an MDP formulation, while past work operated under simplifying assumptions that yielded bandit formulations. Combining the MDP formulation with a novel parameter estimation algorithm, the authors establish regret bounds on the overall procedure.", "questions": "As far as I can tell the work is theoretically sound and is a good contribution. I’m not intimately familiar nor up-to-date on the literature on bidding/autobidding/what is the state of the art with respect to these things in practice. Thus, when reading the paper, some of the important terms like “conversion”, “incrementality bidding”, and so on were unfamiliar to me and I had to do some digging to get some understanding. If somehow the bidding landscape could be set up in a more reader-friendly way, that would be good. But, I think for someone who is more familiar with this field the paper would be accessible and well written.\n\nAre there experiments that the authors could include that would help the theory? I understand that this is a theory paper that complements Lewis and Wong, but perhaps an experimental comparison to the results of that paper would be interesting.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work seems to fill in an important gap in the theoretical learning-to-bid literature. It relaxes assumptions that allowed previous work to study a simplified bandit formulation, and provides theoretical guarantees for an important practical setting introduced by Lewis and Wong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ao62zXSWKw5", "reviewer": "Reviewer_hB3J", "summary": "This work considers an online ad-bidding problem where the reward can be mixed and delayed. It models the problem using an MDP,  and it proposes an UCB-based RL algorithm to solve this problem with an analysis of the regret, with a moment matching routine for parameter estimation that addresses the mixed and delayed reward issue. ", "questions": "This is a very interesting work, some of the following may be suggestions for future consideration, which may or may not be necessary for the current version.\n\n- In the paper, \\beta_h(l) is assumed to be lower bounded by some positive number c_{\\beta} to indicate that any previous ad presence can only increase the conversion rate. However, in practice, sometimes this can be negative as well, for example seeing too many ads that one is not interested in can have a negative impact on certain users. Would the current framework be able to accommodate the settings where  \\beta_h(l) is negative as well?\n\n- The MDP considered currently has states related to how long time ago one user has seen this ad before. Would some other related features be helpful for the modeling too? In that case, the states will be more complicated, would the current solutions still apply, or can be easily modified? \n\n- What is the high-level idea and intuition of this moment matching algorithm? It will be helpful to explain the algorithm and intuition more clearly in the main body of the paper.\n\n- It would be great if some synthetic or even real-life experiments can be presented to further illustrate the power of this method.\n", "limitations": "Not applicable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: This is a very interesting work considering mixed and delayed rewards in an online setting, and it proposes a method to solve these challenges.\n\nWeakness: certain assumptions of the model may be a bit strong (please see the next section).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "JmMzXiyt3MO", "reviewer": "Reviewer_ia2P", "summary": "This paper proposes a new problem setting in reinforcement learning which is specialized for modeling incrementality bidding. Authors claim that this is a more realistic model compared with previous learning-to-bid models. The main difficulty in solving this RL problem is the mixed and delayed feedback, which is tackled by a novel pairwise moment-matching algorithm. The final regret bound is $\\widetilde{O}(H^2\\sqrt{T})$, which does not depends on the number of actions due to the special structure of bidding. ", "questions": "- Can authors provide some examples that satisfy auctions happening every unit time? \n", "limitations": "I am not aware of any potential negative societal impact", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First of all, I need to mention that I am not very familiar with (learning to) bidding literature so there might be some misunderstanding. \n\nStrengths:\n- The MDP model is more realistic than previous learning-to-bid models because it considers multiple bidding for one user and the delayed and mixed feedback while the authors claim that previous models only consider single bid and thus use contextual bandits as the model. \n- A new parameter estimation algorithm for delayed and mixed feedback. \n- The paper is well written and easy to follow. \n\nWeakness:\n- The model still makes some strong assumptions. First, it is assumed that auctions happen per unit time. Second, it is assumed that the dependence of incrementality function on previous actions is only on the elapsed time since the last bid was won, but I understand that this may be necessary for theoretical analysis and it does make sense to me. \n- The related work part is mainly on bidding literature and there should be more discussion on RL literature, e.g., a detailed comparison of previous works on delayed feedback in RL. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Fx_78eqZHVo", "reviewer": "Reviewer_ZwJW", "summary": "This paper first uses the MDP to formulate the sequential bidding process, and then studies the problem of optimizing the a specific sequence with unknown incrementality parameters of the bidding process. \n\nThe incrementality rate function $d_h$ is defined to enjoy the Markovian property in Assumption 1 and plays the role of \"reward\" in the MDP. The state $\\ell$ is formulated by the number of rounds between the last winning of the learner and the current round. Therefore, the transition function is defined by the distribution of the highest bid among other bidders $F_h(\\cdot)$. A special problem is that the incrementality has to be calcuated with Poisson process estimation. \n\nThe algorithm is designed based on the principle \"Optimism on the face of uncertainty\" and the concentration inequalities of the parameters of the MDP. For each episode, it selects the optimal policy associated with the parameters $(\\beta,\\lambda)$ within the confidence sets, and adjust the confidence interval accordingly. It achieves similar regret bound $O(\\sqrt{T})$ without knowing the environments beforehand. ", "questions": "By the work of Simchowitz and Jameison (2019) we know that it is possible to achieve $O(\\log T)$ regret for unknown transition MDP problems. Is it able to achieve some problem-dependent / gap-dependent regret bounds ? ", "limitations": "None.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am willing to reconsider my scores if the authors could provide a more detailed argument or some references for the following concerns.\n\n1. The idea of proposed algorithm is very similar to that of the UCRL2 or UCBVI and the concentration inequalities are the same, which makes the technical contribution of this work less obvious.  \n\n2. It would be better to emphasize the procedure to estimate the incrementality. It is hard to understand the usage of $X$ and $Y$ in the PAMM algorithm and the necessarity. \n\nThere are many notations in this paper and some of them are not clearly defined before using, such as $n_t$ and $k_t$ in lines 293,295. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept