Reinforcement Learning for Bandits with Continuous Actions and Large Context Spaces
Authors: Paul Duckworth
Bruno Lacerda
Katherine Vallis
Nick Hawes
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contextual bandits
Continuous actions
Image context
reinforcement learning

Abstract: We consider the challenging scenario of contextual bandits with continuous actions and large input ``context'' spaces, e.g. images. We posit that by modifying reinforcement learning (RL) algorithms for continuous control, we can outperform hand-crafted contextual bandit algorithms for continuous actions on standard benchmark datasets, i.e. vector contexts. We demonstrate that parametric policy networks outperform recently published tree-based policies in both average regret and costs on held-out samples. Furthermore, in contrast to previous work, we successfully demonstrate that RL algorithms can generalise contextual bandit problems with continuous actions to large context spaces. We obtain state-of-the-art performance using RL and significantly outperform previous methods on image contexts. Lastly, we introduce a new contextual bandits domain with multi-dimensional continuous action space and image context. 

Under review as a conference paper at ICLR 2023
REINFORCEMENT LEARNING FOR BANDITS WITH CONTINUOUS ACTIONS AND LARGE CONTEXT SPACES
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the challenging scenario of contextual bandits with continuous tions and large context spaces, e.g. images. We posit that by modifying reinforce- ment learning (RL) algorithms for continuous control, we can outperform hand- crafted contextual bandit algorithms for continuous actions on standard bench- mark datasets, i.e. vector contexts. We demonstrate that parametric policy net- works outperform recently published tree- tree-based policies (Majzoubi et al. (2200) in both average regret and costs on held-out samples. Furthermore, we successfully demonstrate that RI. algorithms can generalise contextual bandit problems with continuous actions large context spaces and provide state-of-the-a results on image contexts. Lastly, introduce dimensional continuous action space and image contexts which existing methods cannot handle.
contextual bandits domain with multi-
INTRODUCTION
We consider the challenging scenario of contextual bandits with continuous actions and large "con- text" spaces, e.g. images. This setting arises naturally when an agent is repeatedly requested to provide a single continuous action based on observing context only once. In this scenario, an agent acquires context, chooses an action from a continuous action space, and receives an imme- diate reward based on an unknown loss function. The process then repeated with a new context vector. The agent's goal is to learn how to act optimally. This is usually achieved via minimising regret across actions selected over a fixed number trials.
Our work is motivated by increasingly important application area in personalised healthcare. An agent requested to make dosing decisions based on a patient's single 3D image scan (additional scans after treatment can potentially be damaging unsolved domain; current methods fail to handle the large context space associated with 3D scans and the high-dimensional actions required. We consider this problem under the contextual bandits framework. Contextual bandits are a model for single-step decision making under uncertainty where both exploration and exploitation are required in unknown environments. They pose challenges be- yond classical supervised learning, since a ground truth label is not provided with each training sample. This scenario can also be considered as one-step reinforcement learning (RL), where no transition function is available and environment interactions are considered independent and identi- cally distributed (i.i.d).
the patient) (Jarrett et al. 2019). This is an
There are well-established methods for contextual bandit problems with small, discrete action spaces, often known as multi-armed bandits with side information. The optimal trade-off between exploration and exploitation is well studied in this setting and formal bounds on regret have been established (Gittins et al. 1989 Auer et al., 2002 Li et al. 2010 Garivier & Moulines 2011 Agar- wal et al. 2014). However, there is relatively little research into continuous action spaces. Recent works have focused on extreme classification using beee methods to sample actions from a dis- cretized action space with smoothing (Krishnamurthy et al.] 2020; Majzoubi et al.) 2020). However, developments in RL for continuous control beg the question: Can we use one-step policy gradients solve contextual bandits with continuous actions?

---

Under review as a conference paper at ICLR 2023
Real world contextual bandit problems, such those in healthcare, require solution methods to gen- eralise to large context spaces, i.c. directly from images. We posit that existing tree-based methods not well suited to this task in their current form (Krishnamurthy et al. 20200 Majzoubi et al. 2020 |Bejjani & Courtot, 2022). However, recent breakthroughs in deep RI. (Lillicrap et al. 2016 Arulkumaran al.] 2017 Schrittwieser et al. 2020; Hafner et al. (202)) have successfully demon- strated continuous control abilities directly from pixels. In this regard, neural networks have proven powerful and flexible function approximators, often employed as parametric policy and value net- works. The adoption of nonlinear function approximators often reduces any theoretical guarantees of convergence, but works well in practice. Underpinning this recent work in deep RL for continuous control from pixels is the deterministic policy gradient that can be estimated much more efficiently than the usual stochastic policy gradient (Silver al. 2014 Lillicrap et al. 2016).
The contributions of our work are as follows:
1. We modify an existing RI. algorithm for continuous control and demonstrate state-of-the- art results on contextual bandit problems with continuous actions. We evaluate on four OpenML datasets across two settings: online average regret and offline held-out costs.
2. We propose a deep contextual bandit agent that can handle image-hased context represen- tations and continuous actions. To the best of our knowledge, we are the first to tackle the challenging setting of multi-dimensiooal continuous actions and high-dimensional context space. We demonstrate state-of- performance on image contexts.
3. We propose new challenging contextual bandits domain for multi-dimensional continuous actions and image contexts. We provide this challenging domain new testbed to the community, and present initial results with our RL agent.
Our new contextual bandit domain is based on 2D game of Tanks: where two opposing tanks are situated at opposite sides of the game screen. The agent must learn control parameters to accurately fire trajectories at the enemy tank. There are three continuous action dimensions: agent z-location, turret angle and shot power. Example context images are provided in Figure
2 RELATED WORKS
A naive approach to dealing with continuous actions is to simply discretise the action space (Slivkins et al. 2019). A major limitation of this approach is the curse of dimensionality: the number of pos- sible actions increases exponentially with the number of action dimensions. This exacerbated for tasks that require fine control of actions, sation, leading to an explosion of the number of discrete actions. A simple fixed discretization of actions over the continuous space has been shown be wasteful, and has led to adaptive discretiza- tion methods, e.g. the Zooming Algorithm (Kleinberg et al. 2008 2019).
they require correspondingly finer grained discreti-
Building upon the discretisation approach. extreme classification algorithms were recently devel- oped (Krishnamurthy et al.] 2020 Majzoubi et al., 2020). These works use tree-based policies and introduce the idea of smoothing over actions in order create probability density function over the entire action space. The authors provide a immputationalll-trrcabbb algorithm for large-scale ex- periments with continuous actions. However, their optimal performance guarantees scale inversell with the bandwidth parameter (the uniform smooth region around the discretised action). The au- thors of these works provide no theoretical or empirical analysis investigating the effects of large context sizes. For these reasons, we propose an RL agent based on one-step policy gradients (Sutton) ct al. 1999) that scales well with context size to handle continuous actions and large context spaces.
Prior works have framed personalised healthcare problems as contextual bandits problems (Kallus & Zhou 2018 Rindtorff et al. 2019). A one-step actor-critic method was proposed for binary actions relating just-in-time adaptive interventions (Lei et al., 2017). However, until now, these methods have been restricted to discrete actions and small context vectors.
In contrast, the notion of deep contextual bandits has previously been introduced in (Zhu & Rigotti 2021) to handle large context spaces. The authors propose a novel sample average uncertainty method, however, it is only suitable for discrete action spaces. To the best of our knowledge, no prior works have focused on the challenging intersection of continuous actions and large context spaces, and it is an under-explored research area of particular interest in healthcare.
2

---

Under review as a conference paper at ICLR 2023
3 PRELIMINARIES
Setting and key definitions We consider the i..d contextual bandit setting with continuous ac- tions. In this setting, an agent acquires a context vector (or image) I from the context space X' by observing the environment E at timestep t. The agent chooses an action a, from a continu- ous action space A = [0, 1] and receives an immediate cost based on an unknown loss function 4 : x A + 10, 1j. The process is then repeated with new context at time + 1. Unlike the standard RI. setting. there is no transition function in the bandit setting. Existing contextual bandit literature for continuous actions make the restriction N 1, i... one-dimensional action is used. This constraint can be relaxed for our proposed RL agent, but for notational simplicity, we will restrict ourselves this case for now.
In general, an agent's ss behaviour is defined by a policy. We define a stochastic policy TT that maps from the context space X' + ((AA), where P(AA is the set of probability measures on A. A R" is a vector of n parameters, and na (a, x2) is the conditional probability density associated with the policy. We also define a deterministic policy po that maps contexts specific actions in A, similarly parameterised by 0.
For tree-based policy methods (Krishnamurthy et al.) 2020). smoothing operator is also required: AA + (A)). that maps each action a to a Uniform distribution over the interval {a' e A: la a'| 2 h aa- h, + h| n.A.
Smooth
Over T rounds, an agent accumulates a history of interactions. At round I, this experience is com- prised of tuples {(x,; az (z,, a;))
Regret: Online vs Offline One assumption in the contextual bandit setting is the availability of an oracle to provide the loss value for given context and action pair. Traditionally, an agent's 's goal is to minimise the expected cumulative regret across fixed number of T trials, which is equivalent to minimising the online loss: J - Σ (2,, a:)]. Note that this expectation is under both the policy and potentially stochastic loss function, i.e. not transition function| In this online setting, an agent has access to each trial only once and no prior knowledge of the environment is assumed, and no burn-in phases allowed.
Whilst online cumulative average regret over fixed number of trials is popular, it is also com- mon to allow a dedicated learning or planning phase followed by an execution phase to exploit the learning. In these cases, simple regret (Bubeck et al. 2011)) can also be sensible consideration. In this offline setting a dataset of experience can be collected in advance and an agent relies on multiple passes (or epochs) over the dataset of experience followed by an evaluation
a held-out portion. We know that for enough training samples offline validation will obey the Hoeffding bound, guar- anteeing that with high probability the error discrepancy between the training estimate and hold-out estimate will be small (Blum et al., 1999). Therefore, for the offline setting, we report average loss on held-out portion of the dataset.
Methods such as UCB1 are often used to minimise online cumulative regret in discrete action set- tings (Auer et al., 2002 Bubeck et al. 2012), or identify the best action in a best-arm identification setting. However, we cannot maintain statistics over the infinite actions available in the continuous setting.
Continuous Armed Bandits An extension to the multi-armed bandit (MAB) setting, continu- ous armed bandits have an infinite number of action values available in the continuous range. Tractable algorithms. with provable bounds on cumulative regret, often make of Lipschitz condition 2 (Agrawal| 1995 Kleinberg 2004). However, a simple fixed discretization of arms over the continuous space has been shown to be wasteful, and adaptive discretization has become popu- lar approach, e.g. the Zooming Algorithm KKeiiberg et al. [0088] 2019). Adaptive discretization has the same worst-case regret as fixed discretization, but often performs better in practice. However, the regret bounds for these algorithms rely on prior knowledge of the Lipschitz constant.
We provide an example of stochastic loss function, induced by stochastic action outcomes, in our new Tanks domain Section 5.2
A function f X + R which satisfies [f()) f(y)) S T., yl for any two arms x. y X, is called L Lipschizz-comtimoous on X, with Lipschitz constant I..
3

---

Under review as a conference paper at ICLR 2023
A recent sfate-of-the-arr method proposed smoothing over discretised action space in order to create a density over the entire continuous action space (Krishnamurthy et al. 2020 Majzoubi et al.) 2020). Continuous Actions + Trees + Smoothing (CATS) is one such algorithm that uses pre- specified bandwidth and number of discrete action bins to perform the task of extreme classification for the continuous action setting (Majzoubi al. (020). An open-source re-implementation of the CATS algorithm is CATX (written in JAX) (Bejjani & Courtot 2022). CATX additionally shares parameters within tree-layers using multi-class classification networks instead of binary trees, We compare our RL agent to both the CATS and CATX algorithms in Section 5.3 and describe each in more detail next.
routers
centers
"smoothing"
Figure 1: (Left:) CATS binary tree policy where each internal node routes context vector left or right until a leaf node that represents an action center. Smoothing is then applied to create a probability distribution over A, and action is sampled. Image taken from (Vrousgou) (021). (Right): CATX tree policy: re-implementation with multi-class classification networks at each tree- layer. Action discretization of (pink leaves), tree depth of three: blue, red and green classification networks, for input context x. Image taken from (Bejjani & Courtot 2022).
Continuous Action + Trees + Smoothing (CATS) The CATS algorithm (Majzobii et al. 2020) was recently introduced as state-of-the-at for efficient contextual bandits for continuous action spaces. It is based on the idea of extreme classification using a decision tree structure where in- ternal nodes route a context vector probability distribution the algorithm performs successive classifications within a binary tree structure, where the number of nodes in each subsequent layer increases exponentially. The authors define Tree policy T as complete binary tree of depth D with K = 2 leaves, where each leaf has label function K-1 from left to right respectively, and where each internal node maintains a binary classifier f" " the class of binary classifiers from X' + {left, right].
continuous actions. That is,
label(v) K
K-
Successive classifications route an input context to an output bin, or action "center". A CATS tree policy is depicted in Figure [ (left). An action is then sampled e-greddily from uniformly smooth region around the bin (defined by the bandwidth parameter). The key idea is to smooth the actions: each action is mapped to distribution over actions using smoothing operator introduced in Section 3 When the action space the interval [0, 1]. P(A) = (a - h, aa++// /1] n [0, 1]. The authors state that the loss function for smoothed actions is always well-behaved (Krishnamurthy et al. 2020
See (Majzoubi et al. 2020) for more details.
CATX A recent re-implementation Bejjani & Courtot 2022) builds and improves upon the orig- inal CATS algorithm. has been written in JAX and released open-source. The repository authors make two key contributions: they integrate nonlinear function approximators into the tree struc- ture. Instead of binary classifiers at each internal node they implement multi-class classification networks each tree layer; it) each layer is represented by a network with output shape equiva- lent to the nodes in that tree-layer, meaning that each layer shares parameters. This is intended to stabilise training. Figure ( (right) depicts a CATX tree policy of depth three comprising of three classification networks (blue, red and green), providing action discretisation of 8 (bins). An action similarly sampled e-greedily from smoothed (LA), i.e. a smoothed region around the leaf node.
Deterministic Policy Gradients (DPG) RL for continuous control has its origins in the Deep Q-Network (DQN) (Mnih et al.) 2015) algorithm that combines advances in deep learning with rein- forcement learning. This seminal work adapted the standard Q-learing algorithm in order to make effective use of neural networks as function approximators. However, while DQN tackles problems with high-dimensional observation spaces, it can only handle discrete and low-dimensional action
4

---

Under review as a conference paper at ICLR 2023
spaces. For learning in high-dimensional and continuous action spaces, previous work combines an pollcc actor-critic approach with a deterministic policy gradient (DPG) (Silver et al. 2014). DPG maintains an actor function p(s|(0") parameterised by HM, which specifies the current policy by deterministically mapping states to specific actions. The critic Q(s, a) is learned using the Bellman equation. However, since the action space is continuous, the critic is presumed to be differentiable with respect to the action argument. This allows for a policy gradient-basee learning rule, and we can approximate max, Q(s, a) with ((s)..
The standard REINFORCE, or vanilla actor-critic algorithms are restricted to on-policy up- dates (Williams, 1992, Sutton Barto, 2018). This means that training samples are collected ac- cording to the policy currently being optimised for. These samples are then discarded and cannot be reused. However, DPG is an off off-policy actor-critic approach (Degris et al., 2012) that does not require full trajectories to learn and can reuse past interactions by storing them in an experience replay buffer. This facilitates better sample efficiency by allowing behaviour policy to collect new training samples to be decoupled from the deterministic policy being learned, providing better ex- ploration. This allows the behaviour policy to be stochastic for exploration purposes, but the target policy remains deterministic.
Finally, combining DPG with insights from the success of DQN, the deep deterministic policy gra- dients (DDPG) (Lillicrap al. 2016) algorithm demonstrated solving continuous simulated physics tasks directly from pixels. To achieve stability, the authors borrow ideas from DQN, such as a replay buffer, soft update target networks, and added novel noise UhleebenU process generate lemporally-correlan exploration noise.
We propose a deep contextual bandit agent for continuous actions based upon the DDPG algorithm. We describe our notation and the modifications to the standard DDPG algorithm in the next section.
METHODOLOGY
Sutton & Barto| 2018). with state space S and reward function r x A + R. We consider the state space equivalent to the context space X' defined in Section B
An RL problem can be specified by a Markov decision process (MDP) (Kolobov) 2012
In the RL setting, the value of a state can be defined as the expected total discounted future reward from that state onwards. The return is defined to be the total discounted reward from time-step r onwards, R. = Ee=r 7#-Tr(8x (x),, for discount factor 0 < 7 < 1. The agent's 'sss goal is to maximise the immediate reward plus the estimated return. Notice that r refers to the timestep within a trajectory evolving under a policy and transition function up to some infinite horizon. Whereas, specifies the trial from T' total trials in a contextual bandit setting, i.c. the agent is provided with i.i.d context vectors from the environment, i.e. 2₁ ~ E. Given the absence of a transition function in the bandit setting we are not required estimate the return R?, as the value of any future states collapses to the instant reward achieved at the first time point. We can therefore consider this as a one-step RL problem. Instead of maximising rewards, in the contextual bandit setting is common to minimise the equivalent loss le(x,,
We can therefore define the optimal one-step action-value function based on the Bellman equation as: Q" (xz, = E |r(ze, az)], and the mean-squared Bellman error (MSBE) function collapses to: re>²]. - for network parameters 6 and environment E' (Sttonn
L(d, E) (e
E
(QQ(Iz, az)
& Barto 2018). Note we can interchange the rewards r with losses depending on the environment.
One can note that we no longer require an estimate of the action-value for the future states along our trajectory. key innovation in DQN (Mnih et al., 2015) was to introduce target Q network to decouple the Q being optimised and the Q in the temporal difference update. However, since the contextual bandit setting has no transition function, there is no need to estimate future action-valle values. This means that our computation requirements are reduced as we do not need to maintain inefficient target networks, or interleave learning with target-updates.
It is particularly useful to estimate the policy gradient off off-policy from actions sampled using a dis- tinct behaviour (or exploration) policy, Le. 3(a, 2x) A T(a, ε2). We use an off off-cylicy actor-critic algorithm (Degris et al., 2012), coupled with deterministic policy network and deterministic policy
5

---

Under review as a conference paper at ICLR 2023
gradients (Silver et al. 2014). That is, we estimate the action-value function using differentiable function approximator, and then update a deterministic parametric policy μ4 the direction of the approximate action-value gradient.
The fundamental result underlying our approach is the policy gradient theorem (Sutton et al. 1999). We specifically interested in the deterministic policy gradient (Silver al. 2014), and adapt here for the contextual bandit setting:
/ p² (z) Vopoea 2),"(z, a)dx,
VaJa(po)
(1)
E BER.~E/VeHue V.Q (z,, at) loc=no(m,)l
where, (x) defines the state distribution visited under our behaviour policy 3, which the bandit setting is equivalent to sampling context from our environment E.
There is a crucial difference between the stochastic policy gradient and the deterministic policy gradient: In the stochastic case, the policy gradient integrates over both state and action spaces, whereas in the deterministic case, it only integrates over the state space. This has additional benefits for continuous high-dimensional action spaces.
For exploration, we take inspiration from the literature (Lillicrap et al. 2016). We construct a behaviour policy by adding noise sampled from noise process N to deterministic actor policy:
B pl 0%) NN
where / is chosen simply one-step Omnstein-Uhlenbeed process to add exploration.
EXPERIMENTS
5.1 STAGE 1: CONTEXTUAL BANDIT WITH CONTINUOUS ACTIONS FROM VECTOR CONTEXTS
Following the experimental protocol provided in (Majzoubi et al. 2020) and (ietti et al. 2021 we first evaluate the performance of our RL agent on four benchmark OpenML datasets (Vanschoren et al. 2013) each with single continuous action dimension and vector context space:
Wisconsin dataset: 32 dim state vector, action dim, 194 samples
2. CPULact dataset: 21 dim state vector, action dim, 8192 samples
3. Zurich Delays 5% dataset: 17 dim state vector, 1 action dim, 26,670 samples
4. Black friday dataset 9 dim state vector, action dim, 166,821 samples
We demonstrate the performance of our RL agent compared to state-of-the-art tree-based policy method CATS (Majzoubi al. 2020), and the shared parameter JAX re-implementation CATX (Be- ijani & Courtol, 2022) as described in Section 3
We used an implementation of CATS released with publication (Majzoubi al. 2020). It is de- veloped in the open-source Vowpal Wabbit framework (Langford et al. 2022 forms a parameter sweep over discretization and smoothing parameters: (2-13 12-13...22-1 2-1), K [22 .... 213)), }. and we report the best results. The hyperparamer settings for all experiments can found Appendix A.I.I. Similarly, we used the open-source JAX imple- mentation of CATX code (Bejjani & Courto 2022), and also swept for optimal hyperparameters per dataset.
The code per- { (h, K)
Following the literature, we also compare to two baseline approaches: Firstly, dLinear, dis- cretized e-gredyy algorithm which by default uses doubly robust approach for policy evaluation and optimisation (Si et al. 2020p. Secondly, d'ree, is discretized tree-based algorithm which is equivalent to CATS without smoothing, i.e. zero bandwidth. For all experiments we used e = 0.05.

---

Under review as a conference paper at ICLR 2023
Hesfi
Helphh
height: 95
Figure 2: Three example images from our new Tanks Bandit domain. Demonstrated here are three successful actions. Tanks in black (enemy is left, agent is right of barrier), trajectory of action taken shown in red. 3-dimensional continuous actions are interpreted as aj ==locaaton, 02 shot power, ag turret height.
5.2 STAGE 2: CONTEXTUAL BANDITS WITH CONTINUOUS ACTIONS FROM IMAGE CONTEXTS
We significantly increase the complexity of context provided this section. That is, we move away from small vector context spaces, to high-dimensional contexts directly from pixels. This demon- strates the agent's 's ability to adequately represent the context information provided for continuous control. We experiment with two image domains: 1) a single-action regression task based on the widely used MNIST dataset (LeCun et al.] 1998); and 2) we introduce a novel image benchmark domain for multi-dimensional actions based on 2D game of Tanks. See Figure [ for sample game images. We provide an OpenAl Gym (Brockman et al., 2016) interface to existing source game Flair 2022
MNIST contains 60K training and 10K held-out samples, where each is a 28 x 28 image. We define continuous loss function 1(I,, 0z) |a,- Vel. where y: is the label provided supervision (i.e.
the digit value).
The Tanks domain properties over existing contextual bandit benchmark datasets: 1. large image context space (upto 800 600 pixels); and multi-dimensional continuous action space, A [0, _j where N 3. The context Ia can be provided as either a vector describing the locations of key objects in the scene, or as an image.
benchmark that will release with the paper. It has two challenging
Context images contain the ground, random barrier, and an enemy tank (randomly positioned to the left the barrier). single continuous action vector is available the agent: 0₁ the z-location of its own tank (restricted to right of the barrier); the power of its shot; and a3 the height of its turret Resulting trajectories can be see in Figure (note these images contain additional information than the context images).
We have developed two loss functions. The first is smooth loss across the enemy's ' possible loca- tions, defined as:
1xx a) -) | - Fenerry "Fenery/abaarree if (0 5) S Inteersect S (Sbarrier)
(3)
otherwise,
where Wintersects Karemy and Fearrier the a-location where the fired trajectory intersects with the ground plane; the vc-location of the enemy tank; and the v-llocatinn of the random barrier respec- tively. 8 50 pixels by default.
The second is sparse loss, defined only at the enemy's location:
[ [ otherwise,
Fintersect Tenemyl I 30 pixels
11x a)
(4)
Each agent was trained for 50 epochs and 50 batches per epoch (further details in Appendix B)
'In Figure the Power action scaled (0, 1] [0. 100) and turret height [0, - [0, 9) for visual purposes.

---

Under review as a conference paper at ICLR 2023
5.3 RESULTS
To evaluate our agents we provide metrics based upon the two settings introduced in Section online regret minimisation, and offline held-out costs. Online regret is popular in the Bandits liter- ature, where the agent has access each training sample only once. Offline costs is popular in the RL literature, and facilitates multiple passes over the dataset of experience. We simulate this setting by passing over the training dataset for multiple epochs, and report the average cost held-out validation set (10% of the dataset size unless otherwise stated).
Average Regret: OpenML Datasets
Average Costs on Held Out Validation set
0.30
Average Online Regret
Held Out Average Costs
0.25
0.4
0.20
0.15
Agent
Offline
0.10
CATX
0.2
CATX
CATS
0.05
dLinear
RL Agent
RL Agent
0.00
0.0
Wisconsin n=194
CPU
Zurichs
CPU
n=27k
n=167k
n=194
n=8k
n=27k
n=166k
n-60k
Figure 3: (Left:) Average regret in the online setting on four benchmark OpenMI. datasets with single continuous action. Dataset size and number of fixed trials T is provided along the x-axis. (Right:) Average cost on a held-out validation set (10% unless stated). All confidence intervals are calculated with a single run using the Clopper-Pearson interval with 95% confidence level (as is standard in the literature). The confidence intervals are noticeably larger for the small dataset sizes: Wisconsin and CPU datasets.
Online Regret We report online regret on four benchmark OpenML datasets with vector contexts in 3 (where lower is better). We provide the dataset size, and therefore the number of fixed trials T, reported along the z-axis (increasing from left to right). The results shows clear trend: as the number of online trials increases, i.c. for larger datasets, our RL agent begins to noticeably outperform the tree-based methods in average online regret. For example, Wisconsin is a very small dataset, and provides just 194 online trials to the agent. this case, the tree-based method CATS achieves the lowest online regret. However, as the number of trials increases, for example in the Zurich or Friday datasets, our RL agent achieves substantially lower regret. The RL agent's 's policy and value networks require certain number of trials and policy gradient updates to converge, after initialising with random weights. This experiment demonstrates that with increasing number of trials, our RL agent provides lower average regret.
Offline Costs We report average offline costs on the four OpenMI. datasets in Figure 3 (right) (lower is better). We hold out 10% of the available dataset and report average costs on the held-out samples. In Figure (right) we shaded the additional improvement in performance by re-using the training data and iterating between epoch 0 and epoch 10 (30 for Wisconsin dataset), and depicted the methods using "+" in the legend. The results demonstrate clear trend, that on held-out samples our RL agent outperforms the tree- tree-based methods on all datasets. The shared- parameeer implemen- tation CATX also outperforms CATS. This experiment demonstrates that on held-out samples, our RL agent better generalises to new context vectors and outperforms current methods.
Image Context Held-out costs on the MNIST domain are reported in Figure 3 (right) (where lower is better). We can clearly see our RI. agent achieves the lowest average cost, and continues to improve with more epochs of training. We speculate that our actor-critic method is able to learn the required representations of the high-dimensional contexts to provide accurate continuous ac- tions without discretisation. Further, the shared-paramedd CATX implementation outperforms the original CATS that is clearly not able to adequately route image contexts to continuous actions.
We provide the initial results applying our RL agent the Tanks domain with multi multtiddimensional continuous actions in Figure 4 (lower better). We present online regret (left) and held-out costs (right) for four differnt agents: 1) from small vector contexts (blue); 2) from image contexts with
8

---

Under review as a conference paper at ICLR 2023
Tanks Domain: Orline Regret
Tanks Domain: Offline Held Out Costs
Online Avgerage Regiet
Offinn Costs
04-
0.4
m
Vecter Contest
Imapec: Smoothices
Imapes: Sparscuoss
Handam agent
Batches
Batches
Figure 4: (Left:) Average regret in the online setting for Tanks domain. (Right:) Average cost on held-out validation set. We compare using low-dim vector contex, image contexts with smooth and sparse loss functions, and a random agent. Confidence intervals are generated by multiple (5) training runs and each agent was trained for 50 epochs, with 50 batches per epoch.
smooth loss function in Equation 3 (orange); 3) from image contexts with challenging sparse loss function in Equation 4 4) random agent. All images were rescaled to (32 x 32) greyscale images.
DISCUSSION
We have demonstrated that one-step RI. agents, based deterministic policy gradients, can outper- form existing methods for contextual bandit problems with continuous action spaces. Our approach effective both from vector context spaces, as standard in the literature, and also from large context spaces where existing methods struggle.
Historically, the design of contextual bandit algorithms has been steered towards minimising cu- mulative (or average) regret over fixed number of T trials, with no burn-in phase. However, in real world scenarios with image contexts, we propose designing agents, or more specifically policy networks, that minimise either simple regret held-out costs after an initial training period. In this way, we can exploit the potential of policy gradients to re-use previous trials off-policy. We have successfully demonstrated that this is possible from image context spaces, and it is clear this is the problem setting we must solve to address challenges in real world personalised healthcare.
Finally, existing works based tree policies are limited to single action dimension. This is major limitation to their approach. our new Tanks domain, we successfully demonstrated control 3- dimensional continuous action value. Our policy network based approach can be trivially extended to output more than a single continuous action value and share parameters within the network.
7 CONCLUSION
In this paper we have demonstrated state-of-the-art performance on contextual bandit problems with continuous actions by modifying an RL algorithm for continuous control. We outperform hand-crafted contextual bandit algorithms for continuous actions in both online regret and held-out costs. Furthermore, we have successfully demonstrated generalising contextual bandits with multi- dimensional continuous actions to large context spaces, such images. We provided state-of-the-an performance using RL, significantly outperforming tree-policy methods on multiple benchmarks.
Finally, we have introduced new benchmark domain for contextual bandits with multi-dimmeniionll continuous actions and image contexts. We provide this challenging domain to the wider research community in the hope of stimulating additional research towards solving these challenging do- mains, especially when using sparse loss signals.
Whilst the motivation for this work comes from personalised healthcare, we leave the application of our method to medical images for future work. Similarly, for even larger context spaces such as 3D scans, we leave integrating pre-trained image representation or curriculum for future work.

---

Under review as a conference paper at ICLR 2023
7.1 ETHICS STATEMENT
There are no confficts of interest with regards to this work. We aim to contribute to the positive body of research aimed at tackling healthcare problems. We uphold the highest research standards and report honest, trustworthy, and transparent work to the community.
Our paper introduces a new, open-source challenge domain, Tanks. Games have long provided testbed for research into intelligent agents, from DeepBlue to AlphaGo. It is worth noting that although tanks are depicted in this game, game agents do not resemble real world tanks, and the game does not encourage violent actions. This simplified game provides our contextual bandit agent with a control problem: learn to produce a winning trajectory by adjusting parameters of the tank (the bandit actions), based on a single image of the environment (the bandit context).
This paper under double-blind review. For this reason, we honour confidentiality and have pro- vided no links to Github repositories or code.
7.2 REPRODUCIBILLITY STATEMENT
The work in this paper is entirely reproducible. There are three key components to this statement: the datasets: the code; and our new Tanks environment. We address each in turn below.
We use four widely-used benchmark datasets from OpenML. They are open-source, publicly avail- able, and version controlled. We do not manipulate the datasets, except for downloading, normal- ising, and randomly splitting into train and test sets. The same is true for our use of the MNIST dataset.
We will release the code required to reproduce our results in full in a linked Github repository upon publication of the paper. This will include the values of all hyperparameters and neural network architectures. We also provide many of the key hyperparameters in the supplemental material. The code to reproduce the results on benchmark algorithms, e.g. CATS, CATX, dLinear and d'Tree are all available at the citations provided in the main manuscript.
Upon publication we will also provide the source code for the Tanks game, along with Python code to interact with it an OpenAl Gym environment. We will provide the code to reproduce presented results, from both vector and image context spaces.
In conclusion, significant efforts have been made to ensure the reproducibility of this work, and that the community can build upon our research.
REFERENCES
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. International Conference on Machine Learning, pp. 1638-1646 PMLR, 2014.
Rajeev Agrawal. The continuum-armed bandit problem. SIAM journal control and optimization, 33(6):1926 1951,, 1995.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning. arXiv preprint arXiv: 05866, 2017.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235-256, 2002.
Wissam Bejjani and Cyprien Courtot. Catx: contextual bandits library for continuous action trees with smoothing in jax, 2022. URL https:///itthul: com/ wttp.///ssttddeppcccct/////////
Alberto Bietti, Alekh Agarwal, and John Langford. contextual bandit bake-off. J. Mach. Learn. Res., 133-1, 102..
Avrim Blum, Adam Kalai, and John Langford. Beating the hold-out: Bounds for k-fold and progres- sive cross-vale vooss-vali.t In Proceedings of the twelfth annual conference on Computational learning theory, pp. 203-208, 1999.
10

---

Under review as a conference paper at ICLR 2023
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in finitely-armed and
continuous-armed bandits. Theoretical Computer Science, 412 412(119:1832--1822 2011.
Sébastien Bubeck, Nicolo Cesa-Bianchi, al. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Foundations and Trends@ in Machine Learning, 5(1):1-122, 2012.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. In 29th International Conference on Machine Learning, 2012.
Data Flair. Blog post: Tank game in python with source code, 2022. URL https://
data-flair teattriiiii ttaiming/bloog -tgpr////bbb/oooooooootttte--eee
Aurélien Garivier and Eric Moulines. upper-confidence bound policies for switching bandit problems. In International Conference Algorithmic Learning Theory, pp. 174-188. Springer, 2011.
John Gittins, Kevin Glazebrook, and Richard Weber. Multi-armed bandit allocation indices. John Wiley & Sons, 1989.
Danijar Hafner, Timothy P. Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. ArXiv, abs/2010.02199) 2021.
D. Jarrett, E. Stride, K. Vallis, and MJ. Gooding. Applications and limitations of machine learning in radiation oncology. The British journal of radiology, (2(1100)201900) 2019.
Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In International conference on artificial intelligence and statistics, 1243-1251. PMLR, 2018.
Robert Kleinberg. Nearly tight bounds for the cotinuum-armen bandit problem. Advances in Neural Information Processing Systems, 17, 2004.
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Multi-armed bandits in metric spaces. In Proceedings of the fortieth annual ACM symposium Theory of computing, pp. 681-690, 2008.
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Bandits and experts in metrie spaces. J. ACM, 66(4), 2019.
Andrey Kolobov. Plamning with markov decision processes: An ai perspective. Synthesis Lectures Artificial Intelligence and Machine Learning, 6(1):1-210, 2012.
Akshay Krishnamurthy, John Langford, Aleksandrs Slivkins, and Chicheng Zhang. Contextual bandits with continuous actions: Smoothing, zooming, and adapting. The Journal of Machine Learning Research, 21(1):5402-5446, 2020.
John Langford, Alekh Agarwal, Miroslav Dudik, Daniel Hsu, Nikos Karampatziakis, Olivier Chapelle, Paul Mineiro, Matt Hoffman, Jake Hofman, Sudarshan Lamkhede, Shubham Chopra, Ariel Faigon, Lihong Li, Gordon Rios, and Alex Strehl. The Vowpal Wabbit (VW) project is fast out-of-core learning system sponsored by Microsoft Research and (previously) Yahoo! Research, 2022. URL https:: htt///gttttbtett thub. com www /ooo//owall hwwp://wwwbbbbbbccc Wabbi wwwpaabbbtt/wwwwaabbbaaaab wabbit
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):278-2324, 1998.
Huitian Lei, Ambuj Tewari, and Susan A Murphy. An actor-critic contextual bandit algorithm for personalized mobile health interventions. arXiv preprint 1rXiv:1706.09090 2017.
Lihong Li. Wei Chu, John Langford, and Robert E Schapire. A contextual-bann approach to personalized news article recommendation. In Proceedings of the 19th international conference World wide web, pp. 661-670, 2010.
11

---

Under review as a conference paper at ICLR 2023
Timothy P Lillierap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Maryam Majzoubi, Chicheng Zhang, Rajan Chari, Akshay Krishnamurthy, John Langford, and Aleksandrs Slivkins. Efficient contextual bandits with continuous actions. Advances in Neural Information Processing Systems (NeurIPS), 33:349-360, 360, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 8(7544)):5222555 2015.
Niklas Rindtorff, Ming Yu Lu, Nisarg A Patel, Huahua Zheng, and Alexander Amour. A biolog- ically plausible benchmark for contextual bandit algorithms in precision oncology using in vitro data. arXiv preprint arXiv: (et.:9911.0389 2019.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 2020.
Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust policy evaluation and learning in offline contextual bandits. International Conference on Machine Learning, pp. 6884-8894 PMLR, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp. 387-395, PMLR, 2014.
Aleksandrs Slivkins al. Introduction to multi-armed bandits. Foundations and Trends@ Ma- chine Learning, 12(1-2):1-280 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. Openml: Networked science in machine learning. SIGKDD Explorations, 15(2):49 60, 2013. doi: 10.1145/2641190. 198.
URL http:: acm. wtt.......0.111111//211111111111
2641198
Olga Vrousgou. Cats continuous action, contextual bandits. Workshop at International Con- ference Machine Learning ICML, 2021. URL htttps://croosmmnds..ouu w
Ronald J Williams. Simple statistical gradient- algorithms for connectionist reifforcmmetiinnoommmmmnn learning, Machine learning, 8(3):229-256, 1992.
Rong Zhu and Mattia Rigotti. Deep bandits show-off: Simple and efficient exploration with deep
networks. Advances in Neural Information Processing Systems, 34:17592-17603, 2021.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 ADDITIONAL EXPERIMENTAL DETAILS
A.1.1 ONLINE REGRET EXPERIMENT
A parameter sweeep was performed for each dataset, for each algorithm. The results provided in Figure 3 (left) used the following parameters and hyperparameters.
Wisconsin dataset: id 191
RL Agent: Actor-critic hidden layers, each fully connected with dimension = 12; Relu activations; size learning rate (1e-3, le-2) (actor/critic);
batch size = 4; memory buffer size
CATX: Neural Net: Haiku Network with latent dimension: 10 + [2 **(depth + 1)] nodes. batch
size 64; Tree depth 5; Discretisation parameter / 6;
CATS: n=512; h=1;
dTree: n=512;
dLinear: n=128;
CPU dataset: id = 197
RL Agent: Actor-critic 2 hidden layers, each fully connected with dimension = 256; Relu acti-
vations; batch size 4; memory buffer size 6000 batch size learning rate = (3c-4,1e-3) (ac- tor/critic);
CATX: Neural Net: Haiku Network with latent dimension: 128 [2 **(depth I)]; batch size 16; Tree depth = 4; Discretisation parameter h = 13;
CATS: 1=8192, h=512
dTree: n=8;
dlinear: n=8;
Zurich05 dataset: id = 42495
RL Agent: Actor-critic hidden layers, each fully connected with dimension 128; Relu acti-
vations; batch size = 2; memory buffer size 1000xbatch size learning rate = (3e-4,le-3) (ac- tor/critic);
CATX: Neural Net: Haiku Network with latent dimension: 128 + [2 **(depth 1)]; batch size 64; Tree depth = 3; Discretisation parameter / = 13;
CATS: n=92 h=256
dTree: n=2:
dLinear: n=4;
Black Friday dataset: 41540
RL Agent: Actor-critic hidden layers, each fully connected with dimension 128; Relu activa-
tions; batch size = 4; memory buffer size = 10000xbatch size learning rate = (3c-4,1e-3) (ac-
torfcritic);
CATX: Neural Net: Haiku Network with latent dimension: 128 + [2 **(depth + 1)]; batch size 64;
Tree depth = 3; Discretisation parameter h 11;
CATS: n==8192, h=256
d'Tree: n=4;
dlinear: n=4;
13

---

Under review as a conference paper at ICLR 2023
A.1.2 OFFLINE COSTS EXPERIMENT
Similarly, a parameter sweeep was performed for each dataset, for each algorithm. The results provided in (right) used the following parameters and hyperparameters.
Wisconsin dataset: id= 191
RL Agent: Actor-critic hidden layers, each fully connected with dimension 12; Relu activations; batch size = 4; memory buffer size 100x batch size learning rate = (1e-3, le-2) (actor/critic);
CATX: Neural Net: Haiku Network with latent dimension: 10 + [2 "*(depth + 1)] nodes. ; batch
size = 20; Tree depth = 5; Discretisation parameter h = 6;
CATS: n=512; h=1;
d'Tree: n=512;
dlinear: n=128:
CPU dataset: id = 197
RL Agent: Actor-critic hidden layers, each fully connected with dimension 256; Relu acti- vations; batch size = 2; memory buffer size x000xbatch size learning rate = (3e-4.1e-3) (ac- tor/critic);
CATX: Neural Net: Haiku Network with latent dimension: 128 + [2 **(depth 1)]: batch size 16; Tree depth = 4; Discretisation parameter / = 13;
CATS: n=8192, h=512
dTree: n=8;
dLinear: n=8;
Zurich05 dataset: id = 42495
RL Agent: Actor-critic 2 hidden layers, each fully connected with dimension 128; Relu acti- vations; batch size 32; memory buffer size 50000xbatch size ; learning rate = (3e-4,1e-3)
(actor/critic);
CATX: Neural Net: Haiku Network with latent dimension: [10, 10] + [2 "*(epphh + 1)]; batch size 16; Tree depth Discretisation parameter h 11;
CATS: n=1192, h=256
dTTree: n=2;
dLinear: n=4;
Black Friday dataset: id 41540
RL Agent: Actor-critic hidden layers, each fully connected with dimension = 128; Relu activa- tions; batch size memory buffer size = 10thhxbatch size learning rate (3e-4.1e-3) (ac- tor/critic); schedule = 1r+=0.00001 per epoch;
CATX:
Neural Net: Haiku Network with latent dimension: 128 + [2 **(depth 1)]; batch size 64;
Tree depth = 3; Discretisation parameter f = 11;
CATS: n==8192, h=256
d'Tree: n=4;
dLinear: n=4;
MNIST dataset:
RI. Agent: Actor-critie hidden layers, each fully connected with dimension 128; Relu acti- vations; batch size 32; memory buffer size 10000 size learning rate = (3e-4.1e-3) (actor/critic);
Neural Net: Haiku Network with latent dimension: 128 + [2 *= 1)]; batch size 64; Tree depth = 2; Discretisation parameter = 6;
CATX:
14

---

Under review as a conference paper at ICLR 2023
Figure 5: Three example image contexts from our new Tanks Bandit domain. Randomly generated environment, including barrier and enemy tank.
B TANKS DOMAIN
We provide sample context images from our Tanks domain in Figure Note, the information present in the image is the random enemy tank's location and the randomly placed barrier. The additional information in Figure iis for visual purposes only.
Image RL Agent: Actor-critic; 1 convolutional layer: (out channels=6, kernel size=5, stride=1, padding=2), fully connected layers, each fully connected with dimension 128; Relu activations; batch size = 64; memory buffer size x000xbatch size learning rate (5e-4,1e-3) (actor/critic);
schedule 1r+=0.0000) per epoch;
Vector RL Agent: Actor-critic 2 hidden layers, each fully connected with dimension = 32; Relu activations; batch size 64; memory buffer size 1000xbatch size learning rate = (5e-4,1e-3) (actor/critic); schedule = 1r+=0.00001 per epoch;
15


Reviews:
[{"review_id": "fVdrReMkuP", "reviewer": "Reviewer_hxTb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study a class of contextual multi-armed bandit (MAB) problems where the input space is high-dimensional and the action space is continuous. This is a common setting in medical diagnostics where, say, a drug dosing regime must be chosen baed on medical diagnostics/imaging. The continuous action space is particularly challenging as one cannot sample the entire \"arm\" space. To address this challenge, they note that there has already been success in reinforcement learning (RL) for high-dimensional inputs and continuous outputs with the Deterministic Policy Gradient (DPG) algorithm. Furthermore, multi-armed bandits are just a special case of RL where we need not worry about the impact of current actions on future states. They show that a simplified version of DDPG can out-perform tree-based policies on their class of MAB problems. They also introduce a novel OpenAI Gym interface (\"Tanks Bandit\") as a benchmark problem.", "summary_of_the_review": "I am marginally inclined to support this paper for publication. The material is well-presented, and focuses attention on an interesting class of bandits with high-dimensional inputs and continuous output spaces.  The paper is clear and presents a strong comparison between their DDPG-inspired technique and existing tree-based approaches. The novelty is somewhat low as they essentially re-use an existing algorithm leveraging the well-known fact that bandits are a special case of RL. However, this may inspire other applications for advanced RL techniques in the bandit domain. Also, the introduction of an OpenAI Gym interface to the tanks domain should be a valuable benchmark for this class of problems in the future.", "strength_and_weaknesses": "Strengths:\n1. Introduction of the \"Tanks Bandit\" OpenAI Gym interface will be a useful benchmark for the community in studying bandits with continuous output spaces.\n2. Demonstrates state-of-the-art performance on several test datasets.\n3. Focuses on an interesting segment of bandit problems with applications in healthcare.\n\nWeaknesses:\n1. Proposed algorithm (DDPG) is not novel. The authors merely apply it to bandits as a special case of RL. It would be interesting for the authors to explore why this connection was not made sooner. Are there other opportunities to apply well-developed RL algorithms in the bandit setting?\n2. The authors compare their work to the tree-based CATS and CATX algorithms. It is not clear why Gaussian Processes are not mentioned or applied as they are also commonly used to optimize a continuous action space.\n3. I have some ethical concerns about applying exploration-exploitation to determine patient dosages. It would be good to give more context on why we wouldn't always \"exploit\" to help the patient.\n4. Unclear in figure 4 why none of the algorithms in the online setting reduce average regret over time, except perhaps transiently in the first few batches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VybJ3CQvdvc", "reviewer": "Reviewer_rEXW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses a deterministic policy gradient for a deep contextual bandit with additive noises for exploration, and through extensive experiments discusses that this approach is easier for continuous and multi-dimensional action spaces, compared to others.", "summary_of_the_review": "In the end, the most important thing that the authors empirically illustrated is that for larger datasets and continuous action spaces, their method provides a smaller regret (Figure 3). It is a valuable experiment, but not a significant contribution. I will try to finalize my decision after the rebuttal period to one of the accept or reject options.\n\npost rebuttal: the work lacks multiple things, discussed above, to become ready for publication.", "strength_and_weaknesses": "Strengths:\nExtending a recent state-of-the-art method from a single dimension to a multi-dimensional continuous action space, which opens the door to new applications.\nWeaknesses:\nThis paper introduces a new algorithm that works better than alternatives in the literature only when a large amount of data is used. This data-hungriness is a weakness, especially for a work motivated by healthcare applications. More importantly, the paper lacks any reliable analysis. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "R2okb6k-Z-g", "reviewer": "Reviewer_ejwg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a setting of contextual bandits where the space of actions is continuous and contexts are large. Different from existing approaches in contextual bandits, the paper proposes to modify the DDPG algorithm, a popular algorithm in reinforcement learning into a context bandit algorithm with continuous action spaces.  The performance of their algorithm is evaluated on benchmark OpenML datasets.  Their results show that the proposed algorithm is effective for large context spaces.\n\n \n\n", "summary_of_the_review": "Overall, the paper can be improved by adding novelties in the approach, and experiments on large-scale datasets. I recommend rejection. \n\n=========== after rebuttal==================\nI thank the authors for their answers. However, their answers have not addressed my concerns about the novelty and reliable analyses of the proposed methodology as well as the experimental results required for high-dimensional actions. The paper raises an interesting open problem of contextual bandits and their approach is still primeval. I keep my score.        ", "strength_and_weaknesses": "Strengths:\n- The contextual bandit problem with continuous action spaces and large contexts spaces is important and challenging. \n- Related works in contextual bandits are discussed in a clear manner.\n- The experimental results are provided.\n\nWeakness:\n- The description of the proposed approach is not clear and should be improved.  I am wondering whether the approach uses offline data. If so, where are offline data while related works the authors discussed are for online bandits?  \n- The novelty of the approach is limited: the authors adapt the DDPG algorithm with a slight modification to the contextual bandits. I am wondering why their approach with such adaptation does better than existing works which are designed for contextual bandits.  \n- The experimental results are not convincing. For example, the authors claim that their approach is designed for multiple-dimensional action spaces, but all experiments are for one-dimensional actions.  In addition, the number of samples of datasets is inadequate for large-scale contextual bandits.     \n\n\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FEsoZYOrG7w", "reviewer": "Reviewer_QaQR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes use of RL algorithms to solve the contextual bandit problems with continuous contexts/action spaces. Using DDPG and actor-critic framework in RL, the paper achieves good levels of performances as compared to other baselines. ", "summary_of_the_review": "My rating would be a 3. I'd be happy to adjust my scores if the authors address my concerns.", "strength_and_weaknesses": "Strength:\n\nThe paper is well-organized, with each section telling exactly what it does. All the algorithms as well as the intuition behind the algorithms are explained clearly, and experimental results seem to corroborate with the claims.\n\nWeaknesses:\n\n1. The novelty of this paper is low. On a high-level, the paper appears to be no more than a direct application of common RL algorithms onto a special use case of RL(contextual bandits) which does not involve state change. While continuous contexts + continuous action spaces do pose their own challenges, direct application of DDPG + policy gradient based optimization is a common trick in RL.  \n\n2. While the motivation is mostly clearly stated, there seems to be a bit disconnection between the different applications/use-cases. In particular, it is unclear why MNIST is used for the problem when the graphical illustration of the problem doesn't seem to be connected with any MNIST attributes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject