Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study a class of contextual multi-armed bandit (MAB) problems where the input space is high-dimensional and the action space is continuous. This is a common setting in medical diagnostics where, say, a drug dosing regime must be chosen baed on medical diagnostics/imaging. The continuous action space is particularly challenging as one cannot sample the entire "arm" space. To address this challenge, they note that there has already been success in reinforcement learning (RL) for high-dimensional inputs and continuous outputs with the Deterministic Policy Gradient (DPG) algorithm. Furthermore, multi-armed bandits are just a special case of RL where we need not worry about the impact of current actions on future states. They show that a simplified version of DDPG can out-perform tree-based policies on their class of MAB problems. They also introduce a novel OpenAI Gym interface ("Tanks Bandit") as a benchmark problem.
strength_and_weaknesses: Strengths:
1. Introduction of the "Tanks Bandit" OpenAI Gym interface will be a useful benchmark for the community in studying bandits with continuous output spaces.
2. Demonstrates state-of-the-art performance on several test datasets.
3. Focuses on an interesting segment of bandit problems with applications in healthcare.

Weaknesses:
1. Proposed algorithm (DDPG) is not novel. The authors merely apply it to bandits as a special case of RL. It would be interesting for the authors to explore why this connection was not made sooner. Are there other opportunities to apply well-developed RL algorithms in the bandit setting?
2. The authors compare their work to the tree-based CATS and CATX algorithms. It is not clear why Gaussian Processes are not mentioned or applied as they are also commonly used to optimize a continuous action space.
3. I have some ethical concerns about applying exploration-exploitation to determine patient dosages. It would be good to give more context on why we wouldn't always "exploit" to help the patient.
4. Unclear in figure 4 why none of the algorithms in the online setting reduce average regret over time, except perhaps transiently in the first few batches.
clarity,_quality,_novelty_and_reproducibility: Clarity - Well-written and understandable for anyone with basic bandit/RL background.

Quality - Work appears generally correct and reasonable effort was spent in comparing multiple techniques across several benchmark datasets.

Novelty - Novelty comes from applying a previously existing technique to an under-studied bandit domain. However, the insight is somewhat basic -- that bandits are a special case of RL, so we can apply RL algorithms in the bandit domain.

Reproducibility - Fully reproducible, pending open-sourcing of their code. Datasets were open source, and author even open-sourced their Open AI Gym environment "tanks".

summary_of_the_review: I am marginally inclined to support this paper for publication. The material is well-presented, and focuses attention on an interesting class of bandits with high-dimensional inputs and continuous output spaces.  The paper is clear and presents a strong comparison between their DDPG-inspired technique and existing tree-based approaches. The novelty is somewhat low as they essentially re-use an existing algorithm leveraging the well-known fact that bandits are a special case of RL. However, this may inspire other applications for advanced RL techniques in the bandit domain. Also, the introduction of an OpenAI Gym interface to the tanks domain should be a valuable benchmark for this class of problems in the future.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes use of RL algorithms to solve the contextual bandit problems with continuous contexts/action spaces. Using DDPG and actor-critic framework in RL, the paper achieves good levels of performances as compared to other baselines. 
strength_and_weaknesses: Strength:

The paper is well-organized, with each section telling exactly what it does. All the algorithms as well as the intuition behind the algorithms are explained clearly, and experimental results seem to corroborate with the claims.

Weaknesses:

1. The novelty of this paper is low. On a high-level, the paper appears to be no more than a direct application of common RL algorithms onto a special use case of RL(contextual bandits) which does not involve state change. While continuous contexts + continuous action spaces do pose their own challenges, direct application of DDPG + policy gradient based optimization is a common trick in RL.  

2. While the motivation is mostly clearly stated, there seems to be a bit disconnection between the different applications/use-cases. In particular, it is unclear why MNIST is used for the problem when the graphical illustration of the problem doesn't seem to be connected with any MNIST attributes.
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, with good quality. While code is not available, the description of the experiments is adequate.
summary_of_the_review: My rating would be a 3. I'd be happy to adjust my scores if the authors address my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers a setting of contextual bandits where the space of actions is continuous and contexts are large. Different from existing approaches in contextual bandits, the paper proposes to modify the DDPG algorithm, a popular algorithm in reinforcement learning into a context bandit algorithm with continuous action spaces.  The performance of their algorithm is evaluated on benchmark OpenML datasets.  Their results show that the proposed algorithm is effective for large context spaces.

 


strength_and_weaknesses: Strengths:
- The contextual bandit problem with continuous action spaces and large contexts spaces is important and challenging. 
- Related works in contextual bandits are discussed in a clear manner.
- The experimental results are provided.

Weakness:
- The description of the proposed approach is not clear and should be improved.  I am wondering whether the approach uses offline data. If so, where are offline data while related works the authors discussed are for online bandits?  
- The novelty of the approach is limited: the authors adapt the DDPG algorithm with a slight modification to the contextual bandits. I am wondering why their approach with such adaptation does better than existing works which are designed for contextual bandits.  
- The experimental results are not convincing. For example, the authors claim that their approach is designed for multiple-dimensional action spaces, but all experiments are for one-dimensional actions.  In addition, the number of samples of datasets is inadequate for large-scale contextual bandits.     



  
clarity,_quality,_novelty_and_reproducibility: - The paper is well written except for the methodology section.
- The paper considers an important problem of contextual bandits, but the novelty of the proposed method is very limited.  
 
summary_of_the_review: Overall, the paper can be improved by adding novelties in the approach, and experiments on large-scale datasets. I recommend rejection. 

=========== after rebuttal==================
I thank the authors for their answers. However, their answers have not addressed my concerns about the novelty and reliable analyses of the proposed methodology as well as the experimental results required for high-dimensional actions. The paper raises an interesting open problem of contextual bandits and their approach is still primeval. I keep my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper uses a deterministic policy gradient for a deep contextual bandit with additive noises for exploration, and through extensive experiments discusses that this approach is easier for continuous and multi-dimensional action spaces, compared to others.
strength_and_weaknesses: Strengths:
Extending a recent state-of-the-art method from a single dimension to a multi-dimensional continuous action space, which opens the door to new applications.
Weaknesses:
This paper introduces a new algorithm that works better than alternatives in the literature only when a large amount of data is used. This data-hungriness is a weakness, especially for a work motivated by healthcare applications. More importantly, the paper lacks any reliable analysis. 

clarity,_quality,_novelty_and_reproducibility: In formula (3), the authors have created a discontinuity in the loss function at point x=x_barrier. What purpose does it serve? Isn't a continuous function easier for learning here?

Figure 3: In the fourth paragraph of section 5.1, it seems that they have used the CATS algorithm, but in Figure 3 and the paragraph titled "Online regret" it seems that the "RL agent" is showing the result of the experiment. The right panel of Figure 3 has a different legend from the bars shown in the plot, from the legend of the right panel, and it does not include the "RL Agent." The notation needs to be more consistent.

The authors conduct an experiment on the MNIST dataset and define a continuous loss function, while the labels are discrete and it is not clear how and why they define the loss as continuous and what advantages it possesses.

The algorithm CATS [Majzoubi et al., 2020] works on continuous action spaces. So, the innovation of this work is limited to going from one to multi-dimensional action space.

Connections between the sections and subsections are not strong enough to deliver a readable story.

summary_of_the_review: In the end, the most important thing that the authors empirically illustrated is that for larger datasets and continuous action spaces, their method provides a smaller regret (Figure 3). It is a valuable experiment, but not a significant contribution. I will try to finalize my decision after the rebuttal period to one of the accept or reject options.

post rebuttal: the work lacks multiple things, discussed above, to become ready for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes policy gradients for solving contextual bandit problems, with a focus on continuous action and large context spaces. The proposed approach is evaluated empirically but not analyzed.

The initial reviews of the paper were 1x borderline and 3x reject, and this did not change after the discussion. The main issue is the lack of novelty, because the paper is a direct application of a popular RL technique (policy gradients) to a subclass of problems (bandits). I also skimmed through the paper and suggest the authors look at

* [Differentiable Meta-Learning of Bandit Policies](https://proceedings.neurips.cc/paper/2020/hash/171ae1bbb81475eb96287dd78565b38b-Abstract.html)

* [Meta-Learning Bandit Policies by Gradient Ascent](https://arxiv.org/abs/2006.05094)

These two papers treat policy-gradient optimization of bandit policies a bit more systematically than the reviewed paper. For instance, they also show that the optimized controllers have provably bounded regret for some conservative setting of their parameters.
justification_for_why_not_higher_score: This paper suffers from the lack of novelty.
Decision: Reject