High Probability Contextual Bandits for Optimal Dosage Selection

Authors: Spyros Dragazis, Aldo Pacchiano
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Linear Bandits, Dosage Selection, Contextual Bandits

Abstract:
Multi-Armed Bandit ($\textit{MAB}$) formulations are commonly used to model the problem of $\textit{Optimal Dose-Finding}$.
However, in many practical applications, it is necessary to receive data about the patient’s current state and then administer a drug dosage adapted to that state. 
To overcome this issue, we adopt a linear contextual bandit formulation with stage-wise constraints.
At each round, the learner selects a dosage and receives both a reward signal and a cost signal.
The learner’s goal is to maximize the drug's efficacy—captured as the expected cumulative reward—while ensuring that the toxicity, reflected by the cost signal, remains below a known threshold.
Satisfying the cost signal constraint only in expectation can be dangerous, as it may lead to over-dosage complications in certain cases.
To address this issue, we introduce a novel model that controls the realization of the cost signal with high probability, in contrast to previous works where control was only applied to the expected cost signal.
Our algorithm follows the $\textit{UCB}$ approach, for which we establish a regret bound over 
$T$ rounds and run numerical experiments.
We further generalize our results to $\textit{non-linear}$ functions and provide a regret bound in terms of the $\textit{eluder dimension}$, a measure of function class complexity.

Decision: reject
Reviewer scores: 5, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729734842840

thread_type:
official_review

<thread_object>
creator: reviewer_fl19

<content>
summary:
The paper proposes a novel approach for determining optimal drug dosages using a linear contextual bandit model with stage-wise constraints. The key contribution is an algorithm that controls the toxicity of the administered dosage with high probability per step, rather than just in expectation as done in prior literature, thus addressing safety concerns in clinical settings. This method maximizes drug efficacy while minimizing the risk of overdose by ensuring toxicity remains below a threshold. The paper establishes theoretical regret bounds and demonstrates the algorithm's effectiveness through synthetic experiments, highlighting its potential in adaptive dose-finding scenarios.

strengths:
+ The paper pushes the boundary of dose-finding methodology designs by considering the per-step toxicity constraint in a bandit-based solution. This setting has practical relevance and the solution thus can move the MAB-type methodology to be useful in practice. This aspect itself is very important not only to the ML community but probably more importantly, to the clinical trial methodology design community where safety and efficiency have been two of the critical considerations. 
+ The high probability guarantee of cost violation is a novel component of this work. Prior works have only considered the average cost constraint, which as the authors argued may not be useful in practice. The algorithmic and theoretical aspects of linear contextual bandits are also of interest to other problems.
+ The trick to get around the problem of not having an initial safe dosage is clever.
+ The work makes some effort to extend the design to non-linear functions, with some initial results.

weaknesses:
- One aspect that might be interesting to consider is that the toxicity tolerance threshold is often varying across patients. A more practical model would be to set $\tau$ also as a function of $X_t$.
- The argument in Section 4 relies on using $L$ and $S$ to construct the initial safe interval. This depends on the prior knowledge of accurate $L$ and $S$, which is, in some sense, reflecting the initialization of the trial. So fundamentally this is not surprising.
- The algorithm design part in Sec. 4 can be improved by more clearly articulating the differences to LinUCB.

questions:
- Back to the issue of not having an initial safe dosage... why can't we just use the minimum dosage as the initial safe dosage?
- What is the theoretical novelty in deriving Theorem 5.1? How is it different than prior proofs of LinUCB or cost-expectation-based solutions?
- The simulations did not have any comparison to prior solutions. Can you add such comparisons? In particular, you have criticized prior solutions as only caring about expected cost constraint -- then how bad are they when you count step-wise constraint violation? What is the tradeoff of regret and constraint violation for all methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479438250

thread_type:
official_review

<thread_object>
creator: reviewer_tfjc

<content>
summary:
The paper studies the optimal dosage finding problem with two objectives, maximize the drug’s efficacy and ensuring the toxicity especially from a high probability perspective. The authors adopt a linear contextual bandit formulation with stage-wise constraints, and design an efficient algorithm based on the idea of UCB. They establish a regret bound for this high-probability constrained approach, ensuring sublinear regret over time, meaning the model becomes more accurate as it learns.

strengths:
1. The two-objective problem in dosage finding formulated by the paper is meaningful and practical relevant.
2. The way that how the authors adopt the idea of UCB looks interesting. The technical results are solid.
3. The paper is well written and relatively easy to follow.

weaknesses:
1. Formulation. I am not sure whether it is common to think that the efficacy and the toxicity are linear with the dosage. From my experience in clinical trials, it is not usually the case. 
2. Technical contribution. I like the authors' way of adpoting UCB, but the techniques seem very standard, mainly based on the repetitive use of the inequality from Abbasi-Yadkori et al., (2011). Is there any technical contribution that the authors want to highlight?
3. A very minor comment: I do not think you need to have a new paragraph only to say like "Proof. The proof is in A.3." You can save a lot of space to present more interesting results.

questions:
An additional question to the weakness above, what does "stage-wise constraint" mean in the abstract? I can not see any explanation on this in the main text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693674251

thread_type:
official_review

<thread_object>
creator: reviewer_ck7p

<content>
summary:
The paper considers the problem regret version of the optimal dose finding with constraints that need to be satisfied with high probability.

strengths:
The paper is well-written and solves a relevant problem. The paper is well-written and organized and is easy to follow.

weaknesses:
The paper claims that is it the first to consider high-probability constraints on costs. I do not agree, since https://arxiv.org/pdf/2401.08016 ("Contextual Bandits with Stage-wise Constraints") seems to consider the setting with constraints that need to be satisfied with high probability as well. It is true that they need the knowledge of safe action, but I feel that data is benign in this application where there is likely to be a historical data set.

2)The motivation of the regret formulation is not clear. What does high regret mean ? Suppose the safety constraints were easy to satisfy, can then the algorithm just give out the maximum dosage? to make regret negative (Assuming the rewards are positive)

3)The techniques overall are standard in the linear function case. Maybe in future versions, the authors can specifically describe the main challenges faced in the proofs as a separate section

questions:
Please see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694986932

thread_type:
official_review

<thread_object>
creator: reviewer_esg2

<content>
summary:
This work investigates linear contextual bandits, with a brief exploration of a nonlinear case at the end, in the context of optimal dosage selection under a constraint function. A UCB-type algorithm is proposed to minimize regret while ensuring that the constraint is satisfied with high probability. Theoretical results are presented, and the approach is validated through numerical experiments.

strengths:
This work introduces a new area of research: contextual bandits applied to dosage optimization while accounting for a (toxicity) constraint. The reviewer believes that this framework has broad practical relevance, making it a valuable and worthwhile topic to explore further.

weaknesses:
The main concern raised by the reviewer is related to the assumptions. Specifically, the assumptions for the efficacy generation $R_t=\alpha_t <X_t,\theta^*> + \alpha_t\xi_t^r$ and $C_t=\alpha_t <X_t,\mu^*> + \alpha_t\xi_t^c$ are not intuitive to the reviewer. 

Additionally, the presentation of the paper needs improvement for better readability, and the theoretical results should be expanded and explained in greater detail.

There are some minor comments:
$\gamma_\alpha$ is not defined. This should be stated in Assumption 1.
L134-138: not clear
L159-170: Thought X and Y are introduced, they are not explained. Also, the meaning of p_k and q_k should be stated.
L171-176: For the reviewer, it is hard to find some connection between this paragraph and the previous one.
The definition of K(x) should be more clear.
The citation format needs correction, and the language throughout the paper should be made more formal.

questions:
1. Could the authors please provide justifications of the suggested efficacy and toxicity function by providing real-word examples?
2. In the regret, is it guaranteed that alpha_t is less than alpha_t^\star?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732990446077

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their valuable comments, constructive discussions, and dedicated efforts. Their input has been instrumental in significantly improving our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies a bandit problem where the agent chooses drug dosage to maximize reward, which is a linear function of the dosage; and controls cost with a high probability, which is also a linear function of the dosage. Both the reward and cost are linear functions of feature vectors and unknown model parameters. The authors propose an optimistic algorithm, analyze it, and evaluate it in simulations. The scores of this paper are 8, 6, and 2x 5, which is an improvement over the initial 8, 6, 5, and 3. The reviewers had multiple concerns:

* **Modeling assumptions:** Some modeling assumptions are highly stylized. For instance, while the linearity of reward and toxicity is reasonable in many cases, this needs to be rigorously justified. Defining the efficacy of a medicine per unit implies that the efficacy increases linearly with dosage. The assumption that the variance of the error term increases quadratically with dosage is unrealistic.

* **Technical challenge:** The reviewers are unconvinced that this paper is technically challenging. The high-probability confidence intervals used in this work have been known for a decade.

* **Experiments:** A strong experimental section, for instance using off-policy evaluation on real patient data, would carry this paper. Unfortunately, all experiments in this paper are synthetic.

These concerns require a major revision and therefore the paper cannot be accepted at this time.