Reinforcement Learning for Control of Non-Markovian Cellular Population Dynamics
Authors: Josiah C Kratz
Jacob Adamczyk
Venue: ICLR 2025 Spotlight (2025)
Keywords: optimal drug dosing
fractional differential equations
reinforcement learning
control theory
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We train an RL agent to find performant drug dosing strategies in novel non-Markovian models relevant for cancer and bacterial systems.

Abstract: Many organisms and cell types, from bacteria to cancer cells, exhibit a remarkable ability to adapt to fluctuating environments. Additionally, cells can leverage memory of past environments to better survive previously-encountered stressors. From a control perspective, this adaptability poses significant challenges in driving cell populations toward extinction, and is thus an open question with great clinical significance. In this work, we focus on drug dosing in cell populations exhibiting phenotypic plasticity. For specific dynamical models switching between resistant and susceptible states, exact solutions are known. However, when the underlying system parameters are unknown, and for complex memory-based systems, obtaining the optimal solution is currently intractable. To address this challenge, we apply reinforcement learning (RL) to identify informed dosing strategies to control cell populations evolving under novel non-Markovian dynamics. We find that model-free deep RL is able to recover exact solutions and control cell populations even in the presence of long-range temporal dynamics.  To further test our approach in more realistic settings, we demonstrate performant RL-based control strategies in environments with dynamic memory strength.

Published as a conference paper at ICLR 2025
REINFORCEMENT LEARNING FOR CONTROL OF NON- MARKOVIAN CELLULAR POPULATION DYNAMICS
Josiah C. Kratz"
Computational Biology Department Carnegie Mellon University Piusburgh, PA 15213, USA
jkratz@andrew cmu edu
Jacob Adamezyk"
Department of Physics
University of Massachusetts Boston
IAIFI
Boston, MA 02125
jacob. adamczyk00 @umb. edu
ABSTRACT
Many organisms and cell types, from bacteria cancer cells, exhibit remark- able ability to adapt to fluctuating environments. Additionally, cells can leverage memory of past environments to better survive previously-encountered stres- sors. From a control perspective, this adaptability poses significant challenges in driving cell populations toward extinction, and thus poses an open question with great clinical significance. In this work, we focus on drug dosing in cell popula- tions exhibiting phenotypic plasticity. For specific dynamical models switching between resistant and susceptible states, exact solutions are known. However, when the underlying system parameters are unknown, and for complex memory- based systems, obtaining the optimal solution is currently intractable. To address this challenge, we apply reinforcement learning (RL) to identify informed dosing strategies to control cell populations evolving under novel Marrovaoo dynam- ics. We find that model-free deep RL is able to recover exact solutions and control cell populations even in the presence of long-range temporal dynamics. To fur- ther test our approach in more realistic settings, we demonstrate robust RL-based control strategies in environments with measurement noise and dynamic memory strength.
INTRODUCTION
In order survive, organisms must adapt to unpredictable environmental stressors occurring over di- verse timescales. As result, biological systems display remarkable adaptive capabilities, making it exceptionally challenging to control them through environmental modulation, marking an open and significant question in the physics of living systems. Two examples of adaptive systems with great clinical importance are cancer cell resistance to chemotherapy (Vasan et al., 2019), and bacterial resistance to antibiotics (Blair et al.) 2015). In both settings, the control problem of drug application is complicated by unknown model parameters and non-Markovian dynamics. Notably, the constant application of drug does not typically result in population extinction, as drug application also drives a certain fraction of the population to alter their phenotypic state to become drug-resistant. This new resistant state can then persist even after drug removal and across cell lineages, thus en- coding a memory of the stressful environment which allows populations to more quickly adapt if the drug is reapplied Harmange al. 2023); Mathis & Ackermann (2017) hh: Banerjee et al.) (2021). As a result, it is unclear how to devise a control strategy that can effectively mitigate long-term population growth in these scenarios. We address this open problem in the present work.
Equull contribution. Author ordering determined by coin flip Primanti Bros.

---

Published as a conference paper at ICLR 2025
Previous literature (Padmanabban et al. 2017; Engelhardt 2020 Gallagher et al. 2024 Fischer & Bluethgen 2024) has studied temporal drug dosing protocols to slow or prevent adaptation in various cancer bacterial models using various methods. However, these models fail to account for the variety of adaptive timescales present real biological systems. The presence of such memory effects greatly complicates the control problem, making the study of more realistic models imperative. Thus, to study the control of such memory-based adaptive systems, in this work we introduce a novel population model exhibiting phenotypic plasticity with non-Markovian dynamics. Prior work has shown that learning- based methods are able to form the basis of patient-specine treatment protocols (Miotto al., 2016 Ahn et al., 2021). We show that insights from control theory can be coupled with deep reinforcement learning to discover treatment protocols which successfully prevent proliferation. We highlight our main contributions below:
MAIN CONTRIBUTIONS
1. We develop a novel memory-based model for phenotypic switching relevant for bacterial and cancerous population dynamics.
2. We pose the problem of optimal drug dosing in this model and provide insights the solution from control theory.
3. We find robust optimal policy with deep RL that is independent of memory strength, requiring only clinically-cceessbble observations.
In the following sections, we first introduce the new population model, derive bang-bang optimal control, and experimentally show that bang-bang control is required find high-performing policy- Because of the non-Markovian dynamics, we introduce a short history to the agent's 's state which we implement via framestacking. Using ideas from optimal control theory and deep reinforcement learning, we thus simplify the original control problem in well-principled manner. We then apply standard and more advanced RL techniques solve different versions of the posed problem.
PROPOSED MODEL AND APPROACH
In this section, develop our novel non-Markovian model for cellular dynamics. These dynamics are based on a simple but general switching model (Figure with switching rates that are dependent on drug concentration. We then introduce physically motivated non-linear memory kernel, before discussing general approaches to the optimal control problem.
2.1 NON-MARKOVIAN PHENOTYPIC SWITCHING MODEL
To model the treatment response of an adap- tive cell population, we use general pheno- typic switching model which captures the time evolution of a susceptible subpopulation, with size S(t), and a resistant subpopulation, with size R(t). (The notation for susceptible and resistant states should not be confused with state and reward in RL, for which use low- ercase letters.) Phenotypic switching models have been successful in describing a wide va- riety of biological scenarios, including devel- opment of persister cells and resistant cells in bacteria, and development of drug resistance in cancer cells (Balaban et al., 2004 Fischer & Bluethgen] 2024 Witzany et al. 2023 Ku- mar et al. 2019 Kratz& & Banerjee 2024). In our model, susceptible cells with net growth rate (u) (u=(u) < 0 corresponds to net cell death) switch to resistant state at rate a<u). where is the drug concentration. Similarly, when the drug is removed, resistant cells with a net growth rate kn(u uu) switch back to the susceptible state concentration-dependee. rate 5(u) (Fig. 1). All growth and switching rates function of
a(x)
S(O)
R(C)
KR(1))
(1)
8(1)
Figure 1: Depiction of the deterministic pheno- typic switching model. The susceptible subpop- ulation S transitions the resistant state R at concentration-dependeen rate, 8. Similarly, the re- sistant state switches back to a susceptible state at a rate a. The subpopulations each have a consentratioo-depeeddee growth or death rate, k.
2

---

Published as a conference paper at ICLR 2025
drug concentration normalized by the maximum allowable dose, thus E [0, 1]. The time evolution of the size of each subpopulation, xtt) = [s(t), R(t))", is then given by the dynamical system:
x(t) [(x(t), (t)) (uu(t))x(t,
where the state-transition matrix is given by:
Kis ()(u) (u) a(u)
8(u) k 5(u)
(u)
To relate the net growth rate to drug concentration for each subpopulation, we assume that both rates take the general form:
(.)( = Kig
max
min + ( KS )g(u): Kn(u) HR
min
KR
min
)h(()
where g(u), hu()) E [) is a monotonic dose-response function which relates drug dose to net growth rate, KiS <gax > O denotes the maximum growth rate of the susceptible subpopulation in the absence of drug application, and where x < O is the maximum death rate of the susceptible subpopulation caused by application of the maximum drug dose (u 1). Importantly, KR max > denotes the maximum growth rate of the resistant subpopulation, which occurs in the presence of the maximum drug dose, and mim < 0 corresponds to the maximum death rate of the resistant subpopulation, which occurs when the drug is removed. This parametrization corresponds to "drug addiction" " behavior in the resistant subpopulation, a robust phenomenon which has been observed not only in cell culture (Suda et al. 20122 Sun et al. 22144] Moriceau et al. (2015). but in animal mod- els (Das Thakur et al. 2013) and in vivo (Seifert et al. 2016) Dooley et al., 2016). Similarly, 5(u) decreases with while (x increases with u, drug application drives the population to become more resistant, while reduction in drug concentration causes the system to recover susceptibility. These switching rates can then be defined as:
a<())= = Omaxj(u) and = = - k(u)) where j(u), k(u) 10 and where Ommax: 6max switching from the susceptible state to the resistant state, and vice versa.
0 denote the phenotypic switching rates of cells
Recently, cell populations of many types, including human cancer cell lines, yeast, and bacteria, have been shown to maintain a memory of past environments which facilitates adaptation to previously- seen stressors over many timescales (Shaffer al. 2020 Harrmang al. 2023 Larkin al.. 2024
Wolf et al. 2008 Mathis & Ackermann (017). To better capture this memory dependence on treat- ment response, we introduce a memory kernel into the previously-describer dynamics (equation I) making them non-local in time. Specifically, we choose a fractional differential equation (FDE) formulation as a phenomenological way to introduce multiple timescales of adaptation, one which has been used successfully to model memory effects in other biological (Lundstrom et al.) [2008), ecological (Khalighi et al. 2022), and physical contexts (Bonfanti et al., 2020). With this addition the dynamics now become:
xt)) F(x((), 1/" (t 73- u(T))d
(3)
[[(p - 1)]
where I(-)) denotes the Gamma function, f(-) is given by equation ! and here we introduce the parameter j E (0, 1] which controls memory strength. A value of μ corresponds to the memo- ryless case (first order derivative), whereas smaller values of μ correspond to an increased influence of past states the current dynamics (lower order fractional derivative).
We seek to obtain a temporal drug concentration protocol u(t) which minimizes the growth of a population and thus choose the final cost C == (()) = 5(1) + R(()) represents the total population.
== log N(T)/N(0) over the time interval T, where
Thus, aim to solve the following control problem:
min C(x(0), u(-)) subject to x(t) = F(x(t),u(t)) x(0) = No. u(t) E [0, 1]
(4)
where xTT u(-) denotes the state of x at terminal time T subject to control u from 0 < < T. Interestingly, in minimal model (equation 3) as long ks(u). (n)u). a(u), and 5(u)
monotonic functions, we can show that the optimal control solution follows "bang-bang" control, regardless of the precise model parameters and memory strength.
3

---

Published as a conference paper at ICLR 2025
2.2 DERIVATION OF BANG- BANG CONTROL
Using equation D and the fact that (() S(1) + R(I), the model dynamics can be rewritten as single fractional differential equation in terms of the fraction of resistant cells, é R/N, namely:
D = f(0,0) u) = (Ne((u) kn(u))²²+ (kn(w) - - 5(u) a(u)) + a(u). (5)
where Di denotes the Caputo fractional derivative of order l starting / (Garrappal 2018), and here we drop the explicit time dependence for notational clarity. This can equivalently be written as the continuous delay differential equation (we use this form in equation 3):
o(t) - h. (t- (
[T - 1)1 f(((T), u(F))
(6)
Given these definitions, the Hamiltonian associated with this control problem is then
H u(t),, (é()), u(t))
where the trajectory of the Lagrange multiplier X(I) is the solution to the costate equation moyunov 2023):
(Go
X(1) (TT: 1 (o()))(((())) dr [( - 1) I(n) J.
(8)
(r f)
Applying Pontryagin's minimum principle, we obtain the following inequality H (t), "0),,"(())) (t), s (t)()),, u(t), (t))
(9)
which along with equation 5nd and equation can used to obtain the optimal control policy:
"" arg min X ( - (g An(ø )h(m)) Emaxe k(()) O ob" (d))
min and AR = AR min where the following shorthand notation is introduced: As = K K
R
Critically, the fact that kg KR Omax 6max > 0, xim min < O, and g. h J. k, [0. 1] ensures that the bracketed sum which multiplies X is positive for u O. As result, long as g(u), hu)), j(u), and kuu) are monotonically increasing functions of u, then the resulting optimal control is said to be "bang-bang", where u" (t) only takes its extreme values. Furthermore, the switching times between the maximum and minimum values of u are determined by X (t),,, yielding the optimal control solution:
m-ax
u ut) 0 if (t))) 0, (t) 1 if (()t) <0, and
" (t) E j0, 1] otherwise.
In principle, the optimal control trajectory can be obtained through numerical integration of the model dynamics (equation [ along with the corresponding costate equation (equation 8 In the con- text of bang-bang control of non-Markovian systems however, using this approach can be difficult, as requires careful choice integration technique and update rule. Thus, we turn to reinforcement learning.
As shown above, the optimal control solution to Problem follows bang-bang control, regardless of model parameters. This allows the continuous model of equation to be simplified to discrete model with binary controls without altering the optimal solution. This yields:
Tx(t) for u = 1 (Treatment Phase) (x( for u 0 (Pause Phase)
(1) f(x(()) m())
(10)
where now there are two state-trande -tate-transition matrices, given by:
min
O'max
P
8max
(11)
O'max
KR
6mux
We this formulation of the environment when training the reinforcement learning agent.
4

---

Published as a conference paper at ICLR 2025
2.3 CONTROL STRATEGIES
Despite this simplification in action space (from continuous binary controls), the optimal control problem remains difficult, as the number of times and duration of drug application must be opti- mized. Constant application of the drug at the maximum dose (u 1) results in cell adaptation and proliferation (Fig. 3 a highly suboptimal solution to Problem 4 and catastrophic result in the clinical context.
Recently, (Fischer & Bluethgen 2024) has shown that in the memoryless case (μ = 1), the opti- mal solution for this type of model requires an initial drug application phase, followed by pulsing between treatment and pause phases at a regular interval dependent on the model parameters. How- ever, as seen in Fig. g we find that the addition of memory (μ4 1) renders the control strategy of the memoryless case ineffective, as cells which have previously encountered treatment switch to the resistant state faster upon subsequent applications. Furthermore, in the clinical or experimen- tal setting, obtaining the values which parameterize equation 3 is usually not feasible, and one can only rely on direct and more macroscopic measurements from the cell populations. Thus, obtaining the appropriate switching frequencies through direct computation via optimal control (OC) theory becomes impossible. One may propose to learn these values directly before feeding them into an OC solution, but this may result in brittle pipeline, especially the model discussed may not fully describe the underlying biological dynamics. As a result, we seek to learn the optimal policy end-to-end, directly through experience using deep reinforcement learning (Sec.
2.4 OBTAINING THE OPTIMAL SOLUTION IN THE MEMORYLESS CASE
As mentioned above, previous work (Fischer & Bluethgen 2024) showed that the optimal pulsing protocol for the memoryless case requires an initial drug application phase until the resistant fraction reaches some upper threshold Φs. followed by pause phase ("drug holiday") in which the resistant fraction naturally relaxes back to some lower bound Φi. This followed by similarly repeated eycles of treatment and pause phases where the switching time occurs when the resistant fraction reaches da, and Φi, respectively. To obtain the optimal values of Φn and or for our specific model parameterization, swept over values of Φ), and o, between 0.1 and 0.9, with increments of 0.011, selecting the values which yielded the highest net death rate: Φi 0.494 and Φ) = 0.505.
To compare baselines against our learned policy in Markovann environments, we repeated the same procedure for different values of The optimal switching fractions are given in Table
REINFORCEMENT LEARNING
Reinforcement learning (RL) allows an agent to learn an optimal control strategy (policy) directly from the experience it collects. This data may be from sim- ulation (Akkaya et al.] 2201)) or from real-world data (Haarnoja et al. 2024), or
μ
01
OR
1.0
0.494
0.505
0.9
0.483
0.494
0.8
0.466
0.477
from fully offline datasets (Levine et al., 2020). RI. the interactive agent receives observations from the environment. As result of this input, the agent outputs a control or action (de- noted u throughout), yielding a transition and a scalar reward (negative cost). The goal of the agent, analogous OC, to act in a way that maximizes the expected long-term accumulated rewards. In our problem formulation, this will correspond to the use of drug to min-
0.7
0.448
0.459
0.6
0.428
0.439
Table 1: Optimal switching fraction values for the baseline policy for each memory strength
new state
imize the total population count.
We consider the online RL setting in simulation, where a drug protocol is learned through experience (Fig 2 left) by continually improving policy as it is learned. Despite the non-Markovian dynamics and even without access to the underlying euvironment-specine model parameters, we show that RL is capable of providing tigh-performing policies. To formulate the RL problem, define the relevant characteristics as follows (in the following, let denote the simulation time between actions):
5

---

Published as a conference paper at ICLR 2025
Learned Policy y
1.0
Action (drug concentration
0.8
Raeiland lecisam Dwlls
0.4
constant application
A-limited frequency
0.6
Pulsatile application, Increasing frequency
0.4
0.2
0.6
0.7
0.9
1.0
0.0
2.5
7.5
12.5
15.0
17.5
15
Time (hours)
Time thours)
Figure 2: Left: The learned policy shows resemblance the optimal memoryless strategy, with an initial constant application phase followed by pulsatile phase. However, in the case of memory- based dynamics, the frequency of pulsing must be increased over time as discussed in Sec. Since the policy is eventually limited by the simulation time, the pulsing frequency becomes bottlenecker by our choice of time discretization A after = hours. Despite this, the policy is still able to perform well with rapid pulsing. Right: Effect of learned policy resistant fraction. for different memory strengths. The RL agent finds (for distinct j values) appropriate lower and upper bounds for the fraction of resistant cells. Maintaining the subpopulation in this range ensures the population can be controlled.
State: The state (st) of the agent's environment is list of the last K = 5 estimates of the instantaneous growth rate and one-hot encoded actions, 8t [Ct-K...2,, Ue_ where = A log N is pointwise version of the cost defined above. Thus, history of past observations is encoded in the state vector, crucial design choice if the agent is learn control without a recurrent hidden state. Action: As motivated in Section 2.1 we choose binary action space E {0, 1} representing whether the drug applied or not, as we have proven that this suf- ficient recover optimal control. Reward: As we seek solve Problem 4 the reward is simply the negative growth rate, 7 = -c. Notice that with this choice of reward function, the sum of rewards across trajectory simplifies with a reduction in total cell population. Dynamics: The dynamics of the total cell population N₂ governed by the dynamical system described in Section 2. initialized be fully susceptible (Nn = [1000, 0).. After an action is executed, the simulation (a numerical solution Of equation 3h computed with the action (dose) fixed for A 0.01 hours to compute the next state (s++a)- To encourage the agent to reduce the cell population while decreasing computational runtime and allowing for exploration, we terminate the episode if the number of cells ever exceeds 150% of its initial amount.
Ro:T A -- log No/Nz, ensuring the agent's objective is aligned
The goal in RL is learn the policy 't" which maximizes the expected cumulative discounted sum of rewards across trajectory:
n (u|s) arg max Eyop,
Σ
rr(ss, ue)
(12)
To solve this optimization problem, consider discrete-action value-based methods. Within this framework, the optimal action-value function Q" (s, u) is learned based on off-policy data observed during exploration. The value function satisfies the Bellman optimality equation (written match the control notation introduced above):
Q" (st, Ue; 0) = r(st, ue) + max Q" (St+ ul;
(13)
Within this class of methods, DQN (Mnih et al. 2015) learns Q" by exploring with an annealed greedy exploration strategy. After an action taken, the experience tuple (8, u, r, ) stored in an experience replay buffer for later use. In deep RI, the action-value function is parameterized with neural network (denoted by the trainable parameters 0), and trained via mini-batch gradient descent,
'The numerical solution of fractional differential equations requires some care; cf. Garrappa 2018 for details.

---

Published as a conference paper at ICLR 2025
p-088
950
600
Total Colls
900-
550-
250
500
800-
450
750
H1. Policy
400
700
Tractio--aseed Palling
50
100
Memorr was Solution
500
60
00 100
20
60 100
20 000
100
05
(horurs)
Figure 3: Performance comparison of constant drug application, solution for the memoryless case, resistant traction-based pulsing technique, and policy learned by RL. For the fraction based policy, an optimal lower and upper bound for resistant fractions are found through sweeping (Appendix)) 2.4 The RI. policy is capable of controlling the cell population better than any other scheme.
denoted B, uniformly at random from the buffer. The temporal-difeee the Bellman residual - the squared difference between left and right hand sides of equation 13
loss function is defined as
Co Σ (",, 0) (r+ max Q" (s:t,,,d:0) {B.U.T.a")
(14)
As common in value-based algorithms, we use an additional target network to reduce the effect of bootstrapping during training. The target network (denoted with parameters 0) has its parameters frozen during training, and updated periodically directly by copying the online network parameters 0 + 0. We additionally use the double DQN approach to mitigate the over-estimation bias in standard DQN ( (Van Hasselt et al.) 2016). Further details on reinforcement learning can be found e.g. in (Sutton & Barto 2018 Hessel et al. 2018). We tune over several hyperparameters (whose values we list in the Appendix) to optimize performance. Our implementation of double DQN Van Hasselt et al. 2016) (and later FQF) based open-source code from Stable-Basee Baselines3 RRffin et al. 202 Each agent is trained for x 10 environment steps. All code to reproduce our experimental results be found at httpt::///// / httgggtttuuu..og./g com waaoo//JJaoaHaaaaaaa wtw.ooet...AAA.........oo
4 CASE STUDY OF FIXED MEMORY ENVIRONMENTS
We first test DQN in the memoryless case (1+ = 1), for which an optimal controller is known (Fischer) & Bluethgen 2024). In this case, the optimal policy can be derived based only on two consecutive resistant fractions. However, recall that the RL agent is only given access to the instantaneous growth rates, not the resistant fractions. Nevertheless, the RL agent surprisingly able to reliably recover the optimal policy with only two successive frames and without any access to underlying model parameters. The control strategy found by the agent involves first applying the drug and then pulsing regularly, in agreement with the OC solution, seen in Fig- (rightmost plot). Indeed, Fig. 2 (right panel) shows that the agent discovers an internal representation based on growth rates that is compatible with fraction-based switching.
With confirmation that RL can find the optimal dosing strategy in the memoryless case, we turn to the more difficult memory-based dynamics (n < 1). We do have solution to Problem + in this regime, so we compare against two baselines: The memoryless protocol described above and modified version in which switching times occur when the resistant fraction reaches a threshold value. We note that these baselines can have arbitrarily small switching times, which is practically infeasible. Remarkably, we find that using a small but bounded A, our learned policy outperforms both of these baselines (Fig. B.. In addition, our experiments show that decreasing A (increasing measurement frequency) beyond a certain threshold does not considerably increase performance (Fig.
Interestingly, the learned policy reveals that the agent initially maintains constant drug applica- tion before transitioning to memory-depender
pulsing protocol. The agent increases the fre- quency of pulsing throughout the trajectory until saturation at the maximum rate (A -1) (Fig. 2 For cases with memory, this increase in dosing frequency be understood as the result of

---

Published as a conference paper at ICLR 2025
faster cellular adaptation back to the resistant state after multiple drug encounters. Thus, to maintain a susceptible population, the policy's 's pulse frequency must continuously be increased to compensate for the memory-basee udaptability. We also find that the agent maintains lower average resistant fraction at higher memory strengths (Fig. 2 right). This qualitative trend is in agreement with the optimal switching fractions found by optimal control shown in Table T To independently show the benefit of using
discrete actions, the original problem can in-
stead be approached with continuous action al- gorithms, such as PPO (Schulman et al., (0177) or SAC (Haarnoja et al. (0118). As shown in Fig. these algorithms considerably underper- form relative to DQN.
DQN
Cola
Action
Coll
Qualitatively, we find that these continuous action algorithms successfully find the con- stant application period at the beginning of the episode. As the episode continues, though, both PPO and SAC attempt a short-lived bang- bang protocol before defaulting to the average action for the remainder of the episode. These results were obtained from an equal-compute hyper- 0.9 environ-
Cols
10
20
30
40
30
60
70
Time (hours)
Figure 4: PPO and SAC fail to find a bang-bang control policy and have a lower performance than DQN; highlighting the need for discrete action al- gorithms, as informed by optimal control.
parameter sweep in a fixed ment. We conjecture that continuous action al- gorithms struggle because of the saturating gra- dient effect at the limits of action space, which correspond to bang-bang control. Additionally, continuous action algorithms that give bonuses for policy entropy (e.g. SAC) make the task of learning these deterministic policies more difficult.
MODEL SUCCESSFULLY GENERALIZES TO MEMORY AND OBSERVATION NOISE PERTURBATIONS
In the previous sections, we construct RL agents that are each trained on specific dynamics deter- mined by a fixed value. However, this can be problematic in practice, as the memory strength is difficult measure (it would require fitting equation 5 entire episodes data), which could corre- spond to adverse costs in the clinical setting. Furthermore, adaptive cell populations such as cancer can dynamically change their memory capacity over time (Ringrose Paro 2004; Acar et al. 2005). Thus, useful policy should be able to interact immediately without requiring additional samples, and with no access to the value of μ. The unknown value ju thus represents "side information" that is not given to the agent but provides context for the agent to understand the current envi- ronment. This places the new problem setting in the framework of contextual MDPs (Hallak et al.) 2015 Sodhanii al. 2022). To approach this problem while maintaining clinical applicability, train a general agent over a range of memory strengths. This approach of "domain randomization" (DR) has been applied with great success other RL problems (Tobin et al. 2017 2019)
Akkaya al.,
To tackle this more challenging environment, we use FQF YYng et al. 2019) recent algorithm for distributional RL, where the distribution (rather than only the mean) of future returns is learned. FQF and its variants have shown success in discrete control environments. In particular, we use a noisy variant that combines the approach of NoisyNets (Fortunato et al. 2018) for improved exploration. Further, we find it useful to supplement the RI. state with the one-hot encoded actions used in the last K steps. This allows the agent to observe the response of the environment to the presence of drug: At lower μl, application of drug has a drastically different effect than at higher μ values at similar growth rates. With these changes, we sweep model capable of reducing cell populations given any memory strength.
hyperparameters (cf. Appendix) to find
Although the p-specifii models outperform the generalist agent in their own finetuned environments, the generalist agent can achieve good performance across a range of memory strengths (Fig. (5). is worth noting that this model now enables use in settings where memory strength is entirely unknown
8

---

Published as a conference paper at ICLR 2025
1000
1.0
Registant Fraction Number of Gells
800
0.8
002
Number
600
602
Memory strength,
0.6
400
20
40
60
80
100
10
00
80
100
Resistant Fraction
0.21
0.2
20
40
60
80
100
Time (hours)
Time (hours)
Figure 5: Left: Policy is robust observation noise, tested in memoryless environment. To represent potential measurement errors in the clinical setting, noise is drawn from a normal distri- bution with standard deviation a and added each state before given the agent. Despite large amounts of observation noise, the agent was able to drive population reduction and maintain a sim- ilar resistant fraction. Each trace represents the mean over 10 trajectories, with the standard error represented by the shaded region. Right: Learned policy is general and robust to changes in memory strength. Every 20 decision steps the memory strength is reset new value, drawn from uniform distribution over the interval [0.6, 1]. Agent is able to quickly adapt to mitigate population growth in the new environment.
and possibly dynamical or noisy in nature. Experimentally, we find this agent very robust to perturbations of various types: namely large changes in μ and large amounts of observation noise. In both cases, the learned policy employed by the agent is able to successfully adapt and continue to minimize population growth (Fig. 5).
RELATED WORK
The use of reinforcement learning in control of biological systems, especially in dosage control, has been investigated by others. We will briefly highlight the most relevant prior work this section. (Padmanabhan et al., 2017) studies a deterministic memory-less model of cancer dynamics with additional "immune" and "normal" cell types included. The number of tumor cells is used as discrete RL state, and hence tabular method is used. In (Engelhardt (020)) coupled SDE governs dynamics of d-many phenotypic subpopulations and the RL state is composed of measurements of individual phenotype populations. Although the specific dynamical model varies, prior work often requires unrealistic inputs to define the RL observation. An important contribution of the current work is in advancing the model class itself while maintaining clinical relevance by using easily measured quantities and providing policy that generalizes across memory strengths.
FUTURE WORK
We utilize frame-stacking (Miih al.) 2013) encapsulate the history of the agent's trajectory, but in future work we will test the use of recurrent policies to capture more nuanced long-term effects which may further improve performance. We also plan extend our deterministic framework the stochastic setting, as population heterogeneity is known to further complicate the control process. This work demonstrates the benefits of combining OC and RL to simplify the control problem, show- ing how their frameworks can be effectively integrated. This connection can be further developed by using the memoryless OC solution to enhance RL training through reward shaping (Ng et al. 1999
Wiewiora 2003 Adamezyk al. 2023) or pre-training (Uchendu et al. 2023) techniques.
In this work we present a strong proof of concept for RL-guided drug dosing, which we hope will motivate further experimental work on adaptive therapies. Since preparing drug doses of arbitrary concentration can be challenging, the derived bang-bang control simplifies the practical implemen- tation of policies. While results show that the optimality of such policies is robust against

---

Published as a conference paper at ICLR 2025
memory strength, switching rates, and dose-response curves, their optimality in the clinical setting may be influenced by additional factors such as patient well-being, interactions with the host sys- tem, and penalties for drug exposure. Addressing these constraints highlights the need to bridge the "sim-to-real" gap through both novel computational approaches and evaluations on real-world data.
Inspired by its success in robotics applications (Akkaya et al. 2019), domain randomization may prove useful in addressing such computational challenges. By training agents in a diverse set of simulated environments with randomized parameters (e.g., switching rates, dose-response effects, noise levels), policies can become more robust for real-world application. With such general agents successfully trained, an offline RL problem (Levine et al. (220)) with warmstarting (Uchendu al.j |023] Agarwal et al., 2022) could be formulated once real-works data is available. We hope that this approach can provide path toward generalization, thus enabling more reliable application of RI. policies in diverse real-world contexts.
DISCUSSION
In this work, we study the control of highly non-Markovian model of adaptive cellular growth dy- namics using deep reinforcement learning. Although we focus here specific parameterization most relevant to cancer, we expect this methodology to be applied successfully to other scenarios, including resistance development in bacteria. We find that deep RI. is capable of recovering the known optimal policy for the memoryless case and can successfully find policy for memory-based systems which prevents proliferation, all without access the underlying model parameters. Fur- thermore, we obtain policy that is robust to measurement noise and memory strength perturbations.
Exact methods from optimal control require direct access the population's resistant fraction and underlying model parameters to determine switching rates. In practice, experimentally measuring the growth rate of pathogenic population is simpler than determining its corresponding resistant fraction. Surprisingly, we find that model-ffre reinforcement learning can provide successful poli- cies directly from the growth rate itself, making it a promising method for use in clinical settings.
ACKNOWLEDGEMENTS
JA acknowledges funding support the NSF through Award No. PHY-2425180; the use of the su- percomputing facilities managed by the Research Computing Department at UMass Boston; and fruitful discussions with Rahul V. Kulkarn and Stas Tiomkin. JCK acknowledges support from the Computational Biology Department at CMU, and from Shiladitya Banerjee in the Department of Physics at CMU. This work is supported by the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interac- tions, http:: / / /iaifi.fiii// org/). This project developed out of the hackathon the 2024 Summer School hosted by the Institute for Artificial Intelligence and Fundamental Interactions (IAIFI..
REFERENCES
Murat Acar, Attila Becskei, and Alexander van Oudenaarden. Enhancement of cellular memory by reducing stochastic transitions. Nature, 435(7039):228-22) 2005.
Jacob Adamezyk, Argenis Arriojas, Stas Tiomkin, and Rahul V Kulkarni. Utilizing prior solutions for reward shaping and composition in retropy-re reinforcement learning. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 6658-6665, 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Ad- vances in neural information processing systems, 35:28955-28971 2022.
Sang Hee Ahn, EunSook Kim, Chankyu Kim, Wonjoong Cheon, Myeongsoo Kim, Se Byeong Lee, Young Kyung Lim, Haksoo Kim, Dongho Shin, Dae Yong Kim, et al. Deep learning method for prediction of satient-specific dose distribution breast cancer. Radiation Oncology, 16:1-13, 2021.
10

---

Published as a conference paper at ICLR 2025
Ilge Akkaya, Marcin Andrychowicz. Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glemn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXfv:1910.07113. 2019.
Nathalie Q. Balaban, Jack Merrin, Remy Chait, Lukasz Kowalik, and Stanislas Leibler. Bacterial persistence as a phenotypic switch. Science, 305(5690):1622-1665 2004. ISSN 00368075.
Shiladitya Banerjee, Klevin Lo, Nikola Ojkic, Roisin Stephens, Norbert F. Scherer, and Aaron R. Dinner. Mechanical feedback promotes bacterial adaptation antibiotics. Nature Physics, 17(3): 403-409, 2021. ISSN 1745-2481.
Jessica M.A. Blair, Mark A. Webber, Alison J. Baylay, David O. Ogbolu, and Laura J.V. Piddock. Molecular mechanisms antibiotic resistance. Nature Reviews Microbiology, 13(1):42-51, 2015. ISSN 17401534.
Alessandra Bonfanti, Jonathan Louis Kaplan, Guillaume Charras, and Alexandre Kabla. Fractional viscoelastic models for power-law materials. Soft Matter, 16(26):6002-600) 2020.
Meghna Das Thakur, Fernando Salangsang, Allison S. Landman, William R. Sellers, Nancy K. Pryer, Mitchell P. Levesque, Reinhard Dummer, Martin McMahon, and Darrin D. Stuart. Mod- clling vemurafenib resistance melanoma reveals a strategy to forestall drug resistance. Nature, 49417436)251--555, 2013. ISSN 1476-4687.
Andrew Dooley, Avinash Gupta, and Mark R Middleton. Ongoing Response in BRAF V600E- Mutant Melanoma After Cessation of Intermittent Vemurafenib Therapy: A Case Report. Tar- geted Oncology, 11 11:57-5633, 2016.
Dalit Engelhardt. Dynamic control of stochastic evolution: a deep reinforcement learning approach to adaptively targeting emergent drug resistance. Journal of Machine Learning Research, (203):1-38) 2020.
Matthias M Fischer and Nils Bluethgen. On minimising tumoural growth under treatment resistance. Journal of Theoretical Biology, 579:111716.1 2024.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learning Representations, 2018.
Kit Gallagher, Maximilian A.R. Strobl, Derek S. Park, Fabian C. Spoendlin, Robert A. Gatenby, Philip K. Maini, and Alexander R.A. Anderson. Mathematical Model-Driven Deep Learning Enables Personalized Adaptive Therapy. Cancer Research, 84(11):1929-19) 1944,, 2024. ISSN 15387445.
Roberto Garrappa. Numerical solution of fractional differential equations: A survey and a software tutorial. Mathematics, 6(2):16, 2018.
M. I. Gomoyunov. On the Relationship Between the Pontryagin Maximum Principle and the Hamil- ton Jaco-Jccoi--aaa -elllan Equation in Optimal Control Problems for Fractional Onder Systems. Dif- ferential Equations, 59(11); 59(11)11520-152 1520-156. 2023. ISSN 16083083.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: /8/2.05980 2018.
Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H Huang. Dhruva Tirumala, Jan Humplik, Markus Wulfmeier, Saran Tunyasuvunakool, Noah Y Siegel. Roland Hafner, et al. Learning agile soccer skills for a bipedal robot with deep reinforcement learning. Science Robotics, :89)eadi8002, 2024.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: 555.00222222 2015.

---

Published as a conference paper at ICLR 2025
Guillaume Harmange, Raúl A Reyes Hueros, Dylan L Schaff, Benjamin Emert, Michael Saint- Antoine, Laura Kim, Zijian Niu, Shivani Nellore, Mitchell Fane, Gretchen M Alicea, Ashani T Weeraratna, M Celeste Simon, Abhyudai Singh, and Sydney M Shaffer. Disrupting cellular memory to overcome drug resistance. Nature Communications, 14, 2023.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning- In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018
Moein Khalighi, Guilhem Sommeria-Kien Didier Gonze, Karoline Faust, and Leo Lahti. Quan- tifying the impact of ecological memory on the dynamics of interacting communities. PLoS Computational Biology, 18(6):1-21, 2022. ISSN 15537358.
Josiah Kratz and Shiladitya Banerjce. Gene Expression Tradeoffs Determine Bacterial Survival and Adaptation Antibiotic Stress. PRX Life, 2:1-15, 2024.
Niraj Kumar, Gwendolyn M Cramer, Seyed Alireza Zamani Dahaj, Bala Sundaram, Jonathan Celli, and Rahul V Kulkarni. Stochastic modeling of phenotypic switching and chemoresistance in cancer cell populations. Scientific reports, 9()):0045, 2019.
Ajay Larkin, Colin Kunze, Melissa Seman, Alexander Levashkevich, Justin Curran, Dionysus Morris-Evant Sophia Lemieux, Ahmad $ Khalil, and Kaushik Ragunathan. Mapping the dynam- ics of epigenetic adaptation in s. pombe during heterochromatin misregulation. Developmental cell, 59(16):2222-22333 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Brian N. Lundstrom, Matthew H. Higgs, William J. Spain, and Adrienne L. Fairhall. Fractional differentiation by neocortical pyramidal neurons. Nature Neuroscience, 11(11):1335-130) 2008
ISSN 15461726.
Roland Mathis and Martin Ackermann. Asymmetric cellular memory in bacteria exposed to antibi- otics. BMC Evolutionary Biology, 17(1):1-14, 2017. ISSN 14712148.
Riccardo Miotto, Li Li, Brian A Kidd, and Joel T Dudley. Deep patient: unsupervised represen- tation to predict the future of patients from the electronic health records. Scientific reports, 6(1); 1-10, 2016
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, Nature, 51 518(7540):529-533 2015.
Gatien Moriceau, Willy Hugo, Aayoung Hong, Hubing Shi, Xiangju Kong, Clarissa C. Yu, Richard C. Koya, Ahmed A. Samatar, Negar Khanlou, Jonathan Braun, Kathleen Ruchalski, Heike Seifert, James Larkin, Kimberly B. Dahlman, Douglas B. Johnson, Alain Algazi, Jeffrey A. Sosman, Antoni Ribas, and Roger S. Lo. Tunable-Combinatoria Mechanisms of Acquired Re- sistance Limit the Efficacy of BRAF/MEK Cotargeting but Result in Melanoma Drug Addiction. Cancer Cell, 27(2):240255 256, 2 2015. ISSN 1535-6108.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, volume 99, pp. 278-287, 1999.
Regina Padmanabhan, Nader Meskin, and Wassim M Haddad. Reinforcement learning- sramiig-bbseed con- trol of drug dosing for cancer chemotherapy treatment. Mathematical biosciences, 293: 1-20, 2017.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines:; Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021.
12

---

Published as a conference paper at ICLR 2025
Leonie Ringrose and Renato Paro. Epigenetic regulation of cellular memory by the polycomb and trithorax group proteins. Annual Review of Genetics, 184412-445, 2004.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1700770777.7777 2017.
Heike Seifert, Rosalie Fisher, Juan Martin-Libere Kim Edmonds, Peta Hughes, Komel Khabra, Martin Gore, and James Larkin. Prognostic markers and tumour growth kinetics melanoma patients progressing on vemurafenib. Melanoma Research, 26(2):138 144, 2016.
Sydney M. Shaffer, Benjamin Emert, Raúl A. Reyes Hueros, Christopher Cote, Guillaume Harmange, Dylan L. Schaff, Ann E. Sizemore, Rohit Gupte, Eduardo Toree Abhyudai Singh, Danielle S. Bassett, and Arjun Raj. Memory Sequencing Reveals Heritable Single-Cell Gene Expression Programs Associated with Distinct Cellular Behaviors, Cell, 182(4))947-959) 2020. ISSN 10974172.
Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual mdps for continual learning. In Learning for Dynamics and Control Conference, pp. 608-623. PMLR, 2022.
Kenichi Suda, Kenji Tomizawa, Hirotaka Osada, Yoshihiko Machara, Yasushi Yatabe, Yoshitaka Sekido, and Tetsuya Mitsudomi. Conversion from the "oncogene addiction" to "drug addiction" by intensive inhibition of the egfr and met in lung cancer with activating egfr mutation. Lung Cancer, 76(3):292-299, 2012. ISSN 0169-5002.
Chong Sun, Ligin Wang. Sidong Huang, Guus J.J.E. Heynen, Anirudh Prahallad, Caroline Robert, John Haanen, Christian Blank, Jelle Wesseling, Stefan M. Willems, Davide Zecchin, Sebastijan Hobor, Prashanth K. Bajpe, Cor Lieftink, Christina Mateus, Stephan Vagner, Wipawadee Gr- ernrum, Ingrid Hofland, Andreas Schlicker, Lodewyk F.A. Wessels, Roderick L. Beijersbergen, Alberto Bardelli, Federica Di Nicolantonio, Alexander M.M. Eggermont, and Rene Bernards. Reversible and adaptive resistance BRAF(V600E) inhibition in melanoma. Nature, 508(7494): 118-122, 2014. ISSN 1476-4687.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-start re- inforcement learning. Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 34556-345 34583. PMLR, 23-29 Jul 2023.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Neil Vasan, José Baselga, and David M. Hyman. A view on drug resistance in cancer. Nature, 575 (7782):299-309 2019. ISSN 14764687.
Eric Wiewiora. Potential-based shaping and Q-value initialization are equivalent. Journal of Artifi- cial Intelligence Research, 19:205-208, 2003.
Christopher Witzany, Jens Rolff, Roland Regoes, and Claudia Igler. The pharmacokinetic- pharmacodynamie modeling framework tool to predict drug resistance evolution. Micro- biology, pp. 1-37, 2023.
Denise M. Wolf, Lisa Fontaine-Bodde Ilka Bischofs, Gavin Price, Jay Keasling, and Adam P. Arkin. Memory in microbes: Quantifying history-dependent behavior in a bacterium. PLoS ONE, 3(2), 2008. ISSN 19326203.
13

---

Published as a conference paper at ICLR 2025
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tie- Yan Liu. Fully parameterized quan- tile function for distributional reinforcement learning. Advances in neural information processing systems, 32, 2019.
A REINFORCEMENT LEARNING DETAILS
We adapted DQN from Gaable-Baseliness (Rafiin et al. 2021) with Double DQN action selection rule (Van Hasselt al. 2016). We train the RL agent for x 10 total environment steps. We limited the episodes to be of length 10* steps (corresponding to 100 hours in simulation). An MLP of fixed size (2 hidden layers with 64 dimensions each and ReLU activation) was used parameterize the Q-function.
Regarding the environment and training, we list several key implementation choices: At the begin- ning of an episode, the state is zero-padded to always be of length K = 5. We experimented with adding a penalty for allowing the number of cells to increase beyond the initial amount upon termi- nation, but found this was not necessary for successful training. Borrowing terminology from the literature on Atari environments (Mnih et al. 2015), we stack K frames (previous cost values) to form the RL agent's 's state vector. Although initially we let K be p-deppenentt hyperparameter, found constant choice of K to work well across the values of μ studied.
We find exponentially decaying the exploration parameter work better than the typical linear annealing (with constant, positive final E) scheduling. We conjecture that this is because non non-greedy actions be quite detrimental (causing the environment terminate), and exponentially decaying & ensures some exploration continues to occur but with increasingly fewer random actions. To ensure the agent is not overly myopic (especially for such long episodes)
found large discount factor of = 0.999 (corresponding to an effective horizon of H = (1 - - = 10 to be helpful. When training the agent, we wait until the completion of one rollout episode, and take many gradient steps as environment steps have occurred.
A.1 HYPERPARAMETERS
We find that sweeping over a range of hyperparameters (as shown Table 2 did not have signif- icant effect performance, though for reproducibility we list the final hyperparameters used (for = 0.7) below for DQN and FQF experiments, respectively. (N/A indicates the value was not swept over, and fixed to the finetuned value.) The discount factor y 0.999, gradient steps per update (1), frames stacked (5) and buffer size (100,000) are fixed throughout.
Table 2: Hyperparameters for Double DQN
Hyperparameter
Sweep Values
Finetuned Value
batch size exploration rate
16, 32, 64 0.01 0.2
32
0.05
learning rate
10 10-3
3.60 x 10-4
target update interval
1,000, 5,000, 30,000,
1,000
Table 3: Hyperparameters for (NoisyNet) FQF
Hyperparameter
Sweep Values
Finetuned Value
batch size
8, 64 0.02 0.2
8
exploration rate learning rate
0.2
5 x 10 - 5 10-3
2.7 x 10-3
target Polyak averaging n-step TD
0.001, 0.005, 0.01
0.005
1,2,4,32 3,
1
entropy coeff.
0, 0,0.0005, 0.001,0.003
0.003
14

---

Published as a conference paper at ICLR 2025
B
FURTHER EXPERIMENTAL RESULTS
Effect of Simulation Timestep Best Model's Performance
1
450-
Final Number of Cells
440
430
420
410
10a
Simulation Time (1/A)
Figure 6: We find that the simulation time can have a significant effect on RL performance. For each choice of A, we run random sweep of size 30 over various hyperparameters, selecting the highest-performing run for each A. Since smaller values of require longer compute-times for simulations, there is a tradeoff between the amount of time (also, the maximum pulsing frequency, relevant clinical settings) and the best performance. have chosen to use 0.01 throughout.
Robustness Switching Rates
0.12
880
860
0.11
increasing
840
Rate Value,
820
0.10
Difficulty
780
0.09
760
740
0.08
0.08
0.09
0.10
0.i1
0.12
Rate Value,
Figure 7: Robustness experiment on changing switching rates 8. The black box in the center highlights the rates which the agent was trained. On each axis, the rate is varied by +20%, representing a large range of biologically plausible parameters. As a increases (or 5 decreases). the cells are more often in the resistant state (cf. Fig. [ making the control problem more difficult for the agent.
15


Reviews:
[{"review_id": "cTcf9PRECx", "reviewer": "Reviewer_mj4C", "summary": "The paper considers the problem of controlling cell-population by adjusting drug dosage. In line with medical research, the authors propose a dynamics model with memory (non-Markov) for this setting, based on ODEs, and derive the theoretical optimal controller for drug administration, that is found to be a \"bang bang\" controller. Using this insight, the authors formulate this problem as an RL problem, then use RL with reduced action-space to solve it. The method is evaluated against a few naive baselines on a simulated environment, showing both superiority over the baselines, robustness to noise and changes of the memory of the system.", "questions": "1. How many random seeds/trajectories did you use in each experiment? What is the faded area in Fig. 4?\n\n2. Have you tested how robust is the resulted policy to parameter noise of the model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Presentation:\nThe paper is well written, and for me, it was easy to understand, in addition, it cites many relevant papers across all fields (RL, control theory, medicine).\n\n### Methodology:\nI think this paper bridges some of the gap between real-life problems to RL; while it is more applicative in terms of RL, it suggests how to do it right and efficient. The analysis in Sec. 2 enables RL to solve the problem (still need to show it). The experiments are extensive and show results for a few settings, and how it affects the model results.", "weaknesses": "### Presentation:\nIn the current form, the paper is written as two papers: One that formulates the problem and another that utilizes RL to solve the problem. For my opinion, it should have a more solid story, which would require some rephrasing. Considering the title, I suggest noting the difficulties of using RL for the raw problem, and then present your analysis for the optimal control which provides a solution which implies a much simpler problem for RL.\n\n### Experiments:\nI think that the naive baselines are not enough here, maybe add an RL variant that operates over the continuous action-space, showing the importance of your optimal control analysis?\nAlso, there are missing details for the experiments -- for how long the RL agent is trained? what\n\n### Contribution:\nMy concern here, is that the method is motivated by real world, but eventually tested in simulation. Which shakes the ground of this paper, as it all revolves around the suggested dynamics model. I think another step towards bridging RL and real-life should be added. Since it is less likely to perform experiments that involve real-life environment and control, I would suggest showing empirically that the proposed dynamics model could indeed predict cell population -- using real-life data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fj5rZwD7Vo", "reviewer": "Reviewer_yE9z", "summary": "The paper proposes to model the non-markovian process of adaptive cellular growth dynamics in bacterial and cancerous populations to learn optimal drug dosing using reinforcement learning. It shows successful training of an improved policy for memory-based systems compared to optimal control, which is robustness to observation noise and memory strength perturbations.", "questions": "Do the authors have any intuition on how RL would perform in a continuous drug control scenario? \n\nIf the constructed model might not fully describe the underlying biological dynamics (cf. p.4 l 214f.), causing OC to be inaccurate, how does RL solve this issue? \n\nHow are the objectives of maximizing net death and maintaining the stability of the subpopulations connected? \n\nGiven that learning an RL policy in a specific real-world application would not be feasible (similar to OC), how well does the resulting policy generalize regarding the parameters underlying the assumed model (beyond the memory size)? How can overfitting to the chosen training specification be prevented?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-motivated and regards a relevant real-world application. It is generally well-written, explained, and easy to follow. It provides a formally sound theoretical connection of the problem at hand to optimal control theory, extended to the application of reinforcement learning (including realistically obtainable observations) to tackle current shortcomings. Convincing empirical results show improved adaptability and generalization to varying memory levels of the underlying model.", "weaknesses": "While, in general, the problem is well-explained and elaborated, an example or illustration might have been useful. Also, the figure placement and referencing should be improved to further ease readability. The overall objective of reducing the population by minimizing growth to negative growth could be stated more clearly in the former sections (e.g., 2.1). In Fig. 2 (right), a comparison to the memory-less or optimal control approaches used as a baseline later on could have been insightful. \n\nMinor comments: \n\n- p.2 l.89 no middle part in Fig. 2 (referencing the wrong figure?)\n- p.6 l.308 RL uses gradient ascent on the reward, not SGD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWRKzm6pN5", "reviewer": "Reviewer_9huZ", "summary": "The paper addresses the challenge of controlling and driving cell populations, such as cancer cells, toward extinction in environments where exhibiting adaptability and memory of past stressors. This is particularly difficult in systems with unknown parameters or those displaying non-Markovian dynamics such memory-based systems. The authors propose a new memory-based model for switching population dynamics, develop an optimal solution for the memoryless case, and apply model-free deep reinforcement learning (RL) to develop effective drug dosing strategies for cell populations with memory. The proposed approach is tested by comparing its performance against known exact solutions in controlled scenarios, demonstrating that deep RL can accurately recover these solutions. Additionally, the approach is evaluated in more complex environments where the memory strength of cell populations varies. The results demonstrate the method’s robustness in managing dynamic, memory-influenced systems.", "questions": "- Given that mammalian cancer cells are targeted as an application scenario which generally have a cell division rate of about 24h, how realistic are the assumptions in the paper made about the delta parameter and the control frequencies applied by the pulsating policies?\n- Can you give more biological motivation for the experiment with the switching memory strengths in Fig. 4 (right)?\n- How important is the switch to FQF from DQN in section 5? How well would DQN do on the more challenging problems considered there? How important is the use of Noisy Nets for exploration over just using annealed epsilon-greedy? These points are not examined, which makes an assessment of the justification for the methodological choices difficult.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Very clear exposition, introduction to the problem and model development\n- Contribution of a novel model useful for RL-based optimization of drug dosing\n- Useful insights from control theory that allow simplification of the problem\n- Optimal solution given for the memoryless case which serves as a sanity check for the RL-based method\n- Convincing results of the RL-based solution compared to three baselines\n- Clinically highly relevant problem overall", "weaknesses": "- Limited methodological contribution: author's mainly apply existing RL methods to a newly developed simulation model (I see the model itself and the insights gained about it as main contribution)\n- Not clear whether assumed time-scales are realistic", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UhnOCvSd8", "reviewer": "Reviewer_JJJP", "summary": "This paper studies how non-markovian cellular population dynamics can be chemically controlled through the lens of RL.\nWhile prior work focused on markovian assumptions or unrealistic assumptions, this paper tries to apply deep RL (DQN-based methods) to assess how effective this would be to design drug delivery patterns that optimally control the pathogen cellular dynamics.\nThe results of the RL-based method are encouraging: the authors show that in the markovian case, the algorithm is capable of finding the analytical solution, and in the non-markovian case a good handling of the cellular growth rate can be achieved.", "questions": "- Why choose DQN and not another algorithm? SAC and others can also be trained in discrete action spaces. It would be nice to have a notion of how DQN compares to other algorithms (in supp. materials).\n\nMy main questions have to do with the feasibility of the proposed technique:\n- How feasible is it to measure the cell growth periodically in real life settings? I guess one cannot grow cells very frequently and the amount of input data could limit the efficiency of the method proposed. Also, how feasible is it to provide drugs at the proposed rates, and would tighter constraints actually impact the policy much (could it be that a tight constraint on the frequency of the drug administration actually drives the policy to another local minimum?)\n- I did not really understand if the plan here would be to train an algorithm offline on simulated data and test it in the real world (sim2real) or if the purpose of this method is to show on simulated data that RL can be used to solve the problem but this will require training the algorithm on real data. Or is the goal to get a standardized drug administration schedule that would fit all cases? \n- If the goal is sim2real, some evidence or a path forward to the applicability of this technique would be a nice add-on.\n- If the goal is online training, one could maybe consider offline reinforcement learning to help solve this problem?\n- How can one assess how good and representative the growth model is? Is there any evidence that sim2real would work?\n- Alongside the previous points: is 300K steps for training realistic in real life?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I've found the paper compelling and engaging. It is well written, goes through the necessary background effectively, has enough math to thoroughly back its claims and build the necessary background.", "weaknesses": "- The paper mentions that a state encoding the memory is required, transforming the non-markovian dynamic into a Markovian dynamic. The usage of recurrent networks (or perhaps other models like transformers?) could be used to solve that problem without encoding for these states. I find this idea compelling. I wonder if using an RNN is enough material for a follow-up paper, even though it is crucially missing from this one. Hence, not having it within this publication could actually be detrimental for the publicity around this publication altogether.\n- Only one family of RL algorithms is being used. Although the authors show that it is robust enough to discover the solution of the problem, it'd be nice to see how DQN compares to other techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)