Active Observing in Continuous-time Control
Authors: Samuel Holt
Alihan Hüyük
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: Sensing
Model-Based Reinforcement Learning

Abstract: The control of continuous-time environments while actively deciding when to take costly observations in time is a crucial yet unexplored problem, particularly relevant to real-world scenarios such as medicine, low-power systems, and resource management. Existing approaches either rely on continuous-time control methods that take regular, expensive observations in time or discrete-time control with costly observation methods, which are inapplicable to continuous-time settings due to the compounding discretization errors introduced by time discretization. In this work, we are the first to formalize the continuous-time control problem with costly observations. Our key theoretical contribution shows that observing at regular time intervals is not optimal in certain environments, while irregular observation policies yield higher expected utility.  This perspective paves the way for the development of novel methods that can take irregular observations in continuous-time control with costly observations. We empirically validate our theoretical findings in various continuous-time environments, including a cancer simulation, by constructing a simple initial method to solve this new problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner. Although determining the optimal method remains an open problem, our work offers valuable insights and understanding of this unique problem, laying the foundation for future research in this area.

Active Observing in Continuous-time Control
Samuel Holt University of Cambridge sih31@cam, ac.uk
Alihan Hüyük University Cambridge ah2075@cam. ac.
Mihaela van der Schaar University of Cambridge mv4720cam. ac uk
Abstract
The control of continuous-time environments while actively deciding when to take costly observations in time is crucial yet unexplored problem, partieularly relevant to real-world scenarios such as medicine, low-power systems, and resource management. Existing approaches either rely continuous-time control methods that take regular, expensive observations in time or discrete-time control with costly observation methods, which are inapplicable continuous-time settings due the compounding discretization errors introduced by time discretization. In this work, we are the first to formalize the continuous time control problem with costly observations. Our key theoretical contribution shows that observing at regular time intervals not optimal certain environments, while irregular observation policies yield higher expected utility. This perspective paves the way for the development of novel methods that can take irregular observations in continuous- costly observations. We empirically validate our theoretical findings in various continuous- tiemu environments, including cancer simulation, by constructing a simple initial method to solve this new problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner. Although determining the optimal method remains an open problem, our work offers valuable insights and understanding this unique problem, laying the foundation for future research in this area.
control with
Introduction
The problem of continuous control with costly observations is ubiquitous with applications in medicine, biological systems, low power systems, robotics, resource management and surveillance [Yoshioka and Tsujimura, 2020, Brunereau et al., 2012, Mastronarde and van der Schaar, 2012]. A setting that shared across all these domains is that decision-male needs continually control (e.g., chemotherapy dosing, quantities of food, data to transmit, etc.) whilst deciding when to take a costly observation (e.g., a medical computed tomography (CT) scan, measuring the population of a species, measuring bandwidth, etc.). The decision-maker" observing policy must be timely to determine whether the action plan is effective (e.g., errors in treating stage cancer can be fatal [Reinhardt et al., 2019], with further application examples in Appendix B).
In many of these real-world systems (e.g., medicine, low-power systems, resource management), an offline setup is beneficial as it enables catisioonmmkers to learn control policies without incurring excessive costs risking adverse effects. Where an offline setup refers to learning policy from a previously collected dataset of statc-action trajectories, without interacting with the environment [Argenson and Aulac-Arnold, 2020]. The development of novel methods that can take irregular observations in continuous-ting control with costly observations, within an offline setup, provides a safe and cost-effective way to improve real-world decision-maker processes.
Recent work falls into three main categories. First, sensing approaches determine when to informa- tively observe to identify an underlying state, but unable continually control. Second, planning approaches only continually control and have the restrictive assumption that the observing schedule is
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

given given-most often at regularly fixed observation time interval. While these methods can optimally solve many challenging tasks, with regular, frequent observations of the state, frequent observations are overly costly with many being unnecessary, and where the optimal regular observation interval is unknown.
Third, discrete monitoring approaches control and determine when to observe only for discrete time. These are fundamentally incompatible with continuous-time systems which can exhibit irregular observations in time and states that evolve on different time scales. Moreover, these rely on a discretization interval being carefully selected use but still suffer from time discretization errors that can compound, leading to unstable and poor- poor-performing controllers [Yildiz et al., 2021]. For example, in medicine, a crucial test may be administered too late, and therefore delay updating the treatment dosing which can be fatal when treating patients for cancer [Geng et al., 2017].
Contributions: ① We are the first formalize the problem of continuous-timg control whilst deciding when to take costly observations. Theoretically, we show that regular observing in continuous time with costly observations is not optimal for some systems and that irregularly observing can achieve higher expected utility (Section 2). ② Empirically we verify this key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations. We construct a simple initial method to solve this new problem, called Active Observing Control. This uses heuristic threshold on the variance of reward rollouts an offline continuous- time moodel-based model predictive control (MPC) planner (Sections 4 and 5.1). However, the optimal method is still an open problem, and we leave this for future work. We gain insight into this unique problem, uncovering how our initial method is capable of irregularly observing, and identifies observation frequencies that correlate to different stages of cancer fixed observing frequencies of clinicians. Further, we demonstrate how it can avoid discretization errors in time, thus achieving a better utility as result (Section 5.2), and confirm its robustness to its threshold hyperparameter (Section 5.3).
We have released a PyTorch [Paszke et al., 2019b] implementation of the code at and have broader research timeControl.
group codebase at www.m//aaooo///ttttteeeeeinnnnna
2 Problem
States & actions. For a system with state space S = Rds and action space A = RRA,, the state time E 14 is denoted as s(t) = (t)...... s...,,ss(( (t)] S and the action at time E E is denoted as att) action trajectory E AA are both functions of time, where an individual state s(I) E S or an individual action al) E A are points on these trajectories. practical applications, action values are usually bounded by actuator's limits; hence we also restrict the action space a Euclidean box 'dA
...,, ad, (t) E A. We elaborate that state trajectory s S and an
A = amin [max]
Environment dynamics. The dynamics of the system are given by ds(t)/dt = f(s(t), a())) and the system is stationary. We consider the setting where the true state trajectory s is la- tent and, at a given time I, only a noisy ob- servation =()) x(() + e(l) E S can be ob- served, where e(t) ~ NO o²) is Gaussian noise with zero mean and standard deviation o. and E(t) IL e(t',, Vt, t' E R4. We denote as a tuple (t, (t), at)) a discrete sample taken at time t, and with h = {(tz, 2(t)), alt,))) E H a history of samples taken at times (t,) where H Upe OR+ x S x A)" is the space of all possible histories.
Algorithm Policies p, æ interacting with the en- vironment
1: f1 = 0, = (11, (f1),
2: for i E 21.
3: Schedule next observation: 4441
4 + p(h:)
4: Execute actions: a(I) T(hi, l - 4,) for
E [t,, t++1)
5: Take an observation: h41 = hy U
Policies. Policies consist of an observing policy p: H + R and a decision policy π H R - A. These policies interact with the environment as described in Algorithm We denote with A1+1 =
ty+1 -t,, ty. where j is a dummy variable.
2

---

Objective, Suppose we are given reward function r: S + R to maximize, and each observation has fixed cost R. Then, the utility achieved by a policy up a time horizon T is given by
u = /
ft i, E [0,
(1)
Cost C
Reward
Our objective is to find the optimal observing and decision policies ' ,², arg max,, B[ given an offline dataset of histories D = { } but without knowing ʃ and a, or having online access to the system described by f and o.-
2.1 Taking regular observations in time is not optimal for some systems
This is a highly non-trivial task since, in many cases, the obvious solution of taking regular obser- vations in time is not optimal, and rather taking irregular observations achieves a higher expected utility- point we make formally in Proposition 2.1.
Proposition 2.1. For some systems, is not optimal to observe regularly-that is 3f, oe, v, F, C, N:: p (h) ph').
Proof. Full proof is in Appendix C. However, we present the following sketch. Consider the task of maximizing the utility U given a reward function r(s, a,
((s-)))-(ss()-eee -a(() > 0}, observation cost > 0 for the specific system ds/di = s(1) with observation noise (a, > 0). Intuitively this reward function only provides a positive reward of #1 at the end of the episode (1 = T) if the chosen action a(T) has the same sign as the unobserved state s(T), otherwise, the reward is O. Therefore, this is maximized when the sign of the correct state s(f) identified. Since this system defined by ds/ dtt = s(t), we know that state solutions are of the form st) = s(O)e.. Thus, to determine the sign of stt), since et is always strictly positive, the sign of the state is determined by the initial condition s(0). Our observing policy can observe any anywhere inside the interval of [o, T7. We prove that the optimal observing policy, p² cannot be a regular observing policy by showing that, for each p (rreg.)), there exists at least one (iineg.() with E 2 2 that achieves higher expected utility, where I is the number of observations taken. A higher expected utility can be achieved by an irregular observing policy that takes all observations at the end of the episode, i.e. p² (hhs) = T - t, for i E (2...., E..,haht that is when =(0)
the signal-to-noise ratio (s(t)²/ /o?) is the highest, i.e. arg max, of :t E [0, T). which occurs when ( - T.
This motivates us to develop new continuous-time methods for solving this problem, where a solution should be able to flexibly adapt to take irregular observations when it achieves a higher expected utility. Intuitively, based on the above system, it is more informative to take an observation when the predicted state change magnitude is larger, which can occur faster for large state velocity é- indicating observations should taken more frequently. We later show experimentally illustrative cases (Section 5.2) of how irregularly observing can lead to better performance.
3 Related work
Table 1 summarizes the key differences between Active Observing Control from related approaches to planning and monitoring in reinforcement learning (RL). Moreover, we provide an extended related work in Appendix D, which includes discussions of the benefits of offline RI. model-based RI., why model predictive control is our preferred action policy, event-based sampling, Semi-MDP methods, and Linear Quadratic Regression (LQR) & Linear Quadratic Gaussian (LQG) methods.
Sensing approaches have been proposed of when to optimally take an observation in both discrete time [Jarrett and Van Der Schaar, 2020] and continuous time [Alaa and Van Der Schaar, 2016, Barfoot et al., 2014]-where where their goal is to identify an underlying state. However, these approaches cannot also continuously control the system. In contrast, Active Observing Control seeks to both actively observe and control the system.
Planning approaches only focus on optimal decision policies n, and therefore observing has to be provided by schedule, which is often assumed to be at regular constant interval in time A
3

---

Table 1: Comparison with related approaches to planning and monitoring in RL. Plots for the corresponding state trajectories for each approach are illustrated in Figure 1. Our initial method, Active Observing Control, is the only method for continuous-time time control whilst deciding when to
observe with smoothly-varying states.
Appeoarh
Deralin
Rovirnerent Dyunies
State Trajectrkos
Polices
Polley Bormulativn
Disorte Senceg
Jaare Van Der Sevee, 2011)
Gbantty-on
Blanke Plasing
[Claa di 2011
Decsde-lat lane
ne Flnuoii
JAla Van al. 3021)
Desisin deucving
Centraans Senuing
2016)
Contossnn dime
Chuntty-asty
Cantinnnn Plasing
ofidic 3021]
Contaannn-ine:
Norgaa 16601,000) aeaeD no.4 2Tn, Mloitwatealll
s(mbuueedd m elf (Q
Seri-oontranes Moviarring
[flug ot 2019)
Contooonn
chueving
Active Ulterving Control
(Chem)
Decason & ohserving
Sus Deain vidie
Figure I: Comparison of related approaches of an environment's latent state s(t) s(t)- where green lines represent observing times and red dots observations that have observation noise.
Ahtthat is, observations are not actively planned [Yildiz et al., 2021]. Within planning approaches, there exist many discrete-time approaches [Chua et al., 2018, Mnih et al., 2013, Williams al., 2017] and recently more continuous-time approaches [Du et al., 2020, Yildiz et al., 2021J. Specifically, Yildiz et al. [2021] presented a seminal online continuous-time model-based RL algorithm, leveraging a continuous-time dynamics model that can predict the next state at an arbitrary time s(t). However, all these methods are unable to plan when take the next observation, in contrast to Active Observing Control which is able to determine when to take the next observation whilst planning an action trajectory.
Monitoring approaches consist of both a decision policy т and an observing policy p; however, existing approaches only consider the discrete-time and discrete-state setting with observation costs [Sharma et al., 2017, Nam et al., 2021, Aguiar et al., 2020, Bellinger et al., 2020a, Krueger et al., 2020, Bellinger et al., 2020b]. In particular, Krueger et al. [2020] proposed that even in simple setting of a discrete-state multi-armed bandit, computing the optimal time to take the next observation is intractable-therefforr. they must rely on heuristics for their observing policy. Broadly, discrete-time monitoring methods use discrete-time model and propagate the last observed state, often appending either a flag or a counter to the state-action tuple to indicate the time since an observation was taken [Aguiar et al., 2020, Bellinger et al., 2020b]. However, these methods cannot be applied to continuous-time environments or propagate the predicted current state and its associated prediction interval of the uncertainty associated with the state estimate. Moreover, training policy [Aguiar et al., 2020] that decides at each state whether to observe or not, requires discretization of time. Whereas Active Observing Control, which determines the continuous- time interval of when to observe the next state, does not require any discretization of time, and hence it a continuous-time method and avoids compounding time discretization errors.
One approach exists that we term semi-continuous monitoring, where Huang et al. [2019] proposes a discrete-state, constant action policy, that determines when to take an observation in continuous time of a controlled Markov Jump Process. However, this approach is limiting, as it assumes actions are constant until the next sample of the state is taken [Ni and Jang, 2022]-whicd is clearly suboptimal, uses discrete states, and assumes a Markov Jump Process [Gillespie, 1991]. Instead, Active Observing Control is fully continuous in both states and observing controledd by an action trajectory giving rise to smoothly-varying states.
4 Active Observing Control
We now propose Active Observing Control (AOC), an initial method for the defined problem of continuous-time control whilst deciding when to take costly observations. AOC, can plan action trajectories in continuous time and plan when observe next in continuous time. The key idea is to use continuous-time uncertainty-aware dynamics model fo to plan 1) action trajectories and 2) the next time to take an observation, such that it is informative to do so so-Figure 2 provides a block diagram. We adhere to the standard offline model-based setup [Lutter et al., 2021] first learning our dynamics model (Section 4. )), and then using it at run-time with our planning method (Section 4.2).

---

Enviromament
1) Plan action trajeciory
2) Plan next observation time 45+1
a(t)
~ =(4)
x (2(tt), H) = st.
a(()
Exccute actions aft) up 11 and then take observation at 1y41
a(r(1)
1--1
p(a(t:) = A:+1
(f), Vi E [6,,44))
Figure 2: Block diagram of Active Observing Control. An uncertainty dynamics model fa learned from an offline dataset DD of state-action trajectories. At run-time, planning consists of two steps: 1) The actions are determined by a Model Predictive Control (MPC) Planner, and 2) the determined action trajectory a is forward simulated to provide uncertainty a(r(t)) on the planned path reward. We determine the continuous time f to execute the action plan up to, such that (r(()) < т. as in Algorithm 3. We then exccute a(t) Vt E [t,, tx+1) up until the time to take the next observation (t) at.
4.1 Learning a continuous-time uncertainty-aware dynamics model
Fundamentally the goal of offline model-based RL is to learn an accurate dynamics model that can be used for planning from an offline dataset DD [Lutter et al., 2021] In particular, we require a dynamics model that is both uncertainty aware and continuous in time; that is, it can provide prediction uncertainty for the next state future time + 6, i.e. p(z(t + (=((), a(t), time difference input to create a continuous-time dynamics model [Yildiz et al., 2021].
8)). Here we use the
Model-based RL has shown the crucial importance of the performance of learning uncertainty- aware dynamics model that captures both 1) aleatoric uncertainty- due to the inherent stochasticity of the environment (c.g., observation noise and environment process noise) that is irreducible and 2) epistemic uncertainty- due to the lack of data, for a given state-action space (which should vanish when trained on the limit of infinite data) [Chua et al., 2018).
We learn an uncertainty- perceptron (MLP) neural network models that each parameterize multivariate Gaussian distribution- where the ensembling captures the epistemic uncertainty and each model individually captures the aleatoric uncertainty. We note that shown by other works, ensembles of high-capacity neural networks outperform Gaussian process dynamics models, as they have constant-time inference- scaling better to larger offline datasets, while still providing well-calibrated uncertainty estimates LLkshmiiarayaanp et al., 2017 7] and can model more complex functions inciinsvvinnlsivllyyenee non-smooth dynamics [Chua al., 2018].
dynamics model by training an ensemble of high-capacity multi-layer
Precisely, the ensemble consists of M = 5 neural network models that output the parameters for multivariate Gaussian distribution with diagonal covariances, each with 6,, parameters, i.e.,
fe.. Ps... (=(1+) (z))( a()),)) = =N(ue., (z(1), ((),,), E.g., (=(1), a()), d)) where the elements of the diagonal covariances are given by oe (=(t), att), 8), with total of 0 = {0m JM parameters of the ensemble. Moreover, we denote all the models in the ensemble as fa.
To create our ensemble of parameterized Gaussian distribution models, we train each model inde- pendently with unique initialized random weights unique permutation of all the samples in the offline
each base model converges to its own local optima and is more effective than training the models on subsets of the dataset, i.c.. bootstrapping [Lakshminarayanan al., 2017]. Therefore, we minimize the negative log-likelihood for every single model separately, that is
INI-1
[(0m.) = Σ
[oo.. a(t)))) 2(1 E (x a(t;), 6541)
RED 1-1
[no (=(1g), (((1), - =((++1)) log del Ee,, (=(1)) all;), 85+1),
where 6j+1 = Aj+1 1y-1 ly arises from the offline dataset D. Here each parameterized Gaussian distribution model captures heteroskedastic (i.c., the output noise distribution is function of the
5

---

input) aleatoric uncertainty. To capture the heteroskedastic epistemic uncertainty, we combine the individual models as a uniformly weighted mixture model and combine the predictions as po(z(t + 6))(2(t), a(t), d), 0) = + Em= Po.. (z(t + 0))(2(t), (t)), 5), 0m). Therefore, we can compute the effective mean and covariance (diagonal elements) of the mixture
(=(1,,, a()), 6) Iu (=(1), ((t),)) (=()), =o (z(t), (
+ (x()),)))))))))))) - - (((2((),
4.2 Active Observing Control Planning
We desire to use the trained uncertainty-aware dynamics model fo 1) plan the optimal action trajectory a to execute and 2) plan when best to take the next observation. Intuitively, we use the probabilistic dynamics model fo to plan an optimal action trajectory a and execute this until the next observing time- as determined by the heuristic of when the predicted reward distribution over the planned trajectory crosses set threshold, Figure 2. In the following, we detail these two steps.
I) Planning optimal actions. To plan an optimal action trajectory a, specify that any model predictive controller (MPC) planner can be used-that can optimize the action trajectory up to a entire action
specified time horizon H E I... We opt to use MPC planner, as it optimizes trajectory a up to H, does not require the probabilistic dynamics model fo to be differentiable (it uses no gradients), and the reward function r can be changed on the alllooingg changing goals/tasks at run-time. We use our probabilistic dynamics model fe. with the MPC planner of Model Predictive Path Integral Control (MPPI, a zeroth-order particle- based trajectory optimizer [Williams et al., 2017], due to its competitive performance [Wang et al., 2022].
To optimize the action trajectory up to a time horizon H. it discretizes H into smaller constant action time intervals 8, E R+ which are then optimized-when
there are K = x E Z4 time steps in H--ie., (a) = 14 + kō.; k E {O...., K - 1}. It forward simulates several parallel rollouts E Z4. where the next state estimate at each time step is simulated as =( = n. (=((()),, a(((k)), ) 8,) recursively. This requires a state estimate =(1;) to plan from. Therefore, we recompute the planned trajectory when we observe a new sample of the state. Furthermore, we provide MPC MPPI planner pseudocode and details in Appendix E.
2) When to observe. We desire to observe when it most informative to do so. Therefore, we seek to determine the time interval As+1 that the action trajectory a can be followed until an observation taken. Intuitively, we create a reward distribution over continuous time following the MPC planned action trajectory a, which we can follow until the uncertainty crosses a set threshold т E R. hyperparameter to be tuned. Intuitively we prefer the reward uncertainty rather than that of the state because we can achieve the task with a very certain reward despite having uncertain states. For instance, when there might be multiple ways to achieve a task-and we know that our multiple action trajectories guarantee this where we can take any; however, are uncertain about which one. We empirically verify this in Appendix L.
To create the cintinoous-time reward distribution and state distribution, we use our learned proba- bilistic dynamics model fo to forward propagate the last observed state, according to the planned action trajectory a. As noted by others [Chua et al., 2018), computing a closed-form expression for the distribution of the expected state trajectory is generally intractable; therefore, it is common to approximate the uncertainty propagation [Chua et al., 2018, Girard et al., 2002, Candela et al., 2003]. We generate this through Monte Carlo sampling by forward simulating P E Z_ particle rollouts of state trajectories. That is, for a given state particle, for the next step of 8₂ we sample a new state particle zp (t) (H,, (t()), a(t( 88), (t),((((()))) a(t())) 5a)) recursively along the planned action trajectory. This allows us to apply the reward function to the state particles, generating reward sarticles-whereby we can compute the mean and standard deviation of the reward each time step {(k+), i.c., r(t(k) (H,,,(tt)) (t(()))) Therefore, using our estimate of (r()())) over the state trajectory allows us to determine the maximum time interval A until the reward uncertainty becomes greater than a set threshold r. Thus, the next time to observe at is
(f((),
p(=(4))= max A' E R. Vv,,I((4 A/) <T}
This provides an estimate of the next observation interval A++1 that is discretized to an integer number of &g intervals. However, we seek the continuous-time observation interval A therefore,

---

we further refine this estimate through a continuous binary search (root finding algorithm) up to tolerance of 6, E R.,. We outline the AOC planning algorithm pseudocode in Appendix F.
Run-time complexity. Intuitively, we find if the time taken to plan using an MPC controller is feasible in problem setting, then AOC is feasible. As the run-time complexity of AOC is OGGK + P(K + W)) where W = [(log(8a) (6.) log():))/ 1og(2)] E ž_ (Appendix G). Empirically we chose P 10G, W = 4, G 10, 000 for all experiments. Although AOC takes approximately 2.4x longer to plan, which includes both planning the actions and when to observe compared to Continuous Planning methods that only plan actions (an MPC step)-it it can often take fewer observations overall, leading to less time spent planning overall. AOC is still a practical method that can plan in less time than the action interval da. even in robotics environments, where takes 24.9 ms to plan the actions and the next observation time (Appendix G).
An important parameter is the uncertainty threshold T hyperparameter. We find the following procedure to tune this parameter for a benchmark sufficient. For a particular environment after training its dynamics model, one episode is run, where the action trajectory a is re-planned every 5n time step and compute the median of the reward uncertainty over time for each rollout action plan, and then takes the mean of these over an episode to produce т. This step is akin to a calibration step, where calibration is common in medicine [Preston et al., 1988] and robotics [Nubiola and Bonev, 2013]. We also include practical guidelines to select e in Appendix H.
5 Experiments and Evaluation
Benchmark environments. We use four continuous-tin control environments and adapt them to add a fixed cost e for taking an observation these environments were selected as they exhibit a range of dynamical system state region regimes. First, the Cancer environment uses a simulation of a caaaaaooonnnii-Paaaaodyyyyy
(PK-PD) model lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. We note that the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. Moreover, we use the standard continuous-time control environments from the ODE-RL suite [Yildiz et al., 2021], which consists of three environments: Pendulum, CartPole, and Acrobot. We note that all the environments are described by a differential equation (DE) and use DE solver to simulate the fully continuous in-time states and actions, unlike discrete environments [Yildiz et al., 2021, Brockman et al., 2016]. Furthermore, to increase realism, taken, that of (/(0, o?) with standard deviation Ø = 0.01. Specifically, the ODE-RL starting states have the poles hanging down at the stable equilibrium of the DE system, and the goal is to swing up and stabilize the pole(s) upright to the unstable equilibrium [Yildiz et al., 2021]. Additionally, the reward function form is the same across the environments-ice. the exponential of the negative distance from the current state to the desired goal state 8", while penalizing the action magnitudes- and we assume we are given this simple reward function when planning (however, we also show AOC can work with a learned reward model in Appendix L.9). Furthermore, we generate an offline dataset D from these environments that has irregular times between state-action samples A++1 ~ Exp( ()). with a mean of A = da seconds. We detail all environments, including offline dataset generation in Appendix I.
add Gaussian noise to observations
Benchmark methods. We seek to provide more competitive benchmarks than those outlined in Table I; therefore, we compare against the fol- lowing ablations of our method, Active Observ- ing Control. Specifically, we focus on bench- marking observing policies p and use the same action policy т across all the benchmark meth- ods, that of the same MPC MPPI planner with the hyperparameters fixed. We implement two sicrette-imee methods by first learning a discrete-time time uncertainty- aware dynamics model (an abla- tion of the same model and hyperparameters, without the time interval input to predict the next state for) on the same offline dataset D. Second, we use this discrete-ting time uncertainty-awar aware dynamics model to create two benchmark methods, that of Discrete Planning that observes at each action time interval 6, and Discrete Monitoring a discrete ablation of our observing policy (Algorithm 3) that uses the same reward distribution estimation and determines the discrete action sequence to execute up until the reward uncertainty crosses the threshold 7-at discrete- time resolution of 6,. Moreover,
Table 2: Benchmark observing policies
Benclamark
Observing Policy
Discrete Planning
p(=((,)) - by mlz(M)) = 6. (la alli Adh.) <)) p(s((.) d
Discrete Monitoring
Continuous Planning
Active Observing Control ((=(t,)) max|A alt <))
7

---

Table 3: Normalized utilities 44 rewards R. and observations O for the benchmark methods, across each environment. AOC performs the best across all environments. Results are averaged over 1,000 random seeds, with + indicating 95% confidence intervals. Utilities and rewards undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds the expert, that of Continuous Planning. taking state observations every 8a.
Casor
Acrelet 010 $7.1
Corpolo
Poskitun
Pilky
Resden
D110
010
13.15
0.0
$010
0.1
011
50.0
010
010
50.00
Dserds Plasing
1ELAS
01
65.6
4877
127
42.9400522
42.3
130=252
67.3590557
Contreven Purning
800 155
13.10
10010 16S
9010
10010 772
50.00
1001
100L0WI1
50.0
we also benchmark against Continuous Planning that uses our trained continuous-time uncertainty- aware dynamics model and takes observations regular time intervals of multiple of 6g. Finally, we also compare with a random action policy, Random that observes at each action interval 5a. For ease of comparison, we list these observing policies in Table We further provide the dynamics model implementation details, hyperparameters, and training details in Appendix J.
Evaluation. An uncertainty-aware dynamics model is trained for each environment on an offline dataset D of collected trajectories consisting of 10 samples. The trained dynamics model weights are then frozen when used at run- run-time. We record the average undiscounted utility U4 (Equation (1)), average undiscounted reward R and the number of observations taken O. after running one episode of the benchmark method-an 95% confidence intervals throughout. Moreover, we normalize R and U following the standard normalization of offline-RL. Yuu al., normalzzdd the interval of to 100, where a score of O corresponds to random policy performance, and 100 to expert-which we assume here the Continuous Planning benchmark. Furthermore, we detail these metrics and the experimental setup further in Appendix K.
repeat this for 100 random seed runs for each result with their
5.1 Main results
We evaluated all the benchmark methods across all environments with results tabulated in Table 3. Active Observing Control achieves high average utility 24 on all environments. Specifically, AOC can achieve near-optimal state control performance while using fewer observations O compared to laking regular, frequent observations of Continuous Planning. Furthermore, it outperforms Discrete
Active Observing Control
Continuous Plamming
1000
1030
A.).
B)
2.5
2.5
0.0
0.0
(days)
(days)
0.1
0.0
500
1000
500
1000
(emn")
(ctu")
Figure 3: Comparison of Active Observing Control against Continuous Planning on the cancer environment for one episode. Rows: A) Cancer volume v state trajectory-witt green vertical lines representing observation times. B) Reward uncertainty of the planned state trajectory a after an observation is -heeee the red horizontal line indicates AOC's threshold used 7. C) Bar chart of the frequency of observations per state region. AOC automatically determines to observe larger cancer volumes more frequently as they are more informative, as the future state change magnitude is larger (Section 2. which correlates to clinician's 's findings [Geng et al., 2017]. Whereas with Continuous Planning the observing frequency is regular, and therefore observations can be taken when the reward is still certain, suggesting an unnecessary observation.

---

45
315
0.075
Table 4: Normalized utilities 24 and rewards R for the cancer environment- malization as in Table 3. Even when Continuous Planning takes the same number of observations as determined by AOC, it still performs worse, because those observations As Continuous Planning observations are taken blindly regular times, rather than more in- formative points.
0.050
the same nor-
90
270
not well located.
135
180"
Figure 4: Frequency of observations per state region for Pendulum. The intuition from Sec- tion 2.1 indicates that it is more informative sample when Φ is larger, hence near the top goal equilibrium point e s 0 necessitates infrequent observations.
Cancer
Policy
44
Active Observing Cuntrol
108
188.001733
3.39
Cominncus Planning with
102.0.234
6.10.234
3.10
Comiinous Plannng with 4
102::112266
97.34:1:2266
430
Monitoring as can determine when, in continuous time to observe, whereas those methods suffer from discretization errors that compound, leading to worse performance. 5.2 Insight experiments
In the following, we gain insight into why AOC performs better than Continuous Planning, i.e., better than taking regular observations in time and the importance of continuous-time monitoring as opposed to discrete-time monitoring.
How does irregularly observing achieve a higher expected utility than regularly observing" To better understand why monitoring (active observing) approaches outperform planning that regularly observes, we provide a qualitative and quantitative study.
On the one hand, qualitatively, as detailed in Figure 3, observe that for given threshold, AOC can still solve the Cancer environment while taking less costly observations and that AOC automatically determines increase the observing frequency when the tumor volume is large and less when it is smaller. Crucially, this matches what clinicians have already discovered, where they categorize the cancer volume into five discrete bins where the observing frequency is correlated to tumor volume [Geng et al., 2017, Seedat et al., 2022]. Moreover, this provides insight into how irregular observing can achieve higher expected utility (Section 2.1), observations could be more informative (hence taken more frequently) when the future state change magnitude is larger-occurring for large v.
Furthermore, as in Figure 4 for the Pendulum environment, observe that AOC observes infre- quently the state of the pendulum when has been swung up to the upright goal state and is balancing it there. This matches
intuition from Section 2.1, future state changes of ø the angle of the pendulum are large when the pendulum is not at the equilibrium goal state, thus having o > O, which maximally large in the swing up state trajectory phase and necessitates frequent observing, whereas balancing near the top equilibrium point, à Rs 0 necessitates infrequent observing.
On the other hand, quantitatively, as tabulated in Table 4, we make Continuous Planning take the same number of observations as AOC, however, taking them regularly. Since AOC can take better-located observations in time, outperforms Continuous Planning, achieving higher reward R. In summary, these experiments demonstrate that actively observing can be superior to planning in continuous time.
Why is it crucial to actively observe with continuous-time methods, rather than discrete-time methods? We compare Active Observing Control to Discrete Monitoring, which is the closest discrete counterpart (a discrete ablation of AOC) and look at when the observing times are reached, relative to their reward uncertainty estimates, to determine whether time discretization affects observing times. As shown in Figure 5, we see that Discrete Monitoring takes observations that are delayed, and therefore achieves lower state reward R. This allows AOC to capture fast-moving states, where missing these can be catastrophic deadly, as in the case of cancer.

---

100
Altia Chairika Goard
X- Billldd 274 a
9
Sucrita Manttaee
10
Time (days)
Oiberiing
-10
Chellease Plancing
Figure 5: Reward uncertainty o(r) normal- ized by the method specific threshold т on the cancer environment-thus the threshold to observe is at 1.0, indicated by the red horizon- tal line. We observe that AOC can detect and catch when the uncertainty crosses the thresh- old, independent of the time discretization. contrast, Discrete Monitoring suffers from time discretization error and takes a delayed observa- tion, leading to compounding errors and worse performance. Here Discrete Monitoring misses observing critical test by over day.
3.0
10.
15.00
Figure 6: Normalized utility U the cancer environment of Active Observing Control and Continuous Planning, plotted against changing uncertainty threshold т. We plot AOC's thresh- old used in Table 3 the red line. AOC maintains a high utility and reward over a wide using fewer obser-
range of thresholds-whi vations as increases, compared to Continuous Planning which takes expensive frequent obser- vations. The threshold 7 is between the mini- mum feasible Tmin and maximum feasible Tmax values.
5.3 Sensitivity analysis of
Active Observing Control still outperforms Continuous Planning even at variable T on the cancer environment, Figure 6. We note that as is increased, this increases the observing times A;+1, and hence fewer observations of the state are taken, Although we set = 6.67 for the cancer environment, following the procedure in Section 4, we observe robustness to other choices of are also suitable with decreasing reward R; however, AOC still outperforms and with a high utility U. Moreover, we tuned Tmin so that frequent observations were taken to be at least equal to Continuous Planning, and then Tmax to take the fewest observations, which is often minimally 1, as we start by taking an observation, until the entire action trajectory plan is executed, and requires re re-planning by taking another observation.
Conclusion and Future work
This work lays the foundation for the significant yet unexplored real-world problem continuous- time control whilst deciding when to take costly observations, which has profound implications across diverse fields such as medicine, resource management, and low-power systems. Our key theoretical contribution, verified empirically is that regular observing not optimal and that irregularly observing can achieve a higher expected utility. To demonstrate the power of active irregular observing, we provided an initial solution using heuristic threshold based on the variance of reward rollouts in offline continuous-time model-based model predictive control planner. However, the quest for the optimal method remains an open problem, providing fertile ground for future research.
However, this work is not without limitations. The heuristic threshold, while a robust initial solution in our experiments, may not be optimal suitable for all scenarios. Moreover, approach relies on the assumption that the offline dataset provides sufficient coverage the state-action space to leamn useful enough model of the system dynamics and that the observation observes all the states plus Gaussian noise. Also, in some practical applications, the cost of observations could vary or be unknown.
Navigating these limitations illuminates exciting pathways for future research. Potential directions include: optimizing multiple observations concurrently over a planned action trajectory, jointly optimizing action trajectories and observing times, and theoretically deriving the optimal solution to reveal characteristics of optimal observing policies.
10

---

Acknowledgements. SH would like to acknowledge and thank AstraZeneca for funding. This work was udditionally supported by the Office of Naval Research (ONR) and the NSF (Grant number: 1722516). Moreover, we would like to warmly thank all the anonymous reviewers, alongside research group members of the van der Scaar lab, for their valuable input, comments and suggestions as the paper was secelooed-wherr all these inputs ultimately improved the paper.
References
Rui Aguiar, Nikka Mofid, and Hyunji Alex Nam. Exploring optimal control with observations at cost. arXiv preprint arXiv:2006. /5757, 2020.
Ahmed Alaa and Mihaela Van Der Schaar. Balancing suspense and surprise: Timely decision
making with endogenous information acquisition. Advances in Neural Information Processing Systems, 29, 2016.
Arthur Argenson and Gabriel Dulac-Arold. Model-based offline planning. In International Confer- ence on Learning Reprexentations, 2020.
Karl Aström. Event based control. Analysis and design of nonlinear control systems: honor of Alberto Isidori, pages 127-147. Springer, 2008.
Karl Johan Äström and Bo Bernhardsson. Comparison of periodic and event based sampling for first-order stochastic systems. IFAC Proceedings Volumes, (205):500--501, 1999.
Tim D Barfoot, Chi Hay Tong, and Simo Särkkä. Batch continuous-time trajectory estimation as exactly sparse gaussian process regression. In Robotics: Science and Systems, volume 10, pages 1-10. Citeseer, 2014.
Andrew Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834 846, 1983.
Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn. Active measure reinforcement learning for observation cost minimization. arXiv preprint 12X: 2020a.
Colin Bellinger, Andriy Drozdyuk, Mark Crowley, and Isaac Tamblyn. Balancing information with observation costs in deep reinforcement learning. 2020b.
Ioana Bica, Ahmed M Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. arXiv preprint arXiv:2002.04088. 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01144 2016.
Bruunereu,, F Bruyère, C Linassier, and J-L Baulieu. The role of imaging in staging and monitoring testicular cancer. Diagnostic and interventional imaging, 93(4):310-318, 2012.
Shelly Butler, Denise Kirschner, and Suzzane Lenhart. Optimal control of chemotherapy affecting the infectivity of hiv. Advances in mathematical population dynamics: molecules, cells, man, pages 104-120, 1997.
Joaquin Quinonero Candela, Agathe Girard, Jan Larsen, and Carl Edward Rasmussen. Propagation of uncertainty in bayesian kernel models -aplicaappiiiotiin to multiple-step ahead forecasting, In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceed-
ings volume 2, pages II-701. IEEE, 2003.
Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Pratap Tokekar, and Dinesh Manocha. Dealing with sparse rewards in continuous control robotics via heavy-tailed policies. arXiv preprint arXiv:2206. 05652, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.

---

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances neural information processing systems, 31, 2018.
Alicia Curth, Alihan Hüyük, and Mihaela van der Schaar. Adaptively identifying patient populations with treatment benfit clinical trials. In International Conference on Machine Learning, 2023.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472 Citeseer, 2011.
J Du, J Futoma, and F Doshi-Velez. Model-based reinforcement learning for semi-Markov decision processes with neural ODEs. In Proceedings of the 34th Conference on Neural Information Processing Systems, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.
Changran Geng, Harald Paganetti, and Clemens Grassberger. Prediction of treatment response for combined chemo-and radiation therapy for non-small cell lung cancer patients using a bio- mathematical model. Scientific reports, 7(1):1-12, 2017.
Daniel T Gillespie. Markov processes: an introduction for physical scientists. Elsevier, 1991.
Agathe Girard, Carl Edward Rasmussen, Quinonero-Candela, R Murray-Smith, O Winther, and Larsen. Multiple-step ahead prediction for non linear dynamic systems-a a gaussian process treatment with propagation of the uncertainty. Advances in neural information processing systems, 15:529-536, 2002.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 1803. 2018. 2018.
Nicklas Hansen, Xiaolong Wang. and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955 049555, 2022.
David Hoeller, Farbod Farshidian, and Marco Hutter. Deep value model predictive control. In Conference on Robot Learning, pages 990-1004. PMLR, 2020.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2022a.
Samuel Holt, Alihan Hüyük, Zhaozhi Qian, Hao Sun, and Mihaela van der Schaar. Neural Laplace control for continuous time delayed systems. In International Conference on Artificial Intelligence and Statistics, 2023.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Neural laplace: Learning diverse classes of differential equations in the laplace domain. In International Conference on Machine Learning. pages 8811 1- 8832. PMLR, 2022b.
Yunhan Huang and Quanyan Zhu. Infinite-horizon linear-quadrraii-gaausss control with costly measurements. arXiv preprint arXiv:2012.149255 2020.
Yunhan Huang, Veeraruna Kavitha, and Quanyan Zhu. Continuous-time markov decision processes with controlled observations. In 2019 57th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 32-39. IEEE, 2019.
Alihan Hüyük, Zhaozhi Qian, and Mihaela van der Schaar. When make and break commitments? In International Conference on Learning Representations, 2023.
12

---

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Daniel Jarrett and Mihaela Van Der Schaar. Inverse active sensing: Modeling and understanding timely decision-making. International Conference on Machine Learning, pages 4713-4723. PMLR, 2020.
Napat Karnchanachari, Miguel Iglesia Valls, David Hoeller, and Marco Hutter. Practical reinforcement learning for mpe: Learning from sparse objectives in under an hour real robot. In Learning for Dynamics and Control, pages 211 2-142 PMLR, 2020.
Rahul Kidambi, Aravind Rajeswaran, Pranceth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-2188 21823,, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Marin Kobilarov. Cross-entropy motion planning. The International Journal of Robotics Research, 31(7):855 2012. 2012.
David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observ- ing rewards at cost. arXiv preprint arXiv:2011.06709 2020.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1179 1911199919, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Bruce A Larson. Principles stochastic dynamic optimization in resource management: the continuous-time case. Agricultural Economics, 7(2):91-007, 1992.
Suzanne Lenhart and John Workman. Optimal control applied to biological models. CRC press, 2007.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 2015.
Bryan Lim. Forecasting treatment responses time using recurrent marginal structural networks. advances in neural information processing systems, 31, 2018.
Zhichu Lin and Mihaela van der Schaar. Autonomic and distributed joint routing and power control for delay-sensitive applications in multi-hop wireless networks. IEEE Transactions on Wireless Communications, 10(1):102-113: 2010.
Michael Lutter, Leonard Hasenclever, Arunkumar Byravan, Gabriel Dulac-Arnold, Piotr Trochim, Nicolas Heess, Josh Merel, and Yuval Tassa. Learning dynamics models for model predictive agents. arXiv preprint arXiv:2109.143/1 2021.
Ruben Martinez- Mantine Nando de Freitas, Arnaud Doucet, and José A Castellanos. Active policy learning for robot planning and exploration under uncertainty. In Robotics: Science and systems, volume 3, pages 321-328, 2007.
Nicholas Mastronarde and Mihaela van der Schaar. Joint physical-layer and system-level power man- agement for delay-sensitive wireless communications. IEEE Trunsactions on Mobile Computing, 12(4):694-709, 2012.
13

---

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):5295533 2015.
Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Model based reinforcement learning: A survey. Xrviv preprint arXiv:2006. 16712, 2020.
Juliette Morgan, Martin I Meltzer, Brian D Plikaytis, Andre N Sofair, Sharon Huie- White, Steven Wilcox, Lee H Harrison, Eric C Seaberg, Rana A Hajjeh, and Steven M Teutsch. Excess mortality, hospital stay, and cost due to candidemia: a case-control study using data from population- based candidemia surveillance. Infection control & hospital epidemiology, 26(6):540-547, 2005.
HyunJi Alex Nam, Scott Fleming, and Emma Brunskill. Reinforcement learning with state observation costs in action-contingent noiselessly observable markov decision processes. Advances in Neural Information Processing Systems, 34: 15650-1666, 2021.
Tianwei Ni and Eric Jang. Continuous control on time. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
Kenzo Nonami, Farid Kendoul, Satoshi Suzuki, Wei Wang, and Daisuke Nakazawa. Autonomous flying robots: unmanned aerial vehicles and micro aerial vehicles. Springer Science & Business Media, 2010.
Albert Nubiola and Ilian A Bonev. Absolute calibration of abb irb 1600 robot using laser tracker. Robotics and ComputerrIntegraated Manufacturing, 29(1):236 245, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, deep learning library. Advances in neural information processing systems, 32,
2019a.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019b.
Roy C Preston, David R Bacon, and Robert A Smith. Calibration of medical ultrasonic equipment- procedures and accuracy assessment. IEEE transactions on ultrasonics, ferroelectrics, and fre- quency control, 35(2):110-121 1988.
Heike Reinhardt, Petra Otte, Alison G Eggleton, Markus Ruch, Stefan Wöhrl, Stefanie Ajayi, Justus Duyster, Manfred Jung, Martin J Hug, and Monika Engelhardt. Avoiding chemotherapy prescribing errors: analysis and innovative strategies. Cancer, 125(9):554--1557 2019.
Jacques Richalet, André Rault, JL Testud, and J Papon. Model predictive heuristic control: Applica- tions to industrial processes. Automatica, 14(5):41-442 1978.
Tim Salzmann, Elia Kaufmann, Marco Pavone, Davide Scaramuzza, and Markus Ryll. Neural-mpe: Deep learning model predictive control for quadrotors and agile robotic platforms. arXiv preprint 200.0:2203.07777, 2022.
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela van der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. arXiv preprint arXiv:2206.083111 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representations, 2019.
Sahil Sharma, Aravind Srinivas, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforeement learning. arXiv preprint arXiv:1702.06054, 2017.
14

---

Hao Sun, Alihan Hüyük, and Mihaela van der Schaar. Accountable batched control with decision corpus, In Conference on Neural Information Processing Systems, 2023.
Richard S Sutton. Between mdps and semi-mdps: Learning, planning, and representing knowledge at multiple temporal scales. 1998.
Irena Twardowska, Sebastian Stefaniak, Herbert E Allen, and Max M Häggblom. Soil and water pollution monitoring, protection and remediation, volume 69. Springer Science & Business Media, 2007.
Volodymyr Vasyutynskyy and Klaus Kabitzsch. Event-based control: Overview and generic model. In 2010 IEEE International Workshop on Factory Communication Systems Proceedings, pages 271-279. IEEE, 2010.
Pauli Virtanen, Ralf Gommers, Travis Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Eygeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272. 2020.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 14:29420-224333 2021.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprini arXiv: 1 /907.02077 2019.
Ziyi Wang, Augustinos D Saravanos, Hassan Almubarak, Oswin So, and Evangelos A Theodorou. Sampling- based optimization for multi-agent model predictive control. arXiv preprint arXiv:2211. 1878, 2022.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4433-1440. IEEE, 2016.
Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg. Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model- based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1714-1721. IEEE, 2017.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint I31, 2019.
Cagatay Yildiz, Markus Heinonen, and Harri Lähdesmäki. Continuous-tine time model-based reinforce- ment learning. In International Conference on Machine Learning, pages 10009-12018 8. PMLR, 2021.
Hidekazu Yoshioka and Motoh Tsujimura. Analysis and computation of an optimality equation arising in impulse control problem with discrete and costly observations. Journal of Computational and Applied Mathematics, 366: 1 12399, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Mosel-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142, 2020.
Zichen Zhang, Johannes Kirschner, Junxi Zhang, Francesco Zanini, Alex Ayoub, Masood Dehghan, and Dale Schuurmans. Managing temporal resolution in continuous value estimation: funda- mental trade-off. arXiv preprin! www.i::2/22089444 2022.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. arXiv preprint arXiv:1909.12077. 2019.
15

---

Appendix
Table of Contents
A Broader Impact Statement
17
B Further Application Examples
17
Proof for Proposition 2.1
D Extended Related Work
20
E MPC MPPI Pseudocode and Planner Implementation Details
23
F Active Observing Control Planning Algorithm Pseudocode
24
Active Observing Control Run-time Complexity
25
I Practical Guidelines Select
26
Environment Selection and Details
26
111 ODE-RL Suite
27
1.2 Cancer Environment
29
Benchmark Method Implementation Details
J.1
Probabilistic Dynamics Model
J.2
Observing Policies
................................
31
J.3
MPPI Implementation
31
K Evaluation Metrics
32
L Additional Experiments
32
L.1
Ablation of Thresholding the State Uncertainty
L.2
Sparse Reward Environment
33
L.3
Increasing the Planning Resolution of MPC (decreasing 6..) ............
33
L.4
Discrete Time Controller Trained on Equidistant Dataset Sensitivity of AOC to 8,
34
L.5
35
L.6
Evaluation of Reward Uncertainty
36
L.7 Relation Between Rewards R and the Fixed Observation Cost
36
L.8 Dependence on the Accuracy of the Learned Dynamics Model
37
Extending AOC Work with a Learned Reward Model
I.10 AOC Also Empirically Works for Non-linear State Transformations
38
L.1l Further Empirical Validation in Other Real-World Scenarios
38
Code. All code is available at attp:/aihhoooosssm///oooooooooot
16

---

A Broader Impact Statement
Our work on active observing in continuous-time control presents a novel perspective on decision- making in domains such as medicine, low-power systems, and resource management, with the potential to substantially reduce costs and increase efficiency. However, the adoption of these techniques also presents potential risks and ethical concerns. The dependability of these systems could raise issues if they malfunction critical fields like healthcare. Hence, robustness, security, and error handling must be prioritized in implementation.
compromised, potentially leading to harmful decisions in
B Further Application Examples
The problem of continuously controlling continuous time environments whilst actively deciding when to take costly observations in time is ubiquitous in real-world environments, with some of these applications listed in Table 5.
Table 5: Applications of continuous control with costly observations.
Continuous time
Ref.
Continuous control
Costly observation
High stakes
cuvironment
Medical cancer chemotherapy treatment
[Geng 2017, Brunereau et al., 2012]
Continuous chemotherapy dosing
Computed tomography scan
Errors be fatal
Biological fish population
(Yoshioka and Tsujimura, 2020)
Continuous food temperature
Fish population survey
Underfeeding or overlishing can lead extinction
control
management
Low power communication
[Msstroarde and van der Schuar, 2012, Lin and Schuar, 2010]
Continuous channel transmission
Measure the maximum bandwidth
Communication rate emoos lead to loss of data
Mobile Robotics
[Martneez Cantin et al., 2007]
Continuous rohot control
Mensure the rbbo's 's position
Errors can lead to damage the robot or environment
Agricultural resource
[Larson, 1992]
Continuous land allocation for livestock
Meusure healthh
Errors can lead to livestnck death
management
livestock
Nursing surveillance
MMrgan et al., 20055
Continuous treatment plans
Measure the healtth of patient
Errors can be fatal
Water pollution monitoring
(Twardowska 2007]
Continuous treatment
Measure the
Errors lead water contamination
quality
C Proof for Proposition 2.1
Proof. We will consider the system defined by
ds dt = s
(2)
1
with probability 1/2
s0)
(3)
with probability
1/2
Note that solution to this system can written as s(t) s0)e', which is independent of any actions that might have been taken. Also, consider the reward function given by
a(s,a,t) = 5(t - T) 1 0}
(5)
17

---

Given history of samples = ((t), posterior over s(0) can be written
A
P(s(0)) 1/4) P(s(0) = ()) =1
Va(a(())=>)))
(6)
PP(h)
P(h)
2
1=1
A
((s(0) - 11h P(s(0) = 1))p(()s(())) = =
1
-=(=(()))=
(7)
P(h)
P(h)
V2
Using this posterior, we define auxiliary variable
b(h) = log P(=(0) = 1(h) - log P(s((0) -1
(8)
141
1h
Σ
(x(t)))ett) ----------- Σ
(=((g) + c5,2
(9)
2 1=1
==1
Then, the optimal decision policy can be written as
if F(s(0) 1(4) 2 1/2 6(h) 2 0
x" r1, t)
-1 if P(s(0) 1(h) < 1/2 = b(h)
(10)
Next, we derive the expected utility of the optimal decision policy given set of observing times (ty [0, Denoting with h = [tt,, xttz), aft,)))}= the final history, we have
R = /' r(s((),a((),
(11)
(' 8(1 T). 1(s(1) a(l) > 0)d/
T
(12)
((8(7) a(T) > 0}
(13)
((s(T)) - (h,,)) 0}
(14)
=1(s(0) = A A
=
-1)
(15)
= =1(x(0) ((,,)) 20 +1{s(((0 b(l,)) <0}
(16)
Hence,
B[R[())==1 = P(s(0) 1 A AKA.. > P(()) = -1 11 A(hm) (11()))
(17)
-P(s(0) 1KG) 2 (x(())))))
(18)
+ +P(((()) = --1((()))(((),)) <()s(0)) = =(,(0))
(19)
= 4/2 (Pb(h,,) 2 0)s(()= P(b(h,, 0ls(0) = - (t,
(20)
18

---

Observing that
P(b(h)n) 2 0|s(0) = 1, (1))
(21)
b Σ
P
(=(()) 0//
1
Σ
(z(1g) + + s(0) =1)
(22)
2
j=1
==1
P
1
(s(1)++ +
- -(())) +
1
Σ
(e(l))+ + ((()))+(()+)))))))
x(0) =1
(23)
2
2
b
b +
(e(l)) + 225) 22)))200) 20) ))
(s(1)))
Σ
(24)
2
2
Σ
2 + ebe(())) 20)
(25)
= P
2241)
E> Σ 24
(26)
jml
(ow((o.) E2--1 6=-V2n j=1
E2t,
(27)
= (vv Ei
j= 2211
(28)
where F(x) = S(1/V/T) (1/2)2² is the cumulative distribution function of the standard normal
distribution, and similarly that P(b(hn)) 2 0}s(0) = -1, = FF( PVVEE=1 e²2)), we obtain
1 2 j=1 6 [
en=
en
(29)
Next, we introduce two classes of observing policies. The first class consists of observing policies that observe regularly in time with an observing interval of 5 such that ((rog,5) (h) for all When enrolled, the observing times for these policies can be written as t, jō hence
E ploee, a) 1 1 (v
T/81 1=1
T 8
256
(30)
The second class consists of observing policies that take E-many observations all at l = T such that
- if n <
p (ireg.8)) f) (h- = ((t,,,((t)
(31)
if n 2
When enrolled, the observing times for these policies can be written as tj = T hence
B mmmg o [1] 1 F (V el
(32)
Crucially, (imeg.)) with E 2 2 is distinct from all regular observing policies p! with 8 > O, since for no regular observing policy, i = 12. We prove that p" cannot be regular observing policy by showing that, for each ((reg.)), there exists at least one ((ineg.e) with é >> that achieves higher expected utility.
...) For 8 0, E pttm [24] + -00 whereas Eptie, LT/ [4] is bounded for any e e {2, (..3.... (hence
E /aup. [24] > E, Eoos..0) (2)).
19

---

For ò S T/2. we have [T/6] 2 2 and E 17/ R
>
E ppoee, a> [1] since
E Ep [2] - Bsp pmm LTyA1) [4]
(33)
1
[T/J]
T
T
T
F
6216 j=1
c
F
V
e²T
+
(34)
2
6
2
8
1
F
V
Σ
|J/61
e216
F
(V
T
e2T
(35)
1=1
<
1
Σ
[T/6]
eer
1
T'
22T
=0
(36)
j=l
Let the cost per observation be given as
1 F( F(Ve22) F( (((e2T) 1 F(0) >0 0
c 2
min
F
(V2227)
(37)
2
2
Then, for T/2 < 8 S T,, we have [T/6] = 1 and E p(0mp [2]] >> E B)tms,0) [1]] since
EE s) 5) [1]- - E plemp. [2]
(38)
1
26
c マ/4
F
(V2222
+ 2c
(39)
2
2
F( (V)2
F F (222TT)
+
(40)
1
1
-
*((Ve2 22
F( ((V2622
cc= 0
(41)
2
2
min
1
Fr (Veerr
1
F(
1 1 F()) 2e22)
4
1 [
+c
(42)
2
-2 220 0
(43)
Finally, for T < 6, we have [ = 0 and E plinss 2) [2] > E pl [2] since
Eptor 81 [1] E, Epiim..2) 23 [2d]
(44)
1 F(0)
-
F( 2T
+ 2c
(45)
2
2
{
1
1
F(V262T) }
S min
F(0)
F (V2c2T), F F(Ve2T)
+ 2c
(46)
2
2
4c
4c
(47)
D Extended Related Work
In the following, we expand on the existing related work, in Section 3, and we provide additional discussions of the benefits of offline RL, model-based RL, why model predictive control is our preferred action policy, event-based sampling, Semi-MDP methods, and Linear Quadratic Regression (LQR)& & Linear Quadratic Gaussian (LQG) methods.
Why offline RL. The setting of offline reinforcement learning consists of an agent that learns a dynamics model (i.e., a model of the environment dynamics) from dataset of previously collected state-action trajectories from a specific environment and the agent is not allowed to interact with the environment [Wu et al., 2019]. For instance, real-world problem settings, as they rely on expensive trial-and-erros approaches performed either is not often possible. contrast, "offline" model-based RL
challenging to apply online RL methods to
online or in a realistic simulator-tie learns a model of the environment dynamics from previously collected dataset of state-action trajectories, that is often possible. These methods then control the environment to a desired goal state using any available planning method such as training policy [Fujimoto et al., 2018] or using
20

---

model predictive controller (MPC) [Williams et al., 2016]. Specifically, both model-free [Kumar et al., 2019, 2020. Fujimoto and Gu, 2021] and model-based [Kidambi et al., 2020, Wang al.. 2021] approaches have been proposed for offline RI., where in general model-hased methods have shown to be more sample efficient than model-free methods [Moerland et al., 2020].
Why model-based reinforcement learning? Core to Model-based RL methods is the desire to create policies for real world tasks in the offline setting where the true environment dynamies are unknown and rather learn an approximate dynamics model from an offline dataset of state-action trajectories from the environment. A key benefit of model-based reinforcement learning is that it can be significantly more sample efficient compared to model-free RL methods [Lutter et al., 2021] where model-free methods can require millions or billions of interactions with the environment to learn a good policy. Another key benefit is that learning a dynamics model of the environment enables planning with that dynamics model to optimize actions, such as using a model predictive control planner, rather than learning specific policy.
Principally, the dynamics model is independent of the reward and therefore allows changing reward of the planning policy-enablie on the fly run-time. Conversely, policy that was solely trained for a particular task would have to be re-trained, or a conditional policy trained when adapting goals/tasks [Lutter et al., 2021J. This realistic as real-world continuous- time control settings of the dynamics model (e.g., often the physics/bioloiical process of an environment [Holt et al., 2022al) is independent of the reward function. Importantly model-based RL for conventional control tasks has been shown be more sample-efficient than model-free methods [Wang et al., 2019, Moerland et al., 2020]. Historically, model-free methods have demonstrated expert performance on challenging tasks [Mnih et al., 2015, Lillicrap et al., 2015], and was later shown that proper tuning of model-based methods have much higher sample efficiency Ha and Schmidhuber 22018,, Janner et al. [2019] and can achieve expert performance with probabilistic dynamics models on continuous control tasks [Chua et al., 2018].
planning-based policy to easily adapt to changing goal states/tasks
Model predictive control (MPC). Core to model predictive control (MPC) is having a good dynamics model, historically using simple, first principle derived (often linear), known dynamics models [Richalet et al., 1978, Salzmann et al., 2022]. Recently, MPC Model Predictive Path Integral (MPPI) [Williams et al., 2017] is a zeroth-orde particle- -aadd (action) trajectory optimizer [Williams et al., 2017] method that can use a general, non-differentiable nonlinear dynamics model with a defined complex reward function (or cost function).
Particularly, Williams et al. [2017] demonstrated that it was suitable to be used with learned neural network dynamics modch-which was used for the task of driving a vehicle around a dirt track in aggressive, actuator saturating maneuvers. In general, MPC is often computationally infeasible for optimizing actions up to long time horizons, therefore are often used to optimize actions up to fixed time horizon, which is re-computed iteratively when needed. A key benefit of planning, and MPC, is that it can incorporate state constraint fcasibility into its plans as well as easily changing the reward function for a changing goal/task at run-time. We specifically use this MPC method in this paper, as is -tate-of-thheart [Wang et al., 2022] for performing MPC with learned dynamics model, improving upon the widely used cross-entropy method (CEM) MPC planner [Kobilarov, 2012].
Hybrid MPC: Combining a powerful MPC planner with a learned policy is another exciting area developing [Argenson and Dulac-Arnold, 2020]. These existing hybrid works, work on only discrete- time environment dynamics models. These include MBOP [Argenson and Dulac-Arnold, 2020], TD-MPC [Hansen et al.. 2022] and DADS [Sharma et al., 2019]. We highlight that these hybrid methods are unable to determine when to observe in continuous-time and only determine discrete actions to execute.
Sensing approaches have been proposed of when to optimally take an observation in both discrete time [Jarrett and Van Der Schaar, 2020] and continuous time [Alaa and Van Der Schaar, 2016, Barfoot et al., 2014]-whers their goal is to identify an underlying state. However, these approaches cannot also continuously control the system. In contrast, Active Observing Control seeks to both actively observe and control the system. Besides deciding when to take an observation, some work on sensing focuses on when stop taking observations (e.g. Hüylik et al., 2023] or what kind of observations to take [e.g. Curth et al., 2023]. Unlike these works, we focus exclusively on the timing of observations.
Planning approaches only focus on optimal decision policies ग. and therefore observing has to be provided by a schedule, which is often assumed to be at a regular constant interval in time
21

---

Ai+1 = -htthat is, observations are not actively planned [Yildiz et al., 2021]. Within planning approaches, there exist many discrete-time approaches [Chua al., 2018, Mnih al., 2013, Williams et al., 2017, Sun et al., 2023] and recently more continuous-time approaches [Du et al., 2020, Yildiz et al., 2021, Holt et al., 2023] these use continuous-time dynamies model, e.g. [Chen et al., 2018, Holt et al., 2022b]. Specifically, Yildiz et al. [2021] presented a seminal online continuous model-based RL algorithm, leveraging a continuous-ting dynamics model that can predict the next state at an arbitrary time st). However, all these methods are unable to plan when to take the next observation, in contrast to Active Observing Control which is able to determine when to take the next observation whilst planning action trajectory.
Monitoring approaches consist of both a decision policy т and an observing policy pt however, existing approaches only consider the discrete-time and discrete-state setting with observation costs [Sharma et al., 2017, Nam et al., 2021, Aguiar et al., 2020, Bellinger et al., 2020a, Krueger et al., 2020, Bellinger et al., 2020b]. In particular, Krueger et al. [2020] proposed that even in a simple setting of a discrete-stade multi-armed bandit, computing the optimal time to take the next observation is
they must rely on heuristics for their observing policy'.
Broadly, discrete-time monitoring methods use discrete-time model and propagate the last observed state, often appending either a flag or a counter to the state-action tuple to indicate the time since an observation was taken [Aguiar et al., 2020, Bellinger et al., 2020b]. However, these methods cannot be applied to continuous time environments or propagate the predicted current state and its associated prediction interval of the uncertainty associated with the state estimate. Moreover, training a policy [Aguiar et al., 2020] that decides at each state whether to observe it or not, requires discretization of time. Whereas Active Observing Control, which determines the continuous- observe the next state, does not require any discretization of time, and hence it is a continuous-time method and avoids compounding time discretization errors.
interval of when to
One approach exists that we term semi-continuous monitoring, where Huang et al. [2019] proposes a discrete-state, constant action policy, that determines when take an observation in continuous time of a controlled Markov Jump Process. However, this approach is limiting, as it assumes actions are constant until the next sample of the state is taken [Ni and Jang, 2022]-whrl
clearly suboptimal, uses discrete states, and assumes a Markov Jump Process [Gillespie. 1991]. Instead, Active Observing Control is fully continuous in both states and observing comes--ontrolllee by an action trajectory o, giving rise to amoothly-variin states.
Event-based sampling a similar related work area in the control community [Åström and Bern- hardsson, 1999. Aström, 2008], which addresses the similar problem of controlling system, of only taking a full state observation when an external input event occurs, and then providing a control action. To create this event, it assumes part of the state is continuously observed, and often an event is defined when this part of the state (or a function of it) crosses a fixed threshold (e.g., a physical sensor crossing threshold) (Vasyutynskyy and Kabitzsch, 2010]. This finds multiple uses in problems such as electronic analog control systems for audio and mobile telephone systems [Åström and Bernhardsson, 1999], and battery capacity optimization in wireless devices [Vasyutynskyy and Kabitzsch, 2010]. However, this is different from our proposed problem of continuous-time control whilst deciding when to take costly observations as (1) Event-based sampling assumes part of the state is continuously observed. Often in our environments, it is not feasible observe part of the state continuously (e.g., imaging a cancer tumor volume or performing a medical test) or it is prohibitively expensive to continually observe at a high-frequency part of the state (similar to Continuous Planning approaches). (2) The event input (the time to take the next observation) is given as a control input to the agent. However, it precisely the more difficult problem we tackle of coming up with an observing policy that decides when to take the next observation. (3) The event is often defined by a human in advance and is a function of the current partial state. General environments may not have partial state spaces that are predictive of when to observe next, such as a robotic navigation task in two dimensions and only continuously observing one dimension.
'Kuuegrr et al. [2020], focus on the simpler problem of multi-armed bandits (MAB), where there is cost to observe the reward. In the MAB setting, it is possible to formulate it an RL PO-MDP with one state. The underlying static state (the fixed reward distributions of each bandit) is unknown and must be determined by taking successive costly observations (paying a cost to observe a reward). Therefore, Krueger et al. [2020]'s statement that the optimal algorithm for their simple MAB setting is intractable applies to the problem of active observing in continuous-tinng control.
22

---

Semi-MDP methods is another similar related field, where it extends the MDP problem to include options that define temporally abstracted action sequences [Sutton, 1998]. However, some distinct differences prevent using a semi-MDP to address our continuous-time control whilst deciding when to take costly observations problem. These differences include (1) Semi-MDP is still an underlying discrete MDP with discrete state transitions, whereas our problem formulation focuses on continuous- time systems that can handle continuous actions and states. (2) Semi-MDP formulation does not involve an observation cost, whereas our problem formulation does.
LQR/LQ methods. Similar related methods from control exist, as Zhang al. [2022] propose a discrete planning method that works only for linear quadratic regulator (LQR) systems and optimizes the observing frequency for given total budget of observations. Moreover, Huang and Zhu [2020] addresses an infinite horizon discrete-ting linear quadratic Gaussian (LQG) control problem with observation costs. This notably different from the formulation proposed for the problem of continuous-time control with observation costs as (1) Huang and Zhu [2020] only applies to linear systems that are discrete time and have an infinite time horizon; our formulation applies to non- linear systems that are continuous in time and have a fixed time horizon. (2) Huang and Zhu [2020] assume their full system dynamics model is known; ours makes no such assumptions and only assumes access to an offline collected dataset of (possibly irregular in time) state-action trajectories to learn a dynamics model, which is more applicable to real-world environments.
E MPC MPPI Pseudocode and Planner Implementation Details
To plan an optimal action trajectory a, we specify that any model predictive controller (MPC) planner can be used-that can optimize the action trajectory up to specified time horizon H E IR... We opt use MPC planner, as it optimizes an entire action trajectory a up H, does not require the probabilistic dynamics model fo to be differentiable (it uses no gradients), and the reward function r can be changed on the alloownig changing goals/tasks at run-time. Moreover, model-based RL with MPC planners has achieved comparable performance to model-free RL methods [Chua et al., 2018, Lutter et al., 2021]. We use our probabilistic dynamics model fe. with the MPC planner of Model Predictive Path Integral Control (MPPI), zeroth-order particle-based trajectory optimizer [Williams et al., 2017], due to its competitive performance [Wang et al., 2022].
To optimize the action trajectory up to time horizon H, it discretizes H into smaller constant action time intervals 6, E + which are then optimized where there are K hg E Z4 time steps in H- i.e., (()) = 12 + kō h E (O.... K - 1}. It forward simulates number of parallel rollouts G E Z where the next state estimate each time step is simulated as =(((++1))= (=((((())) a((())), 5,) recursively. This requires a state estimate z(1.) to plan from: therefore, we recompute the planned trajectory when we observe new sample of the state. This optimizes the action trajectory by
Ā-1
at (++) arg max E. a(t())
Σ (((2(t())), ((tt))
MPPI is also Monte Carlo based sampler, and thus increasing the number of rollouts improves the input trajectory optimization, however, scales the run-time complexity as OGK)). The standard
MPPI algorithm [Williams et al., 2017] is used with our probabilistic dynamics model fo. We articulate the MPPI pseudocode for the action trajectory optimizer in Algorithm 2.
23

---

Algorithm 2 MPPl--rajecoryy-Qpriiiiaaao
Input: State observation =(t;), Pre-trained probabilistic dynamics model fo. Reward function r. Time Horizon H, Action time interval 5as Number of parallel rollouts G, Noise covariance Σ, Hyper parameters X, Action max Omaxx Action min Qmin- K
! 8e R - O(G)
This holds our G' trajectory returns. This holds our G action trajectories of length K. This holds G action trajectories of length that are perturbed by
A (G, t O(G.K)
A (G.(o:K-l] t- ((C.K)
noise. ((G + (
This holds the generated scaled action noise.
for g = O....,, 1:
) 2f44)
Sample G trajectories over the horizon K.
for = 0, 0..... K 1
e(g.h) < 10 Σ)
Sample action noise. Perturb action by noise. A (g.k) min(max(AA) (llsh)) 1),, +1) Clip normalized perturbed noise (to bound actions
k A(9.k) + clo.k)
to their limits).
cla.k) A ((ak)) A (lak)
Update noise after bounding, do not penalize clipped
noise.
for k= ...., 1
2(1(b+1) t 1r. (z(((()), (1k)) amax; 8n) Sample next state from pre-trained probabilistic
dynamics model fo.
R(s) + R(9) + r( (((+t()))) Ala,k) - AA > Accumulate the current state
reward.
K - ming [
(R)) 2nn exp + (R(a) -x))e
amax Vk E [0 1]
D> Generate the retum-weighted
Σg ( (R())
trajectory update. Return: T
F
Active Observing Control Planning Algorithm Pseudocode
The Active Observing Control planning algorithm pseudocode is outlined in Algorithm In particular, 6, is the continuous search (root finding algorithm) tolerance that is used when searching with binary search for the continuous time duration that the computed action trajectory can followed for, which occurs when the standard deviation of the reward crosses the threshold 7. We note that all numerical root-finding algorithms generally involve a search tolcrance or similar stopping criterion. The d, tolerance ensures that the binary search algorithm stops evaluations once solution time is found that "close enough" to the actual true value. Another way to view the search tolerance 8, is that the numerical precision that the search value is correct up to.
24

---

Algorithm 3 Active Observing Control Policy
Input: State observation =fa),, Pre-trained probabilistic dynamics model fo. Reward uncertainty threshold Reward function r, Time Horizon H, Search tolerance o,, Action time interval 6a-
4- MPC(fo, z(1:), T, H)
b Plan action trajectory.
O.....K- --1: for ((h)) t to + kōg
for lorp- t .:....P: (((ttk))) (H. a(t(())))) ),))), (((
if Vplr(sp(t(k+ a(t(++))))] >T:
(lower) {(k), (lupper) t- {lk+1)
while g(upper) - (lowerr) > d; y {miid) + (b(upper) + ((lower))
-I....,P P: for p 1.
zp (t(midd))) ~ N (2 ((t())))))))))))))..))))))))))))
a(t())), f(mid) - =(0)), 02(2t(1))) a(t(()), slmid) {(h)))
if (()pp(((((mmd)) al ((t/mid)))] T: (upper) &- ((mid)
else:
+ (lower) x f(mid)
Return a(t)vt E [t,, flmid))
Return a())tt [1;, (x)
G Active Observing Control Run-time Complexity
Active Observing Control's s run-time complexity of Algorithm is O(GK + P(K + W)) where W [(log(ó,) log ( log(2)] E 24. As W is determined by continuous binary real line search, on an interval of 6, up to a resolution of ö,.
Proof. By definition binary search halves the search interval 6, every iteration therefore
(1/2)"... 8a <ō.
(48)
51
w (1/2)W
<
(49)
8a
W log(2)
< <log((,)) log(da)
(50)
log(O, 0.) log(6a)
W
(51)
log(2)
W = [(log(6.) log(5.)) log(2)]
(52)
4. 4, 10, 000, K 20 for all ceeerimennss-threfffee the dominating scaling parameter is G the number of particle rollouts. We observe in Table 6 that AOC takes approximately 2.4x longer to plan, which includes both planning the actions and when to observe, compared to Continuous Planning which only plans actions (an MPC step). In the case of the Cancer environment, it can take fewer observations and therefore spend less time planning overall. Moreover, AOC is still practical in environments that require fast decision decision-making, i.e., robotics, as AOC can still produce a plan in less time than the planning action interval 5, for the Cartpole environment, shown in Table 7. Therefore, AOC is a practical method that can be utilized across different environments (planning cancer treatment plans with high accuracy, whilst being fast enough control continuous robots). Furthermore, note that MPC has demonstrated scaling high dimensional environments (Chua et al. [2018] with G 20, Lutter et al. [2021] with G=500).
Empirically we chose 10G, W W=
25

---

Table 6: Normalized utilities U, rewards R. observations O, average time taken to plan policy method T (Average), total time spent planning policy in episode ( for the Cancer environment the same normalization as in Table 3. Even though Active Observing Control takes 2.4x longer to plan both the actions and the observing time T (Average) compared to Continuous Planning that only plans the actions-AOC can achieve less total planning time T(Total) (hence less total compute) because it takes fewer observations in total-here Continuous Planning takes 1.5x longer total planning time (Total)
Cancer
Policy
U
R
O
T (Average) (ms)
(TToaal) (ms)
Active Observing Control
105
98.8
3.4
31.1
105
Continuous Planning
100
100
13
12.8
166
Table 7: Normalized utilities U4 rewards R. observations O, average time taken to plan policy method (Average), total time spent planning policy in episode ((Total) for the Cartpole environment-usiin the same normalization as in Table 3. Even though Active Observing Control takes 1.7x longer to plan both the actions and the observing time T (Average), compared to Continuous Planning that only plans the actions-bont policies can be run at real-time as the action interval 6, = 100 ms for the Cartpole environment-make AOC practical method.
Cartpole (Average) (ms)
Policy
U
T (Total) (ms)
Active Observing Control
140
100
43.2
24.9
1073
Continuous Planning
100
100
50
14.0
700
H Practical Guidelines to Select e
The observation cost should be decided based on the real-world application at hand and include any human preferences in that application if applicable. real-world systems (with resource constraints), this cost might correspond to actual monetary cost, computational expense, or energy consumption. Further, if this cost involves a human, for example, a patient receiving medical treatment, they may have preferences for treatment, their health impact (e.g.. chemotherapy side effects), and or the number, frequency, and timing of treatments that could be included in the cost c. Also, there can often be trade-off between control performance and the number of observations taken, and one way to trade this off is to tune
I Environment Selection and Details
In the following we discuss our reasoning for why we selected the continuous-time environments. Principally, a continuous- environment is defined by an underlying differential equation (DE) system (e.g., physics or biological system), which allows for the environment state trajectories to be sampled and simulated at any continuous-time, which is unlike discrete environments [Yildiz et al., 2021, Brockman et al., 2016]. We used an ordinary differential equation solver [Virtanen et al., 2020] to simulate all environments, using Euler solver at time resolution of dzim as indicated by the environments parameters. We selected the standard continuous-time control environments from the ODE-RL suite [Yildiz et al., 2021], which consists of three well known environments: Pendulum, CartPole, and Acrobot. Furthermore, as detailed in the following. we implemented a Cancer environment that uses simulation of Pharmacokinetic- Pharmacodynamic (PK-PD) model hhtpaa/ooooniit--hhaaaaaaaaaaaa
of lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. Where the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. We further adapt all the environments to have a fixed cost when taking a sample of the state. To learn a continuous-time dynamics model, is a standard assumption to assume the dataset of state-action trajectories is sampled irregularly in time [Yildiz et al., 2021,
WWhere the ODE-RI RL suite of the environments used can be downloaded freely from https: //gtt/hb////b///////// com/ cagatayyildiz/on oderl.
26

---

Chen et al., 2018]. First let us justify the reasoning behind our choice of observing irregularly in time state-action trajectories from these environments and why existing offline datasets are not suitable.
Why we cannot use an offline dataset of agent trajectories from a discrete-time environment. There exist standard discrete-time offline datasets [Fu et al., 2020] of state-action trajectories of agents interacting with discrete-time enviromments-where the state-action trajectories are sampled at regular time intervals Aj+1 A Let hypothetically imagine that there exists a regularly-samples
Aj+1 = A, in time state-action trajectory offline dataset. Is it possible to sample the trajectories irregularly? Two approaches come to mind. 1) One could imagine using some form of interpolation (e.g., splines or similar) to interpolate to irregular time steps between states and actions, however doing so would lead errors in the sampled State-action trajectories in comparison to the true irregularly sampled state-action trajectories at those non- non-uniform time points. These errors could compound over dynamics model being trained on these; therefore, we highlight that this approach unsuitable. 2) Another approach would be to start with regularly sampled state-action trajectory and then sub-sample state-action times from that regular grid of collected times. However, again this approach is unsuitable, as often environments time interval Á-where only observing discrete multiples of this (i.e., nA, 12 E Z,+) would lead to gaps between observations, where the mean of the observation intervals would be larger than that of the environments nominal run-time observation interval A Ssub-sample > Soriginal Trajectory. Therefore, we note that this becomes different problem, as there is less information in the state-action trajectories with large observation interval gaps.
captured at run-time with a particular regular observation
Offline dataset generation D. Rather, to mitigate both of these issues above (1,2) we prefer to collect an offine dataset ourselves by observing irregularly in time state-action trajectories, where the time interval between the state-action time points is sampled from an exponential distribution Ai+1 ~ Exp(A), with a mean of Á = 6a seconds, and state-action values are randomly sampled, collecting a total of 1e6 samples [Yildiz et al., 2021]. Furthermore, to increase realism, we add Gaussian noise to observations taken, that of A (1 o²) with standard deviation a = 0.01.
In all environments the actions are continuous and bounded to defined range [Qmin- Amax]. Here we assume a given state s(t) is composed of the position state g and their respective velocities q. i.e.. s(t) = {q(t), q(t)}. Furthermore, each environment uses the reward function of the exponential of the negative distance from the current state to the goal state z², whilst also penalizing the magnitude of action, and we assume that we are given this reward function when planning as we often know the desired goal state q" and our current state q. Therefore, the reward function for the environments has the following form:
(()()))(()),,(()) a())= exp((1la(()) 2 12 - -())()) =I(a(()]13
(53)
Where and v are specific environment constants [Yildiz et al., 2021J. Specifically, when we use our MPC planner observe that it plans better without the exponential operator, and therefore remove it, and use the following reward function throughout, r r({9(t), ((t)): a(t) = ll(((t) q-112 bl 2l Yildiz et al. [2021] set the environments parameters of b, v to penalize large values and enforce exploration from trivial states, and we use their same values which are also tabulated in Table 8. For all environments we assume a fixed cost of 50 when observing a sample
of the state.
Table 8: Environment specification parameters.
Base Environment
b
Omax
8 Init
q"
Pendulum
le -
le 2
[2]
[0.1, 0.1] [0.05, 0.05, 0.05, 0.05)
[0, L]
Cartpole
le
le 2
3
[0, 0, T.
Acrobot
le -
le 2
4.4
[0.1, 0.1, 1, 0. 0.1] [1138,
[0.2T.]
Cancer
le
le 2
5,2
[0,0]
I.1 ODE-RL Suite
The starting state, for all these tasks, is hanging down, and the goal is to swing up and stabilize the pole(s) upright [Yildiz et al., 2021] in each environment.
Specifically, we set 5n = 0.1 seconds, and simulate each ODE system at a time resolution of daim 0.01 seconds, up until the episode length of T 5 seconds. The goal states for these
27

---

ODE-RL suite environments are when the pole(s), each of length L are fully upright, such that their z, y co-ordinates of the tip of the pole reach the goal state. That is where q" is: [o, L] for the the additional 0 is zero for the cart's x location and in Acrobot is (0, 2T. as there are two poles connected to each other. Furthermore, upon restarting the environment the initial state 8 is sampled from the uniform distribution of 5o ~ 2[ -lnit Stni] [Yildiz et al., 2021J, then the 0 states are added with set angle such that the pole(s) are pointing downwards (i.e., Cartpole O' Otnit + π).
Pendulum environment, [0, 0, T.] for the Cartpole environment-
In the following we describe each of the ODE-RL environments introducing the environment with screenshot figure.
L.11 Cartpole (swing up) Environment
(a)
(c)
Figure 7: Screen shots of the Cartpole environment. The task is to swing up a pole attached to cart that can move horizontally along a rail. In the following we see: (a) the starting downward state with an additional small amount of perturbation, (b) the optimal trajectory solution found by policy that scores R ss 100% including our Active Observing Control policy and (c) the final goal state that has been reached, that is, to swing up the pole and stabilize it upwards-whir task. We note that the control actuator is bounded and limited, and the force is such that the Cartpole cannot be directly swung up- rather it must gain momentum through swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position, as indicated when the tip of the pole reaches the centre of the red target square. Furthermore, we note this environment is an underactuated system.
is challenging control
We can see in Figure 7, an illustration of the starting state Figure 7 (a) with a small perturbed random initial start. Here pole is attached to an un-actuated joint to cart that moves along frictionless track [Barto et al., 1983]. The pendulum starts in the downward position Figure 7 (a) and the goal is to swing the pendulum upwards and then balance the pole upright by applying forces to the left or right horizontal direction of the cart. This environment has the state of [ar, å 0, 0] and a corresponding ()). 0). where E (- x measured from the upward vertical of the pole. We note that this environment is an underactuated system, as it has two degrees of freedom (rc, 6). however only the carts position is actuated, leaving / indirectly controlled. Additionally, for this Cartpole environment only, created a more competitive Random policy than randomly executing actions, that of applying no actions keep the pole in the same starting position. This achieves significantly higher reward than a pure Random policy, as the Cartpole environment is E IR unbounded and a Random policy leads to random drift that significantly moves away from the goal position during the episode length, i.e., |ar| > O.
observation of [z, ă, cos(0) sin(0),
I.1.2 Pendulum Environment
We can see in Figure 8, an illustration of the starting state Figure 8 (a) with a small perturbed random initial start. Here a pole (pendulum) is attached to fixed point at one end with the other end being free [Barto et al., 1983, Yildiz et al., 2021]. The pendulum starts in the downward position Figure 8 (a) and the goal is to swing the pendulum upwards and then balance the pole upright by applying torques about the fixed point, as indicated in the Figure 8 with a visualization showing the torque direction and magnitude based on the size of the arrow. This environment has the state of [0, 0) and a corresponding observation of [sin(0), cos( 0).
28

---

(a)
(b)
(c)
Figure 8: Screen shots of the Pendulum environment. The task is to swing up the pole (pendulum). In the following see: (a) starting downward state with additional small amount of perturbation, (b) the optimal trajectory solution found by policy that scores R. x 100% including our Active Observing Control policy and (c) the final goal state that has been reached, that is, swing up the pole and stabilize it upwards. We note that the control actuator is bounded and limited, and the force such that the Pendulum cannot be directly swung up- up-rather it must gain momentum through swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position.
(a)
(b)
(c)
Figure 9: Screen shots of the Acrobot environment. The task is to swing up the 2-link pendulum. In the following we see: (a) starting downward state with an additional small amount of perturbation, (b) the optimal trajectory solution found by policy that scores R ~ 100% including our Active Observing Control policy and (c) the final goal state that has been reached. that is, swing up the 2-link pendulum and stabilize it upwards. We note that the control actuator is bounded and limited, and the force is such that the 2-link pendulum cannot be directly swung op-rather must gain momentum through a 2-link swing and then stabilize this swing to not overshoot when stabilizing the 2-link pendulum upwards the goal position.
L1.3 Acrobot Environment
We can see in Figure 9, an illustration of the starting state Figure 9 (a) with a small perturbed random initial start. It is a 2-link pendulum with the individual joints actuated [Brockman al., 2016]. The 2-link pole starts in the downward position Figure 9 (a) and the goal is to swing the 2-link pendulum upwards and then balance the pole(s) upright by applying torques about their fixed points. This environment has the state of [01, 0,, 02 0z] and a corresponding observation of (sin(O)), cos( ) 0, sin( cos cos(02), dz]. Here the Acrobot environment is fully actuated, as no method has been able to solve the underactuated balancing problem [Yildiz et al., 2021, Zhong et al., 2019].
1.2 Cancer Environment
We implemented a Cancer environment, that uses simulation of a Pharmacokineticc Pharmacodynamic (PK-PD) (a bio-mathematical model that represents dose-response relationships)
29

---

model of lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. Where the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. Here, the tumor volume V at time after diagnosis defined as
dt())
K
)
dt =
plog
((t)
(tt( - -(a,((t)+A +,,((t)²)
V(t)
(54)
Chemotherapy
Radiotherapy
Tumorgrowth
dC((t)
C(t) + c(1)
(55)
Where the state space is (t) E C(t) E R4J, with the V(t) being the cancer volume cm², and C(t) being the chemotherapy concentration the patient. Whereas the action space inputs are (c(t) [0, 5), d(t) E [0, 2]], the chemotherapy input dose c(t) and radiotherapy dose d(t)-which which are given by the policy. Specifically, p. K, ar, S,, Se are effect parameters [Geng et al., 2017], and are defined as in Table 9.
Table 9: PK-PD model parameters for the Cancer environment.
Model
Variable
Parameter
Value
Tumor growth
Growth parameter
P
1.45 x 10
Carrying capacity
K
30.0
Radio cell kill (a)
0.0398
Radiotherapy
Radio cell kill (3)
3,
Set s.t. a
Chemotherapy
Chemo cell kill
Be
0.028
As detailed above. the chemotherapy drug concentration state Ctt) follows an exponential decay relationship with half-life of one day, and c(t) represents dose of Vinblastine up to the concentra- tion of 5. 5.0mg/ /m' per time day. Whereas the radiotherapy concentration d(t) represents 2.0G, fractions of radiotherapy, where Gy is the Gray ionizing radiation dose. Particularly, we set 5, = .4 days, and simulate each ODE system at a time resolution of &gim 0.04 days, up to an episode length of T = 25 days.
The goal state here is reduce the cancer tumor volume zero, alongside zero chemotherapy concentration in the patient, i.e. d² = [0, 0J. Moreover, upon restarting the environment the initial sampled from the uniform distribution of 8p [1120, 0). [1138, O]]. Here assume the patient starts with stage four cancer, the largest stage, with tumor diameter of approximately 13em diameter, assuming that the tumor is spherical. Furthermore, use larger growth parameter p than the nominal one reported by Geng et al. [2017] to model aggressive cancer which is of high interest.
Benchmark Method Implementation Details
All benchmark policies consist of observing policy and an action policy. To ensure competitive benchmarks, each of the following benchmarks are specific ablations of our method, Active Observ- ing Control. Therefore, for a given trained probabilistic dynamics model, the same action policy planner was used across all the benchmark methods, that of the same MPC MPPI planner with the hyperparameters fixed to the below values.
J.1 Probabilistic Dynamics Model
To train our continuous-thn time and discrete-time probabilistic dynamics model we use deep ensemble of fully connected multi-layer perceptions [Lakshninarayanan et al., 2017], adapted for multi- dimensional inputs [Chua et al., 2018]. Specifically, for each individual model in the ensemble, we use 3-layer multilayer perceptron (MIP) of 256 units, with tanh activation functions. We also use the negative ligeeiillhood loss to train each model in the ensemble separately Srrinnnel-trainingg each model
30

---

for the same number of epochs with a different random seed, where the ensemble has M = 5 total models. We use Xavier weight initialization and oulput the log variance, following the setup by Chua et al. [2018). All dynamics models are implemented in PyTorch [Paszke et al., 2019a], and trained with an Adam optimizer [Kingma and Ba, 2017] with a learning rate of le-4. For each environment we train two dynamics models on the offline dataset, that of continuous-time probabilistic model using the time delta input 6, and a discrete-time probabilistic wodel-which is the exact same model architecture without the additional time delta input 6.
All dynamics models predict the next state difference, Az + 8), and we construct the state as ztt + 6) = z(t) + Az(t + 5) following the standard suggestion of [Deisenroth and Rasmussen, 2011].
For training, using the whole collected dataset we pre-process this by a standardization step, to make each dimension of the samples have zero mean and unit variance (by taking away the mean for each also use this step
dimension and then dividing by the standard deviation for each dimension) during run-time for each dynamics model. Furthermore, we train all the baseline models on all the samples collected in the offline dataset (all samples are training data) Lakshminarayanai et al. [2017]. Specifically, we train all the dynamics models on given offline dataset by training each model until convergence, each for at least 12 hours.
J.2 Observing Policies
We implement two discrete-time methods. First by learning discrete-time uncertainty-aware dynamics model (an ablation of the exact same model and hyperparameters, without the time interval input d to predict the next state for) on the same offline dataset DD. Second, we use this discrete-tima uncertainty-ave dynamics model to create two benchmark methods, that of Discrete Planning that samples the state at each action time interval 8, and Discrete Monitoring discrete ablation of our observing policy (Algorithm 3htthat uses the same reward distribution estimation and determines the discrete action sequence execute up until the reward uncertainty crosses the threshold T at discrete-time resolution of da-
Moreover, we also benchmark against Continuous Planning that uses our trained continuous -immt uncertainty-aware dynamics model and takes sample of the state regular time intervals of multiple of 8a Finally, we also compare with random action policy, Random that samples the state at each action interval 5,-
For each policy method, we tune the hyperparameter T is one exists following the procedure outlined in Section 4.2. We then keep this fixed and constant throughout all run-time experiments for each policy method for each environment-unles explicitly noted that we modify it, as in Section 5.3. For all policies we limit the minimum A++1 to be 8a (as actions take time to execute and interact with the environment, therefore we want to avoid observing the same state again immediately, for fast-moving states), and the maximum H the MPC planned action horizon.
J.3 MPPI Implementation
We use the MPPI algorithm, with pscudocode and is further described in Appendix E. Specifically, as is recommended by Lutter et al. [2021] optimized the MPPI hyperparameters through grid search with the continuous-time probabilistic dynamics model for a single environment setting, that of the Cartpole environment, and fix these for planning with all the learned dynamics models throughout. Particularly, our final optimized hyperparameter combination is N = 20, M 1, 000, X = 0.01, o 1.0. Where E the MPPI action noise is defined as:
[o
if = 1
Σ =
02
0.502 o²
(56)
if d =
0.502
Where the Cartpole and Pendulum environments have d,4 1 and Acrobot and Cancer environments has d 2. These hyperparameters were found by searching over a grid of possible values, which detailed in Table 10.
31

---

Table 10: MPPI hyperparameter grid search sweep values.
Hyperparameter
Grid values searched over
K
21 4, 16, 20, 40, 50, 60, 70, 80, 90, 100, 128, 256, 512, 1024, 2048} {1, 4. 8, 16, 20, 40, 50, 60, 70, 80, 90, 100, 128, 256, 500, 1000, 2000, 4000, 8000) {0.00001, 0.0001,000000000 0.0001, 0.001, 0.,, 0.8, 1.0, 1.0, 10.0. 100.0, 1000.0) {0.00001,
G
A
0
1.0, 1.5, 2.0, 10.0, 100.0, 1000.0}
K
Evaluation Metrics
An uncertainty-aware dynamics model trained for each environment on an offline dataset of collected trajectories consisting of 1e6 samples. The trained dynamics model weights then frozen when used at run-time. We record the average undiscounted utility 24 (Equation (1)), average undiscounted reward R and the number of observations taken O, after running one episode of the benchmark method-and repeat this for 100 random seed runs for each result with their 95% confidence intervals throughout. We use fixed observation cost of 50 throughout. Moreover, normalize R and ZI following the standard normalization of offline-RL. [Yu et al.. normaznd
the interval of 0 to 100, where a score of corresponds whhcc we assume here is the Continuous Planning benchmark. Furthermore, we also track the metric of total planning time taken to plan the next action and time schedule sample and perform all experiments using a single Intel Core i9- 1900KK CPU @ 3.20GHz, 64GB RAM with a Nvidia RTX3090 GPU 24GB.
random policy performance, and 100
to an
L Additional Experiments
L.1 Ablation of Thresholding the State Uncertainty
Intuitively we threshold the variance of the value function rather than the variance of the state as is possible to achieve task with very certain value despite having uncertain states. For example, when there exist multiple ways to achieve a task-and we know that multiple action trajectories guarantee this where we can take any; however, we are uncertain about which one.
We performed additional ablation experiment where we threshold the variance of state instead and compare the utility against Active Observing Control that thresholds the variance of the predicted reward instead.
Table 11: Ablation experiment of using a threshold on the variance of state instead of the predicted reward, for the Cancer environment.
Policy
Utility 4
Active Observing Control (Reward Uncertainty Threshold) Active Observing Control (State Uncertainty Threshold)
106_0.789 103+1.6
For each policy, we tuned the threshold following the tuning setup, in Section 4.2. Here we evaluated the utility metric over 10 random seeds for each policy on the Cancer environment. As tabulated in Table 11, it is possible to threshold on the state variance for an acceptable performing policy, however, it is preferable and more intuitive to threshold the variance of the predicted reward uncertainty.
Furthermore, we performed an additional ablation experiment where we threshold the variance of the state instead and compare this to the utility of Active Observing Control that thresholds the variance of the predicted reward instead, on the Cartpole environment, tabulated in Table 12.
We again tuned each observing policy's 's threshold following the tuning setup in Section 4.2. Here we evaluated the utility metric over 100 random seeds. Empirically we observe that is indeed possible to threshold the state variance for an acceptable performing observing policy. However, is again preferable and more intuitive to threshold the variance of the predicted reward uncertainty.
We also highlight the Cartpole environment notably more complicated than the Cancer environment, as it is an underactuated system, as it has two degrees of freedom [x, 0). however only the carts
32

---

Table 12:
Ablation experiment of using a threshold on the variance of state instead of the predicted
reward, for the Cartpole environment.
Policy
Utility
Active Observing Control (Reward Uncertainty Threshold) Active Observing Control (State Uncertainty Threshold)
144+5,51 98.4+-2.4
position is actuated, leaving 9 indirectly controlled. We note that the control actuator is bounded and limited, and the force is such that the Cartpole cannot be directly swung up-rather it must gain momentum through swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position, indicated when the tip of the pole reaches the center of the red target square.
L.2 Sparse Reward Environment
To investigate how well Active Observing Control generalizes to other environments that have possibly sparse reward functions, we performed an additional sparse reward experiment. We implemented sparse Pendulum environment [Chakraborty et al., 2022] which only provides reward when the Pendulum state angle is within =uCoon degrees from the goal state upright (0" 0). That is using sparse reward function defined by r'(s, a, r(s, a, t) [ Ocmon). This allows us to vary the degree of sparsity by decreasing fcwor to create a sparser reward for the Pendulum environment. This is tabulated in Table 13, with each result averaged over 100 random seeds, and normalize scores to 100 for Continuous Planning with Cuon 180%
Table 13: Sparse reward Pendulum environment.
lcuoi
Policy
Utility U4
Reward R
Observations
Pcuon 180
Random
01:0
0::0
13:10
Ocusoar = 180
Continuous Planning
100)1-9.48
100:19:48
50:10
Ocuront 60
Continuous Planning
18.8+11.77
18.8
5010
fcuoa =
Continuous Planning
-..01=0.00115
-..01000001155
50:10
fcurnn = 180
Active Observing Control
109
98.5+10.2
48100955
fcuor 60%
Active Observing Control
86.7+113.5
12::
35.4=
OCuron = 5
Active Observing Control
223+2.32
-7.8140.00146
4.8-0.452
We note that Active Observing Control relies on MPC being feasible in the environment and that is possible to solve the environment with the chosen MPC decision policy π with regular observing at each time step. In our experimental setup, this translates that the baseline of Continuous Planning should be able to solve the environment to achieve an optimal reward (i.e., R = 100).
The above results demonstrate the well-known [Karnchanachari et al., 2020] limitation of MPC in general, that a vanilla MPC decision policy π struggles in environments with sparse rewards. We highlight that this is not unique to MPC, and standard RL methods in general significantly struggle with sparse rewards in continuous-state and continuous-action tasks (Chakraborty et al., 2022] without additional assumptions data (reward shaping or using expert demonstrations). This arises due to the limited receding planning horizon time H, however, there exists works to combine trajectory optimization (MPC) with learning reward estimation [Karnchanachari et al., 2020, Hoeller et al., 2020]. This is an interesting direction for future work, though we believe this is out of scope for this paper.
L.3 Increasing the Planning Resolution of MPC (decreasing 5..)
We note that planning with a higher resolution MPC planner (smaller 5..) for all the baselines will make the baselines perform on par with Active Observing Control for that 8_. However, motivation to compare the observing policies of each baseline, therefore all the implemented baselines we the same decision policy æ of the same MPC algorithm with the exact same hyperparameters.
We performed an additional re-run of the baselines on the Cancer environment, with varying ó, 1. 0.05, 0.02, 0.01] and each result averaged over 100 random seeds, as tabulated in Table 14. We
33

---

kept the MPC planning horizon time H fixed, to the same value as in our original experiments, and let the number of planning time steps K vary, i.e. K 8
Table 14: Increasing the planning resolution of MPC (decreasing 6x) for the Cancer environment.
Discretization interval 6
Policy
Uiility U
Reward R
Ohservations O
0.1
Random
0++0
0++
13++0
0.1
Discrete Plamning
92.2+t1.11
92.2+1.11
13++0
0.1
Discrete Monitoring
91.1=1.72
8..9+1.7
4.94=0.109
0.1
Continuous Planning
100+0.559
100+0
13++0
0.1
Active Observing Control
105+0.75
98.4:0.707
3.42:00102
0.05
Random
0=:0
0+0
13:00
0.05
Discrete Planning
9.11
95.6#1.68
13+0
0.05
Discrete Monitoring
96.3=11.9
91.6=1.47
7.18=0.153
0.05
Continuous Planning
100+1.36
100
13:00
0.05
Active Observing Control
104:1.229
98.1
4.95=0.118
0.02
Random
0=0
0=0
13::0
0.02
Discrete Planning
9511.25
92.8=1881
13:00
0.02
Discrete Monitoring
95.9:11.17
88,2+1.7
6.510.196
0,02
Continuous Planning
100:11.17
100
13:00
0.02
Active Observing Control
98.6=0.967
88.8+1.43
2.77=0.0839
0.01
Random
010
010
13:10
0,01
Discrete Planning
102.10.806
105.1187
13.00
0.01
Discrete Monitoring
102.10.844
9811.93
5.8910.0744
0.01
Continuous Planning.
10011.03
100.2..99
13.0
0.01
Active Observing Control 103.10.937
97.8
4.79L0.117
Where for each & we normalize utility to be between 100 and 0, where 100 corresponds to Continuous planning at that 6a and O a random policy. We note that 6g = 0.1 was used, as this is the default value for the RI. environments [Yildiz al., 2021].
Moreover, in light of the empirical results (Table 14), we can also ask what the performance looks like if we only normalize utility to Continuous planning with 5, = 0.1 (the best possible reward) for each baseline whilst varying 6a, as tabulated in Table 15.
Interestingly we observe all baselines degrading in performance, however, AOC still has the highest utility amongst the baselines for a set 6g. Intuitively the MPC MPPI algorithm produces less optimal action trajectory a plans when the number of discrete actions K in the discrete action sequence increases for a fixed number of parallel rollouts G. We note that this is inherent to using an MPC planner [Williams et al., 2017], and could be improved by increasing G for decreasing 6,, however, that is out of scope for this work.
L.4 Discrete Time Controller Trained on an Equidistant Dataset
The Continuous Planning baseline (which uses continuous-time time dynamics model) perform better than the Discrete Planning baseline (which uses a discrete-time dynamics model), as it has a more accurate dynamics moeel-when both trained on the irregularly sampled offline dataset D= [(=(4), ((4),, 1- As the offline dataset of state-action trajectories DD Ai41 has irregular times between state-action samples A Exp(A), with mean of A 8, seconds.
We test what would happen the discrete-time dynamics model (for the Discrete Planning and Discrete Monitoring baselines) was instead trained on regular (equidistant) sampled offline dataset D. i.e., A++1 = = 8g, we performed an additional experiment. This is in the Cancer environment, with each result averaged over 100 random seeds, Tabulated in Table 16.
Empirically this agrees with our intuition that the discrete-time dynamics model baselines indeed perform better when trained on offline dataset that is regularly sampled. However, the discrete-tine baselines still underperform Active Observing Control, due to their inherent time discretization error, Figure 6.
34

---

Table 15: Increasing the planning resolution of MPC (decreasing 8,) for the Cancer environment. Here we only normalize utility to Continuous planning with 6, - 0.1 (the best possible reward) for each baseline whilst varying da.
Discretization interval 6
Policy
Utility 2
Reward R
Observations O
0.1
Random
0=:0
0+:0
13:10
0.1
Discrete Planning
92.2.11.11
92.2.11.11
13110
0.1
Discrete Monitoring
91.111..72
85.911.7
4,94.10.10
0.1
Continuous Planning
100.10.597
100.0..99
13.10
0.1
Active Observing Control
105....5
98.410.707
3.42.10.102
0.05
Random
0.10
0.10
13.10
0.05
Discrete Planning
85.6:1
8..6.11.44
13:10
0.05
Discrete Monitoring
8..
82.1
7.18:00.13
0.05
Continuous Planning
8..3
89.3
13::0
0.05
Active Observing Control
92.9:11.2
87.7+1.14
4,95:10.118
0.02
Random
0=:0
01:0
13:0
0.02
Discrete Planning
71.811:.3
71.81.3
13:0
0,02
Discrete Monitoring
72.8:11.22
68.611.23
6.510.196
0.02
Continuous Planning
77.1:1.22
77.1
13:0
0.02
Active Observing Control
75.6:11.01
6911.03
2.7710.0839
0.01
Random
0++0
0+:0
13+0
0.01
Discrete Planning
63.2++1033
63.2+1.03
13++0
0,01
Discrete Monitoring
63.8++1077
19.2+
5.89+0.0744
0.01
Continuous Planning
60.3+1.32
60.3+1.32
13+0
0.01
Active Observing Control
64.4+t1.19
59.1
4.79+0.117
Table 16: Results for the Cancer environment where the discrete-time dynamics model trained on regular (cquidistant) sampled offline dataset D.
4 Distribution
Policy
Utility
Reward
Observations O
Irregularly sampled, A1+) Exp(A)
Random
0.10
0.10
13.00
Regularly sampled, Ai+1 A
Discrete Planning
95.5100.67
95.5110.67
1310
Regularly sampled, Ai+1 A
Discrete Monitoring
96.1=1..3
90.111.83
3.75:0.127
Irregularly sampled, A4+1 (x)(A)
Continuous Planning
100++0.594
100+0.594
13+-0
Irregularly sampled, A441 Exp(A)
Active Observing Control
0.755
98.5+
3.4+:.1122
L,5
Sensitivity of AOC to 6,
8a is often chosen for user, as it is the average time interval between two state-action points in trajectory in the collected offline dataset
Here MPC depends on a given 5a. however, the observing policy is orthogonal to 8,- As outlined in Appendix L.3, reducing 8 whilst keeping the planning time horizon H and the number parallel rollouts G the same, reduces the MPC MPPI action trajectory quality, leading to control performance that achieves lower reward, and hence utility.
We performed an additional experiment to verify this on the Cancer environment, varying 6₂ = (0.1, 0.05, 0.02, 0.01), }, each result averaged over 100 random seeds, as tabulated in Table 17. Here we normalize utility to Continuous planning with 5, = 0.1 (the best possible reward) for each baseline whilst varying 8...
Table 17: Results for the Cancer environment where vary d, for AOC.
Discretization interval 6,
Policy
Utility 4
Reward R
Observations O
0.1
Random
0=:0
01:0
13:10
0.1
Continuous Planning
100:1.597
10010.597
1310
0.1
Active Observing Control
105:10.75
98.4=0.707
3,42:10.102
0.05
Active Observing Control
92.911.2
87.711.14
4.95:101118
0.02
Active Observing Control
75.6+1.01
69+1.03
2.77+0008839
0.01
Active Observing Control
64.4+1.19
59.1+1.13
4.79-H0.117
35

---

L.6 Evaluation of Reward Uncertainty
The reliability of the reward uncertainty depends on the reliability of the probabilistic dynamics model fo provide good predictive uncertainty for future state. We highlight that the predictive uncertainty of the dynamics model depends only on the training. that training on the offline dataset of trajectories D D-which is only performed once. At run-time the dynamics model is only used for inference, therefore its predictive uncertainty for the state- itate-action space will remain the same, independent of the number of samples of the state taken in given evaluation episode.
To illustrate this point, we performed an additional experiment on the Cancer environment where, at run-time, vary the number of observations (samples of the state taken) in an episode and evaluate the negative log-likelihood (NLL) [Lakshminarayanan) al., 2017] of the predicted reward uncertainty to the ground truth reward uncertainty for 100 random seeds, as tabulated in Table 18.
Table 18: Evaluation of reward uncertainty of AOC on the Cancer environment.
State Observations
NLL
1
8,261 = 1.124
10
8.138 = 1.060
100
8.190 = 1.052
1,000
8.241 = 0.973
10,000
7.933 = 0.288
100,000
7.988 = 0.093
L.7 Relation Between Rewards and the Fixed Observation Cost
An exact theoretical relation between rewards R and the fixed observation cost c intractable to provide for the general case.
We encounter this intractability even for our simple analytic system used to prove Proposition 2.1. As let us start with Equation (1,,4 = J/" a(()
- clt, : 1 E [o, T)H, where the Reward is R = r(s(t), a(t), t) and Cost is = cttt : t, E [o, T.H. For even a simple regular observing policy (Appendix C), E ax a3 [2]. which a function of (0, T). well the reward function r(s(t), a(t), t) and the observation cost c.
we keep r, тT C fixed, and attempt to find p" argmax, E[4 by only varying 6, that is by d dB Biu 0, calculating closed-form solution for 25 B[44] is intractable.
Instead, we can empirically simulate different values of 8, then which grid sweep value of maximizes E[(4]. such that approximately 8* Brgm this value can then be used to calculate the Reward R f-_ r(s(t),
We do this below on the Cancer environment where we use a regular observing policy, that of Continuous Planning, and vary 5 over grid sweep such that different settings of the policy take different observations, as tabulated in Table 19. We start with setting e = 1 and only normalize the Reward R to 100 for the most frequent samples possible (O 13), each result calculated for 100 random seeds.
Table 19: Cancer environment where we use regular observing policy, that of Continuous Planning. and vary 8 over grid sweep such that different run settings of the policy take different observations.
Observations O
Reward R
Utility 24
13 + 0
100 + 0.616
87
7
99.3 3.65
92.3
4
96 0.821
92
2+0
90.7 1.01 89.2 - 1.1
88.7
1
88.2
36

---

For a given cost i.e. in this case 1, we see that there exists an approximate number of observations that maximize the Utility, here for observation cost numerically, and for each observation cost, determine the number of observations (or regular observing frequeney) that maximizes the Utility U (O* argmaxx.R((1) and determine what the Reward R for this O* is. This is tabulated in Table 20.
this is O 7. This allows us vary the
Table 20:
Cancer environment where we vary the observation cost c numerically, and for each obser- vation cost, determine the number of observations (or regular observing frequency) that maximizes the Utility (O" = argmaxx)i((() and determine what the Reward R. for this O is.
Fixed Observation Cost
Optimal Utility ur
Optimal number of Observations Or
Reward
100
13
100
92.3
99.3=10.656
2
88
4
9610.821
86.2
1..2+t1.1
Empirically we observe the relation between the and R is that the fixed observation cost is negatively correlated to Reward R. when using policy that maximizes the Utility U1 Equation (1).
L.8 Dependence on the Accuracy of the Learned Dynamics Model
To understand the dependence on the accuracy of the learned dynamics model further, we performed an additional experiment ablation, where we benchmarked all methods with a less accurate dynamics model-traiiee training all the dynamics models with fewer samples (10% of the total amount of samples used training the dynamics models presented in the main paper), here trained on 100,000 samples. This is tabulated in Table 21.
We observe that AOC still achieves high average utility U all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Specifically, this empirically further verifies the key theoretical contribution that regular observing is not optimal and that irregularly observing can achieve higher expected utility.
Table 21: Ablation with less accurate dynamics model-trainee all the dynamics models with fewer samples (10% of the total amount of samples used in training the dynamics presented in the main paper), here trained 100,000 samples. We observe that AOC still achieves high average utility 2 on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities U1 rewards R and observations O for the benchmark methods, across each environment. Results are averaged over 1,000 random seeds, with indicating 95% confidence intervals. Utilities and rewards are undiscounted and normalized be between and 100, where O corresponds to Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations every 8a-
Cancer
Acnoot R
Carpole R
Pondilom 12
Polley
14
Rundon
D&0
0:0
060
0s0
S0ad
Disrete Plauing
99 2..3600001
99.3.5 T00
1340
67.26 d6
67 27.266.4
Suid
117=900
117:5505
3050
$1:313
$163.13
Discrte Meckoiing
69 1113:773
4.5120161
143664
4056.37
5.884 111
705-166
47.1-169
6-0
34961 18
13.61
6.340
Costinues Plasting
705
10060754
100:6.74
103-1.51
100-3.51
10062997
100=2.87
Aetive Olbersing Control
1941 8.767
1821009955
157 (690
S151697
1.291000
R25 2.38
217/1.28
94.611.35
1.91/00577
L.9 Extending AOC to Work with Learned Reward Model
To investigate whether AOC can also be used with a learned reward model, we performed an additional experiment, where we trained an MLP reward model (4-layer MLP with 128 units and Tanh activations) from the offline dataset in all the benchmarks- -ths is tabulated in Table 22.
We observe that AOC still achieves high average utility /4 on the Cancer environment, outperforming the competing Continuous Planning and Discrete Monitoring methods. This empirically verifies that our proposed initial approach can still perform well using learned reward model and that the theoretical contribution still holds.
37

---

Table 22:
We use learned MLP reward model (4-layer MLP with 128 units and Tanh activations) from the offline dataset in all the benchmarks. We observe that AOC still achieves a high average utility U on the Cancer environment, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities Ut rewards and observations O for the benchmark methods, across the Cancer environment. Results are averaged over 1,000 random seeds, with + indicating 95% confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every 8a-
Policy
R
Random
0+0
0=00
13:00
Discrete Planning
91.4
91.4::0.83
13=0
Discrete Monitoring
95,1=0044
91.6=t0.322
7.63=1
Continuous Planning
100
10010.151
13=0
Active Observing Control 105
98.9110168
3.38
L.10 AOC Also Empirically Works for Non-linear State Transformations
We performed an additional experiment to investigate AOC's performance within environments
that utilize observations stemming from non-linear state transformations. We tailored the existing Cancer environment to render observations via the non-linear state transformation function =t) = (st(s + c(t))² (s(t) + (t)). The subsequent results, conducted across 1,000 random seeds, outlined in Table 23. We observe that AOC still achieves a high average utility 2/ on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Specifically, this empirically further verifies our key theoretical contribution that regular observing is not optimal and that irregularly observing can achieve a higher expected utility.
Table 23: We adapted the existing Cancer environment to render observations via the non-linear state transformation function =t)) = (.1 + c(t)² + (s(t) + c(t). We observe that AOC still achieves high average utility U/ all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities 44 rewards R and observations O for the benchmark methods, across the Cancer environment. Results are averaged over 1,000 random seeds, with + indicating 95% confidence intervals. Utilities and rewards are undiscounted and normalized be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds the expert, that of Continuous Planning, taking state observations at every 6a.
Policy
U
R
Random
0+0
0+00
13:1:0
Discrete Planning
91.7:10.997
91.7:10.97
13::0
Discrete Monitoring
90.5:10.559
85.4810.548
5. 5.25:10.0329
Continuous Planning
100-10.21
100-10.21
13:10
Active Observing Control
104+0.222
98.9
4.68+00.0315
L.11 Further Empirical Validation in Other
Scenarios
To verify the theoretical contribution in additional real-world scenarios, we show empirically in the following that AOC can successfully operate in the real-world environments (medical and engineering) of an accurate Glucose environment (controlling the injected insulin for a diabetic patient [Lenhart and Workman, 2007]), a (Human Immunodeficiency Virus) II/V environment (controlling the chemotherapy dose for affecting the infectivity of HIV in a patient [Butler et al., 1997]), and Quadrotor environment (controlling the actuators of an unmanned aerial vehicle [Nonami et al., 2010]). Each environment represents a unique challenge and has direct implications in medical and engineering applications. thus reflecting the broad applicability of our method. These are:
Glucose environment. Controlling the injected insulin for diabetic patient to regulate their blood glucose level-here observations are costly as a blood test must be performed to measure the glucose level [Lenhart and Workman, 2007].
38

---

HIV environment. Controlling the chemotherapy dose for affecting the infectivity of HIV a mattent-where observations are costly blood must be performed measure CD4*T cell levels [Butler et al., 1997].
Quadrotor environment. Controlling the actuators of an unmanned aerial Sthicle-where observations can be costly due performing expensive (power and compute) localization measure [Nonami et al., 2010].
We observe that AOC still achieves high average utility U on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods, reinforcing our theoretical claims, and extending our empirical validation as tabulated Table 24.
Table 24: Normalized utilities U, rewards R. and observations O for the benchmark methods, across each environment. AOC performs the best across all environments. Results are averaged over 1,000 random seeds, with indicating 95% confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every 8a.
Glueoss
HIV
Policy
Randon
0:20
G:1
50.50
O20
50:10
030
0::0
$0.:0
Discrete Planzing
96:80-855
96:20.855
50:00
15250.12 121
152:0.121
50:10
101:0000582
10120000852
50.00
Dieende Monitoing
120H-0-489 020
92.9+0 403
15,7 15,6
2. 77-440--0-55
126-0455
7+0
1.79p-0--0 199
10:+000142
5.99
Coninuoe: Planing
100+0.30 30
100+0.30 39
50:01
100=00-31
100=0-431
50:10
100t-0015
100+0.005
50:00
Active Observing Control
126:10.41
99.5:0039
17.5:000366
2.83e033:1.56
107:0:878
5.75:00069
1898-488:000739
97.9H0..2558
$:6000196
39


Reviews:
[{"review_id": "aLxqFi0l5w", "reviewer": "Reviewer_U2ft", "summary": "The authors present a continuous-time framework for \"active observation\", meaning deciding when to take observations in a control problem, assuming taking observations has a cost. They present two separated controllers, one for taking action and one for deciding when to observe. The controller for acting is an MPC controller, the controller for deciding when to observe is based on binary search. They learn a continuous-time stochastic dynamics model using an MLP with delta time in the input, with ensembling to model epistemic uncertainty and a gaussian prediction to model aleatoric uncertainty.", "questions": "For the experiment in figure 3, what happens if you use discrete monitoring with a slightly smaller timestep? Same for figure 4; to me this feels simply like the time discretization is too big and the gain is simply from the fact that the observation controller has a higher frequency.\n\nHow would you handle the interesting case where the cost of observing depends on the state?\n\nHow would you create a combined act+observe controller?", "limitations": "The authors are honest about the limitations of their method. I would add the limitation of the observation simply being the true state plus some noise. ", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The problem is really interesting, and it's obvious from the introduction that this is something that should be worked on. The paper is well-written.", "weaknesses": "Let's start with a few things that I think are incorrect in the math. Line 203 seems to state that a mixture of gaussian (summing the pdfs, with uniform weight) is a gaussian. That is not correct. You can compute the mean and variance of the mixture, as your equations suggest, but that doesn't make it a gaussian.\n\nSame thinking from 241 to 246. With your ensembling, the z distributions at t^{k+1} won't be gaussian, and neither will be your reward. Yes you can compute their mean and covariance, but the reward distribution won't be gaussian unless your state distribution is gaussian and the reward function is linear.\n\nBeyond the dynamics model which is truly continuous-time (can predict future observation at an arbitrary floating point resolution), I think the proposed control algorithm is decisively discrete-time. The delta_t for observing is smaller than delta_a for control, but that's it: these are two discrete-time controller with differently size timesteps. There is no notion of adaptive step size during control, the controller makes no use of the continuous time formulation in either case. The advantage for the approach, if I understand correctly, does not come from the continuous-time formulation, but merely for allowing the observation controller to control at a higher frequency than the action controller.\n\nThe dynamics model is only conditioned on the previous observation z. But in general, you need the whole history of noisy observations and action to predict the next, especially in noisy scenarios.\n\nThe formulation is limited by the fact that the observation is simply the true state plus some gaussian noise, rather than being a non-linear transformation of the state.\n\nThe claim that ensembling captures epistemic and gaussian prediction captures aleatoric uncertainty is vague to me. Intuitively that both should capture both type of uncertainty to some degree; can you provide a reference?\n\nTable 3: the bold font should be applied to continuous planning when it performs better or just as well. So continuous planning - cancer reward, acrobot reward, and pendulum reward should be bold.\n\nIn Krueger et al, they actively sample rewards, not observations\n\nLine 73, the wiggle should be an equal sign\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions!\n## (A) Distributions of state and reward\nWe agree that a uniformly weighted mixture of Gaussians is not a Gaussian itself, rather it is possible to compute the mean and variance of the mixture to approximate it with a Gaussian, and we have now fixed this typo in line 203.\n\nAllow us first to clarify that we follow the standard model-based uncertainty-aware dynamics model methodological setup from the seminal work of Chau et al. 2018. By doing so, we perform Monte Carlo Sampling of simulating state particle $z_p(t^{(k+1)})$, roll-outs of state trajectories, *to capture the multi-modality* of the state and reward distributions [Chau et al. 2018]. You are correct that when forward propagating to determine the future state and reward distributions, the distribution of the state particles $z_p(t^{(k+1)})$ is not Gaussian. Although we do compute the standard deviation of the reward particles at an evaluation time, we find this *approximate* statistic to empirically work well in AOC—which is the *simplest* instantiation of a method to solve this new problem setting and verify the key theoretical contribution that irregular observing policies can achieve a higher utility than regular observing policies.\n \n## (B) Why can’t you create a discrete observing policy at a higher frequency?\nWe agree that creating a discrete observing policy with a higher frequency is possible, and we already provide an experiment in **Appendix K.4** to show this. Allow us to re-iterate the goal of the paper is to formalize the problem of continuous-time control with observation costs, provide the theoretical proof that irregular observing policies can achieve a higher utility than regularly observing policies, and verify this key theoretical contribution with the simplest initial instantiation of a method, that of AOC. \n\nThe inherent benefit of the continuous-time dynamics model is that it allows arbitrary evaluation at any future time, which is beneficial when determining the time that the reward variance crosses the threshold, which is determined up to a tolerance of $\\delta_t$, as is common for any root finding algorithms (e.g., Newton’s Method). \n\nHowever, using an observing policy that is discrete at a higher frequency is **not practical**, as $\\delta_t << \\delta_a$, computing the reward variance at every $\\delta_t$ within the whole action trajectory duration $H$ scales poorly—commonly known as *linear search*, and scales as $\\mathcal{O}(n)$, where n is the worst case number of evaluations of the reward variance, $n = \\lfloor \\frac{H}{\\delta_t} \\rfloor$. Instead, AOC employs the simplest adaptive search method, that of *binary search*, which scales as $\\mathcal{O}(\\log(n))$, which **is practical**. We now include this discussion in Section 4.2.\n## (C) Discrete monitoring with a smaller timestep\nBuilding on the above, we already provide empirical results in **Appendix K.4** (line 1036) for discrete monitoring with a smaller timestep $\\delta_a$. As we reduce $\\delta_a$, all baselines degrade in performance; however, AOC still has the highest utility amongst the baselines for a set $\\delta_a$—with the gain over discrete monitoring reducing. The improvement gain of AOC compared to discrete monitoring arises from:\n1. Avoiding time discretization errors, finding $\\rho(z(t_i))^*$ up to a tolerance of $\\delta_t$ compared to a large discretization of $\\delta_a$ (where $\\delta_t << \\delta_a$)\n2. scales much better as the tolerance $\\delta_t$ is reduced, $\\mathcal{O}(log(n))$ versus discrete monitoring of $\\mathcal{O}(log(n))$\n3. the continuous-time dynamics model is more accurate as it can learn from the irregularly observed offline dataset, whereas the discrete-time dynamics model does not use $\\delta$ as an input.\n## (D) Dynamics model is only conditioned on the previous observation\nThe implementation dynamics model choice does not affect how the different observing policies $\\rho$ compare, as they *all use the same form of dynamics model*. We chose this as the simplest dynamics model, which happens to be a Markovian neural network. However, we could have used an RNN-based continuous-time model to encode the whole history of observations.\n## (E) Non-linear state transformations\nLet us clarify that we follow the standard assumptions for an environment description in control [Yildiz et al., 2021]. We agree that observations cannot be non-linear in the current formulation. However, it *does allow for non-linear state transitions*. We have clarified this in Section 2.\n## (F) Capturing both aleatoric and epistemic uncertainty with the probabilistic dynamics model\nLet us clarify that this is well known and comes from the key reference of Chau et al. 2018, which provides an excellent explanation in their Section 4, from the top of page 4—we reference it on lines 180,188.\n## (G) Clarification of related work w.r.t. Krueuger at al. 2020\nKrueger et al. 2020, focus on the simpler problem of multi-armed bandits (MAB), where there is a cost to observe the reward. In the MAB setting, it is possible to formulate it as an RL PO-MDP with one state. The underlying static state (the fixed reward distributions of each bandit) is unknown and must be determined by taking successive costly observations (paying a cost to observe a reward). Therefore, Kreuger et al. 2020’s statement that the optimal algorithm for their simple MAB setting is intractable applies to our problem. We now clarify this in Section 3.\n## (H) Extensions\nHandling state-dependent observation costs requires intricate reward scaling. A combined act+observe controller with MPC would scale poorly in complexity. These are beyond our current scope but are valuable directions for future work. We thank the reviewer for their suggestions.\n## (I) Typos\nThank you, we will un-bold the reward in Table 3, as only Utility should be bolded as that is our ultimate objective to maximize; yes, line 73 should be $=$."}, {"review_id": "CNUMqyVnrE", "reviewer": "Reviewer_wAUT", "summary": "The paper addresses the problem of controlling continuous-time environments while actively deciding when to take costly observations in time, which is relevant to real-world scenarios such as medicine, low-power systems, and resource management. Existing approaches either rely on continuous-time control methods that take regular, expensive observations in time or discrete-time control with costly observation methods, which are inapplicable to continuous-time settings due to the compounding discretization errors introduced by time discretization. The paper formalizes the continuous-time control problem with costly observations and shows that observing at regular time intervals is not optimal in certain environments, while irregular observation policies yield higher expected utility. The paper proposes an initial method called Active Observing Control (AOC) to solve the problem of continuous-time control with costly observations and demonstrates how AOC can avoid discretization errors in time and achieve a better utility as a result. The paper empirically verifies the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations.", "questions": "1) The paper proposes an initial method called Active Observing Control (AOC) to solve the problem of continuous-time control with costly observations, but determining the optimal method remains an open problem.\n2) The paper constructs a simple initial method to solve the problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner, which may not be optimal in all scenarios.\n3) The paper assumes that the dynamics model is learned accurately from an offline dataset, which may not always be feasible or accurate in practice.\n4) The paper only empirically validates the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations, and further empirical validation in other real-world scenarios is needed to establish the generalizability of the proposed method.", "limitations": "1) The paper proposes an initial method called Active Observing Control (AOC) to solve the problem of continuous-time control with costly observations, but determining the optimal method remains an open problem.\n2) The paper constructs a simple initial method to solve the problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner, which may not be optimal in all scenarios.\n3) The paper assumes that the dynamics model is learned accurately from an offline dataset, which may not always be feasible or accurate in practice.\n4) The paper only empirically validates the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations, and further empirical validation in other real-world scenarios is needed to establish the generalizability of the proposed method.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The paper addresses an important and unexplored problem of controlling continuous-time environments while actively deciding when to take costly observations in time, which is relevant to real-world scenarios such as medicine, low-power systems, and resource management.\n2) The paper provides a theoretical framework for the problem and shows that irregular observation policies can achieve higher expected utility than regular observation policies in certain environments.\n3) The paper proposes an initial method called Active Observing Control (AOC) to solve the problem of continuous-time control with costly observations, which can plan action trajectories in continuous time and plan when to observe next in continuous time.\n4) The paper demonstrates how AOC can avoid discretization errors in time and achieve a better utility as a result.\n5) The paper empirically verifies the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations.", "weaknesses": "1) The paper proposes an initial method called Active Observing Control (AOC) to solve the problem of continuous-time control with costly observations, but determining the optimal method remains an open problem.\n2) The paper constructs a simple initial method to solve the problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner, which may not be optimal in all scenarios.\n3) The paper assumes that the dynamics model is learned accurately from an offline dataset, which may not always be feasible or accurate in practice.\n4) The paper only empirically validates the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations, and further empirical validation in other real-world scenarios is needed to establish the generalizability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions!\n## (A) The optimal method remains open\nWe agree—in fact, this is the primary motivation for our paper, to first formalize this problem, theoretically contribute a property that an optimal method should have and lay the groundwork for the development of further methods to tackle this “important” and “unexplored” problem of continuous-time control whilst deciding when to take costly observations.\n\nThe theoretical contribution of this work is a proof that shows regular observing is not optimal and that irregularly observing can achieve a higher expected utility. We empirically verify this key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations by constructing the simplest initial method to do so (AOC).\n \nComing up with an optimal method for all scenarios and systems could be *intractable*. Krueger et al. 2020, who focus on a simplified setting of multi-armed bandits where they must pay a cost to observe, propose that for their problem **an optimal method is intractable**—which applies to our problem.\n \nAs we agree that the optimal method remains open, we clearly state this throughout the paper, in the *abstract (line 18)*, in the *contributions (line 58)* and in the *conclusion and future work section (line 391)*. We believe solving the optimal problem is out of scope for the current paper (and is most likely intractable, i.e., there is no efficient algorithm to solve it), however, we have provided a key theoretical property and empirically verified, that an optimal method should have.\n## (B) The simple initial method to solve the problem may not be optimal in all scenarios\nWe agree—building on the above rebuttal response **(A)**, such an optimal method for all scenarios may be intractable. Hence, this motivates the need to rely on heuristics to solve the problem, as suggested by Krueger et al. 2020—which is how our initial method (AOC) works, using a heuristic threshold on the variance of the reward rollouts in an offline continuous-time model-based predictive control planner.\n \nAllow us to kindly re-iterate that we explicitly state in the limitations paragraph in line 392 that this “initial solution in our experiments may not be optimal for all scenarios”. Let us restate the official 2023 NeurIps reviewer guidelines, *“In general, authors should be rewarded rather than punished for being up front about the limitations of their work”*. Furthermore, all model-based RL methods, and RL methods in general, are not optimal in *all scenarios*. A few methods can theoretically find an optimal policy *only in restrictive scenarios* (certain conditions), for example, Q-learning with an MDP, finite and discrete state and action spaces, stationarity, infinite exploration, and a fully observable environment [Sutton et al. 2018].\n## (C) Assumes that the dynamics model is learned accurately from an offline dataset, which may not be feasible or accurate in practice\nWe agree and explicitly state this in the limitations paragraph on line 394. This is a key assumption of *all model-based offline RL methods* in general.\n1.\tAll offline methods assume that there exists a previously collected dataset of state-action trajectories, as they cannot interact online by definition [Ernst et al. 2005, Levine et al. 2020]. We clearly state throughout the paper (abstract, introduction, contributions, problem definition, etc.) that we are working in the offline setup, where it *is feasible* to have access to an offline dataset. Furthermore, we provide an additional discussion in **Appendix D** of the benefits of offline RL and model-based RL.\n2.\tAll offline model-based RL methods rely on the offline dataset covering sufficient state-action space and that the learned dynamics model is accurate enough [Sutton et al. 2018, Levine et al. 2020, Lutter et al., 2021].\n\nWe verified empirically that AOC can perform well even when the dynamics model is *less accurate* by performing an *additional experiment ablation* (R1); see the global response. We include this in the supplementary rebuttal pdf. Table 21 shows that AOC with a less accurate dynamics model outperforms Continuous Planning and Discrete Monitoring.\n\n## (D) Only empirically validates the key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations\nLet us clarify that we empirically validate in more environments than is standard for continuous-time control environments [Yildiz et al. 2021]; as we validate in all three of the continuous-time control environments of the ODE-RL suite from Yildiz et al. 2021, and an additional real-world Cancer environment. Furthermore, we also validate in another environment setup in **Appendix K.3**.\n \nWe highlight that we discuss the selection reasons for these environments in **Appendix H**. We selected the standard continuous-time control environments from the ODE-RL suite [Yildiz et al. 2021], which consists of three well-known environments: Pendulum, Cartpole, and Acrobot. Additionally, we implemented a Cancer environment that simulates a Pharmacokinetic-Pharmacodynamic (PK-PD) model of lung cancer tumor growth [Geng et al. 2017] under continuous dosing treatment of chemotherapy and radiotherapy.\n \nAs we need a continuous-time environment, many standard discrete-time environments are not applicable. Therefore, we empirically verify all the environments from the ODE-RL suite and an additional cancer environment.\n\n## Additional References\n* Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n* Ernst, D., Geurts, P., and Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503–556.\n* Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\" arXiv preprint arXiv:2005.01643 (2020)."}, {"review_id": "jc52oJPOps", "reviewer": "Reviewer_KW9G", "summary": "The paper addresses the problem of continuous control with costly observations. The authors provide a formal definition of the observation problem and the control problem. The continue by proposing a scheme to how to do the two simultaneously in irregular intervals. Finally, the method is evaluated against benchmark methods on 4 problems and the results are analyzed. The paper is well structured and clear.", "questions": "1.\tWhat are the limitation on the dynamics of the system? Can any f(t) be used and the method would work?\n2.\tThe model under non-uniform samples is reduce to semi-MDP, is there something from that line of research that can solve the problem?\n", "limitations": "Not applicable", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper deals with an important problem and provide a formal clean definition for it, and also proposes a solution. The experiments not only show that the method has merit but also help to discuss and analyze the solution and the importance of the problem.", "weaknesses": "The paper is not clearly stationed within previous work on similar topics. The area of event-based sampling in the control community has delt with close problem and analyzed how to control them and even analyzed the stability of such problem under irregular sampling. \n\nAlso, the method in the paper assumes access to the full state (noisy state), which can be unrealistic in real systems (for the purpose of developing theory it is ok but should be discussed). The method assumes the system can be identified and then a controller controls the system for the given interval based on the identified dynamics. This sounds very close to classical control principle such as certainty equivalence and and the separation principle. Those are applied to nonlinear systems as well (while some of the proofs are for linear systems), and since there is no proof of stability or optimality in this paper should be discussed in light of that.\n\nIt is unclear what are the limitation of the method in regard of system dynamics, should the system be initially stable? should it be of specific structure?\nWhile the experiments are compared to simple control methods, there are other approaches out there, that addresses similar problem and can be adjusted to this setting. As paper does not compare to other methods (other than naïve ones) it is hard to assess the true value of this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions!\n## (A) Add similar related work of event-based sampling\nWe agree that it is helpful for the reader to expand the related work section to include a discussion of the similar topic of event-based sampling. We now extend our related work section to include the following. \n\nThere exists a similar related work area, that of *event-based sampling* in the control community [Åström et al. 1999, Åström et al. 2008], which addresses the similar problem of controlling a system, of only taking a full state observation when an external input *event* occurs, and then providing a control action. To create this *event*, it **assumes part of the state is continuously observed**, and often an *event* is defined when this part of the state (or a function of it) crosses a fixed threshold (e.g., a physical sensor crossing a threshold) [Vasyutynskyy et al. 2010]. This finds multiple uses in problems such as electronic analog control systems for audio and mobile telephone systems [Åström et al. 1999], and battery capacity optimization in wireless devices [Vasyutynskyy et al. 2010]. However, this is different from our proposed problem of continuous-time control whilst deciding when to take costly observations as:\n1. Event-based sampling *assumes part of the state is continuously observed*. Often in our environments, it is not feasible to observe part of the state continuously (e.g., imaging a cancer tumor volume or performing a medical test) or it is prohibitively expensive to continually observe at a high-frequency part of the state (similar to Continuous Planning approaches).\n2. The *event* input (the time to take the next observation) is given as a control input to the agent. However, it is precisely the more difficult problem we tackle of coming up with an observing policy that decides *when* to take the next observation.\n3. The *event* is often defined by a human in advance and is a function of the current partial state. General environments may not have partial state spaces that are predictive of when to observe next, such as a robotic navigation task in two dimensions and only continuously observing one dimension.\nThis is now included in Section 3.\n## (B) Assumption of full noisy state access\nWe acknowledge your concern about the assumption of full state access, which may not always be feasible in real-world systems, however is often taken as a standard assumption in other works [Yildiz et al. 2021]. Indeed, we made this assumption primarily to facilitate the development of the theory. We agree, that our proposed problem and formalism could be extended to partial state observations, however, we leave this for exciting future work and development of such methods. We have now included this discussion and assumption in the limitations paragraph in the Conclusion and Future work, section 6.\n## (C) Proof of Stability or Optimality\nWe agree; however, optimality or stability proofs are *not* typically provided for deep RL methods. Allow us to clarify that this current paper contributes the first step to formalizing and understanding theoretically a key property an optimal method should have, which is verified empirically with an initial method. \n\nFurthermore, all model-based RL methods, and RL methods in general, are not optimal in *all scenarios*. A few methods can theoretically find an optimal policy *only in restrictive scenarios*, for example, Q-learning with an MDP, finite and discrete state and action spaces, stationarity, infinite exploration, and a fully observable environment.\n\nWe have now added this additional promising future work of a proof of stability and clearly state this as a limitation in the limitation paragraph in the Conclusion and Future work section (section 6).\n## (D) Limitations of the dynamics of the system\nOur formulation does not impose any specific structure on the dynamics of the system $f$ or require the system to be initially stable. We allow non-linear dynamics and unstable systems. However, we do assume stationarity in the dynamics, i.e., the dynamics are not changing within a single collected offline dataset. Allow us to reiterate that $f$ is defined as a differential equation, which is continuous in time, meaning that existing discrete-time and discrete system dynamics methods are not applicable. We agree this is beneficial to clarify, and we have now added this discussion to the Problem section (section 2).\n## (E) Related work clarification of Semi-MDP literature\nWe agree that semi-MDP literature is a similar related field, where it extends the MDP problem to include *options* that define temporally abstracted action sequences [Sutton et al. 1998]. However, some distinct differences prevent using a semi-MDP to address our continuous-time control whilst deciding when to take costly observations problem. These differences include:\n1. Semi-MDP is still an underlying discrete MDP with discrete state transitions, whereas our problem formulation focuses on continuous-time systems that can handle continuous actions and states.\n2. Semi-MDP formulation does not involve an observation cost, whereas our problem formulation does.\nWe have now expanded the related work section (section 3) to include the above discussion.\n## Additional References \n* Åström, Karl Johan, and Bo Bernhardsson. \"Comparison of periodic and event based sampling for first-order stochastic systems.\" IFAC Proceedings Volumes 32.2 (1999): 5006-5011.\n* Aström, Karl J. \"Event based control.\" Analysis and design of nonlinear control systems: In honor of Alberto Isidori. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008. 127-147.\n* Vasyutynskyy, Volodymyr, and Klaus Kabitzsch. \"Event-based control: Overview and generic model.\" 2010 IEEE International Workshop on Factory Communication Systems Proceedings. IEEE, 2010.\n* Sutton, Richard S. \"Between MDPs and Semi-MDPs: Learning, planning, and representing knowledge at multiple temporal scales.\" (1998).\n"}, {"review_id": "GPhs6iJWLS", "reviewer": "Reviewer_mUFv", "summary": "This paper tackles the issue of determining the optimal timing for observations in continuous-time control with costly observations. \nThe authors formulate the problem and develop theoretical findings to provide insights about why observing at irregular intervals is advantageous in certain environments.\nThey then introduce a novel method called Active Observing Control, which involves taking observations at irregular intervals and performing continuous-time control. The adaptive time interval is achieved using a heuristic that compares the reward variance of the rollout with a threshold, leveraging a model learned from an offline dataset.\n\nThe effectiveness of this method is validated through experimental results, demonstrating that it outperforms alternative approaches that either utilize a discrete-time model for planning or employ a continuous-time model with regular observation intervals. Ablation studies illustrate how the proposed method can mitigate the discretization error associated with discrete-time formulation and show that the performance is not sensitive to the choice of the threshold.\n", "questions": "- See weakness section\n- How to choose the parameter \\delta_a and \\delta_t? Grid search?\n- In adaptive quadrature, the time interval for estimating the integral of a function is adjusted based on a comparison of a threshold with the relative estimation error of two different time intervals. Although different to the proposed method, it appears to bear some similarities. Have the authors explored the possibility of applying algorithms from adaptive quadrature to determine the time interval in their proposed method? \n", "limitations": "It is discussed in the abstract and conclusion. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed approach is novel.\n- It provides insights from both theoretical and experimental results.\n- It includes comprehensive implementation details and ablation results.\n", "weaknesses": "1. The proposed method appears to be heavily reliant on the accuracy of the learned model. This could limit the generalizability and robustness of the approach.\nSpecifically, the time interval is directly impacted by the uncertainty of the reward. Knowing the true reward function simplifies the problem to estimating uncertainty of the dynamics.\nIt is not clear whether the approach would work well in the general case, where the reward function is not known and need to be learned. \n\n2. The following related works are not included:\n\n   [1] Huang, Y. and Zhu, Q., 2020. Infinite-horizon linear-quadratic-gaussian control with costly measurements. arXiv preprint arXiv:2012.14925.\n\n   [2]  Zhang, Z., Kirschner, J., Zhang, J., Zanini, F., Ayoub, A., Dehghan, M. and Schuurmans, D., 2022. Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-off. arXiv preprint arXiv:2212.08949.\n\n   In particular, [1] adds the cost of observations to the original cost function based on the state and action, similar to this work, but in a discrete-time LQG setting.\n   [2] studies the continuous-time stochastic LQR setting, where they formulate the costly measurement differently through the data budget.\n\n3. The paper claims that the formulation is novel. However, it appears that Reference [1] above proposes a similar objective function, albeit within the Linear Quadratic Gaussian (LQG) setting. It would be beneficial if the authors could provide clarifications on the differences between their proposed formulation and that found in Reference [1]. \n\n4. I may be missing something but the method might not be truly continuos-time. The time resolution seems to be lower-bounded by the \\delta_t. \n\n5. While an ablation study regarding the cost parameter 'c' in the Cancer environment is included in Appendix K.8, the process of selecting an appropriate 'c' in a real-world context remains unclear. Practical guidance or heuristic for choosing 'c' would greatly increase the method's applicability and ease of use.\n\n6. The uncertainty threshold \\tau is determined from a single episode and subsequently fixed.\nIt seems that further adjusting \\tau as we collect more episodes could potentially enhance the method’s performance. Can the authors comment on this?\n\n7. The submission of code via an external link is not ideal, particularly because the linked material appears to have been last updated on June 11, which is after the supplementary materials deadline. \n\n8. Typos:\n- Line 60: “our initial method capable of irregularly observing”, missing “is”\t\t\t\t\n- Appendix 939: “Random that samples the state” should this be “action” instead?\n- There seems to be an inconsistency in the numbers:\nLine 762: G=1k but G=10k in the main paper\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions!\n## (A) Dependence on the accuracy of the learned model\nWe agree; allow us to clarify that it is standard in all offline model-based RL methods to learn an accurate dynamics model [Levine et al. 2020, Lutter et al., 2021].\n\nWe verified empirically that AOC can perform well even when the dynamics model is *less accurate* by performing an *additional experiment ablation* (R1); see the global response. We include this in the supplementary rebuttal pdf. Table 21 shows that AOC with a less accurate dynamics model outperforms Continuous Planning and Discrete Monitoring.\n## (B) Extending to work with a learned reward model\nYou are correct; it is useful to also work in the more general case of learning a reward model. We performed an additional experiment, where we learned a separate reward model from the reward values from the offline dataset—**(R2)**; see the global response. We include this in the supplementary rebuttal pdf. Table 22 shows that AOC with a learned reward model still outperforms Continuous Planning and Discrete Monitoring.\n## (C) Include related work of Huang et al. 2020, Zhang et al. 2022\nWe agree they are relevant and now include them in the paper. In the following (D) we discuss Huang et al. 2020. Zhang et al. 2022 is a discrete planning method that works for *linear* (LQR) systems and optimizes the observing frequency for a given total budget of observations.\n## (D) Formulation clarification between related work of Huang et al. 2020\nWe find the formulation differences between ours and the neat work of Huang et al. 2020 that address an infinite horizon discrete time Linear-Quadratic-Gaussian control problem with observation costs be:\n* Huang et al. 2020 only applies to *linear systems* that are *discrete in time* and have an *infinite time horizon*; our formulation applies to *non-linear systems* that are *continuous in time* and have a *fixed time horizon*.\n* Huang et al. 2020 assume their full system dynamics model is *known*; ours makes no such assumptions and only assumes access to an offline collected dataset of (possibly irregular in time) state-action trajectories to learn a dynamics model, which is more applicable to real-world environments.\n\nWe now include a version of this discussion in the related work Section 3.\n## (E) Clarification of the $\\delta_t$ parameter for a continuous time method\nWe agree \\delta_t can be better explained—it is the continuous search (root finding algorithm) tolerance that is used when searching with binary search for the continuous time duration that the computed action trajectory can be followed for, which occurs when the standard deviation of the reward crosses the threshold $\\tau$. We note that all numerical root-finding algorithms generally involve a search tolerance or a similar stopping criterion. The $\\delta_t$ tolerance ensures that the binary search algorithm stops evaluations once a solution time is found that is \"close enough\" to the actual true value. Another way to view the search tolerance $\\delta_t$, is that the numerical precision that the search value is correct up to. We included a form of this discussion when introducing $\\delta_t$. \n## (F) Practical guidance for selecting $c$ in a real-world context\nWe agree such guidance increases greatly the applicability of the method. The observation cost $c$ should be decided based on the real-world application at hand and include any human preferences in that application if applicable. In real-world systems (with resource constraints), this cost might correspond to actual monetary cost, computational expense, or energy consumption. Further, if this cost involves a human, for example, a patient receiving medical treatment, they may have preferences for treatment, their health impact (e.g., chemotherapy side effects), and or the number, frequency, and timing of treatments that could be included in the cost $c$. Also, there can often be a trade-off between control performance and the number of observations taken, and one way to trade this off is to tune $c$. We have expanded this discussion to create an additional detailed **new Appendix L**, labeled “Practical guidelines to select $c$”.\n## (G) Updating $\\tau$ when more data is collected\nWe agree. Currently, we assume an initial offline dataset of state-action trajectories for training the dynamics model. However, we can easily extend this to when more state-action trajectories are collected; we retrain the dynamics model and determine $\\tau$ again. This would be most helpful when the initial offline dataset is small and has limited state-action space coverage. We now include this discussion in section 4.\n## (H) Code submission\nWe kindly highlight code submission is *not* compulsory and that the NeurIps policy for code provision is after acceptance, not before. We only updated the readme.\n## (I) Typos\nThank you, we agree with all the typos. Yes, line 762 should read $G=10,000$.\n## (K) How to choose $\\delta_a$ and $\\delta_t$\nThis depends on the real-world environment; $\\delta_a$ is often determined for us by the offline dataset of irregular time state-action samples, where it is the mean time between the samples in the dataset. However, if it had to be chosen, it would be a trade-off between the min and max range, where each of the max and min (see Appendix K.4, for decreasing $\\delta_a$) lead to poor performance. For the search tolerance $\\delta_t$, a sufficiently small value achieves good control performance. Bayesian optimization or grid search would be suitable.\n## (L) Applying adaptive quadrature algorithms\nWe agree; however, our initial method focuses on a simple initial method to verify the theoretical contribution empirically.\nIt is precisely more purpose-designed methods we hope to inspire for future work such as this.\n## Additional References\n* Levine, Sergey, et al. \"Offline reinforcement learning\" arXiv preprint arXiv:2005.01643 (2020)."}]

Author rebuttal:
We thank all reviewers for their thoughtful comments and suggestions! We respond to each reviewer with an individual rebuttal and share all **additional new experimental** results here.

## (R1) Dependence on the accuracy of the learned dynamics model
To understand this further, we performed an *additional experiment ablation*, where we benchmarked all methods with a less accurate dynamics model—training all the dynamics models with fewer samples (10\% of the total amount of samples used in training the dynamics models presented in the main paper), here trained on 100,000 samples. 

We provide the experimental results in the rebuttal supplemental pdf as Table 21.
We observe that AOC still achieves a high average utility $\mathcal{U}$ on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods.
Specifically, this empirically further verifies our key theoretical contribution that regular observing is not optimal and that irregularly observing can achieve a higher expected utility. We now include this discussion and new experiment in an **additional new Appendix L**.

## (R2) Extending AOC to work with a learned reward model
To investigate whether AOC can also be used with a learned reward model, we performed an *additional experiment*, where we trained an MLP reward model (4-layer MLP with 128 units and Tanh activations) from the offline dataset in all the benchmarks. 

We provide the experimental results in the rebuttal supplemental pdf as Table 22.
We observe that AOC still achieves a high average utility $\mathcal{U}$ on the Cancer environment, outperforming the competing Continuous Planning and Discrete Monitoring methods.
This empirically verifies that our proposed initial approach can still perform well using a learned reward model and that our theoretical contribution still holds. We now include this additional experiment in an **additional new Appendix M**.

Meta-review: This paper formalizes the problem of continuous-time control with observation costs (for non-linear, continuous-time systems with fixed time horizon and unknown dynamics), proves theoretically that observing at regular time intervals is not always optimal, constructs an initial method (using MPC for selecting actions and a binary search-based method to determine when observations should be taken), and validates this method on four continuous-time control environments (three from ODE-RL and a cancer environment). The reviewers agreed that the statement of this problem is interesting and novel, with solid theoretical and experimental insight. While they raised several issues in their initial reviews -- including dependence on the accuracy of the learned model, uncertainty as to whether the proposed control algorithm is actually discrete-time, questions about the relationship to prior work proposing a similar formulation, and questions about/corrections to several mathematical details -- the authors were able to address these concerns to the reviewers' satisfaction through additional experiments and adding additional discussion/clarification to the paper.
Decision: Accept (poster)