SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Samuel Holt
University of Cambridge sih31@cam.ac.uk
Alihan Hüyük
University of Cambridge ah2075@cam.ac.uk
Mihaela van der Schaar
University of Cambridge mv472@cam.ac.uk
The control of continuous-time environments while actively deciding when to take costly observations in time is a crucial yet unexplored problem, particularly relevant to real-world scenarios such as medicine, low-power systems, and resource management. Existing approaches either rely on continuous-time control methods that take regular, expensive observations in time or discrete-time control with costly observation methods, which are inapplicable to continuous-time settings due to the compounding discretization errors introduced by time discretization. In this work, we are the first to formalize the continuous-time control problem with costly observations. Our key theoretical contribution shows that observing at regular time intervals is not optimal in certain environments, while irregular observation policies yield higher expected utility. This perspective paves the way for the development of novel methods that can take irregular observations in continuous-time control with costly observations. We empirically validate our theoretical findings in various continuous-time environments, including a cancer simulation, by constructing a simple initial method to solve this new problem, with a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner. Although determining the optimal method remains an open problem, our work offers valuable insights and understanding of this unique problem, laying the foundation for future research in this area.
The problem of continuous control with costly observations is ubiquitous with applications in medicine, biological systems, low power systems, robotics, resource management and surveillance [Yoshioka and Tsujimura, 2020, Brunereau et al., 2012, Mastronarde and van der Schaar, 2012]. A setting that is shared across all these domains is that a decision-maker needs to continually control (e.g., chemotherapy dosing, quantities of food, data to transmit, etc.) whilst deciding when to take a costly observation (e.g., a medical computed tomography (CT) scan, measuring the population of a species, measuring bandwidth, etc.). The decision-maker's observing policy must be timely to determine whether the action plan is effective (e.g., errors in treating stage 4 cancer can be fatal [Reinhardt et al., 2019], with further application examples in Appendix B).
In many of these real-world systems (e.g., medicine, low-power systems, resource management), an offline setup is beneficial as it enables decision-makers to learn control policies without incurring excessive costs or risking adverse effects. Where an offline setup refers to learning a policy from a previously collected dataset of state-action trajectories, without interacting with the environment [Argenson and Dulac-Arnold, 2020]. The development of novel methods that can take irregular observations in continuous-time control with costly observations, within an offline setup, provides a safe and cost-effective way to improve real-world decision-making processes.
Recent work falls into three main categories. First, sensing approaches determine when to informatively observe to identify an underlying state, but are unable to continually control. Second, planning approaches only continually control and have the restrictive assumption that the observing schedule is
given—most often at a regularly fixed observation time interval. While these methods can optimally solve many challenging tasks, with regular, frequent observations of the state, frequent observations are overly costly with many being unnecessary, and where the optimal regular observation interval is unknown.
Third, discrete monitoring approaches control and determine when to observe only for discrete time. These are fundamentally incompatible with continuous-time systems which can exhibit irregular observations in time and states that evolve on different time scales. Moreover, these rely on a discretization interval being carefully selected to use but still suffer from time discretization errors that can compound, leading to unstable and poor-performing controllers [Yildiz et al., 2021]. For example, in medicine, a crucial test may be administered too late, and therefore delay updating the treatment dosing—which can be fatal when treating patients for cancer [Geng et al., 2017].
Contributions: ① We are the first to formalize the problem of continuous-time control whilst deciding when to take costly observations. Theoretically, we show that regular observing in continuous time with costly observations is not optimal for some systems and that irregularly observing can achieve a higher expected utility (Section 2). ② Empirically we verify this key theoretical result in a cancer simulation and standard continuous-time control environments with costly observations. We construct a simple initial method to solve this new problem, called Active Observing Control. This uses a heuristic threshold on the variance of reward rollouts in an offline continuous-time model-based model predictive control (MPC) planner (Sections 4 and 5.1). However, the optimal method is still an open problem, and we leave this for future work. We gain insight into this unique problem, uncovering how our initial method is capable of irregularly observing, and identifies observation frequencies that correlate to different stages of cancer fixed observing frequencies of clinicians. Further, we demonstrate how it can avoid discretization errors in time, thus achieving a better utility as a result (Section 5.2), and confirm its robustness to its threshold hyperparameter (Section 5.3).
We have released a PyTorch [Paszke et al., 2019b] implementation of the code at https://github.com/samholt/ActiveObservingInContinuous-timeControl and have a broader research group codebase at https://github.com/vanderschaarlab/ActiveObservingInContinuous-timeControl.
2 Problem
States & actions. For a system with state space  $\mathcal{S} = \mathbb{R}^{d_S}$  and action space  $\mathcal{A} = \mathbb{R}^{d_A}$ , the state at time  $t\in \mathbb{R}_+$  is denoted as  $s(t) = [s_1(t),\dots ,s_{d_S}(t)]\in S$  and the action at time  $t\in \mathbb{R}$  is denoted as  $a(t) = [a_1(t),\ldots ,a_{d_A}(t)]\in \mathcal{A}$ . We elaborate that a state trajectory  $s\in S^{\mathbb{R}}$  and an action trajectory  $a\in \mathcal{A}^{\mathbb{R}}$  are both functions of time, where an individual state  $s(t)\in S$  or an individual action  $a(t)\in \mathcal{A}$  are points on these trajectories. In practical applications, action values are usually bounded by an actuator's limits; hence we also restrict the action space to a Euclidean box  $\mathcal{A} = [a_{\mathrm{min}},a_{\mathrm{max}}]^{d_A}$ .
Environment dynamics. The dynamics of the system are given by  $\mathrm{ds}(t) / \mathrm{dt} = f(s(t),a(t))$  and the system is stationary. We consider the setting where the true state trajectory  $s$  is latent and, at a given time  $t$ , only a noisy observation  $z(t) = s(t) + \varepsilon (t)\in S$  can be observed, where  $\varepsilon (t)\sim \mathcal{N}(0,\sigma_{\epsilon}^{2})$  is Gaussian noise with zero mean and standard deviation  $\sigma_{\epsilon}$  and  $\varepsilon (t)\perp \varepsilon (t^{\prime}),\forall t,t^{\prime}\in \mathbb{R}_{+}$ . We denote as a tuple  $(t,z(t),a(t))$  a discrete sample taken at time  $t$ , and with  $h = \{(t_j,z(t_j),a(t_j))\} \in \mathcal{H}$  a history of samples taken at times  $\{t_j\}$  where  $\mathcal{H} = \cup_{n = 0}^{\infty}(\mathbb{R}_+ \times S\times \mathcal{A})^n$  is the space of all possible histories.
Algorithm 1 Policies  $\rho, \pi$  interacting with the environment
1:  $t_1 = 0, h_1 = \{(t_1, z(t_1), a(t_1))\}$
2: for  $i \in \{1, 2, \ldots\}$ :
3: Schedule next observation:  $t_{i+1} = t_i + \rho(h_i)$
4: Execute actions:  $a(t) = \pi(h_i, t - t_i)$  for  $t \in [t_i, t_{i+1})$
5: Take an observation:  $h_{i+1} = h_i \cup \{(t_{i+1}, z(t_{i+1}), a(t_{i+1}))\}$
Policies. Policies consist of an observing policy  $\rho : \mathcal{H} \to \mathbb{R}$  and a decision policy  $\pi : \mathcal{H} \times \mathbb{R} \to \mathcal{A}$ . These policies interact with the environment as described in Algorithm 1. We denote with  $\Delta_{j+1} = t_{j+1} - t_j$ , where  $j$  is a dummy variable.
Objective. Suppose we are given a reward function  $r: \mathcal{S} \to \mathbb{R}$  to maximize, and each observation has a fixed cost  $c \in \mathbb{R}_+$ . Then, the utility achieved by a policy up to a time horizon  $T$  is given by
$$
\mathcal {U} = \underbrace {\int_ {0} ^ {T} r (s (t) , a (t) , t) \mathrm {d} t} _ {\text {R e w a r d} \mathcal {R}} - \underbrace {c \left| \left\{t _ {i} : t _ {i} \in [ 0 , T ] \right\} \right|} _ {\text {C o s t} \mathcal {C}} \tag {1}
$$
Our objective is to find the optimal observing and decision policies  $\rho^{*},\pi^{*} = \arg \max_{\rho ,\pi}\mathbb{E}[\mathcal{U}]$  given an offline dataset of histories  $\mathcal{D} = \{h^{(i)}\}$  but without knowing  $f$  and  $\sigma_{\epsilon}$  or having online access to the system described by  $f$  and  $\sigma_{\epsilon}$ .
2.1 Taking regular observations in time is not optimal for some systems
This is a highly non-trivial task since, in many cases, the obvious solution of taking regular observations in time is not optimal, and rather taking irregular observations achieves a higher expected utility—a point we make formally in Proposition 2.1.
Proposition 2.1. For some systems, it is not optimal to observe regularly—that is  $\exists f, \sigma_{\epsilon}, r, c, h, h': \rho^{*}(h) \neq \rho^{*}(h')$ .
Proof. Full proof is in Appendix C. However, we present the following sketch. Consider the task of maximizing the utility  $\mathcal{U}$  given a reward function  $r(s,a,t) = \delta (t - T)\cdot \mathbb{1}\{s(t)\cdot a(t) > 0\}$ , observation cost  $c > 0$  for the specific system  $ds / dt = s(t)$  with observation noise  $(\sigma_{\epsilon} > 0)$ . Intuitively this reward function only provides a positive reward of  $+1$  at the end of the episode ( $t = T$ ) if the chosen action  $a(T)$  has the same sign as the unobserved state  $s(T)$ , otherwise, the reward is 0. Therefore, this  $r$  is maximized when the sign of the correct state  $s(t)$  is identified. Since this system is defined by  $ds / dt = s(t)$ , we know that state solutions are of the form  $s(t) = s(0)e^{t}$ . Thus, to determine the sign of  $s(t)$ , since  $e^t$  is always strictly positive, the sign of the state is determined by the initial condition  $s(0)$ . Our observing policy can observe anywhere inside the interval of  $[0,T]$ . We prove that the optimal observing policy,  $\rho^{*}$  cannot be a regular observing policy by showing that, for each  $\rho^{(\mathrm{reg},\delta)}$ , there exists at least one  $\rho^{(\mathrm{irreg},\ell)}$  with  $\ell \geq 2$  that achieves higher expected utility, where  $\ell$  is the number of observations taken. A higher expected utility can be achieved by an irregular observing policy that takes all observations at the end of the episode, i.e.  $\rho^{*}(h_{i}) = T - t_{i}$  for  $i\in \{2,\dots ,\ell \}$  that is when the signal-to-noise ratio  $(s(t)^2 /\sigma_\epsilon^2)$  is the highest, i.e.  $\arg \max_t\frac{s(0)^2e^{2t}}{\sigma_\epsilon^2}:t\in [0,T]$ , which occurs when  $t = T$ .
This motivates us to develop new continuous-time methods for solving this problem, where a solution should be able to flexibly adapt to take irregular observations when it achieves a higher expected utility. Intuitively, based on the above system, it is more informative to take an observation when the predicted state change magnitude is larger, which can occur faster for a large state velocity  $\dot{s}$  — indicating observations should be taken more frequently. We later show experimentally illustrative cases (Section 5.2) of how irregularly observing can lead to better performance.
3 Related work
Table 1 summarizes the key differences between Active Observing Control from related approaches to planning and monitoring in reinforcement learning (RL). Moreover, we provide an extended related work in Appendix D, which includes discussions of the benefits of offline RL, model-based RL, why model predictive control is our preferred action policy, event-based sampling, Semi-MDP methods, and Linear Quadratic Regression (LQR) & Linear Quadratic Gaussian (LQG) methods.
Sensing approaches have been proposed of when to optimally take an observation in both discrete time [Jarrett and Van Der Schaar, 2020] and continuous time [Alaa and Van Der Schaar, 2016, Barfoot et al., 2014]—where their goal is to identify an underlying state. However, these approaches cannot also continuously control the system. In contrast, Active Observing Control seeks to both actively observe and control the system.
Planning approaches only focus on optimal decision policies  $\pi$ , and therefore observing has to be provided by a schedule, which is often assumed to be at a regular constant interval in time  $\Delta_{i+1} =$
Table 1: Comparison with related approaches to planning and monitoring in RL. Plots for the corresponding state trajectories for each approach are illustrated in Figure 1. Our initial method, Active Observing Control, is the only method for continuous-time control whilst deciding when to observe with smoothly-varying states.
Approach	Ref.	Domain	Environment Dynamics	State Trajectories	Policies	Policy Formulation
Discrete Sensing	[Jarrett and Van Der Schaar, 2020]	Discrete-time	si~sconstant	A	Observing-only	i+1=i+ρ(hi)
Discrete Planning	[Chua et al., 2018]	Discrete-time	si+1~f(si, ai)	B	Decision-only	ai=π(hi)
Discrete Monitoring	[Nam et al., 2021]	Discrete-time	si+1~f(si, ai)	C	Decision & observing	a′i∈{t1,...,pi(hi)}=πi′-i(hi)
Continuous Sensing	[Alaa and Van Der Schaar, 2016]	Continuous-time	st~sconstant	D	Observing-only	ti+1=t1+ρ(hi)
Continuous Planning	[Yildiz et al., 2021]	Continuous-time	dx(t)/dt~f(s(t), a(t))	E	Decision-only (w/regular observations)	a(τ∈{t1,t1+Δ})=π(hi,t-t1)
Semi-continuous Monitoring	[Huang et al., 2019]	Continuous-time	s(te{ts,t+1}~s(k)(sk+1,t+1)-f(sk,a(ℓ{ts,t+1}))	F	Decision & observing	a(τ∈{t1,t1+ρ(hi)})=π(hi,t-t1)
Active Observing Control	(Ours)	Continuous-time	dx(t)/dt~f(s(t), a(t))	G	Decision & observing	a(τ∈{t1,t1+ρ(hi)})=π(hi,t-t1)
Figure 1: Comparison of related approaches of an environment's latent state  $s(t)$  —where green lines represent observing times and red dots observations that have observation noise.
$\Delta_{i}$  that is, observations are not actively planned [Yildiz et al., 2021]. Within planning approaches, there exist many discrete-time approaches [Chua et al., 2018, Mnih et al., 2013, Williams et al., 2017] and recently more continuous-time approaches [Du et al., 2020, Yildiz et al., 2021]. Specifically, Yildiz et al. [2021] presented a seminal online continuous-time model-based RL algorithm, leveraging a continuous-time dynamics model that can predict the next state at an arbitrary time  $s(t)$ . However, all these methods are unable to plan when to take the next observation, in contrast to Active Observing Control which is able to determine when to take the next observation whilst planning an action trajectory.
Monitoring approaches consist of both a decision policy  $\pi$  and an observing policy  $\rho$ ; however, existing approaches only consider the discrete-time and discrete-state setting with observation costs [Sharma et al., 2017, Nam et al., 2021, Aguiar et al., 2020, Bellinger et al., 2020a, Krueger et al., 2020, Bellinger et al., 2020b]. In particular, Krueger et al. [2020] proposed that even in a simple setting of a discrete-state multi-armed bandit, computing the optimal time to take the next observation is intractable—therefore, they must rely on heuristics for their observing policy. Broadly, discrete-time monitoring methods use a discrete-time model and propagate the last observed state, often appending either a flag or a counter to the state-action tuple to indicate the time since an observation was taken [Aguiar et al., 2020, Bellinger et al., 2020b]. However, these methods cannot be applied to continuous-time environments or propagate the predicted current state and its associated prediction interval of the uncertainty associated with the state estimate. Moreover, training a policy [Aguiar et al., 2020] that decides at each state whether to observe it or not, requires a discretization of time. Whereas Active Observing Control, which determines the continuous-time interval of when to observe the next state, does not require any discretization of time, and hence it is a continuous-time method and avoids compounding time discretization errors.
One approach exists that we term semi-continuous monitoring, where Huang et al. [2019] proposes a discrete-state, constant action policy, that determines when to take an observation in continuous time of a controlled Markov Jump Process. However, this approach is limiting, as it assumes actions are constant until the next sample of the state is taken [Ni and Jang, 2022]—which is clearly suboptimal, uses discrete states, and assumes a Markov Jump Process [Gillespie, 1991]. Instead, Active Observing Control is fully continuous in both states and observing times—controlled by an action trajectory  $a$ , giving rise to smoothly-varying states.
4 Active Observing Control
We now propose Active Observing Control (AOC), an initial method for the defined problem of continuous-time control whilst deciding when to take costly observations. AOC, can plan action trajectories in continuous time and plan when to observe next in continuous time. The key idea is to use a continuous-time uncertainty-aware dynamics model  $\hat{f}_{\theta}$  to plan 1) action trajectories and 2) the next time to take an observation, such that it is informative to do so—Figure 2 provides a block diagram. We adhere to the standard offline model-based setup [Lutter et al., 2021] of first learning our dynamics model (Section 4.1), and then using it at run-time with our planning method (Section 4.2).
Figure 2: Block diagram of Active Observing Control. An uncertainty-aware dynamics model  $\hat{f}_{\theta}$  is learned from an offline dataset  $\mathcal{D}$  of state-action trajectories. At run-time, planning consists of two steps: 1) The actions are determined by a Model Predictive Control (MPC) Planner, and 2) the determined action trajectory  $a$  is forward simulated to provide uncertainty  $\sigma(r(t))$  on the planned path reward. We determine the continuous time  $t_{i+1}$  to execute the action plan  $a$  up to, such that  $\sigma(r(t)) < \tau$ , as in Algorithm 3. We then execute  $a(t) \forall t \in [t_i, t_{i+1})$  up until the time to take the next observation  $z(t)$  at.
4.1 Learning a continuous-time uncertainty-aware dynamics model
Fundamentally the goal of offline model-based RL is to learn an accurate dynamics model that can be used for planning from an offline dataset  $\mathcal{D}$  [Lutter et al., 2021]. In particular, we require a dynamics model that is both uncertainty aware and continuous in time; that is, it can provide a prediction uncertainty for the next state at a future time  $t + \delta$ , i.e.  $p(z(t + \delta)|(z(t),a(t),\delta))$ . Here we use the time difference  $\delta$  input to create a continuous-time dynamics model [Yildiz et al., 2021].
Model-based RL has shown the crucial importance of the performance of learning an uncertainty-aware dynamics model that captures both 1) aleatoric uncertainty—due to the inherent stochasticity of the environment (e.g., observation noise and environment process noise) that is irreducible and 2) epistemic uncertainty—due to the lack of data, for a given state-action space (which should vanish when trained on the limit of infinite data) [Chua et al., 2018].
We learn an uncertainty-aware dynamics model by training an ensemble of high-capacity multi-layer perceptron (MLP) neural network models that each parameterize a multivariate Gaussian distribution—where the ensembling captures the epistemic uncertainty and each model individually captures the aleatoric uncertainty. We note that as shown by other works, ensembles of high-capacity neural networks outperform Gaussian process dynamics models, as they have constant-time inference—scaling better to larger offline datasets, while still providing well-calibrated uncertainty estimates [Lakshminarayanan et al., 2017] and can model more complex functions—inclusively non-smooth dynamics [Chua et al., 2018].
Precisely, the ensemble consists of  $M = 5$  neural network models that output the parameters for a multivariate Gaussian distribution with diagonal covariances, each with  $\hat{\theta}_{m}$  parameters, i.e.,  $\hat{f}_{\theta_{m}} = p_{\theta_{m}}(z(t + \delta)|(z(t),a(t),\delta)) = \mathcal{N}(\mu_{\theta_{m}}(z(t),a(t),\delta),\Sigma_{\theta_{m}}(z(t),a(t),\delta))$  where the elements of the diagonal covariances are given by  $\sigma_{\theta_m}^2 (z(t),a(t),\delta)$ , with a total of  $\theta = \{\theta_{m}\}_{m = 1}^{M}$  parameters of the ensemble. Moreover, we denote all the models in the ensemble as  $\hat{f}_{\theta}$ .
To create our ensemble of parameterized Gaussian distribution models, we train each model independently with unique initialized random weights on a unique permutation of all the samples in the offline dataset—whereby each base model converges to its own local optima and is more effective than training the models on subsets of the dataset, i.e., bootstrapping [Lakshminarayanan et al., 2017]. Therefore, we minimize the negative log-likelihood for every single model separately, that is
$$
\begin{array}{l} \mathcal {L} (\theta_ {m}) = \sum_ {h \in \mathcal {D}} \sum_ {j = 1} ^ {| h | - 1} [ \mu_ {\theta_ {m}} (z (t _ {j}), a (t _ {j}), \delta_ {j + 1}) - z (t _ {j + 1}) ] ^ {\top} \Sigma_ {\theta_ {m}} ^ {- 1} (z (t _ {j}), a (t _ {j}), \delta_ {j + 1}) \\ \left[ \mu_ {\theta_ {m}} (z (t _ {j}), a (t _ {j}), \delta_ {j + 1}) - z (t _ {j + 1}) \right] + \log \det \Sigma_ {\theta_ {m}} (z (t _ {j}), a (t _ {j}), \delta_ {j + 1}), \\ \end{array}
$$
where  $\delta_{j + 1} = \Delta_{j + 1} = t_{j + 1} - t_j$  arises from the offline dataset  $\mathcal{D}$ . Here each parameterized Gaussian distribution model captures heteroskedastic (i.e., the output noise distribution is a function of the
input) aleatoric uncertainty. To capture the heteroskedastic epistemic uncertainty, we combine the individual models as a uniformly weighted mixture model and combine the predictions as  $p_{\theta}(z(t + \delta)|(z(t),a(t),\delta),\theta) = \frac{1}{M}\sum_{m = 1}^{M}p_{\theta_m}(z(t + \delta)|(z(t),a(t),\delta),\theta_m)$ . Therefore, we can compute the effective mean and covariance (diagonal elements) of the mixture as
$$
\mu_ {*} (z (t), a (t), \delta) = \mathbb {E} _ {m} [ \mu_ {\theta_ {m}} (z (t), a (t), \delta) ]
$$
$$
\sigma_ {*} ^ {2} (z (t), a (t), \delta) = \mathbb {E} _ {m} \big [ (\sigma_ {\theta_ {m}} ^ {2} (z (t), a (t), \delta) + \mu_ {\theta_ {m}} ^ {2} (z (t), a (t), \delta) ] - \mu_ {*} ^ {2} (z (t), a (t), \delta)
$$
4.2 Active Observing Control Planning
We desire to use the trained uncertainty-aware dynamics model  $\hat{f}_{\theta}$  to 1) plan the optimal action trajectory  $a$  to execute and 2) plan when best to take the next observation. Intuitively, we use the probabilistic dynamics model  $\hat{f}_{\theta}$  to plan an optimal action trajectory  $a$  and execute this until the next observing time—as determined by the heuristic of when the predicted reward distribution over the planned trajectory crosses a set threshold, Figure 2. In the following, we detail these two steps.