SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1) Planning optimal actions. To plan an optimal action trajectory  $a$ , we specify that any model predictive controller (MPC) planner can be used—that can optimize the action trajectory up to a specified time horizon  $H \in \mathbb{R}_+$ . We opt to use an MPC planner, as it optimizes an entire action trajectory  $a$  up to  $H$ , does not require the probabilistic dynamics model  $\hat{f}_{\theta}$  to be differentiable (it uses no gradients), and the reward function  $r$  can be changed on the fly—allowing changing goals/tasks at run-time. We use our probabilistic dynamics model  $\hat{f}_{\theta}$ , with the MPC planner of Model Predictive Path Integral Control (MPPI), a zeroth-order particle-based trajectory optimizer [Williams et al., 2017], due to its competitive performance [Wang et al., 2022].
To optimize the action trajectory  $a$  up to a time horizon  $H$ , it discretizes  $H$  into smaller constant action time intervals  $\delta_{a} \in \mathbb{R}_{+}$  which are then optimized—where there are  $K = \left\lceil \frac{H}{\delta_a} \right\rceil \in \mathbb{Z}_+$  time steps in  $H$ —i.e.,  $t^{(k)} = t_i + k\delta_a, k \in \{0, \dots, K-1\}$ . It forward simulates several parallel rollouts  $G \in \mathbb{Z}_{+}$  where the next state estimate at each time step is simulated as  $z(t^{(k+1)}) = \mu_*(z(t^{(k)}), a(t^{(k)}), \delta_a)$  recursively. This requires a state estimate  $z(t_i)$  to plan from. Therefore, we recompute the planned trajectory when we observe a new sample of the state. Furthermore, we provide MPC MPPI planner pseudocode and details in Appendix E.
2) When to observe. We desire to observe when it is most informative to do so. Therefore, we seek to determine the time interval  $\Delta_{i+1}$  that the action trajectory  $a$  can be followed until an observation is taken. Intuitively, we create a reward distribution over continuous time following the MPC planned action trajectory  $a$ , which we can follow until the uncertainty crosses a set threshold  $\tau \in \mathbb{R}_+$  —a hyperparameter to be tuned. Intuitively we prefer the reward uncertainty rather than that of the state because we can achieve the task with a very certain reward despite having uncertain states. For instance, when there might be multiple ways to achieve a task—and we know that our multiple action trajectories guarantee this where we can take any; however, are uncertain about which one. We empirically verify this in Appendix L.1.
To create the continuous-time reward distribution and state distribution, we use our learned probabilistic dynamics model  $\hat{f}_{\theta}$  to forward propagate the last observed state, according to the planned action trajectory  $a$ . As noted by others [Chua et al., 2018], computing a closed-form expression for the distribution of the expected state trajectory is generally intractable; therefore, it is common to approximate the uncertainty propagation [Chua et al., 2018, Girard et al., 2002, Candela et al., 2003]. We generate this through Monte Carlo sampling by forward simulating  $P \in \mathbb{Z}_+$  particle rollouts of state trajectories. That is, for a given state particle, for the next step of  $\delta_{a}$  we sample a new state particle  $z_{p}(t^{(k + 1)}) \sim \mathcal{N}(\mu_{*}(z_{p}(t^{(k)}), a(t^{(k)}), \delta_{a}), \sigma_{*}^{2}(z_{p}(t^{(k)}), a(t^{(k)}), \delta_{a}))$  recursively along the planned action trajectory. This allows us to apply the reward function to the state particles, generating reward particles—whereby we can compute the mean and standard deviation of the reward at each time step  $t^{(k)}$ , i.e.,  $r(t^{(k)}) \sim \mathcal{N}(\mu_r(t^{(k)}), \sigma_r^2(t^{(k)}))$ . Therefore, using our estimate of  $\sigma(r(t^{(k)}))$  over the state trajectory  $a$  allows us to determine the maximum time interval  $\Delta_{i + 1}$  until the reward uncertainty becomes greater than a set threshold  $\tau$ . Thus, the next time to observe at is
$$
\rho (z (t _ {i})) = \max \{\Delta^ {\prime} \in \mathbb {R} _ {+}: \sqrt {\mathbb {V} _ {z _ {p}} [ r (t _ {i} + \Delta^ {\prime}) ]} <   \tau \}
$$
This provides an estimate of the next observation interval  $\Delta_{i+1}$  that is discretized to an integer number of  $\delta_a$  intervals. However, we seek the continuous-time observation interval  $\Delta_{i+1}$ ; therefore,
we further refine this estimate through a continuous binary search (root finding algorithm) up to a tolerance of  $\delta_t \in \mathbb{R}_+$ . We outline the AOC planning algorithm pseudocode in Appendix F.
Run-time complexity. Intuitively, we find if the time taken to plan using an MPC controller is feasible in a problem setting, then AOC is feasible. As the run-time complexity of AOC is  $O(GK + P(K + W))$  where  $W = \lceil (\log(\delta_a) - \log(\delta_t)) / \log(2) \rceil \in \mathbb{Z}_+$  (Appendix G). Empirically we chose  $P = 10G$ ,  $W = 4$ ,  $G = 10,000$  for all experiments. Although AOC takes approximately  $2.4 \times$  longer to plan, which includes both planning the actions and when to observe compared to Continuous Planning methods that only plan actions (an MPC step)—it can often take fewer observations overall, leading to less time spent planning overall. AOC is still a practical method that can plan in less time than the action interval  $\delta_a$ , even in robotics environments, where it takes 24.9 ms to plan the actions and the next observation time (Appendix G).
An important parameter is the uncertainty threshold  $\tau$  hyperparameter. We find the following procedure to tune this parameter for a benchmark sufficient. For a particular environment after training its dynamics model, one episode is run, where the action trajectory  $a$  is re-planned at every  $\delta_{a}$  time step and compute the median of the reward uncertainty over time for each rollout action plan, and then takes the mean of these over an episode to produce  $\tau$ . This step is akin to a calibration step, where calibration is common in medicine [Preston et al., 1988] and robotics [Nubiola and Bonev, 2013]. We also include practical guidelines to select  $c$  in Appendix H.
5 Experiments and Evaluation
Benchmark environments. We use four continuous-time control environments and adapt them to add a fixed cost  $c$  for taking an observation—these environments were selected as they exhibit a range of dynamical system state region regimes. First, the Cancer environment uses a simulation of a Pharmacokinetic-Pharmacodynamic (PK-PD) model of lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. We note that the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. Moreover, we use the standard continuous-time control environments from the ODE-RL suite [Yildiz et al., 2021], which consists of three environments: Pendulum, CartPole, and Acrobot. We note that all the environments are described by a differential equation (DE) and use a DE solver to simulate the fully continuous in-time states and actions, unlike discrete environments [Yildiz et al., 2021, Brockman et al., 2016]. Furthermore, to increase realism, we add Gaussian noise to observations taken, that of  $\mathcal{N}(0,\sigma_{\epsilon}^{2})$  with standard deviation  $\sigma = 0.01$ . Specifically, the ODE-RL starting states have the poles hanging down at the stable equilibrium of the DE system, and the goal is to swing up and stabilize the pole(s) upright to the unstable equilibrium [Yildiz et al., 2021]. Additionally, the reward function form is the same across the environments—i.e., the exponential of the negative distance from the current state to the desired goal state  $s^*$ , while penalizing the action magnitudes—and we assume we are given this simple reward function when planning (however, we also show AOC can work with a learned reward model in Appendix L.9). Furthermore, we generate an offline dataset  $\mathcal{D}$  from these environments that has irregular times between state-action samples  $\Delta_{i+1} \sim \mathrm{Exp}(\bar{\Delta})$  with a mean of  $\bar{\Delta} = \delta_a$  seconds. We detail all environments, including offline dataset generation in Appendix I.
Benchmark methods. We seek to provide more competitive benchmarks than those outlined in Table 1; therefore, we compare against the following ablations of our method, Active Observing Control. Specifically, we focus on benchmarking observing policies  $\rho$  and use the same action policy  $\pi$  across all the benchmark meth
ods, that of the same MPC MPPI planner with the hyperparameters fixed. We implement two discrete-time methods by first learning a discrete-time uncertainty-aware dynamics model (an ablation of the same model and hyperparameters, without the time interval input  $\delta$  to predict the next state for) on the same offline dataset  $\mathcal{D}$ . Second, we use this discrete-time uncertainty-aware dynamics model to create two benchmark methods, that of Discrete Planning that observes at each action time interval  $\delta_{a}$  and Discrete Monitoring a discrete ablation of our observing policy (Algorithm 3)—that uses the same reward distribution estimation and determines the discrete action sequence to execute up until the reward uncertainty crosses the threshold  $\tau$ —at a discrete-time resolution of  $\delta_{a}$ . Moreover,
Table 2: Benchmark observing policies
Benchmark	Observing Policy
Discrete Planning	ρ(z(t_i)) = δa
Discrete Monitoring	ρ(z(t_i)) = δa max{k: σ(t_i + kδa) < τ}
Continuous Planning	ρ(z(t_i)) = δa
Active Observing Control	ρ(z(t_i)) = max{Δ: σ(t_i + Δ) < τ}
Table 3: Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$  and observations  $\mathcal{O}$  for the benchmark methods, across each environment. AOC performs the best across all environments. Results are averaged over 1,000 random seeds, with  $\pm$  indicating  $95\%$  confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every  $\delta_{a}$ .
Policy	Cancer	Acrobot	Cartpole	Pendulum
U	R	O	U	R	O	U	R	O	U	R	O
Random	0±0	0±0	13±0	0±0	0±0	50±0	0±0	0±0	50±0	0±0	0±0	50±0
Discrete Planning	91.7±0.368	91.7±0.368	13±0	87.1±1.05	87.1±1.05	50±0	83.6±0.56	83.6±0.56	50±0	87.2±0.962	87.2±0.962	50±0
Discrete Monitoring	91±0.532	85.8±0.522	5.08±0.0327	89.6±1.02	80.2±1.14	43.7±0.189	127±0.846	82.9±0.532	42.3±0.107	130±2.52	87.3±0.957	42.1±0.293
Continuous Planning	100±0.153	100±0.153	13±0	100±0.462	100±0.462	50±0	100±0.772	100±0.772	50±0	100±0.904	100±0.904	50±0
Active Observing Control	105±0.18	98.8±0.169	3.37±0.0302	107±0.911	90.8±0.878	39±0.177	151±1.54	99.5±0.774	41.1±0.196	177±2.18	98.8±0.912	35.6±0.239
we also benchmark against Continuous Planning that uses our trained continuous-time uncertainty-aware dynamics model and takes observations at regular time intervals of a multiple of  $\delta_{a}$ . Finally, we also compare with a random action policy, Random that observes at each action interval  $\delta_{a}$ . For ease of comparison, we list these observing policies in Table 2. We further provide the dynamics model implementation details, hyperparameters, and training details in Appendix J.
Evaluation. An uncertainty-aware dynamics model is trained for each environment on an offline dataset  $\mathcal{D}$  of collected trajectories consisting of  $10^{6}$  samples. The trained dynamics model weights are then frozen when used at run-time. We record the average undiscounted utility  $\mathcal{U}$  (Equation (1)), average undiscounted reward  $\mathcal{R}$  and the number of observations taken  $\mathcal{O}$ , after running one episode of the benchmark method—and repeat this for 100 random seed runs for each result with their  $95\%$  confidence intervals throughout. Moreover, we normalize  $\mathcal{R}$  and  $\mathcal{U}$  following the standard normalization of offline-RL [Yu et al., 2020]—normalized to the interval of 0 to 100, where a score of 0 corresponds to a random policy performance, and 100 to an expert—which we assume here is the Continuous Planning benchmark. Furthermore, we detail these metrics and the experimental setup further in Appendix K.
5.1 Main results
We evaluated all the benchmark methods across all our environments with results tabulated in Table 3. Active Observing Control achieves high average utility  $\mathcal{U}$  on all environments. Specifically, AOC can achieve near-optimal state control performance  $\mathcal{R}$  while using fewer observations  $\mathcal{O}$  compared to taking regular, frequent observations of Continuous Planning. Furthermore, it outperforms Discrete
Figure 3: Comparison of Active Observing Control against Continuous Planning on the cancer environment for one episode. Rows: A) Cancer volume  $v$  state trajectory—with green vertical lines representing observation times. B) Reward uncertainty of the planned state trajectory  $a$  after an observation is taken—where the red horizontal line indicates AOC's threshold used  $\tau$ . C) Bar chart of the frequency of observations per state region. AOC automatically determines to observe larger cancer volumes more frequently as they are more informative, as the future state change magnitude is larger (Section 2.1)—which correlates to clinician's findings [Geng et al., 2017]. Whereas with Continuous Planning the observing frequency is regular, and therefore observations can be taken when the reward is still certain, suggesting an unnecessary observation.
Figure 4: Frequency of observations per state region for Pendulum. The intuition from Section 2.1 indicates that it is more informative to sample when  $\dot{\phi}$  is larger, hence near the top goal equilibrium point  $\dot{\phi} \approx 0$  necessitates infrequent observations.
Table 4: Normalized utilities  $\mathcal{U}$  and rewards  $\mathcal{R}$  for the cancer environment—using the same normalization as in Table 3. Even when Continuous Planning takes the same number of observations as determined by AOC, it still performs worse, because those observations are not well located. As Continuous Planning observations are taken blindly at regular times, rather than at more informative points.
Policy	Cancer
U	R	O
Active Observing Control	105±0.183	98.8±0.173	3.39±0.0306
Continuous Planning with O = 3	102±0.234	95.6±0.234	3±0
Continuous Planning with O = 4	103±0.226	97.3±0.226	4±0
Monitoring as it can determine when, in continuous time to observe, whereas those methods suffer from discretization errors—that compound, leading to worse performance.
5.2 Insight experiments
In the following, we gain insight into why AOC performs better than Continuous Planning, i.e., better than taking regular observations in time and the importance of continuous-time monitoring as opposed to discrete-time monitoring.
How does irregularly observing achieve a higher expected utility than regularly observing? To better understand why monitoring (active observing) approaches outperform planning that regularly observes, we provide a qualitative and quantitative study.
On the one hand, qualitatively, as detailed in Figure 3, we observe that for a given threshold, AOC can still solve the Cancer environment while taking less costly observations and that AOC automatically determines to increase the observing frequency when the tumor volume  $v$  is large and less when it is smaller. Crucially, this matches what clinicians have already discovered, where they categorize the cancer volume into five discrete bins where the observing frequency is correlated to tumor volume [Geng et al., 2017, Seedat et al., 2022]. Moreover, this provides insight into how irregular observing can achieve a higher expected utility (Section 2.1), as observations could be more informative (hence taken more frequently) when the future state change magnitude is larger—occurring for large  $v$ .
Furthermore, as in Figure 4 for the Pendulum environment, we observe that AOC observes infrequently the state of the pendulum when it has been swung up to the upright goal state and is balancing it there. This matches our intuition from Section 2.1, as future state changes of  $\phi$  the angle of the pendulum are large when the pendulum is not at the equilibrium goal state, thus having  $\dot{\phi} >0$ , which is maximally large in the swing up state trajectory phase and necessitates frequent observing, whereas balancing near the top equilibrium point,  $\dot{\phi}\approx 0$  necessitates infrequent observing.
On the other hand, quantitatively, as tabulated in Table 4, we make Continuous Planning take the same number of observations as AOC, however, taking them regularly. Since AOC can take better-located observations in time, it outperforms Continuous Planning, achieving a higher reward  $\mathcal{R}$ . In summary, these experiments demonstrate that actively observing can be superior to planning in continuous time.
Why is it crucial to actively observe with continuous-time methods, rather than discrete-time methods? We compare Active Observing Control to Discrete Monitoring, which is the closest discrete counterpart (a discrete ablation of AOC) and look at when the observing times are reached, relative to their reward uncertainty estimates, to determine whether time discretization affects observing times. As shown in Figure 5, we see that Discrete Monitoring takes observations that are delayed, and therefore achieves a lower state reward  $\mathcal{R}$ . This allows AOC to capture fast-moving states, where missing these can be catastrophic or deadly, as in the case of cancer.
Figure 5: Reward uncertainty  $\sigma(r)$  normalized by the method specific threshold  $\tau$  on the cancer environment—thus the threshold to observe is at 1.0, as indicated by the red horizontal line. We observe that AOC can detect and catch when the uncertainty crosses the threshold, independent of the time discretization. In contrast, Discrete Monitoring suffers from time discretization error and takes a delayed observation, leading to compounding errors and worse performance. Here Discrete Monitoring misses observing a critical test by over a day.
Figure 6: Normalized utility  $\mathcal{U}$  on the cancer environment of Active Observing Control and Continuous Planning, plotted against changing uncertainty threshold  $\tau$ . We plot AOC's threshold as used in Table 3 as the red line. AOC maintains a high utility and reward over a wide range of thresholds—whilst using fewer observations as  $\tau$  increases, compared to Continuous Planning which takes expensive frequent observations. The threshold  $\tau$  is between the minimum feasible  $\tau_{\mathrm{min}}$  and maximum feasible  $\tau_{\mathrm{max}}$  values.
5.3 Sensitivity analysis of  $\tau$
Active Observing Control still outperforms Continuous Planning even at variable  $\tau$  on the cancer environment, Figure 6. We note that as  $\tau$  is increased, this increases the observing times  $\Delta_{i+1}$ , and hence fewer observations of the state are taken. Although we set  $\tau = 6.67$  for the cancer environment, following the procedure in Section 4, we observe robustness to other choices of  $\tau$  are also suitable with decreasing reward  $\mathcal{R}$ ; however, AOC still outperforms and with a high utility  $\mathcal{U}$ . Moreover, we tuned  $\tau_{\mathrm{min}}$  so that frequent observations were taken to be at least equal to Continuous Planning, and then  $\tau_{\mathrm{max}}$  to take the fewest observations, which is often minimally 1, as we start by taking an observation, or until the entire action trajectory plan is executed, and requires re-planning by taking another observation.
6 Conclusion and Future work
This work lays the foundation for the significant yet unexplored real-world problem of continuous-time control whilst deciding when to take costly observations, which has profound implications across diverse fields such as medicine, resource management, and low-power systems. Our key theoretical contribution, verified empirically is that regular observing is not optimal and that irregularly observing can achieve a higher expected utility. To demonstrate the power of active irregular observing, we provided an initial solution using a heuristic threshold based on the variance of reward rollouts in an offline continuous-time model-based model predictive control planner. However, the quest for the optimal method remains an open problem, providing fertile ground for future research.
However, this work is not without limitations. The heuristic threshold, while a robust initial solution in our experiments, may not be optimal or suitable for all scenarios. Moreover, our approach relies on the assumption that the offline dataset provides sufficient coverage of the state-action space to learn a useful enough model of the system dynamics and that the observation observes all the states plus Gaussian noise. Also, in some practical applications, the cost of observations could vary or be unknown.
Navigating these limitations illuminates exciting pathways for future research. Potential directions include: optimizing multiple observations concurrently over a planned action trajectory, jointly optimizing action trajectories and observing times, and theoretically deriving the optimal solution to reveal characteristics of optimal observing policies.
Acknowledgements. SH would like to acknowledge and thank AstraZeneca for funding. This work was additionally supported by the Office of Naval Research (ONR) and the NSF (Grant number: 1722516). Moreover, we would like to warmly thank all the anonymous reviewers, alongside research group members of the van der Scaar lab, for their valuable input, comments and suggestions as the paper was developed—where all these inputs ultimately improved the paper.