SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Aguiar, Nikka Mofid, and Hyunjie Alex Nam. Exploring optimal control with observations at a cost. arXiv preprint arXiv:2006.15757, 2020.
Ahmed M Alaa and Mihaela Van Der Schaar. Balancing suspense and surprise: Timely decision making with endogenous information acquisition. Advances in Neural Information Processing Systems, 29, 2016.
Arthur Argenson and Gabriel Dulac-Arnold. Model-based offline planning. In International Conference on Learning Representations, 2020.
Karl J Astrom. Event based control. In Analysis and design of nonlinear control systems: In honor of Alberto Isidori, pages 127-147. Springer, 2008.
Karl Johan Åström and Bo Bernhardsson. Comparison of periodic and event based sampling for first-order stochastic systems. IFAC Proceedings Volumes, 32(2):5006-5011, 1999.
Tim D Barfoot, Chi Hay Tong, and Simo Särkkä. Batch continuous-time trajectory estimation as exactly sparse gaussian process regression. In Robotics: Science and Systems, volume 10, pages 1-10. CiteSeer, 2014.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834-846, 1983.
Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn. Active measure reinforcement learning for observation cost minimization. arXiv preprint arXiv:2005.12697, 2020a.
Colin Bellinger, Andriy Drozdyuk, Mark Crowley, and Isaac Tamblyn. Balancing information with observation costs in deep reinforcement learning. 2020b.
Ioana Bica, Ahmed M Alaa, James Jordon, and Mihaela van der Schaar. Estimating counterfactual treatment outcomes over time through adversarially balanced representations. arXiv preprint arXiv:2002.04083, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
L Brunereau, F Bruyère, C Linassier, and J-L Baulieu. The role of imaging in staging and monitoring testicular cancer. Diagnostic and interventional imaging, 93(4):310-318, 2012.
Shelly Butler, Denise Kirschner, and Sussane Lenthart. Optimal control of chemotherapy affecting the infectivity of hiv. Advances in mathematical population dynamics: molecules, cells, man, pages 104-120, 1997.
Joaquin Quinonero Candela, Agathe Girard, Jan Larsen, and Carl Edward Rasmussen. Propagation of uncertainty in bayesian kernel models-application to multiple-step ahead forecasting. In 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings (ICASSP'03), volume 2, pages II-701. IEEE, 2003.
Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Pratap Tokekar, and Dinesh Manocha. Dealing with sparse rewards in continuous control robotics via heavy-tailed policies. arXiv preprint arXiv:2206.05652, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
A Broader Impact Statement 17
B Further Application Examples 17
C Proof for Proposition 2.1 17
D Extended Related Work 20
E MPC MPPI Pseudocode and Planner Implementation Details 23
F Active Observing Control Planning Algorithm Pseudocode 24
G Active Observing Control Run-time Complexity 25
H Practical Guidelines to Select  $c$  26
I Environment Selection and Details 26
I.1 ODE-RL Suite 27
I.2 Cancer Environment 29
J Benchmark Method Implementation Details 30
J.1 Probabilistic Dynamics Model 30
J.2 Observing Policies 31
J.3 MPPI Implementation 31
K Evaluation Metrics 32
L Additional Experiments 32
L.1 Ablation of Thresholding the State Uncertainty 32
L.2 Sparse Reward Environment 33
L.3 Increasing the Planning Resolution of MPC (decreasing  $\delta_{a}$  ) 33
L.4 Discrete Time Controller Trained on an Equidistant Dataset 34
L.5 Sensitivity of AOC to  $\delta_{a}$  35
L.6 Evaluation of Reward Uncertainty 36
L.7 Relation Between Rewards  $\mathcal{R}$  and the Fixed Observation Cost  $c$  36
L.8 Dependence on the Accuracy of the Learned Dynamics Model 37
L.9 Extending AOC to Work with a Learned Reward Model 37
L.10 AOC Also Empirically Works for Non-linear State Transformations 38
L.11 Further Empirical Validation in Other Real-World Scenarios 38
Code. All code is available at https://github.com/samholt/ActiveObservingInContinuous-timeControl.
A Broader Impact Statement
Our work on active observing in continuous-time control presents a novel perspective on decision-making in domains such as medicine, low-power systems, and resource management, with the potential to substantially reduce costs and increase efficiency. However, the adoption of these techniques also presents potential risks and ethical concerns. The dependability of these systems could raise issues if they malfunction or are compromised, potentially leading to harmful decisions in critical fields like healthcare. Hence, robustness, security, and error handling must be prioritized in implementation.
B Further Application Examples
The problem of continuously controlling continuous time environments whilst actively deciding when to take costly observations in time is ubiquitous in real-world environments, with some of these applications listed in Table 5.
Table 5: Applications of continuous control with costly observations.
Continuous time environment	Ref.	Continuous control	Costly observation	High stakes
Medical cancer chemotherapy treatment	[Geng et al., 2017, Brunereau et al., 2012]	Continuous chemotherapy dosing	Computed tomography scan	Errors can be fatal
Biological fish population management	[Yoshioka and Tsujimura, 2020]	Continuous food and temperature control	Fish population survey	Underfeeding or overfishing can lead to extinction
Low power communication	[Mastronarde and van der Schaar, 2012, Lin and van der Schaar, 2010]	Continuous channel transmission	Measure the maximum bandwidth	Communication rate errors lead to loss of data
Mobile Robotics	[Martinez-Cantin et al., 2007]	Continuous robot control	Measure the robot's position	Errors can lead to damage to the robot or environment
Agricultural resource management	[Larson, 1992]	Continuous land allocation for livestock	Measure the health of livestock	Errors can lead to livestock death
Nursing surveillance	[Morgan et al., 2005]	Continuous treatment plans	Measure the health of a patient	Errors can be fatal
Water pollution monitoring	[Twardowska et al., 2007]	Continuous water treatment	Measure the water quality	Errors can lead to water contamination
C Proof for Proposition 2.1
Proof. We will consider the system defined by
$$
\frac {\mathrm {d} s}{\mathrm {d} t} = s \tag {2}
$$
$$
s (0) = \left\{ \begin{array}{l l} 1 & \text {w i t h p r o b a b i l i t y} \quad 1 / 2 \\ - 1 & \text {w i t h p r o b a b i l i t y} \quad 1 / 2 \end{array} \right. \tag {3}
$$
$$
\sigma_ {\epsilon} = 1 \tag {4}
$$
Note that solution to this system can be written as  $s(t) = s(0)e^{t}$ , which is independent of any actions that might have been taken. Also, consider the reward function given by
$$
r (s, a, t) = \delta (t - T) \cdot \mathbb {1} \{s \cdot a > 0 \} \tag {5}
$$
Given a history of samples  $h = \{(t_j, z(t_j), a(t_j))\}$ , a posterior over  $s(0)$  can be written as
$$
\mathbb {P} (s (0) = 1 | h) = \frac {\mathbb {P} (s (0) = 1) \mathbb {P} (h | s (0) = 1)}{\mathbb {P} (h)} \quad = \frac {1}{\mathbb {P} (h)} \prod_ {j = 1} ^ {| h |} \frac {1}{\sqrt {2 \pi}} e ^ {- 1 / 2 \left(z \left(t _ {j}\right) - e ^ {t _ {j}}\right) ^ {2}} \tag {6}
$$
$$
\mathbb {P} (s (0) = - 1 | h) = \frac {\mathbb {P} (s (0) = - 1) \mathbb {P} (h | s (0) = - 1)}{\mathbb {P} (h)} = \frac {1}{\mathbb {P} (h)} \prod_ {j = 1} ^ {| h |} \frac {1}{\sqrt {2 \pi}} e ^ {- 1 / 2 \left(z \left(t _ {j}\right) + e ^ {t _ {j}}\right) ^ {2}} \tag {7}
$$
Using this posterior, we define an auxiliary variable
$$
b (h) = \log \mathbb {P} (s (0) = 1 | h) - \log \mathbb {P} (s (0) = - 1 | h) \tag {8}
$$
$$
= - \frac {1}{2} \sum_ {j = 1} ^ {| h |} \left(z \left(t _ {j}\right) - e ^ {t _ {j}}\right) ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {| h |} \left(z \left(t _ {j}\right) + e ^ {t _ {j}}\right) ^ {2} \tag {9}
$$
Then, the optimal decision policy can be written as
$$
\pi^ {*} (h, t) = \left\{ \begin{array}{l l} 1 & \text {i f} \quad \mathbb {P} (s (0) = 1 | h) \geq 1 / 2 \Longleftrightarrow b (h) \geq 0 \\ - 1 & \text {i f} \quad \mathbb {P} (s (0) = 1 | h) <   1 / 2 \Longleftrightarrow b (h) <   0 \end{array} \right. \tag {10}
$$
Next, we derive the expected utility of the optimal decision policy given a set of observing times  $\{t_j\in [0,T]\}_{j = 1}^n$ . Denoting with  $h_n = \{(t_j,z(t_j),a(t_j))\}_{j = 1}^n$  the final history, we have
$$
\mathcal {R} = \int_ {0} ^ {T} r (s (t), a (t), t) \mathrm {d} t \tag {11}
$$
$$
= \int_ {0} ^ {T} \delta (t - T) \cdot \mathbb {1} \{s (t) \cdot a (t) > 0 \} \mathrm {d} t \tag {12}
$$
$$
= \mathbb {1} \{s (T) \cdot a (T) > 0 \} \tag {13}
$$
$$
= \mathbb {1} \left\{s (T) \cdot \pi^ {*} \left(h _ {n}, T\right) > 0 \right\} \tag {14}
$$
$$
= \mathbb {1} \{s (0) = 1 \wedge \pi^ {*} \left(h _ {n}, T\right) = 1 \} + \mathbb {1} \{s (0) = - 1 \wedge \pi^ {*} \left(h _ {n}, T\right) = - 1 \} \tag {15}
$$
$$
= \mathbb {1} \left\{s (0) = 1 \wedge b \left(h _ {n}\right) \geq 0 \right\} + \mathbb {1} \left\{s (0) = - 1 \wedge b \left(h _ {n}\right) <   0 \right\} \tag {16}
$$
Hence,
$$
\begin{array}{l} \mathbb {E} [ \mathcal {R} | \{t _ {j} \} _ {j = 1} ^ {n} ] = \mathbb {P} (s (0) = 1 \wedge b \left(h _ {n}\right) \geq 0 \mid \left\{t _ {j} \right\}) + \mathbb {P} (s (0) = - 1 \wedge b \left(h _ {n}\right) <   0 | | \left\{t _ {j} \right\}) (17) \\ = \mathbb {P} (s (0) = 1 | \{t _ {j} \}) \mathbb {P} (b \left(h _ {n}\right) \geq 0 | s (0) = 1, \left\{t _ {j} \right\}) (18) \\ + \mathbb {P} (s (0) = - 1 | \{t _ {j} \}) \mathbb {P} (b \left(h _ {n}\right) <   0 | s (0) = - 1, \left\{t _ {j} \right\}) (19) \\ = 1 / 2 \cdot \mathbb {P} (b \left(h _ {n}\right) \geq 0 | s (0) = 1, \left\{t _ {j} \right\}) + 1 / 2 \cdot \mathbb {P} (b \left(h _ {n}\right) <   0 | s (0) = - 1, \left\{t _ {j} \right\}) (20) \\ \end{array}
$$
Observing that
$$
\begin{array}{l} \mathbb {P} \left(b \left(h _ {n}\right) \geq 0 \mid s (0) = 1, \left\{t _ {j} \right\}\right) (21) \\ = \mathbb {P} \left(- \frac {1}{2} \sum_ {j = 1} ^ {b} \left(z \left(t _ {j}\right) - e ^ {t _ {j}}\right) ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {b} \left(z \left(t _ {j}\right) + e ^ {t _ {j}}\right) ^ {2} \geq 0 \mid s (0) = 1\right) (22) \\ = \mathbb {P} \left(- \frac {1}{2} \sum_ {j = 1} ^ {b} \left(\varepsilon \left(t _ {j}\right) + s \left(t _ {j}\right) - e ^ {t _ {j}}\right) ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {b} \left(\varepsilon \left(t _ {j}\right) + s \left(t _ {j}\right) + e ^ {t _ {j}}\right) ^ {2} \geq 0 \mid s (0) = 1\right) (23) \\ = \mathbb {P} \left(- \frac {1}{2} \sum_ {j = 1} ^ {b} \left(\varepsilon \left(t _ {j}\right)\right) ^ {2} + \frac {1}{2} \sum_ {j = 1} ^ {b} \left(\varepsilon \left(t _ {j}\right) + 2 e ^ {t _ {j}}\right) ^ {2} \geq 0\right) (24) \\ = \mathbb {P} \left(\sum_ {j = 1} ^ {b} \left(e ^ {2 t _ {j}} + e ^ {t _ {j}} \varepsilon \left(t _ {j}\right)\right) \geq 0\right) (25) \\ = \mathbb {P} _ {\bar {\varepsilon} \sim \mathcal {N} \left(0, \sum_ {j = 1} ^ {n} e ^ {2 t _ {j}}\right)} \left(\bar {\varepsilon} > - \sum_ {j = 1} ^ {n} e ^ {2 t _ {j}}\right) (26) \\ = \mathbb {P} _ {\bar {\varepsilon} \sim \mathcal {N} (0, 1)} \left(\bar {\varepsilon} > - \sqrt {\sum_ {j = 1} ^ {n} e ^ {2 t _ {j}}}\right) (27) \\ = F \left(\sqrt {\sum_ {j = 1} ^ {n} e ^ {2 t _ {j}}}\right) (28) \\ \end{array}
$$
where  $F(x) = \int (1 / \sqrt{2\pi})e^{-(1 / 2)x^2}$  is the cumulative distribution function of the standard normal distribution, and similarly that  $\mathbb{P}(b(h_n)\geq 0|s(0) = -1,\{t_j\}) = F(\sqrt{\sum_{j = 1}^n e^{2t_j}})$ , we obtain
$$
\mathbb {E} [ \mathcal {U} | \{t _ {j} \} _ {j = 1} ^ {n} ] = \mathbb {E} [ \mathcal {R} | \{t _ {j} \} _ {j = 1} ^ {n} ] - c n = \frac {1}{2} F \left(\sqrt {\sum_ {j = 1} ^ {n} e ^ {2 t _ {j}}}\right) - c n \tag {29}
$$
Next, we introduce two classes of observing policies. The first class consists of observing policies that observe regularly in time with an observing interval of  $\delta$  such that  $\rho^{(\mathrm{reg},\delta)}(h) = \delta$  for all  $h$ . When enrolled, the observing times for these policies can be written as  $t_j = j\delta$  hence
$$
\mathbb {E} _ {\rho^ {(\operatorname {r e g}, \delta)}} [ \mathcal {U} ] = \frac {1}{2} F \left(\sqrt {\sum_ {j = 1} ^ {\lfloor T / \delta \rfloor} e ^ {2 j \delta}}\right) - c \left\lfloor \frac {T}{\delta} \right\rfloor \tag {30}
$$
The second class consists of observing policies that take  $\ell$ -many observations all at  $t = T$  such that
$$
\rho^ {(\text {i r r e g}, \ell)} (h = \left\{\left(t _ {j}, z \left(t _ {j}\right), a \left(t _ {j}\right)\right) \right\} _ {j = 1} ^ {n}) = \left\{ \begin{array}{l l} T - t _ {j} & \text {i f} n <   \ell \\ \infty & \text {i f} n \geq \ell \end{array} \right. \tag {31}
$$
When enrolled, the observing times for these policies can be written as  $t_j = T$  hence
$$
\mathbb {E} _ {\rho^ {(\text {i r e g}, \ell)}} [ \mathcal {U} ] = \frac {1}{2} F \left(\sqrt {\ell e ^ {2 T}}\right) - c \ell \tag {32}
$$
Crucially,  $\rho^{(\mathrm{irreg},\ell)}$  with  $\ell \geq 2$  is distinct from all regular observing policies  $\rho^{(\mathrm{reg},\delta)}$  with  $\delta >0$ , since for no regular observing policy,  $t_1 = t_2$ . We prove that  $\rho^*$  cannot be a regular observing policy by showing that, for each  $\rho^{(\mathrm{reg},\delta)}$ , there exists at least one  $\rho^{(\mathrm{irreg},\ell)}$  with  $\ell \geq 2$  that achieves higher expected utility.
For  $\delta = 0$ ,  $\mathbb{E}_{\rho^{(\mathrm{reg},\delta)}}[\mathcal{U}]\to -\infty$  whereas  $\mathbb{E}_{\rho^{(\mathrm{irreg},[T / \delta])}}[\mathcal{U}]$  is bounded for any  $\ell \in \{2,3,\ldots \}$  (hence  $\mathbb{E}_{\rho^{(\mathrm{irreg},\ell)}}[\mathcal{U}] > \mathbb{E}_{\rho^{(\mathrm{reg},\delta)}}[\mathcal{U}]$ ).
For  $\delta \leq T / 2$ , we have  $\lfloor T / \delta \rfloor \geq 2$  and  $\mathbb{E}_{\rho^{(\mathrm{irreg},\lfloor T / \delta \rfloor)}}[\mathcal{U}] > \mathbb{E}_{\rho^{(\mathrm{reg},\delta)}}[\mathcal{U}]$  since
$$
\begin{array}{l} \mathbb {E} _ {\rho^ {(\operatorname {r e g}, \delta)} \left[ \mathcal {U} \right]} - \mathbb {E} _ {\rho^ {(\operatorname {i r r e g}, [ T / \delta ])} \left[ \mathcal {U} \right]} (33) \\ = \frac {1}{2} F \left(\sqrt {\sum_ {j = 1} ^ {\lfloor T / \delta \rfloor} e ^ {2 j \delta}}\right) - c \left\lfloor \frac {T}{\delta} \right\rfloor - \frac {1}{2} F \left(\sqrt {\left\lfloor \frac {T}{\delta} \right\rfloor e ^ {2 T}}\right) + c \left\lfloor \frac {T}{\delta} \right\rfloor (34) \\ = \frac {1}{2} F \left(\sqrt {\sum_ {j = 1} ^ {\lfloor T / \delta \rfloor} e ^ {2 j \delta}}\right) - \frac {1}{2} F \left(\sqrt {\left\lfloor \frac {T}{\delta} \right\rfloor e ^ {2 T}}\right) (35) \\ <   \frac {1}{2} F \left(\sqrt {\sum_ {j = 1} ^ {\lfloor T / \delta \rfloor} e ^ {2 T}}\right) - \frac {1}{2} F \left(\sqrt {\left\lfloor \frac {T}{\delta} \right\rfloor e ^ {2 T}}\right) = 0 (36) \\ \end{array}
$$
Let the cost per observation be given as
$$
c = \frac {1}{2} \min  \left\{\frac {1}{2} F (\sqrt {2 e ^ {2 T}}) - \frac {1}{2} F (\sqrt {e ^ {2 T}}), \frac {1}{4} F (\sqrt {e ^ {2 T}}) - \frac {1}{4} F (0) \right\} > 0 \tag {37}
$$
Then, for  $T / 2 < \delta \leq T$ , we have  $\lfloor T / \delta \rfloor = 1$  and  $\mathbb{E}_{\rho(\mathrm{irreg}, 2)}[\mathcal{U}] > \mathbb{E}_{\rho(\mathrm{reg}, \delta)}[\mathcal{U}]$  since
$$
\begin{array}{l} \mathbb {E} _ {\rho^ {(\text {r e g}, \delta)}} [ \mathcal {U} ] - \mathbb {E} _ {\rho^ {(\text {i r r e g}, 2)}} [ \mathcal {U} ] (38) \\ = \frac {1}{2} F \left(\sqrt {e ^ {2 \delta}}\right) - c - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right) + 2 c (39) \\ = \frac {1}{2} F \left(\sqrt {e ^ {2 \delta}}\right) - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right) + c (40) \\ \leq \frac {1}{2} F \left(\sqrt {e ^ {2 T}}\right) - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right) + c = 0 (41) \\ \leq \min  \left\{\frac {1}{2} F \left(\sqrt {e ^ {2 T}}\right) - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right), \frac {1}{4} F (0) - \frac {1}{4} F \left(\sqrt {e ^ {2 T}}\right) \right\} + c (42) \\ = - 2 c + c <   0 (43) \\ \end{array}
$$
Finally, for  $T < \delta$ , we have  $\lfloor T / \delta \rfloor = 0$  and  $\mathbb{E}_{\rho(\mathrm{irreg},2)}[\mathcal{U}] > \mathbb{E}_{\rho(\mathrm{reg},\delta)}[\mathcal{U}]$  since
$$
\begin{array}{l} \mathbb {E} _ {\rho^ {(\text {r e g}, \delta)}} [ \mathcal {U} ] - \mathbb {E} _ {\rho^ {(\text {i r r e g}, 2)}} [ \mathcal {U} ] (44) \\ = \frac {1}{2} F (0) - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right) + 2 c (45) \\ \leq \min  \left\{\frac {1}{2} F (0) - \frac {1}{2} F \left(\sqrt {2 e ^ {2 T}}\right), F \left(\sqrt {e ^ {2 T}}\right) - F \left(\sqrt {2 e ^ {2 T}}\right) \right\} + 2 c (46) \\ = - 4 c + 2 c <   0 (47) \\ \end{array}
$$
D Extended Related Work
In the following, we expand on the existing related work, in Section 3, and we provide additional discussions of the benefits of offline RL, model-based RL, why model predictive control is our preferred action policy, event-based sampling, Semi-MDP methods, and Linear Quadratic Regression (LQR) & Linear Quadratic Gaussian (LQG) methods.
Why offline RL. The setting of offline reinforcement learning consists of an agent that learns a dynamics model (i.e., a model of the environment dynamics) from a dataset of previously collected state-action trajectories from a specific environment and the agent is not allowed to interact with the environment [Wu et al., 2019]. For instance, it is challenging to apply online RL methods to real-world problem settings, as they rely on expensive trial-and-error approaches performed either online or in a realistic simulator—that is not often possible. In contrast, "offline" model-based RL learns a model of the environment dynamics from a previously collected dataset of state-action trajectories, that is often possible. These methods then control the environment to a desired goal state using any available planning method such as training a policy [Fujimoto et al., 2018] or using
a model predictive controller (MPC) [Williams et al., 2016]. Specifically, both model-free [Kumar et al., 2019, 2020, Fujimoto and Gu, 2021] and model-based [Kidambi et al., 2020, Wang et al., 2021] approaches have been proposed for offline RL, where in general model-based methods have shown to be more sample efficient than model-free methods [Moerland et al., 2020].
Why model-based reinforcement learning? Core to Model-based RL methods is the desire to create policies for real world tasks in the offline setting where the true environment dynamics are unknown and rather learn an approximate dynamics model from an offline dataset of state-action trajectories from the environment. A key benefit of model-based reinforcement learning is that it can be significantly more sample efficient compared to model-free RL methods [Lutter et al., 2021]—where model-free methods can require millions or billions of interactions with the environment to learn a good policy. Another key benefit is that learning a dynamics model of the environment enables planning with that dynamics model to optimize actions, such as in using a model predictive control planner, rather than learning a specific policy.
Principally, the dynamics model is independent of the reward and therefore allows changing reward of the planning policy—enabling a planning-based policy to easily adapt to changing goal states/tasks on the fly at run-time. Conversely, a policy that was solely trained for a particular task would have to be re-trained, or a conditional policy trained when adapting goals/tasks [Lutter et al., 2021]. This is realistic as real-world continuous-time control settings of the dynamics model (e.g., often the physics/biological process of an environment [Holt et al., 2022a]) is independent of the reward function. Importantly model-based RL for conventional control tasks has been shown to be more sample-efficient than model-free methods [Wang et al., 2019, Moerland et al., 2020]. Historically, model-free methods have demonstrated expert performance on challenging tasks [Mnih et al., 2015, Lillicrap et al., 2015], and was later shown that proper tuning of model-based methods have much higher sample efficiency Ha and Schmidhuber [2018], Janner et al. [2019] and can achieve expert performance with probabilistic dynamics models on continuous control tasks [Chua et al., 2018].
Model predictive control (MPC). Core to model predictive control (MPC) is having a good dynamics model, historically using simple, first principle derived (often linear), known dynamics models [Richalet et al., 1978, Salzmann et al., 2022]. Recently, MPC Model Predictive Path Integral (MPPI) [Williams et al., 2017] is a zeroth-order particle-based (action) trajectory optimizer [Williams et al., 2017] method that can use a general, non-differentiable nonlinear dynamics model with a defined complex reward function (or cost function).
Particularly, Williams et al. [2017] demonstrated that it was suitable to be used with a learned neural network dynamics model—which was used for the task of driving a vehicle around a dirt track in aggressive, actuator saturating maneuvers. In general, MPC is often computationally infeasible for optimizing actions up to long time horizons, therefore are often used to optimize actions up to a fixed time horizon, which is re-computed iteratively when needed. A key benefit of planning, and MPC, is that it can incorporate state constraint feasibility into its plans as well as easily changing the reward function for a changing goal/task at run-time. We specifically use this MPC method in this paper, as it is state-of-the-art [Wang et al., 2022] for performing MPC with a learned dynamics model, improving upon the widely used cross-entropy method (CEM) MPC planner [Kobilarov, 2012].
Hybrid MPC: Combining a powerful MPC planner with a learned policy is another exciting area developing [Argenson and Dulac-Arnold, 2020]. These existing hybrid works, work on only discrete-time environment dynamics models. These include MBOP [Argenson and Dulac-Arnold, 2020], TD-MPC [Hansen et al., 2022] and DADS [Sharma et al., 2019]. We highlight that these hybrid methods are unable to determine when to observe in continuous-time and only determine discrete actions to execute.
Sensing approaches have been proposed of when to optimally take an observation in both discrete time [Jarrett and Van Der Schaar, 2020] and continuous time [Alaa and Van Der Schaar, 2016, Barfoot et al., 2014]—where their goal is to identify an underlying state. However, these approaches cannot also continuously control the system. In contrast, Active Observing Control seeks to both actively observe and control the system. Besides deciding when to take an observation, some work on sensing focuses on when to stop taking observations [e.g. Hüyük et al., 2023] or what kind of observations to take [e.g. Curth et al., 2023]. Unlike these works, we focus exclusively on the timing of observations.
Planning approaches only focus on optimal decision policies  $\pi$ , and therefore observing has to be provided by a schedule, which is often assumed to be at a regular constant interval in time
$\Delta_{i+1} = \Delta_i$  — that is, observations are not actively planned [Yildiz et al., 2021]. Within planning approaches, there exist many discrete-time approaches [Chua et al., 2018, Mnih et al., 2013, Williams et al., 2017, Sun et al., 2023] and recently more continuous-time approaches [Du et al., 2020, Yildiz et al., 2021, Holt et al., 2023]—where these use a continuous-time dynamics model, e.g. [Chen et al., 2018, Holt et al., 2022b]. Specifically, Yildiz et al. [2021] presented a seminal online continuous-time model-based RL algorithm, leveraging a continuous-time dynamics model that can predict the next state at an arbitrary time  $s(t)$ . However, all these methods are unable to plan when to take the next observation, in contrast to Active Observing Control which is able to determine when to take the next observation whilst planning an action trajectory.