SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Monitoring approaches consist of both a decision policy  $\pi$  and an observing policy  $\rho$ ; however, existing approaches only consider the discrete-time and discrete-state setting with observation costs [Sharma et al., 2017, Nam et al., 2021, Aguiar et al., 2020, Bellinger et al., 2020a, Krueger et al., 2020, Bellinger et al., 2020b]. In particular, Krueger et al. [2020] proposed that even in a simple setting of a discrete-state multi-armed bandit, computing the optimal time to take the next observation is intractable—therefore, they must rely on heuristics for their observing policy<sup>1</sup>.
Broadly, discrete-time monitoring methods use a discrete-time model and propagate the last observed state, often appending either a flag or a counter to the state-action tuple to indicate the time since an observation was taken [Aguiar et al., 2020, Bellinger et al., 2020b]. However, these methods cannot be applied to continuous-time environments or propagate the predicted current state and its associated prediction interval of the uncertainty associated with the state estimate. Moreover, training a policy [Aguiar et al., 2020] that decides at each state whether to observe it or not, requires a discretization of time. Whereas Active Observing Control, which determines the continuous-time interval of when to observe the next state, does not require any discretization of time, and hence it is a continuous-time method and avoids compounding time discretization errors.
Event-based sampling a similar related work area in the control community [Åström and Bernhardsson, 1999, Aström, 2008], which addresses the similar problem of controlling a system, of only taking a full state observation when an external input event occurs, and then providing a control action. To create this event, it assumes part of the state is continuously observed, and often an event is defined when this part of the state (or a function of it) crosses a fixed threshold (e.g., a physical sensor crossing a threshold) [Vasyutynskyy and Kabitzsch, 2010]. This finds multiple uses in problems such as electronic analog control systems for audio and mobile telephone systems [Åström and Bernhardsson, 1999], and battery capacity optimization in wireless devices [Vasyutynskyy and Kabitzsch, 2010]. However, this is different from our proposed problem of continuous-time control whilst deciding when to take costly observations as (1) Event-based sampling assumes part of the state is continuously observed. Often in our environments, it is not feasible to observe part of the state continuously (e.g., imaging a cancer tumor volume or performing a medical test) or it is prohibitively expensive to continually observe at a high-frequency part of the state (similar to Continuous Planning approaches). (2) The event input (the time to take the next observation) is given as a control input to the agent. However, it is precisely the more difficult problem we tackle of coming up with an observing policy that decides when to take the next observation. (3) The event is often defined by a human in advance and is a function of the current partial state. General environments may not have partial state spaces that are predictive of when to observe next, such as a robotic navigation task in two dimensions and only continuously observing one dimension.
$^{1}$ Krueger et al. [2020], focus on the simpler problem of multi-armed bandits (MAB), where there is a cost to observe the reward. In the MAB setting, it is possible to formulate it as an RL PO-MDP with one state. The underlying static state (the fixed reward distributions of each bandit) is unknown and must be determined by taking successive costly observations (paying a cost to observe a reward). Therefore, Krueger et al. [2020]'s statement that the optimal algorithm for their simple MAB setting is intractable applies to the problem of active observing in continuous-time control.
Semi-MDP methods is another similar related field, where it extends the MDP problem to include options that define temporally abstracted action sequences [Sutton, 1998]. However, some distinct differences prevent using a semi-MDP to address our continuous-time control whilst deciding when to take costly observations problem. These differences include (1) Semi-MDP is still an underlying discrete MDP with discrete state transitions, whereas our problem formulation focuses on continuous-time systems that can handle continuous actions and states. (2) Semi-MDP formulation does not involve an observation cost, whereas our problem formulation does.
LQR/LQG methods. Similar related methods from control exist, as Zhang et al. [2022] propose a discrete planning method that works only for linear quadratic regulator (LQR) systems and optimizes the observing frequency for a given total budget of observations. Moreover, Huang and Zhu [2020] addresses an infinite horizon discrete-time linear quadratic Gaussian (LQG) control problem with observation costs. This is notably different from the formulation proposed for the problem of continuous-time control with observation costs as (1) Huang and Zhu [2020] only applies to linear systems that are discrete in time and have an infinite time horizon; our formulation applies to nonlinear systems that are continuous in time and have a fixed time horizon. (2) Huang and Zhu [2020] assume their full system dynamics model is known; ours makes no such assumptions and only assumes access to an offline collected dataset of (possibly irregular in time) state-action trajectories to learn a dynamics model, which is more applicable to real-world environments.
E MPC MPPI Pseudocode and Planner Implementation Details
To plan an optimal action trajectory  $a$ , we specify that any model predictive controller (MPC) planner can be used—that can optimize the action trajectory up to a specified time horizon  $H \in \mathbb{R}_+$ . We opt to use an MPC planner, as it optimizes an entire action trajectory  $a$  up to  $H$ , does not require the probabilistic dynamics model  $\hat{f}_{\theta}$  to be differentiable (it uses no gradients), and the reward function  $r$  can be changed on the fly—allowing changing goals/tasks at run-time. Moreover, model-based RL with MPC planners has achieved comparable performance to model-free RL methods [Chua et al., 2018, Lutter et al., 2021]. We use our probabilistic dynamics model  $\hat{f}_{\theta}$ , with the MPC planner of Model Predictive Path Integral Control (MPPI), a zeroth-order particle-based trajectory optimizer [Williams et al., 2017], due to its competitive performance [Wang et al., 2022].
To optimize the action trajectory  $a$  up to a time horizon  $H$ , it discretizes  $H$  into smaller constant action time intervals  $\delta_{a} \in \mathbb{R}_{+}$  which are then optimized—where there are  $K = \lceil \frac{H}{\delta_a} \rceil \in \mathbb{Z}_+$  time steps in  $H$ —i.e.,  $t^{(k)} = t_i + k\delta_a, k \in \{0, \dots, K - 1\}$ . It forward simulates a number of parallel rollouts  $G \in \mathbb{Z}_{+}$  where the next state estimate at each time step is simulated as  $z(t^{(k + 1)}) = \mu_*(z(t^{(k)}), a(t^{(k)}), \delta_a)$  recursively. This requires a state estimate  $z(t_i)$  to plan from; therefore, we recompute the planned trajectory when we observe a new sample of the state. This optimizes the action trajectory by
$$
a ^ {*} (t ^ {(\cdot)}) = \underset {a (t ^ {(\cdot)})} {\arg \max } \mathbb {E} _ {z} \left[ \sum_ {k = 0} ^ {K - 1} r (z (t ^ {(k)}), a (t ^ {(k)}), t ^ {(k)}) \right]
$$
MPPI is also a Monte Carlo based sampler, and thus increasing the number of rollouts improves the input trajectory optimization, however, scales the run-time complexity as  $O(GK)$ . The standard MPPI algorithm [Williams et al., 2017] is used with our probabilistic dynamics model  $\hat{f}_{\theta}$ . We articulate the MPPI pseudocode for the action trajectory optimizer in Algorithm 2.
Algorithm 2 MPPI-Trajectory-Optimization
Input: State observation  $z(t_{i})$ , Pre-trained probabilistic dynamics model  $\hat{f}_{\theta}$ , Reward function  $r$ , Time Horizon  $H$ , Action time interval  $\delta_{a}$ , Number of parallel rollouts  $G$ , Noise covariance  $\Sigma$ , Hyper parameters  $\lambda$ , Action max  $a_{\mathrm{max}}$ , Action min  $a_{\mathrm{min}}$ .
$$
K \leftarrow \frac {H}{\delta_ {n}}
$$
$$
\boldsymbol {R} ^ {(G)} \leftarrow \mathbf {0} ^ {(\mathbf {G})}
$$
$$
\boldsymbol {A} ^ {(G, K)} \leftarrow \mathbf {0} ^ {(\mathbf {G}, \mathbf {K})}
$$
$$
\begin{array}{l} \boldsymbol {A} ^ {\prime (G, [ 0: K - 1 ])} \leftarrow \mathbf {0} ^ {(\mathbf {G}, \mathbf {K})} \\ \text {n o i s e .} \end{array}
$$
$$
\varepsilon^ {(G, K)} \leftarrow \mathbf {0} ^ {(\mathbf {G}, \mathbf {K})}
$$
This holds our  $G$  trajectory returns.
$\triangleright$  This holds our  $G$  action trajectories of length  $K$ .
$\triangleright$  This holds  $G$  action trajectories of length  $K$  that are perturbed by
$$
\mathbf {f o r} q = 0, \dots , G - 1:
$$
$$
\boldsymbol {z} (t ^ {(0)}) \leftarrow z (t _ {i})
$$
$\triangleright$  Sample  $G$  trajectories over the horizon  $K$ .
$$
\mathbf {f o r} k = 0, \dots , K - 1:
$$
$$
\varepsilon^ {(g, k)} \leftarrow \mathcal {N} (\mathbf {0}, \boldsymbol {\Sigma})
$$
$\triangleright$  Sample action noise.
$$
\boldsymbol {A} ^ {\prime (g, k)} \leftarrow \boldsymbol {A} ^ {(g, k)} + \boldsymbol {\epsilon} ^ {(g, k)}
$$
$\triangleright$  Perturb action by noise.
$\pmb{A}^{\prime(g,k)} \gets \min(\max(\pmb{A}^{\prime(g,k)}, -1), +1) \triangleright$  Clip normalized perturbed noise (to bound actions to their limits).
$$
\varepsilon^ {(g, k)} \leftarrow A ^ {\prime (g, k)} - A ^ {(g, k)}
$$
$\triangleright$  Update noise after bounding, so we do not penalize clipped
$\pmb{z}(t^{(k + 1)}) \gets \mu_{*}(\pmb{z}(t^{(k)}), \pmb{A}'^{(k)} \cdot \pmb{a}_{\max}, \delta_{a}) \triangleright$  Sample next state from pre-trained probabilistic mics model  $\hat{f}_{\theta}$ .
$\pmb{R}^{(g)} \gets \pmb{R}^{(g)} + r(\pmb{z}(t^{(k+1)}), \pmb{A}^{(g,k)}) - \lambda \pmb{A}^{(g,k)T}\Sigma^{-1}\pmb{\varepsilon}^{(g,k)} \triangleright$  Accumulate the current state reward.
$$
\kappa \leftarrow \min  _ {g} [ \boldsymbol {R} ^ {(g)} ]
$$
[ \pmb{T}^{(k)} = \frac{\sum_{g=0}^{G-1} \exp\left(\frac{1}{\lambda}(\pmb{R}^{(g)} - \kappa)\right) \pmb{\varepsilon}^{(g,k)}}{\sum_{g=0}^{G-1} \left(\frac{1}{\lambda}(\pmb{R}^{(g)} - \kappa)\right)} \cdot \pmb{a}_{\max}, \forall k \in [0, K-1] \quad \triangleright \text{Generate the return-weighted trajectory update.} ]
Return:  $T$
F Active Observing Control Planning Algorithm Pseudocode
The Active Observing Control planning algorithm pseudocode is outlined in Algorithm 3. In particular,  $\delta_t$  is the continuous search (root finding algorithm) tolerance that is used when searching with binary search for the continuous time duration that the computed action trajectory can be followed for, which occurs when the standard deviation of the reward crosses the threshold  $\tau$ . We note that all numerical root-finding algorithms generally involve a search tolerance or a similar stopping criterion. The  $\delta_t$  tolerance ensures that the binary search algorithm stops evaluations once a solution time is found that is "close enough" to the actual true value. Another way to view the search tolerance  $\delta_t$ , is that the numerical precision that the search value is correct up to.
Algorithm 3 Active Observing Control Policy
Input: State observation  $z(t_{i})$ , Pre-trained probabilistic dynamics model  $\hat{f}_{\theta}$ , Reward uncertainty threshold  $\tau$ , Reward function  $r$ , Time Horizon  $H$ , Search tolerance  $\delta_{t}$ , Action time interval  $\delta_{a}$ .
$a\gets \mathrm{MPC}(f_{\theta},z(t_i),r,H)$  Plan action trajectory.   
for  $k = 0,\dots ,K - 1$  ..   
 $t^{(k)}\gets t_i + k\delta_a$    
for  $p\gets 1,\ldots ,P$  ..   
 $z_{p}(t^{(k + 1)})\sim \mathcal{N}(\mu_{*}(z(t^{(k)}),a(t^{(k)}),\delta_{a}),\sigma_{*}^{2}(z(t^{(k)}),a(t^{(k)}),\delta_{a}))$    
if  $\sqrt{\mathbb{V}_p[r(z_p(t^{(k + 1)}),a(t^{(k + 1)}))]} >\tau$  ..   
 $t^{\mathrm{(lower)}}\gets t^{(k)},t^{\mathrm{(upper)}}\gets t^{(k + 1)}$    
while  $t^{\mathrm{(upper)}} - t^{\mathrm{(lower)}} > \delta_t$  ..   
 $t^{\mathrm{(mid)}}\gets (t^{\mathrm{(upper)}} + t^{\mathrm{(lower)}}) / 2$    
for  $p = 1,\ldots ,P$  ..   
 $z_{p}(t^{\mathrm{(mid)}})\sim \mathcal{N}(\mu_{*}(z(t^{(k)}),a(t^{(k)}),t^{\mathrm{(mid)}} - t^{(k)}),\sigma_{*}^{2}(z(t^{(k)}),a(t^{(k)}),t^{\mathrm{(mid)}} - t^{(k)}))$    
if  $\sqrt{\mathbb{V}_p[r(z_p(t^{\mathrm{(mid}}),a(t^{\mathrm{(mid}})))]} >\tau$  ..   
 $t^{\mathrm{(upper)}}\gets t^{\mathrm{(mid)}}$    
else:   
 $t^{\mathrm{(lower)}}\gets t^{\mathrm{(mid)}}$    
Return  $a(t)\forall t\in [t_i,t^{(\mathrm{mid})})$    
Return  $a(t)\forall t\in [t_i,t_K)$
G Active Observing Control Run-time Complexity
Active Observing Control's run-time complexity of Algorithm 3 is  $O(GK + P(K + W))$  where  $W = \lceil (\log(\delta_a) - \log(\delta_t)) / \log(2) \rceil \in \mathbb{Z}_+$ . As  $W$  is determined by a continuous binary real line search, on an interval of  $\delta_a$  up to a resolution of  $\delta_t$ .
Proof. By definition binary search halves the search interval  $\delta_{a}$  every iteration therefore
$$
(1 / 2) ^ {W} \delta_ {a} <   \delta_ {t} \tag {48}
$$
$$
(1 / 2) ^ {W} <   \frac {\delta_ {t}}{\delta_ {a}} \tag {49}
$$
$$
- W \log (2) <   \log \left(\delta_ {t}\right) - \log \left(\delta_ {a}\right) \tag {50}
$$
$$
W > \frac {\log (\delta_ {t}) - \log (\delta_ {a})}{\log (2)} \tag {51}
$$
$$
W = \left[ \left(\log \left(\delta_ {a}\right) - \log \left(\delta_ {t}\right)\right) / \log (2) \right] \tag {52}
$$
Empirically we chose  $P = 10G$ ,  $W = 4$ ,  $G = 10,000$ ,  $K = 20$  for all experiments—therefore the dominating scaling parameter is  $G$  the number of particle rollouts. We observe in Table 6 that AOC takes approximately  $2.4 \times$  longer to plan, which includes both planning the actions and when to observe, compared to Continuous Planning which only plans actions (an MPC step). In the case of the Cancer environment, it can take fewer observations and therefore spend less time planning overall. Moreover, AOC is still practical in environments that require fast decision-making, i.e., robotics, as AOC can still produce a plan in less time than the planning action interval  $\delta_{a}$  for the Cartpole environment, as shown in Table 7. Therefore, AOC is a practical method that can be utilized across different environments (planning cancer treatment plans with high accuracy, whilst being fast enough to control continuous robots). Furthermore, we note that MPC has demonstrated scaling to high dimensional environments (Chua et al. [2018] with  $G = 20$ , Lutter et al. [2021] with  $G = 500$ ).
Table 6: Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$ , observations  $\mathcal{O}$ , average time taken to plan policy method  $\mathcal{T}^{(\text{Average})}$ , total time spent planning policy in episode  $\mathcal{T}^{(\text{Total})}$  for the Cancer environment—using the same normalization as in Table 3. Even though Active Observing Control takes  $2.4 \times$  longer to plan both the actions and the observing time  $\mathcal{T}^{(\text{Average})}$ , compared to Continuous Planning that only plans the actions—AOC can achieve less total planning time  $\mathcal{T}^{(\text{Total})}$  (hence less total compute) because it takes fewer observations in total—here Continuous Planning takes  $1.5 \times$  longer total planning time  $\mathcal{T}^{(\text{Total})}$ .
Policy	Cancer
U	R	O	T(Average) (ms)	T(Total) (ms)
Active Observing Control	105	98.8	3.4	31.1	105
Continuous Planning	100	100	13	12.8	166
Table 7: Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$ , observations  $\mathcal{O}$ , average time taken to plan policy method  $\mathcal{T}^{(\text{Average})}$ , total time spent planning policy in episode  $\mathcal{T}^{(\text{Total})}$  for the Cartpole environment—using the same normalization as in Table 3. Even though Active Observing Control takes  $1.7 \times$  longer to plan both the actions and the observing time  $\mathcal{T}^{(\text{Average})}$ , compared to Continuous Planning that only plans the actions—both policies can be run at real-time as the action interval  $\delta_{a} = 100$  ms for the Cartpole environment—making AOC a practical method.
Policy	Cartpole
U	R	O	T(Average) (ms)	T(Total) (ms)
Active Observing Control	140	100	43.2	24.9	1073
Continuous Planning	100	100	50	14.0	700
H Practical Guidelines to Select  $c$
The observation cost  $c$  should be decided based on the real-world application at hand and include any human preferences in that application if applicable. In real-world systems (with resource constraints), this cost might correspond to actual monetary cost, computational expense, or energy consumption. Further, if this cost involves a human, for example, a patient receiving medical treatment, they may have preferences for treatment, their health impact (e.g., chemotherapy side effects), and or the number, frequency, and timing of treatments that could be included in the cost  $c$ . Also, there can often be a trade-off between control performance and the number of observations taken, and one way to trade this off is to tune  $c$ .
I Environment Selection and Details
In the following we discuss our reasoning for why we selected the continuous-time environments. Principally, a continuous-time environment is defined by an underlying differential equation (DE) system (e.g., physics or biological system), which allows for the environment state trajectories to be sampled and simulated at any continuous-time, which is unlike discrete environments [Yildiz et al., 2021, Brockman et al., 2016]. We used an ordinary differential equation solver [Virtanen et al., 2020] to simulate all environments, using an Euler solver at a time resolution of  $\delta_{\mathrm{sim}}$  as indicated by the environments parameters. We selected the standard continuous-time control environments from the ODE-RL suite [Yildiz et al., 2021], which consists of three well known environments: Pendulum, CartPole, and Acrobot. Furthermore, as detailed in the following, we implemented a Cancer environment that uses a simulation of a Pharmacokinetic-Pharmacodynamic (PK-PD) model of lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. Where the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. We further adapt all the environments to have a fixed cost when taking a sample of the state. To learn a continuous-time dynamics model, it is a standard assumption to assume the dataset of state-action trajectories is sampled irregularly in time [Yildiz et al., 2021,
<sup>2</sup>Where the ODE-RL suite of the environments used can be downloaded freely from https://github.com/cagatayildiz/oderl.
[Chen et al., 2018]. First let us justify the reasoning behind our choice of observing irregularly in time state-action trajectories from these environments and why existing offline datasets are not suitable.
Why we cannot use an offline dataset of agent trajectories from a discrete-time environment. There exist standard discrete-time offline datasets [Fu et al., 2020] of state-action trajectories of agents interacting with discrete-time environments—where the state-action trajectories are sampled at regular time intervals  $\Delta_{j+1} = \Delta_j$ . Let us hypothetically imagine that there exists a regularly-sampled  $\Delta_{j+1} = \Delta_j$  in time state-action trajectory offline dataset. Is it possible to sample the trajectories irregularly? Two approaches come to mind. 1) One could imagine using some form of interpolation (e.g., splines or similar) to interpolate to irregular time steps between states and actions, however doing so would lead to errors in the sampled state-action trajectories in comparison to the true irregularly sampled state-action trajectories at those non-uniform time points. These errors could compound over a dynamics model being trained on these; therefore, we highlight that this approach is unsuitable. 2) Another approach would be to start with a regularly sampled state-action trajectory and then sub-sample state-action times from that regular grid of collected times. However, again this approach is unsuitable, as often environments are captured at run-time with a particular regular observation time interval  $\Delta$ —where only observing discrete multiples of this (i.e.,  $n\Delta$ ,  $n \in \mathbb{Z}_+$ ) would lead to gaps between observations, where the mean of the observation intervals would be larger than that of the environments nominal run-time observation interval  $\bar{\Delta}_{\text{Sub-sample}} > \bar{\Delta}_{\text{Original Trajectory}}$ . Therefore, we note that this becomes a different problem, as there is less information in the state-action trajectories with large observation interval gaps.
Offline dataset generation  $\mathcal{D}$ . Rather, to mitigate both of these issues above (1,2) we prefer to collect an offline dataset ourselves by observing irregularly in time state-action trajectories, where the time interval between the state-action time points is sampled from an exponential distribution  $\Delta_{i+1} \sim \mathrm{Exp}(\bar{\Delta})$ , with a mean of  $\bar{\Delta} = \delta_a$  seconds, and state-action values are randomly sampled, collecting a total of 1e6 samples [Yildiz et al., 2021]. Furthermore, to increase realism, we add Gaussian noise to observations taken, that of  $\mathcal{N}(0, \sigma_\epsilon^2)$  with standard deviation  $\sigma = 0.01$ .
In all environments the actions are continuous and bounded to a defined range  $[a_{\min}, a_{\max}]$ . Here we assume a given state  $s(t)$  is composed of the position state  $q$  and their respective velocities  $\dot{q}$ , i.e.,  $s(t) = \{q(t), \dot{q}(t)\}$ . Furthermore, each environment uses the reward function of the exponential of the negative distance from the current state to the goal state  $q^*$ , whilst also penalizing the magnitude of action, and we assume that we are given this reward function when planning—as we often know the desired goal state  $q^*$  and our current state  $q$ . Therefore, the reward function for the environments has the following form:
$$
r (\{q (t), \dot {q} (t) \}, a (t)) = \exp (- | | q (t) - q ^ {*} | | _ {2} ^ {2} - b | | \dot {q} (t) | | _ {2} ^ {2}) - v | | a (t) | | _ {2} ^ {2} \tag {53}
$$
Where  $b$  and  $v$  are specific environment constants [Yildiz et al., 2021]. Specifically, when we use our MPC planner we observe that it plans better without the exponential operator, and therefore remove it, and use the following reward function throughout,  $r(\{q(t),\dot{q} (t)\} ,a(t)) = -||q(t) - q^{*}||_{2}^{2} - b||\dot{q} (t)||_{2}^{2} - v||a(t)||_{2}^{2}$ . Yildiz et al. [2021] set the environments parameters of  $b,v$  to penalize large values and enforce exploration from trivial states, and we use their same values which are also tabulated in Table 8. For all environments we assume a fixed cost of  $c = 50$  when observing a sample of the state.
Table 8: Environment specification parameters.
Base Environment	b	v	amax	sInit	q*
Pendulum	1e-2	1e-2	[2]	[0.1, 0.1]	[0, L]
Cartpole	1e-2	1e-2	[3]	[0.05, 0.05, 0.05, 0.05]	[0, 0, L]
Acrobot	1e-4	1e-2	[4, 4]	[0.1, 0.1, 0.1, 0.1]	[0, 2L]
Cancer	1e-3	1e-2	[5, 2]	[1138, 0]	[0, 0]
I.1 ODE-RL Suite
The starting state, for all these tasks, is hanging down, and the goal is to swing up and stabilize the pole(s) upright [Yildiz et al., 2021] in each environment.
Specifically, we set  $\delta_{a} = 0.1$  seconds, and simulate each ODE system at a time resolution of  $\delta_{\mathrm{sim}} = 0.01$  seconds, up until the episode length of  $T = 5$  seconds. The goal states for these