SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Interestingly we observe all baselines degrading in performance, however, AOC still has the highest utility amongst the baselines for a set  $\delta_{a}$ . Intuitively the MPC MPPI algorithm produces less optimal action trajectory  $a$  plans when the number of discrete actions  $K$  in the discrete action sequence increases for a fixed number of parallel rollouts  $G$ . We note that this is inherent to using an MPC planner [Williams et al., 2017], and could be improved by increasing  $G$  for decreasing  $\delta_{a}$ , however, that is out of scope for this work.
L.4 Discrete Time Controller Trained on an Equidistant Dataset
The Continuous Planning baseline (which uses a continuous-time dynamics model) can perform better than the Discrete Planning baseline (which uses a discrete-time dynamics model), as it has a more accurate dynamics model—when both are trained on the irregularly sampled offline dataset  $\mathcal{D} = \{(z(t_i),a(t_i),\Delta_i)\}_{i = 1}^N$ . As the offline dataset of state-action trajectories  $\mathcal{D}$ ,  $\Delta_{i + 1}$  has irregular times between state-action samples  $\Delta_{i + 1}\sim \mathrm{Exp}(\bar{\Delta})$ , with a mean of  $\bar{\Delta} = \delta_{a}$  seconds.
We test what would happen if the discrete-time dynamics model (for the Discrete Planning and Discrete Monitoring baselines) was instead trained on a regular (equidistant) sampled offline dataset  $\mathcal{D}$ , i.e.,  $\Delta_{i+1} = \bar{\Delta} = \delta_a$ , we performed an additional experiment. This is in the Cancer environment, with each result averaged over 100 random seeds, as Tabulated in Table 16.
Empirically this agrees with our intuition that the discrete-time dynamics model baselines do indeed perform better when trained on an offline dataset that is regularly sampled. However, the discrete-time baselines still underperform Active Observing Control, due to their inherent time discretization error, Figure 6.
Table 15: Increasing the planning resolution of MPC (decreasing  $\delta_{a}$ ) for the Cancer environment. Here we only normalize utility to Continuous planning with  $\delta_{a} = 0.1$  (the best possible reward) for each baseline whilst varying  $\delta_{a}$ .
Discretization interval δa	Policy	UtilityU	RewardR	ObservationsO
0.1	Random	0±0	0±0	13±0
0.1	Discrete Planning	92.2±1.11	92.2±1.11	13±0
0.1	Discrete Monitoring	91.1±1.72	85.9±1.7	4.94±0.109
0.1	Continuous Planning	100±0.597	100±0.597	13±0
0.1	Active Observing Control	105±0.75	98.4±0.707	3.42±0.102
0.05	Random	0±0	0±0	13±0
0.05	Discrete Planning	85.6±1.44	85.6±1.44	13±0
0.05	Discrete Monitoring	85.9±1.3	82.1±1.26	7.18±0.153
0.05	Continuous Planning	89.3±1.27	89.3±1.27	13±0
0.05	Active Observing Control	92.9±1.2	87.7±1.14	4.95±0.118
0.02	Random	0±0	0±0	13±0
0.02	Discrete Planning	71.8±1.3	71.8±1.3	13±0
0.02	Discrete Monitoring	72.8±1.22	68.6±1.23	6.5±0.196
0.02	Continuous Planning	77.1±1.22	77.1±1.22	13±0
0.02	Active Observing Control	75.6±1.01	69±1.03	2.77±0.0839
0.01	Random	0±0	0±0	13±0
0.01	Discrete Planning	63.2±1.03	63.2±1.03	13±0
0.01	Discrete Monitoring	63.8±1.07	59.2±1.06	5.89±0.0741
0.01	Continuous Planning	60.3±1.32	60.3±1.32	13±0
0.01	Active Observing Control	64.4±1.19	59.1±1.13	4.79±0.117
Table 16: Results for the Cancer environment where the discrete-time dynamics model is trained on a regular (equidistant) sampled offline dataset  $\mathcal{D}$ .
Δi+1Distribution	Policy	Utility U	Reward R	Observations O
Irregularly sampled, Δi+1~Exp(Δ)	Random	0±0	0±0	13±0
Regularly sampled, Δi+1=Δ	Discrete Planning	95.5±0.667	95.5±0.667	13±0
Regularly sampled, Δi+1=Δ	Discrete Monitoring	96.1±1.83	90.1±1.83	3.75±0.127
Irregularly sampled, Δi+1~Exp(Δ)	Continuous Planning	100±0.594	100±0.594	13±0
Irregularly sampled, Δi+1~Exp(Δ)	Active Observing Control	105±0.755	98.5±0.713	3.4±0.102
L.5 Sensitivity of AOC to  $\delta_{a}$
$\delta_{a}$  is often chosen for a user, as it is the average time interval between two state-action points in a trajectory in the collected offline dataset  $\mathcal{D}$ .
Here MPC depends on a given  $\delta_{a}$ , however, the observing policy is orthogonal to  $\delta_{a}$ . As outlined in Appendix L.3, reducing  $\delta_{a}$  whilst keeping the planning time horizon  $H$  and the number of parallel rollouts  $G$  the same, reduces the MPC MPPI action trajectory quality, leading to control performance that achieves a lower reward, and hence utility.
We performed an additional experiment to verify this on the Cancer environment, varying  $\delta_{a} = \{0.1, 0.05, 0.02, 0.01\}$ , each result averaged over 100 random seeds, as tabulated in Table 17. Here we normalize utility to Continuous planning with  $\delta_{a} = 0.1$  (the best possible reward) for each baseline whilst varying  $\delta_{a}$ .
Table 17: Results for the Cancer environment where vary  ${\delta }_{a}$  for AOC.
Discretization interval δa	Policy	Utility U	Reward R	Observations O
0.1	Random	0±0	0±0	13±0
0.1	Continuous Planning	100±0.597	100±0.597	13±0
0.1	Active Observing Control	105±0.75	98.4±0.707	3.42±0.102
0.05	Active Observing Control	92.9±1.2	87.7±1.14	4.95±0.118
0.02	Active Observing Control	75.6±1.01	69±1.03	2.77±0.0839
0.01	Active Observing Control	64.4±1.19	59.1±1.13	4.79±0.117
L.6 Evaluation of Reward Uncertainty
The reliability of the reward uncertainty depends on the reliability of the probabilistic dynamics model  $\hat{f}_{\theta}$  to provide a good predictive uncertainty for a future state. We highlight that the predictive uncertainty of the dynamics model depends only on the training, that of training on the offline dataset of trajectories  $\mathcal{D}$  which is only performed once. At run-time the dynamics model is only used for inference, therefore its predictive uncertainty for the state-action space will remain the same, independent of the number of samples of the state taken in a given evaluation episode.
To illustrate this point, we performed an additional experiment on the Cancer environment where, at run-time, we vary the number of observations (samples of the state taken) in an episode and evaluate the negative log-likelihood (NLL) [Lakshminarayanan et al., 2017] of the predicted reward uncertainty to the ground truth reward uncertainty for 100 random seeds, as tabulated in Table 18.
Table 18: Evaluation of reward uncertainty of AOC on the Cancer environment.
State Observations	NLL
1	8.261 ± 1.124
10	8.138 ± 1.060
100	8.190 ± 1.052
1,000	8.241 ± 0.973
10,000	7.933 ± 0.288
100,000	7.988 ± 0.093
L.7 Relation Between Rewards  $\mathcal{R}$  and the Fixed Observation Cost  $c$
An exact theoretical relation between rewards  $\mathcal{R}$  and the fixed observation cost  $c$  is intractable to provide for the general case.
We encounter this intractability even for our simple analytic system used to prove Proposition 2.1. As let us start with Equation (1),  $\mathcal{U} = \int_0^T r(s(t),a(t),t)\mathrm{d}t - c|\{t_i:t_i\in [0,T]\} |$  , where the Reward is  $\mathcal{R} = \int_0^T r(s(t),a(t),t)\mathrm{d}t$  and Cost is  $\mathcal{C} = c|\{t_i:t_i\in [0,T]\} |$  . For even a simple regular observing policy (Appendix C),  $\mathbb{E}_{\rho^{(\mathrm{reg},\delta)}}[\mathcal{U}]$  , which is a function of  $\delta \in (0,T)$  , as well as the reward function  $r(s(t),a(t),t)$  and the observation cost  $c$
If we keep  $r$ ,  $\pi$ ,  $c$  fixed, and attempt to find  $\rho^{*} = \operatorname{argmax}_{\rho}\mathbb{E}[\mathcal{U}]$  by only varying  $\delta$ , that is by  $\frac{\mathrm{d}}{\mathrm{d}\delta}\mathbb{E}[\mathcal{U}] = 0$ , calculating a closed-form solution for  $\frac{\mathrm{d}}{\mathrm{d}\delta}\mathbb{E}[\mathcal{U}]$  is intractable.
Instead, we can empirically simulate different values of  $\delta$ , then see which grid sweep value of  $\delta$  maximizes  $\mathbb{E}[\mathcal{U}]$ , such that approximately  $\delta^{*} = \operatorname{argmax}_{\delta}\mathbb{E}[\mathcal{U}]$ , this value can then be used to calculate the Reward  $\mathcal{R} = \int_0^T r(s(t),a(t),t)\mathrm{d}t$ .
We do this below on the Cancer environment where we use a regular observing policy, that of Continuous Planning, and vary  $\delta$  over a grid sweep such that different run settings of the policy take different observations, as tabulated in Table 19. We start with setting  $c = 1$  and only normalize the Reward  $\mathcal{R}$  to 100 for the most frequent samples possible  $(\mathcal{O} = 13)$ , each result calculated for 100 random seeds.
Table 19: Cancer environment where we use a regular observing policy, that of Continuous Planning, and vary  $\delta$  over a grid sweep such that different run settings of the policy take different observations.
Observations O	Reward R	Utility U
13 ± 0	100 ± 0.616	87
7 ± 0	99.3 ± 0.656	92.3
4 ± 0	96 ± 0.821	92
2 ± 0	90.7 ± 1.01	88.7
1 ± 0	89.2 ± 1.1	88.2
For a given cost  $c$ , i.e., in this case  $c = 1$ , we see that there exists an approximate number of observations that maximize the Utility, here for  $c = 1$  this is  $\mathcal{O} = 7$ . This allows us to vary the observation cost  $c$  numerically, and for each observation cost, determine the number of observations (or regular observing frequency) that maximizes the Utility  $\mathcal{U}(\mathcal{O}^* = \operatorname{argmax}_{\mathcal{O}} \mathbb{E}[\mathcal{U}])$  and determine what the Reward  $\mathcal{R}$  for this  $\mathcal{O}^*$  is. This is tabulated in Table 20.
Table 20: Cancer environment where we vary the observation cost  $c$  numerically, and for each observation cost, determine the number of observations (or regular observing frequency) that maximizes the Utility  $\mathcal{U}\left( {{\mathcal{O}}^{ * } = {\operatorname{argmax}}_{\mathcal{O}}\mathbb{E}\left\lbrack  \mathcal{U}\right\rbrack  }\right)$  and determine what the Reward  $\mathcal{R}$  for this  ${\mathcal{O}}^{ * }$  is.
Fixed Observation Cost c	Optimal Utility U*	Optimal number of Observations O*	Reward R
0	100	13	100
1	92.3	7	99.3±0.656
2	88	4	96±0.821
3	86.2	1	89.2±1.1
Empirically we observe the relation between the  $c$  and  $\mathcal{R}$  is that the fixed observation cost  $c$  is negatively correlated to the Reward  $\mathcal{R}$ , when using a policy that maximizes the Utility  $\mathcal{U}$ , Equation (1).
L.8 Dependence on the Accuracy of the Learned Dynamics Model
To understand the dependence on the accuracy of the learned dynamics model further, we performed an additional experiment ablation, where we benchmarked all methods with a less accurate dynamics model—training all the dynamics models with fewer samples (10% of the total amount of samples used in training the dynamics models presented in the main paper), here trained on 100,000 samples. This is tabulated in Table 21.
We observe that AOC still achieves a high average utility  $\mathcal{U}$  on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Specifically, this empirically further verifies the key theoretical contribution that regular observing is not optimal and that irregularly observing can achieve a higher expected utility.
Table 21: Ablation with a less accurate dynamics model—training all the dynamics models with fewer samples (10% of the total amount of samples used in training the dynamics presented in the main paper), here trained on 100,000 samples. We observe that AOC still achieves a high average utility  $\mathcal{U}$  on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$  and observations  $\mathcal{O}$  for the benchmark methods, across each environment. Results are averaged over 1,000 random seeds, with  $\pm$  indicating  $95\%$  confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every  $\delta_{a}$ .
Policy	Cancer	Acrobot	Cartpole	Pendulum
U	R	O	U	R	O	U	R	O	U	R	O
Random	0±0	0±0	13±0	0±0	0±0	50±0	0±0	0±0	50±0	0±0	0±0	50±0
Discrete Planning	99.3±0.701	99.3±0.701	13±0	67.2±6.46	67.2±6.46	50±0	117±9.05	117±9.05	50±0	51±3.13	51±3.13	50±0
Discrete Monitoring	74.4±2.76	69.1±2.73	4.51±0.161	142±6.4	40±6.37	5.48±0.111	709±36.9	-97.1±36.9	6±0	249±1.18	13.6±1.94	6.3±0.24
Continuous Planning	100±0.754	100±0.754	13±0	100±6.74	100±6.74	50±0	100±2.51	100±2.51	50±0	100±2.87	100±2.87	50±0
Active Observing Control	104±0.767	98.3±0.735	3.62±0.0968	157±6.93	54.5±6.97	5.29±0.0948	825±2.35	0.427±2.35	5±0	277±1.28	39.4±1.25	5.99±0.0527
L.9 Extending AOC to Work with a Learned Reward Model
To investigate whether AOC can also be used with a learned reward model, we performed an additional experiment, where we trained an MLP reward model (4-layer MLP with 128 units and Tanh activations) from the offline dataset in all the benchmarks—this is tabulated in Table 22.
We observe that AOC still achieves a high average utility  $\mathcal{U}$  on the Cancer environment, outperforming the competing Continuous Planning and Discrete Monitoring methods. This empirically verifies that our proposed initial approach can still perform well using a learned reward model and that the theoretical contribution still holds.
Table 22: We use a learned MLP reward model (4-layer MLP with 128 units and Tanh activations) from the offline dataset in all the benchmarks. We observe that AOC still achieves a high average utility  $\mathcal{U}$  on the Cancer environment, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$  and observations  $\mathcal{O}$  for the benchmark methods, across the Cancer environment. Results are averaged over 1,000 random seeds, with  $\pm$  indicating  $95\%$  confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every  $\delta_{a}$ .
Policy	U	R	O
Random	0±0	0±0	13±0
Discrete Planning	91.4±0.383	91.4±0.383	13±0
Discrete Monitoring	95.1±0.4	91.6±0.392	7.63±0.0631
Continuous Planning	100±0.151	100±0.151	13±0
Active Observing Control	105±0.178	98.9±0.168	3.38±0.0303
L.10 AOC Also Empirically Works for Non-linear State Transformations
We performed an additional experiment to investigate AOC's performance within environments that utilize observations stemming from non-linear state transformations. We tailored the existing Cancer environment to render observations via the non-linear state transformation function  $z(t) = 0.1(s(t) + \epsilon(t))^2 + (s(t) + \epsilon(t))$ . The subsequent results, conducted across 1,000 random seeds, are outlined in Table 23. We observe that AOC still achieves a high average utility  $\mathcal{U}$  on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Specifically, this empirically further verifies our key theoretical contribution that regular observing is not optimal and that irregularly observing can achieve a higher expected utility.
Table 23: We adapted the existing Cancer environment to render observations via the non-linear state transformation function  $z(t) = 0.1(s(t) + \epsilon (t))^2 +(s(t) + \epsilon (t))$ . We observe that AOC still achieves a high average utility  $\mathcal{U}$  on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods. Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$  and observations  $\mathcal{O}$  for the benchmark methods, across the Cancer environment. Results are averaged over 1,000 random seeds, with  $\pm$  indicating  $95\%$  confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every  $\delta_{a}$ .
Policy	U	R	O
Random	0±0	0±0	13±0
Discrete Planning	91.7±0.397	91.7±0.397	13±0
Discrete Monitoring	90.5±0.559	85.4±0.548	5.25±0.0329
Continuous Planning	100±0.21	100±0.21	13±0
Active Observing Control	104±0.221	98.9±0.208	4.68±0.0315
L.11 Further Empirical Validation in Other Real-World Scenarios
To verify the theoretical contribution in additional real-world scenarios, we show empirically in the following that AOC can successfully operate in the real-world environments (medical and engineering) of an accurate Glucose environment (controlling the injected insulin for a diabetic patient [Lenhart and Workman, 2007]), a (Human Immunodeficiency Virus) HIV environment (controlling the chemotherapy dose for affecting the infectivity of HIV in a patient [Butler et al., 1997]), and a Quadrotor environment (controlling the actuators of an unmanned aerial vehicle [Nonami et al., 2010]). Each environment represents a unique challenge and has direct implications in medical and engineering applications, thus reflecting the broad applicability of our method. These are:
- Glucose environment. Controlling the injected insulin for a diabetic patient to regulate their blood glucose level—here observations are costly as a blood test must be performed to measure the glucose level [Lenhart and Workman, 2007].
- HIV environment. Controlling the chemotherapy dose for affecting the infectivity of HIV in a patient—where observations are costly as a blood test must be performed to measure CD4\*T cell levels [Butler et al., 1997].
- Quadrotor environment. Controlling the actuators of an unmanned aerial vehicle—where observations can be costly due to performing an expensive (power and compute) localization measure [Nonami et al., 2010].
We observe that AOC still achieves a high average utility  $\mathcal{U}$  on all environments, outperforming the competing Continuous Planning and Discrete Monitoring methods, reinforcing our theoretical claims, and extending our empirical validation—as tabulated in Table 24.
Table 24: Normalized utilities  $\mathcal{U}$ , rewards  $\mathcal{R}$ , and observations  $\mathcal{O}$  for the benchmark methods, across each environment. AOC performs the best across all environments. Results are averaged over 1,000 random seeds, with  $\pm$  indicating  $95\%$  confidence intervals. Utilities and rewards are undiscounted and normalized to be between 0 and 100, where 0 corresponds to a Random agent, and 100 corresponds to the expert, that of Continuous Planning, taking state observations at every  $\delta_{a}$ .
Policy	Glucose	HIV	Quadcoptor
U	R	O	U	R	O	U	R	O
Random	0±0	0±0	50±0	0±0	0±0	50±0	0±0	0±0	50±0
Discrete Planning	96±0.485	96±0.485	50±0	152±0.121	152±0.121	50±0	101±0.00882	101±0.00882	50±0
Discrete Monitoring	120±0.489	92.9±0.493	15.7±0.0466	2.77e+03±0.455	126±0.455	7±0	1.79e+03±0.199	101±0.0142	5.99±0.00518
Continuous Planning	100±0.39	100±0.39	50±0	100±0.431	100±0.431	50±0	100±0.015	100±0.015	50±0
Active Observing Control	126±0.41	99.9±0.39	17.5±0.0336	2.83e+03±1.56	107±0.878	5.75±0.0269	1.83e+03±0.0789	97.9±0.0258	5±0.00196