SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ODE-RL suite environments are when the pole(s), each of length  $L$  are fully upright, such that their  $x, y$  co-ordinates of the tip of the pole reach the goal state. That is where  $q^{*}$  is:  $[0, L]$  for the Pendulum environment,  $[0, 0, L]$  for the Cartpole environment—where the additional 0 is zero for the cart's  $x$  location and in Acrobot is  $[0, 2L]$  as there are two poles connected to each other. Furthermore, upon restarting the environment the initial state  $s$  is sampled from the uniform distribution of  $s_0 \sim \mathcal{U}[-s_{\mathrm{Init}}, s_{\mathrm{Init}}]$  [Yildiz et al., 2021], then the  $\theta$  states are added with set angle such that the pole(s) are pointing downwards (i.e., Cartpole  $\theta' = \theta_{\mathrm{Init}} + \pi$ ).
In the following we describe each of the ODE-RL environments introducing the environment with a screenshot figure.
I.1.1 Cartpole (swing up) Environment
(b)
Figure 7: Screen shots of the Cartpole environment. The task is to swing up a pole attached to a cart that can move horizontally along a rail. In the following we see: (a) the starting downward state with an additional small amount of perturbation, (b) the optimal trajectory solution found by a policy that scores  $\mathcal{R} \approx 100\%$  including our Active Observing Control policy and (c) the final goal state that has been reached, that is, to swing up the pole and stabilize it upwards—which is a challenging control task. We note that the control actuator is bounded and limited, and the force is such that the Cartpole cannot be directly swung up—rather it must gain momentum through a swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position, as indicated when the tip of the pole reaches the centre of the red target square. Furthermore, we note this environment is an underactuated system.
We can see in Figure 7, an illustration of the starting state Figure 7 (a) with a small perturbed random initial start. Here a pole is attached to an un-actuated joint to a cart that moves along a frictionless track [Barto et al., 1983]. The pendulum starts in the downward position Figure 7 (a) and the goal is to swing the pendulum upwards and then balance the pole upright by applying forces to the left or right horizontal direction of the cart. This environment has the state of  $[x,\dot{x},\theta ,\dot{\theta} ]$  and a corresponding observation of  $[x,\dot{x},\cos (\theta),\sin (\theta),\dot{\theta} ]$ , where  $\theta \in (-\pi ,\pi)$  is measured from the upward vertical of the pole. We note that this environment is an underactuated system, as it has two degrees of freedom  $[x,\theta ]$ , however only the carts position is actuated, leaving  $\theta$  indirectly controlled. Additionally, for this Cartpole environment only, created a more competitive Random policy than randomly executing actions, that of applying no actions to keep the pole in the same starting position. This achieves a significantly higher reward than a pure Random policy, as the Cartpole environment is  $x\in \mathbb{R}$  unbounded—and a Random policy leads to random drift that significantly moves away from the goal position during the episode length, i.e.,  $|x|\gg 0$ .
I.1.2 Pendulum Environment
We can see in Figure 8, an illustration of the starting state Figure 8 (a) with a small perturbed random initial start. Here a pole (pendulum) is attached to a fixed point at one end with the other end being free [Barto et al., 1983, Yildiz et al., 2021]. The pendulum starts in the downward position Figure 8 (a) and the goal is to swing the pendulum upwards and then balance the pole upright by applying torques about the fixed point, as indicated in the Figure 8 with a visualization showing the torque direction and magnitude based on the size of the arrow. This environment has the state of  $[\theta ,\dot{\theta} ]$  and a corresponding observation of  $[\sin (\theta),\cos (\theta),\dot{\theta} ]$
(a)
Figure 9: Screen shots of the Acrobot environment. The task is to swing up the 2-link pendulum. In the following we see: (a) starting downward state with an additional small amount of perturbation, (b) the optimal trajectory solution found by a policy that scores  $\mathcal{R} \approx 100\%$  including our Active Observing Control policy and (c) the final goal state that has been reached, that is, to swing up the 2-link pendulum and stabilize it upwards. We note that the control actuator is bounded and limited, and the force is such that the 2-link pendulum cannot be directly swung up—rather it must gain momentum through a 2-link swing and then stabilize this swing to not overshoot when stabilizing the 2-link pendulum upwards in the goal position.
Figure 8: Screen shots of the Pendulum environment. The task is to swing up the pole (pendulum). In the following we see: (a) starting downward state with an additional small amount of perturbation, (b) the optimal trajectory solution found by a policy that scores  $\mathcal{R} \approx 100\%$  including our Active Observing Control policy and (c) the final goal state that has been reached, that is, to swing up the pole and stabilize it upwards. We note that the control actuator is bounded and limited, and the force is such that the Pendulum cannot be directly swung up—rather it must gain momentum through a swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position.
(b)
I.1.3 Acrobot Environment
We can see in Figure 9, an illustration of the starting state Figure 9 (a) with a small perturbed random initial start. It is a 2-link pendulum with the individual joints actuated [Brockman et al., 2016]. The 2-link pole starts in the downward position Figure 9 (a) and the goal is to swing the 2-link pendulum upwards and then balance the pole(s) upright by applying torques about their fixed points. This environment has the state of  $[\theta_1,\dot{\theta}_1,\theta_2,\dot{\theta}_2]$  and a corresponding observation of  $[\sin (\theta_1),\cos (\theta_1),\dot{\theta}_1,\sin (\theta_2),\cos (\theta_2),\dot{\theta}_2]$ . Here the Acrobot environment is fully actuated, as no method has been able to solve the underactuated balancing problem [Yildiz et al., 2021, Zhong et al., 2019].
I.2 Cancer Environment
We implemented a Cancer environment, that uses a simulation of a Pharmacokinetic-Pharmacodynamic (PK-PD) (a bio-mathematical model that represents dose-response relationships)
model of lung cancer tumor growth [Geng et al., 2017] under continuous dosing treatment of chemotherapy and radiotherapy. Where the same underlying model has also been used by others [Lim, 2018, Bica et al., 2020, Seedat et al., 2022]. Here, the tumor volume  $V$  at time  $t$  after diagnosis is defined as
$$
\frac {d V (t)}{d t} = \left(\underbrace {\rho \log \left(\frac {K}{V (t)}\right)} _ {\text {T u m o r g r o w t h}} - \underbrace {\beta_ {c} C (t)} _ {\text {C h e m o t h e r a p y}} - \underbrace {\left(\alpha_ {r} d (t) + \beta_ {r} d (t) ^ {2}\right)} _ {\text {R a d i o t h e r a p y}}\right) V (t) \tag {54}
$$
$$
\frac {d C (t)}{d t} = - \frac {C (t)}{2} + c (t) \tag {55}
$$
Where the state space is  $[V(t) \in \mathbb{R}_+, C(t) \in \mathbb{R}_+]$ , with the  $V(t)$  being the cancer volume  $\mathrm{cm}^3$ , and  $C(t)$  being the chemotherapy concentration in the patient. Whereas the action space inputs are  $[c(t) \in [0,5], d(t) \in [0,2]]$ , the chemotherapy input dose  $c(t)$  and radiotherapy dose  $d(t)$ —which are given by the policy. Specifically,  $\rho, K, \alpha_r, \beta_r, \beta_c$  are effect parameters [Geng et al., 2017], and are defined as in Table 9.
Table 9: PK-PD model parameters for the Cancer environment.
Model	Variable	Parameter	Value
Tumor growth	Growth parameter	ρ	1.45 × 10-2
Carrying capacity	K	30.0
Radiotherapy	Radio cell kill (α)	αr	0.0398
Radio cell kill (β)	βr	Set s.t. α/β = 10
Chemotherapy	Chemo cell kill	βc	0.028
As detailed above, the chemotherapy drug concentration state  $C(t)$  follows an exponential decay relationship with a half-life of one day, and  $c(t)$  represents a dose of Vinblastine up to the concentration of  $5.0mg / m^3$  per time in a day. Whereas the radiotherapy concentration  $d(t)$  represents  $2.0G_{y}$  fractions of radiotherapy, where  $G_{y}$  is the Gray ionizing radiation dose. Particularly, we set  $\delta_{a} = .4$  days, and simulate each ODE system at a time resolution of  $\delta_{\mathrm{sim}} = 0.04$  days, up to an episode length of  $T = 25$  days.
The goal state here is to reduce the cancer tumor volume to zero, alongside zero chemotherapy concentration in the patient, i.e.  $q^{*} = [0,0]$ . Moreover, upon restarting the environment the initial state  $s$  is sampled from the uniform distribution of  $s_0 \sim \mathcal{U}[[1120,0],[1138,0]]$ . Here we assume the patient starts with stage four cancer, the largest stage, with a tumor diameter of approximately 13cm diameter, assuming that the tumor is spherical. Furthermore, we use a larger growth parameter  $\rho$  than the nominal one reported by Geng et al. [2017] to model aggressive cancer tumors—which is of high interest.
J Benchmark Method Implementation Details
All benchmark policies consist of an observing policy and an action policy. To ensure competitive benchmarks, each of the following benchmarks are specific ablations of our method, Active Observing Control. Therefore, for a given trained probabilistic dynamics model, the same action policy planner was used across all the benchmark methods, that of the same MPC MPPI planner with the hyperparameters fixed to the below values.
J.1 Probabilistic Dynamics Model
To train our continuous-time and discrete-time probabilistic dynamics model we use a deep ensemble of fully connected multi-layer perceptions [Lakshminarayanan et al., 2017], adapted for multidimensional inputs [Chua et al., 2018]. Specifically, for each individual model in the ensemble, we use a 3-layer multilayer perceptron (MLP) of 256 units, with tanh activation functions. We also use the negative log-likelihood loss to train each model in the ensemble separately—training each model
for the same number of epochs with a different random seed, where the ensemble has  $M = 5$  total models. We use Xavier weight initialization and output the log variance, following the setup by Chua et al. [2018]. All dynamics models are implemented in PyTorch [Paszke et al., 2019a], and trained with an Adam optimizer [Kingma and Ba, 2017] with a learning rate of 1e-4. For each environment we train two dynamics models on the offline dataset, that of a continuous-time probabilistic model using the time delta input  $\delta$ , and a discrete-time probabilistic model—which is the exact same model architecture without the additional time delta input  $\delta$ .
All dynamics models predict the next state difference,  $\Delta z(t + \delta)$ , and we construct the state as  $z(t + \delta) = z(t) + \Delta z(t + \delta)$  following the standard suggestion of [Deisenroth and Rasmussen, 2011].
For training, using the whole collected dataset we pre-process this by a standardization step, to make each dimension of the samples have zero mean and unit variance (by taking away the mean for each dimension and then dividing by the standard deviation for each dimension)—we also use this step during run-time for each dynamics model. Furthermore, we train all the baseline models on all the samples collected in the offline dataset (all samples are training data) Lakshminarayanan et al. [2017]. Specifically, we train all the dynamics models on a given offline dataset by training each model until convergence, each for at least 12 hours.
J.2 Observing Policies
We implement two discrete-time methods. First by learning a discrete-time uncertainty-aware dynamics model (an ablation of the exact same model and hyperparameters, without the time interval input  $\delta$  to predict the next state for) on the same offline dataset  $\mathcal{D}$ . Second, we use this discrete-time uncertainty-aware dynamics model to create two benchmark methods, that of Discrete Planning that samples the state at each action time interval  $\delta_{a}$  and Discrete Monitoring a discrete ablation of our observing policy (Algorithm 3)—that uses the same reward distribution estimation and determines the discrete action sequence to execute up until the reward uncertainty crosses the threshold  $\tau$ —at a discrete-time resolution of  $\delta_{a}$ .
Moreover, we also benchmark against Continuous Planning that uses our trained continuous-time uncertainty-aware dynamics model and takes a sample of the state at regular time intervals of a multiple of  $\delta_{a}$ . Finally, we also compare with a random action policy, Random that samples the state at each action interval  $\delta_{a}$ .
For each policy method, we tune the hyperparameter  $\tau$  is one exists following the procedure outlined in Section 4.2. We then keep this fixed and constant throughout all run-time experiments for each policy method for each environment—unless explicitly noted that we modify it, as in Section 5.3. For all policies we limit the minimum  $\Delta_{i+1}$  to be  $\delta_a$  (as actions take time to execute and interact with the environment, therefore we want to avoid observing the same state again immediately, for fast-moving states), and the maximum  $H$  the MPC planned action horizon.
J.3 MPPI Implementation
We use the MPPI algorithm, with pseudocode and is further described in Appendix E. Specifically, as is recommended by Lutter et al. [2021] we optimized the MPPI hyperparameters through a grid search with the continuous-time probabilistic dynamics model for a single environment setting, that of the Cartpole environment, and fix these for planning with all the learned dynamics models throughout. Particularly, our final optimized hyperparameter combination is  $N = 20$ ,  $M = 1,000$ ,  $\lambda = 0.01$ ,  $\sigma = 1.0$ . Where  $\Sigma$  the MPPI action noise is defined as:
$$
\Sigma = \left\{ \begin{array}{l l} {[ \sigma^ {2} ]} & {\text {i f} d _ {\mathcal {A}} = 1} \\ {\left[ \begin{array}{c c} \sigma^ {2} & 0. 5 \sigma^ {2} \\ 0. 5 \sigma^ {2} & \sigma^ {2} \end{array} \right]} & {\text {i f} d _ {\mathcal {A}} = 2} \end{array} \right. \tag {56}
$$
Where the Cartpole and Pendulum environments have  $d_{\mathcal{A}} = 1$  and Acrobot and Cancer environments has  $d_{\mathcal{A}} = 2$ . These hyperparameters were found by searching over a grid of possible values, which is detailed in Table 10.
Table 10: MPPI hyperparameter grid search sweep values.
Hyperparameter	Grid values searched over
K	{1,2,4,8,16,20,40,50,60,70,80,90,100,128,256,512,1024,2048}
G	{1,2,4,8,16,20,40,50,60,70,80,90,100,128,256,500,1000,2000,4000,8000}
λ	{0.00001,0.0001,0.001,0.01,0.1,0.5,0.8,1.0,1.5,2.0,10.0,100.0,1000.0}
σ	{0.00001,0.0001,0.001,0.01,0.1,0.5,0.8,1.0,1.5,2.0,10.0,100.0,1000.0}
K Evaluation Metrics
An uncertainty-aware dynamics model is trained for each environment on an offline dataset  $\mathcal{D}$  of collected trajectories consisting of 1e6 samples. The trained dynamics model weights are then frozen when used at run-time. We record the average undiscounted utility  $\mathcal{U}$  (Equation (1)), average undiscounted reward  $\mathcal{R}$  and the number of observations taken  $\mathcal{O}$ , after running one episode of the benchmark method—and repeat this for 100 random seed runs for each result with their  $95\%$  confidence intervals throughout. We use a fixed observation cost of  $c = 50$  throughout. Moreover, we normalize  $\mathcal{R}$  and  $\mathcal{U}$  following the standard normalization of offline-RL [Yu et al., 2020]—normalized to the interval of 0 to 100, where a score of 0 corresponds to a random policy performance, and 100 to an expert—which we assume here is the Continuous Planning benchmark. Furthermore, we also track the metric of total planning time taken to plan the next action and time to schedule a sample and perform all experiments using a single Intel Core i9-12900K CPU @ 3.20GHz, 64GB RAM with a Nvidia RTX3090 GPU 24GB.
L Additional Experiments
L.1 Ablation of Thresholding the State Uncertainty
Intuitively we threshold the variance of the value function rather than the variance of the state as it is possible to achieve a task with a very certain value despite having uncertain states. For example, when there exist multiple ways to achieve a task—and we know that multiple action trajectories guarantee this where we can take any; however, we are uncertain about which one.
We performed an additional ablation experiment where we threshold the variance of state instead and compare the utility against Active Observing Control that thresholds the variance of the predicted reward instead.
Table 11: Ablation experiment of using a threshold on the variance of state instead of the predicted reward, for the Cancer environment.
Policy	Utility U
Active Observing Control (Reward Uncertainty Threshold)	106±0.789
Active Observing Control (State Uncertainty Threshold)	103±1.6
For each policy, we tuned the threshold following the tuning setup, in Section 4.2. Here we evaluated the utility metric over 10 random seeds for each policy on the Cancer environment. As tabulated in Table 11, it is possible to threshold on the state variance for an acceptable performing policy, however, it is preferable and more intuitive to threshold the variance of the predicted reward uncertainty.
Furthermore, we performed an additional ablation experiment where we threshold the variance of the state instead and compare this to the utility of Active Observing Control that thresholds the variance of the predicted reward instead, on the Cartpole environment, as tabulated in Table 12.
We again tuned each observing policy's threshold following the tuning setup in Section 4.2. Here we evaluated the utility metric over 100 random seeds. Empirically we observe that is indeed possible to threshold the state variance for an acceptable performing observing policy. However, it is again preferable and more intuitive to threshold the variance of the predicted reward uncertainty.
We also highlight the Cartpole environment is notably more complicated than the Cancer environment, as it is an underactuated system, as it has two degrees of freedom  $[x,\theta ]$ , however only the carts
Table 12: Ablation experiment of using a threshold on the variance of state instead of the predicted reward, for the Cartpole environment.
Policy	UtilityU
Active Observing Control (Reward Uncertainty Threshold)	144±5.51
Active Observing Control (State Uncertainty Threshold)	98.4±2.64
position is actuated, leaving  $\theta$  indirectly controlled. We note that the control actuator is bounded and limited, and the force is such that the Cartpole cannot be directly swung up—rather it must gain momentum through a swing and then stabilize this swing to not overshoot when stabilizing the pole upwards in the goal position, as indicated when the tip of the pole reaches the center of the red target square.
L.2 Sparse Reward Environment
To investigate how well Active Observing Control generalizes to other environments that have possibly sparse reward functions, we performed an additional sparse reward experiment. We implemented a sparse Pendulum environment [Chakraborty et al., 2022] which only provides a reward when the Pendulum state angle  $\theta$  is within  $\pm \theta_{\mathrm{CutOff}}$  degrees from the goal state upright ( $\theta^{*} = 0$ ). That is using a sparse reward function defined by  $r'(s,a,t) = r(s,a,t)\cdot \mathbf{1}\{| \theta | < \theta_{\mathrm{CutOff}}\}$ . This allows us to vary the degree of sparsity by decreasing  $\theta_{\mathrm{CutOff}}$  to create a sparser reward for the Pendulum environment. This is tabulated in Table 13, with each result averaged over 100 random seeds, and we normalize scores to 100 for Continuous Planning with  $\theta_{\mathrm{CutOff}} = 180^{\circ}$ .
Table 13: Sparse reward Pendulum environment.
θCutOff	Policy	UtilityU	RewardR	ObservationsO
θCutOff = 180°	Random	0±0	0±0	13±0
θCutOff = 180°	Continuous Planning	100±9.48	100±9.48	50±0
θCutOff = 60°	Continuous Planning	18.8±17.7	18.8±17.7	50±0
θCutOff = 5°	Continuous Planning	-7.81±0.00125	-7.81±0.00125	50±0
θCutOff = 180°	Active Observing Control	109±13.8	98.5±10.2	48±0.954
θCutOff = 60°	Active Observing Control	86.7±13.5	12±16.5	35.4±4.08
θCutOff = 5°	Active Observing Control	223±2.32	-7.81±0.00146	4.8±0.452
We note that Active Observing Control relies on MPC being feasible in the environment and that it is possible to solve the environment with the chosen MPC decision policy  $\pi$  with regular observing at each time step. In our experimental setup, this translates that the baseline of Continuous Planning should be able to solve the environment to achieve an optimal reward (i.e.,  $\mathcal{R} = 100$ ).
The above results demonstrate the well-known [Karnchanachari et al., 2020] limitation of MPC in general, that a vanilla MPC decision policy  $\pi$  struggles in environments with sparse rewards. We highlight that this is not unique to MPC, and standard RL methods in general significantly struggle with sparse rewards in continuous-state and continuous-action tasks [Chakraborty et al., 2022] without additional assumptions or data (reward shaping or using expert demonstrations). This arises due to the limited recoding planning horizon time  $H$ , however, there exists works to combine trajectory optimization (MPC) with learning a reward estimation [Karnchanachari et al., 2020, Hoeller et al., 2020]. This is an interesting direction for future work, though we believe this is out of scope for this paper.
L.3 Increasing the Planning Resolution of MPC (decreasing  $\delta_{a}$ )
We note that planning with a higher resolution MPC planner (smaller  $\delta_{a}$ ) for all the baselines will make the baselines perform on par with Active Observing Control for that  $\delta_{a}$ . However, our motivation is to compare the observing policies  $\rho$  of each baseline, therefore in all the implemented baselines we use the same decision policy  $\pi$  of the same MPC algorithm with the exact same hyperparameters.
We performed an additional re-run of the baselines on the Cancer environment, with varying  $\delta_{a} = \{0.1, 0.05, 0.02, 0.01\}$  and each result averaged over 100 random seeds, as tabulated in Table 14. We
kept the MPC planning horizon time  $H$  fixed, to the same value as in our original experiments, and let the number of planning time steps  $K$  vary, i.e.,  $K = \lceil \frac{H}{\delta_a} \rceil$ .
Table 14: Increasing the planning resolution of MPC (decreasing  ${\delta }_{a}$  ) for the Cancer environment.
Discretization interval δa	Policy	Utility U	Reward R	Observations O
0.1	Random	0±0	0±0	13±0
0.1	Discrete Planning	92.2±1.11	92.2±1.11	13±0
0.1	Discrete Monitoring	91.1±1.72	85.9±1.7	4.94±0.109
0.1	Continuous Planning	100±0.597	100±0.597	13±0
0.1	Active Observing Control	105±0.75	98.4±0.707	3.42±0.102
0.05	Random	0±0	0±0	13±0
0.05	Discrete Planning	96±1.54	95.6±1.68	13±0
0.05	Discrete Monitoring	96.3±1.39	91.6±1.47	7.18±0.153
0.05	Continuous Planning	100±1.36	100±1.48	13±0
0.05	Active Observing Control	104±1.29	98.1±1.33	4.95±0.118
0.02	Random	0±0	0±0	13±0
0.02	Discrete Planning	95±1.25	92.8±1.81	13±0
0.02	Discrete Monitoring	95.9±1.17	88.2±1.7	6.5±0.196
0.02	Continuous Planning	100±1.17	100±1.7	13±0
0.02	Active Observing Control	98.6±0.967	88.8±1.43	2.77±0.0839
0.01	Random	0±0	0±0	13±0
0.01	Discrete Planning	102±0.806	105±1.87	13±0
0.01	Discrete Monitoring	102±0.844	98±1.93	5.89±0.0741
0.01	Continuous Planning	100±1.03	100±2.39	13±0
0.01	Active Observing Control	103±0.937	97.8±2.06	4.79±0.117
Where for each  $\delta_{a}$  we normalize utility to be between 100 and 0, where 100 corresponds to Continuous planning at that  $\delta_{a}$  and 0 a random policy. We note that  $\delta_{a} = 0.1$  was used, as this is the default value for the ODE-RL environments [Yildiz et al., 2021].
Moreover, in light of the empirical results (Table 14), we can also ask what the performance looks like if we only normalize utility to Continuous planning with  $\delta_{a} = 0.1$  (the best possible reward) for each baseline whilst varying  $\delta_{a}$ , as tabulated in Table 15.