SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We experimentally evaluate our models across several PO regular/irregular observation domains. In regular observation domains, we consider conventional partially observable control and meta-RL tasks by employing MuJoCo [Todorov et al., 2012] and PyBullet [Greff et al., 2022] environments. In irregular observation domains, we consider modifying classical environments: Pendulum and CartPole tasks[Brockman et al., 2016] into irregular time intervals with surrogate rewards.
4.1 Partially Observable Classic Control Tasks
The Pendulum and CartPole tasks are classic control tasks for evaluating RL algorithms. For the Pendulum task, the goal is to learn a policy to swing the pendulum up and maintain it at the highest position to obtain more rewards. For the CartPole task, the goal is to learn a policy to preclude the pole from falling down and forestall the cart from running away by exerting force on the cart.
Figure 2: Classic partial observable control tasks. The shaded region represents a standard deviation of average evaluation over five runs and Curves are smoothed for visual clarity.
The observable information consists of the coordinates of the cart, the angle of the pole, and their velocities.
These classic control tasks are relatively easy to solve in fully observable domains. Therefore, these PO tasks can effectively underscore the problem of representation learning. Experiments are performed in PO cases of these two control tasks, in which only velocities or positions could be observed (we use the suffix -V/-P to name). The PO setting is more practically significant, as in many real-world applications, agents may only be able to estimate partial state information.
We compare our method with SLAC, VRM, and SAC-LSTM. SLAC [Lee et al., 2020] combines off-policy model-free RL with representation learning via a sequential stochastic state space model. SLAC was developed for pixel observations, we follow the modifications in Han et al. [2019] to compare it with our method. VRM [Han et al., 2019] is a recent, model-based POMDP algorithm, which uses variational recurrent models to encode context variables and separate representation learning from dynamic programming. In the SAC-LSTM, soft actor-critic algorithms with recurrent networks are used as function approximators. We follow the settings in [Han et al., 2019] to construct the above methods.
As expected, our method succeeds in learning to solve all these tasks and performs better than VRM which is the state-of-the-art method for solving POMDP tasks. While SLAC performs well in the CartPole tasks and less sample efficiency in Pendulum tasks. SAC-LSTM needs more epochs for training and fails to solve the Pendulum-V task. We reckon these results are attributed to the ability of ODEs to model physical systems and infer unknown information.
To assess the capability of our model to deduce unobservable environmental information, we project the latent variable  $z_{t}$  in an episode to 2D using UMAP [McInnes et al., 2018]. We estimate the unobservable state information such as the angular velocities in the Pendulum-P task through observations, actions, and rewards (as the rewards are calculated based on state and actions, the unobservable state could be computed in reverse), and scatter plot color based on the logarithmic scaling
(a)
Figure 3: The visualization of the latent variables  $z_{t}$  in one episode after dimension reduction. Points are colored according to (a) the logarithm of the calculated angular velocity and (b) the logarithm of the calculated angular position (angle difference from the initial position). Each figure consists of 201 points including the initialized Gaussian  $z_{0}$ , as the maximum step in the Pendulum task is 200.
Figure 4: Learning curve of the partially observable PyBullet robotic control tasks, plotted in the same way as in Fig. 2. The PyBullet environments are ported from deprecated Roboschool environments which are harder than the MuJoCo Gym environments<sup>1</sup>.
of their values. Fig. 3 shows the clustering of latent variables corresponds closely to the unobservable physical parameters, which demonstrates the efficacy of our GRU-ODE context encoder.
4.2 Partially Observable Robotic Control Tasks
We evaluate the performance of our proposed method in the PyBullet control environments [Greff et al., 2022] which are more challenging than MuJoCo. The PyBullet environments contain several continuous robotic control tasks and the state of fully observable environments includes the robot's coordinates, trigonometric functions of joint angles, and angular velocities. We modify the observations into PO versions with the same criteria and compare them with the same methods as the PO classic control tasks. As experimental results demonstrated our method obtains policy improvement in the majority of PO robotic control tasks, especially in environments where only the position information is observable. For tasks with only positions observable, our method outperforms the other three methods and presumably shows efficiency in extracting useful information from historical position observations. We reckon that velocity could be simply estimated by one-step differentiation in coordinates and joint angles of the robot, which eases representation and policy learning. Environments with only velocities that could be observed are much harder to solve. However, deducing the position information solely from the given velocity is not practical. In environments where only velocity can be observed, our method performs similarly to VRM. Additionally, our results indicate that the learning processes of the SLAC method exhibited instability, where it intermittently achieved a nearly optimal policy, but frequently converged to a suboptimal one. As a result, the average performance of SLAC was generally less favorable than that of our approach across the majority of PO robotic control tasks. SAC-LSTM fails to solve almost all these tasks.
We further compare our methods with two more baselines RMF (Recurrent Model-Free)[Ni et al., 2022] and TD3-FPOW (TD3 with fixed previous observation window). The complete results are shown in Table 6.
4.3 Meta-learning MuJoCo Control Tasks
We further show our method can also scale to other types of PO tasks, such as Meta-RL, by applying our method to a diverse set of domains, such as point robot and MuJuCo locomotion tasks, which are commonly used in meta-RL literature. Our tasks involve utilizing the semi-circle, which refers to a point robot that navigates along a semi-circular path in search of a sparsely located reward, as well as wind tasks, where a point robot navigates to a fixed goal amidst fluctuating wind conditions. For MuJuCo locomotion tasks, we consider using the HalfCheetah-Dir environment where the agent has
<https://github.com/openai/roboschool#deprecated-please-use-pybullet-instead>
Figure 5: Learning curves with compared methods on meta-RL tasks. Unlike other methods, we only plot the final return of Oracle and Pearl methods in the semi-circle and wind tasks. The learning curve data of on-policy VariBAD, Oracle, and RL2 in the HalfCheetah-Dir task are copied from the public GitHub repository of variBAD<sup>2</sup>.
to control two legs run either forward or backward. The MuJuCo-relevant tasks are much harder than the prior point robot tasks.
We compare our method with various methods, such as VariBAD, RL2 [Duan et al., 2016], Pearl [Rakelly et al., 2019], and Oracle policy. VariBAD utilizes a variational, model-based objective to learn task embeddings explicitly. The originally proposed variBAD [Zintgraf et al., 2019] uses PPO [Schulman et al., 2017], and the same method was implemented with SAC [Dorfman et al., 2020]. These two methods could be named on-policy variBAD and off-policy variBAD. An oracle policy can access the POMDP hidden state to transform the POMDP into an MDP, thus this policy could be treated as an upper bound on the performance of related methods. We follow the settings in Ni et al. [2022] to construct the above methods.
Fig. 5 shows that our method outperforms off-policy variBAD and Pearl in the semi-circle and wind environments, and on-policy variBAD and RL2 in the HalfCheetah-Dir environment, both in terms of sample efficiency and asymptotic return. As our model is trained end-to-end, without using pre-trained encoders to represent task contexts like off-policy variBAD, which do not suffer out-of-date representation problems. On-policy variBAD uses autoencoder architecture to reconstruct the state and reward, which are more advantageous to context representation. Our method shows simple recurrent encoder could also stabilize representation learning and improve policy performance.
4.4 Temporally Irregular Observable Control Tasks
In standard RL frameworks such as OpenAI Gym, the states/observations arrive at constant intervals as  $\Delta t = \delta$ . To contrast the robustness of our continuous-time method with other discrete POMDP methods, we evaluate related methods with irregularly sampled observations. We consider using an irregular sampling scenario in which observations arrive uniformly within a range  $\Delta t \sim \mathcal{U}(0,2\delta]$ . We choose the mean time difference  $\delta = 0.05$  for observation spacings, which is the standard interval setting of the Gym Pendulum task.
Fig. 6 exhibits the final average returns with a standard deviation of VRM, SLAC, and our method
in irregular Pendulum and CartPole tasks. The results show our ODE-based recurrent model is more robust with stochastic time increments, which is reasonable as ODEs build discrete-time sequences into continuous ones. This environment has more practical significance, as observation intervals are usually disturbed in practical application.
Figure 6: A comparison of PODMP methods with uniformly irregular observations.
$^{2}$ https://github.com/lmzintgraf/varibad
Although we have demonstrated that ODE-based encoders have advantages in robotic control tasks, their efficiency in discrete game problems, such as Atari, is not promising. In Atari, the states of the agents are not updated based on numerical differential equations, resulting in the inability to represent the state and action as integration processes. Also, the reward function is typically sparse and predefined. Implementing ODE-based encoders to model related processes does not make sense.
In addition, since the integration process is implemented by neural networks, the training procedure of ODE-based encoders requires more computation resources, which results in increased training time, especially for high-order numerical differential solvers. Although we find that using simple numerical solvers is enough in the majority of tasks (see Appendix Section 10), accelerating the integration process is still significant in solving differential equations and downstream tasks.
6 Conclusion and future works
We incorporate ODE-based recurrent neural networks with model-free reinforcement learning to solve POMDP control tasks. Through our empirical assessment across several partially observable domains, we showed that the policy network and value function conditioned on context variables encoded by GRU-ODE help the agents infer unknown observations and maximize expected returns. We experimentally demonstrate that our method is robust to environmental changes such as irregularity.
Reinforcement learning has been widely used in physical or biological systems to explain some of the mechanisms. However, some mechanisms in the brain are still unknown, such as how the brain arbitrates or allocates control over various sub-systems and the role of the prefrontal cortex and striatum in controlling. The classic leaky integrate and fire (LIF) neuron model is quite similar to ODEs if we ignore the firing process. We reckon combining ODEs with RL could help biologists explore mechanisms in the human brain.
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA27010404), the Beijing Nova Program (Grant No. 20230484369), the Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX), and the Youth Innovation Promotion Association of Chinese Academy of Sciences.
S. Ainsworth, K. Lowrey, J. Thickstun, Z. Harchaoui, and S. Srinivasa. Faster policy learning with continuous-time gradients. In Learning for Dynamics and Control, pages 1054-1067. PMLR, 2021.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
S. Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254, 2017.
S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informed neural networks: where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
E. De Brouwer, J. Simm, A. Arany, and Y. Moreau. Gru-ode-bayes: Continuous modeling of sporadically-observed time series. In Advances in Neural Information Processing Systems, pages 7377-7388, 2019.
R. Dorfman, I. Shenfeld, and A. Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02598, 2020.
J. Du, J. Futoma, and F. Doshi-Velez. Model-based reinforcement learning for semi-markov decision processes with neural odes. Advances in Neural Information Processing Systems, 33:19805-19816, 2020.
In this supplementary material, we provide further information about our implementation details and ablation studies.
7 Truncation Error
The update procedure of the Euler Equation
$$
y _ {i + 1} = y _ {i} + h f \left(x _ {i}, y _ {i}\right), \tag {10}
$$
and the accurate update equation is
$$
y _ {i + 1} = y _ {i} + h f \left(x _ {i}, y \left(x _ {i}\right)\right) + \frac {h ^ {2}}{2} f ^ {\prime \prime} \left(\xi_ {i}\right), \quad \xi_ {i} \in \left(x _ {i}, x _ {i + 1}\right). \tag {11}
$$
The lost item  $\frac{h^2}{2} f''(\xi_i)$  is the local truncation error of the Euler method. The global truncation error is defined as
$$
\begin{array}{l} e _ {i + 1} = y (x _ {i + 1}) - y (i + 1) \\ = \left(y _ {i} + h f \left(x _ {i}, y \left(x _ {i}\right)\right) + \frac {h ^ {2}}{2} f ^ {\prime \prime} \left(\xi_ {i}\right)\right) - \left(y _ {i} + h f \left(x _ {i}, y _ {i}\right)\right) \\ = e _ {i} + h \left(f \left(x _ {i}, y \left(x _ {i}\right)\right) - f \left(x _ {i}, y _ {i}\right)\right) - \frac {h ^ {2}}{2} f ^ {\prime \prime} (\xi_ {i}) \\ \end{array}
$$
In Euler method  $f(x,y)$  satisfy the Lipschitz condition for  $y$  and denote  $T_{i + 1} = -\frac{h^2}{2} f''(\xi_i)$  is the local truncation error and  $T = \max_{i}|T_{i}| = O(h^{2})$
$$
\begin{array}{l} \left| e _ {i + 1} \right| \leq \left| e _ {i} \right| + h \left| f \left(x _ {i}, y \left(x _ {i}\right)\right) - f \left(x _ {i}, y _ {i}\right) \right| + \left| T _ {i + 1} \right| \\ \leq \left| e _ {i} \right| + h \left| y \left(x _ {i}\right) - y _ {i} \right| + \left| T _ {i + 1} \right| \\ \leq \left| e _ {i} \right| (1 + h L) + T \\ \leq (1 + h L) ^ {i + 1} \left(\left| e _ {0} \right| + \frac {T}{h L}\right) \\ \leq e ^ {(i + 1) h L} \left(\left| e _ {0} \right| + \frac {T}{h L}\right) \\ = O (h) \\ \end{array}
$$
where  $e^0 = y(x - 0) - y_0 = 0$
8 Environment Specification
We incorporate the occlusion benchmark proposed by VRM and replace the deprecated roboschool with PyBullet, as the official GitHub repository suggests. We transform the original MDP task into a POMDP version by removing all position/angle-related entries in the observation space for "-V" environments and velocity-related entries for "-P" environments.
{Pendulum, CartPole, AntBLT, WalkerBLT, HopperBLT}-P. The "-P" denotes environments that retain position-related observations by eliminating velocity-related observations.
{Pendulum, CartPole, AntBLT, WalkerBLT, HopperBLT}-V. The "-V" denotes environments that retain velocity-related observations by eliminating position-related observations.
9 Implementetaion Details
We use the PyTorch framework for our experiments. Some basic hyperparameters about the network architectures are listed below
Also, there are some unique parameters in TD3 and SAC, we report them together
Table 1: Environment information
Environment name	dim(o)	dim(a)	Maximum step
Pendulum-P(position only)	1	1	200
Pendulum-V(velocity only)	1	1	200
CartPole-P(position only)	2	1	1000
CartPole-V(velocity only)	2	1	1000
AntBLT-P(position only)	17	8	1000
AntBLT-V(velocity only)	11	8	1000
WalkerBLT-P(position only)	13	6	1000
WalkerBLT-V(velocity only)	9	6	1000
HopperBLT-P(position only)	9	3	1000
HopperrBLT-V(velocity only)	6	3	1000
HalfCheetah-Dir	17	6	200
Table 2: Hyperparameters
Hyperparameters	Description	Value
γ	Discount factor	0.99
lr	Learning rate	0.0003
λ	Balance weight between KL divergence and RL loss	0.5
τ	Fraction of updating the target network each gradient step	0.005
dqn_layers	MLP layer sizes of value function	[256,256]
policy_layers	MLP layer sizes of policy network	[256,256]
sampled_seq_len	The number of steps in a sampled sequence for each update	64
batch_size	The number of sequences to sample for each update	64
buffer_size	The number of saved transitions	1e6
action_embedding_size	Embedding dimension of action	16
observ_embedding_size	Embedding dimension of observation	32
reward_embedding_size	Embedding dimension of reward	16
gru Hidden_size	Hidden layer size of recurrent encoder	128
9.1 Encoding procedure of GRU-ODE
The input observations  $x_{t}$  is the concatenation of observations, actions and rewards, as we mentioned above. The time increment  $dt$  is set to a fixed value in regular observation environments. In this paper, we set  $dt = 0.1$  for all POMDP tasks.
9.2 Integrating with TD3
We use separate recurrent encoders for the actor and critic in order to improve the performance of our method. Owing to the policy network of TD3 updating less frequently than the value function. We modify the iteration equation of the policy network to:
$$
\mathcal {L} \left(\phi_ {a}, \theta , \psi\right) = - \underset {o \sim \mathcal {D}} {\mathbb {E}} \left[ q _ {\psi_ {1}} \left(o, \pi_ {\theta} (o, z), z\right) \right] + \lambda_ {a} \mathcal {K} \left(\phi_ {a}\right), \tag {12}
$$
Algorithm 1 The GRU-ODE algorithm.
Input: Observations and time difference between observations  $(x_{t},dt)_{t = 1..T}$ $h_0 = 0$    
for t in 1,2,...,T do   
 $\tilde{h}_t = \mathrm{GRUCell}(h_{t - 1},x_t)$  {Update hidden state}   
 $h_t = \mathrm{ODESolve}\left(f_\theta ,\tilde{h}_t,dt\right)$  {Solve ODE}   
 $z_{t} = \mathrm{MLP}(h_{t})$  for all  $t = 1..T$    
Return:  $\{z_t\}_{t = 1..T};h_t$
Table 3: Hyperparameters of SAC and TD3
Method	Hyperparameters	Value
SAC	entropy_alpha	0.2
SAC	automatic_entropy_tuning	True
SAC	alpha_lr	0.0003
TD3	exploration_noise	0.1
TD3	target_noise	0.2
TD3	target_noise Clip	0.5
$\mathcal{D}$  denotes the replay buffer. The state  $s$  is replaced with observation  $o$  owing to the partially observable environments.
The update of the value function is modified to:
$$
y (r, o ^ {\prime}, z ^ {\prime}) = r + \gamma \min _ {i = 1, 2} Q _ {\psi_ {i}, t a r g} (o ^ {\prime}, \pi_ {\theta , t a r g} (o ^ {\prime}), z ^ {\prime}),
$$
$$
\mathcal {L} \left(\psi_ {i}\right) = \underset {\left(o, a, r, o ^ {\prime}\right) \sim \mathcal {D}} {\mathbb {E}} \left[ \left(Q _ {\psi_ {i}} \left(o, a, z\right) - y \left(r, o ^ {\prime}, z ^ {\prime}\right)\right) ^ {2} \right], \tag {13}
$$
$$
\mathcal {L} (\phi_ {c}, \psi) = \mathcal {L} (\psi_ {1}) + \mathcal {L} (\psi_ {2}) + \lambda_ {c} \mathcal {K} (\phi_ {c}),
$$
where subscript 1, 2 denotes two different critic networks implemented in TD3 and  $\text{targ}$  denotes the target network. The context variable is computed by  $g_{\phi}(z_t|\tau ;t)$  as the policy network and value function update, we omit this procedure to simplify the equation.
9.3 Integrating with SAC
Similar to TD3, we implement our GRU-ODE in SAC. The training loss of the policy network is modified to
$$
\mathcal {L} \left(\phi_ {a}, \theta , \psi\right) = - \underset {o \sim \mathcal {D}} {\mathbb {E}} \left[ \min  _ {j = 1, 2} q _ {\psi_ {j}} (o, \pi_ {\theta} (o, z), z) - \alpha \log \pi_ {\theta} (a | o) \right] + \lambda_ {a} \mathcal {K} \left(\phi_ {a}\right), \tag {14}
$$
and the value function becomes
$$
y (r, o ^ {\prime}, z ^ {\prime}) = r + \gamma \min _ {j = 1, 2} (Q _ {\psi_ {j}, t a r g} (o ^ {\prime}, \pi_ {\theta , t a r g} (o ^ {\prime}), z ^ {\prime}) - \alpha \log \pi_ {\theta} (\tilde {a} | o ^ {\prime})), \tilde {a} \sim \pi_ {\theta} (\cdot | o ^ {\prime}),
$$
$$
\mathcal {L} \left(\psi_ {i}\right) = \underset {\left(o, a, r, o ^ {\prime}\right) \sim \mathcal {D}} {\mathbb {E}} \left[ \left(Q _ {\psi_ {i}} \left(o, a, z\right) - y \left(r, o ^ {\prime}, z ^ {\prime}\right)\right) ^ {2} \right], \tag {15}
$$
$$
\mathcal {L} \left(\phi_ {c}, \psi\right) = \mathcal {L} \left(\psi_ {1}\right) + \mathcal {L} \left(\psi_ {2}\right) + \lambda_ {c} \mathcal {K} \left(\phi_ {c}\right),
$$
where symbols have the same meaning as in Eq. 13. The parameter  $\lambda$  is set to 0.5 both in TD3 and SAC algorithms.
10 ODE Solver Comparison
In this ablation study, we ask two questions in relation to numerical integration. (i) how different numerical solvers influence the final performance of POMDP tasks, and (ii) the effect on the training runtime hours. We experimentally evaluate using different numerical methods as ODESolve for integrating  $\tilde{h}_t$  in GRU-ODE. From Table 4, we find that there is no obvious relationship between the
Table 4: Final performance of different numerical integration methods
Ant-P	Ant-V	Walker-P	Walker-V
Euler	1243 ± 43	998 ± 93	1487 ± 81	612 ± 27
RK4	1183 ± 67	1045 ± 112	1405 ± 64	683 ± 45
Heun	1123 ± 53	1023 ± 69	1457 ± 45	625 ± 41
complexity of the numerical solvers and the final performance of the PODMP tasks. However, the
RK4 and Heun need much more time for training. We train these methods on a server with NVIDIA TITAN Xp and Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz as GPU and CPU respectively. The following are rough estimates of average run times for the AntBLT-P environments.
- Ours (with Euler) 12 hours
- Ours (with RK4) 20 hours
- Ours (with Heun) 17 hours
As the results show for context variable coding, the simplest numerical Euler method can work well as ODESolve and save training time. We speculate that though the environment is complicated to solve, the context variables always reflect the essential dynamic information of the environments which are more refined than the observations. Thus, simple numerical solvers are enough.
11 Time Cost of various baselines
We evaluate the time costs of different baselines on Walker-P environments. From the results, we find our method does not increase the computation a lot compared to other baselines. We analyze that there are two reasons. (I). Within our implementation of ODESolve(), we employ the "Euler" method for computation. This offers substantial reductions in computation time when contrasted with the default "dopri5" approach (Runge-Kutta of order 5 of Dormand-Prince-Shampine) utilized in ODE-RNN. We observe that employing complex ODESolve() methods does not notably enhance performance. Instead, it increases computational time significantly. (II). We are working within the context of RL tasks. When training RL algorithms, it's not just the ODE-GRU that needs training, other networks such as policy networks and value networks also require training. During the training process, a significant amount of time is consumed by sampling transitions from the data buffer and updating the data buffer. Therefore, in terms of the overall training time of our approach, it doesn't significantly exceed the time required by the previous method that used an RNN as an encoder.
Table 5: Time Cost of different methods
Methods	Time Cost
Ours	12h
RMF	10h
VRM	18h
SAC-LSTM	6h
SLAC	3h
12 Further Performance Comparison