ODE-based Recurrent Model-free Reinforcement Learning for POMDPs
Authors: Xuanle Zhao
Duzhen Zhang
Liyuan Han
Tielin Zhang
Bo XU
Venue: NeurIPS 2023 poster (2023)
Keywords: neural ode
POMDPs
reinforcement learning
TL;DR: We propose a novel ODE-based recurrent model combine with model-free RL framework to solve POMDP problems.

Abstract: Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combines with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs
Xuanle Zhao! Duzhen Z Liyuan Han Tielin Zhangh Xu
Institute of Automation, Chinese Academy of Sciences, Beijing, China School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai, China
(zhaoxuanle2022, zhangduzhen2019, hanliyuan2019, tielin.zhang, xubo] @ia.ac.cn
Abstract
Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how infer unseen information from observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-relater infor- mation, present novel ODE-based recurrent model combined with model-free reinforcement learning (RL) framework to solve partially observable Markov de- cision processes (POMDPs). We experimentally demonstrate the efficacy of methods across various PO continuous control and meta-RI. tasks. Furthermore, experiments illustrate that method is robust against irregular observations, owing to the ability of ODEs to model rregularly-samples time series.
Introduction
Conventional reinforcement learning (RL) is typically cast as problem of solving fully observable Markov decision process (MDP) tasks which are trained and tested on the same task. However, most practical applications require the agents to handle some degree of partially observable and irregular observations. Humans are always good at solving these kinds of tasks by extracting crucial information from past observations and actions, while conventional RL agents do not have the ability to extract information relevant to the tasks.
In recent works, several categories of methods have been proposed to solve PO problems. The most straightforward one include all historical observations input (Lee et al. 2020). However, this kind of method is impractical in application owing to the dimension increment of the observation inputs. Another category based on model-free methods which generally use recurrent neural networks (RNNs) as memory units and function approximators to extract historical task-related information solve partial observable problems [Igl al. 2018 Kapturowski al. 2019 Han et Daderberg et al. [019]. Recurrent neural networks are commonly employed to encode historical state transitions into contextual representations, leveraging their capacity process sequential information. These representations are then used in conjunction with the current state to find the optimal policy. This baseline is simple and efficient, as it obtains near-optimal results and only requires changing the policy and value network into recurrent version. The third category considers using a model to estimate a belief state from historical transitions [Lee et al. 2020 Agent receives
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

belief states as observations that contain useful information to solve the tasks, this category is much similar to the second one. However, almost all notable breakthroughs have been showcased in discrete-time problems such as games and discrete control problems [Miih et al. 2015 Silver et al. 2016], whereas most physical and biological systems in the real world are inherently continuous in time and follow differential equation dynamics.
Differential equations are commonly considered the benchmark for modeling physical mechanisms [Schölkopt et al. 2021]. They permit people to forecast the future behaviors of physical systems and the statistical interdependencies among variables. Moreover, they provide physical insights. explain the system's 's functioning, and justify reasons for their causal structure [Peters et al. 2022 Zhang et al.] 2023b). A discretized control system will converge to an ODE as the time increment approaches zero. However, there is no consensus on how to choose the right discretization stepsizes and account for irregular observation times. In current paradigms, stepsizes are fixed advance and vary in environments. For example, in classic control tasks (such as CartPole and Pendulum), observations are sampled by regularly discretized timesteps [Brockman et al. 2016].
In this study, similar to context-based RL, we developed an ODE-based recurrent model (GRU-ODE) that encodes historical observations, actions, and rewards into a latent variable. Previous works have shown that ODE-based recurrent models could simulate the physical dynamic system in an auto-regressive fashion, such as Latent-ODE [Rubanova et al. 2019 and ODE-LSTM [Lechner and Hasani 2020). The GRU-ODE is based on recurrent networks. which modify the GRU topology by separating its memory cell into time-continuous states. At each time step, the model first iterates using the conventional GRU equations and then computing the differentiation based on the ODEs (Chen et al., 2018).. We then propose method to solve PO tasks by training the GRU-ODE and the actor-critic algorithm. We show owing to the differential process, our model could extract more accurate unobservable information from partially observable environments. Our approach outperforms baselines in several regular PO and Meta-RL control tasks and irregular PO tasks.
2 Preliminary and Related work
Typical POMDPs are used to describe decision or control problems in which the underlying states of the environments cannot be directly observed. A POMDP a tuple AS, O, T, O, R. 7). S set of states, A is a set of actions, and T: S x A is the nate-trannition probability function. Let O be the observation set and let O: S A x O - [0, 1] be the observation probability. The reward function R S x A decides the reward during state transition and is the discount factor. The objective is
to learn a policy function to maximize expected discounted rewards E
Σ
with a finite
horizon
Common PO tasks include scenarios of partially occluded states [Heess et al. 2015 randomly
dropped frames (Hausknecht and Stone 2015), egocentric images [Zhu et al. 2017] and randomly noised observations [Meng et al., 2021]. These kinds of problems are hard to solve because the dimension of historical observations grows linearly with timesteps. Prior works consider learning a belief state to encode underlying environments by extracting information from historical state transitions [Zintgraf et al., [019]. Some other methods use memory- -based policy, which takes the entire history states inputs [Lee et al. 2020). Recent works have widely used recurrent models to equip relevant algorithms [Ni et al.) 2022 Han et 2019). These recurrent methods could be further divided into model-free and model- methods according to whether their training objectives contain transition functions.
In the meta-RL setting, some indicator of the task is unobserved and methods are quite similar to that of PO tasks. Prominent approaches utilize recurrent networks for fast adaption (Duan et al. 2016). Recent studies demonstrate that the recurrent model-fre -reee algorithm serves as robust baseline for meta-RL [Ni et al., [022]. This is accomplished by equipping the policy network with a recurrent model, which aggregates past state and action transitions into an auxiliary context input [Zintgraf et al.
2019 Recent works also consider training non-recurrent models to encode the context [Mu et al.
2022
Guo et al. 2022]. These approaches can be categorized into coneext-based meta-RI. methods.
Another category uses policy gradients [Finn et al. 2017 Xu et al.) 2018a] or hyperparameters XXu
al. 2018b to learn from aggregated experience defined as gradient- gradient-based methods.

---

Neural ODEs have been widely used to tackle irregular time series [Chen et al., 2018). Standard RNN treats observations as sequence of tokens and does not account for variable timestep between observations. When facing irregular-sized data, another interpolation or generative adversarial network is used to perform interpolation and imputation. However, these methods are not robust in unseen environments. [Chnn et al. (2018) use residual neural networks with time-invariant dynamics to solve the ODE initial-value problem (IVP). Recent works build continuous time models by adding ODEs to the recurrent cells update process (De Brouwer et al. 2019 Kidger et al. 2020]. CT-LSTM Mei and Eisner 2017] combines the LSTM architecture with the continuous- time neural Hawker process. ODE-LSTM Lechnner and Hasani 2020] transforms the internal dynamical flow of LSTM to a continuous- cartiuuuus-tme model and demonstrates its efficacy in learning kinematics simulation. Liquid time-constant networks (LTC) [Hasani et al. 2021 Lechner et al.) 2020) represent dynamic systems with liquid time constants coupled to their hidden states and compute outputs by numerical differential equation solvers.
Recent studies in scientific machine learning have attempted to utilize dynamic encoding or physical mechanisms within their respective fields, such as DeepONet (Lu et al., 2019] and PINNs [Cuomo
ct al. 2022 Krishnapriyan et al. 2021]. Incorporating physical insights has been shown to enhance the efficiency of neural networks and the ability of representation learning (Zhagg et al. 2022 Yang et al.
2021 Han et al. 2023 Zhang et al. 2023a]. Some previous works combine ODEs with model-
decision processes (SMDPs), which also shows variables encoded by Latent- Latent-ODE capture unobservable state representations in PO environment. Yildiz al. []02]] infer the unknown state evolution differentials with Bayesian neural ODEs. Some other works use ODEs to learn the dynamic or transition functions
RL. Du et al. (2020) use Latent- Latent-ODE to learn the transition model to solve
and apply them continuous and irregular time interval tasks (Salehi al. 2022
Ainsworth et al.)
2021
3 Approach
This section first describes constructing ODE-based recurrent models to encode historical information into context variables for model-free RI. Then, we describe the overall training objective and how to use our model to optimize the policy network and value function.
In the typical POMDP setting, the reward and transition functions share some common structures across the MDPs. Some embeddings must represent the detail of the task information, which could not be accessed in advance. Context-based RL methods attempt to solve partial observable problems by allowing the policy network to receive an auxiliary context variable which includes historical information. The context variable at time step t is inferred from the agents' experience up to the current states,
(op. Go. 01,00 11.
There should be a superseript i to denote various episodes, we drop it for ease of notation. In POMDP environments, observations are denoted as o, instead of st, as the underlying states are only partially observable and must be inferred based on the available observations. Based on the above T., it is sufficient to encode historical information into context variable 2,, rather than modeling transition and reward dynamics which consist of millions of parameters. Generally, the memory- based policy is defined as т (aa T.)), conditioning on the whole history.
3.1 Model definition
Following ODE-RNN [Rubanova et al.] 2019). extend hidden state transitions in RNNs to continuous-time dynamics defined by Neural ODEs. As RNNs with exponentially-decaye hidden state follow implicit ODE of the form dh(C)
di -Th with h((l)) = ho. where is parameter, the hidden states could be modeled by Neural ODEs. Therefore, we propose our ODE-based recurrent model.
3

---

GRU-ODE. In our models, transformations of hidden states and transitions between latent states are computed using recurrent cells and neural ODEs, respectively. The resulting model is
h GRUCell (1
h13 ODESolve (for (For he, dt
(2)
2 (uhu Th.) o(he),
where the context variable 2e is defined as Gaussian distribution parameterized by mean ph, and standard deviation Th, 2tz is the concatenation of oz, a;-1 and rs. The gate of the GRUCell is controlled by the output of ODESolve hy, instead of its own output h2. The ODESolve are numerical differential solvers implemented by neural networks
pltd ODESolve (fo. ((t) dt) = h(t) + f, fo(h(t))dt,
(3)
where fo computes the differential of h(t) which are implemented by neural networks. The time increment dt the numerical ODE solver is fixed for conventional control tasks and varied in irregular observation tasks. The input of our model s, is the concatenation of observations or, actions d=- and rewards Tz- We found that the reward signals could help the agent learn unobservable information, leading improved agent performance in PO environments. As the reward function typically consists of r(ss, a) = d(s,,* 8*) + c||a||2. where the first term measures the distance between the current state and the objective state and the latter term contains the penalty of the action magnitude. Therefore. the unobservable state information could be inferred based on tuples of observations, actions, and rewards. To verify the rationality of our proposed model, we simplify the classic control task. In PO dynamic systems, the underlying physics of the task is always supposed to be second-order, the observable states are expressed in terms of either position @ or angular velocity w. As the state and action space could be denoted as s(t) = [0(t), w()) and a(t) respectively, the dynamic system with explicit control is defined as follows:
dB(t)
(t), dw()) = ((t()), dt
dt
where A(-) is referred to as acceleration field. Simplifying the update equation, this procedure could be modeled as differential equations with explicit actions. The current state could be modeled using ODESolve with initial state as s(t) = =0) + f fe(s(T), a(T))dr,, which could be solved by neural ODEs [Chen et al., 2018). Previous neural ODE models were always used for time-series predictions, which do not contain explicit actions. Chiappa et al. (2017) incorporate actions as auxiliary inputs of recurrent cells build recurrent simulators. We combine them to build our recurrent models. In addition to states, actions and rewards can also be solved as initial value problems.
Proposition 1. by a deterministic policy function x (-)) The derivative of a(l) with respect to time could be expressed as
The action a(l) is differential with time like the state s(l), as the action is controlled
da(t) dx(s(t) dr(s((t) ds(t) ds(t) dt
(5)
dt
dt
The policy function (T-) is implemented with neural networks, so dr (s(1))/ds() is the differential between the output and input of the policy network and could be computed by the chain rule. Following the above formulation, the reward function is also differentiable as it is always the linear combination of state s(t) and action a(t). Therefore, the update process of GRU-ODE input xt)). the concatenation of state, action and reward, could be modeled as ODEs and solved by ODESolve. As the latent context variable 2, reflects the general dynamic across different episodes, we reckon the update process of 2, could be modeled in the same way with explicit input zxt). Our model explicitly changes the discrete update into a continuous case, which represents latent variables more precisely. Existing implementations of state transitions perform inaccurate numerical integration routines, which results in rough estimation. For example, in the CartPole task crude numerical solver lead approximations bifurcate near the pole hangs up positions (Yildiz et al. 2021). The accuracy of the approximation dependent on the stepsize of the solver. Consequently, continuous methods tend to provide a closer approximation to the true value.

---

Encoder
Encoder critic
GRU-ODE
GRU-ODE
0z
GRU-ODE
-x-1
0 (x)l::))
Op-1
(gleelt.))
ODESole
MiF
7>
Ft
T(-11
10g 4-1
Policy network
Value function
Buffer Ztt
Actor Ty (z 2,)
02
Critic
Ge
qup (Qclo ac)
Figure I: Network architecture. Trajectories of observations, actions, and rewards are sampled and stored in the buffer. During the training procedure, Tt-1:t processed online using GRUCell to produce the embedding ht Subsequently, h4 passed through an ODE solver and MLP to encode the context variable 25. We use separate encoders for the actor and critic networks to improve the performance.
The global and local truncation errors of the Euler numerical solver are O(h) and O(h²) respectively. As the stepsize approaches zero, the truncation error tends to lim [O(h) = h-
Proposition 2.
Assuming the increment dt the ODEs equals the observation interval, the discretized system will converge to the actual ODE as the time increment approaches zero.
Similar relevant context-based methods, we use a recurrent ODE-based model encode past trajectories into a variable. Compared with ODE-based models with incodr-ddecode structures, models transformed from typical recurrent models could predict online at each time step. As opposed to standard RNNs, ODE-based RNNs learn the dynamics between observations, rather than having them pre-defined, which enables handling irregular data and modeling continuous sequences without relying pre- assumpptions about the dynamics of the observation sequences (Rubanova et al. 2219). Morcover, we found that owing to their underlying physical principles, ODEs are suitable for inferring unknown physical information, which is significant in partially observable environments.
3.2 Training procedure
Now we have the GRU-ODE model represent the context variable, move the question of identifying learning procedure and optimal policy. As mentioned above, with partial observations, latent context 2, is used to provide historically unknown information. Prior works have shown using separate recurrent encoders for the actor and critic could achieve high rewards [Meng et al. 2021]. Ni et al. [2022] show the gradient norms of recurrent context encoders vary in the actor and critic resulting in the gradient critic loss dominating the actors'. which will impact the training process of the actor's is context encoder. Therefore,
separate GRU-ODE models for context representation and implement them into Actor-Critic algorithms to improve the final performance.
Model implementation. The general training objective maximizes the discounted reward in an episode. We implement method based on TD3 Fujimoto al. [2018] and SAC Haarnojj al.) [2118. so the policy network and value function should be parameterized differently. Deep neural networks are used to represent individual components. There are
1. The recurrent context encoder GRU-ODE model ga(=)lTa), parameterised by d. We use superscripts a or e denote the encoder and context variables used for the policy network or the value function.
2. The conditioned policy network (actor) (a lov, 98 (.+)) parameterized by 0 with con- text variable zf-
3. The conditioned value function (critic) (Qrlor, (zf()T)) parameterized by o with context variable 2f.
5

---

The network architecture is shown in Fig. [ The context variable 2, is regarded as posterior and represented by the distribution's 's parameters, the details are discussed later. Compared to modeling the transition or reward function directly using the network, learning distribution embeddings need fewer parameters which makes inference easier.
Training objectives. the expected return
The conventional training objective of the policy network (.) is to maximize
T [(0) E Σ R((ccrills
(6)
After experimental analysis, it was discovered that utilizing J(0) directly in conditioned policy network and value function with recurrent encoders results in significant variation of the embedding variable 2, within an episode. This variation can lead to a degradation of the context encoder's s representation ability in PO environments. Therefore, we adjust the original training objective with an embedding constraint term to regularize 2t. Now, the overall training objective is maximize
[(o, 0, v = E. [Tb,, Φ, 8) XK(
where / denotes the trajectory distribution induced by the sample policy. Expectations are approxi- mated by Monte Carlo samples. Conditioned delayed policy update algorithms are implemented in our method, thus (1b, o, 0) are used to denote actor and critic networks with different parameters. To regularize the context variable 24, we add the Kullback-Leible (KL) divergence K(()) between posterior and prior distributions
K(Ø) = Σ DKL (20(2)
(8)
t=
We set the prior to our previous posterior, po(22_1/::t-1) with initial prior (2)) (0. DxL denotes the KL divergence, which could be optimized by the reparameterization trick. Since both terms in (C)) are parameterized by Gaussian distributions, the KL--iiergence can be analytically calculated
The + (Hort - Mo,t-1) o2. 1 2
DKL (=)=()) log
(9)
ge.t-1
20 20
In this work, past trajectories are encoded by GRU-ODE. Eq. 7 is optimized end-to-end, and À the balance weight that supervises the encoder learning objective against the RL objective J (r), db 0). which is objective to find optimal policy and accurate value function. To improve the efficiency of the training process, we sample fixed-length truncated sequences mini-batches for training model, rather than using the whole episodes.
Previous work mentioned that training the encoder and the policy network separately using data from different buffers can prevent the gradients of opposing losses from interfering with each other. Zintgraf et al. [2019] follow this setting to pre-train train the encoder in advance before the policy network. We found this setting unnecessary in practice, the data buffer shared in our method.
Experiments
We experimentally evaluate our models across several PO regular/irregular observation domains. In regular observation domains, we consider conventional partially observable control and meta-RL tasks by employing MuJoCo [Todorov al. 2012) and PyBullet [Greff et al. 2022 environments. In irregular observation domains, we consider modifying classical environments: Pendulum and CartPole tasks Brockman et al. 2016 into irregular time intervals with surrogate rewards.
4.1 Partially Observable Classic Control Tasks
The Pendulum and CartPole tasks are classic control tasks for evaluating RL algorithms. For Pendulum task, the goal is to learn policy to swing the pendulum up and maintain it at the highest position to obtain more rewards. For the CartPole task, the goal to learn policy to preclude the pole from falling down and forestall the cart from running away by exerting force on the cart.

---

Perdulum
Pendulum
Pendulum-V
-290
Average Return
-480
-000
Average
-403
-1.00
-10
GLAC
-182
Thousand Stape
Thousand Skops
CarPole
CartPole
CartPoe-VV
190
100
Average Retum
Average Retium
Oum SAC- SLAC
Thousand Staps
Thousand Steps
Figure 2: Classic partial observable control tasks. The shaded region represents standard deviation of average evaluation over five runs and Curves are smoothed for visual clarity.
The observable information consists of the coordinates of the cart, the angle of the pole, and their velocities.
These classic control tasks are relatively easy to solve in fully observable domains. Therefore, these PO tasks can effectively underscore the problem of representation learning. Experiments are performed in PO cases of these two control tasks, in which only velocities or positions could be observed (we use the suffix -V/-P to name). The PO setting is more practically significant, as many real-world applications, agents may only be able to estimate partial state information.
We compare our method with SLAC, VRM, and SAC-LSTM. SLAC (Lee al., 2020) combines off-policy model-free RL with representation learning via sequential stochastic state space model. SLAC was developed for pixel observations, we follow the modifications in Han et (2019] to compare with our method. VRM [Han et al., 2019] is recent, model- -ased POMDP algorithm, which uses variational recurrent models to encode context variables and separate representation learning from dynamic programming. In the SAC-LSTM, soft actor-critic algorithms with recurrent
networks are used as function approximators. We follow the settings in [Han et al. 2019 to construct the above methods.
As expected, our method succeeds learning to solve all these tasks and performs better than VRM which is the state-of-the-art method for solving POMDP tasks. While SLAC performs well in the CartPole tasks and less sample efficiency in Pen- dulum tasks. SAC-LSTM needs more epochs for training and fails to solve the Pendulum- V task. We reckon these results are attributed to the abil- ity of ODEs to model physical systems and infer unknown information.
Pendulum-P
Pendulum-V
UNAA
UMAP1
UMAP1 1 (b)
(a)
To assess the capability of our model deduce un- observable environmental information, we project the latent variable 2, in an episode to 2D using UMAP [ccInnes et al.] 2018). We estimate the unobservable state information such as the angular velocities in the Pendulum- task through obser- vations, actions, and rewards (as the rewards are calculated based on state and actions, the unob- servable state could be computed in reverse), and scatter plot color based on the logarithmic scaling
Figure 3: The visualization of the latent vari- ables 2; one episode after dimension reduc- tion. Points are colored according to (a) the logarithm of the calculated angular velocity and (b) the logarithm of the calculated angular posi- tion (angle difference from the initial position). Each figure consists of 201 points including the initialized Gaussian 2o, as the maximum step in the Pendulum task is 200.
7

---

AndLLT-P
WalkerBLT-P
HopperBLTTP
Our SAGAUTTM
2003
Averagg Returr
1/00
2000
Thousard Shapa
190
Trousand Stope
Thouoard Etace HopperBLT-V
Jm0
AntBLT-V
Walker
Chans
1403
Average Rokur
1700
RACA NMM
1200
BLAG
1903
1.00
100
401
Figure 4: Learning curve of the partially observable PyBullet robotic control tasks, plotted in the same way in Fig. [ The PyBullet environments are ported from deprecated Roboschool environments which are harder than the MuloCo Gym environments!.
of their values. Fig. shows the clustering of latent variables corresponds closely to the unobservable physical parameters, which demonstrates the efficacy of our GRU-ODE context encoder.
4.2 Partially Observable Robotic Control Tasks
We evaluate the performance of our proposed method the PyBullet control environments (Greff ctt al. [002]] which are more challenging than MuJoCo. The PyBullet environments contain several continuous robotic control tasks and the state of fully observable environments includes the robot's co- ordinates, trigonometric functions of joint angles, and angular velocities. We modify the observations into PO versions with the same criteria and compare them with the same methods as the PO classic control tasks. As experimental results demonstrated our method obtains policy improvement in the majority of PO robotic control tasks, especially in environments where only the position information observable. For tasks with only positions observable, our method outperforms the other three methods and presumably shows efficiency in extracting useful information from historical position observations. We reckon that velocity could be simply estimated by one-step differentiation in coordi- nates and joint angles of the robot, which eases representation and policy learning. Environments with only velocities that could be observed are much harder to solve. However, deducing the position information solely from the given velocity is not practical. In environments where only velocity can be observed, our method performs similarly to VRM. Additionally, our results indicate that the learning processes of the SLAC method exhibited instability, where it intermittently achieved nearly optimal policy, but frequently converged to a suboptimal one. As a result, the average performance of SLAC was generally less favorable than that of our approach across the majority of PO robotic control tasks. SAC-LSTM fails to solve almost all these tasks.
We further compare our methods with two more baselines RMF (Recurrent Model-Free) Ni et al.. 2022] and FPO- (TD3 with fixed previous observation window). The complete results are shown in Table 6
4.3 Meta-learning MuJoCo Control Tasks
We further show our method can also scale to other types of PO tasks, such as Meta-RL, by applying our method to a diverse set of domains, such as point robot and MuJuCo locomotion tasks, which are commonly used in meta-RL literature. Our tasks involve utilizing the semi-circle, which refers to a point robot that navigates along a semi-circular path in search of sparsely located reward, as well as wind tasks, where a point robot navigates fixed goal amidst fluctuating wind conditions. For MuJuCo locomotion tasks, we consider using the HalfCheetah-Dii environment where the agent has

---

Semi-Circl
Wind
HalfCheetah-
Average Returns
2006
Orock
196
polky
Oroda
Onyp VerDAo
RLL
Our
Frames
145
Frames
02
Frames
1eB
Figure 5: Learning curves with compared methods meta-RL RL tasks. Unlike other methods, only plot the final return of Oracle and Pearl methods in the semi-circle and wind tasks. The learning curve data of on-policy VariBAD, Oracle, and RL2 in the HalfCheetah-Dee Dir task are copied from the public GitHub repository of variBAD*.
to control two legs run either forward or backward. The MuJuCo-relevant tasks are much harder than the prior point robot tasks.
We compare our method with various methods, such as VariBAD, RL2 [Duan al., 2016]. Pearl
[Rakelly et al. 2019), and Oracle policy. VariBAD utilizes variational, model- based objective to learn task embeddings explicitly. The originally proposed variBAD (Zintgraf et al.) 2019] uses PPO
[Schulman et al., 2017], and the same method was implemented with SAC [Dorfman et al. 2020).
These two methods could be named on-policy variBAD and off-policy variBAD. An oracle policy can access the POMDP hidden state to transform the POMDP into an MDP, thus this policy could be treated as an upper bound on the performance related methods. We follow the settings Ni et al.
2022 to construct the above methods.
Fig. shows that our method outperforms off- off-policy variBAD and Pearl in the semi-cirele and wind environments, and on-policy variBAD and RL2 in the HalfCheetah- Dir environment, both terms sample efficiency and asymptotic return. our model trained end-to-end, without using pre-trained encoders to represent task contexts like off-polic variBAD, which do not suffer out-of-date representation problems. On-policy variBAD uses autoencoder architecture reconstruct the state and reward, which are more advantageous to context representation. Our method shows simple recurrent encoder could also stabilize representation learning and improve policy performance.
4.4 Temporally Irregular Observable Control Tasks
In standard RL frameworks such as OpenAl Gym, the states/observations arrive at constant inter- vals as A 8. To contrast the robustness of our continuous-time method with other discrete POMDP methods, we evaluate related methods with irregularly sampled observations. We con- sider using an irregular sampling scenario in which observations arrive uniformly within range At 4 20)2 We choose the mean time difference 5 0.05 for observation spacings, which is the standard interval setting of the Gym Pendulum task.
1000
200
800
400
600
500
400
Ours
300
VRM
200
SLAC
-1000
Pendutum
CanPob-VV
Figure 6: A comparison of PODMP methods standard deviation VRM, SLAAC, and our method with uniformly irregular observations.
Fig. 6 exhibits the final average returns with
in irregular Pendulum and CartPole tasks. The results show ODE-based recurrent model is more robust with stochastic time increments, which reasonable as ODEs build discrete-time sequences into continuous ones. This environment has more practical significance, as observation intervals are usually disturbed practical application.
htte::://gthhbbb.ommmmmmmggggggs

---

Limitation
Although we have demonstrated that ODE-based encoders have advantages in robotic control tasks, their efficiency in discrete game problems, such as Atari, is not promising. In Atari, the states of the agents are not updated based on numerical differential equations, resulting in the inability to represent the state and action as integration processes. Also, the reward function is typically sparse and predefined. Implementing ODE-based encoders to model related processes does not make sense.
In addition, since the integration process is implemented by neural networks, the training procedure of ODE-based encoders requires more computation resources, which results in increased training time, especially for high-order numerical differential solvers. Although we find that using simple numerical solvers is enough in the majority of tasks (see Appendix Section [TP,. accelerating the integration process is still significant in solving differential equations and downstream tasks.
6 Conclusion and future works
incorporate ODE-based recurrent neural networks with model-free reinforcement learning to solve POMDP control tasks. Through empirical assessment across several partially observable domains, we showed that the policy network and value function conditioned on context variables encoded by GRU-ODE help the agents infer unknown observations and maximize expected returns. We experimentally demonstrate that our method is robust to environmental changes such as irregularity.
Reinforcement learning has been widely used in physical or biological systems explain some of the mechanisms. However, some mechanisms in the brain are still unknown, such how the brain arbitrates or allocates control over various sub-systems and the role of the prefrontal cortex and striatum in controlling. The classic leaky integrate and fire (LIF) neuron model quite similar to ODEs if we ignore the firing process. We reckon combining ODEs with RL could help biologists explore mechanisms in the human brain.
Acknowledgment
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA27010404) the Beijing Nova Program (Grant No. 20230484369), the Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX), and the Youth Innovation Promotion Association of Chinese Academy of Sciences.
References
S. Ainsworth, K. Lowrey, J. Thickstun, Z. Harchaoui, and S. Srinivasa. Faster policy learning with continuous- time gradients. In Learning for Dynamics and Control, pages 1054-1067. PMLR, 2021.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: (60.:6150600400 2016.
R. T. Chen, Y. Rubanova, J. Bettencourt, and K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.022 2017.
Cuomo, Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informet neural networks: where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
E. De Brouwer, J. Simm, A. Arany, and Moreau. Gru-ode-bayes: Continuous modeling of sporadically- observed time series. In Advances in Neural Information Processing Systems, pages 7377-7388, 2019.
R. Dorfman, Shenfeld, and A. Tamar. Offline meta lcarning of exploration. arXiv preprint arXi: 2020.
J. Du. J. Futoma, and F. Doshi-Veled Model based reinforcement learning for coni-makovv decision processes with neural odes. Advances Neural Information P Processing Systems, 33:19805-1 19816, 2020.
10

---

Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sulskever. and P. Abbeel. RI 2 Fast reinforcement learning via slow reinforcement learning. arXiv preprint urXiv:1611.0277,, 2016.
C. Finn, P. Abheel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1136-1355. PMIR, 2017.
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018.
K. Greff, F. Belletti, L. Beyer. C. Doersch, Y. Du. D. Duckworth, D. J. Fleet. D. Gnanapragasam, Golemo, C. Herrmann, al. Kubric: A scalable dataset generator. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022.
Guo, M. Gong, D. Tao, A relational intervention approach unsupervised dynamics generalization model-based reinforcement learning. arXiv preprint arXiv:2206.04551 2022.
Haarnoja, A. Zhou, Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
D. Han, K. Doya, and J. Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint 1912. 10703, 2019.
M. Han, Q. Wang, T. Zhang, Y. Wang, D. Zhang, and B. Xu. Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition. arXiv preprint arXiv:2302.011 230.00092.1194 2023.
R. Hasani, M. Lechner, A, Amini, D. Rus, and R. Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7657-7666, 2021.
M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv: 1507.:1507.065222 2015.
N. Heess, J. Hunt, T. P. Lillicrap. and D. Silver. Memory- control with recurrent neural networks. arXiv preprint 1: 2015.
M. Igl. L. Zintgraf, T. A. Le. F. Wood, and S. Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pages 2117-2126 PMLR, 2018.
M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz.
A. S. Morcos, A. Ruderman, al. Human-level performance 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859-866) 2019.
Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2019.
Kidger, Morrill, Foster, and Lyons. Neural Controlled Differential Equations Irregular Time Series. arXiv:2005.0999 08926, 2020.
A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in
neural networks. Advances Neural Information Processing Systems, $4:26548-26566
2021.
M. Lechner and R. Hasani, Learning long term-termmm dependencies in irreularty-sampled time series. arXiv preprint arXiv:2006.0444 2020.
M. Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence, 2(10):642 652, 2020.
A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neurul Information Processing Systems, 33:741-7522 2020.
L. Lu, P. Jin, and G. E. Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv: 1910.03193 2019.
L. Mcinnes, J. Healy, and J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1802.03426, 2018
H. Mei and J. M. Eisner. The neural hawkes process: A neurally ceff-modulatin multivariate point process. Advances neural information processing systems, 30, 2017.

---

L. Meng, R. Gorbet, and D. Kulić. Memory-based deep reinforcement learning for pomdps. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-5626. IEEE, 2021.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540); 529-533, 2015.
Y. Mu, Y. Zhuang, F. Ni, B. Wang. J. Chen, J. Hao, and P. Luo. Domino: Decomposed mutual information optimization for generalized context in meta- reinforeett learning. Advances in Neural Information Processing Systems, 35:27563 27575, 2022.
T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. In International Conference on Machine Learning, pages 16691-16723. PMLR, 2022.
J. Peters, S. Bauer, and N. Pfister. Causal models for dynamical systems. In Probabilistic and Causal Inference: The Works of Judea Pearl, pages 671-690. 2022.
K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off policy meta-reinforcemmnt learning via probabilistic context variables. In International conference on machine learning. pages 5331-5340. PMLR, 2019.
Y. Rubanova, R. T. Chen, and D. Duvenaud. Latent odes for ireeullrly-ssappled time series. arXiv preprint arXiv: 2019.
A. Salchi, S. Rühl, and S. Doncicux, Meta neural ordinary differential equations for adaptive asynchronous control. arXiv preprint axiv:2207.12066. 2022.
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, Kalchhrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)x612-634, 2021.
Schulman, Wolski, Dhariwal, Radford, and Klimov, Proximal policy optimization algorithms. arXiv preprint 177. 2017.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
T. Xu, Q. Liu, L. Zhao, and J. Peng. Learning to explore via meta-policy gradient. In International Conference on Machine Learning, pages 5463-544 5472. PMLR, 2018a.
Z. Xu, H. P. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems, 31, 2018b.
S. Yang, Z. Li, G. Song, and L. Cai. Deep molccular representation learning via fusing physical and chemical information. Advances in Neural Information Processing Systems, 01116666-165551 2021.
C. Yildiz, M. Heinonen, and H. Lähdesmäki. Continuous-tine model-basee reinforcement learning. In International Conference on Machine Learning, pages 12009-12018. PMLR, 2021.
D. Zhang, T. Zhang, S. Jia, and B. Xu. Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 59-67, 2022.
D. Zhang. T. Zhang, S. Jia, Q. Wang, and B. Xu. Tuning synaptic connections instead of weights by genetic algorithm in spiking policy network. CoRR, abs/2301.10292 2023a. doi: 21.455500aXX12223111022.. URL https://doi.. org/10. 2tte://1..8500//88 48550/arXiv> 2301 10292
T. Zhang, X. Cheng, S. Jia, C. Li, M.-m. Poo, and B. Xu. A brain-inspired algorithm that mitigates catastrophic forgetting of artificial and spiking neural networks with low computational cost. Science Advances, 9(4)): eadi2947, 2023b.
Y. Zhu, R. Mottaghi, F. Kolve, Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEFE international conference robotics and automation (ICRA). pages 3357-3364. IEEE, 2017.
L. Zintgraf, K. Shiarlis, M. Igl. S. Schulze, Y. Gal, K. Hofmman, and S. Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- meta-Learning arXiv preprint ar.00:191,.08348, 2019.
12

---

Supplementary Material
In this supplementary material, we provide further information about our implementation details and ablation studies.
7 Truncation Error
The update procedure the Euler Equation
Mi+1 = yi (x1.
(10)
and the accurate update equation is
122 (")),, E (o,, (141)
Vi+1 y/ hffr,, 2
(11)
The lost item (E:) is the local truncation enror of the Euler method. The global truncation defined as
c++1 = y(æ1+1) - y(i + 1)
h2
(yx + hos, v(x:)) + 2 ("())) - (yi + h1(,
h²
ei + (M)(((aa)(()))) (x; v:))
(E)
2
In Euler method f(x, y) satisfy the Lipschitz condition for and denote T: local truncation error and T max [T.] Ohh)
12 (E.) is the
le++1
$ lesl + fzl, y(x:)) - f(z,, vi) + [T+il
< leil + hl)((z;) Vil + (Ti+1l
s les[(1 + hu) + T
T
s (1 + hL) ++1 (leal +
hL
c(tt1)DL((eo + hL
T
O(h)
where e =y(x 0) y0 0.
8 Enivronment Specification
We incorporate the occlusion benchmark proposed by VRM and replace the deprecated roboschool with PyBullet, as the official GitHub repository suggests. We transform the original MDP task into POMDP version by removing all position/angll--elate entries in the observation space for "-V" environments and velocity-related entries for " "-P" environments.
(Pendulum, CartPole, AntBLT, WalkerBLT, HopperBLT)-P. The "-P" denotes environments that retain position-relaae related observations by eliminating velocity-related observations.
(Pendulum, CartPole, AntBLT, WalkerBLT, Hopper The "-V" denotes environments that retain velocity-related observations by eliminating position-related observations.
Implemnetaion Details
We the PyTorch framework for experiments. Some basic hyperparameters about the network architectures are listed below
Also, there unique parameters in TD3 and SAC, report them together
13

---

Table 1: Environment information
Environment name
dim(o)
dim(a)
Maximum step
Pendulum--P(position only)
200
Pendulum--(vvlocctty only)
1
200
CartPol-PPpossition only)
2
1000
CartPole-V(velocity only)
2
1
1000
AntBLT-P(position only)
17
8
1000
AnteLi-V(velocity only)
11
8
1000
WalkerBLTTP(positionn only)
13
1000
WalkerBLTTVVvelociiyy only)
9
6
1000
HopperBLTTP(position only)
1000
HopperrBLLT-VVvelocciyy only)
6
3
1000
HalfCheetah-Dir
17
200
Table 2: Hyperparameters
Hyperparameters
Description
Value
Discount factor
0.99
Learning rate
0.0003
A
Balance weight between KL divergence and RL loss
0.5
т
Fraction updating the target network each gradient step
0.005
layers
MLP layer sizes of value function
[256,256]
policy layers
MLP layer sizes of policy network
[256,256]
sampled_sey len
The number of steps in sampled sequence for each update
64
batch_size
The number sequences to sample for each update
64
buffer_size
The number of saved transitions Embedding dimension of action Embedding dimension of observation Embedding dimension of reward
le6
action_embal
seten__mmeddiiiigeee size
16
size
32
reward_ember eewieddeeeeddiigggee size cru_hidden_sine
16
Hidden layer size of recurrent encoder
128
9.1 Encoding procedure of GRU-ODE
The input observations is the concatenation of observations, actions and rewards, mentioned above. The time increment dt is set to a fixed value in regular observation environments. In this paper, we set dt 0.1 for all POMDP tasks.
9.2 Integrating with TD3
We use separate recurrent encoders for the actor and critic order to improve the performance of our method. Owing to the policy network of TD3 updating less frequently than the value function. We modify the iteration equation of the policy network to:
L( 0, = E on
law, (o, To(o, z), + Aa(C(d))
(12)
Algorithm
The GRU-ODE algorithm.
Input: Observations and time difference between observations (Ie, di) 11.TT
ho 0
for in 2,... do .....
h, = GRUCell zx) (Update hidden state)
hr = ODESolve (Jo, hr, dl) (Solve ODE)
2: MLP(h;) for all I..
Return: h=
14

---

Table 3: Hyperparameters of SAC and TD3
Method
Hyperparameters
Value
SAC
entropy alpha
0.2
SAC
automatic cattice_eetttttteeeeee ttnnngg
True
SAC
alpha_Ir
0.0003
TD3
exploration noise
0.1
TD3
target_noise noise
0.2
TD3
target_noise noise clip
0.5
DD denotes the replay buffer. The state s is replaced with observation o owing to the partially observable environments.
The update of the value function is modified to:
u(r,o',2
y min Quintary(d', no (oaorg(o')))))))) (o)), 2/), ==1,2
((1)))
(o.0, [[Q (0,0, slr o'r z =²))²]+
E
(13)
[(o., 1/) [(v)) ((v2) A.K(d.).
where subscript 1, 2 denotes two different critic networks implemented in TD3 and targ denotes the target network. The context variable computed by g (T²)) as the policy network and value function update, we omit this procedure to simplify the equation.
9.3 Integrating with SAC
Similar TD3, we implement GRU-ODE in SAC. The training loss of the policy network is modified to
L( -- =
min go, (o,00))))) (loo) + AaX((aa)
(14)
anD 1=112
and the value function becomes
u(r,o', = min (Q (ary (d', mo.targ(d)) clogro(()o')), = j=1.2
((v)) =
E
[Q0, (o, 2) y(r, 2'))))
(15)
(o.a, >D
C(d., [(v)) + [(w2) + \ck))
where symbols have the same meaning as Eq. 13 The parameter set to 0.5 both in TD3 and SAC algorithms.
10 ODE Solver Comparison
In this ablation study, we ask two questions in relation to numerical integration. (i) how different numerical solvers influence the final performance of POMDP tasks, and (ii) the effect the training runtime hours. We experimentally evaluate using different numerical methods as ODESolve for
integrating be in GRU-ODE, From Table 4 we find that there is no obvious relationship between the
Table 4: Final performance different numerical integration methods
Ant-P
Ant-V
Walker-P
Walker-V
Euler 1243 +:
998 + 93
1487 81
612 27
RK4
1183 4: 67
1045 + 112
1405 + 64
683 + 45
Heun 1123 + 1023 69
1457 45
625 41
complexity of the numerical solvers and the final performance of the PODMP tasks. However, the
15

---

RK4 and Heun need much more time for training. We train these methods on server with NVIDIA TITAN Xp and Intel(R) Xe) CPU E5-2620 @ 2.10GHz as GPU and CPU respectively. The following are rough estimates of average run times for the AntBI.T-P environments.
Ours (with Euler) 12 hours
Ours (with RK4) 20 hours
Ours (with Heun) 17 hours
As the results show for context variable coding, the simplest numerical Fuler method can work well as ODESolve and save training time. We speculate that though the environment is complicated to solve, the context variables always reflect the essential dynamic information the environments which are more refined than the observations. Thus, simple numerical solvers are enough.
11 Time Cost of various baselines
We evaluate the time costs of different baselines Walker-P environments. From the results, find our method does not increase the computation a lot compared to other baselines. We analyze that there two reasons. (I). Within our implementation of ODESolve(, method for computation. This offers substantial reductions in computation time when contrasted with the default "dopri5" approach Stunge-Kutte of order of Dormand- Gormand-Prince-S5 Prince-Shamping utilized in ODE-RNN. We observe that employing complex ODESolve() methods does not notably enhance performance, Instead, increases computational time significantly.
employ the "Euler"
(II). We are working within the context of RL tasks. When training RL algorithms, it's not just the ODE-GRU that needs training, other networks such as policy networks and value networks also require training. During the training process, significant amount of time is consumed by sampling transitions from the data buffer and updating the data buffer. Therefore, terms of the overall training time of our approach, it doesn't significantly exceed the time required by the previous method that used an RNN as an encoder.
Table 5: Time Cost of different methods
Methods
Time Cost
Ours
12h
RMF
10h
VRM
18h
SAC-LSTM
6h
SLAC
3h
12 Further Performance Comparison
We add two baselines for performance comparison, the RMF (Recurrent Model-free) is a recent SOTA method POMDP problems and the TD3-FPOW (TD3 with fixed previous observation window) is a modified version of TD3, which takes 3 concatenation frames as the input.
Table 6: Performance comparison
Environments
SLAC
VRM
SAC-LSTM
RMF
TD3-FPOW
Ours
Ant-P
950-1129
1040175
9461147
1048+74
9741143
1243-143
Ant-V
663188
981113
690114
1021+165
903:154
998--93
Walker-P
277+121
1121+167
971+103
1123+176
1043+103
1487--1
Walker-V V
138+25
551+:30
491138
586+:52
482-:32
612-t27
Hopper-P
222+:21
1851+61
1236-t39
2133+326
1654+217
2455+887
Hopper-V
310+41
1652.167
890-L43
1495 +
13124887
1545+104
The result shows that method achieved the best performance in 4 out of 6 environments.
16

---

13
Additional ablation study
We study the influence of the shared/separate encoders, and different concatenations inputs as the ablation study. Due to the time limitation, we conducted these experiments on the Walker-P and Walker- environments.
Table Performance comparison about shared/separate encoders
Environments shared
separate
Walker-P
1336
1487
Walker-V
584
612
The result shows that separate context encoders for the policy network and value function improve the performance.
Table 8: Performance comparison different kinds of inputs
Environments
Walker-P
1276
1065
1396
1487
The result shows that taking the input as the concatenation of and a;-1 for the context encoder (GRU-ODE) improves the performance.
17


Reviews:
[{"review_id": "t04JSu0cUW", "reviewer": "Reviewer_FbJ4", "summary": "The author present a method for doing deep reinforcement learning for POMDPs with uneven time between observations. Their method uses TD3 with Neural-ODE+GRU encoders for the actor and the critic. They test on various partially observed control tasks, as well as tasks with uneven observation times.", "questions": "Why are there no experiments on pixels observations? It seems the algorithm is scalable.\n\nWhy did you use the deprecated environments? What makes them harder and why does that matter as long as the comparison is fair?\n\nModel implementation, item 2 and 3. Why do you add back o_t as input to the network. It's already encoded by g, no?", "limitations": "In their limitation section, the authors mention atari. I don't think this is a very important limitation; it's OK, nobody expected ODEs to do well here. What worries me more is the ability to scale to pixels at all. They mention that their method is slower and I suspected that.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is sound and the results are good even though limited. To the best of my knowledge this is a novel approach for dealing with continuous time observation in RL and generally for using Neural-ODEs in RL. The problem of dealing with non-uniform time between observation is interesting for a lot of problems, especially robotics. The presentation is good (except a few minor things which I will highlight below). I find the integration of continuous-time actions with equation (5) clever, I would have gone with a constant action in between timesteps if I was going to tackle this problem. There will be a lot of small complaints below, but keep in mind that my overall opinion of the paper is positive.", "weaknesses": "I think the evaluation environments are a bad choice for POMDPs. Just removing part of the state is a very basic way to simulate partial observability. In the case of observing position and not velocity, think of all of the typical control benchmarks (Deepmind control suite for example) with pixel observations. They're equivalent (you can't observe velocities from a single frame) and yet people mostly treat them as MDPs and ignore the partial observability. How? Because you simply have to stack a couple of frames and use regular deep RL approaches. The partial observability aspect is easily addressed and is simply not interesting. There's no information gathering happening. This is a wider complaint because that's the way that a lot of POMDP papers do evaluation, so I won't take off to many points for this. But as a research community, we really need to sit down and find better benchmarks for POMDPs which highlight their ability to combine information gathering and reward maximization, which is the beauty of the framework.\n\nThe way I read the first result section is this way: we do well on the easy problem, position only (but I'd like to see basic TD3 with a concatenation of 3-4 previous observations as a baseline), but on the harder, actually interesting problem, we do just as bad as the other approaches.\n\nI find the use of non-standard deprecated environments odd, even though you mention they're \"more difficult\" (how so?).\n\nBecause all of the evaluation seems to be on non-standard environments, it's difficult to assess the significance of the results. My intuition is that pendulum from position only is actually easily solved, so it raises suspicion that's you are able to beat everyone else by a large margin. No information on how other algorithms were tuned was provided. Convergence speed, which is what your algorithm seems to be better at, varies wildly depending on hyperparameters.\n\nSome sentences I found hard to read: line 201 to 203, 170. 52-54.\n\nLine 56 is a bit hand-wavy. What do you mean by extract more accurate unobservable information? What the experiments show is faster reward maximization. No evaluation of the method's ability to gather information is shown.\n\nThe sentence about POMDP starting at line 107 is not correct. You're confusing meta-RL and POMDPs. Please give a formal mathematical definition of a POMDP and provide a citation. You will see that for most the the literature the reward function is the same across episodes. Yes, you can transform a meta-RL problem into a POMDP, by adding a hidden task variable as input to the reward function.\n\nEquation (2) $x_t$ is defined too far from it's appearance in the equation, I was confused at first\n\n(minor) Line 178, the comment about \"physical principles\" feels a bit hand-wavy to me. The type of Neural ODEs that are used here are no more physically inspired than typical RNNs; no physics knowledge is given to them. Yes a lot of mechanics is usually written down by engineers and physicists as ODEs, but I don't think that means Neural ODEs have \"underlying physical principles\", unless some actual physics guide the learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the positive aspects of our paper, and we will address the reviewer’s concerns in the following parts.\n\n**W1**: POMDP setting\n\nWe agree with your opinion about the POMDP framework, during our experiments, we found almost all the baselines are proposed on position/velocity only environments, so it is harder for us to compare in other benchmarks.\n\n**W2**: TD3 baseline\n\nWe add the basic TD3 method with a fixed partial observations window (TD3-FPOW, the window size is 3) as the input and compare it with our methods. We also compare a recent SOTA method RMF [1] here. Due to time constraints, we conducted experiments only on a subset of environments.\n\n|   |  Ours   |   VRM    | TD3-FPOW |   RMF    |\n| :------: | :-----: | :------: | :------: | :------: |\n|  Ant-P   | **1243±43** | 1040±75  |  974±43| 1048±74  |\n|  Ant-V   | 998±93  |  981±63  |  903±54  | **1021±165** |\n| Walker-P | **1487±81** | 1121±167 | 1043±103 | 1123±176 |\n\n**W3**: deprecated environment\n\nThe reason we used PyBullet as our experimental environment is that the baseline we are comparing against, such as VRM, was evaluated on this environment (specifically, it was evaluated on the roboschool environment, which has now been updated to PyBullet). We believe that this setting makes the comparison more fair because it eliminates the issue of adapting the baseline to different environments.\n\n**W4**: evaluation problem\n\nIn fact, the baselines we compared against were all open-source codes from relevant papers. In the pendulum experiment, our method achieves similar performance to VRM after the models converge, but with a faster convergence speed. We believe this is reasonable because the structure of the VRM encoder is similar to an autoencoder, which requires encoding and decoding of inputs, thus needing more steps to optimize the encoder. Although we have ODE steps, it is still faster than the decoding procedure.\n\n**W5**: sentence modification\n\nWe will modify lines 201 to 203 into “Compared to modeling the transition or reward function directly using the network, learning distribution embeddings need fewer parameters which makes inference easier.”\n\nLine 170 into \"Assuming the increment $dt$ in the ODEs equals the observation interval,\"\n\nLine 52-54 \"At each time step, the model first iterates using the conventional GRU equations and then computing the differentiation based on the ODEs.\"\n\nThe description on line 56 may not have been precise enough in explaining the unobservable information. Our intention was based on our visualization results and experimental performance, which led us to believe that our model can better extract unobservable information. As in the visualization results, similar values were mostly clustered together.\n\n**W6**: POMDP definition\n\nThe definition of pomdp is \"Typical POMDPs are used to describe decision or control problems in which the underlying states of the environments cannot be directly observed. A POMDP is a tuple $(\\mathcal{S}, \\mathcal{A}, \\mathcal{O}, \\mathit{T}, \\mathit{O}, \\mathit{R}, \\mathit{\\gamma})$. $\\mathcal{S}$ is a set of states, $\\mathcal{A}$ is a set of actions, and $\\mathit{T}: \\mathcal{S}\\times\\mathcal{A}$ is the state-transition probability function. Let $\\mathcal{O}$ be the observation set and let $\\mathit{O}: \\mathcal{S}\\times\\mathcal{A}\\times\\mathcal{O}\\to[0,1]$ be the observation probability. The reward function $\\mathit{R}:\\mathcal{S}\\times\\mathcal{A}$ decides the reward during state transition and $\\mathit{\\gamma}$ is the discount factor. The objective is to learn a policy function to maximize expected discounted rewards $\\mathbb{E}\\left[\\sum\\limits_{t=0}^{T-1}\\gamma^tr_{t+1}\\right]$ with a finite horizon $T$.\n\n**W7**: notation definition \n\nWe will add \"$x_t$ is the concatation of $o_t$, $a_{t-1}$ and $r_t$ \" after the Equation (2)\n\n**W8**: physical principle\n\nWe have considered your views carefully. We acknowledge that the \"underlying physical principle\" is not clear in ODE, but we still reckon Neural ODE could help to learn the underlying dynamic as the intrinsic structure of Neural ODE contains the differentiation process.\n\n**Q1**: pixel observations\n\nR1: Because our primary focus is on POMDP problems, there is no standard paradigm and baseline for POMDP problems in image/pixel observation environments. We attempted to use a two-layer convolutional network as the image encoder, but the results were unsatisfactory. We believe that the representation ability of the image encoder significantly affects the performance. Combining some image encoding methods may help our method scale to the pixel observation domain.\n\n**Q2**: environment setting\n\nR2: I apologize for any misunderstanding caused by our writing. The environment we used, Pybullet, is not deprecated. We chose Pybullet as our experiment baseline because VRM, which we are comparing against, was initially evaluated on roboschool. However, roboschool has been deprecated and updated to Pybullet. To compare consistently, we selected Pybullet as our experimental environment. The \"harder\" means PyBullet is harder than MuJoCo, as its observation space contains robots’ coordinates, joint angles, and velocities. \n\n**Q3**: model implementation\n\nR3: Yes, the observation $o_t$ is already encoded by GRU-ODE $g()$. However, the GRU-ODE $g()$ is regarded as a context encoder in our experiment, which output a context variable $z_t$. According to the definition of context-based RL, the context variables are regarded as auxiliary inputs to help the policy and value networks better calculate action distributions and state-action values. Thus, $g()$ is an auxiliary encoder, which does not modify the framework of the policy network and value function. Also, in this context-based paradigm, the encoder and RL algorithm are more flexible to revise.\n\n[1] Ni, T., Eysenbach, B., & Salakhutdinov, R. (2021). Recurrent model-free rl can be a strong baseline for many pomdps. *arXiv preprint arXiv:2110.05038*."}, {"review_id": "7OJT15gSr9", "reviewer": "Reviewer_6E9a", "summary": "The authors propose a neural ODE-based GRU for use in partially observable RL. They effectively \"filter\" GRU recurrent states using an ODE gate, before feeding the filtered state back into the GRU. They use a variational approach, where their model $g$ spits out a mean and variance. They find that training with just reward results in a highly-variable latent state $z_t$, so they constrain $z_t$ via an additional KL loss term with the prior recurrent state.\n\nThey go on to evaluate their method across partially observable and meta RL tasks. They examine cartpole with masked velocity/position, as well as various PyBullet/MuJoCo masked control tasks, comparing against SAC+LSTM and a few other methods. They go on to evaluate tasks with irregularly-spaced observations in time. The idea is that a continuous temporal representation using the ODE is more amenable to continuous time than a discrete RNN.", "questions": "- If you are going to modify the internals of a GRU, I think it makes sense to write out the full GRU equations.\n- \"We found that the reward signals could help the agent learn unobservable information, leading to improve performance of the agent in PO environments.\"\n    - What does this mean? What is \"unobservable\" information in this context? Generally POMDP observations are modeled as $o \\sim O(s)$, in that the state is always indirectly observed.\n- Line 135/136: Are you providing the true Markov state to your model via the loss function? This limits your approach to tasks where we know the underlying state, like toy tasks. After reading the training section, I don't think this is the case but it might confuse readers.\n- I would be interested in further information/experiments about highly-variable recurrent states and how a KL regularizer helps\n    - I do also wonder how responsible this is for your results. How much is the ODE that is helping compared to the variational state constraints.", "limitations": "- The authors claim that their method performs well in discrete tasks, such as atari. This is to be expected, as a continuous-time representation does not make sense in these scenarios. \n- They also find that solving ODEs is computationally expensive ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Their method is novel -- I was not able to find other papers using ODEs within RNNs for POMDPs.\n- Their half-cheetah experiment is very promising, showing that their method matches oracle performance while others cannot\n- The paper is generally well written", "weaknesses": "- Although their method seems interesting, it seems impractical given current hardware. They must solve an ODE at each timestep (even during inference). Given that interesting RL tasks require collecting tens/hundreds/thousands of millions of transitions, their method is likely to be restricted to toy tasks, at least for now.\n- Many of their experiments are limited to 1M frames, which seems low. This is likely due to the high computational complexity of their method. For most environments, it is not clear that their conclusions would hold at 10M or 50M steps as their policies do not appear to have converged. In their defence, the half-cheeta experiment appears very promising, and the Ni et al. paper they reference trains the PyBullet tasks for the same number of environment steps.\n- They do not ablate components of their approach", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the positive aspects of our paper, and we will address the reviewer’s concerns in the following parts.\n\n**W1**: computation \n\nWe agree with your statement that calculating ODE at each step requires additional computational resources. However, we would like to clarify that in our paper, we primarily used the Euler method to approximate the solution of the differential equation. Compared to not performing ODE iterations, this method does not significantly increase the computational complexity. Therefore, we believe that there is potential for this work to be extended to more environments.\n\n**W2**: frame limitation\n\nAbout the number of frames, not only us, all the baselines we compare are limited to about 1M frames. According to the time cost, running 10M or 50M frames takes a long time for all methods.\n\n**W3**: ablation components\n\nWe add some ablation studies to answer Q4. Due to time constraints, we only conducted experiments in some environments and listed typical problems.\n\n**Q1**: GRU equations\n\nR1: There are some errors in Openreview that do not display the equations correctly, which we will directly add the equations in the appendix.\n\n**Q2**: \"unobservable\" meaning\n\nR2: In this context, the term 'unobservable' refers to the velocity/position information that the agent cannot directly obtain. This information is not contained in the observation $o$. Previous research has demonstrated that adding reward signals to inputs can improve performance. As discussed in our paper, the reward function typically consists of $r(s, a)=d(s,s^*)+c||a||_2$, which can be considered as computed from the state $s$ and action $a$ (where the observation $o$ represents a partial value of the state $s$). Therefore, in reverse, if the reward $r$, action $a$, and observations $o$ are known, the unobservable state can be inferred through computation.\n\n**Q3**: observation in the loss function\n\nR3: Sorry for our writing makes you misunderstanding, we provide the observation instead of the state. In lines 135/136, we aim to show that the reward signals consist of action and state, as the observations represent a partial value of the state, the agent could infer the unobservable information from reward, actions, and observations. We will modify  the sentence into \"As the reward function typically consists of $r(s, a)=d(s,s^*)+c||a||_2$, where state $s$ is implicitly measured by the environments and can not be fully observed.\"\n\n**Q4**: ablation study\n\nR4: We remove the variational process for the ablation study. During the training process, there are significant performance oscillations. The best performance is good, but the average is not very high (in Walker-P the performance is 1246±193). Then, we only ablated the KL divergence constraint in the loss function and found that the experimental results depended on the environment. In the Walker-P environment, the drop in the average performance was not significant, but the variance increased substantially (1355±201). In the Hopper-P environment, the mean performance was greatly affected (1931±176). Due to time constraints, we were unable to conduct more experiments on other environments. However, based on the current situation, we believe that adding the KL divergence constraint and the variational process can better constrain the learning of the context variable. For comparing ODE with the variational state. We remove the ODE process and variational process to do an ablation study, the performances are 1057±78 and 1246±193 respectively in the Walker-P task, which shows the significant roles of both processes in our method."}, {"review_id": "muoRYV8jOx", "reviewer": "Reviewer_BSsg", "summary": "The paper proposes GRU-ODE, which encodes historical observations, actions, and rewards into a latent variable. This model is based on recurrent networks and modifies the GRU topology by separating its memory cell into time-continuous states. The integration of variables is accomplished through neural ODEs. Given the differential process, this formulation can extract more accurate unobservable information from partially observable settings. This approach outperforms baselines in several partially observable and Meta-RL control tasks and irregular partially observable tasks.", "questions": "-\tIs this the first combination of GRU-ODE – or the first applied to control? \n-\tCould you compare this recent SOTA POMDP formulations e.g., Dreamer with the same partial inputs?  \n-\tHow different is the performance if the same latent context variable (i.e., same GRU-ODE encoder) is used for both the action and critic?\n-\tCurrently, only positions / velocities are removed? How does performance change if multiple observation information is removed. Would it be possible to evaluate this for some environments? \n-\tFor $x_t$ why does it have to be a concatenation of $o_{0:t}, a_{0:t}$ and $r_{0:t}$? How much does the performance vary when you remove this extra signal? \n-\tHow is the incremental $dt$ varied in the numerical ODE solver for irregular observations? If the $\\delta = 0.05$ is varied, does it cause issues when the time is different between the generative process and model?\n-\tCan you relax the deterministic policy function assumption? Would this cause issues with differentiating $a(t)$ wrt to time. \n-\tHow is discount formulated?\n-\tHow does this formulation perform in stochastic settings?\n-\tWould it be include a metric that ranks performance conditioned on the time budget? \n-\tWas there any pretraining?", "limitations": "-\tThe authors have adequately addressed the concerns about the  Model Complexity. This is because the proposed GRU-ODE model, while powerful, is likely more complex than traditional models. This could make it more computationally intensive and harder to implement.\n-\tFurthermore, these setting are relevant for lower-level control e.g., the Hopper tasks. However, for environments with discrete dynamics these might not be relevant.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-\tThe use of ordinary differential equations (ODEs) allows the model to handle continuous time dynamics, which is a more accurate representation of many real-world systems. This can lead to more accurate and realistic models. \n-\tThe proposed GRU-ODE model is designed to handle partially observable Markov decision processes (POMDPs), a challenging problem in reinforcement learning. By encoding historical observations, actions, and rewards into a latent variable, the model can extract unobservable information from the environment. I found this quite interesting specifically, the results presented in Fig~3 are quite convincing. \n-\tThe paper demonstrates that the proposed method is robust against irregular observations, which is a significant advantage in real-world applications where observations may not always be regularly or uniformly sampled.\n-\tThe authors show that their approach outperforms baselines in several regular partially observable control tasks. \n-\tIts nice to see that the encoder and the policy network together from the same data buffer. Simplifies the training process.", "weaknesses": "-\tBecause networks were trained from the same data buffer – does this cause exploration problems? Potentially why this approach is unable to handle sparse rewards?  \n-\tThe models performance should be tested on robotics tasks e.g., Franka kitchen(https://github.com/Farama-Foundation/D4RL/tree/master/d4rl/kitchen/third_party/franka). It's unclear how well the model would generalize to other slightly more complicated dynamics.\n-\tThe model makes several assumptions, such as the use of ordinary differential equations (ODEs) to model the dynamics of the system. If these assumptions do not hold in a particular application, the model's performance will be impacted i.e., the stationarity and time-dependence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the contributions of our paper and giving us constructive suggestions. We have endeavored to address all of your inquiries and provide explanations for the issues raised in the weaknesses section. We aspire to resolve your concerns about assumptions and ablation studies and earn your recognition for the contributions of our paper. We look forward to receiving your feedback and further questions.\n\n**W1**: exploration and sparse reward\n\nIn our experiment, there is no exploration problem, as we use the Gaussian network for policy learning. From our perspective, rewards are crucial information that contains relevant details about states and actions in POMDP settings. Additionally, since we use recurrent neural networks as encoders, sparse rewards may dissipate dependence over long iterations.\n\n**W2**: robotic tasks\n\nFor further experiments, we train our network in the Kitchen-Burner environment[1]. After training for 100,000 steps, our reward reached around 1.2, which is competitive with relevant methods at the same steps. However, we found that methods such as SAC perform badly. Since we do not utilize demonstrations to guide agent learning, direct training methods are expected to perform less effectively compared to demonstration-guided RL approaches. We believe that, combined with demonstration, our approach can be generalized to relevant tasks.\n\n**W3**: assumption\n\nFor continuous control tasks, almost all the physical engines update the state according to the Euler method, which follows the ordinary differential equations. As our aim is to solve continuous control tasks, we reckon that our method can be used in most environments.\n\n**Q1**: first proposal\n\nR1: We are the first to apply the ODE-based recurrent model to RL and robotic tasks, particularly in the context of POMDP settings.\n\n**Q2**: comparison\n\nR2: With our understanding, the dreamer is applied to pixel input and is not used for POMDP, which is not suitable for our baseline. We have compared a recent SOTA method RMF [2]. **Bold** denotes the best performance on each task.\n\n|  | VRM| RMF|Ours|\n| -| -| -| -|\n| Ant-P| 1040±75 |1048±74|**1243±43**|\n| Ant-V| 981±63| **1021±165**|998±93|\n| Walker-P |1121±167| 1123±176| **1487±81** |\n| Walker-V |551±30 | 586±52|**612±27**|\n| Hopper-P |1851±61| 2133±326| **2455±87** |\n| Hopper-V | **1652±67**| 1495 ± 38|1545±104|\n\n**Q3**: shared/separate encoders\n\nR3: We have compared that using the shared and separate encoders, as shown in the table below. Shared and separate recurrent actor-critic architectures have been compared and show a huge performance gap [2]. In our method, we do not find such big gaps, but there are still some performance gaps.\n\n| |shared|seprarate|\n|-|-|-|\n|Walker-P|1336|1487|\n|Walker-V|584|612|\n\n**Q4**: positions/veloicties\n\nR4: Actually, in the POMDP setting, the state space is divided into two parts, the positions, and the velocities. You can also regard the experimental environment as having only position/velocity information, which is equivalent to the removal of velocity/position information. Thus, multiple observation information has been removed in this paradigm.\n\n**Q5**: input influence\n\nR5: There is a misunderstanding, the input of our encoder is the concatenation of $o_t$,$a_{t-1}$, and $r_t$. We do not take the whole historical information as input. As our proposition, the action $a_t$ and reward $r_t$ could help to infer the unobservable observations, previous works also mentioned that containing action and reward as input could improve the performance[2]. We do ablation study in four scenarios, $[o_t]$,  $[o_t,a_{t-1}]$, $[o_t,r_t]$, and $[o_t,a_{t-1},r_t]$ in Walker-P environments.\n\n|  |o|oa|or|oar|\n|-|-|-|-|-|\n| Walker-P|1276±134|1065±65|1396±74|1487±81|\n\n**Q6**: $dt$ variation\n\nR6: In irregular setting, $dt$ follows $dt\\sim\\mathcal{U}\\left(0,2\\delta\\right]$. Only encoders take $dt$ as input, so there is no inconsistency in our model. $\\delta=0.05$ is the default setting, it could be changed to any rational value.  \n\n**Q7**: deterministic assumption\n\nR7: The deterministic policy assumption is for the sake of rigor. This assumption is still valid in a stochastic setting. For the stochastic policy,  the probability distribution of actions is computed from the state first, and then actions are sampled from this distribution. In this process, the former part is more critical. This part is differentiable and aligns with our assumption. However, the latter half is non-differentiable and does not align with our assumption. Therefore, if $a(t)$ is a random value represented by the probability distributions, our assumption is still valid. However, if $a(t)$ represents the sampled actions, this assumption needs to be revised. Overall, because Gaussian random policy networks are commonly used in modern networks to learn action probability distributions, assuming $a(t)$ is the output of the network, this assumption still holds for stochastic policies. \n\n**Q8**: discount factor\n\nR8: The discount factors keep 0.99 across all the experiments.\n\n**Q9**: stochastic setting \n\nR9: The formulation does not need to be changed in the stochastic setting as explained in R7.\n\n**Q10**: time rank\n\nR10: It is hard to design an evaluation index that both concerns time cost and performance, maybe a figure is meaningful, but we could not show it here. We list the time cost according to the performance in the Walker-P environment.\n\n|   | Time Cost | Performance |\n|-| -| -|\n|Ours|12h|1487±81|\n|VRM|18h|1121±167|\n|RMF|10h|982±339|\n|SAC-LSTM|6h| 971±103|\n|SLAC|3h|277±121|\n\n**Q11**: pretrain\n\nR11: There is no pretraining.\n\n[1] Hakhamaneshi, K., Zhao, R., Zhan, A., Abbeel, P., & Laskin, M. (2021). Hierarchical few-shot imitation with skill transition models. *arXiv preprint arXiv:2107.08981*.\n\n[2] Ni, T., Eysenbach, B., & Salakhutdinov, R. (2021). Recurrent model-free rl can be a strong baseline for many pomdps. *arXiv preprint arXiv:2110.05038*."}, {"review_id": "HzxfiMl20S", "reviewer": "Reviewer_kjqZ", "summary": "This manuscript presents GRU-ODE, a new ODE-based recurrent model designed to tackle POMDP control tasks. By utilizing a numerical ODE solver, GRU-ODE effectively incorporates past observations, actions, and rewards into its context. In comparison to previous RNN-based methods, GRU-ODE offers several advantages such as the ability to handle irregular observations and infer unfamiliar physical environments. Experimental results demonstrate the superior performance of GRU-ODE compared to baselines across standard POMDP tasks, meta RL tasks, and tasks with temporally irregular observations.\n\n**After rebuttal**: The authors addressed most of my concerns and the clarification is valid. I have increased my score to 5.", "questions": "1. Could you elaborate on the distinction between the GRU-ODE module and the model used in prior studies [1, 2]?\n2. How are baseline methods used for tasks with irregular temporal observations?\n\n\n\n[1] Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[J]. Advances in neural information processing systems, 2019, 32.\n\n[2] Lechner M, Hasani R. Learning long-term dependencies in irregularly-sampled time series[J]. arXiv preprint arXiv:2006.04418, 2020.", "limitations": "The authors adequately acknowledge the limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental results in this work adequately verify the proposed GRU-ODE method, which demonstrates competitive evaluation performance.\n2. The paper is well-organized and easy to read.", "weaknesses": "1. My main concern is the applicability of the proposed method. The authors assume certain conditions, such as the structure of the reward function and the underlying physics of the task, which may limit the applicability of GRU-ODE. Furthermore, ODE-based encoders fall short in handling discrete game environments such as Atari, as mentioned in line 337.\n2. The baselines considered are not sufficiently recent, which weakens the persuasiveness of the evaluation. It is recommended to include results from published papers from the past two years, such as [1, 2].\n3. Using ODEs would inevitably incur more training time and inference time. Although the authors compare runtimes with different numerical solvers, it is recommended to also include comparisons to baseline methods.\n\n\n\n[1] Ni T, Eysenbach B, Salakhutdinov R. Recurrent model-free rl can be a strong baseline for many pomdps[J]. arXiv preprint arXiv:2110.05038, 2021.\n\n[2] Morad S, Kortvelesy R, Bettini M, et al. POPGym: Benchmarking Partially Observable Reinforcement Learning[J]. arXiv preprint arXiv:2303.01859, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the contributions of our paper and giving us constructive suggestions. We will address the reviewer’s concerns in the following parts. We address all of your concerns and show distinctions between our methods with previous works [1]. We hope that our work will receive your recognition and garner positive feedback. We sincerely hope to receive more of your questions and feedback.\n\n**W1**: applicability concern\n\nIn our paper, the reward function proposition is employed to provide a theoretical analysis of the rationality behind concatenating both actions and rewards with observations as inputs. Previous works typically validate the rationality of this setting just through experimental results. Actually, the structure of the reward function does not need to be conditioned on such a format. Furthermore, as we mainly focus on continuous control tasks, the underlying physics of such tasks are quite similar, which are generally updated according to the Euler method, thus our method is suitable to apply in such environments. It is true that ODE-based encoders fall short in handling discrete game environments. However, we do not pay much attention to this, our aim is to solve continuous control tasks. \n\n**W2**: performance comparison\n\nAs for the baseline problems, we have rerun the method RMF proposed in [2], the comparison results are listed as follows. **Bold** denotes the best performance on each task. Our method holds the best performance in 4 out of 6 environments.\n\n|          | SLAC    | VRM         | SAC-LSTM | RMF          | Ours        |\n| :------- | :------ | :---------- | :------- | :----------- | :---------- |\n| Ant-P    | 950±129 | 1040±75     | 946±47   | 1048±74      | **1243±43** |\n| Ant-V    | 663±87  | 981±63      | 690±34   | **1021±165** | 998±93      |\n| Walker-P | 277±121 | 1121±167    | 971±103  | 1123±176     | **1487±81** |\n| Walker-V | 138±25  | 551±30      | 491±38   | 586±52       | **612±27**  |\n| Hopper-P | 222±21  | 1851±61     | 1236±39  | 2133±326     | **2455±87** |\n| Hopper-V | 310±41  | **1652±67** | 890±43   | 1495 ± 38    | 1545±10     |\n\nIt is not fair to compare our method with POPGYM [3], as the environments in POPGYM are almost game and navigation tasks. The control tasks only contain Cartpole and Pendulum, which is similar to the Gym/MuJoCo environment. So we do not compare our method with POPGYM.\n\n**W3**: runtime\n\nFor the time cost of concern, we evaluate all the baselines on the Walker-P environments with 1M steps.\n\n| Methods  | Time cost |\n| -------- | --------- |\n| Ours     | 12h       |\n| RMF      | 10h       |\n| VRM      | 18h       |\n| SAC-LSTM | 6h        |\n| SLAC     | 3h        |\n\n**Q1**: distinction\n\nR1: Regarding the distinction, we have made modifications to the computation order of the ODE and GRU updates. This alteration was motivated by the fact that ODE-RNN [1] was originally designed to encode time series backward in time from $t_N$ to $t_0$. Changing the update order has proven beneficial for forward-in-time encoding in our scenario. Additionally, we utilize variational outputs at each step instead of just the last step, and we optimize the KL divergence loss function to regularize the embeddings. Experiments demonstrate that this constraint enhances the agent's performance in the context of RL. In this paper, our primary contribution lies in applying an ODE-based recurrent model to POMDP reinforcement learning environments and putting forward a theoretically sound proposition for combining ODEs with continuous control and partially observable tasks. As shown in Equation (5), we combine continuous action with differential equations, which proposes a method to calculate the action differentiation. As far as I know, this is the first instance of utilizing ODEs within RNNs for POMDPs.\n\n**Q2**: irregular baselines\n\nR2: In irregular tasks, to fairly compare, we assume the incremental $dt$ is known to the agent. In our method, $dt$ is one of the inputs of the $\\text{ODESolve}$() function, as the update equation is Equation (2). In the other compared baselines, the incremental $dt$ is added as an additional input, which allows the agent to infer the irregular environments.\n\n[1] Rubanova, Y., Chen, R. T., & Duvenaud, D. K. (2019). Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32.\n\n[2] Ni, T., Eysenbach, B., & Salakhutdinov, R. (2021). Recurrent model-free rl can be a strong baseline for many pomdps. *arXiv preprint arXiv:2110.05038*.\n\n[3] Morad, S., Kortvelesy, R., Bettini, M., Liwicki, S., & Prorok, A. (2023). POPGym: Benchmarking Partially Observable Reinforcement Learning. *arXiv preprint arXiv:2303.01859*."}]

Author rebuttal:
We are incredibly grateful for the positive assessment and highly detailed and constructive feedback on our paper. In this rebuttal, we have addressed the comments of the review team. This has led to multiple improvements including a better exposition of the proposition, more complete ablation studies, and new applications. Below we address each question and minor questions point by point. Due to character limitations, we have simplified the tables in the rebuttal. The complete tables can be referenced in the PDF file. We hope that our contributions will receive your positive evaluation.

Meta-review: This paper tackles the problem of partial observability in deep reinforcement learning (RL). The authors introduce a method combining recurrent neural networks (GRU) and Neural ODE for the encoder of actor/critic networks. 
The paper initially received mixed reviews, with two accept and two reject recommendations. The main concerns pointed out by reviewers related to the novelty of the approach, its applicability for more complex control/robotics tasks and for partial observability including perception (e.g. from pixels), its scalability due to the the Neural-ODE's cost, and the experimental comparison with stronger and more recent baselines. The rebuttal resolved the most important points raised by the reviewers, who were inclined to accept the paper.

The AC has thoroughly reviewed the submission and the discussions. The problem of partial observability is central in robotics and RL. Although the approach may seem incremental given the extensive attempts for using neural-ODE in dynamical models, the combination N-ODE-GRU for improving POMDP solutions open a promising paths. It could inspire researchers for improving the current limitation of the works, e.g. its scalability with advanced solvers, or including perception for more advanced robotic tasks. The rebuttal also clarifies several points regarding experiments. 
Therefore, the AC recommends acceptance but highly recommends the authors to carefully take into account the remarks of the reviewers when preparing the final paper.
Decision: Accept (poster)