Title: ODE-based Recurrent Model-free Reinforcement Learning for POMDPs
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural ode

POMDPs

reinforcement learning
Abstract: Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combines with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.

ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

Abstract

Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combined with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.

1 Introduction

Conventional reinforcement learning (RL) is typically cast as a problem of solving fully observable Markov decision process (MDP) tasks which are trained and tested on the same task. However, most practical applications require the agents to handle some degree of partially observable and irregular observations. Humans are always good at solving these kinds of tasks by extracting crucial information from past observations and actions, while conventional RL agents do not have the ability to extract information relevant to the tasks.

In recent works, several categories of methods have been proposed to solve PO problems. The most straightforward one is to include all historical observations as input [Lee et al., 2020]. However, this kind of method is impractical in application owing to the dimension increment of the observation inputs. Another category is based on model-free methods which generally use recurrent neural networks (RNNs) as memory units and function approximators to extract historical task-related information to solve partial observable problems [Igl et al., 2018, Kapturowski et al., 2019, Han et al., 2019, Jaderberg et al., 2019]. Recurrent neural networks are commonly employed to encode historical state transitions into contextual representations, leveraging their capacity to process sequential information. These representations are then used in conjunction with the current state to find the optimal policy. This baseline is simple and efficient, as it obtains near-optimal results and only requires changing the policy and value network into a recurrent version. The third category considers using a model to estimate a belief state from historical transitions [Lee et al., 2020]. Agent receives

belief states as observations that contain useful information to solve the tasks, this category is much similar to the second one. However, almost all notable breakthroughs have been showcased in discrete-time problems such as games and discrete control problems [Mnih et al., 2015, Silver et al., 2016], whereas most physical and biological systems in the real world are inherently continuous in time and follow differential equation dynamics.

Differential equations are commonly considered the benchmark for modeling physical mechanisms [Schölkopf et al., 2021]. They permit people to forecast the future behaviors of physical systems and the statistical interdependencies among variables. Moreover, they provide physical insights, explain the system's functioning, and justify reasons for their causal structure [Peters et al., 2022, Zhang et al., 2023b]. A discretized control system will converge to an ODE as the time increment approaches zero. However, there is no consensus on how to choose the right discretization stepsizes and account for irregular observation times. In current paradigms, stepsizes are fixed in advance and vary in environments. For example, in classic control tasks (such as CartPole and Pendulum), observations are sampled by regularly discretized timesteps [Brockman et al., 2016].

In this study, similar to context-based RL, we developed an ODE-based recurrent model (GRU-ODE) that encodes historical observations, actions, and rewards into a latent variable. Previous works have shown that ODE-based recurrent models could simulate the physical dynamic system in an auto-regressive fashion, such as Latent-ODE [Rubanova et al., 2019] and ODE-LSTM [Lechner and Hasani, 2020]. The GRU-ODE is based on recurrent networks, which modify the GRU topology by separating its memory cell into time-continuous states. At each time step, the model first iterates using the conventional GRU equations and then computing the differentiation based on the ODEs [Chen et al., 2018]. We then propose a method to solve PO tasks by training the GRU-ODE and the actor-critic algorithm. We show owing to the differential process, our model could extract more accurate unobservable information from partially observable environments. Our approach outperforms baselines in several regular PO and Meta-RL control tasks and irregular PO tasks.

2 Preliminary and Related work

Typical POMDPs are used to describe decision or control problems in which the underlying states of the environments cannot be directly observed. A POMDP is a tuple ( S , A , O , T , O , R , γ ) . S is a set of states, A is a set of actions, and T : S ×A is the state-transition probability function. Let O be the observation set and let O : S × A × O → [0 , 1] be the observation probability. The reward function R : S × A decides the reward during state transition and γ is the discount factor. The objective is to learn a policy function to maximize expected discounted rewards E [ T -1 ∑ t =0 γ t r t +1 ] with a finite

horizon T .

Common PO tasks include scenarios of partially occluded states [Heess et al., 2015], randomly dropped frames [Hausknecht and Stone, 2015], egocentric images [Zhu et al., 2017], and randomly noised observations [Meng et al., 2021]. These kinds of problems are hard to solve because the dimension of historical observations grows linearly with timesteps. Prior works consider learning a belief state to encode underlying environments by extracting information from historical state transitions [Zintgraf et al., 2019]. Some other methods use memory-based policy, which takes the entire history states as inputs [Lee et al., 2020]. Recent works have widely used recurrent models to equip relevant algorithms [Ni et al., 2022, Han et al., 2019]. These recurrent methods could be further divided into model-free and model-based methods according to whether their training objectives contain transition functions.

In the meta-RL setting, some indicator of the task is unobserved and methods are quite similar to that of PO tasks. Prominent approaches utilize recurrent networks for fast adaption [Duan et al., 2016]. Recent studies demonstrate that the recurrent model-free algorithm serves as a robust baseline for meta-RL [Ni et al., 2022]. This is accomplished by equipping the policy network with a recurrent model, which aggregates past state and action transitions into an auxiliary context input [Zintgraf et al., 2019]. Recent works also consider training non-recurrent models to encode the context [Mu et al., 2022, Guo et al., 2022]. These approaches can be categorized into context-based meta-RL methods. Another category uses policy gradients [Finn et al., 2017, Xu et al., 2018a] or hyperparameters [Xu et al., 2018b] to learn from aggregated experience are defined as gradient-based methods.

Neural ODEs have been widely used to tackle irregular time series [Chen et al., 2018]. Standard RNN treats observations as a sequence of tokens and does not account for variable timestep between observations. When facing irregular-sized data, another interpolation or generative adversarial network is used to perform interpolation and imputation. However, these methods are not robust in unseen environments. Chen et al. [2018] use residual neural networks with time-invariant dynamics to solve the ODE initial-value problem (IVP). Recent works build continuous time models by adding ODEs to the recurrent cells update process [De Brouwer et al., 2019, Kidger et al., 2020]. CT-LSTM [Mei and Eisner, 2017] combines the LSTM architecture with the continuous-time neural Hawkes process. ODE-LSTM [Lechner and Hasani, 2020] transforms the internal dynamical flow of LSTM to a continuous-time model and demonstrates its efficacy in learning kinematics simulation. Liquid time-constant networks (LTC) [Hasani et al., 2021, Lechner et al., 2020] represent dynamic systems with liquid time constants coupled to their hidden states and compute outputs by numerical differential equation solvers.

Recent studies in scientific machine learning have attempted to utilize dynamic encoding or physical mechanisms within their respective fields, such as DeepONet [Lu et al., 2019] and PINNs [Cuomo et al., 2022, Krishnapriyan et al., 2021]. Incorporating physical insights has been shown to enhance the efficiency of neural networks and the ability of representation learning [Zhang et al., 2022, Yang et al., 2021, Han et al., 2023, Zhang et al., 2023a]. Some previous works combine ODEs with model-based RL. Du et al. [2020] use Latent-ODE to learn the transition model to solve semi-Markov decision processes (SMDPs), which also shows variables encoded by Latent-ODE capture unobservable state representations in PO environment. Yildiz et al. [2021] infer the unknown state evolution differentials with Bayesian neural ODEs. Some other works use ODEs to learn the dynamic or transition functions and apply them to continuous and irregular time interval tasks [Salehi et al., 2022, Ainsworth et al., 2021].

3 Approach

This section first describes constructing ODE-based recurrent models to encode historical information into context variables for model-free RL. Then, we describe the overall training objective and how to use our model to optimize the policy network and value function.

In the typical POMDP setting, the reward and transition functions share some common structures across the MDPs. Some embeddings must represent the detail of the task information, which could not be accessed in advance. Context-based RL methods attempt to solve partial observable problems by allowing the policy network to receive an auxiliary context variable which includes historical information. The context variable at time step t is inferred from the agents' experience up to the current states,

There should be a superscript i to denote various episodes, we drop it for ease of notation. In POMDP environments, observations are denoted as o t instead of s t , as the underlying states are only partially observable and must be inferred based on the available observations. Based on the above τ : t , it is sufficient to encode historical information into a context variable z t , rather than modeling transition and reward dynamics which consist of millions of parameters. Generally, the memory-based policy is defined as π ( a t | τ : t ) , conditioning on the whole history.

3.1 Model definition

Following ODE-RNN [Rubanova et al., 2019], we extend hidden state transitions in RNNs to continuous-time dynamics defined by Neural ODEs. As RNNs with exponentially-decayed hidden state follow an implicit ODE of the form dh ( t ) dt = -τh with h ( t 0 ) = h 0 , where τ is a parameter, the hidden states could be modeled by Neural ODEs. Therefore, we propose our ODE-based recurrent model.

GRU-ODE. In our models, transformations of hidden states and transitions between latent states are computed using recurrent cells and neural ODEs, respectively. The resulting model is

where the context variable z t is defined as a Gaussian distribution parameterized by mean µ h t and standard deviation σ h t . x t is the concatenation of o t , a t -1 and r t . The gate of the GRUCell is controlled by the output of ODESolve h t , instead of its own output ˜ h t . The ODESolve are numerical differential solvers implemented by neural networks

where f θ computes the differential of h ( t ) which are implemented by neural networks. The time increment dt in the numerical ODE solver is fixed for conventional control tasks and varied in irregular observation tasks. The input of our model x t is the concatenation of observations o t , actions a t -1 and rewards r t . We found that the reward signals could help the agent learn unobservable information, leading to improved agent performance in PO environments. As the reward function typically consists of r ( s, a ) = d ( s, s ∗ ) + c || a || 2 , where the first term measures the distance between the current state and the objective state and the latter term contains the penalty of the action magnitude. Therefore, the unobservable state information could be inferred based on tuples of observations, actions, and rewards. To verify the rationality of our proposed model, we simplify the classic control task. In PO dynamic systems, the underlying physics of the task is always supposed to be second-order, the observable states are expressed in terms of either position θ or angular velocity ω . As the state and action space could be denoted as s ( t ) = [ θ ( t ) , ω ( t )] T and a ( t ) respectively, the dynamic system with explicit control is defined as follows:

where A ( · ) is referred to as acceleration field. Simplifying the update equation, this procedure could be modeled as differential equations with explicit actions. The current state could be modeled using ODESolve with initial state as s ( t ) = s (0) + ∫ t 0 f θ ( s ( τ ) , a ( τ )) dτ , which could be solved by neural ODEs [Chen et al., 2018]. Previous neural ODE models were always used for time-series predictions, which do not contain explicit actions. Chiappa et al. [2017] incorporate actions as auxiliary inputs of recurrent cells to build recurrent simulators. We combine them to build our recurrent models. In addition to states, actions and rewards can also be solved as initial value problems.

Proposition 1. The action a ( t ) is differential with time like the state s ( t ) , as the action is controlled by a deterministic policy function π ( · ) . The derivative of a ( t ) with respect to time t could be expressed as

The policy function π ( · ) is implemented with neural networks, so dπ ( s ( t )) /ds ( t ) is the differential between the output and input of the policy network and could be computed by the chain rule. Following the above formulation, the reward function is also differentiable as it is always the linear combination of state s ( t ) and action a ( t ) . Therefore, the update process of GRU-ODE input x ( t ) , the concatenation of state, action and reward, could be modeled as ODEs and solved by ODESolve . As the latent context variable z t reflects the general dynamic across different episodes, we reckon the update process of z t could be modeled in the same way with explicit input x ( t ) . Our model explicitly changes the discrete update into a continuous case, which represents latent variables more precisely. Existing implementations of state transitions perform inaccurate numerical integration routines, which results in rough estimation. For example, in the CartPole task crude numerical solver lead approximations bifurcate near the pole hangs up positions [Yildiz et al., 2021]. The accuracy of the approximation is dependent on the stepsize of the solver. Consequently, continuous methods tend to provide a closer approximation to the true value.

𝐺𝑅𝑈𝐶𝑒𝑙𝑙

! ℎ

!

𝑂𝐷𝐸𝑆𝑜𝑙𝑣𝑒

ℎ

!

𝑀𝐿𝑃

GRU-ODE

𝑔

!

"

𝑧

#

𝜏

:#

GRU-ODE

𝜋

#

(𝑧

$

|𝑜

$

, 𝑧

$

)

Actor

𝑞

%

(𝑄

$

|𝑜

$

, 𝑎

$

, 𝑧

$

)

Critic

𝜏

#%&:#

=

𝑜

#

, 𝑎

#%&

, 𝑟

#

𝑜

0

𝑟

0

𝑎

012

𝑔

!

'

𝑧

#

𝜏

:#

GRU-ODE

𝑜

0

𝑟

0

𝑎

012

𝑜

0

𝑜

0

𝑎

0

𝑎

0

𝑄

0

Encoder for actor

Encoder for critic

Policy network

Value function

𝜏

:$

Buffer

Proposition 2. The global and local truncation errors of the Euler numerical solver are O ( h ) and O ( h 2 ) respectively. As the stepsize h approaches zero, the truncation error tends to lim h → 0 | O ( h ) | = 0 . Assuming the increment dt in the ODEs equals the observation interval, the discretized system will converge to the actual ODE as the time increment approaches zero.

Similar to relevant context-based methods, we use a recurrent ODE-based model to encode past trajectories into a variable. Compared with ODE-based models with encoder-decoder structures, models transformed from typical recurrent models could predict online at each time step. As opposed to standard RNNs, ODE-based RNNs learn the dynamics between observations, rather than having them pre-defined, which enables handling irregular data and modeling continuous sequences without relying on pre-assumptions about the dynamics of the observation sequences [Rubanova et al., 2019]. Moreover, we found that owing to their underlying physical principles, ODEs are suitable for inferring unknown physical information, which is significant in partially observable environments.

3.2 Training procedure

Now we have the GRU-ODE model to represent the context variable, we move on to the question of identifying learning procedure and optimal policy. As mentioned above, with partial observations, latent context z t is used to provide historically unknown information. Prior works have shown using separate recurrent encoders for the actor and critic could achieve high rewards [Meng et al., 2021]. Ni et al. [2022] show the gradient norms of recurrent context encoders vary in the actor and critic resulting in the gradient of critic loss dominating the actors', which will impact the training process of the actor's context encoder. Therefore, we use separate GRU-ODE models for context representation and implement them into Actor-Critic algorithms to improve the final performance.

Model implementation. The general training objective maximizes the discounted reward in an episode. We implement our method based on TD3 Fujimoto et al. [2018] and SAC Haarnoja et al. [2018], so the policy network and value function should be parameterized differently. Deep neural networks are used to represent individual components. There are :

The recurrent context encoder GRU-ODE model g ϕ ( z t | τ : t ) , parameterised by ϕ . We use superscripts a or c to denote the encoder and context variables used for the policy network or the value function.

The conditioned policy network (actor) π θ ( a t | o t , g a ϕ ( z a t | τ : t )) parameterized by θ with context variable z a t .

The conditioned value function (critic) q ψ ( Q t | o t , a t , g c ϕ ( z c t | τ : t )) parameterized by ϕ with context variable z c t .

The network architecture is shown in Fig. 1. The context variable z t is regarded as a posterior and represented by the distribution's parameters, the details are discussed later. Compared to modeling the transition or reward function directly using the network, learning distribution embeddings need fewer parameters which makes inference easier.

Training objectives. The conventional training objective of the policy network π θ ( · ) is to maximize the expected return

After experimental analysis, it was discovered that utilizing J ( θ ) directly in conditioned policy network and value function with recurrent encoders results in significant variation of the embedding variable z t within an episode. This variation can lead to a degradation of the context encoder's representation ability in PO environments. Therefore, we adjust the original training objective with an embedding constraint term to regularize z t . Now, the overall training objective is to maximize

where ρ denotes the trajectory distribution induced by the sample policy. Expectations are approximated by Monte Carlo samples. Conditioned delayed policy update algorithms are implemented in our method, thus J ( ψ, ϕ, θ ) are used to denote actor and critic networks with different parameters. To regularize the context variable z t , we add the Kullback-Leibler (KL) divergence K ( ϕ ) between posterior and prior distributions

We set the prior to our previous posterior, g ϕ ( z t -1 | τ : t -1 ) , with initial prior g ϕ ( z 0 ) = N (0 , I ) . D KL denotes the KL divergence, which could be optimized by the reparameterization trick. Since both terms in K ( ϕ ) are parameterized by Gaussian distributions, the KL-divergence can be analytically calculated as

In this work, past trajectories are encoded by GRU-ODE. Eq. 7 is optimized end-to-end, and λ is the balance weight that supervises the encoder learning objective against the RL objective J ( ψ, ϕ, θ ) , which is objective to find an optimal policy and accurate value function. To improve the efficiency of the training process, we sample fixed-length truncated sequences as mini-batches for training our model, rather than using the whole episodes.

Previous work mentioned that training the encoder and the policy network separately using data from different buffers can prevent the gradients of opposing losses from interfering with each other. Zintgraf et al. [2019] follow this setting to pre-train the encoder in advance before the policy network. We found this setting is unnecessary in practice, the data buffer is shared in our method.

4 Experiments

We experimentally evaluate our models across several PO regular/irregular observation domains. In regular observation domains, we consider conventional partially observable control and meta-RL tasks by employing MuJoCo [Todorov et al., 2012] and PyBullet [Greff et al., 2022] environments. In irregular observation domains, we consider modifying classical environments: Pendulum and CartPole tasks[Brockman et al., 2016] into irregular time intervals with surrogate rewards.

4.1 Partially Observable Classic Control Tasks

The Pendulum and CartPole tasks are classic control tasks for evaluating RL algorithms. For the Pendulum task, the goal is to learn a policy to swing the pendulum up and maintain it at the highest position to obtain more rewards. For the CartPole task, the goal is to learn a policy to preclude the pole from falling down and forestall the cart from running away by exerting force on the cart.

Pendulum

Pendulum-P

Pendulum-V

-200

200

Average Return

400

Average Return

400

600

600

800

800

1000

-1000

Ours

VRM

-1200

1200

SLAC

SAC-LSTM

-1400

1400

10

Thousand Steps

20

30

40

0S

10

Thousand Steps

20

30

40

50

CartPole

CartPole-P

CartPole-V

1000

1000

Average Return

800

Average Return

800

600

600

400

400

Ours

200

VRM

200

SAC-LSTM

SLAC

D

50

100

150

50

100

150

Thousand Steps

Thousand Steps

The observable information consists of the coordinates of the cart, the angle of the pole, and their velocities.

These classic control tasks are relatively easy to solve in fully observable domains. Therefore, these PO tasks can effectively underscore the problem of representation learning. Experiments are performed in PO cases of these two control tasks, in which only velocities or positions could be observed (we use the suffix -V/-P to name). The PO setting is more practically significant, as in many real-world applications, agents may only be able to estimate partial state information.

We compare our method with SLAC, VRM, and SAC-LSTM. SLAC [Lee et al., 2020] combines off-policy model-free RL with representation learning via a sequential stochastic state space model. SLAC was developed for pixel observations, we follow the modifications in Han et al. [2019] to compare it with our method. VRM [Han et al., 2019] is a recent, model-based POMDP algorithm, which uses variational recurrent models to encode context variables and separate representation learning from dynamic programming. In the SAC-LSTM, soft actor-critic algorithms with recurrent networks are used as function approximators. We follow the settings in [Han et al., 2019] to construct the above methods.

As expected, our method succeeds in learning to solve all these tasks and performs better than VRM which is the state-of-the-art method for solving POMDP tasks. While SLAC performs well in the CartPole tasks and less sample efficiency in Pendulum tasks. SAC-LSTM needs more epochs for training and fails to solve the Pendulum-V task. We reckon these results are attributed to the ability of ODEs to model physical systems and infer unknown information.

To assess the capability of our model to deduce unobservable environmental information, we project the latent variable z t in an episode to 2D using UMAP [McInnes et al., 2018]. We estimate the unobservable state information such as the angular velocities in the Pendulum-P task through observations, actions, and rewards (as the rewards are calculated based on state and actions, the unobservable state could be computed in reverse), and scatter plot color based on the logarithmic scaling

Pendulum-P

Pendulum-V

海福

UMAP 2

UMAP2

UMAP 1

UMAP 1

(a)

(b)

AntBLT-P

WalkerBLT-P

HopperBLT-P

2500

Return

2000

1750

VRM

Ours

SAC-LSTM

2000

1750

2000

1500

SLAC

1500

Average

1250

1250

1500

1000

1000

1000

750

750

500

500

500

250

250

200

Thousand Steps

400

600

800

1000

0

200

Thousand Steps

400

600

800

1000

0

0

200

400

Thousand Steps

600

800

1000

WalkerBLT-V

HopperBLT-V

Return

2000

1750

Ours

VRM

SAC-LSTM

1400

1200

2500

2000

1500

SLAC

1000

Average

1250

800

1500

1000

750

600

1000

500

400

500

250

200

200

400

600

800

1000

0

0

200

400

600

800

1000

0

0

200

400

600

800

1000

Thousand Steps

Thousand Steps

Thousand Steps

of their values. Fig. 3 shows the clustering of latent variables corresponds closely to the unobservable physical parameters, which demonstrates the efficacy of our GRU-ODE context encoder.

4.2 Partially Observable Robotic Control Tasks

We evaluate the performance of our proposed method in the PyBullet control environments [Greff et al., 2022] which are more challenging than MuJoCo. The PyBullet environments contain several continuous robotic control tasks and the state of fully observable environments includes the robot's coordinates, trigonometric functions of joint angles, and angular velocities. We modify the observations into PO versions with the same criteria and compare them with the same methods as the PO classic control tasks. As experimental results demonstrated our method obtains policy improvement in the majority of PO robotic control tasks, especially in environments where only the position information is observable. For tasks with only positions observable, our method outperforms the other three methods and presumably shows efficiency in extracting useful information from historical position observations. We reckon that velocity could be simply estimated by one-step differentiation in coordinates and joint angles of the robot, which eases representation and policy learning. Environments with only velocities that could be observed are much harder to solve. However, deducing the position information solely from the given velocity is not practical. In environments where only velocity can be observed, our method performs similarly to VRM. Additionally, our results indicate that the learning processes of the SLAC method exhibited instability, where it intermittently achieved a nearly optimal policy, but frequently converged to a suboptimal one. As a result, the average performance of SLAC was generally less favorable than that of our approach across the majority of PO robotic control tasks. SAC-LSTM fails to solve almost all these tasks.

We further compare our methods with two more baselines RMF (Recurrent Model-Free)[Ni et al., 2022] and TD3-FPOW (TD3 with fixed previous observation window). The complete results are shown in Table 6.

4.3 Meta-learning MuJoCo Control Tasks

We further show our method can also scale to other types of PO tasks, such as Meta-RL, by applying our method to a diverse set of domains, such as point robot and MuJuCo locomotion tasks, which are commonly used in meta-RL literature. Our tasks involve utilizing the semi-circle, which refers to a point robot that navigates along a semi-circular path in search of a sparsely located reward, as well as wind tasks, where a point robot navigates to a fixed goal amidst fluctuating wind conditions. For MuJuCo locomotion tasks, we consider using the HalfCheetah-Dir environment where the agent has

Semi-Circle

Wind

HalfCheetah-Dir

70

100

2500

Returns

60

80

50

2000

Oracle

60

Ours

40

1500

erag

40

Off-policy VariBAD

30

1000

Aver

20

Oracle

500

On-policy VariBAD

20

10

RL2

Ours

0

0.0

0.5

Frames

1.0

1.5

1e6

2.0

0

0

2

Frames

1e5

6

0.0

0.2

0.4

Frames

0.6

0.8

1e8

1.0

to control two legs run either forward or backward. The MuJuCo-relevant tasks are much harder than the prior point robot tasks.

We compare our method with various methods, such as VariBAD, RL2 [Duan et al., 2016], Pearl [Rakelly et al., 2019], and Oracle policy. VariBAD utilizes a variational, model-based objective to learn task embeddings explicitly. The originally proposed variBAD [Zintgraf et al., 2019] uses PPO [Schulman et al., 2017], and the same method was implemented with SAC [Dorfman et al., 2020]. These two methods could be named on-policy variBAD and off-policy variBAD. An oracle policy can access the POMDP hidden state to transform the POMDP into an MDP, thus this policy could be treated as an upper bound on the performance of related methods. We follow the settings in Ni et al. [2022] to construct the above methods.

Fig. 5 shows that our method outperforms off-policy variBAD and Pearl in the semi-circle and wind environments, and on-policy variBAD and RL2 in the HalfCheetah-Dir environment, both in terms of sample efficiency and asymptotic return. As our model is trained end-to-end, without using pre-trained encoders to represent task contexts like off-policy variBAD, which do not suffer out-of-date representation problems. On-policy variBAD uses autoencoder architecture to reconstruct the state and reward, which are more advantageous to context representation. Our method shows simple recurrent encoder could also stabilize representation learning and improve policy performance.

4.4 Temporally Irregular Observable Control Tasks

In standard RL frameworks such as OpenAI Gym, the states/observations arrive at constant intervals as ∆ t = δ . To contrast the robustness of our continuous-time method with other discrete POMDP methods, we evaluate related methods with irregularly sampled observations. We consider using an irregular sampling scenario in which observations arrive uniformly within a range ∆ t ∼ U (0 , 2 δ ] . We choose the mean time difference δ = 0 . 05 for observation spacings, which is the standard interval setting of the Gym Pendulum task.

Fig. 6 exhibits the final average returns with a standard deviation of VRM, SLAC, and our method

1000

-200

800

=0.05

-400

009

-600

400

Ours

-800

VRM

200

SLAC

-1000

D

Pendulum-P

Pendulum-V

CartPole-P

CartPole-V

in irregular Pendulum and CartPole tasks. The results show our ODE-based recurrent model is more robust with stochastic time increments, which is reasonable as ODEs build discrete-time sequences into continuous ones. This environment has more practical significance, as observation intervals are usually disturbed in practical application.

5 Limitation

Although we have demonstrated that ODE-based encoders have advantages in robotic control tasks, their efficiency in discrete game problems, such as Atari, is not promising. In Atari, the states of the agents are not updated based on numerical differential equations, resulting in the inability to represent the state and action as integration processes. Also, the reward function is typically sparse and predefined. Implementing ODE-based encoders to model related processes does not make sense.

In addition, since the integration process is implemented by neural networks, the training procedure of ODE-based encoders requires more computation resources, which results in increased training time, especially for high-order numerical differential solvers. Although we find that using simple numerical solvers is enough in the majority of tasks (see Appendix Section 10), accelerating the integration process is still significant in solving differential equations and downstream tasks.

6 Conclusion and future works

We incorporate ODE-based recurrent neural networks with model-free reinforcement learning to solve POMDP control tasks. Through our empirical assessment across several partially observable domains, we showed that the policy network and value function conditioned on context variables encoded by GRU-ODE help the agents infer unknown observations and maximize expected returns. We experimentally demonstrate that our method is robust to environmental changes such as irregularity.

Reinforcement learning has been widely used in physical or biological systems to explain some of the mechanisms. However, some mechanisms in the brain are still unknown, such as how the brain arbitrates or allocates control over various sub-systems and the role of the prefrontal cortex and striatum in controlling. The classic leaky integrate and fire (LIF) neuron model is quite similar to ODEs if we ignore the firing process. We reckon combining ODEs with RL could help biologists explore mechanisms in the human brain.

Acknowledgment

This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDA27010404), the Beijing Nova Program (Grant No. 20230484369), the Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZDZX), and the Youth Innovation Promotion Association of Chinese Academy of Sciences.

References

Ainsworth, K. Lowrey, J. Thickstun, Z. Harchaoui, and S. Srinivasa. Faster policy learning with continuoustime gradients. In Learning for Dynamics and Control , pages 1054-1067. PMLR, 2021.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems , 31, 2018.

Chiappa, S. Racaniere, D. Wierstra, and S. Mohamed. Recurrent environment simulators. arXiv preprint arXiv:1704.02254 , 2017.

Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi, and F. Piccialli. Scientific machine learning through physics-informed neural networks: where we are and what's next. Journal of Scientific Computing , 92(3):88, 2022.

De Brouwer, J. Simm, A. Arany, and Y. Moreau. Gru-ode-bayes: Continuous modeling of sporadicallyobserved time series. In Advances in Neural Information Processing Systems , pages 7377-7388, 2019.

Dorfman, I. Shenfeld, and A. Tamar. Offline meta learning of exploration. arXiv preprint arXiv:2008.02598 , 2020.

Du, J. Futoma, and F. Doshi-Velez. Model-based reinforcement learning for semi-markov decision processes with neural odes. Advances in Neural Information Processing Systems , 33:19805-19816, 2020.

Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Greff, F. Belletti, L. Beyer, C. Doersch, Y. Du, D. Duckworth, D. J. Fleet, D. Gnanapragasam, F. Golemo, C. Herrmann, et al. Kubric: A scalable dataset generator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3749-3761, 2022.

Guo, M. Gong, and D. Tao. A relational intervention approach for unsupervised dynamics generalization in model-based reinforcement learning. arXiv preprint arXiv:2206.04551 , 2022.

Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Han, K. Doya, and J. Tani. Variational recurrent models for solving partially observable control tasks. arXiv preprint arXiv:1912.10703 , 2019.

Han, Q. Wang, T. Zhang, Y. Wang, D. Zhang, and B. Xu. Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition. arXiv preprint arXiv:2302.01194 , 2023.

Hasani, M. Lechner, A. Amini, D. Rus, and R. Grosu. Liquid time-constant networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7657-7666, 2021.

Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv:1507.06527 , 2015.

Heess, J. J. Hunt, T. P. Lillicrap, and D. Silver. Memory-based control with recurrent neural networks. arXiv preprint arXiv:1512.04455 , 2015.

Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning , pages 2117-2126. PMLR, 2018.

Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, et al. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science , 364(6443):859-865, 2019.

Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2019.

Kidger, J. Morrill, J. Foster, and T. Lyons. Neural Controlled Differential Equations for Irregular Time Series. arXiv:2005.08926 , 2020.

Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems , 34:26548-26560, 2021.

Lechner and R. Hasani. Learning long-term dependencies in irregularly-sampled time series. arXiv preprint arXiv:2006.04418 , 2020.

Lechner, R. Hasani, A. Amini, T. A. Henzinger, D. Rus, and R. Grosu. Neural circuit policies enabling auditable autonomy. Nature Machine Intelligence , 2(10):642-652, 2020.

X. Lee, A. Nagabandi, P. Abbeel, and S. Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems , 33:741-752, 2020.

Lu, P. Jin, and G. E. Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193 , 2019.

McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Mei and J. M. Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems , 30, 2017.

Meng, R. Gorbet, and D. Kuli´ c. Memory-based deep reinforcement learning for pomdps. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5619-5626. IEEE, 2021.

Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540): 529-533, 2015.

Mu, Y. Zhuang, F. Ni, B. Wang, J. Chen, J. Hao, and P. Luo. Domino: Decomposed mutual information optimization for generalized context in meta-reinforcement learning. Advances in Neural Information Processing Systems , 35:27563-27575, 2022.

Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps. In International Conference on Machine Learning , pages 16691-16723. PMLR, 2022.

Peters, S. Bauer, and N. Pfister. Causal models for dynamical systems. In Probabilistic and Causal Inference: The Works of Judea Pearl , pages 671-690. 2022.

Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

Rubanova, R. T. Chen, and D. Duvenaud. Latent odes for irregularly-sampled time series. arXiv preprint arXiv:1907.03907 , 2019.

Salehi, S. Rühl, and S. Doncieux. Meta neural ordinary differential equations for adaptive asynchronous control. arXiv preprint arXiv:2207.12062 , 2022.

Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612-634, 2021.

Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Xu, Q. Liu, L. Zhao, and J. Peng. Learning to explore via meta-policy gradient. In International Conference on Machine Learning , pages 5463-5472. PMLR, 2018a.

Xu, H. P. van Hasselt, and D. Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems , 31, 2018b.

Yang, Z. Li, G. Song, and L. Cai. Deep molecular representation learning via fusing physical and chemical information. Advances in Neural Information Processing Systems , 34:16346-16357, 2021.

Yildiz, M. Heinonen, and H. Lähdesmäki. Continuous-time model-based reinforcement learning. In International Conference on Machine Learning , pages 12009-12018. PMLR, 2021.

Zhang, T. Zhang, S. Jia, and B. Xu. Multi-sacle dynamic coding improved spiking actor network for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 59-67, 2022.

Zhang, T. Zhang, S. Jia, Q. Wang, and B. Xu. Tuning synaptic connections instead of weights by genetic algorithm in spiking policy network. CoRR , abs/2301.10292, 2023a. doi: 10.48550/arXiv.2301.10292. URL https://doi.org/10.48550/arXiv.2301.10292 .

Zhang, X. Cheng, S. Jia, C. T. Li, M.-m. Poo, and B. Xu. A brain-inspired algorithm that mitigates catastrophic forgetting of artificial and spiking neural networks with low computational cost. Science Advances , 9(34): eadi2947, 2023b.

Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 3357-3364. IEEE, 2017.

Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y . Gal, K. Hofmann, and S. Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. arXiv preprint arXiv:1910.08348 , 2019.

Supplementary Material

In this supplementary material, we provide further information about our implementation details and ablation studies.

7 Truncation Error

The update procedure of the Euler Equation

and the accurate update equation is

The lost item h 2 2 f ′′ ( ξ i ) is the local truncation error of the Euler method. The global truncation error is defined as

In Euler method f ( x, y ) satisfy the Lipschitz condition for y and denote T i +1 = -h 2 2 f ′′ ( ξ i ) is the local truncation error and T = max i | T i | = O ( h 2 )

where e 0 = y ( x -0) -y 0 = 0 .

8 Enivronment Specification

We incorporate the occlusion benchmark proposed by VRM and replace the deprecated roboschool with PyBullet, as the official GitHub repository suggests. We transform the original MDP task into a POMDP version by removing all position/angle-related entries in the observation space for "-V" environments and velocity-related entries for "-P" environments.

{Pendulum, CartPole, AntBLT, WalkerBLT, HopperBLT}-P. The "-P" denotes environments that retain position-related observations by eliminating velocity-related observations.

{Pendulum, CartPole, AntBLT, WalkerBLT, HopperBLT}-V. The "-V" denotes environments that retain velocity-related observations by eliminating position-related observations.

9 Implemnetaion Details

We use the PyTorch framework for our experiments. Some basic hyperparameters about the network architectures are listed below

Also, there are some unique parameters in TD3 and SAC, we report them together

9.1 Encoding procedure of GRU-ODE

The input observations x t is the concatenation of observations, actions and rewards, as we mentioned above. The time increment dt is set to a fixed value in regular observation environments. In this paper, we set dt = 0 . 1 for all POMDP tasks.

9.2 Integrating with TD3

We use separate recurrent encoders for the actor and critic in order to improve the performance of our method. Owing to the policy network of TD3 updating less frequently than the value function. We modify the iteration equation of the policy network to:

D denotes the replay buffer. The state s is replaced with observation o owing to the partially observable environments.

The update of the value function is modified to:

where subscript 1 , 2 denotes two different critic networks implemented in TD3 and targ denotes the target network. The context variable is computed by g ϕ ( z t | τ ; t ) as the policy network and value function update, we omit this procedure to simplify the equation.

9.3 Integrating with SAC

Similar to TD3, we implement our GRU-ODE in SAC. The training loss of the policy network is modified to

and the value function becomes

where symbols have the same meaning as in Eq. 13. The parameter λ is set to 0.5 both in TD3 and SAC algorithms.

10 ODE Solver Comparison

In this ablation study, we ask two questions in relation to numerical integration. (i) how different numerical solvers influence the final performance of POMDP tasks, and (ii) the effect on the training runtime hours. We experimentally evaluate using different numerical methods as ODESolve for integrating ˜ h t in GRU-ODE. From Table 4, we find that there is no obvious relationship between the

complexity of the numerical solvers and the final performance of the PODMP tasks. However, the

RK4 and Heun need much more time for training. We train these methods on a server with NVIDIA TITAN Xp and Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz as GPU and CPU respectively. The following are rough estimates of average run times for the AntBLT-P environments.

Ours (with Euler) 12 hours

Ours (with RK4) 20 hours

Ours (with Heun) 17 hours

As the results show for context variable coding, the simplest numerical Euler method can work well as ODESolve and save training time. We speculate that though the environment is complicated to solve, the context variables always reflect the essential dynamic information of the environments which are more refined than the observations. Thus, simple numerical solvers are enough.

11 Time Cost of various baselines

We evaluate the time costs of different baselines on Walker-P environments. From the results, we find our method does not increase the computation a lot compared to other baselines. We analyze that there are two reasons. (I). Within our implementation of ODESolve(), we employ the "Euler" method for computation. This offers substantial reductions in computation time when contrasted with the default "dopri5" approach (Runge-Kutta of order 5 of Dormand-Prince-Shampine) utilized in ODE-RNN. We observe that employing complex ODESolve() methods does not notably enhance performance. Instead, it increases computational time significantly. (II). We are working within the context of RL tasks. When training RL algorithms, it's not just the ODE-GRU that needs training, other networks such as policy networks and value networks also require training. During the training process, a significant amount of time is consumed by sampling transitions from the data buffer and updating the data buffer. Therefore, in terms of the overall training time of our approach, it doesn't significantly exceed the time required by the previous method that used an RNN as an encoder.

12 Further Performance Comparison

We add two baselines for performance comparison, the RMF (Recurrent Model-free) is a recent SOTA method on POMDP problems and the TD3-FPOW (TD3 with fixed previous observation window) is a modified version of TD3, which takes 3 concatenation frames as the input.

The result shows that our method achieved the best performance in 4 out of 6 environments.

13 Additional ablation study

We study the influence of the shared/separate encoders, and different concatenations of inputs as the ablation study. Due to the time limitation, we conducted these experiments on the Walker-P and Walker-V environments.

The result shows that separate context encoders for the policy network and value function improve the performance.

The result shows that taking the input x t as the concatenation of o t , r t and a t -1 for the context encoder (GRU-ODE) improves the performance.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript presents GRU-ODE, a new ODE-based recurrent model designed to tackle POMDP control tasks. By utilizing a numerical ODE solver, GRU-ODE effectively incorporates past observations, actions, and rewards into its context. In comparison to previous RNN-based methods, GRU-ODE offers several advantages such as the ability to handle irregular observations and infer unfamiliar physical environments. Experimental results demonstrate the superior performance of GRU-ODE compared to baselines across standard POMDP tasks, meta RL tasks, and tasks with temporally irregular observations.\n\n**After rebuttal**: The authors addressed most of my concerns and the clarification is valid. I have increased my score to 5.\n\nSTRENGTHS:\n1. The experimental results in this work adequately verify the proposed GRU-ODE method, which demonstrates competitive evaluation performance.\n2. The paper is well-organized and easy to read.\n\nWEAKNESSES:\n1. My main concern is the applicability of the proposed method. The authors assume certain conditions, such as the structure of the reward function and the underlying physics of the task, which may limit the applicability of GRU-ODE. Furthermore, ODE-based encoders fall short in handling discrete game environments such as Atari, as mentioned in line 337.\n2. The baselines considered are not sufficiently recent, which weakens the persuasiveness of the evaluation. It is recommended to include results from published papers from the past two years, such as [1, 2].\n3. Using ODEs would inevitably incur more training time and inference time. Although the authors compare runtimes with different numerical solvers, it is recommended to also include comparisons to baseline methods.\n\n\n\n[1] Ni T, Eysenbach B, Salakhutdinov R. Recurrent model-free rl can be a strong baseline for many pomdps[J]. arXiv preprint arXiv:2110.05038, 2021.\n\n[2] Morad S, Kortvelesy R, Bettini M, et al. POPGym: Benchmarking Partially Observable Reinforcement Learning[J]. arXiv preprint arXiv:2303.01859, 2023.\n\nQUESTIONS:\n1. Could you elaborate on the distinction between the GRU-ODE module and the model used in prior studies [1, 2]?\n2. How are baseline methods used for tasks with irregular temporal observations?\n\n\n\n[1] Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[J]. Advances in neural information processing systems, 2019, 32.\n\n[2] Lechner M, Hasani R. Learning long-term dependencies in irregularly-sampled time series[J]. arXiv preprint arXiv:2006.04418, 2020.\n\nLIMITATIONS:\nThe authors adequately acknowledge the limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GRU-ODE, which encodes historical observations, actions, and rewards into a latent variable. This model is based on recurrent networks and modifies the GRU topology by separating its memory cell into time-continuous states. The integration of variables is accomplished through neural ODEs. Given the differential process, this formulation can extract more accurate unobservable information from partially observable settings. This approach outperforms baselines in several partially observable and Meta-RL control tasks and irregular partially observable tasks.\n\nSTRENGTHS:\n-\tThe use of ordinary differential equations (ODEs) allows the model to handle continuous time dynamics, which is a more accurate representation of many real-world systems. This can lead to more accurate and realistic models. \n-\tThe proposed GRU-ODE model is designed to handle partially observable Markov decision processes (POMDPs), a challenging problem in reinforcement learning. By encoding historical observations, actions, and rewards into a latent variable, the model can extract unobservable information from the environment. I found this quite interesting specifically, the results presented in Fig~3 are quite convincing. \n-\tThe paper demonstrates that the proposed method is robust against irregular observations, which is a significant advantage in real-world applications where observations may not always be regularly or uniformly sampled.\n-\tThe authors show that their approach outperforms baselines in several regular partially observable control tasks. \n-\tIts nice to see that the encoder and the policy network together from the same data buffer. Simplifies the training process.\n\nWEAKNESSES:\n-\tBecause networks were trained from the same data buffer – does this cause exploration problems? Potentially why this approach is unable to handle sparse rewards?  \n-\tThe models performance should be tested on robotics tasks e.g., Franka kitchen(https://github.com/Farama-Foundation/D4RL/tree/master/d4rl/kitchen/third_party/franka). It's unclear how well the model would generalize to other slightly more complicated dynamics.\n-\tThe model makes several assumptions, such as the use of ordinary differential equations (ODEs) to model the dynamics of the system. If these assumptions do not hold in a particular application, the model's performance will be impacted i.e., the stationarity and time-dependence.\n\nQUESTIONS:\n-\tIs this the first combination of GRU-ODE – or the first applied to control? \n-\tCould you compare this recent SOTA POMDP formulations e.g., Dreamer with the same partial inputs?  \n-\tHow different is the performance if the same latent context variable (i.e., same GRU-ODE encoder) is used for both the action and critic?\n-\tCurrently, only positions / velocities are removed? How does performance change if multiple observation information is removed. Would it be possible to evaluate this for some environments? \n-\tFor $x_t$ why does it have to be a concatenation of $o_{0:t}, a_{0:t}$ and $r_{0:t}$? How much does the performance vary when you remove this extra signal? \n-\tHow is the incremental $dt$ varied in the numerical ODE solver for irregular observations? If the $\\delta = 0.05$ is varied, does it cause issues when the time is different between the generative process and model?\n-\tCan you relax the deterministic policy function assumption? Would this cause issues with differentiating $a(t)$ wrt to time. \n-\tHow is discount formulated?\n-\tHow does this formulation perform in stochastic settings?\n-\tWould it be include a metric that ranks performance conditioned on the time budget? \n-\tWas there any pretraining?\n\nLIMITATIONS:\n-\tThe authors have adequately addressed the concerns about the  Model Complexity. This is because the proposed GRU-ODE model, while powerful, is likely more complex than traditional models. This could make it more computationally intensive and harder to implement.\n-\tFurthermore, these setting are relevant for lower-level control e.g., the Hopper tasks. However, for environments with discrete dynamics these might not be relevant.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a neural ODE-based GRU for use in partially observable RL. They effectively \"filter\" GRU recurrent states using an ODE gate, before feeding the filtered state back into the GRU. They use a variational approach, where their model $g$ spits out a mean and variance. They find that training with just reward results in a highly-variable latent state $z_t$, so they constrain $z_t$ via an additional KL loss term with the prior recurrent state.\n\nThey go on to evaluate their method across partially observable and meta RL tasks. They examine cartpole with masked velocity/position, as well as various PyBullet/MuJoCo masked control tasks, comparing against SAC+LSTM and a few other methods. They go on to evaluate tasks with irregularly-spaced observations in time. The idea is that a continuous temporal representation using the ODE is more amenable to continuous time than a discrete RNN.\n\nSTRENGTHS:\n- Their method is novel -- I was not able to find other papers using ODEs within RNNs for POMDPs.\n- Their half-cheetah experiment is very promising, showing that their method matches oracle performance while others cannot\n- The paper is generally well written\n\nWEAKNESSES:\n- Although their method seems interesting, it seems impractical given current hardware. They must solve an ODE at each timestep (even during inference). Given that interesting RL tasks require collecting tens/hundreds/thousands of millions of transitions, their method is likely to be restricted to toy tasks, at least for now.\n- Many of their experiments are limited to 1M frames, which seems low. This is likely due to the high computational complexity of their method. For most environments, it is not clear that their conclusions would hold at 10M or 50M steps as their policies do not appear to have converged. In their defence, the half-cheeta experiment appears very promising, and the Ni et al. paper they reference trains the PyBullet tasks for the same number of environment steps.\n- They do not ablate components of their approach\n\nQUESTIONS:\n- If you are going to modify the internals of a GRU, I think it makes sense to write out the full GRU equations.\n- \"We found that the reward signals could help the agent learn unobservable information, leading to improve performance of the agent in PO environments.\"\n    - What does this mean? What is \"unobservable\" information in this context? Generally POMDP observations are modeled as $o \\sim O(s)$, in that the state is always indirectly observed.\n- Line 135/136: Are you providing the true Markov state to your model via the loss function? This limits your approach to tasks where we know the underlying state, like toy tasks. After reading the training section, I don't think this is the case but it might confuse readers.\n- I would be interested in further information/experiments about highly-variable recurrent states and how a KL regularizer helps\n    - I do also wonder how responsible this is for your results. How much is the ODE that is helping compared to the variational state constraints.\n\nLIMITATIONS:\n- The authors claim that their method performs well in discrete tasks, such as atari. This is to be expected, as a continuous-time representation does not make sense in these scenarios. \n- They also find that solving ODEs is computationally expensive", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author present a method for doing deep reinforcement learning for POMDPs with uneven time between observations. Their method uses TD3 with Neural-ODE+GRU encoders for the actor and the critic. They test on various partially observed control tasks, as well as tasks with uneven observation times.\n\nSTRENGTHS:\nThe approach is sound and the results are good even though limited. To the best of my knowledge this is a novel approach for dealing with continuous time observation in RL and generally for using Neural-ODEs in RL. The problem of dealing with non-uniform time between observation is interesting for a lot of problems, especially robotics. The presentation is good (except a few minor things which I will highlight below). I find the integration of continuous-time actions with equation (5) clever, I would have gone with a constant action in between timesteps if I was going to tackle this problem. There will be a lot of small complaints below, but keep in mind that my overall opinion of the paper is positive.\n\nWEAKNESSES:\nI think the evaluation environments are a bad choice for POMDPs. Just removing part of the state is a very basic way to simulate partial observability. In the case of observing position and not velocity, think of all of the typical control benchmarks (Deepmind control suite for example) with pixel observations. They're equivalent (you can't observe velocities from a single frame) and yet people mostly treat them as MDPs and ignore the partial observability. How? Because you simply have to stack a couple of frames and use regular deep RL approaches. The partial observability aspect is easily addressed and is simply not interesting. There's no information gathering happening. This is a wider complaint because that's the way that a lot of POMDP papers do evaluation, so I won't take off to many points for this. But as a research community, we really need to sit down and find better benchmarks for POMDPs which highlight their ability to combine information gathering and reward maximization, which is the beauty of the framework.\n\nThe way I read the first result section is this way: we do well on the easy problem, position only (but I'd like to see basic TD3 with a concatenation of 3-4 previous observations as a baseline), but on the harder, actually interesting problem, we do just as bad as the other approaches.\n\nI find the use of non-standard deprecated environments odd, even though you mention they're \"more difficult\" (how so?).\n\nBecause all of the evaluation seems to be on non-standard environments, it's difficult to assess the significance of the results. My intuition is that pendulum from position only is actually easily solved, so it raises suspicion that's you are able to beat everyone else by a large margin. No information on how other algorithms were tuned was provided. Convergence speed, which is what your algorithm seems to be better at, varies wildly depending on hyperparameters.\n\nSome sentences I found hard to read: line 201 to 203, 170. 52-54.\n\nLine 56 is a bit hand-wavy. What do you mean by extract more accurate unobservable information? What the experiments show is faster reward maximization. No evaluation of the method's ability to gather information is shown.\n\nThe sentence about POMDP starting at line 107 is not correct. You're confusing meta-RL and POMDPs. Please give a formal mathematical definition of a POMDP and provide a citation. You will see that for most the the literature the reward function is the same across episodes. Yes, you can transform a meta-RL problem into a POMDP, by adding a hidden task variable as input to the reward function.\n\nEquation (2) $x_t$ is defined too far from it's appearance in the equation, I was confused at first\n\n(minor) Line 178, the comment about \"physical principles\" feels a bit hand-wavy to me. The type of Neural ODEs that are used here are no more physically inspired than typical RNNs; no physics knowledge is given to them. Yes a lot of mechanics is usually written down by engineers and physicists as ODEs, but I don't think that means Neural ODEs have \"underlying physical principles\", unless some actual physics guide the learning.\n\nQUESTIONS:\nWhy are there no experiments on pixels observations? It seems the algorithm is scalable.\n\nWhy did you use the deprecated environments? What makes them harder and why does that matter as long as the comparison is fair?\n\nModel implementation, item 2 and 3. Why do you add back o_t as input to the network. It's already encoded by g, no?\n\nLIMITATIONS:\nIn their limitation section, the authors mention atari. I don't think this is a very important limitation; it's OK, nobody expected ODEs to do well here. What worries me more is the ability to scale to pixels at all. They mention that their method is slower and I suspected that.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}