Variational Recurrent Models for Solving Partially Observable Control Tasks
Authors: Dongqi Han
Kenji Doya
Jun Tani
Venue:  (2020)
Keywords: Reinforcement Learning
Deep Learning
Variational Inference
Recurrent Neural Network
Partially Observable
Robotic Control
Continuous Control

Abstract: In partially observable (PO) environments, deep reinforcement learning (RL) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how to improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: a variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types of PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner.

Published as a conference paper at ICLR 2020
VARIATIONAL RECURRENT MODELS FOR SOLVING PARTIALLY OBSERVABLE CONTROL TASKS
Dongqi Han
Kenji Doya
Cognitive Neurorobotics Research Unit
Neural Computation Unit Okinawa Institute of Science and Technology Okinawa, Japan doya@oist.jp
Okinawa Institute of Science and Technology Okinawa, Japan
dongqi.han@oist. www.aannnnnnttt. ip
Jun Tani*
Cognitive Neurorobotics Research Unit
Okinawa Institute of Science and Technology
Okinawa, Japan
jun. taniiiist..iiiii.... ip
ABSTRACT
In partially observable (PO) environments, deep reinforcement learning (RI.) agents often suffer from unsatisfactory performance, since two problems need to be tackled together: how to extract information from the raw observations to solve the task, and how improve the policy. In this study, we propose an RL algorithm for solving PO tasks. Our method comprises two parts: variational recurrent model (VRM) for modeling the environment, and an RL controller that has access to both the environment and the VRM. The proposed algorithm was tested in two types PO robotic control tasks, those in which either coordinates or velocities were not observable and those that require long-term memorization. Our experiments show that the proposed algorithm achieved better data efficiency and/or learned more optimal policy than other alternative approaches in tasks in which unobserved states cannot be inferred from raw observations in a simple manner"
INTRODUCTION
Model-free deep reinforcement learning (RL) algorithms have been developed to solve difficult
control and decision-making tasks by self-exploration (Sutton & Barto 1998 Mnih et al. 2015 Silver et al. 2016). While various kinds of fully observable environments have been well investigated,
recently, partially observable (PO) environments (Hafner et al. 2018 Igl et al. 2018 Lee et al. 2019
Jaderberg et al. 2019) have commanded greater attention, since real worl-world applications often need to tackle incomplete information and non-trivial solution is highly desirable.
There are many types of PO tasks; however, those that can be solved by taking the history of observations into account are more common. These tasks are often encountered in real life, such as videos games that require memorization of previous events (Kapturowski et al. [2018] Jaderberg al. 2019) and robotic control using real-time images as input (Hafner al.] 2018 Lee al. 2019). While humans are good at solving these tasks by extracting crucial information from the past observations, deep RL agents often have difficulty acquiring satisfactory policy and achieving good data efficiency. compared to those in fully observable tasks (Hainer et al. [018]] Lee et al. 2019).
For solving such PO tasks, several categories of methods have been proposed. One simple, straight- forward solution is to include history of raw observations in the current "observation" (McCallum 1993 Lee et al. 2019). Unfortunately, this method can be impractical when decision-ne maaionn requires a long term memory because dimension of observation become unacceptably large if long history included.
"Corresponding author.
Codes are available
wttp:///mm....ttototoee/aaaaaaaa onal kecurrent-Moos -

---

Published as a conference paper at ICLR 2020
Another category is based on model-free RL methods with recurrent neural networks (RNN) as func- tion approximators (Schmidhuber, 1990; 1991; gg et al. 2018 Kapturowski et al. 2018 Jaderberg et al. 2019), which is usually more tractable to implement. In this case. RNNs need to tackle two problems simultaneously (Lee et al. 2019): learning representation (encoded by hidden states of the RNN) of the underlying states of the environment from the state-transition data, and learning to maximize returns using the learned representation. As most RL algorithms use a bootstrapping strategy to learn the expected return and to improve the policy (Sutton & Barto 1998), is challenging to train the RNN stably and efficiently, since RNNs are relatively more difficult to train (Pascanu al. 2013) than feedforward neural networks.
The third category considers learning model of the environment and estimating belief state, extracted from sequence of state-transitions (Kaelbling et al., 1998; Ha & Schmidhuber 2018] Lee al. 2019). The belief state is an agent-estimated variable encoding underlying states of the environment that determines state-transition and rewards. thus be taken as "observations" of an RL agent that contains complete information for solving the task. Therefore, solving a PO task is segregated into a representation learning problem and a fully observable RL problem. Since fully observable RL problems have been well explored by the RL community, the critical challenge here is how to estimate the belief state.
belief states can
In this study, developed variational recurrent model (VRM) that models sequential observations and rewards using a latent stochastic variable. The VRM is an extension of the variational recurrent neural network (VRNN) model (Chung et al. 2015) that takes actions into account. Our approach falls into the third category by taking the internal states of the VRM together with raw observations as the belief state. We then propose an algorithm to solve PO tasks by training the VRM and feed-forward RL controller network, respectively. The algorithm can be applied in an end-to-end manner, without fine tuning of a hyperparameters.
We then experimentally evaluated the proposed algorithm in various PO versions of robotic control tasks. The agents showed substantial policy improvement in all tasks, and in some tasks the algorithm performed essentially as in fully observable cases. In particular, our algorithm demonstrates greater performance compared to alternative approaches in environments where only velocity information observable in which long-term memorization is needed.
2
RELATED WORK
Typical model-based RL approaches utilize learned models for dreaming, i.c. generating state- transition data for training the agent (Deisenroth & Rasmussen] 2011: Ha & Schmidhuber (0188) Kaiser al. 2019) for planning of future state-transitions (Watter al. 2015 Hafner al. 2018; Ke et al. 2019). This usually requires a well-designed and finely tuned model so that its predictions are accurate and robust. In our case, we do not use VRMs for dreaming and planning, but for auto-encoding state-transitions. Actually, PO tasks can be solved without requiring VRMs to predict accurately (see Appendix D. This distinguishes our algorithm from typical model-based RL methods.
The work our method most closely resembles is known as stochastic latent actor-critic (SLAC, Lee et al. (2019), in which a latent variable model is trained and uses the latent state as the belief state for the critic. SLAAC showed promising results using pixels-based robotic control tasks, in which velocity information needs to be inferred from third-person images of the robot. Here we consider more general PO environments in which the reward may depend on a long history of inputs, e.g., in a snooker game one has to remember which ball was potted previously. The actor network of SLAC did not take advantage of the latent variable, but instead used some steps of raw observations as input, trattttransiiions
which creates problems in achieving long-tern memorization of reward-relate Furthermore, SLAC did not include raw observations in the input of the critic, which may complicate training the critic before the model converges.
2

---

Published as a conference paper at ICLR 2020
3 BACKGROUND
3.1 PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
The scope of problems study can be formulated into framework known partially observable Markov decision processes (POMDP) (Kaelbling et al. 1998). POMDPs are used to describe decision or control problems in which a part of underlying states of the environment, which determine triteerranstions and rewards, cannot be directly observed by agent.
POMDP is usually defined as 7-tuple AS, R. 7). in which S a set of states, A is a set of actions, and T S X A + p(S) is the state-transition probability function that determines the distribution of the next state given current state and action. The reward function R: decides the reward during state-transition, which can also be probabilistic. Moreover, X is a set of observations, and observations are determined by the observation probability function S x A -> p(X). By defining POMDP, the goal is to maximize expected discounted future rewards >, >fre by learning good strategy to select actions (policy function).
IR
Our algorithm was designed for general POMDP problems by learning the representation of underly- ing states 8, E via modeling obsarvation-transiiions and reward functions. However, is expected to work in PO tasks in which 8, p(s:) can be (at least partially) estimated from the history of observations Ti:t-
3.2 VARIATIONAL RECURRENT NEURAL NETWORKS
To model general state-transitions that can be stochastic and complicated, we employ a modified version of the VRNN (Chung et al.) 2015). The VRNN was developed as recurrent version of the variational auto-encoder (VAE, Kingma & Welling (2013)), composed of variational generation model and a variational inference model. It is recurrent latent variable model that can learn to encode and predict complicated sequential observations o; with a stochastic latent variable 2!.
The generation model predicts future observations given the its internal states,
2t N dia.:,iag(o"
[Mp., of. fprior (d(-1);
E1/Z2 N (My.e di)g(o)))) [My,e, pdecoder (ze, de-2),
(1)
where i's parameterized mappings, such as feed-forward neural networks, and de is the state variable of the RNN, which is recurrently updated by
d: f (d.11:z),;;).
(2)
The inference model approximates the latent variable 2, given x, and de.
æ N (Mz,e diag(o",)) where [M,,,: eencoder (x, d.-1).
(3)
For sequential data that contain T time steps, learning conducted by maximizing the evidence lower bound ELBO, like that in a VEA (Kingma & Welling 2013), where
T
ELBO Σ | D 21
+ t (aa [log (p(x,/=1 x1)-
(4)
where p and q are parameterized PDFs z, by the generative model and the inference model, respectively. In a POMDP, a VRNN can be used to model the environment and to represent underlying states its state variable d, Thus an RL agent can benefit from well-learned VRNN model since d, provides additional information about the environment beyond the current raw observation
3.3 SOFT ACTOR CRITIC
Soft actor-critic (SAC) is a state-of-ther toeff-thearr model-free RL that uses experience replay for dynamic programming, which been tested various robotic control tasks and that shows promising perfor- mance (Haamoja et al.) 2018a|b). SAC agent learns to maximize reinforcement returns as well as entropy of its policy,
obtain rewards while keeping actions sufficiently stochastic.
3

---

Published as a conference paper at ICLR 2020
(b)
Generative model
(c) Inference model
RL Controller
Rr-1
X
A--1
Zt
Internal states
Variational
action
d=-1
d2
de-1
observation
Environment
Xt
Figure 1:
Diagrams of the proposed algorithm. (a) Overview. (b, c) The generative model and the inference model of VRM.
typical SAC implementation can be described as follows. The state value function Vs)), the state-action value function (s, a) and the policy function T[ass) are parameterized by neural networks, indicated by v, A, respectively. Also, an entropy coefficient factor (also known as the temperature parameter), denoted by is learned control the degree of stochasticity of the policy. The parameters are learned by simultaneously minimizing the following loss functions.
Jv(V) Ea,-8 [ (Vo())) Ba,-T, (Qx, a.)) log Ty (s)a)
(5)
[ JQ(A) as)
2 (Qx(s), az) (r(s;, at) + BB - (+2+1)))))
(6)
(n) Eg,88 (a)(())))))) la nh (a, ) s.)) ) as,,,
[a) = Eg.~8 [Eawn,(.)) I-a (a)(a))) s;) - a Marll
(8)
where B is the replay buffer from which 8; is sampled, and Htar is the target entropy. To compute the gradient of J. (Equation.)) 7). the reparameterization trick (Kingma & Welling, 2013 used
action, indicated by a, (st). Reparameterization of action is not required in minimizing J (a)
(Equation. since log ,, (a)s)) does not depends on a.
SAC was originally developed for fully observable environments: thus, the raw observation the current step I was used as network input. In this work, we apply SAC in PO tasks by including the state variable d, of the VRNN in the input of function approximators of both the actor and the critic.
4 METHODS
4.1
VARIATIONAL RECURRENT STATE-TRANSITION MODELS
An overall diagram the proposed algorithm summarized in Fig. Ta). while a more detailed computational graph is plotted in Fig. We extend the original VRNN model (Chung al.) 2015) to the proposed VRM model by adding action feedback, i.e., actions taken by the agent are used in the inference model and the generative model. Also, since we are modeling sate-trrasstion and reward functions, we include the reward rt-1 in the current raw observation for convenience. Thus, we have the inference model (Fig. Tc)). denoted by Φ, as
Z k0 N (no diag(o where [mo,e, 03.1 6l2, (d-1,a1-))
1 The generative model (Fig. ()), denoted by 0 here, is
21 (mat diag o..)) [Ho [4,] = øprior 1, a1-1);
2c 21 ~ N (Mu,,, ()..)
[Mr,er 22.1] = gdecoder (ze, d)-1).
(10)
For building recurrent connections, the choice of RNN types is not limited. In study, the long- short term memory (LSTM) (Hochreiter & Schmidhuber 1997] is used since works well in general cases. So we have d, LSTM(d 21,
4

---

Published as a conference paper at ICLR 2020
(a)
(b)
RL controller
Excecution phase
Deterministic node
Q
Stochastic node
MLP
MLLP
MLL
Generative model
dg.
dg.%g
Environment
Inference model
(c)
Learning phase
RL controller network
Interacting with the environment
Error back- propagation
Figure 2: Computation graph of the proposed algorithm. (a) The RL controller. (b) The execution phase. (c) The learning phase of VRM. a: action; z: latent variable; d: RNN state variable; æ:: raw observation (including reward); Q: state-action value function; V: state value function. A bar on variable means that it is the actual value from the replay buffer the environment. Each stochastic variable follows parameterized diagonal Gaussian distribution.
As in training VRNN, the VRM is trained by maximizing an evidence lower bound (Fig. Ic)
ELBO Σ (Ego [log
x1:-1)]
-DKL (2) -1: Li.t, @1:) pa (21 11: a1:)]}.
(11)
In practice, the first term En, [log pappolaa (wal Zi,t, +1 )) can be obtained by unrolling the RNN using the inference model (Fig. [() with sampled sequences of æ;, Since do and po are parameterized Gaussian distributions, the KL-divergence term can be analytically expressed as
DEL [90 (p:))lpo(zz) = log Tbet T
(Mo.t 14,,,) 0 0.1
1
(12)
20
2
For computation efficiency in experience replay, we train a VRM by sampling minibatchs of truncated sequences of fixed length, instead of whole episodes. Details found in Appendix A.1
Since training of a VRM segregated from training of the RL controllers, there are several strategies for conducting them in parallel. For the RL controller, we adopted smooth update strategy in Haarnoja et al. (2)18)), i.e., performing one time of experience replay every n steps. To train the VRM. one can also conduct smooth update. However, in that case, RL suffers from instability of the representation of underlying states in the VRM before it converges. Also, stochasticity of RNN state variables d can be meaninglessly high early stage of training, which may create problems in RI. Another strategy is to pre-train the VRM for abundant epochs only before RL starts, which unfortunately, can fail if novel observations from the environment appear after some degree policy improvement. Moreover, if pre-training and smooth update are both applied to the VRM, RL may suffer from large representation shift of the belief state.
To resolve this conflict, we propose using two VRMs, which we call the first-impression model and the keep-leand leep-leanning model, respectively. As the names suggest, we pre-train the iisstiimressioon model and stop updating when RL controllers and the keep-learning model start smooth updates. Then take state variables from both VRMs, together with raw observations, as input for the RL controller. We found that this method yields better overall performance than using single VRM (Appendix C).
5

---

Published as a conference paper at ICLR 2020
Algorithm 1 Variational Recurrent Models with Soft Actor Critic
Initialize the first-impression VRM M and the keep-learning VRM Mg. the RL controller C, and the replay buffer D, global step + 0. repeat
Initialize an episode, assign M with zero initial states. while episode not terminated do
Sample an action a, from π ((a, zz)) and execute az, L + L 1+1.
Record (I: at, done)) into B.
Compute 1-step forward of both VRMs using inference models. if == step_start_ then
For N epochs, sample minibatch of samples from B to update Mj (Eq. end if
if > step_start, R.L and mod(t, train inttiiintttall...m LetervaalLLVVV RM) 0 then
Sample minibatch of samples from B to update Ma (Eq. end if
8
if fft > step_start_Rll and mod(t, train_interen _RL) == 0 then
Sample minibatch of samples from to update (Eq. end if' end while
until training stopped
4.2 REINFORCEMENT LEARNING CONTROLLERS
As shown in Fig. a) we use multi-layer perceptrons (MLP) as function approximators for V, Q. respectively. Inputs for the Qe network are (æ,, di, at), and V₂ is mapped from (æ,, de). Following Haarnoja et al. (2)18a), we use two Q networks λ and X2 and compute Q min QA Qxa) in 5 and for better performance and stability. Furthermore, we also used a target value network for computing V in Eq. G as in Haarnoja et al. 2018a). The policy function T, follows parameterized Gaussian distribution A (H,, d., J, diag x+))) where Un and on are also MLPs.
In the execution phase (Fig. bb), observation and reward (x,, ) are received as VRM inputs to compute internal states d, using inference models. Then, the agent selects an action, sampled from (a, .2,), to interact with the environment.
To train RL networks, we first sample sequences of steps from the replay buffer as minibatches; thus, d, can be computed by the inference models using recorded observations H, and actions a, (See Appendix 1.1.2 Then RL networks are updated by minimizing the loss functions with gradient descent. Gradients stop d, so that training of RL networks does not involve updating VRMs.
5 RESULTS
To empirically evaluate our algorithm, we performed experiments in range of (partially observable) continuous control tasks and compared to the following alternative algorithms. The overall procedure is summarized in Algorithm| T For the RL controllers, we adopted hyperparameters from the original SAC implementation (Haarnoja et al. (0088). Both the keep-learning and first-impression VRMs were trained using learning rate 0.0008. We pre-trained the first-impression VRM for 5,000 epochs, and updated the keep-learning VRM every 5 steps. Batches of size 4, each containing a sequence of 64 steps, were used for training both the VRMs and the RL controllers. All tasks used the same hyperparameters (Appendix A.I.
SAC-MLP: The vanilla soft actor-critic implementation (Haarnoja 2018a b). in which each function is approximated by 2-layer MLP taking raw observations as input.
SAC-LSTM: Soft actor-critic with recurrent networks as function approximators, where raw observations are processed through an LSTM layer followed by 2 layers of MLPs. This allows the agent to make decisions based on the whole history of raw observations. In this case, the network has to conduct representation learning and dynamic programming collectively. Our algorithm is compared with SAC-LSTM to demonstrate the effect of separating representation learning from dynamic programming.

---

Published as a conference paper at ICLR 2020
Pendulum
Pendulum
Pendulum velocities only
Pendulum velocities
everege retum
SWerage Perage retur
-500
Ours
SAC-LSTM SAC-MLP
average
-1000
SLAC
-1500
thousand sieps
thousand steps
thousand steps
CartPole
CartPole
CartPole velocities only
CartPole no velocities
1000
1000
1000
average retum
average retum
retum
500
500
500
300
200
300
thousand steps
thousand steps
thousand stepe
Figure 3: Learning curves of the classic control tasks. Shaded areas indicate S.E.M..
The stochastic latent actor-critic algorithm introduced in Lee et al. (2019). which is a ofute-of-the-ar RL algorithm for solving POMDP tasks. It was shown that SLAC outperformed other model-based and model-free algorithms, such as (Igl et al. 2018 Hafner et al. 2)18). in robotic control tasks with third-person image of the robot as observation
SLAC:
Note that in our algorithm, apply pre-training of the first-impression model. For fair comparison, we also perform pre-training for the alternative algorithm with the same epochs. For SAC-MLP and SAC-LSTM, pre-training is conducted RL networks; while for SLAC, its model is pre-trained.
5.1 PARTIALLY OBSERVABLE CLASSIC CONTROL TASKS
The Pendlum and CartPole (Barto et al. 1983) tasks are the classic control tasks for evaluating RL algorithms (Fig. 3 Left). The CartPole task requires learning of policy that prevents the pole from falling down and keeps the cart from running away by applying a (1-dimensional) force to the cart, in which observable information is the coordinate of the cart, the angle of the pole, and their derivatives w.r.t time (i.e., velocities). For the Pendulum task, the agent needs to learn a policy to swing inverse-pente perruss-e up and to maintain it at the highest position in order to obtain more rewards.
We are interested in classic control tasks because they are relatively easy to solve when fully observable, and thus the PO cases can highlight the representation learning problem. Experiments performed in these two tasks, well as their PO versions, in which either velocities cannot be observed or only velocities can be observed. The latter case is meaningful in real-life applications because an agent may be able perceive its own position, but can estimate its speed.
As expected, SAC-MLP failed to solve the PO tasks (Fig- 3p. While our algorithm succeeded in learning solve all these tasks, SAC-LSTM showed poorer performance in some of them. In particular, in the pendulum task with only angular velocity observable, SAC-LSTM may suffer from the periodicity of the angle. SLAC performed well in the CartPole tasks, but showed less satisfactory sample efficiency in the Pendulum tasks.
5.2 PARTIALLY OBSERVABLE ROBOTIC CONTROL TASKS
To examine performance of the proposed algorithm in more challenging control tasks with higher degrees of freedom (DOF), also evaluated performance of the proposed algorithm in the OpenAl Roboschool environments (Brockman et al. 2016). The Roboschool environments include a number
*SLAC was developed for pixel observations. To compare it with our algorithm, we made some modifications of its implementation (see Appendix (.223) Nonetheless, expect the comparison can demonstrate the effect of the key differences aforementioned in Section

---

Published as a conference paper at ICLR 2020
Roboschoolttopper
RoboschoolHopper
RoboschoolHopperr velocities only RoboschoolHoppeer velocilies
2000
2000
aussaga redurn
2000
ausrage
1500
1500
1000
Gura
1000
1000
500
SAC-ALF ALP
500
500
500
1000
500
1000
500
1000
thousand
thousand
thousand
RobeschooWWaleerdd
RoboschoolWalkerd
RobeschoolWolkee22 velocities only RobeschoolWalee22 velocities
1500
averagg raturn
1000
1000
8000
500
500
500
ED0
1000
500
1000
500
1000
thousand steps
thousand slecs
thousand stops
RoboschoolAnt
RoboschoolAnt
RoboschoolAnt velocities
RoboschoolAnt. velocities
2000
2000
2000
avernge ratur
1000
1000
2000
600
800
500
500
1000
500
1000
500
1000
thousand steps
thousand staps
thousand staps
Figure 4: Learning curves of the robotic control tasks, plotted in the same way as in Fig.
of continuous robotic control tasks, such as teaching a multiple-join robot to walk as fast as possible without falling down (Fig. Left). The original Roboschool environments nearly fully observable since observations include the robot's coordinates and (trigonometric functions of) joint angles, as well as (angular and coordinate) velocities. As in the PO classic control tasks, we also performed experiments in the PO versions of the Roboschool environments.
Using algorithm, experimental results (Fig. demonstrated substantial policy improvement in all PO tasks (visualization of the trained agents is in Appendix D). In some PO cases, the agents achieved comparable performance that fully observable cases. For tasks with unobserved velocities, our algorithm performed similarly to SAC-LSTM. This is because velocities can be simply estimated by one-step differences in robot coordinates and joint angles, which eases representation learning. However, in environments where only velocities can be observed, our algorithm significantly outperformed SAC-LSTM, presumably because SAC-LSTM is less efficient at encoding underlying states from velocity observations. Also, found that learning of SLAC agent unstable, i.c., sometimes could acquire near-optimal policy, but often its policy converged to poor one. Thus, average performance of SLAC was less promising than ours in most of the PO robotic control tasks.
5.3 LONG-TERE MEMORIZATION TASKS
Another common type of PO task requires long-term memorization of past events. To solve these tasks, an agent needs learn extract and to remember critical information from the whole history of raw observations. Therefore, we also examined our algorithm and other alternatives in long-tem term memorization task known as the sequential target reaching task (Han al. 2019). which a robot agent needs to reach 3 different targets in a certain sequence (Fig.) 5 Left). The robot can control its two wheels to move or turn, and will get one-step small, medium, and large rewards when it reaches the first, second, and third targets, respectively, in the correct sequence. The robot senses distances and angles from the 3 targets, but does not receive any signal indicating which target to reach. each episode, the robot's initial position and those of the three targets randomly initialized. In order to obtain rewards, the agent needs to infer the current correct target using historical observations.
We found that agents using algorithm achieved almost 100% success rate (reaching 3 targets in the correct sequence within maximum steps). SAC-LSTM also achieved similar success rate after convergence, but spent training steps learning encode underlying goal-related information
8

---

Published as a conference paper at ICLR 2020
Bequential larget reaching Lskk
WUP
Figure 5: Learning curves of the sequential target reaching task.
from sequential observations. Also, SLAC struggled hard to solve this task since its actor only received a limited steps of observations, making it difficult to infer the correct target.
5.4 CONVERGENCE OF THE
VRM
One of the most concerned problems of our algorithm is that input of the RL controllers can experience representation change, because the keep-learning model is not guaranteed to converge if novel observation appears due to improved policy (c.g. for a hopper robot, "in-the-air" state can only happen after it learns to hop). To empirically investigate how convergence of the keep-learning VRM affect policy improvement, we plot the loss functions (negative ELBOs) of the the keep-learning VRM for example tasks (Fig. (6. For simpler task (CartPole), the policy was already near optimal before the VRM fully converged. We also saw that the policy was gradually improved after the VRM mostly converged (RoboschoolAnt no velocities), and that the policy and the VRM were being improved in parallel (RoboschoolAnt velocities only).
The results suggested that policy could be improved with sufficient sample efficiency even the keep-lcarning VRM did not converge. This can be explained by that the RL controller also extract information from the first-impression model and the raw observations, which did not experience representation change during RL. Indeed, our ablation study showed performance degradation in many tasks without the first-impression VRM (Appendix C).
CartPole
RoboschoolHoppper veolocities
RoboschoolAnt veolocities
1000
2000
800
negalive ELBO
average retum
1000
ELLBO
1500
800
3
average
1000
500
400
200
100
200
300
400
500
1000
0
500
1000
Thouseind steps
Thousand steps
thousand slepe
Figure 6: Example tasks showing relationship between average return of the agent and negative ELBO (loss function, dashed) of the keep-learning VRM.
DISCUSSION
In this paper, we proposed variational recurrent model for learning to represent underlying states of PO environments and the corresponding algorithm for solving POMDPs. Our experimental results demonstrate effectiveness of the proposed algorithm in tasks in which underlying states cannot be simply inferred using a short sequence of observations. Our work can be considered attempt to understand how RL benefits from stochastic Bayesian inference of state-transitions, which actually happens in the brain (Funamizu et al.] 2016), but has been considered less often in RL studies.
We used stochastic models in this work which we actually found perform better than deterministic ones, through the environments used are deterministic (Appendix C The VRNN can

---

Published as a conference paper at ICLR 2020
be replaced with other alternatives (Bayer & Osendorfer 2014
Goyal al. 2017) to potentially
improve performance, although developing model architecture is beyond the scope of the current study. Morcover, recent study (Ahmadi & Tani 2019) showed a novel way of inference using back-propagation of prediction errors, which may also benefit our future studies.
Many researchers think that there two distinct systems for model-based and model-free RL in the brain (Gläscher et al., 2010 Lee et al. (2044) and a number of studies investigated how and when the brain switches between them (Smittenaar et al.) 2013 Lee al. 2014). However, /ttachenfeld et al. (2017) suggested that the hippocampus can learn successor representation of the environment that benefits both model-free and model-based RL, contrary the aforementioned conventional view. We further propose another possibility, that model is learned, but not used for planning or dreaming. This blurs the distinction between model- and model-free RI..
ACKNOWLEDGEMENT
This work supported by Okinawa Institute of Science and Technology Graduate University funding, and was also partially supported by a Grant-in-Aid Aid for Scientific Research on Innovative Areas: Elucidation of the Mathematical Basis and Neural Mechanisms of Multi-layer Representation Learning 16H06563. We would like to thank the lab members in the Cognitive Neurorobotics Research Unit and the Neural Computation Unit of Okinawa Institute of Science and Technology. In particular, we would like to thank Ahmadreza Ahmadi for his help during model development. We also would like to thank Steven Aird for assisting improving the manuscript.
REFERENCES
Ahmadreza Ahmadi and Jun Tani. A novel predictive-coding-- -iniiigiinppireddd variational rmn model for online prediction and recognition. Neural computation, pp. 1-50, 2019.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, pp. 834 846, 1983.
Justin Bayer and Christian Osendorfer. Learning stochastic recurrent networks. In NIPS 2014 Workshop on Advances in Variational Inference, 2014.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXivv1606.015500 2016.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
lan Danforth. Continuous CartPole for openAl
2018.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model- and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (C),L-I1), pp. 465-472, 2011.
Akihiro Funamizu, Bernd Kuhn, and Kenji Doya. Neural substrate of dynamic Bayesian inference in the cerebral cortex. Nature neuroscience, 19(12): 1682, 2016.
Jan Gläscher, Nathaniel Daw, Peter Dayan, and John O' O'Doherty. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4))585-595 2010.
Anirudh Goyal Alias Parth Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pp. 6713-6723, 2017.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2450-2462, 2018.
10

---

Published as a conference paper at ICLR 2020
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1856-1865, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rX2:
2018b.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint 1811::811.0453 2018.
Dongqi Han, Kenji Doya, and Jun Tani. Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent networks. arXiv preprint arXiv:/901.1011: 2019.
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. arXiv preprint
2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. arXiv preprint NEEi::88000022666 2018.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):859-888) -659999-66611 2019.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint (tt:::9903000777 2019.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. OpenReview, 2018.
Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra, Learning dynamics model in reinforcement learning by incorporating the long term future. arXiv preprint 1903019003 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 132.6113 2013.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic:
Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00953 2019.
Sang Wan Lee, Shinsuke Shimojo, and John P O' O'Doherty. Neural computations underlying arbitration
between model-based and model-free learning. Neuron, 81(3):687-665 699, 2014.
Andrew McCallum. Overcoming incomplete perception with utile distinction memory. In Proceedings of the Tenth International Conference on Machine Learning, pp. 190-196, 1993.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540):529, 2015.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.

---

Published as a conference paper at ICLR 2020
Jürgen Schmidhuber. Making the world differentiable: On using fully recurrent self self-super neural networks for dynamic reinforcement learning and planning in non- non-stationary environments. Institut für Informatik, Technische Universität Münnchen. Technical Report FKI-126, 90, 1990.
Jürgen Schmidhuber. Reinforcement learning in Markovian and non-Markovian environments. In Advances in neural information processing systems, pp. 500-506, 1991.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Peter Smittenaar, Thomas HB FitzGerald, Vincenzo Romei, Nicholas D Wright, and Raymond J Dolan, Disruption dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans. Neuron, 80(4):914 1194499999111 2013.
Kimberly L Stachenfeld, Matthew M Botvinick, and Samuel Gershman. The hippocampus predictive map. Nature neuroscience, 20(11); 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from images. In Advances in neural information processing systems, pp. 2746-2754, 2015.
12

---

Published as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
In this section we describe the details of implementing our algorithm as well as the alternative ones. Summaries of hyperparameters be found in Table T and
Table 1: Shared hyperparameters for all the algorithms and tasks in the paper, adopted from the original SAC implementation (Haarnoja et al. 2018b).
Hyperparameter
Description
Value
y step_start RL
Discount factor
0.99
From how many steps to start training the RL controllers
1,000
train interval RL
Interval training the RL controllers
Ir_actor
Learning rate for the actor
0.0003
Ir.critic
Learning rate for the critic
0.0003
Ir_a Htar optimizer
Learning rate for the entropy coefficient Target entropy
0.0003
-DOF
Optimizers for all the networks
Adam (Kingma & Ba 2014
Fraction of updating the target network each gradient step
0.005
policy_layen
MLP layer sizes for (4, and T. MLP layer sizes for Vo and Qx
256, 256
value_layers
256, 256
Table 2: Hyperparameters for the proposed algorithm.
Hyperparameter
Description
Value
train times FIVRM
Epoches of training the first-mppeesionn model.
5,000
train_intent vraininteerva__. KLVRM
Interval of training the keep-learning model.
5
Ir_model
Learning rate for the VRMs
0,0008
seqleen
How many steps in a sampled sequence for each update
64
batch size
How many sequences to sample for each update
4
A.1 THE PROPOSED ALGORITHM
A.1.1 NETWORK ARCHITECTURES
The first-impression model and the keep-learning model adopted the same architecture. Size of d and is 256 and 64, respectively. We used one-hidden-layer fully-connected networks with 128 hidden
neurons for the inference models [ 03.1 = Φ(x,, d, a,. as well as for |e 2b11 =
oprior (d--1, ax-1) in the generative models. For the decoder [Mr,,e a = gdecoder d:-1) in the generative models, we used 2-layers MIPs with 1 128 neurons in each layer. The input processing layer fa is also an one-layer MLP with size-128. For all the Gaussian variables, output functions for mean are linear and output functions for variance are softplus. Other activation functions of the VRMs are tanh.
The RL controllers are the same as those in SAC-MLP (Section A.2. except that network inputs raw observations together with the RNN states from the frrt-iimpresioon model and the keep-learning model.
A.1.2 INITIAL STATES OF THE VRMS
To train the VRMs, one can use number of entire episodes mini-batch, using zero initial states, as in Heess et al. (2015). However, when tackling with long episodes (e.g. there can be 1,000 steps in each episode in the robotic control tasks we used) or even infinite-horizon problems, the computation consumption will be huge in nack-propagation through time (BPTT). For better computation efficiency, method for providing the initial states (Kapturowski et al.) 2018), or more specifically, unrolling the RNNs using a portion of the replay sequence (burn-in period, up 64 steps in case) from
used 4 length-64 sequences for training the RNNs, and applied the burn-in
13

---

Published as a conference paper at ICLR 2020
initial states. We assume that proper initial states be obtained in this way. This is crucial for the tasks that require long-term memorization, and helpful to reduce bias introduces by incorrect initial states in general cases.
A.2 ALTERNATIVE ALGORRITHMS
A.2.1 SAC-MLP
We followed the original implementation of SAC in (Haarnoja et al. 201 including hyperparame- ters. However, we also applied automatic learning of the entropy coefficient (inverse of the the reward scale in Haarnoja et al. (2018a)) introduced by the authors in Haarnoja al. (2018b) to avoid tuning the reward scale for each task.
A.2.2 SAC-LSTM
To apply recurrency to SAC's function approximators, added an LSTM network with size-256 receiving raw observations as imput. The function approximators of actor and critic were the same those in SAC except receiving the LSTM's output input. The gradients pass through the LSTM so that the training the LSTM and MLPs were synchronized. The training the network also followed Section 1.2
A.2.3 SLAC
We mostly followed the implementation of SLAC explained in the authors' paper (Lee et al. 2019). One modification that since their work was using pixels observations. convolutional neural networks (CNN) and transposed CNNs were chosen for input feature extracting and output decoding layers; in our case, we replaced the CNN and transposed CNNs by 2-layers MLPs with 256 units in each layer. In addition, the authors set the output variance of for each image pixel as 0.1. However, oy. 0.1 can be too large for joint states/velocittes as observations. We found that it will lead to better performance by setting y,t as trainable parameters (as that in our algorithm). We also used 2-layer MLP with 256 units for approximating σy (22, d.-1). To avoid network weights being divergent, all the activation functions of the model were tanh except those for outputs.
B ENVIRONMENTS
For the robotic control tasks and the Pendulum task, we used environments (and modified them for PO versions) from OpenAl Gym (Brockman et al.) 2016). The CartPole environment with continuous action space was from Danforth (2018), and the codes for the sequential target reaching tasks were provided by the authors (Han al. 2019).
In the no-velocities cases, velocity information was removed from raw observations; while in the velocities-only cases, only velocity information retained in observations. We summarize key information of each environment in Table
The performance curves were obtained in evaluation phases in which agents used same policy but did not update networks or record teteetrransttion data. Each experiment was repeated using 5 different random seeds.
C ABLATION STUDY
This section demonstrated ablation study in which we compared the performance of the proposed algorithm to the same but with some modification:
With single VRM. In this case, we used only one VRM and applied both pre-training and smooth update to it.
Only first-impression model. In this case, only the first frst-impression model was used and pre-trained.
14

---

Published as a conference paper at ICLR 2020
Table 3: Information of environments used.
Name
dim(X)
DOF
Maximum steps
Pendulum
200
Pendulum (velocities only)
1
I
200
Pendulum (no velocities)
200
CartPole
4
1
1,000
CartPole (velocities only)
1,000
CartPolc (no velocities)
2
1,000 1
RoboschoolHopper
15
1,000 3
RoboschoolHHppeer (velocities only)
6
1,000 3
RoboschoolHopper (no velocities)
9
3
1,000
Roboschool
22
1,000 6
Roboschool Walker2d (velocities only)
9
1,000
ToooschoolWalkerrre (no velocities)
13
1,000 8
RoboschoolAnt
28
1,000
Roboschool (velocities only)
8
1,000
RoboschoolAnt Ant (no velocities)
17
8
1,000
Sequential goal reaching task
12
2
128
Only keep-learning model. In this case, only the keep-learning model was used and smooth-update was applied.
Deterministic model. this case, the fist--mpprression model and the keep-learning model were deterministic RNNs which learned to model the tiee-trrasiiions by minimizing mean-square error between prediction and observations instead of ELBO. The network architecture was mostly the same as the VRM expect that the inference model and the generative model were merged into a deterministic one.
The learning curves are shown in Fig. 7 It can be seen that the proposed algorithm consistently performed similar better than the modified ones.
D VISUALIZATION OF TRAINED AGENTS
Here show actual movements of the trained robots in the PO robotic control tasks (Fig. (). It can be seen that the robots succeeded in learning to hop or walk, although their policy may be sub-optimal.
E MODEL ACCURACY
As discussed in Section 2 our algorithm relies mostly on encoding capacity of models, but does not require models to make accurate prediction of future observations. Fig. shows open-loop (using the inference model to compute the latent variable 2) and close-loop (purely using the generative model) prediction of raw observation by the keep-learning models of randomly selected trained agents. Here we showcase "RohoschoolHopper- velocities only" and "Pendulum no velocities" because in these tasks our algorithm achieved similar performance to those in fully-observable versions (Fig. although the prediction accuracy of the models was imperfect.
F SENSITIVITY TO HYPERPARAMETER. OF THE VRMS
To empirically show how choice of hyperparameters of the VRMs affect RL performance, we conducted experiments using hyperparameters different from those used in the main study, More specifically, the learning rate for both VRMs was randomly selected from (0.0004, 0.0006. 0.0008, 0.001} and the sequence length was randomly selected from {16, 32, 64) (the batch size was 256/ (66//seuen length) to ensure that the total number samples in batch was 256 which matched with the alternative approaches). The other hyperparameters unchanged.
15

---

Published as a conference paper at ICLR 2020
The results can be checked in Fig 10 for all the environments used. The overall performance did not significantly change using different, random hyperparameters of the VRMs, although we could observe significant performance improvement (e.g. RoboshoolWalker2)) or degradation (e.g. RoboshoolHopper velocities only) in a few tasks using different haperparameters. Therefore, the representation learning part (VRMs) of our algorithm does not suffer from high sensitivity to hyperparameters. This can be explained by the fact that we do not use a bootstrapping (e.g. the estimation of targets of value functions depends the estimation of value functions) (Sutton & Barto 1998) update rule to train the VRMs.
G SCALABILITY
Table showed scalability of our algorithm and the alternative ones.
Algorithm
wall-clock time (100,000 steps)
# parameters
Ours
8 hr
2.8M
SAC-MLP
1 hr
0.4M
SAC-LSTM
12 hr
1.1M
SLAC
5 hr
2.8M
Table Wall-clock time and number of parameters of our algorithm and the alternative ones. The working environment was a desktop computer using Intel i7-6850K CPU and the task is "Velocities- only RoboschoolHopper". The wall-clock time include training the first-impression VRM or pre- trainings.
16

---

Published as a conference paper at ICLR 2020
Pandulum
Pendulun velocites only
Pendulun veloctios
CartPole
-703
200
-300
1900
Dun
-20
-10
400
403
Or6 kaep lounting
.1400
-5000
-440
mrud trigs
Piciion tayss
misyaan ctrps
CartPele velecilles only
CartPole: velocilies
RoboschoeHogger
Pobeschooologoer velecitee only
2003
500
Fassonelsige
Brousand steas
Ficsson stps
Chosuand
pteesrc no velocites
RobeschootWW Barld: velocites only
RobeschhooWuBaree velocitos
anctage
rsuu tras
Poliiin taps
morrand evps
Roboschosiket
Robeschhookktt velecitis only
Rebescihoolänt velocilles
Sequential beget maching task
ESCON (hl
1000
1000
2003
(housard
Chansadd STEPS
Figure 7: Learning curves of our algorithms and the modified ones.
RoboschoolHopper velltttts only
RoboschoolHopper no velocities
RoboschoollWalleero velocities only
RoboschoolWalkerze velocities
RoboschoolAnt- velocities only
RoboschoolAnt velocitics
Figure 8: Robots learned to hop or walk in PO environments using our algorithm. Each panel shows trajectory of trained agent (randomly selected) within one episode.
17

---

Published as a conference paper at ICLR 2020
RoboschoolHopper -velocities only (open loop)
RoboschoolHopper velocities velloooit only (close loop)
Pendulum no velocities (open loop)
Pendulum no velocities (close loop)
C
Figure Examples of observation predictions by keep-learning VRMs of trained agents.
18

---

Published as a conference paper at ICLR 2020
Posfslmm
Puslelan vibcito only
Perduker- se relccies
Carthile
-50
-20
-20
20000 03.0
-40
500
(00
-00
1007
Carthiee selecite only
CartPole veler
ReboschaslHogper
Reboschaslloapeer velocites only
23:0
2003
2006
1103
19n
13
L000
3300
220
300
123
5:0
20
1883
MitoschssPoogeer vdeccies
Rebouchnefttbkeel
RoboschaslMtkhaee- velocites only
Roboschonttidkeect an vedeciees
1504
1206
200
690
501
300
100
200
300
209
100
1387
300
3003
400
130
1030 B0
Roboschostket
Boboschosikot weleiits only
Bobeschookket- so velcctios
Sequentid terpetracchegg tak
2006
120
1000
1200
200
430
502
300
1003
3000
302
3206
500
2002
230
250
230
150
Figure 10: The learning curves of our algorithm using the hyperparameters for the VRMs used in the paper (Table 2 and using range random hyperparameters (Appendix(F) Data are Mean = S. S.E.M., obtained from 20 repeats using different random seeds.
19


Reviews:
[{"review_id": "SylTqP96FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Summary\n\nThe work considers reinforcement learning in partially observed environments, targeting in particular the case when the agent needs to remember some important information over a longer period of time. \n\nThe work gives an interesting connection between variational recurrent neural networks and reinforcement learning. The paper is well written and technically sound. The method achieves improved performance in several (simpler) benchmark problems in comparison to strong baselines.\n\n* Detailed discussion\n\nThe work allocates latent variables to describe unobserved part of the state and learns the state transition probability model (the action-dependent generator (10)). Learning these state transitions becomes an unsupervised learning task, for which the lower bound on the likelihood of the observed data is optimized. This learning of the state transition model is actually similar to classical model-based RL. The policy and value functions are then learned on top of this model using SoTA techniques. From this perspective I would say this is a classical approach to partially observed reinforcement learning with advanced models for transition matrix and value and policy functions.\n\nThe work combines together variational recurrent neural network (VRNN) and soft actor critic (SAC) models into a rather advanced system for reinforcement learning. These parts are somewhat extended to fit together, but it is not clear whether all components are needed. For example, why z and d need to be distinguished, cannot they be combined into one latent variable “zd” in order to simplify the number of connections? Or I am missing some important property necessary for the inference?\n\nIn (7),(8) are the expectations over the same state / actions or different? The notation is not clear about it.\n\nAfter (2), given x_t and d_{t-1} instead of d_t?\n\nThe experiments are conducted in the setting that the observations were all relevant measurements for the agent (positions, velocities). From Fig 7 RoboscoolHooper (close loop) we see that actually all measurements can be well predicted by generative model up to 8 steps ahead (in dim 2 that stays zero the noise is amplified). This is a nice and desirable property and is the consequence of the generative modeling and the marginal likelihood ELBO optimization. However the paper seem to argue (Appendix E) that the algorithm does not require the model to make accurate predictions and relies on the encoding capacity, which I find confusing. Furthermore, it would be interesting to see how this approach should scale to the setting when there are more sensory inputs, some less relevant than the other. In the end, is a good generative model to predict the future observation needed or not needed in this approach?\n\nThe exploration seems to be addressed just by randomizing the policy,  are there any better options?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sklk8C2pKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes that for partially observable reinforcement learning tasks it might be simpler to decompose the problem in two parts: a recurrent world model and a feedforward agent, as opposed to using just a recurrent agent. Intuitively the decomposition makes sense, though it's not very clear to me that the problems encountered training a recurrent agent should be dramatically simpler when training a recurrent world model instead.\n\nFor the world model the paper proposes using a variational recurrent state-transition model, which is essentially a VRNN which also conditions on the actions. It is not clear to me that this is easier to learn than a recurrent agent because the probability distribution of the data used to train the modified VRNN is highly dependent on the current policy in the same way that the non-iid training data makes it tricky for the RNN policy models to converge.\n\nWhile the experimental results seem to show the new algorithm outperforming the existing ones over a large set of random seeds the paper does not specify how the fixed hyperparameters for all models were chosen, leaving open the possibility that different hyperparameter settings would have shown reversals in the experimental results. It's also not clear that the complexity of the alternate models was adequately accounted for (specially since two VRNNs are required to match the performance of a single RNN agent). That said the incompleteness of the experimental results is my only reservation against accepting this paper.\n\n\n----\n\nThe author response somewhat addressed my major concern, which was the lack of hyperparameter tuning in the baselines, as well as clarifying some of the other questions I had. Given that my score has been updated.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gTJc8QqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the paper is written quite well and addresses a relevant topic. The experiments are thorough and make sense given the research question.  The paper furthermore provide additional ablation studies and does a good job in model analysis.\n\nThere are some critical points I have about the paper summarized below:\n\n\nIntroduction:\nThe 3 categories to solve POMDPs appear artificial. All 3  (whether using a window directly, using a RNN or building a surrogate model) aim to transform the POMDP to an MDP by taking history of observations (either explicit, or implicit).\n\nFigure 1 is too  complicated for an illustrative figure  and the presentation and clarity needs to be improved.\n\n\nMethod:\nThe idea is sound and described well. However, some design decisions appear ad-hoc  (only justified by empirical observations). For instance, the authors argue it is better to keep 2 models (the \"first-impression\" and \"keep-learning\" model) and show an ablation study in appendix C.  \nNow, one could wonder if the proposed method would still perform better than the baselines if with, say, the just using a single VRM.   The improvement compared to SAC-LSTM is not very large except in a few cases, so possibly just using the VRM would perform no better.\n\nThis could then defeat the theoretical argument of the method that \"[..] the actor network of SLAC\ndid not take advantage of the latent variable, but instead used some steps of raw observations as input,\nwhich creates problems in achieving long-term memorization of reward-related state-transitions.\"\n\n\nReference that should be included and possibly compared to:\n[1] Watter, Manuel, et al. \"Embed to control: A locally linear latent dynamics model for control from raw images.\" Advances in neural information processing systems. 2015.\n\n\nQuestions:\n1. What is the scalability, such as wall-clock time and #parameters  of the approach compared to the baselines? (When using 2 VRMs the # of parameters is doubled)\n2. Why the need to input the original observation x (and not just the latent representation) into the RL controller?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to decompose control in a POMDP into learning a model of the environment (via a VRNN) and learning a feed-forward policy that has access to both the environment and environment model. They argue that learning the recurrent environment model is easier than learning a recurrent policy. They demonstrate improved performance over existing state-of-the-art approaches on several PO tasks.

Reviewers found the motivation for the proposed approach convincing and the experimental results proved the effectiveness of the method. The authors response resolved reviewers concerns, so as a result, I recommend acceptance.
Decision: Accept (Poster)