SVQN: Sequential Variational Soft Q-Learning Networks
Authors: Shiyu Huang
Hang Su
Jun Zhu
Ting Chen
Venue:  (2020)
Keywords: reinforcement learning
POMDP
variational inference
generative model

Abstract: Partially Observable Markov Decision Processes (POMDPs) are popular and flexible models for real-world decision-making applications that demand the information from past observations to make optimal decisions. Standard reinforcement learning algorithms for solving Markov Decision Processes (MDP) tasks are not applicable, as they cannot infer the unobserved states. In this paper, we propose a novel algorithm for POMDPs, named sequential variational soft Q-learning networks (SVQNs), which formalizes the inference of hidden states and maximum entropy reinforcement learning (MERL) under a unified graphical model and optimizes the two modules jointly. We further design a deep recurrent neural network to reduce the computational complexity of the algorithm. Experimental results show that SVQNs can utilize past information to help decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.

Published as a conference paper at ICLR 2020
SVQN: SEQUENTIAL LEARNING NETWORKS
VARIATIONAL
SOFT
Q-
Shiyu Huang, Hang Su, Jun Zhu", Ting Chen'
Dept. of Comp. Sci. & Tech., BNRist Center, Institute for AI, THBI Lab, Tsinghua University hsy? 17 @ wwwall.tiinnguaa.oom
(suhangss, deszj. tingchen] @tsinghua.edu.cn
ABSTRACT
Partially Observable Markov Decision Processes (POMDPs) are popular and flex- ible models for real-world decision- making applications that demand the infor- mation from past observations to make optimal decisions. Standard reinforcement learning algorithms for solving Markov Decision Processes (MDP) tasks are not applicable, as they cannot infer the unobserved states. In this paper, we propose novel algorithm for POMDPs, named sequential variational soft Q-learning net- works (SVQNs), which formalizes the inference of hidden states and maximum entropy reinforcement learning (MERL) under a unified graphical model and opti- mizes the two modules jointly. We further design deep recurrent neural network to reduce the computational complexity of the algorithm. Experimental results show that SVQNs can utilize past information to help decision making for effi- cient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.
INTRODUCTION
In recent years, substantial progress has been made in deep reinforcement learning for solving var- ious challenging tasks, including the computer Go game (Silver et al., 2016), Atari games (Mnih) eet al. 2015), StarCraft (Zambaldi et al. 2018 Pang al, 2018) and the first-person shooting (FPS) games (Lample & Chaplot 2017, Wu & Tian 2016, Huang et al, 2019). However, in many real-world applications, decision-making problems are partially observable (Aström) (965), preventing such problems from being solved by standard reinforcement learning algorithms. For- mally, these kinds of problems often defined as Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998), which demand information from past observations to help in the decision-mmaing process
McCallum 1993).
Although numerous efforts (Hausknecht & Stone 2015 Foerster et al. 2016 Igl et al. 2018 Zhu
al. 2018) have been paid to tackle this problem, there still exist various challenges, For example,
Egorov (2015) tries to solve POMDPs by using the belief of the agent as the input of DQN (Mnih) al.] 2015), this algorithm needs access to the environment model. However, in many rein- forcement learning tasks, it is not possible for the agent to acquire the underlying transition function. making such algorithms inapplicable. Some recent work (Karkus al., [017 McAllester & Singh) 2013 Babayan et al. 2018) tries to solve POMDPs under the model-free setting, i.e.. the agent does not need to know and learn the transition function of the environment. For instance, Karkus et al. (017) trained an agent to navigate in partially observable grid world under the model-free setting, lee, the agent can only observe a part of the grid world and does not learn the transition function. The agent uses its local observations to update its beliefs (McAllester & Singh 2013 [Babayan] et al., 2018). In their experiments, the ground truth of the state is the full map plus the location of the agent, which means the representation of the state is explicit. However, in some complex tasks, it is impossible to acquire or design the state or beliefs.
To solve the unknown representation problem of the state, Hausknechh & Stone 2015) and Zhu et al. (2018) try to represent the state as latent variables of neural networks. However, they only use
"J. Zhu and T. Chen corresponding authors. J. Zhu is also with RealAI.

---

Published as a conference paper at ICLR 2020
deep recurrent neural network to capture the historical information and fail to utilize the Markov property of the state in POMDPs. [g et al. (2018) apply sequential Monte Carlo (SMC) (Le et al. 2017) to introduce inductive bias to the neural network, which can embody the Markov properties of the state. They can infer the state from the past observations online. However, they separate the planning algorithm from the inference of the state.
To infer the hidden states and optimize the planning module jointly, we represent POMDPs as a unified probabilistic graphical model (PGM) and derive a single evidence lower bound (ELBO). We apply structured variational inference to optimize the ELBO. In our implementation, we design generative models to infer the hidden variables, however, the distribution of the latent variables is conditioned on previous hidden states. This is different from standard VAEs (Kingma & Welling 2013), whose prior of the latent variables can be standard Gaussian distribution. So, we ap- ply an additional approximate function to tackle the conditional prior problem. The planning can also be solved under the PGM framework. Fortunately, maximum entropy reinforcement learning (MERL) (Levine 2018) provide tool to formalize the planning probabilistic inference task.
In this paper, we propose a novel end-to-end neural network called the sequential variational soft Q-leariing network (SVQN), which integrates the learning of hidden states and the optimization of the planning within the same framework. A deep recurrent neural network (RNN) (Cho et al.) 2014 [Hocreiterr & Schmidhuber 1997) in SVQNs is used to reduce the computational complexity, because the feature extraction can share the same weights in the RNN. Experimental results show that the SVQN can utilize past information to help in decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.
Contributions: (1)
We derive the variational lower bound for POMDPs, which allows us to integrate the optimization of the control problem and learning of the hidden state under a unified graphical model. (2) We propose to tackle the difficulty of the inference of the hidden state and solve the problem of a conditional prior using the generative models. (3) We design an end-to-end deep
recurrent neural network, which can reduce the computational complexity and be trained efficiently.
RELATED WORK
We summarize some related work in POMDPs and inference methods for sequential data.
Model-Based and Model-Free methods for POMDPs: When the environment model is accessible, POMDPs can be solved by model-based method. |Egorov] (2015) used model-based methods to solve POMDPs, but their agents need to know the belief-update function and the transition function. When the environment model is unknown, model-free methods should be applicd. Recently, some researchers (Hausknecht & Stone, 2015 Zhu et al. 2018) used recurrent neural networks to capture the historical information, but they failed to utilize the Markov property of the state in POMDPs. Our work proposes generative models for the algorithm learning. which tackles the difficulty of the inference of hidden states and introduces inductive bias to the network structure. Igl et al. (2018) applied sequential Monte Carlo (SMC) to the POMDPs. They can infer the hidden state from the past observations online. However, they separate the planning algorithm and the inference of the hidden state. Our algorithm is derived from a unified graphical model, which can train the inference model and the planning algorithm jointly.
Explorations in POMDPs: In contrast to MDP methods, POMDP methods should do exploration to gather information with no immediate reward which can then be used in the future to gain higher rewards. Pathak et al. (017)) and Choi et al.] (2018) designed deep reinforcement learning algo- rithms to help the exploration in their tasks. Actually, their tasks are POMDP tasks. However, they just treat their tasks as MDP tasks and intuitively add exploration tricks to standard reinforcement learning algorithms. Their exploration tricks include the reconstruction of observations and actions, which comes from intuitive concepts such as curiosity and attention. The generative models in our method also need to reconstruct the observations and actions, but our algorithm is derived from solid theoretical foundations.
The observation in POMDP tasks is sequential data, and need to do inference for the hidden state from observations. Coquelin et al. (2009) used a particle filter to estimate the belief state given past observations. However, their method needs access to the
Inference for Sequential Data:
2

---

Published as a conference paper at ICLR 2020
St
91+1
8t
8141
Gc
O2
OL
O
O1+1
(h)
Figure I: The graphical models for Markov decision processes (MDPs) (a) and partially observable Markov processes (POMDPs) (b). Grey nodes observed, white nodes are hidden. In POMDPs, the state is not observable and must be inferred from past observations. The O₂ a binary random variable, where O₂ 1 denotes that the action is optimal at time and O2 0 denotes that the action is not optimal.
environment model. Chung et al. (2015) derived the evidence lower bound of latent variables for sequential data and designed a novel deep neural network to infer recurrent latent variables. Our inference method has some superficial resemblances to their method, but we derived the evidence lower bound from different graphical model and tackle the difficulty of how to integrate the RL. planning algorithm instead of just inferring the hidden variables.
PRELIMINARIES
start by briefly reviewing the backgrounds and notations related to our method, including par- tially observable Markov decision processes and maximum entropy reinforcement learning which solves optimal control problems using a probabilistic framework.
3.1 PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
In many control tasks, the complete state of the environment is unknown the agent. Instead, the agent can only receive local observations, which are typically conditioned on the current state of the system. The agent needs make decisions based on the historical information. This kind of problems can be formalized as partially observable Markov decision processes (POMDPs) (Smith Simmons] 2004). Formally, a POMDP is represented as tuple (S, A, Z. r), where A and O are state space, action space and observation space, respectively. The reward function rss,a) the received reward when taking action a in state s. T'ss, a, s') the state-transition function, which defines the probability of the succeeding state s' after taking action in state Z(s, a, o) is the observation function, which defines the probability of emitted observation o after taking action in state
POMDPs use the belief (s) to maintain the distribution of the unknown state at time t, and the agent updates its belief with Bayesian filter when receiving observation as
b,(s') = 7z ( Ot 0c)
Σ 8 - 18),
eES
where y is normalizing constant and 0g is the action at time t. POMDP planning needs to find policy π to maximize the accumulative reward as
(bb) = E (rr( (b)
(2)
where s; the state at time and 7 E (0.1) 1) discount factor.
3.2 MAXIMUM ENTROPY REINFORCEMENT LEARNING
POMDPs require solving two problems: the inference of state and optimal control. To integrate both them under a unified framework, we represent POMDPs as probabilistic graphical model. So the optimal control problem needs to be solved the maximum entropy reinforcement learning (MERL) algorithm provides an effective tool to solve optimal control problem under PGM framework. The graphical model of Markov decision processes
probabilistic inference task. Fortunately,
3

---

Published as a conference paper at ICLR 2020
is shown in Fig. Ia). We borrow the notation from Levine 2018
illustrate the algorithm. Levine
(2018) introduces a binary random variable O2 to the graphical model, where O = 1 denotes that the action is optimal at time t, and O, 0 denotes that the action is not optimal. The probability distribution of O is p(O, = 1/8, o+) = exp(r(*t, at)), and the variational lower bound is given by:
logp(O (1:1) 2 ((L
Er(s),a)) - logn((a(ss))
(3)
where r[ass) is the policy function. Standard reinforcement learning only needs maximize the cumulative reward. However, MERL will maximize an extra term, which is the policy entropy each visited state. To get the optimal solution for MERL, two messages are introduced, i.e., Be(s;, a.) = p(Ourls,, at) and (st) = p(Oer(ss), and their relations are given by:
Be(st) = / (s,,,, at)p(acs:)
Be(st, at) = =So a.t...
at) a dss411
and the optimal policy is given by:
Be(s a1)
(a St, OCT)
(5)
(st)
We can define the Q-value function and V-value function as below:
(st, 1st at); V(st) = (oe);
(6)
(7)
and the update functions are:
V(s:) = log / p(at (x))e)(((((())))))))))) (()(,,t))))))) d:))d
(8)
((sr, = r(ss, ax) + logl po (xxp(()))))))))))))))))))) (st+1))].
To maximize the variational lower bound in Eq. (3). can use parameterized Q-function Qo at), where 0 is the function parameter, and can be learned via gradient descent:
Qo(s,, as) (r(s), a.) log /
0 = 0 - aB
(9)
d0
where a is the learning rate. This algorithm is called soft Q-learning, because the update function for Q-value can be considered soft-update version of the standard Bellman backup. For better understanding of MERL, we recommend readers reference this tutorial (Levine) 2018).
4 SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS
We now present our algorithm in detail. We first derive the variational lower bound for POMDPs, and then illustrate how to deal with the conditional prior.
4.1 VARIATIONAL LOWER BOUND FOR POMDPS
Different from MDPs, the state of POMDPs in the PGM (shown in Fig. [b)) is unobservable, which need to be inferred from the action and the observation We need to derive different variational lower bound for POMDPs, which can used to infer the hidden state and do planning jointly.
Unlike the observation function Z(s, a, o) in Section 2.1, we assume that the observation is emitted from the hidden state s, which means the probability distribution of observations tioned on states, i.e., O, ~ p(), This assumption can hold in many tasks. For example, the observation in partially observed maze is only determined by the full map and the agent's location.
only condi-
4

---

Published as a conference paper at ICLR 2020
SVON
hidden state
(x(* [Ap-1)
St-1
Feature.
Og
W/ /o(n: 04- Pt
quisi]be)
Qa(
retulit-sshh
recurrent unit
inner hidden state!
S
XT0.
The structure of sequential variational soft Q-learning networks (SVQNs). Black solid lines represent forward paths of the neural network, gray dashed lines represent reconstruction paths and blue arrows stand for sampling latent variables using the re-parameterization trick. Double- arrows indicate that the algorithm needs to minimize the KL-divergence between two probability dis- tributions. The model takes the observation or previous action d:- and reward 7 as inputs, and it uses neural network fo dz-1, ( (wne(We, (W,, 1) is used to capture the historical information ht. (5 (hg) and (s a1-1) proposed distributions of hidden states. The hidden state 8, and inner hidden state & are sampled from go(ss Ih,) and parior (s, 1t- ar-1) respectively. The Q-function Qu(st, 0₂) is learned via the temporal difference (TD) algorithm with soft update.
Figure 2:
Pr-1) extract the low-dim hidden feature wy. The recurrent
unit rnng
We apply the structured variational inference to optimize the evidence lower bound of POMDPs. In structured variational inference, different parts of the proposal distributions can be optimized separately, which means we can fix some approximate functions and optimize other approximate functions. In POMDPs, we will use two approximate functions 9-r (agls;) and [ 1t-1. ot). gx() approximates the optimal policy and go approximates the function of inferring hidden states, where 0 is the parameter of the approximate function. When (a(:) is fixed, the learning procedure is same as MERL,, so that ạ- (.) can be learned via the soft Q-learning algorithm. Conversely, when Ja (-) is fixed as the optimal policy, we can learn the inference function qs (-) for hidden states. We can derive the evidence lower boLnd(ELBO) as bellow:
p(S1.T, Oo.T, a0.r, 01.T)
logp(On:r ag:r, 01:T) loglE yo (
4o (s(rr u.rr 0o.r 01:1) p(s1er, Ourr ao:r, 01:T)
(10)
Bu log
(s Ou.T, 0o.T: 01:T)
and the ELBO can be written as:
L(Oo.T, 00.T, = Egga Surrapor.oo
Σ r(se, ar) + log p(ar)(oo:ss)
(II)
DKL
To get the optimal action in POMDPs, need to maximize the ELBO above. The first term in the ELBO the cumulative reward ΣT r(se, 0e), which can be optimized via maximum entropy reinforcement learning algorithm.
We apply generative models to optimize the rest terms in the ELBO. The term p(o, (s;) means that the hidden state 8; needs to have the ability to generate current observation
The Kullback-Leible di- vergence term Dx L (gr(s11tt111 dc-1, ot) ( at at-1)] 1]] indicates that we should minimize the gap between approximate function go(s; -1, 0t-1, or) and the prior |s Gt-1). In standard VAEs, the prior p(-) is generally fixed and can be set as a standard Gaussian. However, in this gen- erative model, the prior of the hidden state s, is conditioned We will show how we deal with the conditional prior in next sub-section. More details about the derivation process of the ELBO can be found in Appendix A.
previous state $;-1 and action a--1-
5

---

Published as a conference paper at ICLR 2020
4.2 VARIATIONAL AUTOENCODERS FOR THE CONDITIONAL PRIOR
The KL-divergence term in Eq. (11) introduces a conditional prior for the hidden state Sr, but the true conditional distribution p( (t-1) is unknown, making it intractable do inference. Hence we propose new parameterized function pa parior (st|8:-1, at-1) to approximate the true distribution 1 1) We use a standard VAE to learn the Po the inner observed data in this VAE model and the inner hidden state & is inferred from (st-1, (z-1)
prior (-). Hence (st-1, 0t-1) can be treated as
We can write down the ELBO for the conditional prior directly by using the ELBO of standard VAEs (see Appendix
11 -DxI(PD (vgrir (8e(81-1, at- at-1))lp(s=))
(12)
+ E Bygrior ee-1 [logp(s-- (1- ds)1(
where p(() be set a standard Gaussian. paradigm.
The p/ perior() (-) can be learned via standard VAE
The outputs of pg (.) are perior and oerior, so that the KL-divergence term in Eq. 11 can be written as:
DKL. (ga 8 ar-1 "0
DEL go (1 0z) prior (st $111, (()
(13)
DKL (90()))) (1(s))))-1,04111 ot) diag( diag(grrrio?)))
Unlike standard VAEs, the second distribution of the KL-divergence function in Eq. (13) is not a standard normal distribution. But we can still get the final loss function if we expand the KL- divergence function. We present the final formula of the loss function for Eq. (13) in Appendix
4.3 SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS
We design a deep recurrent neural network to improve the ELBO derived in Section 4.1 and Section 4.2. Fig. 2 shows the overall structure of algorithm.
In our implementation, there are two generative models, i.c., one of the generate model learns the conditional prior %
prior (.) and reconstructs the inner observed data (st-1, and another gener- model learns the qo(-) in Eq. (13) and reconstructs the current observation Oz.
For the first generative model, there are two losses, i.c., Linner and L Lige is the negated KL-divergence between paiior (s a--1) and standard Gaussian:
L inner = -DKI(P, prior (sr -e (+/55-1..4-
(14)
And L ioner is the reconstruction loss of (st-1, Ct-1):
Linner MSE ((s-11, pp (fnner(ss))
(15)
where MSE(.) is the mean-square error function, the inner hidden state 8, is sampled from pg prior (se|8 41-1), and gonner (.) is reconstruction function.
For the second generative model, there are also two losses, i.e., Telho and Leke Lelbo is defined as:
I LKL -DxL
o1)l1 aol (x(88(1,,))))
(16)
And L elloo the reconstruction loss o::
Lakee = MSE @gtho
(17)
where MSE(.) is the mean-square enror function, the hidden state s, is sampled from 0z), and gelbo() (-) is reconstruction function.

---

Published as a conference paper at ICLR 2020
Figure 3: Screen shots of Atari games and ViZDoom tasks. From left right, they are Pong, Chop- perCommand, DoubleDunk, Asteroids, Health Gather,Healt, Gather v2 and Defend Center. When only given one frame the input, these Atari games are POMDPs because we can not obtain the ve- locity of the moving object from a single observation. Because the agent in ViZDoom environment can only see in just one direction, naturally introduces partial observability to these tasks.
Finally, we get kinds of loss functions for the two generative models, i.e., the reconstruction loss LMSS = LMSE + LLMS and the KL-divergence loss LKL Linner + cllo
KL
And for the planning algorithm, use the soft Q-learning algorithm (Levine 2018 Its loss func-
tion is the temporal difference eπror LTD. which is updated via Eq. (9) All these losses can jointly be optimized via stochastic gradient descent algorithms.
To reduce the computation complexity, we use a deep recurrent neural network to capture the his- torical information. We first use the fo(o,, at-1; to extract a low-dim hidden feature Wy from current input, and then feed this features to a recurrent unit rnno(-) to get the recurrent output h4- Because h2 contains the information of past observations, the loss function in Eq. (16) can be rewritten as:
= DK (0e()))))) 1o? lpgrin (1-1) L KL
elha
(18)
In the training, we tried both LSTM cell (Hochreiter & Schmidhuber 1997) and GRU cell (Cho et al. 2014) as basic recurrent units. Because of the generalization ability of the recurrent neural network (Lample & Chaplot 2017). can just sample fixed length H of sequential data for training instead of using the full-length data. In the experiment, we also studied how the training length H influences the final performance. We hosts multiple games in parallel and they all send batched data to a central data memory. This is quite similar the data collection method in ELF (Tian al. 2017). More details about the training strategy can be found in Appendix
parallel training strategy, i.e., the program
5 EXPERIMENTS
We evaluate our algorithm on flickering Atari (Hausknecht & Stone 2015) and ViZDoom plat- form (Kempka al., 2016). Flickering Atari was previously used as the test environment in DRON (Hausknecht & Stone 2015), ADRQN (Zhu et al. 2018) and DVRL (Igl et al., 2018). The ViZDoom platform
3D FPS game for Al research. The agent in ViZDoom needs to navigate in the 3D environment to accomplish various tasks, such gathering resources, shooting enemies and looking for the exit. Because the agent can only see in just one direction each time step, this naturally introduces partial observability to the task.
5.1 EXPERIMENT SETUP
We used some recent algorithms as baselines. Because the true state and transition function are un- known, only the methods which can be trained under model-free setting will be used for comparison. Baselines are listed as below:
Deep Q-Networks (DQN) (Mnih et al., 2015): method which uses standard Bellman backup and uses the temporal difference error as the objective function.
Deep Soft Q-Networks (DSQN) (Levine 2018): A method which uses soft Bellman backup and uses the temporal difference error as the objective function.
Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone 2015): A method which applies LSTM the top of the DQN.
Action-Specific Deep Recurrent Q-Networks (ADRQN) (Zuu et al. 2018): A method which ex- tends the DRQN, i.e., it uses both the observation and action inputs.

---

Published as a conference paper at ICLR 2020
Deep Variational Reinforcement Learning (DVRL) (gl et al. 2018 A method which combines sequential Monte Carlo and A2C (Dhariwal et al. 2017 7 to solve POMDPs.
5.2 EVALUATION ON FLICKERING ATARI
Atari environments (Bellemare et al. 2013) are widely used as the benchmark for deep reinforce- ment learning algorithms due to its high dimensional observation spaces and numerous challenging tasks. Flickering Atari was introduced by (Hausknecht Stone 2015). In each running step, the observation may be obscured with certain probability, L.e., the screen will be either fully ob- servable or fully obscured with black pixels. The settings of experiments keep in line with DVRL (Igl ctt al. 2018). i.c., only one frame is used, the frameskip is set to four and each frame is obscured with a probability 0.5.
We choose four Atrai games (i.e., Pong, ChopperCommand, DoubleDunk and Asteroids) for eval- uation. Fig. 3 shows the screen shots of these games. When only given one frame as the input, these tasks are POMDPs because observation. These tasks have high dimensional and continuous observation spaces, while discrete action spaces. The basic network architecture and syper--arameters are similar to DQN (Mnih al.) 2015). For the recurrent neural networks, we use a sequence length of for training. All the algo- rithms train for 10000,000 steps and run for 100 episodes during evaluation. The training details can be found in Appendix
cannot obtain the velocity of the moving object from a single
Table shows the performance results of different algorithms on these Atari games. We can see that our algorithms significantly outperform other baselines on three of the games and get close score to DVRL on ChopperCommand. Compared with DRQN and ADRQN, our method introduces inductive bias to the network, which helps state estimate and RI. planning for POMDPs. Compared with DVRL, our method can achieve competitive performance with lower sampling complexity.
DQN
DSQN
DRQN
ADRON
DVRL
SVQN(GRU) SVQN(LSTM)
Pong -4.9(2..6)
-2.1(22.3)
1.6(#7.8)
7(44.6)
18.2(22.7)
19.2(#1.3)
18.6(+1.9)
Chopper 1350(+731)
1090(:4409 1250(1522)
1608(1707) 6602()449)
5805(#312) 6005(#258)
DDunk -16.2(=3.1)
-16.9(+2.8)
-14.4(#3.2)
-12.9(+3.6)
-6.0(=1.3)
-5.8 (#1.4)
-..((+1.2)
Asteroids
935(#410) 940(#320)
871(#340)
1040(#432)
1539(#73)
1645(=102)
1585(+86)
Table 1:
Evaluation results of different models on flickering Atari. The values are the final evaluation scores after training for different algorithms. Values in parentheses indicate the standard deviation. Mann-Whitney
Evaluations Mann- Whitney rank test and bold numbers indicate statistical significance the 5% level. Our algorithms outperform other baselines on three of the games and get close score to DVRL on ChopperCommand.
5.3 EVALUATION ON VIZDOOM TASKS
We designed three tasks in ViZDoom as the evaluation tasks for our algorithm.
Health Gather: The agent needs to gather health supplies in a flat map. The agent will lose health every time step. Accordingly, it can't 't gather enough health supplies, it will die and the game is over. At each time step, the agent will receive reward of 0.001, which encourages the agent to live a longer life. When it collects one health supply, it will receive a reward of The observation for the agent is the grayscale image of its vision and its health value. The agent has three actions to choose, i.c., TURN LEFT, TURN RIGHT and MOVE FORWARD.
Health Gather v2: This is more difficult task than previous task with a more complex map. There is some poison in the map. When the agent encounters poison, it will lose health. We use the same reward scenario, observations and actions in Health Gather.
In this task, the agent stands in the center a flat map. Monsters will walk toward the agent from the edge of the map. When the monsters touch the agent, the agent will lose health. The agent holds a gun with limited ammo, and it can kill the monsters by pressing down the shooting button. The agent gets reward of by killing monsters, reward of -0.2 by using ammo and reward of
Defend Center:
8

---

Published as a conference paper at ICLR 2020
40
40
Reward
Reward
22
DCN
DSON
DSCN
DRON
DROW
ADAON
10
ADRON
SVGNIGRU) SVONILSTMO
10
15
SVONILSTIO
50300 100000 150000
220300 250200
50000 LC0020 150000
200000 230020
0.1 022 0.3 0. as as 0.7 88 a9
Step
Step
Observation Probability
(c)
Figure 4: (a) Training curves on Health Gather. Both SVQN models outperform other baselines.
(b) Performances of the models with different training sequence lengths Health Gather. When the training sequence length is too long, it is hard for the algorithm to gather useful information through a long gradient flow. (c) Evaluation results of different models under different observation probabilities of the Health Gather, and all models are trained with full observation. The results show that SVQN models are more robust to the disturbance of the observation than other algorithms.
DQN
DSQN
DRQN ADRQN SVQN(GRU) SVQN(LSTM)
Health Gather 27.8 1..8(+1.4) 23.3(+1.7)
32.4(+-2.) 29.9(+1.6)
37.7(+0.9)
42.3(+1.7)
Health Gather v2
9.5(#1.55 6.7(22.1) 13.2(+2.5)
14.3(#1.2)
15.((=1.))
18.2(=0.9) 48(=1.4)
Defend Center 35.4(+:.3) 38.5(+1.5) 46.0(+0.5) 45.8(+2.2) 50.3(42.1)
Table 2:
Evaluation results of different models on ViZDoom. The values are the final evaluation scores after training for different algorithms. Values in parentheses indicate the standard deviation. Mann-Whitney
Evaluations use Mann- Whitney rank test and bold numbers indicate statistical significance at the 5% level. The SVQN models achieve the best performance these three tasks.
when losing health. The observation for the agent is the grayscale image of its vision, its health value and the ammo remained. The agent has three actions to choose, i.e., TURN LEFT, TURN RIGHT and ATTACK.
Fig. 3 shows the screen shots of these three tasks. All the algorithms are trained with the same basic network architecture and use the same hyperparameters. All the models take only one observation as input at each time step and the vision inputs resized to the resolution of 84 84. For the recurrent neural networks, we use a sequence length of 5 for training. All the algorithms train for 300,000 steps and run for 20 episodes during evaluation. The discount factor 7 is set to 0.95, learning rate is 0.0001 and the Adam optimizer (Kingma & Ba (014)) used for training. The network architecture for these tasks and training details are shown in Appendix [
Fig. 4[a) shows the training curves of different methods on the Health Gather. Both SVQN models outperform other baselines. Table the final scores of different models on these three tasks. The SVQN models achieve the best performance on these three tasks. Experiments on ViZDoom indicate that generative models of SVQNs can improve agent's exploration ability in complicated unknown environment
5.4 ABLATION STUDY
Training sequence length: We study how the training sequence length H impacts the algorithm's performance, We use the SVQN(LSTM) model and the environment of Health Gather for this study. Fig. 4 b) shows training processes models with different training sequence lengths(HH = 2, 10, 15). We can see that, when the sequence length is too short(H 2). the model can't 't gather enough historical information, which causes performance degradation. When the sequence length is too long(H 15), it is arduous for the network to do optimization, which will also

---

Published as a conference paper at ICLR 2020
lead to performance degradation. This experiment also shows that with limited training length, the algorithm can generalize to the time of arbitrary lengths during testing. Thanks to this generalization abillity, we can train the agent with fixed length of data instead of the full-length data, which can reduce the computation complexity.
Observation Probability: We also study whether different models are robust the noise of the observation. We add a modification to the game Health Gather, i.e., at each time step, the ob- servation of the screen is either fully revealed or fully obscured with fixed observation prob- ability p. Fig. c show the evaluation results of different models under different observation probabilities(p 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). All the models are trained under the stan- dard environment with full observations. The results show that SVQN models are relatively robust to the disturbance of the observation compared to other algorithms.
CONCLUSIONS
We propose a novel algorithm named Sequential Variational Soft Q-Learning Networks (SVQN) to solve POMDPs with the discrete action space. SVQN is model-free and does not need know the true state's 's representation. We apply generative models to deal with the conditional prior of hidden states and use a recurrent neural network to reduce the computational complexity, i.e., with a small length of training data, it can generalize to the test data with an arbitrary length. Our designed deep neural network can be trained end-to-end, which optimizes the planning and inference of hidden states jointly. Experimental results show that SVQN outperforms previous methods on challenging tasks and has the robustness to the disturbance of the observation. SVQN is also flexible and can be integrated with other maximum entropy reinforcement learning algorithms, such soft actor- critic (Haarnoja et al.) 2018). In the future, we will try to develop algorithms for POMDPs problems with the continuous action space.
ACKNOWLEDGMENT
This work was supported by the National Key Research and Development Program of China (No. 2017YFA0700904), NSFC Projects (Nos. 61620106010, U19B2034, U1811461), Beijing NSF Project (No. L172037), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelli- gent Computing, the JP Morgan Faculty Research Program and the NVIDIA NVAIL Program with GPU/DGX Acceleration.
REFERENCES
K. Aström. Optimal control of Markov decision processes with incomplete state estimation. J. Math. Anal. Appl., 10:174-205, 1965.
Benedicte M Babayan, Naoshige Uchida, and Samuel J Gershman. Belief state representation in the dopamine system. Nature communications, 9(1):1891, 2018
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Jongwook Choi, Yijie Guo, Marcin Moczulski, Junhyuk Oh, Neal Wu, Mohammad Norouzi, and Honglak Lee. Contingency-awart exploration in reinforcement learning. arXiv preprint 1rXiv:1811.01443 2018
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Adyances in neural information processing systems, pp. 2980-2988, 2015.
10

---

Published as a conference paper at ICLR 2020
Pierre-Araud Coquelin, Romain Deguest, and Rémi Munos. Particle filter-based policy gradient in pomdps. In Advances in Neural Information Processing Systems, pp. 337-344, 2009.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford,
John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:
2017.
Maxim Egorov. Deep reinforcement learning with pomdps. 2015.
Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communi- cate solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv:1602.02677, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01220, 2018.
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Seriex, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Shiyu Huang, Hang Su, Jun Zhu, and Ting Chen. Combo-action: Training agent for fiss game with auxiliary tasks. AAAL, 2019.
Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep variational reinforcement learning for pomdps. arXiv preprint arXiv: (tt1::v80002222 2018.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-334 1998.
Peter Karkus, David Hsu, and Wee Sun Lee. Qmdp-net: Deep learning for planning under partial observability. In Advances in Neural Information Processing Systems, pp. 4694 470444704, 2017.
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 6980, 2014.
Diederik Kingma and Max Welling, Auto-cncoding variational bayes. arXiv preprint 1rXiv: 14, 2013.
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learn- ing. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Tuan Anh Le, Maximilian Igl, Tom Rainforth, Tom Jin, and Frank Wood. Auto-encoding sequential monte carlo. arXiv preprint arXiv: /705. 10306, 2017.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review.
arXiv preprint arXiv: 180500 2018
David A McAllester and Satinder Singh. Approximate planning for factored pomdps using belief state simplification. arXiv preprint arXiv:1301.67199 2013.
Andrew McCallum. Overcoming incomplete perception with utile distinction memory. In Pro- ceedings of the Tenth International Conference on Machine Learning, pp. 190-196, 1993.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7500))529, 2015.
11

---

Published as a conference paper at ICLR 2020
Zhen-Jia Pang, Ruo-Ze Liu, Zhou- Yu Meng, Yi Zhang, Yang Yu, and Tong Lu. On reinforcement learning for full-length game of starcraft. arXiv preprint 1rXiv:1809 809.09099, 2018.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484445) 2016.
Trey Smith and Reid Simmons. Heuristic search value iteration for pomdps. In Proceedings of the 20th conference on Uncertainty in artificial intelligence, pp. 520-527. AUAI Press, 2004.
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C Lawrence Zitnick. Elf: An extensive, lightweight and flexible research platform for real-time strategy games. In Advances in Neural Information Processing Systems, pp. 2659-2669, 2017.
Yuxin Wu and Yuandong Tian. Training agent for first-person shooter game with actor-critic cur- riculum learning. 2016.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. Relational deep reinforcement learning. arXiv preprint arXiv: 001./000..8800 2018.
Pengfei Zhu, Xin Li, Pascal Poupart, and Guanghui Miao. On improving deep reinforcement learn- ing for pomdps. arXiv preprint 1rXiv:1804.063100 00000, 2018.
12

---

Published as a conference paper at ICLR 2020
A DERIVATION OF THE VARIATIONAL LOWER BOUND
logT(Oo.T: do.T, 01:T)
((1.Tr OuT, aorr, 01:1)
= logg 10..00
(81. Oo.T corT, 01:T)
((i.TT Ourr aorr, 01:1)
2 Oun our) log
((1.T OorT, Go:T, 01:T)
= f d(SIT/O0TT 00. 01. (0
p(31:T, Oor, ap.r; 01:))
d81.T
[((1: Oo.T, 40:7; 01.7)
T
pla, )p O, (8,, a, Da
Σ
d3ie
Σ / q(31:e aoa, oog
as haappp00sss.nnnssssssoooooooooo
ds
Σ { ((s1n100a ane pat.
(19)
/ q($ Ores aoes on
p p(selst-1, dsia
[q() (11,)15-11,011,,10
Σ { a(51nllOe aon, bat
/
(atr::/D.........oo.....P.aa.aa. a-1) dsya)
-DKL bo
a1) (oggnra)))(al)))
DEL 11- a1- where 81:7 ~ q(81:r(11.1 Oo:r, 01:))
13

---

Published as a conference paper at ICLR 2020
B VARIATIONAL AUTOENCODERS
Variational auto-encoders (VAEs) (Kingma & Welling 2013 are effective generative models that can recover complex distributions over the data space. In the VAE, there are observed data æ and the underlying causal factor 2. Usually, it's intractable do inference for the posterior p(=|z)). The VAE uses a variational function (<z[x)) to approximate the true posterior. The lower bound of the marginal distribution the observed data is given by:
p(a,2)) =) [((([x]]
2 (o(a)) log p(r, 2) (((22)
logp(x)
loglE
(20)
and the lower bound can be equivalently written as
((x) = - - + (a(a))) [logp(z|2)]
(21)
where (-c1(-11-) is Kullback-Leibler divergence between two distributions. The approximate poste- rior (==/z) often set as Gaussian N (ju, diag(o?)), where μ and a are the outputs of non-linear function of æ. The generative model p(z|2) and inference model g(=[2) can be trained jointly via standard backpropagation techniques.
C OPTIMIZE THE KLS IN EQ. (15)
We give out the final form of the loss function as below:
/ p(x) p(x) dx o(x))
DRi(pllg)
/ p(x) log logg(z)de - ff p(o)g(a)de
Og
o
(pp -2g)2
log
0p
20²
2
1
2
log
0g
p
+ (pp - -2y)2
(22)
2
Dp
0²
1
0 + (Ap - m₂)
- 2 2logg
2
+
2
Og
o²
o + (11
mty)
2(log(o,) - log(o,)) +
2
o²
where p and q are two Gaussian distributions. on, og My and Mp are their standard deviations and means respectively.
14

---

Published as a conference paper at ICLR 2020
D NETWORK ARCHITECTURE
D.1 ATARI
The network architecture for Atari is shown as below:
Observation Reconstruction Module
Sowdd1
84
Image
Sear112
FINU un!
Scar512
Navs
Conv1:8x844 Bervl//xBeyoor2....k Conv2:4x4s2
21
32
32
Action
Sbar10
Inner Reconstruction Module
Reward
Observation Reconstruction Module
Size:21 168
Size:512
84
84
84
hidden state
84
84
84
Reshape
Convifxtt1
Sigmoid
Inner Reconstruction Module
Size:512
Size:512
Sire:512+action size
Size:612 1aatiinoonn sizo
Size:512
Innar mean
Size:512
hidden state
Sized5
variance
Action
15

---

Published as a conference paper at ICLR 2020
D.2 VIZDOOM
The network architecture ViZDoom is shown as below:
Eriteding
Stae32
Observation Reconstruction Module
Carra Variel
Intecky
Staed122
84
Conv8:3x31
8ec/512
Blco112
Image
Sbed112
Stzr112
same
84
1oxBe:8xBis4 4x422:4x4e2
32
32
Inner Reconstruction Module
Sex
Observation Reconstruction Module
Size:7056
Size:512
84
84
84
hidden state
84
84
84
Reshape
Convc1xta1
Sigmoid
Inner Reconstruction Module
Size:812
Size:512
Size:512 size
Size:512-action size
Size:512
Size:512
hidden
Size:512
sample
inner varianca
Action
16

---

Published as a conference paper at ICLR 2020
E
TRAINING DETAILS
Training architechture:
Game
Game
Game
Data Collector
Game
Game
Data Collector
Game
Game
Data Collector
Actor
Game
training batch
Game
parameter update
Evaluator
Trainer
parameter update
The data produce module, training module and evaluation module are run in parallel. The framework implemented by Python and Tensorflow.
E.
ATARI
All the models take only one observation input at each timestep and the vision inputs are resized the resolution of 84 x 84. For the recurrent neural networks, we use sequence length of for training. All the algorithms train for 1,0000,000 steps and run for 100 episodes during evaluation.
E.2 VIZDOOM
All the algorithms are trained with the same basic network architectures and use the same hyperpa- rameters. All the models take only one observation input each timestep and the vision inputs are resized to the resolution of 84 x 84. For the recurrent neural networks, we use sequence length of 5 for training. All the algorithms train for 300,000 steps and run for 20 episodes during evalua- tion. The discount factor 7 is set to 0.95, learning rate is 0.0001 and the Adam optimizer is used for training.
17


Reviews:
[{"review_id": "SJls05-ptS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes SVQN, an algorithm for POMDPs based on the soft Q-learning framework which uses recurrent neural networks to capture historical information for the latent state inference. In order to obtain this formulation, the author first derive the variational bound for POMDPs and then present a practical algorithm.\n\nThe key idea of the paper is to replace DQN with Soft Q-learning that already demonstrated better performance on a variety of tasks. This seems to be an obvious extension of DRQNs (Hausknecht & Stone, 2015) even though it did not appear in the literature.\n\nThe authors evaluate the final algorithm on a set of ALE and DoomViz tasks. The algorithm outperforms the previous methods, in particular, DRQNs. The set of tasks and prior methods is adequate.\n\nOverall, the contribution of the paper is not significant enough to be accepted to ICLR.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgdqX8HKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new sequential model-free Q-learning methodology for POMDPs that relies on variational autoencoders to represent the hidden state. The approach is generic, well-motivated and has  clear applicability in the presence of partial observability. The idea is to create a joint model for optimizing the hidden-state inference and planning jointly. For that reason variational inference is used to optimize the ELBO objective in this particular setting. All this is combined with a recurrent architecture that makes the whole process feasible and efficient.\n\nThe work is novel and it comes with the theoretical derivation of a variational lower bound for POMDPs in general. This intuition is exploited to create a VAE based recurrent architecture. One motivation comes from maximal entropy reinforcement learning (MERL), but which has the ad hoc objective of maximizing the policy entropy. On the other hand SVQN optimizes both a variational approximation of the policy and that of the hidden state. Here the rest terms of the ELBO objective can be approximated generatively and some of them are conditioned on the previous state which calls for a recurrent architecture. The other parts are modeled by a VAE.\n\nThe paper also explores two different recurrent models in this context: GRU and LSTM are both evaluated. Besides the nice theoretical derivation the paper presents compelling evidence by comparing this approach to competing approaches on four games of the flickering ATARI benchmark and outperforming the baselines significantly. Also both the GRU and LSTM version outperforms the baseline methods on various tasks of the VIZDoom benchmark as well.\n\nIn general, I find that this well written paper presents a significant progress in modelling POMDPS in a model-free manner with nice theoretical justification and compelling empirical evidence.\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel model-free solution to POMDPs, which proposes a unified graphical model for hidden state inference and max entropy RL. The method is principled and provides good empirical results on a set of experiments that relatively comprehensive. I would have liked to see more POMDP tasks instead of Atari, but the results are good. Overall this is good work.
Decision: Accept (Poster)