Title: SVQN: SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS

Abstract: Partially Observable Markov Decision Processes (POMDPs) are popular and flexible models for real-world decision-making applications that demand the information from past observations to make optimal decisions. Standard reinforcement learning algorithms for solving Markov Decision Processes (MDP) tasks are not applicable, as they cannot infer the unobserved states. In this paper, we propose a novel algorithm for POMDPs, named sequential variational soft Q-learning networks (SVQNs), which formalizes the inference of hidden states and maximum entropy reinforcement learning (MERL) under a unified graphical model and optimizes the two modules jointly. We further design a deep recurrent neural network to reduce the computational complexity of the algorithm. Experimental results show that SVQNs can utilize past information to help decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation. * J. Zhu and T. Chen are corresponding authors.

1. INTRODUCTION:

In recent years, substantial progress has been made in deep reinforcement learning for solving various challenging tasks, including the computer Go game (Silver et al., 2016) , Atari games (Mnih et al., 2015) , StarCraft (Zambaldi et al., 2018; Pang et al., 2018) and the first-person shooting (FPS) games (Lample & Chaplot, 2017; Wu & Tian, 2016; Huang et al., 2019) . However, in many real-world applications, decision-making problems are partially observable (Aström, 1965) , preventing such problems from being solved by standard reinforcement learning algorithms. Formally, these kinds of problems are often defined as Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998) , which demand information from past observations to help in the decision-making process (McCallum, 1993) .

Although numerous efforts (Hausknecht & Stone, 2015; Foerster et al., 2016; Igl et al., 2018; Zhu et al., 2018) have been paid to tackle this problem, there still exist various challenges. For example, Egorov (2015) tries to solve POMDPs by using the belief of the agent as the input of DQN (Mnih et al., 2015) , but this algorithm needs access to the environment model. However, in many reinforcement learning tasks, it is not possible for the agent to acquire the underlying transition function, making such algorithms inapplicable. Some recent work (Karkus et al., 2017; McAllester & Singh, 2013; Babayan et al., 2018) tries to solve POMDPs under the model-free setting, i.e., the agent does not need to know and learn the transition function of the environment. For instance, Karkus et al. (2017) trained an agent to navigate in a partially observable grid world under the model-free setting, i.e., the agent can only observe a part of the grid world and does not learn the transition function. The agent uses its local observations to update its beliefs (McAllester & Singh, 2013; Babayan et al., 2018) . In their experiments, the ground truth of the state is the full map plus the location of the agent, which means the representation of the state is explicit. However, in some complex tasks, it is impossible to acquire or design the state or beliefs.

To solve the unknown representation problem of the state, Hausknecht & Stone (2015) and Zhu et al. (2018) try to represent the state as latent variables of neural networks. However, they only use a deep recurrent neural network to capture the historical information and fail to utilize the Markov property of the state in POMDPs. Igl et al. (2018) apply sequential Monte Carlo (SMC) (Le et al., 2017) to introduce inductive bias to the neural network, which can embody the Markov properties of the state. They can infer the state from the past observations online. However, they separate the planning algorithm from the inference of the state.

To infer the hidden states and optimize the planning module jointly, we represent POMDPs as a unified probabilistic graphical model (PGM) and derive a single evidence lower bound (ELBO). We apply structured variational inference to optimize the ELBO. In our implementation, we design generative models to infer the hidden variables, however, the distribution of the latent variables is conditioned on previous hidden states. This is different from standard VAEs (Kingma & Welling, 2013) , whose prior of the latent variables can be a standard Gaussian distribution. So, we apply an additional approximate function to tackle the conditional prior problem. The planning can also be solved under the PGM framework. Fortunately, maximum entropy reinforcement learning (MERL) (Levine, 2018) provide a tool to formalize the planning as a probabilistic inference task.

In this paper, we propose a novel end-to-end neural network called the sequential variational soft Q-learning network (SVQN), which integrates the learning of hidden states and the optimization of the planning within the same framework. A deep recurrent neural network (RNN) (Cho et al., 2014; Hochreiter & Schmidhuber, 1997) in SVQNs is used to reduce the computational complexity, because the feature extraction can share the same weights in the RNN. Experimental results show that the SVQN can utilize past information to help in decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.

Contributions::

(1) We derive the variational lower bound for POMDPs, which allows us to integrate the optimization of the control problem and learning of the hidden state under a unified graphical model. (2) We propose to tackle the difficulty of the inference of the hidden state and solve the problem of a conditional prior using the generative models. (3) We design an end-to-end deep recurrent neural network, which can reduce the computational complexity and be trained efficiently.

2. RELATED WORK:

We summarize some related work in POMDPs and inference methods for sequential data.

Model-Based and Model-Free methods for POMDPs: When the environment model is accessible, POMDPs can be solved by model-based method. Egorov (2015) used model-based methods to solve POMDPs, but their agents need to know the belief-update function and the transition function. When the environment model is unknown, model-free methods should be applied. Recently, some researchers (Hausknecht & Stone, 2015; Zhu et al., 2018) used recurrent neural networks to capture the historical information, but they failed to utilize the Markov property of the state in POMDPs. Our work proposes generative models for the algorithm learning, which tackles the difficulty of the inference of hidden states and introduces inductive bias to the network structure. Igl et al. (2018) applied sequential Monte Carlo (SMC) to the POMDPs. They can infer the hidden state from the past observations online. However, they separate the planning algorithm and the inference of the hidden state. Our algorithm is derived from a unified graphical model, which can train the inference model and the planning algorithm jointly.

Explorations in POMDPs::

In contrast to MDP methods, POMDP methods should do exploration to gather information with no immediate reward which can then be used in the future to gain higher rewards. Pathak et al. (2017) and Choi et al. (2018) designed deep reinforcement learning algorithms to help the exploration in their tasks. Actually, their tasks are POMDP tasks. However, they just treat their tasks as MDP tasks and intuitively add exploration tricks to standard reinforcement learning algorithms. Their exploration tricks include the reconstruction of observations and actions, which comes from intuitive concepts such as curiosity and attention. The generative models in our method also need to reconstruct the observations and actions, but our algorithm is derived from solid theoretical foundations.

Inference for Sequential Data: The observation in POMDP tasks is sequential data, and we need to do inference for the hidden state from observations. Coquelin et al. (2009) used a particle filter to estimate the belief state given past observations. However, their method needs to access to the Figure 1 : The graphical models for Markov decision processes (MDPs) (a) and partially observable Markov processes (POMDPs) (b). Grey nodes are observed, white nodes are hidden. In POMDPs, the state s is not observable and must be inferred from past observations. The O t is a binary random variable, where O t = 1 denotes that the action is optimal at time t, and O t = 0 denotes that the action is not optimal.

environment model. Chung et al. (2015) derived the evidence lower bound of latent variables for sequential data and designed a novel deep neural network to infer recurrent latent variables. Our inference method has some superficial resemblances to their method, but we derived the evidence lower bound from a different graphical model and tackle the difficulty of how to integrate the RL planning algorithm instead of just inferring the hidden variables.

3. PRELIMINARIES:

We start by briefly reviewing the backgrounds and notations related to our method, including partially observable Markov decision processes and maximum entropy reinforcement learning which solves optimal control problems using a probabilistic framework.

3.1. PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES:

In many control tasks, the complete state of the environment is unknown to the agent. Instead, the agent can only receive local observations, which are typically conditioned on the current state of the system. The agent needs to make decisions based on the historical information. This kind of problems can be formalized as partially observable Markov decision processes (POMDPs) (Smith & Simmons, 2004) . Formally, a POMDP is represented as a tuple (S, A, O, T, Z, r), where S, A and O are state space, action space and observation space, respectively. The reward function r(s, a) is the received reward when taking action a in state s. T (s, a, s ) is the state-transition function, which defines the probability of the succeeding state s after taking action a in state s. Z(s, a, o) is the observation function, which defines the probability of emitted observation o after taking action a in state s.

POMDPs use the belief b t (s) to maintain the distribution of the unknown state at time t, and the agent updates its belief with a Bayesian filter when receiving a new observation as

EQUATION

where η is a normalizing constant and a t is the action at time t. POMDP planning needs to find a policy π to maximize the accumulative reward as

EQUATION

where s t is the state at time t, and γ ∈ (0, 1) is a discount factor.

3.2. MAXIMUM ENTROPY REINFORCEMENT LEARNING:

POMDPs require solving two problems: the inference of state and optimal control. To integrate both of them under a unified framework, we represent POMDPs as a probabilistic graphical model. So the optimal control problem needs to be solved as a probabilistic inference task. Fortunately, the maximum entropy reinforcement learning (MERL) algorithm provides an effective tool to solve optimal control problem under PGM framework. The graphical model of Markov decision processes is shown in Fig. 1 (a). We borrow the notation from Levine (2018) to illustrate the algorithm. Levine (2018) introduces a binary random variable O t to the graphical model, where O t = 1 denotes that the action is optimal at time t, and O t = 0 denotes that the action is not optimal. The probability distribution of O is p(O t = 1|s t , a t ) = exp(r(s t , a t )), and the variational lower bound is given by:

EQUATION

where π(a|s) is the policy function. Standard reinforcement learning only needs to maximize the cumulative reward. However, MERL will maximize an extra term, which is the policy entropy at each visited state. To get the optimal solution for MERL, two messages are introduced, i.e., β t (s t , a t ) = p(O t:T |s t , a t ) and β t (s t ) = p(O t:T |s t ), and their relations are given by:

EQUATION

and the optimal policy is given by:

EQUATION

)

We can define the Q-value function and V-value function as below:

EQUATION

EQUATION

and the update functions are:

EQUATION

To maximize the variational lower bound in Eq. ( 3), we can use a parameterized Q-function Q θ (s t , a t ), where θ is the function parameter, and it can be learned via gradient descent:

EQUATION

where α is the learning rate. This algorithm is called soft Q-learning, because the update function for Q-value can be considered a soft-update version of the standard Bellman backup. For a better understanding of MERL, we recommend readers reference this tutorial (Levine, 2018) .

4. SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS:

We now present our algorithm in detail. We first derive the variational lower bound for POMDPs, and then illustrate how to deal with the conditional prior.

4.1. VARIATIONAL LOWER BOUND FOR POMDPS:

Different from MDPs, the state s of POMDPs in the PGM (shown in Fig. 1(b) ) is unobservable, which need to be inferred from the action a and the observation o. We need to derive different variational lower bound for POMDPs, which can be used to infer the hidden state and do planning jointly.

Unlike the observation function Z(s, a, o) in Section 2.1, we assume that the observation is emitted from the hidden state s, which means the probability distribution of observations are only conditioned on states, i.e., o t ∼ p(o t |s t ). This assumption can hold in many tasks. For example, the observation in a partially observed maze is only determined by the full map and the agent's location. We apply the structured variational inference to optimize the evidence lower bound of POMDPs. In structured variational inference, different parts of the proposal distributions can be optimized separately, which means we can fix some approximate functions and optimize other approximate functions. In POMDPs, we will use two approximate functions q π (a t |s t ) and q θ (s t |s t-1 , a t-1 , o t ). q π (•) approximates the optimal policy and q θ (•) approximates the function of inferring hidden states, where θ is the parameter of the approximate function. When q θ (•) is fixed, the learning procedure is same as MERL, so that q π (•) can be learned via the soft Q-learning algorithm. Conversely, when q π (•) is fixed as the optimal policy, we can learn the inference function q θ (•) for hidden states. We can derive the evidence lower bound(ELBO) as bellow:

EQUATION

and the ELBO can be written as:

EQUATION

To get the optimal action in POMDPs, we need to maximize the ELBO above. The first term in the ELBO is the cumulative reward T t=1 r(s t , a t ), which can be optimized via maximum entropy reinforcement learning algorithm.

We apply generative models to optimize the rest terms in the ELBO. The term p(o t |s t ) means that the hidden state s t needs to have the ability to generate current observation o t . The Kullback-Leibler divergence term D KL [q θ (s t |s t-1 , a t-1 , o t )||p(s t |s t-1 , a t-1 )] indicates that we should minimize the gap between approximate function q θ (s t |s t-1 , a t-1 , o t ) and the prior p(s t |s t-1 , a t-1 ). In standard VAEs, the prior p(•) is generally fixed and can be set as a standard Gaussian. However, in this generative model, the prior of the hidden state s t is conditioned on previous state s t-1 and action a t-1 . We will show how we deal with the conditional prior in next sub-section. More details about the derivation process of the ELBO can be found in Appendix A.

4.2. VARIATIONAL AUTOENCODERS FOR THE CONDITIONAL PRIOR:

The KL-divergence term in Eq. ( 11) introduces a conditional prior for the hidden state s t , but the true conditional distribution p(s t |s t-1 , a t-1 ) is unknown, making it intractable to do inference. Hence we propose a new parameterized function p prior θ (s t |s t-1 , a t-1 ) to approximate the true distribution p(s t |s t-1 , a t-1 ). We use a standard VAE to learn the p prior θ (•). Hence (s t-1 , a t-1 ) can be treated as the inner observed data in this VAE model and the inner hidden state ŝt is inferred from (s t-1 , a t-1 ). We can write down the ELBO for the conditional prior directly by using the ELBO of standard VAEs (see in Appendix B):

EQUATION

where p(s t ) can be set as a standard Gaussian. The p prior θ (•) can be learned via a standard VAE paradigm.

The outputs of p prior θ (•) are µ prior and σ prior , so that the KL-divergence term in Eq. ( 11) can be written as:

EQUATION

Unlike standard VAEs, the second distribution of the KL-divergence function in Eq. ( 13) is not a standard normal distribution. But we can still get the final loss function if we expand the KLdivergence function. We present the final formula of the loss function for Eq. ( 13) in Appendix C.

We design a deep recurrent neural network to improve the ELBO derived in Section 4.1 and Section 4.2. Fig. 2 shows the overall structure of our algorithm.

In our implementation, there are two generative models, i.e., one of the generate model learns the conditional prior q prior θ (•) and reconstructs the inner observed data (s t-1 , a t-1 ), and another generate model learns the q θ (•) in Eq. ( 13) and reconstructs the current observation o t .

For the first generative model, there are two losses, i.e., L inner KL and L inner M SE . L inner Finally, we get two kinds of loss functions for the two generative models, i.e., the reconstruction loss L M ES = L inner M SE + L elbo M SE and the KL-divergence loss L KL = L inner KL + L elbo KL . And for the planning algorithm, we use the soft Q-learning algorithm (Levine, 2018) . Its loss function is the temporal difference error L T D , which is updated via Eq. ( 9). All these losses can jointly be optimized via stochastic gradient descent algorithms.

To reduce the computation complexity, we use a deep recurrent neural network to capture the historical information. We first use the f θ (o t , a t-1 , r t-1 ) to extract a low-dim hidden feature w t from current input, and then feed this features to a recurrent unit rnn θ (•) to get the recurrent output h t . Because h t contains the information of past observations, the loss function in Eq. ( 16) can be rewritten as:

EQUATION

In the training, we tried both LSTM cell (Hochreiter & Schmidhuber, 1997) and GRU cell (Cho et al., 2014) as basic recurrent units. Because of the generalization ability of the recurrent neural network (Lample & Chaplot, 2017) , we can just sample a fixed length H of sequential data for training instead of using the full-length data. In the experiment, we also studied how the training length H influences the final performance. We use a parallel training strategy, i.e., the program hosts multiple games in parallel and they all send batched data to a central data memory. This is quite similar to the data collection method in ELF (Tian et al., 2017) . More details about the training strategy can be found in Appendix E.

5. EXPERIMENTS:

We evaluate our algorithm on flickering Atari (Hausknecht & Stone, 2015) and ViZDoom platform (Kempka et al., 2016) . Flickering Atari was previously used as the test environment in DRQN (Hausknecht & Stone, 2015) , ADRQN (Zhu et al., 2018) and DVRL (Igl et al., 2018) . The ViZDoom platform is a 3D FPS game for AI research. The agent in ViZDoom needs to navigate in the 3D environment to accomplish various tasks, such as gathering resources, shooting enemies and looking for the exit. Because the agent can only see in just one direction each time step, this naturally introduces partial observability to the task.

5.1. EXPERIMENT SETUP:

We used some recent algorithms as baselines. Because the true state and transition function are unknown, only the methods which can be trained under model-free setting will be used for comparison.

Baselines are listed as below: Deep Q-Networks (DQN) (Mnih et al., 2015) : A method which uses standard Bellman backup and uses the temporal difference error as the objective function.

Deep Soft Q-Networks (DSQN) (Levine, 2018) : A method which uses soft Bellman backup and uses the temporal difference error as the objective function.

Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone, 2015) : A method which applies LSTM on the top of the DQN. Action-Specific Deep Recurrent Q-Networks (ADRQN) (Zhu et al., 2018) : A method which extends the DRQN, i.e., it uses both the observation and action as inputs.

Deep Variational Reinforcement Learning (DVRL) (Igl et al., 2018) A method which combines sequential Monte Carlo and A2C (Dhariwal et al., 2017) to solve POMDPs.

5.2. EVALUATION ON FLICKERING ATARI:

Atari environments (Bellemare et al., 2013) are widely used as the benchmark for deep reinforcement learning algorithms due to its high dimensional observation spaces and numerous challenging tasks. Flickering Atari was introduced by (Hausknecht & Stone, 2015) . In each running step, the observation may be obscured with a certain probability, i.e., the raw screen will be either fully observable or fully obscured with black pixels. The settings of experiments keep in line with DVRL (Igl et al., 2018) , i.e., only one frame is used, the frameskip is set to four and each frame is obscured with a probability of 0.5.

We choose four Atrai games (i.e., Pong, ChopperCommand, DoubleDunk and Asteroids) for evaluation. Fig. 3 shows the screen shots of these games. When only given one frame as the input, these tasks are POMDPs because we cannot obtain the velocity of the moving object from a single observation. These tasks have high dimensional and continuous observation spaces, while discrete action spaces. The basic network architecture and hyper-parameters are similar to DQN (Mnih et al., 2015) . For the recurrent neural networks, we use a sequence length of 5 for training. All the algorithms train for 10000,000 steps and run for 100 episodes during evaluation. The training details can be found in Appendix E.

Table 1 shows the performance results of different algorithms on these Atari games. We can see that our algorithms significantly outperform other baselines on three of the games and get close score to DVRL on ChopperCommand. Compared with DRQN and ADRQN, our method introduces inductive bias to the network, which helps state estimate and RL planning for POMDPs. Compared with DVRL, our method can achieve competitive performance with lower sampling complexity.

DQN DSQN DRQN ADRQN DVRL SVQN(GRU) SVQN(LSTM)

Pong -4.9(±2.6) -2.1(±2.3) 1.6(±7.8) 7(±4.6) 18.2(±2.7) 19.2(±1.3) 18.6(±1.9) Chopper 1350(±731) 1250(±522) 1090(±409) 1608(±707) 6602(±449) 6005(±258) 5805(±312) DDunk -16.2(±3.1) -16.9(±2.8) -14.4(±3.2) -12.9(±3.6) -6.0(±1.3) -5.8 (±1.4) -5.5(±1.2) Asteroids 935(±410) 940(±320) 871(±340) 1040(±432) 1539(±73) 1645(±102) 1585(±86) 

5.3. EVALUATION ON VIZDOOM TASKS:

We designed three tasks in ViZDoom as the evaluation tasks for our algorithm.

Health Gather: The agent needs to gather health supplies in a flat map. The agent will lose health every time step. Accordingly, if it can't gather enough health supplies, it will die and the game is over. At each time step, the agent will receive a reward of 0.001, which encourages the agent to live a longer life. When it collects one health supply, it will receive a reward of 1. The observation for the agent is the grayscale image of its vision and its health value. The agent has three actions to choose, i.e., TURN LEFT, TURN RIGHT and MOVE FORWARD.

Health Gather v2: This is a more difficult task than previous task with a more complex map. There is some poison in the map. When the agent encounters poison, it will lose health. We use the same reward scenario, observations and actions in Health Gather.

Defend Center: In this task, the agent stands in the center of a flat map. Monsters will walk toward the agent from the edge of the map. When the monsters touch the agent, the agent will lose health. The agent holds a gun with limited ammo, and it can kill the monsters by pressing down the shooting button. The agent gets reward of 4 by killing monsters, reward of -0.2 by using ammo and reward of The SVQN models achieve the best performance on these three tasks.

-1 when losing health. The observation for the agent is the grayscale image of its vision, its health value and the ammo remained. The agent has three actions to choose, i.e., TURN LEFT, TURN RIGHT and ATTACK. 2, 5, 10, 15) . We can see that, when the sequence length is too short(H = 2), the model can't gather enough historical information, which causes performance degradation. When the sequence length is too long(H = 15), it is arduous for the network to do optimization, which will also lead to performance degradation. This experiment also shows that with a limited training length, the algorithm can generalize to the time of arbitrary lengths during testing. Thanks to this generalization ability, we can train the agent with a fixed length of data instead of the full-length data, which can reduce the computation complexity.

Observation Probability: We also study whether different models are robust to the noise of the observation. We add a modification to the game Health Gather, i.e., at each time step, the observation of the screen is either fully revealed or fully obscured with a fixed observation probability p. Fig. 4 (c) show the evaluation results of different models under different observation probabilities(p = 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). All the models are trained under the standard environment with full observations. The results show that SVQN models are relatively robust to the disturbance of the observation compared to other algorithms.

6. CONCLUSIONS:

We propose a novel algorithm named Sequential Variational Soft Q-Learning Networks (SVQN) to solve POMDPs with the discrete action space. SVQN is model-free and does not need to know the true state's representation. We apply generative models to deal with the conditional prior of hidden states and use a recurrent neural network to reduce the computational complexity, i.e., with a small length of training data, it can generalize to the test data with an arbitrary length. Our designed deep neural network can be trained end-to-end, which optimizes the planning and inference of hidden states jointly. Experimental results show that SVQN outperforms previous methods on challenging tasks and has the robustness to the disturbance of the observation. SVQN is also flexible and can be integrated with other maximum entropy reinforcement learning algorithms, such as soft actorcritic (Haarnoja et al., 2018) . In the future, we will try to develop algorithms for POMDPs problems with the continuous action space.

A DERIVATION OF THE VARIATIONAL LOWER BOUND logp(O 0:T , a 0:T , o 1:T )

= logE q(s 1:T |O 1:T ,a 0:T ,o 1:T ) p(s 1:T , O 0:T , a 0:T , o 1:T ) q(s 1:T |O 0:T , a 0:T , o 1:T )

≥ E q(s 1:T |O 1:T ,a 0:T ,o 1:T ) log p(s 1:T , O 0:T , a 0:T , o 1:T ) q(s 1:T |O 0:T , a 0:T , o 1:T ) = q(s 1:T |O 1:T , a 0:T , o 1:T )log p(s 1:T , O 0:T , a 0:T , o 1:T ) q(s 1:T |O 0:T , a 0:T , o 1:T ) ds 1:T = T t=1 q(s 1:T |O 1:T , a 0:T , o , where s 1:T ∼ q(s 1:T |O 1:T , a 0:T , o 1:T )

B VARIATIONAL AUTOENCODERS Variational auto-encoders (VAEs) (Kingma & Welling, 2013) are effective generative models that can recover complex distributions over the data space. In the VAE, there are observed data x and the underlying causal factor z. Usually, it's intractable to do inference for the posterior p(z|x). The VAE uses a variational function q(z|x) to approximate the true posterior. The lower bound of the marginal distribution of the observed data is given by: logp(x) = logE q(z|x) p(x, z) q(z|x) ≥ E q(z|x) log p(x, z) q(z|x) ,

and the lower bound can be equivalently written as :

EQUATION

where D KL (•||•) is Kullback-Leibler divergence between two distributions. The approximate posterior q(z|x) is often set as Gaussian N (µ, diag(σ 2 )), where µ and σ are the outputs of a non-linear function of x. The generative model p(x|z) and inference model q(z|x) can be trained jointly via standard backpropagation techniques.

C OPTIMIZE THE KLS IN EQ. (15)

We give out the final form of the loss function as below:

D KL (p||q) = p(x) p(x) q(x) dx = p(x) log p(x)dx -p(x)q(x)dx

= log σ q σ p + σ 2 p + (µ p -µ q ) 2 2σ 2 q - 1 2 = 1 2 2 log σ q σ p + σ 2 p + (µ p -µ q ) 2 σ 2 q -1 = 1 2 -2 log σ p σ q + σ 2 p + (µ p -µ q ) 2 σ 2 q -1 = - 1 2 2(log(σ p ) -log(σ q )) + 1 - σ 2 p + (µ p -µ q ) 2 σ 2 q (22)

, where p and q are two Gaussian distributions. σ p , σ q , µ q and µ p are their standard deviations and means respectively.

D NETWORK ARCHITECTURE D.1 ATARI:

The network architecture for Atari is shown as below:

:



Figures and Tables:

Figure2: The structure of sequential variational soft Q-learning networks (SVQNs). Black solid lines represent forward paths of the neural network, gray dashed lines represent reconstruction paths and blue arrows stand for sampling latent variables using the re-parameterization trick. Doublearrows indicate that the algorithm needs to minimize the KL-divergence between two probability distributions. The model takes the observation o t , previous action a t-1 and reward r t-1 as inputs, and it uses a neural network f θ (o t , a t-1 , r t-1 ) to extract the low-dim hidden feature w t . The recurrent unit rnn θ (w t , h t-1 ) is used to capture the historical information h t . q θ (s t |h t ) and p θ (s t |s t-1 , a t-1 ) are proposed distributions of hidden states. The hidden state s t and inner hidden state ŝt are sampled from q θ (s t |h t ) and p prior

Figure 3: Screen shots of Atari games and ViZDoom tasks. From left to right, they are Pong, Chop-perCommand, DoubleDunk, Asteroids, Health Gather,Health Gather v2 and Defend Center. When only given one frame as the input, these Atari games are POMDPs because we can not obtain the velocity of the moving object from a single observation. Because the agent in ViZDoom environment can only see in just one direction, it naturally introduces partial observability to these tasks.

Figure 4: (a) Training curves on Health Gather. Both SVQN models outperform other baselines. (b) Performances of the models with different training sequence lengths on Health Gather. When the training sequence length is too long, it is hard for the algorithm to gather useful information through a long gradient flow. (c) Evaluation results of different models under different observation probabilities of the Health Gather, and all models are trained with full observation. The results show that SVQN models are more robust to the disturbance of the observation than other algorithms.

Fig.3shows the screen shots of these three tasks. All the algorithms are trained with the same basic network architecture and use the same hyperparameters. All the models take only one observation as input at each time step and the vision inputs are resized to the resolution of 84×84. For the recurrent neural networks, we use a sequence length of 5 for training. All the algorithms train for 300,000 steps and run for 20 episodes during evaluation. The discount factor γ is set to 0.95, learning rate is 0.0001 and the Adam optimizer (Kingma & Ba, 2014) is used for training. The network architecture for these tasks and training details are shown in Appendix D. Fig. 4(a) shows the training curves of different methods on the Health Gather. Both SVQN models outperform other baselines. Table2reports the final scores of different models on these three tasks. The SVQN models achieve the best performance on these three tasks. Experiments on ViZDoom indicate that generative models of SVQNs can improve agent's exploration ability in complicated unknown environment

Evaluation results of different models on flickering Atari. The values are the final evaluation scores after training for different algorithms. Values in parentheses indicate the standard deviation. Evaluations use Mann-Whitney rank test and bold numbers indicate statistical significance at the 5% level. Our algorithms outperform other baselines on three of the games and get close score to DVRL on ChopperCommand.

Table content: <table/>

Evaluation results of different models on ViZDoom. The values are the final evaluation scores after training for different algorithms. Values in parentheses indicate the standard deviation. Evaluations use Mann-Whitney rank test and bold numbers indicate statistical significance at the 5% level.

Table content: <table/>

1:T )log p(a t )p(O t |s t , a t )p(s t |s t-1 , a t-1 )p(o t |s t ) q(s t |s t-1 , a t-1 , o t ) ds 1:T |O 1:t , a 0:t , o 1:t )log p(a t )p(O t |s t , a t )p(s t |s t-1 , a t-1 )p(o t |s t ) q(s t |s t-1 , a t-1 , o t ) ds 1:t |O 1:t , a 0:t , o 1:t )log [p(a t )p(O t |s t , a t )p(o t |s t )] ds 1:t + q(s 1:t |O 1:t , a 0:t , o 1:t )log p(s t |s t-1 , a t-1 ) q(s t |s t-1 , a t-1 , o t ) ds 1:t |O 1:t , a 0:t , o 1:t )log [p(a t )p(O t |s t , a t )p(o t |s t )] ds 1:t-q(s 1:t-1 |O 1:t-1 , a 0:t-1 , o 1:t-1 )D KL [q(s t |s t-1 , a t-1 , o t )||p(s t |s t-1 , a t-1 )] ds 1:t = E q(s 1:T |O 1:T ,a 0:T ,o 1:T ) )p(O t|s t , a t )p(o t |s t )] -D KL [q(s t |s t-1 , a t-1 , o t )||p(s t |s t-1 , a t-1 )] , a t ) + log [p(a t )p(o t |s t )] -D KL [q(s t |s t-1 , a t-1 , o t )||p(s t |s t-1 , a t-1 )]

Table content: <table><tr><td>T</td></tr><tr><td>= q(s 1:t = t=1 T t=1 T q(s 1:t = t=1 q(s 1:t T</td></tr><tr><td>t=1 log [p(a t T</td></tr><tr><td>r(s t</td></tr><tr><td>t=1</td></tr></table>

Peer review (iclr2020):
This paper proposes a new sequential model-free Q-learning methodology for POMDPs that relies on variational autoencoders to represent the hidden state. The approach is generic, well-motivated and has clear applicability in the presence of partial observability. The idea is to create a joint model for optimizing the hidden-state inference and planning jointly. For that reason variational inference is used to optimize the ELBO objective in this particular setting. All this is combined with a recurrent architecture that makes the whole process feasible and efficient. The work is novel and it comes with the theoretical derivation of a variational lower bound for POMDPs in general. This intuition is exploited to create a VAE based recurrent architecture. One motivation comes from maximal entropy reinforcement learning (MERL), but which has the ad hoc objective of maximizing the policy entropy. On the other hand SVQN optimizes both a variational approximation of the policy and that of the hidden state. Here the rest terms of the ELBO objective can be approximated generatively and some of them are conditioned on the previous state which calls for a recurrent architecture. The other parts are modeled by a VAE. The paper also explores two different recurrent models in this context: GRU and LSTM are both evaluated. Besides the nice theoretical derivation the paper presents compelling evidence by comparing this approach to competing approaches on four games of the flickering ATARI benchmark and outperforming the baselines significantly. Also both the GRU and LSTM version outperforms the baseline methods on various tasks of the VIZDoom benchmark as well. In general, I find that this well written paper presents a significant progress in modelling POMDPS in a model-free manner with nice theoretical justification and compelling empirical evidence.