SVQN: SEQUENTIAL VARIATIONAL SOFT Q- LEARNING NETWORKS
Year: 2020

Abstract:
Partially Observable Markov Decision Processes (POMDPs) are popular and flexible models for real-world decision-making applications that demand the information from past observations to make optimal decisions. Standard reinforcement learning algorithms for solving Markov Decision Processes (MDP) tasks are not applicable, as they cannot infer the unobserved states. In this paper, we propose a novel algorithm for POMDPs, named sequential variational soft Q-learning networks (SVQNs), which formalizes the inference of hidden states and maximum entropy reinforcement learning (MERL) under a unified graphical model and optimizes the two modules jointly. We further design a deep recurrent neural network to reduce the computational complexity of the algorithm. Experimental results show that SVQNs can utilize past information to help decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.

heading: 1 INTRODUCTION
text: In recent years, substantial progress has been made in deep reinforcement learning for solving various challenging tasks, including the computer Go game (Silver et al., 2016), Atari games (Mnih et al., 2015), StarCraft (Zambaldi et al., 2018; Pang et al., 2018) and the first-person shooting (FPS) games (Lample & Chaplot, 2017; Wu & Tian, 2016; Huang et al., 2019). However, in many real-world applications, decision-making problems are partially observable (Aström, 1965), preventing such problems from being solved by standard reinforcement learning algorithms. Formally, these kinds of problems are often defined as Partially Observable Markov Decision Processes (POMDPs) (Kaelbling et al., 1998), which demand information from past observations to help in the decision-making process (McCallum, 1993).
Although numerous efforts (Hausknecht & Stone, 2015; Foerster et al., 2016; Igl et al., 2018; Zhu et al., 2018) have been paid to tackle this problem, there still exist various challenges. For example, Egorov (2015) tries to solve POMDPs by using the belief of the agent as the input of DQN (Mnih et al., 2015), but this algorithm needs access to the environment model. However, in many reinforcement learning tasks, it is not possible for the agent to acquire the underlying transition function, making such algorithms inapplicable. Some recent work (Karkus et al., 2017; McAllester & Singh, 2013; Babayan et al., 2018) tries to solve POMDPs under the model-free setting, i.e., the agent does not need to know and learn the transition function of the environment. For instance, Karkus et al. (2017) trained an agent to navigate in a partially observable grid world under the model-free setting, i.e., the agent can only observe a part of the grid world and does not learn the transition function. The agent uses its local observations to update its beliefs (McAllester & Singh, 2013; Babayan et al., 2018). In their experiments, the ground truth of the state is the full map plus the location of the agent, which means the representation of the state is explicit. However, in some complex tasks, it is impossible to acquire or design the state or beliefs.
To solve the unknown representation problem of the state, Hausknecht & Stone (2015) and Zhu et al. (2018) try to represent the state as latent variables of neural networks. However, they only use
∗J. Zhu and T. Chen are corresponding authors. J. Zhu is also with RealAI.
a deep recurrent neural network to capture the historical information and fail to utilize the Markov property of the state in POMDPs. Igl et al. (2018) apply sequential Monte Carlo (SMC) (Le et al., 2017) to introduce inductive bias to the neural network, which can embody the Markov properties of the state. They can infer the state from the past observations online. However, they separate the planning algorithm from the inference of the state.
To infer the hidden states and optimize the planning module jointly, we represent POMDPs as a unified probabilistic graphical model (PGM) and derive a single evidence lower bound (ELBO). We apply structured variational inference to optimize the ELBO. In our implementation, we design generative models to infer the hidden variables, however, the distribution of the latent variables is conditioned on previous hidden states. This is different from standard VAEs (Kingma & Welling, 2013), whose prior of the latent variables can be a standard Gaussian distribution. So, we apply an additional approximate function to tackle the conditional prior problem. The planning can also be solved under the PGM framework. Fortunately, maximum entropy reinforcement learning (MERL) (Levine, 2018) provide a tool to formalize the planning as a probabilistic inference task.
In this paper, we propose a novel end-to-end neural network called the sequential variational soft Q-learning network (SVQN), which integrates the learning of hidden states and the optimization of the planning within the same framework. A deep recurrent neural network (RNN) (Cho et al., 2014; Hochreiter & Schmidhuber, 1997) in SVQNs is used to reduce the computational complexity, because the feature extraction can share the same weights in the RNN. Experimental results show that the SVQN can utilize past information to help in decision making for efficient inference, and outperforms other baselines on several challenging tasks. Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation.
Contributions: (1) We derive the variational lower bound for POMDPs, which allows us to integrate the optimization of the control problem and learning of the hidden state under a unified graphical model. (2) We propose to tackle the difficulty of the inference of the hidden state and solve the problem of a conditional prior using the generative models. (3) We design an end-to-end deep recurrent neural network, which can reduce the computational complexity and be trained efficiently.

heading: 2 RELATED WORK
text: We summarize some related work in POMDPs and inference methods for sequential data.
Model-Based and Model-Free methods for POMDPs: When the environment model is accessible, POMDPs can be solved by model-based method. Egorov (2015) used model-based methods to solve POMDPs, but their agents need to know the belief-update function and the transition function. When the environment model is unknown, model-free methods should be applied. Recently, some researchers (Hausknecht & Stone, 2015; Zhu et al., 2018) used recurrent neural networks to capture the historical information, but they failed to utilize the Markov property of the state in POMDPs. Our work proposes generative models for the algorithm learning, which tackles the difficulty of the inference of hidden states and introduces inductive bias to the network structure. Igl et al. (2018) applied sequential Monte Carlo (SMC) to the POMDPs. They can infer the hidden state from the past observations online. However, they separate the planning algorithm and the inference of the hidden state. Our algorithm is derived from a unified graphical model, which can train the inference model and the planning algorithm jointly.
Explorations in POMDPs: In contrast to MDP methods, POMDP methods should do exploration to gather information with no immediate reward which can then be used in the future to gain higher rewards. Pathak et al. (2017) and Choi et al. (2018) designed deep reinforcement learning algorithms to help the exploration in their tasks. Actually, their tasks are POMDP tasks. However, they just treat their tasks as MDP tasks and intuitively add exploration tricks to standard reinforcement learning algorithms. Their exploration tricks include the reconstruction of observations and actions, which comes from intuitive concepts such as curiosity and attention. The generative models in our method also need to reconstruct the observations and actions, but our algorithm is derived from solid theoretical foundations.
Inference for Sequential Data: The observation in POMDP tasks is sequential data, and we need to do inference for the hidden state from observations. Coquelin et al. (2009) used a particle filter to estimate the belief state given past observations. However, their method needs to access to the
environment model. Chung et al. (2015) derived the evidence lower bound of latent variables for sequential data and designed a novel deep neural network to infer recurrent latent variables. Our inference method has some superficial resemblances to their method, but we derived the evidence lower bound from a different graphical model and tackle the difficulty of how to integrate the RL planning algorithm instead of just inferring the hidden variables.

heading: 3 PRELIMINARIES
text: We start by briefly reviewing the backgrounds and notations related to our method, including partially observable Markov decision processes and maximum entropy reinforcement learning which solves optimal control problems using a probabilistic framework.

heading: 3.1 PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES
text: In many control tasks, the complete state of the environment is unknown to the agent. Instead, the agent can only receive local observations, which are typically conditioned on the current state of the system. The agent needs to make decisions based on the historical information. This kind of problems can be formalized as partially observable Markov decision processes (POMDPs) (Smith & Simmons, 2004). Formally, a POMDP is represented as a tuple (S,A,O, T, Z, r), where S, A and O are state space, action space and observation space, respectively. The reward function r(s, a) is the received reward when taking action a in state s. T (s, a, s′) is the state-transition function, which defines the probability of the succeeding state s′ after taking action a in state s. Z(s, a, o) is the observation function, which defines the probability of emitted observation o after taking action a in state s.
POMDPs use the belief bt(s) to maintain the distribution of the unknown state at time t, and the agent updates its belief with a Bayesian filter when receiving a new observation as
bt(s ′) = ηZ(s′, at, ot) ∑ s∈S T (s, at, s ′)bt−1(s), (1)
where η is a normalizing constant and at is the action at time t. POMDP planning needs to find a policy π to maximize the accumulative reward as
Vπ(b0) = E( T∑ t=0 γtr(st, at)|b0, π), (2)
where st is the state at time t, and γ ∈ (0, 1) is a discount factor.

heading: 3.2 MAXIMUM ENTROPY REINFORCEMENT LEARNING
text: POMDPs require solving two problems: the inference of state and optimal control. To integrate both of them under a unified framework, we represent POMDPs as a probabilistic graphical model. So the optimal control problem needs to be solved as a probabilistic inference task. Fortunately, the maximum entropy reinforcement learning (MERL) algorithm provides an effective tool to solve optimal control problem under PGM framework. The graphical model of Markov decision processes
is shown in Fig. 1(a). We borrow the notation from Levine (2018) to illustrate the algorithm. Levine (2018) introduces a binary random variable Ot to the graphical model, where Ot = 1 denotes that the action is optimal at time t, and Ot = 0 denotes that the action is not optimal. The probability distribution of O is p(Ot = 1|st, at) = exp(r(st, at)), and the variational lower bound is given by:
logp(O1:T ) ≥ E(s1:T ,a1:T )∼π(s1:T ,a1:T ) [ T∑ t=1 r(st, at)− logπ(at|st) ] , (3)
where π(a|s) is the policy function. Standard reinforcement learning only needs to maximize the cumulative reward. However, MERL will maximize an extra term, which is the policy entropy at each visited state. To get the optimal solution for MERL, two messages are introduced, i.e., βt(st, at) = p(Ot:T |st, at) and βt(st) = p(Ot:T |st), and their relations are given by:
βt(st) = ∫ A βt(st, at)p(at|st)dat,
βt(st, at) = ∫ S βt+1(st+1)p(st+1|st, at)p(Ot|st, at)dst+1, (4)
and the optimal policy is given by:
π(at|st,Ot:T ) = βt(st, at)
βt(st) . (5)
We can define the Q-value function and V-value function as below:
Q(st, at) = logβt(st, at), (6) V (st) = logβt(st), (7)
and the update functions are:
V (st) = log ∫ A p(at|st)exp(Q(st, at))dat,
Q(st, at) = r(st, at) + logEst+1∼p(st+1|st,at)[exp(V (st+1))]. (8)
To maximize the variational lower bound in Eq. (3), we can use a parameterized Q-function Qθ(st, at), where θ is the function parameter, and it can be learned via gradient descent:
θ = θ − αE[Qθ(st, at) dθ − (r(st, at) + log ∫ A p(at+1|st+1)exp(Qθ(st+1, at+1))dat+1)], (9)
where α is the learning rate. This algorithm is called soft Q-learning, because the update function for Q-value can be considered a soft-update version of the standard Bellman backup. For a better understanding of MERL, we recommend readers reference this tutorial (Levine, 2018).

heading: 4 SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS
text: We now present our algorithm in detail. We first derive the variational lower bound for POMDPs, and then illustrate how to deal with the conditional prior.

heading: 4.1 VARIATIONAL LOWER BOUND FOR POMDPS
text: Different from MDPs, the state s of POMDPs in the PGM (shown in Fig. 1(b)) is unobservable, which need to be inferred from the action a and the observation o. We need to derive different variational lower bound for POMDPs, which can be used to infer the hidden state and do planning jointly.
Unlike the observation function Z(s, a, o) in Section 2.1, we assume that the observation is emitted from the hidden state s, which means the probability distribution of observations are only conditioned on states, i.e., ot ∼ p(ot|st). This assumption can hold in many tasks. For example, the observation in a partially observed maze is only determined by the full map and the agent’s location.
We apply the structured variational inference to optimize the evidence lower bound of POMDPs. In structured variational inference, different parts of the proposal distributions can be optimized separately, which means we can fix some approximate functions and optimize other approximate functions. In POMDPs, we will use two approximate functions qπ(at|st) and qθ(st|st−1, at−1, ot). qπ(·) approximates the optimal policy and qθ(·) approximates the function of inferring hidden states, where θ is the parameter of the approximate function. When qθ(·) is fixed, the learning procedure is same as MERL, so that qπ(·) can be learned via the soft Q-learning algorithm. Conversely, when qπ(·) is fixed as the optimal policy, we can learn the inference function qθ(·) for hidden states. We can derive the evidence lower bound(ELBO) as bellow:
logp(O0:T , a0:T , o1:T ) = logEqθ(s1:T |O1:T ,a0:T ,o1:T ) [ p(s1:T ,O0:T , a0:T , o1:T ) qθ(s1:T |O0:T , a0:T , o1:T ) ] ≥ Eqθ(s1:T |O1:T ,a0:T ,o1:T )log [ p(s1:T ,O0:T , a0:T , o1:T ) qθ(s1:T |O0:T , a0:T , o1:T ) ] ,
(10)
and the ELBO can be written as:
L(O0:T , a0:T , o1:T ) = Eqθ(s1:T |O1:T ,a0:T ,o1:T ) T∑ t=1 { r(st, at) + log [p(at)p(ot|st)]
−DKL [qθ(st|st−1, at−1, ot)||p(st|st−1, at−1)] } .
(11)
To get the optimal action in POMDPs, we need to maximize the ELBO above. The first term in the ELBO is the cumulative reward ∑T t=1 r(st, at), which can be optimized via maximum entropy reinforcement learning algorithm.
We apply generative models to optimize the rest terms in the ELBO. The term p(ot|st) means that the hidden state st needs to have the ability to generate current observation ot. The Kullback-Leibler divergence term DKL [qθ(st|st−1, at−1, ot)||p(st|st−1, at−1)] indicates that we should minimize the gap between approximate function qθ(st|st−1, at−1, ot) and the prior p(st|st−1, at−1). In standard VAEs, the prior p(·) is generally fixed and can be set as a standard Gaussian. However, in this generative model, the prior of the hidden state st is conditioned on previous state st−1 and action at−1. We will show how we deal with the conditional prior in next sub-section. More details about the derivation process of the ELBO can be found in Appendix A.

heading: 4.2 VARIATIONAL AUTOENCODERS FOR THE CONDITIONAL PRIOR
text: The KL-divergence term in Eq. (11) introduces a conditional prior for the hidden state st, but the true conditional distribution p(st|st−1, at−1) is unknown, making it intractable to do inference. Hence we propose a new parameterized function ppriorθ (st|st−1, at−1) to approximate the true distribution p(st|st−1, at−1). We use a standard VAE to learn the ppriorθ (·). Hence (st−1, at−1) can be treated as the inner observed data in this VAE model and the inner hidden state ŝt is inferred from (st−1, at−1). We can write down the ELBO for the conditional prior directly by using the ELBO of standard VAEs (see in Appendix B):
L(st−1, at−1) = −DKL(ppriorθ (st|st−1, at−1)||p(st)) + Eppriorθ (st|st−1,at−1) [logp(st−1, at−1|st)] ,
(12)
where p(st) can be set as a standard Gaussian. The p prior θ (·) can be learned via a standard VAE paradigm.
The outputs of ppriorθ (·) are µprior and σprior, so that the KL-divergence term in Eq. (11) can be written as:
DKL [qθ(st|st−1, at−1, ot)||p(st|st−1, at−1)] ≈ DKL [ qθ(st|st−1, at−1, ot)||ppriorθ (st|st−1, at−1) ] = DKL [ qθ(st|st−1, at−1, ot)||N (µprior,diag(σprior 2 )) ] .
(13)
Unlike standard VAEs, the second distribution of the KL-divergence function in Eq. (13) is not a standard normal distribution. But we can still get the final loss function if we expand the KLdivergence function. We present the final formula of the loss function for Eq. (13) in Appendix C.

heading: 4.3 SEQUENTIAL VARIATIONAL SOFT Q-LEARNING NETWORKS
text: We design a deep recurrent neural network to improve the ELBO derived in Section 4.1 and Section 4.2. Fig. 2 shows the overall structure of our algorithm.
In our implementation, there are two generative models, i.e., one of the generate model learns the conditional prior qpriorθ (·) and reconstructs the inner observed data (st−1, at−1), and another generate model learns the qθ(·) in Eq. (13) and reconstructs the current observation ot. For the first generative model, there are two losses, i.e., LinnerKL and L inner MSE . L inner KL is the negated KL-divergence between ppriorθ (st|st−1, at−1) and a standard Gaussian:
LinnerKL = −DKL[p prior θ (st|st−1, at−1)||N (0, I)]. (14)
And LinnerMSE is the reconstruction loss of (st−1, at−1): LinnerMSE =MSE ( (st−1, at−1), ϕ inner θ (ŝt) ) , (15)
where MSE(·) is the mean-square error function, the inner hidden state ŝt is sampled from ppriorθ (st|st−1, at−1), and ϕinnerθ (·) is a reconstruction function.
For the second generative model, there are also two losses, i.e., LelboKL and L elbo MSE . L elbo KL is defined as:
LelboKL = −DKL [ qθ(st|st−1, at−1, ot)||ppriorθ (st|st−1, at−1) ] . (16)
And LelboMSE is the reconstruction loss of ot: LelboMSE =MSE ( ot, ϕ elbo θ (st) ) , (17)
where MSE(·) is the mean-square error function, the hidden state st is sampled from qθ(st|st−1, at−1, ot), and ϕelboθ (·) is a reconstruction function.
Finally, we get two kinds of loss functions for the two generative models, i.e., the reconstruction loss LMES = LinnerMSE + L elbo MSE and the KL-divergence loss LKL = L inner KL + L elbo KL .
And for the planning algorithm, we use the soft Q-learning algorithm (Levine, 2018). Its loss function is the temporal difference error LTD, which is updated via Eq. (9). All these losses can jointly be optimized via stochastic gradient descent algorithms.
To reduce the computation complexity, we use a deep recurrent neural network to capture the historical information. We first use the fθ(ot, at−1, rt−1) to extract a low-dim hidden feature wt from current input, and then feed this features to a recurrent unit rnnθ(·) to get the recurrent output ht. Because ht contains the information of past observations, the loss function in Eq. (16) can be rewritten as:
LelboKL = −DKL [ qθ(st|ht)||ppriorθ (st|st−1, at−1) ] . (18)
In the training, we tried both LSTM cell (Hochreiter & Schmidhuber, 1997) and GRU cell (Cho et al., 2014) as basic recurrent units. Because of the generalization ability of the recurrent neural network (Lample & Chaplot, 2017), we can just sample a fixed length H of sequential data for training instead of using the full-length data. In the experiment, we also studied how the training length H influences the final performance. We use a parallel training strategy, i.e., the program hosts multiple games in parallel and they all send batched data to a central data memory. This is quite similar to the data collection method in ELF (Tian et al., 2017). More details about the training strategy can be found in Appendix E.

heading: 5 EXPERIMENTS
text: We evaluate our algorithm on flickering Atari (Hausknecht & Stone, 2015) and ViZDoom platform (Kempka et al., 2016). Flickering Atari was previously used as the test environment in DRQN (Hausknecht & Stone, 2015), ADRQN (Zhu et al., 2018) and DVRL (Igl et al., 2018). The ViZDoom platform is a 3D FPS game for AI research. The agent in ViZDoom needs to navigate in the 3D environment to accomplish various tasks, such as gathering resources, shooting enemies and looking for the exit. Because the agent can only see in just one direction each time step, this naturally introduces partial observability to the task.

heading: 5.1 EXPERIMENT SETUP
text: We used some recent algorithms as baselines. Because the true state and transition function are unknown, only the methods which can be trained under model-free setting will be used for comparison. Baselines are listed as below: Deep Q-Networks (DQN) (Mnih et al., 2015): A method which uses standard Bellman backup and uses the temporal difference error as the objective function. Deep Soft Q-Networks (DSQN) (Levine, 2018): A method which uses soft Bellman backup and uses the temporal difference error as the objective function. Deep Recurrent Q-Networks (DRQN) (Hausknecht & Stone, 2015): A method which applies LSTM on the top of the DQN. Action-Specific Deep Recurrent Q-Networks (ADRQN) (Zhu et al., 2018): A method which extends the DRQN, i.e., it uses both the observation and action as inputs.
Deep Variational Reinforcement Learning (DVRL) (Igl et al., 2018) A method which combines sequential Monte Carlo and A2C (Dhariwal et al., 2017) to solve POMDPs.

heading: 5.2 EVALUATION ON FLICKERING ATARI
text: Atari environments (Bellemare et al., 2013) are widely used as the benchmark for deep reinforcement learning algorithms due to its high dimensional observation spaces and numerous challenging tasks. Flickering Atari was introduced by (Hausknecht & Stone, 2015). In each running step, the observation may be obscured with a certain probability, i.e., the raw screen will be either fully observable or fully obscured with black pixels. The settings of experiments keep in line with DVRL (Igl et al., 2018), i.e., only one frame is used, the frameskip is set to four and each frame is obscured with a probability of 0.5.
We choose four Atrai games (i.e., Pong, ChopperCommand, DoubleDunk and Asteroids) for evaluation. Fig. 3 shows the screen shots of these games. When only given one frame as the input, these tasks are POMDPs because we cannot obtain the velocity of the moving object from a single observation. These tasks have high dimensional and continuous observation spaces, while discrete action spaces. The basic network architecture and hyper-parameters are similar to DQN (Mnih et al., 2015). For the recurrent neural networks, we use a sequence length of 5 for training. All the algorithms train for 10000,000 steps and run for 100 episodes during evaluation. The training details can be found in Appendix E.
Table 1 shows the performance results of different algorithms on these Atari games. We can see that our algorithms significantly outperform other baselines on three of the games and get close score to DVRL on ChopperCommand. Compared with DRQN and ADRQN, our method introduces inductive bias to the network, which helps state estimate and RL planning for POMDPs. Compared with DVRL, our method can achieve competitive performance with lower sampling complexity.

heading: 5.3 EVALUATION ON VIZDOOM TASKS
text: We designed three tasks in ViZDoom as the evaluation tasks for our algorithm.
Health Gather: The agent needs to gather health supplies in a flat map. The agent will lose health every time step. Accordingly, if it can’t gather enough health supplies, it will die and the game is over. At each time step, the agent will receive a reward of 0.001, which encourages the agent to live a longer life. When it collects one health supply, it will receive a reward of 1. The observation for the agent is the grayscale image of its vision and its health value. The agent has three actions to choose, i.e., TURN LEFT, TURN RIGHT and MOVE FORWARD.
Health Gather v2: This is a more difficult task than previous task with a more complex map. There is some poison in the map. When the agent encounters poison, it will lose health. We use the same reward scenario, observations and actions in Health Gather.
Defend Center: In this task, the agent stands in the center of a flat map. Monsters will walk toward the agent from the edge of the map. When the monsters touch the agent, the agent will lose health. The agent holds a gun with limited ammo, and it can kill the monsters by pressing down the shooting button. The agent gets reward of 4 by killing monsters, reward of -0.2 by using ammo and reward of
-1 when losing health. The observation for the agent is the grayscale image of its vision, its health value and the ammo remained. The agent has three actions to choose, i.e., TURN LEFT, TURN RIGHT and ATTACK.
Fig. 3 shows the screen shots of these three tasks. All the algorithms are trained with the same basic network architecture and use the same hyperparameters. All the models take only one observation as input at each time step and the vision inputs are resized to the resolution of 84×84. For the recurrent neural networks, we use a sequence length of 5 for training. All the algorithms train for 300,000 steps and run for 20 episodes during evaluation. The discount factor γ is set to 0.95, learning rate is 0.0001 and the Adam optimizer (Kingma & Ba, 2014) is used for training. The network architecture for these tasks and training details are shown in Appendix D.
Fig. 4(a) shows the training curves of different methods on the Health Gather. Both SVQN models outperform other baselines. Table 2 reports the final scores of different models on these three tasks. The SVQN models achieve the best performance on these three tasks. Experiments on ViZDoom indicate that generative models of SVQNs can improve agent’s exploration ability in complicated unknown environment

heading: 5.4 ABLATION STUDY
text: Training sequence length: We study how the training sequence length H impacts the algorithm’s performance. We use the SVQN(LSTM) model and the environment of Health Gather for this study. Fig. 4(b) shows training processes of models with different training sequence lengths(H = 2, 5, 10, 15). We can see that, when the sequence length is too short(H = 2), the model can’t gather enough historical information, which causes performance degradation. When the sequence length is too long(H = 15), it is arduous for the network to do optimization, which will also
lead to performance degradation. This experiment also shows that with a limited training length, the algorithm can generalize to the time of arbitrary lengths during testing. Thanks to this generalization ability, we can train the agent with a fixed length of data instead of the full-length data, which can reduce the computation complexity.
Observation Probability: We also study whether different models are robust to the noise of the observation. We add a modification to the game Health Gather, i.e., at each time step, the observation of the screen is either fully revealed or fully obscured with a fixed observation probability p. Fig. 4(c) show the evaluation results of different models under different observation probabilities(p = 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). All the models are trained under the standard environment with full observations. The results show that SVQN models are relatively robust to the disturbance of the observation compared to other algorithms.

heading: 6 CONCLUSIONS
text: We propose a novel algorithm named Sequential Variational Soft Q-Learning Networks (SVQN) to solve POMDPs with the discrete action space. SVQN is model-free and does not need to know the true state’s representation. We apply generative models to deal with the conditional prior of hidden states and use a recurrent neural network to reduce the computational complexity, i.e., with a small length of training data, it can generalize to the test data with an arbitrary length. Our designed deep neural network can be trained end-to-end, which optimizes the planning and inference of hidden states jointly. Experimental results show that SVQN outperforms previous methods on challenging tasks and has the robustness to the disturbance of the observation. SVQN is also flexible and can be integrated with other maximum entropy reinforcement learning algorithms, such as soft actorcritic (Haarnoja et al., 2018). In the future, we will try to develop algorithms for POMDPs problems with the continuous action space.

heading: ACKNOWLEDGMENTS
text: This work was supported by the National Key Research and Development Program of China (No. 2017YFA0700904), NSFC Projects (Nos. 61620106010, U19B2034, U1811461), Beijing NSF Project (No. L172037), Beijing Academy of Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, a grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for Intelligent Computing, the JP Morgan Faculty Research Program and the NVIDIA NVAIL Program with GPU/DGX Acceleration.

heading: A DERIVATION OF THE VARIATIONAL LOWER BOUND
text: logp(O0:T , a0:T , o1:T ) = logEq(s1:T |O1:T ,a0:T ,o1:T ) [ p(s1:T ,O0:T , a0:T , o1:T ) q(s1:T |O0:T , a0:T , o1:T ) ] ≥ Eq(s1:T |O1:T ,a0:T ,o1:T )log [ p(s1:T ,O0:T , a0:T , o1:T ) q(s1:T |O0:T , a0:T , o1:T )
] = ∫ q(s1:T |O1:T , a0:T , o1:T )log [ p(s1:T ,O0:T , a0:T , o1:T ) q(s1:T |O0:T , a0:T , o1:T ) ] ds1:T
= ∫ T∑ t=1 q(s1:T |O1:T , a0:T , o1:T )log [ p(at)p(Ot|st, at)p(st|st−1, at−1)p(ot|st) q(st|st−1, at−1, ot) ] ds1:T
= T∑ t=1 ∫ q(s1:t|O1:t, a0:t, o1:t)log [ p(at)p(Ot|st, at)p(st|st−1, at−1)p(ot|st) q(st|st−1, at−1, ot) ] ds1:t
= T∑ t=1 {∫ q(s1:t|O1:t, a0:t, o1:t)log [p(at)p(Ot|st, at)p(ot|st)] ds1:t
+ ∫ q(s1:t|O1:t, a0:t, o1:t)log [ p(st|st−1, at−1) q(st|st−1, at−1, ot) ] ds1:t } =
T∑ t=1 {∫ q(s1:t|O1:t, a0:t, o1:t)log [p(at)p(Ot|st, at)p(ot|st)] ds1:t
− ∫ q(s1:t−1|O1:t−1, a0:t−1, o1:t−1)DKL [q(st|st−1, at−1, ot)||p(st|st−1, at−1)] ds1:t } = Eq(s1:T |O1:T ,a0:T ,o1:T )
T∑ t=1 { log [p(at)p(Ot|st, at)p(ot|st)]
−DKL [q(st|st−1, at−1, ot)||p(st|st−1, at−1)] }
' T∑ t=1 { r(st, at) + log [p(at)p(ot|st)]
−DKL [q(st|st−1, at−1, ot)||p(st|st−1, at−1)] } ,where s1:T ∼ q(s1:T |O1:T , a0:T , o1:T )
(19)
B VARIATIONAL AUTOENCODERS
Variational auto-encoders (VAEs) (Kingma & Welling, 2013) are effective generative models that can recover complex distributions over the data space. In the VAE, there are observed data x and the underlying causal factor z. Usually, it’s intractable to do inference for the posterior p(z|x). The VAE uses a variational function q(z|x) to approximate the true posterior. The lower bound of the marginal distribution of the observed data is given by:
logp(x) = logEq(z|x) [ p(x, z)
q(z|x)
] ≥ Eq(z|x) [ log p(x, z)
q(z|x)
] , (20)
and the lower bound can be equivalently written as :
L(x) = −DKL(q(z|x)||p(z)) + Eq(z|x) [logp(x|z)] , (21)
where DKL(·||·) is Kullback-Leibler divergence between two distributions. The approximate posterior q(z|x) is often set as Gaussian N (µ,diag(σ2)), where µ and σ are the outputs of a non-linear function of x. The generative model p(x|z) and inference model q(z|x) can be trained jointly via standard backpropagation techniques.

heading: C OPTIMIZE THE KLS IN EQ. (15)
text: We give out the final form of the loss function as below:
DKL(p||q) = ∫ p(x) p(x)
q(x) dx
= ∫ p(x) log p(x)dx− ∫ p(x)q(x)dx
= log σq σp
+ σ2p + (µp − µq)2
2σ2q − 1 2
= 1
2
[ 2 log
σq σp
+ σ2p + (µp − µq)2
σ2q − 1
]
= 1
2
[ −2 log σp
σq + σ2p + (µp − µq)2 σ2q − 1
]
= −1 2
[ 2(log(σp)− log(σq)) + 1− σ2p + (µp − µq)2
σ2q
]
(22)
, where p and q are two Gaussian distributions. σp, σq, µq and µp are their standard deviations and means respectively.

heading: D NETWORK ARCHITECTURE
text: D.1 ATARI
The network architecture for Atari is shown as below:
D.2 VIZDOOM
The network architecture for ViZDoom is shown as below:

heading: E TRAINING DETAILS
text: Training architechture:
The data produce module, training module and evaluation module are run in parallel. The framework is implemented by Python and Tensorflow.
E.1 ATARI
All the models take only one observation as input at each timestep and the vision inputs are resized to the resolution of 84 × 84. For the recurrent neural networks, we use sequence length of 5 for training. All the algorithms train for 1,0000,000 steps and run for 100 episodes during evaluation.
E.2 VIZDOOM
All the algorithms are trained with the same basic network architectures and use the same hyperparameters. All the models take only one observation as input at each timestep and the vision inputs are resized to the resolution of 84 × 84. For the recurrent neural networks, we use sequence length of 5 for training. All the algorithms train for 300,000 steps and run for 20 episodes during evaluation. The discount factor γ is set to 0.95, learning rate is 0.0001 and the Adam optimizer is used for training.

Review summaries:
This paper proposes a new sequential model-free Q-learning methodology for POMDPs that relies on variational autoencoders to represent the hidden state. The approach is generic, well-motivated and has  clear applicability in the presence of partial observability. The idea is to create a joint model for optimizing the hidden-state inference and planning jointly. For that reason variational inference is used to optimize the ELBO objective in this particular setting. All this is combined with a recurrent architecture that makes the whole process feasible and efficient.