SVQN: Sequential Variational Soft Q-Learning Networks

1 INTRODUCTION . In recent years , substantial progress has been made in deep reinforcement learning for solving various challenging tasks , including the computer Go game ( Silver et al. , 2016 ) , Atari games ( Mnih et al. , 2015 ) , StarCraft ( Zambaldi et al. , 2018 ; Pang et al. , 2018 ) and the first-person shooting ( FPS ) games ( Lample & Chaplot , 2017 ; Wu & Tian , 2016 ; Huang et al. , 2019 ) . However , in many real-world applications , decision-making problems are partially observable ( Aström , 1965 ) , preventing such problems from being solved by standard reinforcement learning algorithms . Formally , these kinds of problems are often defined as Partially Observable Markov Decision Processes ( POMDPs ) ( Kaelbling et al. , 1998 ) , which demand information from past observations to help in the decision-making process ( McCallum , 1993 ) . Although numerous efforts ( Hausknecht & Stone , 2015 ; Foerster et al. , 2016 ; Igl et al. , 2018 ; Zhu et al. , 2018 ) have been paid to tackle this problem , there still exist various challenges . For example , Egorov ( 2015 ) tries to solve POMDPs by using the belief of the agent as the input of DQN ( Mnih et al. , 2015 ) , but this algorithm needs access to the environment model . However , in many reinforcement learning tasks , it is not possible for the agent to acquire the underlying transition function , making such algorithms inapplicable . Some recent work ( Karkus et al. , 2017 ; McAllester & Singh , 2013 ; Babayan et al. , 2018 ) tries to solve POMDPs under the model-free setting , i.e. , the agent does not need to know and learn the transition function of the environment . For instance , Karkus et al . ( 2017 ) trained an agent to navigate in a partially observable grid world under the model-free setting , i.e. , the agent can only observe a part of the grid world and does not learn the transition function . The agent uses its local observations to update its beliefs ( McAllester & Singh , 2013 ; Babayan et al. , 2018 ) . In their experiments , the ground truth of the state is the full map plus the location of the agent , which means the representation of the state is explicit . However , in some complex tasks , it is impossible to acquire or design the state or beliefs . To solve the unknown representation problem of the state , Hausknecht & Stone ( 2015 ) and Zhu et al . ( 2018 ) try to represent the state as latent variables of neural networks . However , they only use ∗J . Zhu and T. Chen are corresponding authors . J. Zhu is also with RealAI . a deep recurrent neural network to capture the historical information and fail to utilize the Markov property of the state in POMDPs . Igl et al . ( 2018 ) apply sequential Monte Carlo ( SMC ) ( Le et al. , 2017 ) to introduce inductive bias to the neural network , which can embody the Markov properties of the state . They can infer the state from the past observations online . However , they separate the planning algorithm from the inference of the state . To infer the hidden states and optimize the planning module jointly , we represent POMDPs as a unified probabilistic graphical model ( PGM ) and derive a single evidence lower bound ( ELBO ) . We apply structured variational inference to optimize the ELBO . In our implementation , we design generative models to infer the hidden variables , however , the distribution of the latent variables is conditioned on previous hidden states . This is different from standard VAEs ( Kingma & Welling , 2013 ) , whose prior of the latent variables can be a standard Gaussian distribution . So , we apply an additional approximate function to tackle the conditional prior problem . The planning can also be solved under the PGM framework . Fortunately , maximum entropy reinforcement learning ( MERL ) ( Levine , 2018 ) provide a tool to formalize the planning as a probabilistic inference task . In this paper , we propose a novel end-to-end neural network called the sequential variational soft Q-learning network ( SVQN ) , which integrates the learning of hidden states and the optimization of the planning within the same framework . A deep recurrent neural network ( RNN ) ( Cho et al. , 2014 ; Hochreiter & Schmidhuber , 1997 ) in SVQNs is used to reduce the computational complexity , because the feature extraction can share the same weights in the RNN . Experimental results show that the SVQN can utilize past information to help in decision making for efficient inference , and outperforms other baselines on several challenging tasks . Our ablation study shows that SVQNs have the generalization ability over time and are robust to the disturbance of the observation . Contributions : ( 1 ) We derive the variational lower bound for POMDPs , which allows us to integrate the optimization of the control problem and learning of the hidden state under a unified graphical model . ( 2 ) We propose to tackle the difficulty of the inference of the hidden state and solve the problem of a conditional prior using the generative models . ( 3 ) We design an end-to-end deep recurrent neural network , which can reduce the computational complexity and be trained efficiently . 2 RELATED WORK . We summarize some related work in POMDPs and inference methods for sequential data . Model-Based and Model-Free methods for POMDPs : When the environment model is accessible , POMDPs can be solved by model-based method . Egorov ( 2015 ) used model-based methods to solve POMDPs , but their agents need to know the belief-update function and the transition function . When the environment model is unknown , model-free methods should be applied . Recently , some researchers ( Hausknecht & Stone , 2015 ; Zhu et al. , 2018 ) used recurrent neural networks to capture the historical information , but they failed to utilize the Markov property of the state in POMDPs . Our work proposes generative models for the algorithm learning , which tackles the difficulty of the inference of hidden states and introduces inductive bias to the network structure . Igl et al . ( 2018 ) applied sequential Monte Carlo ( SMC ) to the POMDPs . They can infer the hidden state from the past observations online . However , they separate the planning algorithm and the inference of the hidden state . Our algorithm is derived from a unified graphical model , which can train the inference model and the planning algorithm jointly . Explorations in POMDPs : In contrast to MDP methods , POMDP methods should do exploration to gather information with no immediate reward which can then be used in the future to gain higher rewards . Pathak et al . ( 2017 ) and Choi et al . ( 2018 ) designed deep reinforcement learning algorithms to help the exploration in their tasks . Actually , their tasks are POMDP tasks . However , they just treat their tasks as MDP tasks and intuitively add exploration tricks to standard reinforcement learning algorithms . Their exploration tricks include the reconstruction of observations and actions , which comes from intuitive concepts such as curiosity and attention . The generative models in our method also need to reconstruct the observations and actions , but our algorithm is derived from solid theoretical foundations . Inference for Sequential Data : The observation in POMDP tasks is sequential data , and we need to do inference for the hidden state from observations . Coquelin et al . ( 2009 ) used a particle filter to estimate the belief state given past observations . However , their method needs to access to the environment model . Chung et al . ( 2015 ) derived the evidence lower bound of latent variables for sequential data and designed a novel deep neural network to infer recurrent latent variables . Our inference method has some superficial resemblances to their method , but we derived the evidence lower bound from a different graphical model and tackle the difficulty of how to integrate the RL planning algorithm instead of just inferring the hidden variables . 3 PRELIMINARIES . We start by briefly reviewing the backgrounds and notations related to our method , including partially observable Markov decision processes and maximum entropy reinforcement learning which solves optimal control problems using a probabilistic framework . 3.1 PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES . In many control tasks , the complete state of the environment is unknown to the agent . Instead , the agent can only receive local observations , which are typically conditioned on the current state of the system . The agent needs to make decisions based on the historical information . This kind of problems can be formalized as partially observable Markov decision processes ( POMDPs ) ( Smith & Simmons , 2004 ) . Formally , a POMDP is represented as a tuple ( S , A , O , T , Z , r ) , where S , A and O are state space , action space and observation space , respectively . The reward function r ( s , a ) is the received reward when taking action a in state s. T ( s , a , s′ ) is the state-transition function , which defines the probability of the succeeding state s′ after taking action a in state s. Z ( s , a , o ) is the observation function , which defines the probability of emitted observation o after taking action a in state s. POMDPs use the belief bt ( s ) to maintain the distribution of the unknown state at time t , and the agent updates its belief with a Bayesian filter when receiving a new observation as bt ( s ′ ) = ηZ ( s′ , at , ot ) ∑ s∈S T ( s , at , s ′ ) bt−1 ( s ) , ( 1 ) where η is a normalizing constant and at is the action at time t. POMDP planning needs to find a policy π to maximize the accumulative reward as Vπ ( b0 ) = E ( T∑ t=0 γtr ( st , at ) |b0 , π ) , ( 2 ) where st is the state at time t , and γ ∈ ( 0 , 1 ) is a discount factor . 3.2 MAXIMUM ENTROPY REINFORCEMENT LEARNING . POMDPs require solving two problems : the inference of state and optimal control . To integrate both of them under a unified framework , we represent POMDPs as a probabilistic graphical model . So the optimal control problem needs to be solved as a probabilistic inference task . Fortunately , the maximum entropy reinforcement learning ( MERL ) algorithm provides an effective tool to solve optimal control problem under PGM framework . The graphical model of Markov decision processes is shown in Fig . 1 ( a ) . We borrow the notation from Levine ( 2018 ) to illustrate the algorithm . Levine ( 2018 ) introduces a binary random variable Ot to the graphical model , where Ot = 1 denotes that the action is optimal at time t , and Ot = 0 denotes that the action is not optimal . The probability distribution of O is p ( Ot = 1|st , at ) = exp ( r ( st , at ) ) , and the variational lower bound is given by : logp ( O1 : T ) ≥ E ( s1 : T , a1 : T ) ∼π ( s1 : T , a1 : T ) [ T∑ t=1 r ( st , at ) − logπ ( at|st ) ] , ( 3 ) where π ( a|s ) is the policy function . Standard reinforcement learning only needs to maximize the cumulative reward . However , MERL will maximize an extra term , which is the policy entropy at each visited state . To get the optimal solution for MERL , two messages are introduced , i.e. , βt ( st , at ) = p ( Ot : T |st , at ) and βt ( st ) = p ( Ot : T |st ) , and their relations are given by : βt ( st ) = ∫ A βt ( st , at ) p ( at|st ) dat , βt ( st , at ) = ∫ S βt+1 ( st+1 ) p ( st+1|st , at ) p ( Ot|st , at ) dst+1 , ( 4 ) and the optimal policy is given by : π ( at|st , Ot : T ) = βt ( st , at ) βt ( st ) . ( 5 ) We can define the Q-value function and V-value function as below : Q ( st , at ) = logβt ( st , at ) , ( 6 ) V ( st ) = logβt ( st ) , ( 7 ) and the update functions are : V ( st ) = log ∫ A p ( at|st ) exp ( Q ( st , at ) ) dat , Q ( st , at ) = r ( st , at ) + logEst+1∼p ( st+1|st , at ) [ exp ( V ( st+1 ) ) ] . ( 8 ) To maximize the variational lower bound in Eq . ( 3 ) , we can use a parameterized Q-function Qθ ( st , at ) , where θ is the function parameter , and it can be learned via gradient descent : θ = θ − αE [ Qθ ( st , at ) dθ − ( r ( st , at ) + log ∫ A p ( at+1|st+1 ) exp ( Qθ ( st+1 , at+1 ) ) dat+1 ) ] , ( 9 ) where α is the learning rate . This algorithm is called soft Q-learning , because the update function for Q-value can be considered a soft-update version of the standard Bellman backup . For a better understanding of MERL , we recommend readers reference this tutorial ( Levine , 2018 ) .

Summary: This paper proposes a new sequential model-free Q-learning methodology for POMDPs that relies on variational autoencoders to represent the hidden state. The approach is generic, well-motivated and has  clear applicability in the presence of partial observability. The idea is to create a joint model for optimizing the hidden-state inference and planning jointly. For that reason variational inference is used to optimize the ELBO objective in this particular setting. All this is combined with a recurrent architecture that makes the whole process feasible and efficient.