Variational oracle guiding for reinforcement learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has undergone rapid development in recent year ( Sutton & Barto , 2018 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ) . However , there is an common and important but under-explored aspect in RL : imagine that after playing a Poker game , a human player may look at the replay to check opponents ’ hands and analyze this information to improve his/her playing strategy ( or policy ) for the next time . We refer to the information like opponents ’ hands as oracle observation , defined as the information invisible to the agent during online task execution but available during offline training . By contrast , the information available during task execution is called executor observation . Such a scenario has been refered to as oracle guiding for RL ( Li et al. , 2020 ; Fang et al. , 2021 ) ( see Sec . 3 for a formal definition ) . Oracle guiding is common in real life . For example , when taking an examination ( the oracle observations are the answers to similar questions , which are available only during preparing ) ; and training a robot to perform some tasks on the Moon ( when training the robot , we can provide it with the information of the territory , which are not available during execution ) . The type of oracle observations can be diverse , including hindsight information ( Harutyunyan et al. , 2019 ; Guez et al. , 2020 ) , human feedback ( Knox & Stone , 2009 ; Loftin et al. , 2016 ; MacGlashan et al. , 2017 ) , re-calibrated data with post-processing , and hidden states in a partially observed setting ( Li et al. , 2020 ) . While humans naturally perform oracle guiding when learning to make decisions , it remains challenging in RL . The difficulties include : ( 1 ) how to guarantee that learning with oracle observation improves the main decision model using executor observation only , and ( 2 ) if introducing an auxiliary loss leveraging oracle observation , how to tradeoff between the main loss and auxiliary loss . While recent studies attempted to model oracle guiding in RL ( Guez et al. , 2020 ; Li et al. , 2020 ; Fang et al. , 2021 ) , none of them addressed these difficulties ( refer to the Related Work section for more details ) . In particular , all these proposed methods are heuristic : although empirical results showed performance gain with oracle guiding , it is not theoretically guaranteed that the usage of oracle observation improves execution performance . In this paper , we propose a fundamentally new idea for oracle guiding based on Bayesian theory . Taking Poker as an example , we know that the learning of optimal strategy is tractable if knowing the global , true state of the environment ( or simply state1 ) , including all visible or invisible cards , the opponents ’ playing style , etc . ( Azar et al. , 2017 ; Jin et al. , 2018 ; 2020 ) . A key part of skill improvement is learning to estimate the probabilistic distribution of environmental state from executor observation . The common way to do this by human experts is to watch match replays where the oracle observation ( e.g . opponents ’ hands ) is available , and then use the oracle-estimated state to correct the executor-estimated state . We interpret this to Bayesian language : executor-estimated state as the prior distribution , and the oracle-estimated one as the posterior distribution . Thus , the training objective can be considered two-fold : learning to make decisions based on posterior estimation of state , and learning a prior distribution of state closer to the posterior one . We formulate this idea by proposing a novel learning framework for general oracle guiding problems based on variational Bayes ( VB ) ( Kingma & Welling , 2014 ) , referred to as variational latent oracle guiding ( VLOG ) . VLOG owns several preferable properties . First , VLOG is theoretically guaranteed to leverage oracle observation for improving the decision model using executor observation . Second , VLOG is a versatile DRL framework that can be integrated into any value-based RL algorithm and is agnostic to the type of oracle observation . Third , VLOG does not suffer from the necessity of tuning additional hyper-parameters . Finally , we empirically show that VLOG contributes to better performance in a variety of decision-making tasks in both online and offline RL domains . The tasks include a simple maze navigation , video games playing , and a particularly challenging tile-based game Mahjong in which humans heavily leverage oracle observations in learning ( Li et al. , 2020 ) . We also contribute to the community by taking Mahjong as a benchmarking task for oracle guiding and publish the corresponding simulation environment and dataset to facilitate future research . 2 RELATED WORK . In the past few years , research interests have grown on DRL and imitating learning ( Chen et al. , 2020 ) with leveraging oracle or hindsight information . For DRL , Guez et al . ( 2020 ) ; Fang et al . ( 2021 ) considered hindsight observation ( executor observation at future steps ) as the oracle observation during training . Guez et al . ( 2020 ) used hindsight observation to facilitate learning a representation of current state . Another method ( Fang et al. , 2021 ) was used for stock trading : the authors trained a teacher ( oracle ) policy with hindsight information , and employed network distillation to make the student policy behaves more similar to the teacher policy . Both the methods ( Guez et al. , 2020 ; Fang et al. , 2021 ) are heuristic and focused on making use of future observation for a better sequential modeling , while VLOG is theoretically guaranteed for any kind of oracle observation . For applications in imperfect-information games , Suphx ( Li et al. , 2020 ) , a DRL-based AI for Mahjong , also introduced a method to leverage oracle observation ( opponents ’ hand ) for stronger performance . They concatenate oracle observation with executor observation as the input of policy network , where the oracle observation is timed by a scalar variable which is annealed from 1 to 0 during the training course . However , the method used in Li et al . ( 2020 ) is also heuristic and has only been tested in one task . Variational Bayes ( VB ) is a well-established method and has been taken advantage of in RL . For example , control as probabilistic inference uses VB to connect the objective function of RL and variational lower bound of a probabilistic inference problem ( Furmston & Barber , 2010 ; Weber et al. , 2015 ; Levine , 2018 ) . Our idea differs since it frames a value regression problem by the maximumlikelihood problem , and then , it applies VB to solve it ( see Sec . 4 ) . Also , the usage of VBian network models for DRL has captured attention from researchers recently . For example , Ha & Schmidhuber ( 2018 ) proposed to employ a VAE to reduce the high dimensionality of image observation ; Igl et al . ( 2018 ) ; Han et al . ( 2020 ) ; Lee et al . ( 2020 ) proposed variational RNNs as the state-transition models to encode the belief states of the agent ; Yin et al . ( 2021 ) utilized a variational sequential generative model for predicting future observation and used the prediction error to infer intrinsic reward to encourage exploration ; and Okada et al . ( 2020 ) demonstrated performance gain by using a deep Bayesian planning model in continuous control tasks . Our study differs from the mentioned works by focusing on oracle guiding , and VLOG does not involve learning a state transition model . 1Generally , oracle observation does not necessarily contain all the information of environmental state . 3 ORACLE GUIDING POMDP . Here we define the problem based on the Partially Observable Markov decision process ( POMDP ) ( Sondik , 1978 ) . An oracle guiding POMDP distinguishes from the original POMDP by having two types of observations : executor and oracle . The executor observation x is always available to the agent , on which the agent ’ s decision making ( execution ) relies . The oracle observation x̂ is not accessible during execution , but can be obtained afterward . x is included in x̂ since the former is always available . Thus x̂ contain no less information of the underlying environment state than x . A formal definition of oracle guiding POMDP is a tuple 〈S , A , P0 , T , X , X̂ , O , Ô , γ〉 , where S and A are the state and action spaces , respectively . P0 specifies the initial state distribution such that P0 ( s ) is the probability of a state s ∈ S being an initial state . T specifies the state transition probability such that T ( s′ , r|s , a ) is the probability of reaching to a new state s′ ∈ S with an immediate reward r ∈ R after taking an action a ∈ A at a state s ∈ S. X denotes the executor observation space and X̂ denotes the oracle observation space . O specifies the executor observation probability such that O ( x|s ) is the probability of an executor observation x ∈ X at a state s ∈ S. Similarly , Ô specifies the oracle observation probability such that Ô ( x|s ) is the probability of an oracle observation x̂ ∈ X̂ at a state s ∈ S. γ ∈ [ 0 , 1 ) is the discount factor . Value functions are the expected value of return from a particular state or state-action pair . For a policy π , its Q-function is defined by qπ ( s , a ) : = Eπ [ ∑∞ n=0 γ nrt+n|st = s , at = a ] , where Eπ indicates the expectation when the policy π is followed . The state value function vπ is defined by vπ ( s ) : = Eπ ( a|s ) [ qπ ( s , a ) ] . The agent aims to maximize EP0 ( s ) vπ ( s ) with respect to π , and the value-functions play an important role to this end ( Sutton & Barto , 2018 ) . 4 VLOG : VARATIONAL LATENT ORACLE GUIDING . Let us introduce a latent vector zt , which is a probabilistic variable representing the environmental state st. From a Bayesian perspective , we consider the prior distribution p ( zt|xt ) as the agent ’ s estimated probability density function ( PDF ) for zt based on executor observation xt . Meanwhile , the posterior distribution PDF q ( zt|x̂t ) is modeled based on oracle observation x̂t . In RL , the most basic requirement is to make a good estimation of return by a value function approximator v ( xt ) : = ∫ v ( zt ) p ( zt|xt ) dzt ( we denote it by v , but it can also be a Q function by simply replacing xt with ( xt , at ) ) based on available information , i.e . executor observation xt . The target of return , denoted by vtart , can be estimated by any value learning algorithm such as TD ( 0 ) ( Sutton & Barto , 2018 ) , Peng ’ s Q ( λ ) ( Kozuno et al. , 2021 ) , etc . ( generally , vtar can always be given by Bellman equations . However , one can also use Monte-Carlo return as vtar if available ) . In particular , we employed double Q-learning with dueling architecture ( Wang et al. , 2016 ) to compute vtar due to its effectiveness and simplicity ( Sec . 5 and B.1 ) . We want to maximize the log-likelihood objective of the estimation of return based on executor observation xt ( i.e. , for the executor model ) L : = logP ( v ( xt ) = v tar t |xt ) = log ∫ zt P ( v ( zt ) = v tar t |zt ) p ( zt|xt ) dzt = log ∫ zt q ( zt|x̂t ) q ( zt|x̂t ) P ( v ( zt ) = v tar t |zt ) p ( zt|xt ) dzt . By Jensen ’ s inequality , we have L ≥ ∫ zt q ( zt|x̂t ) log P ( v ( zt ) = vtart |zt ) p ( zt|xt ) q ( zt|x̂t ) dzt = ∫ zt [ q ( zt|x̂t ) logP ( v ( zt ) = v tar t |zt ) − q ( zt|x̂t ) log q ( zt|x̂t ) p ( zt|xt ) ] dzt = Eq ( zt|x̂t ) [ logP ( v ( zt ) = v tar t |zt ) ] ︸ ︷︷ ︸ oracle prediction error −DKL ( q ( zt|x̂t ) ‖p ( zt|xt ) ) ︸ ︷︷ ︸ regularization term : = LVLOG . ( 1 ) Thus we can maximize our initial objectiveL via maximizingLVLOG , which is also known as the variational lower bound ( Kingma & Welling , 2014 ) , but in our oracle-guiding scheme . Since p ( zt|xt ) and q ( zt|x̂t ) represents the PDF of latent vector obtained from executor observation and oracle observation , respectively , the meanings of the two terms in LVLOG appear clear now : the first term , i.e. , oracle prediction error , helps to improve value estimation from posterior latent state distribution ( zt computed with oracle observation ) ; and the second term , i.e. , regularization term , helps to shape the prior representation of zt closer to the posterior one as latent oracle guiding . We would like to highlight that the VLOG objective is the lower bound of the objective for the prior executor model v ( xt ) ( the estimation of return using executor observation xt ) with the usage of oracle observation x̂t . This lower bound guarantees that the usage of oracle observation is positively affecting the learning of executor model , which is what we finally want . Now we have derived the theoretical objective of VLOG , the next section will explain its implementation in practice . Remark 1 . One may use any shape of the approximate posterior q , depending on which different instance of VLOG is possible . Furthermore , one may directly use v ( xt ) instead of p ( zt|xt ) . These design choices allow users to incorporate any prior knowledge on oracle observation . For example , if one knows the range of a state-value at xt is a closed interval [ l , u ] , the approximate posterior q ( vt|x̂t ) can be restricted to a family of probability distributions supported on [ l , u ] .

Summary: In this paper, reinforcement learning (RL) with training-time privileged information is studied. This scenario is the relaxation of traditional POMDP by allowing additional observations to be available during training time. A variational Bayes approach is proposed under this scenario. The main idea lies in learning a joint latent space for both testing-stage partial observations and the training-stage full observations.  Experiments over several benchmark tasks, including Mahjong, a large-scale multi-player game with imperfect information are conducted to verify the performance of the approach.