Variational oracle guiding for reinforcement learning

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has undergone rapid development in recent year ( Sutton & Barto , 2018 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ) . However , there is an common and important but under-explored aspect in RL : imagine that after playing a Poker game , a human player may look at the replay to check opponents ’ hands and analyze this information to improve his/her playing strategy ( or policy ) for the next time . We refer to the information like opponents ’ hands as oracle observation , defined as the information invisible to the agent during online task execution but available during offline training . By contrast , the information available during task execution is called executor observation . Such a scenario has been refered to as oracle guiding for RL ( Li et al. , 2020 ; Fang et al. , 2021 ) ( see Sec . 3 for a formal definition ) . Oracle guiding is common in real life . For example , when taking an examination ( the oracle observations are the answers to similar questions , which are available only during preparing ) ; and training a robot to perform some tasks on the Moon ( when training the robot , we can provide it with the information of the territory , which are not available during execution ) . The type of oracle observations can be diverse , including hindsight information ( Harutyunyan et al. , 2019 ; Guez et al. , 2020 ) , human feedback ( Knox & Stone , 2009 ; Loftin et al. , 2016 ; MacGlashan et al. , 2017 ) , re-calibrated data with post-processing , and hidden states in a partially observed setting ( Li et al. , 2020 ) . While humans naturally perform oracle guiding when learning to make decisions , it remains challenging in RL . The difficulties include : ( 1 ) how to guarantee that learning with oracle observation improves the main decision model using executor observation only , and ( 2 ) if introducing an auxiliary loss leveraging oracle observation , how to tradeoff between the main loss and auxiliary loss . While recent studies attempted to model oracle guiding in RL ( Guez et al. , 2020 ; Li et al. , 2020 ; Fang et al. , 2021 ) , none of them addressed these difficulties ( refer to the Related Work section for more details ) . In particular , all these proposed methods are heuristic : although empirical results showed performance gain with oracle guiding , it is not theoretically guaranteed that the usage of oracle observation improves execution performance . In this paper , we propose a fundamentally new idea for oracle guiding based on Bayesian theory . Taking Poker as an example , we know that the learning of optimal strategy is tractable if knowing the global , true state of the environment ( or simply state1 ) , including all visible or invisible cards , the opponents ’ playing style , etc . ( Azar et al. , 2017 ; Jin et al. , 2018 ; 2020 ) . A key part of skill improvement is learning to estimate the probabilistic distribution of environmental state from executor observation . The common way to do this by human experts is to watch match replays where the oracle observation ( e.g . opponents ’ hands ) is available , and then use the oracle-estimated state to correct the executor-estimated state . We interpret this to Bayesian language : executor-estimated state as the prior distribution , and the oracle-estimated one as the posterior distribution . Thus , the training objective can be considered two-fold : learning to make decisions based on posterior estimation of state , and learning a prior distribution of state closer to the posterior one . We formulate this idea by proposing a novel learning framework for general oracle guiding problems based on variational Bayes ( VB ) ( Kingma & Welling , 2014 ) , referred to as variational latent oracle guiding ( VLOG ) . VLOG owns several preferable properties . First , VLOG is theoretically guaranteed to leverage oracle observation for improving the decision model using executor observation . Second , VLOG is a versatile DRL framework that can be integrated into any value-based RL algorithm and is agnostic to the type of oracle observation . Third , VLOG does not suffer from the necessity of tuning additional hyper-parameters . Finally , we empirically show that VLOG contributes to better performance in a variety of decision-making tasks in both online and offline RL domains . The tasks include a simple maze navigation , video games playing , and a particularly challenging tile-based game Mahjong in which humans heavily leverage oracle observations in learning ( Li et al. , 2020 ) . We also contribute to the community by taking Mahjong as a benchmarking task for oracle guiding and publish the corresponding simulation environment and dataset to facilitate future research . 2 RELATED WORK . In the past few years , research interests have grown on DRL and imitating learning ( Chen et al. , 2020 ) with leveraging oracle or hindsight information . For DRL , Guez et al . ( 2020 ) ; Fang et al . ( 2021 ) considered hindsight observation ( executor observation at future steps ) as the oracle observation during training . Guez et al . ( 2020 ) used hindsight observation to facilitate learning a representation of current state . Another method ( Fang et al. , 2021 ) was used for stock trading : the authors trained a teacher ( oracle ) policy with hindsight information , and employed network distillation to make the student policy behaves more similar to the teacher policy . Both the methods ( Guez et al. , 2020 ; Fang et al. , 2021 ) are heuristic and focused on making use of future observation for a better sequential modeling , while VLOG is theoretically guaranteed for any kind of oracle observation . For applications in imperfect-information games , Suphx ( Li et al. , 2020 ) , a DRL-based AI for Mahjong , also introduced a method to leverage oracle observation ( opponents ’ hand ) for stronger performance . They concatenate oracle observation with executor observation as the input of policy network , where the oracle observation is timed by a scalar variable which is annealed from 1 to 0 during the training course . However , the method used in Li et al . ( 2020 ) is also heuristic and has only been tested in one task . Variational Bayes ( VB ) is a well-established method and has been taken advantage of in RL . For example , control as probabilistic inference uses VB to connect the objective function of RL and variational lower bound of a probabilistic inference problem ( Furmston & Barber , 2010 ; Weber et al. , 2015 ; Levine , 2018 ) . Our idea differs since it frames a value regression problem by the maximumlikelihood problem , and then , it applies VB to solve it ( see Sec . 4 ) . Also , the usage of VBian network models for DRL has captured attention from researchers recently . For example , Ha & Schmidhuber ( 2018 ) proposed to employ a VAE to reduce the high dimensionality of image observation ; Igl et al . ( 2018 ) ; Han et al . ( 2020 ) ; Lee et al . ( 2020 ) proposed variational RNNs as the state-transition models to encode the belief states of the agent ; Yin et al . ( 2021 ) utilized a variational sequential generative model for predicting future observation and used the prediction error to infer intrinsic reward to encourage exploration ; and Okada et al . ( 2020 ) demonstrated performance gain by using a deep Bayesian planning model in continuous control tasks . Our study differs from the mentioned works by focusing on oracle guiding , and VLOG does not involve learning a state transition model . 1Generally , oracle observation does not necessarily contain all the information of environmental state . 3 ORACLE GUIDING POMDP . Here we define the problem based on the Partially Observable Markov decision process ( POMDP ) ( Sondik , 1978 ) . An oracle guiding POMDP distinguishes from the original POMDP by having two types of observations : executor and oracle . The executor observation x is always available to the agent , on which the agent ’ s decision making ( execution ) relies . The oracle observation x̂ is not accessible during execution , but can be obtained afterward . x is included in x̂ since the former is always available . Thus x̂ contain no less information of the underlying environment state than x . A formal definition of oracle guiding POMDP is a tuple 〈S , A , P0 , T , X , X̂ , O , Ô , γ〉 , where S and A are the state and action spaces , respectively . P0 specifies the initial state distribution such that P0 ( s ) is the probability of a state s ∈ S being an initial state . T specifies the state transition probability such that T ( s′ , r|s , a ) is the probability of reaching to a new state s′ ∈ S with an immediate reward r ∈ R after taking an action a ∈ A at a state s ∈ S. X denotes the executor observation space and X̂ denotes the oracle observation space . O specifies the executor observation probability such that O ( x|s ) is the probability of an executor observation x ∈ X at a state s ∈ S. Similarly , Ô specifies the oracle observation probability such that Ô ( x|s ) is the probability of an oracle observation x̂ ∈ X̂ at a state s ∈ S. γ ∈ [ 0 , 1 ) is the discount factor . Value functions are the expected value of return from a particular state or state-action pair . For a policy π , its Q-function is defined by qπ ( s , a ) : = Eπ [ ∑∞ n=0 γ nrt+n|st = s , at = a ] , where Eπ indicates the expectation when the policy π is followed . The state value function vπ is defined by vπ ( s ) : = Eπ ( a|s ) [ qπ ( s , a ) ] . The agent aims to maximize EP0 ( s ) vπ ( s ) with respect to π , and the value-functions play an important role to this end ( Sutton & Barto , 2018 ) . 4 VLOG : VARATIONAL LATENT ORACLE GUIDING . Let us introduce a latent vector zt , which is a probabilistic variable representing the environmental state st. From a Bayesian perspective , we consider the prior distribution p ( zt|xt ) as the agent ’ s estimated probability density function ( PDF ) for zt based on executor observation xt . Meanwhile , the posterior distribution PDF q ( zt|x̂t ) is modeled based on oracle observation x̂t . In RL , the most basic requirement is to make a good estimation of return by a value function approximator v ( xt ) : = ∫ v ( zt ) p ( zt|xt ) dzt ( we denote it by v , but it can also be a Q function by simply replacing xt with ( xt , at ) ) based on available information , i.e . executor observation xt . The target of return , denoted by vtart , can be estimated by any value learning algorithm such as TD ( 0 ) ( Sutton & Barto , 2018 ) , Peng ’ s Q ( λ ) ( Kozuno et al. , 2021 ) , etc . ( generally , vtar can always be given by Bellman equations . However , one can also use Monte-Carlo return as vtar if available ) . In particular , we employed double Q-learning with dueling architecture ( Wang et al. , 2016 ) to compute vtar due to its effectiveness and simplicity ( Sec . 5 and B.1 ) . We want to maximize the log-likelihood objective of the estimation of return based on executor observation xt ( i.e. , for the executor model ) L : = logP ( v ( xt ) = v tar t |xt ) = log ∫ zt P ( v ( zt ) = v tar t |zt ) p ( zt|xt ) dzt = log ∫ zt q ( zt|x̂t ) q ( zt|x̂t ) P ( v ( zt ) = v tar t |zt ) p ( zt|xt ) dzt . By Jensen ’ s inequality , we have L ≥ ∫ zt q ( zt|x̂t ) log P ( v ( zt ) = vtart |zt ) p ( zt|xt ) q ( zt|x̂t ) dzt = ∫ zt [ q ( zt|x̂t ) logP ( v ( zt ) = v tar t |zt ) − q ( zt|x̂t ) log q ( zt|x̂t ) p ( zt|xt ) ] dzt = Eq ( zt|x̂t ) [ logP ( v ( zt ) = v tar t |zt ) ] ︸ ︷︷ ︸ oracle prediction error −DKL ( q ( zt|x̂t ) ‖p ( zt|xt ) ) ︸ ︷︷ ︸ regularization term : = LVLOG . ( 1 ) Thus we can maximize our initial objectiveL via maximizingLVLOG , which is also known as the variational lower bound ( Kingma & Welling , 2014 ) , but in our oracle-guiding scheme . Since p ( zt|xt ) and q ( zt|x̂t ) represents the PDF of latent vector obtained from executor observation and oracle observation , respectively , the meanings of the two terms in LVLOG appear clear now : the first term , i.e. , oracle prediction error , helps to improve value estimation from posterior latent state distribution ( zt computed with oracle observation ) ; and the second term , i.e. , regularization term , helps to shape the prior representation of zt closer to the posterior one as latent oracle guiding . We would like to highlight that the VLOG objective is the lower bound of the objective for the prior executor model v ( xt ) ( the estimation of return using executor observation xt ) with the usage of oracle observation x̂t . This lower bound guarantees that the usage of oracle observation is positively affecting the learning of executor model , which is what we finally want . Now we have derived the theoretical objective of VLOG , the next section will explain its implementation in practice . Remark 1 . One may use any shape of the approximate posterior q , depending on which different instance of VLOG is possible . Furthermore , one may directly use v ( xt ) instead of p ( zt|xt ) . These design choices allow users to incorporate any prior knowledge on oracle observation . For example , if one knows the range of a state-value at xt is a closed interval [ l , u ] , the approximate posterior q ( vt|x̂t ) can be restricted to a family of probability distributions supported on [ l , u ] .

Summary: This paper tackles reinforcement learning scenarios when the raw observation obtained from the executor is not very descriptive and there are high quality oracle observations available for the learning process to guide the policy training. The proposed method aims to derive a representative latent state via incorporating a variational inference perspective. Specifically, the oracle and raw observations are processed by two individual encoders and then the two latent spaces are fused and fed to a shared decoder. The latent distribution obtained from the oracle observations serve as the posterior distribution and that from the raw observation serves as the prior distribution. The distance two distributions is minimized via minimizing a KL divergence term.