A Provably Efficient Model-Free Posterior Sampling Method for Episodic Reinforcement Learning

Thompson Sampling is one of the most effective methods for contextual bandits and has been generalized to posterior sampling for certain MDP settings . However , existing posterior sampling methods for reinforcement learning are limited by being model-based or lack worst-case theoretical guarantees beyond linear MDPs . This paper proposes a new model-free formulation of posterior sampling that applies to more general episodic reinforcement learning problems with theoretical guarantees . We introduce novel proof techniques to show that under suitable conditions , the worst-case regret of our posterior sampling method matches the best known results of optimization based methods . In the linear MDP setting with dimension , the regret of our algorithm scales linearly with the dimension as compared to a quadratic dependence of the existing posterior sampling-based exploration algorithms . 1 Introduction . A key challenge in reinforcement learning problems is to balance exploitation and exploration . The goal is to make decisions that are currently expected to yield high reward and that help identify less known but potentially better alternate decisions . In the special case of contextual bandit problems , this trade-off is well understood and one of the most effective and widely used algorithms is Thompson sampling [ Thompson , 1933 ] . Thompson sampling is a Bayesian approach that maintains a posterior distribution of each arm being optimal for the given context . At each round , the algorithm samples an action from this distribution and updates the posterior with the new observation . The popularity of Thompson sampling stems from strong empirical performance [ Li et al. , 2010 ] , as well as competitive theoretical guarantees in the form of Bayesian [ Russo and Van Roy , 2014 ] and frequentist regret bounds [ Kaufmann et al. , 2012 ] . The results in the contextual bandit setting have motivated several adaptations of Thompson sampling to the more challenging Markov decision process ( MDP ) setting . Most common are model-based adaptations such as PSRL [ Strens , 2000 , Osband et al. , 2013 , Agrawal and Jia , 2017 ] or BOSS [ Asmuth et al. , 2012 ] , which maintain a posterior distributions over MDP models . These algorithms determine their current policy by sampling a model from this posterior and computing the optimal policy for it . The benefit of maintaining a posterior over models instead of the optimal policy or optimal value function directly is that posterior updates can be more easily derived and algorithms are easier to analyze . However , model-based approaches are limited to small-scale problems where a realizable model class of moderate size is available and where computing the optimal policy of a model is computationally tractable . This rules out many practical problems where the observations are rich , e.g . images or text . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are also model-free posterior sampling algorithms that are inspired by Thompson sampling . These aim to overcome the limitation of model-based algorithms by only requiring a value-function class and possibly weaker assumptions on the MDP model . Several algorithms have been proposed [ e.g . Osband et al. , 2016a , Fortunato et al. , 2017 , Osband et al. , 2018 ] with good empirical performance but with no theoretical performance guarantees . A notable exception is the randomized least-squares value iteration ( RLSVI ) algorithm by Osband et al . [ 2016b ] that admits frequentist regret bounds in tabular [ Russo , 2019 ] and linear Markiov decision processes [ Zanette et al. , 2020a ] . However , to the best of our knowledge , no such results are available beyond the linear setting . In contrast , there has been impressive recent progress in developing and analyzing provably efficient algorithms for more general problem classes based on the OFU ( optimism in the face of uncertainty ) principle . These works show that OFU-based algorithm can learn a good policy with small sample-complexity or regret as long as the value-function class and MDP satisfies general structural assumptions . Those assumptions include bounded Bellman rank [ Jiang et al. , 2017 ] , low inherent Bellman error [ Zanette et al. , 2020b ] , small Eluder dimension [ Wang et al. , 2020 ] or Bellman-Eluder dimension [ Jin et al. , 2021 ] . This raises the question of whether OFU-based algorithms are inherently more suitable for such settings or whether it is possible to achieve similar results with a model-free posterior sampling approach . In this work , we answer this question by analyzing a posterior sampling algorithm that works with a Q-function class and admits worst-case regret guarantees under general structural assumptions . Our main contributions are : • We derive a model-free posterior sampling algorithm for reinforcement learning in general Markov decision processes and value function classes . • We introduce a new proof technique for analyzing posterior sampling with optimistic priors . • We prove that this algorithm achieves near-optimal worst-case regret bounds that match the regret of OFU-based algorithms and improve the best known regret bounds for posterior sampling approaches . 1.1 Further Related Work . Several extension of TS have been proposed for non-linear function classes in contextual bandits [ Zhang et al. , 2020 , Kveton et al. , 2020 , Russo and Van Roy , 2014 ] . For tabular MDPs , Agrawal et al . [ 2020 ] improves the regret bounds for RLSVI and Xiong et al . [ 2021 ] show that an algorithm with randomized value functions can be optimal . In the same setting , Pacchiano et al . [ 2020 ] proposed an optimism based algorithm that uses internal noise , which can be interpreted as a posterior sampling method . Jafarnia-Jahromi et al . [ 2021 ] analyzes a posterior sampling algorithm for tabular stochastic shortest path problems . Considering the Bayesian regret , the seminal paper of Osband and Van Roy [ 2014 ] provides a general posterior sampling RL method that can be applied to general model classes including linear mixture MDPs . 2 Setting and Notation . Episodic Markov decision process . We consider the episodic Markov decision process ( MDP ) setting where the MDP is defined by the tuple ( X , A , H , P , r ) . Here , X and A are state and action spaces . The number H ∈ N is the length of each episode and P = { Ph } Hh=1 and R = { Rh } Hh=1 are the state transition probability measures and the random rewards . The agent interacts with the MDP in T episodes of length H . In each episode t ∈ [ T ] = { 1 , 2 , . . . , T } , the agent first observes an initial state x1t ∈ X and then , for each time step h ∈ [ H ] , the agent takes an action aht ∈ A and receives the next state xh+1t ∼ Ph ( ·|xht , aht ) and reward rht ∼ Rh ( ·|xht , aht ) . To simplify our exposition , we assume that the initial states x1t = x 1 are identical across episodes . When initial states are stochastic , we can simply add a dummy initial state and increase H by 1 to ensure this . Our approach can also be generalized to adversarial initial states which we discuss in the appendix . We denote by πt : X × [ H ] → A the agent ’ s policy in the t-th episode , that is , aht = πt ( xht , h ) . For a given policy π , the value functions are defined for all h ∈ [ H ] and state-action pairs ( x , a ) as Qπh ( x , a ) = r h ( x , a ) + Ex′∼Ph ( x , a ) [ V πh+1 ( x ′ ) ] , V πh ( x ) = Q π h ( x , π ( x , h ) ) , where rh ( x , a ) = Erh∼Rh ( x , a ) [ rh ] ∈ [ 0 , 1 ] is the average immediate reward and V πH+1 ( x ) = 0 for convenience . We further denote by T ? h the Bellman optimality operator that maps any state-action function f to [ T ? h f ] ( x , a ) = rh ( x , a ) + Ex′∼Ph ( x , a ) [ max a′∈A f ( x′ , a′ ) ] . The optimal Q-function is given by Q ? h = T ? h Q ? h+1 for all h ∈ [ H ] where again Q ? H+1 = 0 . Value function approximation . We assume the agent is provided with a Q-function class F = F1 × F2 × · · · × FH of functions f = { fh } h∈ [ H ] where fh : X × A → R. For convenience , we also consider FH+1 = { 0 } which only contains the constant zero function , and we define fh ( x ) = maxa∈A f h ( x , a ) . Before each episode t , our algorithm selects a Q-function ft ∈ F and then picks actions in this episode with the greedy policy πft w.r.t . this function . That is , aht = πft ( x h t , h ) ∈ arg maxa∈A fht ( xht , a ) . We make the following assumptions on the value-function class : Assumption 1 ( Realizability ) . The optimal Q-function is in the class Q ? h ∈ Fh for all h ∈ [ H ] . Assumption 2 ( Boundedness ) . There exists b ≥ 1 such that fh ( x , a ) ∈ [ 0 , b−1 ] for all x , a ∈ X×A and f = { fh } h∈ [ H ] ∈ F . Assumption 3 ( Completeness ) . For all h and fh+1 ∈ Fh+1 , there is a fh ∈ Fh such that fh = T ? h fh+1 . Additional notation . For any fh ∈ Fh , we define the short-hand notation fh ( x ) = maxa∈A f h ( x , a ) and for any f ∈ F , h ∈ [ H ] and state-action pair x , a , we define the Bellman residual as Eh ( f ; x , a ) = E ( fh , fh+1 ; x , a ) = fh ( x , a ) − T ? h fh+1 ( x , a ) . We measure the performance of an algorithm that produces a sequence of policies π1 , π2 , . . . , by its regret Reg ( T ) = T∑ t=1 ( V ? 1 ( x 1 ) − V πt1 ( x1 ) ) after T episodes . 3 Conditional Posterior Sampling Algorithm . We derive our posterior sampling algorithm by first defining the prior over the function class and our likelihood model of an episode given a value function . Optimistic prior . We assume that the prior p0 over F has the form p0 ( f ) ∝ exp ( λf1 ( x1 ) ) H∏ h=1 ph0 ( f h ) ( 1 ) where ph0 are distributions over each Fh and λ > 0 is a parameter . This form assumes that the prior factorizes over time steps and that the prior for the value functions of the first time step prefers large values for the initial state . This optimistic preference helps initial exploration and allows us to achieve frequentist regret bounds . A similar mechanism can be found in existing sampling based algorithms in the form of optimistic value initializations [ Osband et al. , 2018 ] or forced initial exploration by default values [ Zanette et al. , 2020a ] . Temporal difference error likelihood . Consider a set of observations acquired in t episodes St = { xhs , ahs , rhs } s∈ [ t ] , h∈ [ H ] . To formulate the likelihood of these observations , we make use of the squared temporal difference ( TD ) error , that for time h , is Lh ( f ; St ) = L h ( fh , fh+1 ; St ) = t∑ s=1 ( fh ( xhs , a h s ) − rhs − fh+1 ( xh+1s ) ) 2 . For h = H , we can choose xH+1s arbitrarily since by definition f H+1 ( x ) = 0 for all x . We now define the likelihood of St given a value function f ∈ F as p ( St|f ) ∝ H∏ h=1 exp ( −ηLh ( fh , fh+1 ; St ) ) Ef̃h∼ph0 exp ( −ηL h ( f̃h , fh+1 ; St ) ) , ( 2 ) where η > 0 is a parameter . Readers familiar with model-free reinforcement learning methods likely find the use of squared temporal difference error in the numerator natural as it makes transition samples with small TD error more likely . Most popular model free algorithm such as Q-learning [ Watkins and Dayan , 1992 ] rely on the TD error as their loss function . However , the normalization in the denominator of ( 2 ) may seem surprising . This term makes those transitions more likely that have small TD error under the specific fh , fh+1 pair compared to pairs ( f̃ , fh+1 ) with f̃ is drawn from the prior . Thus , transitions are encouraged to explain the specific choice of fh for fh+1 . This normalization is one of our key algorithmic innovations . It allows us to related a small loss Lh to a small bellman error and circumvent the double-sample issue [ Baird , 1995 , Dann et al. , 2014 ] of the square TD error . Combining the prior in ( 1 ) and likelihood in ( 2 ) , we obtain the posterior of our algorithm after t episodes p ( f |St ) ∝ exp ( λf1 ( x1 ) ) H∏ h=1 q ( fh|fh+1 , St ) , ( 3 ) where q ( fh|fh+1 , St ) = ph0 ( f h ) exp ( −ηLh ( fh , fh+1 ; St ) ) Ef̃h∼ph0 exp ( −ηL h ( f̃h , fh+1 ; St ) ) . Note that the conditional probability q ( fh|fh+1 , St ) samples fh using the data St and the TD error Lh ( fh , fh+1 ; St ) , which mimics the model update process of Q-learning , where we fit the model fh at each step h to the target computed from fh+1 . The optimistic prior encourages exploration , which is needed in our analysis . It was argued that such a term is necessary to achieve the optimal frequentist regret bound for Thompson sampling in the bandit case Zhang [ 2021 ] . For the same reason , we employ it for the analysis of posterior sampling in episodic RL . The losses Lh will grow with the number of rounds t and once the effect of the first exp ( λf1 ( x1 ) ) factor has become small enough , the posterior essentially performs Bayesian least-squares regression of fh conditioned on fh+1 . We thus call our algorithm conditional posterior sampling . This algorithm , shown in Algorithm 1 simply samples a Q-function ft from the posterior before the each episode , follows the greedy policy of ft for one episode and then updates the posterior . Algorithm 1 : Conditional Posterior Sampling Algorithm Input : value function class F , learning rate η , prior optimism parameter λ , number of rounds T 1 for t = 1 , 2 , . . . , T do 2 Draw Q-function ft ∼ p ( ·|St−1 ) according to posterior ( 3 ) 3 Play episode t using the greedy policy πft and add observations to St We are not aware of a provably computationally efficient sampling procedure for ( 3 ) . Since the primary focus of our work is statistical efficiency we leave it as an open problem for future work to investigate the empirical feasibility of approximate samplers or to identify function classes that allow for efficient sampling . Another potential improvement is to derive a conditional sampling rule that allows to sample ( fht ) H h=1 iteratively instead of jointly from a single posterior .

Summary: This work introduces a conditional posterior sampling algorithm that is a model-free posterior sampling reinforcement learning algorithm and can be applied to general Markov decision processes and value function classes. Authors provide theoretical guarantees for the proposed algorithm.  Actually, it has been proved that the worst-case regret of the proposed algorithm is near-optimal and that improves the best known regret bounds for posterior sampling approaches (OPT-RLSVI, UCB-LSVI). The key component of the proposed algorithm is the definition of the likelihood over the collected samples (observations - state, action, reward) given a value function. To conclude, it is a theoretical work that shows that there is no statistical efficiency gap between OFU and posterior sampling algorithms.