Variational Bayesian Reinforcement Learning with Regret Bounds

√ SAT ) , where L is the time horizon , S is the number of states , A is the number of actions , and T is the total number of elapsed timesteps . We show deep connections of this approach to the soft-max and maximum-entropy strands of research in reinforcement learning . 1 Introduction and related work . In reinforcement learning ( RL ) an agent interacts with an environment in an episodic manner and attempts to maximize its return [ 54 , 45 ] . In this work the environment is a Markov decision process ( MDP ) and we consider the Bayesian case where the agent has some prior information and as it gathers data it updates its posterior beliefs about the environment . In this setting the agent is faced with the choice of visiting well understood states or exploring the environment to determine the value of other states which might lead to a higher return . This trade-off is called the explorationexploitation dilemma . One way to measure how well an agent balances this trade-off is a quantity called regret , which measures how sub-optimal the rewards the agent has received are so far , relative to the ( unknown ) optimal policy [ 12 ] . In the Bayesian case the natural quantity to consider is the Bayes regret , which is the expected regret under the agents prior information [ 17 ] . The optimal Bayesian policy can be formulated using belief states , but this is believed to be intractable for all but small problems [ 17 ] . Approximations to the optimal Bayesian policy exist , one of the most successful being Thompson sampling [ 53 , 55 ] wherein the agent samples from the posterior over value functions and acts greedily with respect to that sample [ 38 , 42 , 27 , 41 ] . It can be shown that this strategy yields both Bayesian and frequentist regret bounds under certain assumptions [ 3 ] . In practice , maintaining a posterior over value functions is intractable , and so instead the agent maintains the posterior over MDPs , and at each episode an MDP is sampled from this posterior , the value function for that sample is computed , and the policy acts greedily with respect to that value function . Due to the repeated sampling and computing of value functions this is practical only for small problems , though attempts have been made to extend it [ 37 , 43 ] . Bayesian algorithms have the advantage of being able to incorporate prior information about the problem and , as we show in the numerical experiments , they tend to perform better than nonBayesian approaches in practice [ 17 , 51 , 49 ] . Although typically Bayes regret bounds hold for any prior that satisfies the assumptions , the requirement that the prior over the MDP is known in advance 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is a disadvantage for Bayesian methods . One common concern is about performance degradation when the prior is misspecified . In this case it can be shown that the regret increases by a multiplicative factor related to the Radon-Nikodym derivative of the true prior with respect to the assumed prior [ 48 , §3.1 ] . In other words , a Bayesian algorithm with sub-linear Bayes regret operating under a misspecified prior will still have sub-linear regret so long as the true prior is absolutely continuous with respect to the misspecified prior . Moreover , for any algorithm that satisfies a Bayes regret bound it is straightforward to derive a high-probability regret bound for any family of MDPs that has support under the prior , in a sense translating Bayes regret into frequentist regret ; see [ 48 , §3.1 ] , [ 38 , Appendix A ] for details . In this work we endow an agent with a particular epistemic risk-seeking utility function , where ‘ epistemic risk ’ refers to the Bayesian uncertainty that the agent has about the optimal value function of the MDP . In the context of RL , acting so as to maximize a risk-seeking utility function which assigns higher values to more uncertain actions is a form of optimism in the face of uncertainty , a well-known heuristic to encourage exploration [ 21 , 5 ] . Any increasing convex function could be used as a risk-seeking utility , however , only the exponential utility function has a decomposition property which is required to derive a Bellman recursion [ 1 , 44 , 20 , 46 ] . We call the fixed point of this Bellman operator the ‘ K-values ’ for knowledge since they compress the expected downstream reward and the downstream epistemic uncertainty at any state-action into a single quantity . A high K-value captures the fact that the state-action has a high expected Q-value or high uncertainty , or both . Following a Boltzmann policy over the K-values yields a practical algorithm that we call ‘ K-learning ’ which attains a Bayes regret upper bounded by Õ ( L3/2 √ SAT ) where L is the time horizon , S is the number of states , A is the number of actions per state , and T is the total number of elapsed timesteps [ 11 ] . This regret bound matches the best known bound for Thompson sampling up to log factors [ 38 ] and is within a factor of √ L of the known information theoretic lower bound of Ω ( L √ SAT ) [ 23 , Appendix D ] . The update rule we derive is similar to that used in ‘ soft ’ Q-learning ( so-called since the ‘ hard ’ max is replaced with a soft-max ) [ 6 , 16 , 18 , 30 , 47 ] . These approaches are very closely related to maximum entropy reinforcement learning techniques which add an entropy regularization ‘ bonus ’ to prevent early convergence to deterministic policies and thereby heuristically encourage exploration [ 57 , 59 , 28 , 35 , 2 , 25 ] . In our work the soft-max operator and entropy regularization arise naturally from the view of the agent as maximizing a risk-seeking exponential utility . Furthermore , in contrast to these other approaches , the entropy regularization is not a fixed hyper-parameter but something we explicitly control ( or optimize for ) in order to carefully trade-off exploration and exploitation . The algorithm we derive in this work is model-based , i.e. , requires estimating the full transition function for each state . There is a parallel strand of work deriving regret and complexity bounds for model-free algorithms , primarily based on extensions of Q-learning [ 23 , 58 , 26 ] . We do not make a detailed comparison between the two approaches here other than to highlight the advantage that model-free algorithms have both in terms of storage and in computational requirements . On the other hand , in the numerical experiments model-based approaches tend to outperform the modelfree algorithms . We conjecture that an online , model-free version of K-learning with similar regret guarantees can be derived using tools developed by the model-free community . We leave exploring this to future work . 1.1 Summary of main results . • We consider an agent endowed an epistemic risk-seeking utility function and derive a new optimistic Bellman operator that incorporates the ‘ value ’ from epistemic uncertainty about the MDP . The new operator replaces the usual max operator with a soft-max and it incorporates a ‘ bonus ’ that depends on state-action visitation . In the limit of zero uncertainty the new operator reduces to the standard optimal Bellman operator . • At each episode we solve the optimistic Bellman equation for the ‘ K-values ’ which represent the utility of a particular state and action . If the agent follows a Boltzmann policy over the K-values with a carefully chosen temperature schedule then it will enjoy a sub-linear Bayes regret bound . • To the best of our knowledge this is the first work to show that soft-max operators and maximum entropy policies in RL can provably yield good performance as measured by Bayes regret . Similarly , we believe this is the first result deriving a Bayes regret bound for a Boltzmann policy in RL . This puts maximum entropy , soft-max operators , and Boltzmann exploration in a principled Bayesian context and shows that they are naturally derived from endowing the agent with an exponential utility function . 2 Markov decision processes . In a Markov decision process ( MDP ) an agent interacts with an environment in a series of episodes and attempts to maximize the cumulative rewards . A finite horizon , discrete-time , discrete MDP is given by the tuple M = { S , A , R , P , L , ρ } , where S = { 1 , . . . , S } is the state-space , A = { 1 , . . . , A } is the action-space , Rl ( s , a ) is a probability distribution over the rewards received by the agent at state s taking action a at timestep l , Pl ( s′ | s , a ) ∈ [ 0 , 1 ] is the probability the agent will transition to state s′ after taking action a in state s at timestep l , L ∈ N is the episode length , and ρ is the initial state distribution . Concretely , the initial state s ∈ S of the agent is sampled from ρ , then for timesteps l = 1 , . . . , L the agent is in state s ∈ S , selects action a ∈ A , receives reward r ∼ Rl ( s , a ) with mean µl ( s , a ) ∈ R and transitions to the next state s′ with probability Pl ( s ′ | s , a ) . After timestep L the episode terminates and the state is reset . We assume that at the beginning of learning the agent does not know the reward or transition probabilities and must learn about them by interacting with the environment . We consider the Bayesian case in which the mean reward µ and the transition probabilities P are sampled from a known prior φ . We assume that the agent knows S , A , L , and the reward noise distribution . An agent following policy πl : S × A → [ 0 , 1 ] at state s ∈ S at time l selects action a with probability πl ( s , a ) . The Bellman equation relates the value of actions taken at the current timestep to future returns through the Q-values and the associated value function [ 8 ] , which for policy π are denoted Qπl ∈ RS×A and V πl ∈ RS for l = 1 , . . . , L+ 1 , and satisfy Qπl = T πl Qπl+1 , V πl ( s ) = ∑ a∈A πl ( s , a ) Q π l ( s , a ) , ( 1 ) for l = 1 , . . . , L where QL+1 ≡ 0 and where the Bellman operator for policy π at step l is defined as ( T πl Qπl+1 ) ( s , a ) : = µl ( s , a ) + ∑ s′∈S Pl ( s ′ | s , a ) ∑ a′∈A πl ( s ′ , a′ ) Qπl+1 ( s ′ , a′ ) . The expected performance of policy π is denoted Jπ = Es∼ρV π1 ( s ) . An optimal policy satisfies π ? ∈ argmaxπ Jπ and induces associated optimal Q-values and value function given by Q ? l = T ? l Q ? l+1 , V ? l ( s ) = max a Q ? l ( s , a ) . ( 2 ) for l = 1 , . . . , L , where Q ? L+1 ≡ 0 and where the optimal Bellman operator is defined at step l as ( T ? l Q ? l+1 ) ( s , a ) : = µl ( s , a ) + ∑ s′∈S Pl ( s ′ | s , a ) max a′ Q ? l+1 ( s ′ , a′ ) . ( 3 )

Summary: The author considers the setting of the MDP is sampled from a known prior $\phi$ rather than a fixed unknown one, which belongs to Bayesian reinforcement learning. Under this setting, the author firstly introduces an expected upper bound on the optimal Q-value $Q^{\star}_l(s, a)$, then introduces a new operator called optimistic Bellman operator. This operator combines the principle of using upper confidence bound to encourage exploration and the randomness influence of the transition kernel and rewards, providing a sequence of optimistic estimation of the optimal Q-value. Furthermore, it also uses a soft-max exploration policy to sample from the environment.  Based on these components, K-learning is proposed and has been proved to enjoy a tight Bayes regret bound $O(\sqrt{L^3SAT})$. More numerical experiments are conducted to show the performance with comparisons to existing methods.