Variational Bayesian Reinforcement Learning with Regret Bounds

√ SAT ) , where L is the time horizon , S is the number of states , A is the number of actions , and T is the total number of elapsed timesteps . We show deep connections of this approach to the soft-max and maximum-entropy strands of research in reinforcement learning . 1 Introduction and related work . In reinforcement learning ( RL ) an agent interacts with an environment in an episodic manner and attempts to maximize its return [ 54 , 45 ] . In this work the environment is a Markov decision process ( MDP ) and we consider the Bayesian case where the agent has some prior information and as it gathers data it updates its posterior beliefs about the environment . In this setting the agent is faced with the choice of visiting well understood states or exploring the environment to determine the value of other states which might lead to a higher return . This trade-off is called the explorationexploitation dilemma . One way to measure how well an agent balances this trade-off is a quantity called regret , which measures how sub-optimal the rewards the agent has received are so far , relative to the ( unknown ) optimal policy [ 12 ] . In the Bayesian case the natural quantity to consider is the Bayes regret , which is the expected regret under the agents prior information [ 17 ] . The optimal Bayesian policy can be formulated using belief states , but this is believed to be intractable for all but small problems [ 17 ] . Approximations to the optimal Bayesian policy exist , one of the most successful being Thompson sampling [ 53 , 55 ] wherein the agent samples from the posterior over value functions and acts greedily with respect to that sample [ 38 , 42 , 27 , 41 ] . It can be shown that this strategy yields both Bayesian and frequentist regret bounds under certain assumptions [ 3 ] . In practice , maintaining a posterior over value functions is intractable , and so instead the agent maintains the posterior over MDPs , and at each episode an MDP is sampled from this posterior , the value function for that sample is computed , and the policy acts greedily with respect to that value function . Due to the repeated sampling and computing of value functions this is practical only for small problems , though attempts have been made to extend it [ 37 , 43 ] . Bayesian algorithms have the advantage of being able to incorporate prior information about the problem and , as we show in the numerical experiments , they tend to perform better than nonBayesian approaches in practice [ 17 , 51 , 49 ] . Although typically Bayes regret bounds hold for any prior that satisfies the assumptions , the requirement that the prior over the MDP is known in advance 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is a disadvantage for Bayesian methods . One common concern is about performance degradation when the prior is misspecified . In this case it can be shown that the regret increases by a multiplicative factor related to the Radon-Nikodym derivative of the true prior with respect to the assumed prior [ 48 , §3.1 ] . In other words , a Bayesian algorithm with sub-linear Bayes regret operating under a misspecified prior will still have sub-linear regret so long as the true prior is absolutely continuous with respect to the misspecified prior . Moreover , for any algorithm that satisfies a Bayes regret bound it is straightforward to derive a high-probability regret bound for any family of MDPs that has support under the prior , in a sense translating Bayes regret into frequentist regret ; see [ 48 , §3.1 ] , [ 38 , Appendix A ] for details . In this work we endow an agent with a particular epistemic risk-seeking utility function , where ‘ epistemic risk ’ refers to the Bayesian uncertainty that the agent has about the optimal value function of the MDP . In the context of RL , acting so as to maximize a risk-seeking utility function which assigns higher values to more uncertain actions is a form of optimism in the face of uncertainty , a well-known heuristic to encourage exploration [ 21 , 5 ] . Any increasing convex function could be used as a risk-seeking utility , however , only the exponential utility function has a decomposition property which is required to derive a Bellman recursion [ 1 , 44 , 20 , 46 ] . We call the fixed point of this Bellman operator the ‘ K-values ’ for knowledge since they compress the expected downstream reward and the downstream epistemic uncertainty at any state-action into a single quantity . A high K-value captures the fact that the state-action has a high expected Q-value or high uncertainty , or both . Following a Boltzmann policy over the K-values yields a practical algorithm that we call ‘ K-learning ’ which attains a Bayes regret upper bounded by Õ ( L3/2 √ SAT ) where L is the time horizon , S is the number of states , A is the number of actions per state , and T is the total number of elapsed timesteps [ 11 ] . This regret bound matches the best known bound for Thompson sampling up to log factors [ 38 ] and is within a factor of √ L of the known information theoretic lower bound of Ω ( L √ SAT ) [ 23 , Appendix D ] . The update rule we derive is similar to that used in ‘ soft ’ Q-learning ( so-called since the ‘ hard ’ max is replaced with a soft-max ) [ 6 , 16 , 18 , 30 , 47 ] . These approaches are very closely related to maximum entropy reinforcement learning techniques which add an entropy regularization ‘ bonus ’ to prevent early convergence to deterministic policies and thereby heuristically encourage exploration [ 57 , 59 , 28 , 35 , 2 , 25 ] . In our work the soft-max operator and entropy regularization arise naturally from the view of the agent as maximizing a risk-seeking exponential utility . Furthermore , in contrast to these other approaches , the entropy regularization is not a fixed hyper-parameter but something we explicitly control ( or optimize for ) in order to carefully trade-off exploration and exploitation . The algorithm we derive in this work is model-based , i.e. , requires estimating the full transition function for each state . There is a parallel strand of work deriving regret and complexity bounds for model-free algorithms , primarily based on extensions of Q-learning [ 23 , 58 , 26 ] . We do not make a detailed comparison between the two approaches here other than to highlight the advantage that model-free algorithms have both in terms of storage and in computational requirements . On the other hand , in the numerical experiments model-based approaches tend to outperform the modelfree algorithms . We conjecture that an online , model-free version of K-learning with similar regret guarantees can be derived using tools developed by the model-free community . We leave exploring this to future work . 1.1 Summary of main results . • We consider an agent endowed an epistemic risk-seeking utility function and derive a new optimistic Bellman operator that incorporates the ‘ value ’ from epistemic uncertainty about the MDP . The new operator replaces the usual max operator with a soft-max and it incorporates a ‘ bonus ’ that depends on state-action visitation . In the limit of zero uncertainty the new operator reduces to the standard optimal Bellman operator . • At each episode we solve the optimistic Bellman equation for the ‘ K-values ’ which represent the utility of a particular state and action . If the agent follows a Boltzmann policy over the K-values with a carefully chosen temperature schedule then it will enjoy a sub-linear Bayes regret bound . • To the best of our knowledge this is the first work to show that soft-max operators and maximum entropy policies in RL can provably yield good performance as measured by Bayes regret . Similarly , we believe this is the first result deriving a Bayes regret bound for a Boltzmann policy in RL . This puts maximum entropy , soft-max operators , and Boltzmann exploration in a principled Bayesian context and shows that they are naturally derived from endowing the agent with an exponential utility function . 2 Markov decision processes . In a Markov decision process ( MDP ) an agent interacts with an environment in a series of episodes and attempts to maximize the cumulative rewards . A finite horizon , discrete-time , discrete MDP is given by the tuple M = { S , A , R , P , L , ρ } , where S = { 1 , . . . , S } is the state-space , A = { 1 , . . . , A } is the action-space , Rl ( s , a ) is a probability distribution over the rewards received by the agent at state s taking action a at timestep l , Pl ( s′ | s , a ) ∈ [ 0 , 1 ] is the probability the agent will transition to state s′ after taking action a in state s at timestep l , L ∈ N is the episode length , and ρ is the initial state distribution . Concretely , the initial state s ∈ S of the agent is sampled from ρ , then for timesteps l = 1 , . . . , L the agent is in state s ∈ S , selects action a ∈ A , receives reward r ∼ Rl ( s , a ) with mean µl ( s , a ) ∈ R and transitions to the next state s′ with probability Pl ( s ′ | s , a ) . After timestep L the episode terminates and the state is reset . We assume that at the beginning of learning the agent does not know the reward or transition probabilities and must learn about them by interacting with the environment . We consider the Bayesian case in which the mean reward µ and the transition probabilities P are sampled from a known prior φ . We assume that the agent knows S , A , L , and the reward noise distribution . An agent following policy πl : S × A → [ 0 , 1 ] at state s ∈ S at time l selects action a with probability πl ( s , a ) . The Bellman equation relates the value of actions taken at the current timestep to future returns through the Q-values and the associated value function [ 8 ] , which for policy π are denoted Qπl ∈ RS×A and V πl ∈ RS for l = 1 , . . . , L+ 1 , and satisfy Qπl = T πl Qπl+1 , V πl ( s ) = ∑ a∈A πl ( s , a ) Q π l ( s , a ) , ( 1 ) for l = 1 , . . . , L where QL+1 ≡ 0 and where the Bellman operator for policy π at step l is defined as ( T πl Qπl+1 ) ( s , a ) : = µl ( s , a ) + ∑ s′∈S Pl ( s ′ | s , a ) ∑ a′∈A πl ( s ′ , a′ ) Qπl+1 ( s ′ , a′ ) . The expected performance of policy π is denoted Jπ = Es∼ρV π1 ( s ) . An optimal policy satisfies π ? ∈ argmaxπ Jπ and induces associated optimal Q-values and value function given by Q ? l = T ? l Q ? l+1 , V ? l ( s ) = max a Q ? l ( s , a ) . ( 2 ) for l = 1 , . . . , L , where Q ? L+1 ≡ 0 and where the optimal Bellman operator is defined at step l as ( T ? l Q ? l+1 ) ( s , a ) : = µl ( s , a ) + ∑ s′∈S Pl ( s ′ | s , a ) max a′ Q ? l+1 ( s ′ , a′ ) . ( 3 )

Summary: The paper proposes K-learning, a reinforcement learning exploration heuristic with Bayesian regret bounds. The authors adopt a Bayesian perspective under which long-term (Q-)values have epistemic uncertainty associated with them. They argue that exploration behaviour can be induced by maximising a particular (exponential) utility function of the resulting distributions. It is shown, that these utility scores obey a Bellman-like operator and can be computed using dynamic programming and an algorithm for episodic settings is proposed.  The authors link the optimal policy maximising the new objective to entropy regularised approaches, discuss the optimal selection of a temperature parameter governing risk-sensitivity and provide formal regret bounds under certain assumptions regarding the structure of the agents prior. Finally, K-learning is compared empirically to a number of exploration baselines where it performs competitively.