Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learning

1 Introduction . Reinforcement Learning ( RL ) is a general scenario where agents interact with the environment to achieve some goal . The environment and an agent ’ s interactions are typically modeled as a Markov decision process ( MDP ) [ 29 ] , which can represent a rich variety of tasks . But , for which MDPs can an agent or an RL algorithm succeed ? This requires a theoretical analysis of the complexity of an MDP . This paper studies this question in the tabular episodic setting , where an agent interacts with the environment in episodes of fixed length H and where the size of the state and action space is finite ( S and A respectively ) . While the performance of RL algorithms in tabular Markov decision processes has been the subject of many studies in the past [ e.g . 11 , 22 , 28 , 7 , 4 , 20 , 34 , 6 ] , the vast majority of existing analyses focuses on worst-case problem-independent regret bounds , which only take into account the size of the MDP , the horizon H and the number of episodes K. Recently , however , some significant progress has been achieved towards deriving more optimistic ( problem-dependent ) guarantees . This includes more refined regret bounds for the tabular episodic setting that depend on structural properties of the specific MDP considered [ 30 , 25 , 21 , 13 , 17 ] . Motivated by instance-dependent analyses in multi-armed bandits [ 24 ] , these analyses derive gapdependent regret-bounds of the form O ( ∑ ( s , a ) ∈S×A H log ( K ) gap ( s , a ) ) , where the sum is over state-actions pairs ( s , a ) and where the gap notion is defined as the difference of the optimal value function V ∗ of the Bellman optimal policy π∗ and the Q-function of π∗ at a sub-optimal action : gap ( s , a ) = ∗Author was at Johns Hopkins University during part of this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . V ∗ ( s ) −Q∗ ( s , a ) . We will refer to this gap definition as value-function gap in the following . We note that a similar notion of gap has been used in the infinite horizon setting to achieve instance-dependent bounds [ 1 , 31 , 2 , 12 , 27 ] , however , a strong assumption about irreducibility of the MDP is required . While regret bounds based on these value function gaps generalize the bounds available in the multiarmed bandit setting , we argue that they have a major limitation . The bound at each state-action pair depends only on the gap at the pair and treats all state-action pairs equally , ignoring their topological ordering in the MDP . This can have a major impact on the derived bound . In this paper , we address this issue and formalize the following key observation about the difficulty of RL in an episodic MDP through improved instance-dependent regret bounds : Learning a policy with optimal return does not require an RL agent to distinguish between actions with similar outcomes ( small value-function gap ) in states that can only be reached by taking highly suboptimal actions ( large value-function gap ) . To illustrate this insight , consider autonomous driving , where each episode corresponds to driving from a start to a destination . If the RL agent decides to run a red light on a crowded intersection , then a car crash is inevitable . Even though the agent could slightly affect the severity of the car crash by steering , this effect is small and , hence , a good RL agent does not need to learn how to best steer after running a red light . Instead , it would only need a few samples to learn to obey the traffic light in the first place as the action of disregarding a red light has a very large value-function gap . To understand how this observation translates into regret bounds , consider the toy example in Figure 1 . This MDP has deterministic transitions and only terminal rewards with c > 0 . There are two decision points , s1 and s2 , with two actions each , and all other states have a single action . There are three policies which govern the regret bounds : π∗ ( red path ) which takes action a1 in state s1 ; π1 which takes action a2 at s1 and a3 at s2 ( blue path ) ; and π2 which takes action a2 at s1 and a4 at s2 ( green path ) . Since π∗ follows the red path , it never reaches s2 and achieves optimal return c+ , while π1 and π2 are both suboptimal with return and 0 respectively . Existing value-function gaps evaluate to gap ( s1 , a2 ) = c and gap ( s2 , a4 ) = which yields a regret bound of order H log ( K ) ( 1/c+ 1/ ) . The idea behind these bounds is to capture the necessary number of episodes to distinguish the value of the optimal policy π∗ from the value of any other sub-optimal policy on all states . However , since π∗ will never reach s2 it is not necessary to distinguish it from any other policy at s2 . A good algorithm only needs to determine that a2 is sub-optimal in s1 , which eliminates both π1 and π2 as optimal policies after only log ( K ) /c2 episodes . This suggests a regret of order O ( log ( K ) /c ) . The bounds presented in this paper achieve this rate up to factors of H by replacing the gaps at every state-action pair with the average of all gaps along certain paths containing the state action pair . We call these averaged gaps return gaps . The return gap at ( s , a ) is denoted as gap ( s , a ) . Our new bounds replace gap ( s2 , a4 ) = by gap ( s2 , a4 ) ≈ 12 gap ( s1 , a2 ) + 1 2 gap ( s2 , a4 ) = Ω ( c ) . Notice that and c can be selected arbitrarily in this example . In particular , if we take c = 0.5 and = 1/ √ K our bounds remain logarithmic O ( log ( K ) ) , while prior regret bounds scale as √ K. This work is motivated by the insight just discussed . First , we show that improved regret bounds are indeed possible by proving a tighter regret bound for STRONGEULER , an existing algorithm based on the optimism-in-the-face-of-uncertainty ( OFU ) principle [ 30 ] . Our regret bound is stated in terms of our new return gaps that capture the problem difficulty more accurately and avoid explicit dependencies on the smallest value function gap gapmin . Our technique applies to optimistic algorithms in general and as a by-product improves the dependency on episode length H of prior results . Second , we investigate the difficulty of RL in episodic MDPs from an information-theoretic perspective by deriving regret lower-bounds . We show that existing value-function gaps are indeed sufficient to capture difficulty of problems but only when each state is visited by an optimal policy with some probability . Finally , we prove a new lower bound when the transitions of the MDP are deterministic that depends only on the difference in return of the optimal policy and suboptimal policies , which is closely related to our notion of return gap . 2 Problem setting and notation . We consider reinforcement learning in episodic tabular MDPs with a fixed horizon . An MDP can be described as a tuple ( S , A , P , R , H ) , where S and A are state- and action-space of size S and A respectively , P is the state transition distribution with P ( ·|s , a ) ∈ ∆S−1 the next state probability distribution , given that action a was taken in the current state s. R is the reward distribution defined over S ×A and r ( s , a ) = E [ R ( s , a ) ] ∈ [ 0 , 1 ] . Episodes admit a fixed length or horizon H . We consider layered MDPs : each state s ∈ S belongs to a layer κ ( s ) ∈ [ H ] and the only non-zero transitions are between states s , s′ in consecutive layers , with κ ( s′ ) = κ ( s ) + 1 . This common assumption [ see e.g . 23 ] corresponds to MDPs with time-dependent transitions , as in [ 20 , 7 ] , but allows us to omit an explicit time-index in value-functions and policies . For ease of presentation , we assume there is a unique start state s1 with κ ( s1 ) = 1 but our results can be generalized to multiple ( possibly adversarial ) start states . Similarly , for convenience , we assume that all states are reachable by some policy with non-zero probability , but not necessarily all policies or the same policy . We denote by K the number of episodes during which the MDP is visited . Before each episode k ∈ [ K ] , the agent selects a deterministic policy πk : S → A out of a set of all policies Π and πk is then executed for all H time steps in episode k. For each policy π , we denote by wπ ( s , a ) = P ( Sκ ( s ) = s , Aκ ( s ) = a | Ah = π ( Sh ) ∀h ∈ [ H ] ) and wπ ( s ) = ∑ a w π ( s , a ) probability of reaching stateaction pair ( s , a ) and state s respectively when executing π . For convenience , supp ( π ) = { s ∈ S : wπ ( s ) > 0 } is the set of states visited by π with non-zero probability . The Q- and value function of a policy π are Qπ ( s , a ) = Eπ [ H∑ h=κ ( s ) r ( Sh , Ah ) ∣∣∣∣∣ Sκ ( s ) = s , Aκ ( s ) = a ] , and V π ( s ) = Qπ ( s , π ( s ) ) and the regret incurred by the agent is the sum of its regret over K episodes R ( K ) = K∑ k=1 v∗ − vπk = K∑ k=1 V ∗ ( s1 ) − V πk ( s1 ) , ( 1 ) where vπ = V π ( s1 ) is the expected total sum of rewards or return of π and V ∗ is the optimal value function V ∗ ( s ) = maxπ∈Π V π ( s ) . Finally , the set of optimal policies is denoted as Π∗ = { π ∈ Π : V π = V ∗ } . Note that we only call a policy optimal if it satisfies the Bellman equation in every state , as is common in literature , but there may be policies outside of Π∗ that also achieve maximum return because they only take suboptimal actions outside of their support . The variance of the Q function at a state-action pair ( s , a ) of the optimal policy is V∗ ( s , a ) = V [ R ( s , a ) ] + Vs′∼P ( ·|s , a ) [ V ∗ ( s′ ) ] , where V [ X ] denotes the variance of the r.v . X . The maximum variance over all state-action pairs is V∗ = max ( s , a ) V∗ ( s , a ) . Finally , our proofs will make use of the following clipping operator clip [ a|b ] = χ ( a ≥ b ) a that sets a to zero if it is smaller than b , where χ is the indicator function .

Summary: This work studies instance-dependent regret bound in episodic tabular MDP. The main observation is that the gaps in state-action pairs that can only be reached by optimal/suboptimal policy should have little contribution in the regret. Based on this observation, the authors defined a new notion "return gap", which is never smaller than the original gap notion. Then the authors proved that STRONGEULER enjoys a regret upper bound that only depends on return gap. In addition, instance-dependent regret lower bound based on return gap is provided.