Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learning

1 Introduction . Reinforcement Learning ( RL ) is a general scenario where agents interact with the environment to achieve some goal . The environment and an agent ’ s interactions are typically modeled as a Markov decision process ( MDP ) [ 29 ] , which can represent a rich variety of tasks . But , for which MDPs can an agent or an RL algorithm succeed ? This requires a theoretical analysis of the complexity of an MDP . This paper studies this question in the tabular episodic setting , where an agent interacts with the environment in episodes of fixed length H and where the size of the state and action space is finite ( S and A respectively ) . While the performance of RL algorithms in tabular Markov decision processes has been the subject of many studies in the past [ e.g . 11 , 22 , 28 , 7 , 4 , 20 , 34 , 6 ] , the vast majority of existing analyses focuses on worst-case problem-independent regret bounds , which only take into account the size of the MDP , the horizon H and the number of episodes K. Recently , however , some significant progress has been achieved towards deriving more optimistic ( problem-dependent ) guarantees . This includes more refined regret bounds for the tabular episodic setting that depend on structural properties of the specific MDP considered [ 30 , 25 , 21 , 13 , 17 ] . Motivated by instance-dependent analyses in multi-armed bandits [ 24 ] , these analyses derive gapdependent regret-bounds of the form O ( ∑ ( s , a ) ∈S×A H log ( K ) gap ( s , a ) ) , where the sum is over state-actions pairs ( s , a ) and where the gap notion is defined as the difference of the optimal value function V ∗ of the Bellman optimal policy π∗ and the Q-function of π∗ at a sub-optimal action : gap ( s , a ) = ∗Author was at Johns Hopkins University during part of this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . V ∗ ( s ) −Q∗ ( s , a ) . We will refer to this gap definition as value-function gap in the following . We note that a similar notion of gap has been used in the infinite horizon setting to achieve instance-dependent bounds [ 1 , 31 , 2 , 12 , 27 ] , however , a strong assumption about irreducibility of the MDP is required . While regret bounds based on these value function gaps generalize the bounds available in the multiarmed bandit setting , we argue that they have a major limitation . The bound at each state-action pair depends only on the gap at the pair and treats all state-action pairs equally , ignoring their topological ordering in the MDP . This can have a major impact on the derived bound . In this paper , we address this issue and formalize the following key observation about the difficulty of RL in an episodic MDP through improved instance-dependent regret bounds : Learning a policy with optimal return does not require an RL agent to distinguish between actions with similar outcomes ( small value-function gap ) in states that can only be reached by taking highly suboptimal actions ( large value-function gap ) . To illustrate this insight , consider autonomous driving , where each episode corresponds to driving from a start to a destination . If the RL agent decides to run a red light on a crowded intersection , then a car crash is inevitable . Even though the agent could slightly affect the severity of the car crash by steering , this effect is small and , hence , a good RL agent does not need to learn how to best steer after running a red light . Instead , it would only need a few samples to learn to obey the traffic light in the first place as the action of disregarding a red light has a very large value-function gap . To understand how this observation translates into regret bounds , consider the toy example in Figure 1 . This MDP has deterministic transitions and only terminal rewards with c > 0 . There are two decision points , s1 and s2 , with two actions each , and all other states have a single action . There are three policies which govern the regret bounds : π∗ ( red path ) which takes action a1 in state s1 ; π1 which takes action a2 at s1 and a3 at s2 ( blue path ) ; and π2 which takes action a2 at s1 and a4 at s2 ( green path ) . Since π∗ follows the red path , it never reaches s2 and achieves optimal return c+ , while π1 and π2 are both suboptimal with return and 0 respectively . Existing value-function gaps evaluate to gap ( s1 , a2 ) = c and gap ( s2 , a4 ) = which yields a regret bound of order H log ( K ) ( 1/c+ 1/ ) . The idea behind these bounds is to capture the necessary number of episodes to distinguish the value of the optimal policy π∗ from the value of any other sub-optimal policy on all states . However , since π∗ will never reach s2 it is not necessary to distinguish it from any other policy at s2 . A good algorithm only needs to determine that a2 is sub-optimal in s1 , which eliminates both π1 and π2 as optimal policies after only log ( K ) /c2 episodes . This suggests a regret of order O ( log ( K ) /c ) . The bounds presented in this paper achieve this rate up to factors of H by replacing the gaps at every state-action pair with the average of all gaps along certain paths containing the state action pair . We call these averaged gaps return gaps . The return gap at ( s , a ) is denoted as gap ( s , a ) . Our new bounds replace gap ( s2 , a4 ) = by gap ( s2 , a4 ) ≈ 12 gap ( s1 , a2 ) + 1 2 gap ( s2 , a4 ) = Ω ( c ) . Notice that and c can be selected arbitrarily in this example . In particular , if we take c = 0.5 and = 1/ √ K our bounds remain logarithmic O ( log ( K ) ) , while prior regret bounds scale as √ K. This work is motivated by the insight just discussed . First , we show that improved regret bounds are indeed possible by proving a tighter regret bound for STRONGEULER , an existing algorithm based on the optimism-in-the-face-of-uncertainty ( OFU ) principle [ 30 ] . Our regret bound is stated in terms of our new return gaps that capture the problem difficulty more accurately and avoid explicit dependencies on the smallest value function gap gapmin . Our technique applies to optimistic algorithms in general and as a by-product improves the dependency on episode length H of prior results . Second , we investigate the difficulty of RL in episodic MDPs from an information-theoretic perspective by deriving regret lower-bounds . We show that existing value-function gaps are indeed sufficient to capture difficulty of problems but only when each state is visited by an optimal policy with some probability . Finally , we prove a new lower bound when the transitions of the MDP are deterministic that depends only on the difference in return of the optimal policy and suboptimal policies , which is closely related to our notion of return gap . 2 Problem setting and notation . We consider reinforcement learning in episodic tabular MDPs with a fixed horizon . An MDP can be described as a tuple ( S , A , P , R , H ) , where S and A are state- and action-space of size S and A respectively , P is the state transition distribution with P ( ·|s , a ) ∈ ∆S−1 the next state probability distribution , given that action a was taken in the current state s. R is the reward distribution defined over S ×A and r ( s , a ) = E [ R ( s , a ) ] ∈ [ 0 , 1 ] . Episodes admit a fixed length or horizon H . We consider layered MDPs : each state s ∈ S belongs to a layer κ ( s ) ∈ [ H ] and the only non-zero transitions are between states s , s′ in consecutive layers , with κ ( s′ ) = κ ( s ) + 1 . This common assumption [ see e.g . 23 ] corresponds to MDPs with time-dependent transitions , as in [ 20 , 7 ] , but allows us to omit an explicit time-index in value-functions and policies . For ease of presentation , we assume there is a unique start state s1 with κ ( s1 ) = 1 but our results can be generalized to multiple ( possibly adversarial ) start states . Similarly , for convenience , we assume that all states are reachable by some policy with non-zero probability , but not necessarily all policies or the same policy . We denote by K the number of episodes during which the MDP is visited . Before each episode k ∈ [ K ] , the agent selects a deterministic policy πk : S → A out of a set of all policies Π and πk is then executed for all H time steps in episode k. For each policy π , we denote by wπ ( s , a ) = P ( Sκ ( s ) = s , Aκ ( s ) = a | Ah = π ( Sh ) ∀h ∈ [ H ] ) and wπ ( s ) = ∑ a w π ( s , a ) probability of reaching stateaction pair ( s , a ) and state s respectively when executing π . For convenience , supp ( π ) = { s ∈ S : wπ ( s ) > 0 } is the set of states visited by π with non-zero probability . The Q- and value function of a policy π are Qπ ( s , a ) = Eπ [ H∑ h=κ ( s ) r ( Sh , Ah ) ∣∣∣∣∣ Sκ ( s ) = s , Aκ ( s ) = a ] , and V π ( s ) = Qπ ( s , π ( s ) ) and the regret incurred by the agent is the sum of its regret over K episodes R ( K ) = K∑ k=1 v∗ − vπk = K∑ k=1 V ∗ ( s1 ) − V πk ( s1 ) , ( 1 ) where vπ = V π ( s1 ) is the expected total sum of rewards or return of π and V ∗ is the optimal value function V ∗ ( s ) = maxπ∈Π V π ( s ) . Finally , the set of optimal policies is denoted as Π∗ = { π ∈ Π : V π = V ∗ } . Note that we only call a policy optimal if it satisfies the Bellman equation in every state , as is common in literature , but there may be policies outside of Π∗ that also achieve maximum return because they only take suboptimal actions outside of their support . The variance of the Q function at a state-action pair ( s , a ) of the optimal policy is V∗ ( s , a ) = V [ R ( s , a ) ] + Vs′∼P ( ·|s , a ) [ V ∗ ( s′ ) ] , where V [ X ] denotes the variance of the r.v . X . The maximum variance over all state-action pairs is V∗ = max ( s , a ) V∗ ( s , a ) . Finally , our proofs will make use of the following clipping operator clip [ a|b ] = χ ( a ≥ b ) a that sets a to zero if it is smaller than b , where χ is the indicator function .

Summary: This paper is on instance-dependent regret bounds for RL in finite episodic MDPs. The authors observe that existing bounds fail to capture some fundamental properties of the problem. Specifically, bounds based on value function gaps $gap(s,a)=V^{\star}(s)-Q^{\star}(s,a)$ ignore the fact that only state-action pairs that are relevant from telling optimal policies apart from suboptimal ones should contribute to the regret. So, the authors introduce a new concept of gap, the return gap, denoted $\overline{gap}(s,a)$, that is the average gap encountered up to reaching $(s,a)$ through some suboptimal action. Regret upper bounds are established for the StrongEuler algorithm by Simchowitz and Jamieson (2019). The main result replaces the inverse dependence on value function gaps with (larger) return gaps, and eliminates the dependence on the minimum gap. When transitions are deterministic, the result can be improved by ruling out state-action pairs that can only be reached by optimal policies. These results are complemented by an example MDP that shows why existing bounds can predict a much larger regret. The authors also provide asymptotic lower bounds for consistent algorithms. First they provide a general bound expressed in terms of an abstract optimization problem. Then they consider two special cases. For the case when every state is visited by some optimal policy, they lower bound the regret in terms of value function gaps, generalizing a result by Simchowitz and Jamieson (albeit for a worse dependence on the horizon). For MDPs with deterministic transitions, the lower bound is in terms of the return gaps, and only state-actions pairs that are not visited by any optimal policy count. The corresponding upper bound pays for these pairs if they can also be visited after playing a suboptimal action. This shows the intrinsic limitations of optimistic algorithms.