Title: Regret-Optimal Model-Free Reinforcement Learning for Discounted MDPs with Short Burn-In Time
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning theory

regret minimization

minimax optimality
Abstract: A crucial problem in reinforcement learning is learning the optimal policy. We study this in tabular infinite-horizon discounted Markov decision processes under the online setting. The existing algorithms either fail to achieve regret optimality or have to incur a high memory and computational cost. In addition, existing optimal algorithms all require a long burn-in time in order to achieve optimal sample efficiency, i.e., their optimality is not guaranteed unless sample size surpasses a high threshold. We address both open problems by introducing a model-free algorithm that employs variance reduction and a novel technique that switches the execution policy in a slow-yet-adaptive manner. This is the first regret-optimal model-free algorithm in the discounted setting, with the additional benefit of a low burn-in time.

Regret-Optimal Model-Free Reinforcement Learning for Discounted MDPs with Short Burn-In Time

Abstract

A crucial problem in reinforcement learning is learning the optimal policy. We study this in tabular infinite-horizon discounted Markov decision processes under the online setting. The existing algorithms either fail to achieve regret optimality or have to incur a high memory and computational cost. In addition, existing optimal algorithms all require a long burn-in time in order to achieve optimal sample efficiency, i.e., their optimality is not guaranteed unless sample size surpasses a high threshold. We address both open problems by introducing a model-free algorithm that employs variance reduction and a novel technique that switches the execution policy in a slow-yet-adaptive manner. This is the first regret-optimal model-free algorithm in the discounted setting, with the additional benefit of a low burn-in time.

1 Introduction

In reinforcement learning (RL), a crucial task is to find the optimal policy that maximizes its expected cumulative reward in any given environment with unknown dynamics. An immense body of literature is dedicated to finding algorithms that solve this task with as few samples as possible, which is the prime goal under this task. Ideally, one hopes to find an algorithm with a theoretical guarantee of optimal sample efficiency. At the same time, this task might be accompanied with additional requirements such as low space complexity and computational cost, as it is common that the state and action spaces exhibit high dimensions in modern applications. The combination of these various goals and requirements presents an important yet challenging problem in algorithm design.

The task of searching for optimal policy has been well-studied by existing work in the generative setting [33; 34; 21; 1]. This fundamental setting allows the freedom of querying samples at any state-action pair. In contrast, it is more realistic but difficult to consider the same task in the online setting, in which samples can only be collected along trajectories generated from executing a policy in the unknown Markov decision process (MDP). Solving this task with optimal sample efficiency requires a careful balance between exploration and exploitation, especially when coupled with other goals such as memory and computational efficiency.

MDPs can be divided into two types: the episodic finite-horizon MDPs and the infinite-horizon MDPs. Although these two types of MDPs can be approached in similar ways under the generative setting, there is a clear dichotomy between them in the online setting. In an episodic MDP, sample trajectories are only defined in fixed-length episodes, so samples are collected in episodes, and a reset to an arbitrary initial state occurs at the end of every online episode. Its transition kernel is usually assumed to be non-stationary over time. In contrast, the transition kernel of an infinite-horizon MDP stays stationary over time, and the online sample collection process amounts to drawing a single infinitely long sample trajectory with no reset. These differences render most optimal algorithms for episodic

MDPs suboptimal when applied to infinite-horizon MDPs. Without reset and non-stationarity, the high dependency between consecutive trajectory steps in the infinite-horizon setting presents a new challenge over the episodic setting. In this work, we consider the infinite-horizon discounted MDPs, which is widely used in practice but still has some fundamental questions unanswered in theory.

1.1 Sample Efficiency in Infinite-Horizon MDPs

To evaluate the sample efficiency of online RL algorithms, a natural and widely-accepted metric is the cumulative regret . It captures the performance difference between the optimal policy and the learned policy of an algorithm over its online interactions with a given MDP. The notion of cumulative regret was first introduced in the bandit literature and later adopted in the RL literature [2; 15]. It is profusely used in the online episodic RL literature. Such works aim to prove regret guarantees for algorithms and provide analyses that characterize such regret guarantees in terms of all problem parameters such as state space, action space and sample size in a non-asymptotic fashion. A cumulative regret guarantee can also suggest the sample complexity needed to reach a certain level of average regret.

In the online infinite-horizon setting, many works study a different metric called the sample complexity of exploration, first introduced in [17]. In essence, given a target accuracy level ϵ , this metric characterizes the total number of ϵ -suboptimal steps committed by an algorithm over an infinitelylong trajectory in the MDP. While this is indicative of the sample efficiency of an algorithm, the focus of this metric is very different from that of cumulative regret, as it only reflects the total number of failures but does not distinguish their sizes. As [24; 12] point out, even an optimal guarantee on the sample complexity of exploration can only be converted to a very suboptimal guarantee on the cumulative regret. To obtain a more quantitative characterization of the total volume of failures in the regime of finite samples, some works have turned to studying cumulative regret guarantees for algorithms.

It was not until recently that some works [24; 51; 12; 18] begin to research into the problem of cumulative regret minimization in infinite-horizon discounted MDPs. Among them, [51] focus on linear MDPs while others study tabular MDPs. In this work, we study the regret minimization problem in the tabular case. Hereafter and throughout, we denote the size of the state space, the size of the action space and the discount factor of the problem MDP with S , A and γ , respectively, and let T denote the sample size.

1.2 Model-Based and Model-Free Methods

Since modern RL applications are often large-scale, algorithms with low space complexity and computational complexity are much desired. This renders the distinction between model-based algorithms and model-free algorithms particularly important. The procedure of a model-based method includes a model estimation stage that involves estimating the transition kernel and a subsequent planning stage that searches the optimal policy in the learned model. Thus, O ( S 2 A ) space is required to store the estimated model. This is unfavorable when the state space is large and a memory constraint is present. Additionally, updating the transition kernel estimate brings a large computational burden. In comparison, model-free methods do not learn the entire model and thus can run with o ( S 2 A ) space. Notably, most value-based methods such as Q-learning only require storage of an estimated Q-function, which can take as little as O ( SA ) memory. In the infinite-horizon discounted setting, although UCBVIγ in [12] can achieve optimal regret, its model-based nature exacts a O ( S 2 A ) memory and computational cost; conversely, the algorithms in [24] and [18] are model-free but have suboptimal regret guarantee.

1.3 Burn-in Cost in Regret-Optimal RL

Naturally, one aims to develop algorithms that find the optimal policy with the fewest number of samples. In regards to regret, this motivates numerous works to work towards algorithms with minimax-optimal cumulative regret. However, the job is not done once such an algorithm is found. As can be seen in the episodic RL literature, algorithms that achieve optimal regret as sample size T tends towards infinity can still have different performance in the regime when T is limited. Specifically, for every existing algorithm, there exists a certain sample size threshold such that regret is suboptimal before T exceeds it. Such threshold is commonly referred to as the initial burn-in time of the algorithm. Therefore, it is of great interest to find an algorithm with low burn-in time so that it

can still attain optimal regret in the sample-starved regime. Such effort has been made by [21; 1] in the generative setting and by [23; 26] in the online episodic setting. Yet, this important issue has not been addressed in the infinite-horizon setting, as optimal algorithms all suffer long burn-in times.

Specifically, while UCBVIγ in [12] achieves a state-of-the-art regret guarantee of ˜ O (√ SAT (1 -γ ) 3 ) , which they prove minimax-optimal, their theory does not guarantee optimality unless the samples size T becomes as large as

This threshold can be prohibitively large when S and A are huge, which is true in most applications. Thus, this makes reducing these factors in the burn-in cost particularly important. For instance, a 5-by-5 tic-tac-toe has a state space of size 3 25 . While this is a manageable number in modern machine learning, any higher power of it may cause computational difficulties; in contrast, the horizon of this game is much smaller-no more than 25 . Since no lower bound precludes regret optimality for T ≥ SA (1 -γ ) 4 , one might hope to design an algorithm with smaller S and A factors in the burn-in cost so that it can achieve optimality even in the sample-starved regime.

1.4 Summary of Contributions

While it is encouraging to see recent works have shown that in the discounted setting, modelfree methods can provide nearly optimal guarantees on sample complexity of exploration and that model-based methods can provide nearly optimal finite-sample regret guarantees, there still lacks a model-free approach that can attain regret optimality . In the orthogonal direction, there is still a vacancy for algorithms that can attain optimal regret for a broader sample size range, i.e., with fewer samples than S 3 A 2 poly(1 -γ ) .

In fact, we can summarize these two lingering theoretical questions as follows:

Is there an algorithm that can achieve minimax regret optimality with low space complexity and computational complexity in the infinite-horizon discounted setting, even when sample size is limited?

We answer this question affirmatively with a new algorithm Q-SlowSwitch-Adv , which uses variance reduction and a novel adaptive switching technique. It is the first model-free algorithm that achieves optimal regret in the infinite-horizon discounted setting. This result can be summarized as follows:

Theorem (informal) . For any sample size T ≥ SA poly(1 -γ ) , Q-SlowSwitch-Adv is guaranteed to achieve near-optimal cumulative regret ˜ O (√ SAT (1 -γ ) 3 ) with space complexity O ( SA ) and computational complexity O ( T ) .

A formal theorem is presented in Section 4; its proof can be found in the full version [14]. We also provide a complete summary of related prior results in Table 1. A discussion about the additional related work is deferred to Appendix A.

2 Problem Formulation

Let us specify the problem we aim to study in this section. Throughout this paper, we let ∆( X ) denote the probability simplex over any set X . We also introduce the notation [ m ] := { 1 , 2 , · · · , m } for a positive integer m .

2.1 Infinite-Horizon Discounted Markov Decision Process

We consider an infinite-horizon discounted Markov decision process (MDP) represented with ( S , A , γ, P, r ) . Notably, we consider a tabular one, in which S := { 1 , 2 , · · · , S } denotes the state space with size S and A := { 1 , 2 , · · · , A } denotes the action space with size A . P : S × A → ∆( S ) denotes the probability transition kernel in that P ( ·| s, a ) ∈ ∆( S ) is the transition probability vector from state s ∈ S when action a ∈ A is taken. r : S × A → [0 , 1] denotes the reward function, which is assumed to be deterministic in this work. Specifically, r ( s, a ) is the immediate reward for taking action a ∈ A at state s ∈ S . Lastly, γ denotes the discount factor for the reward, which makes 1 1 -γ the effective horizon.

A (stationary) policy π : S → ∆( A ) specifies a rule for action selection in that π ( ·| s ) ∈ ∆( A ) is the action selection probability vector at state s ∈ S . We overload this notation by letting π ( s ) denote the action policy π takes at state s . Given a policy π , the Q-function of π is defined as

in which s t +1 ∼ P ( ·| s t , a t ) for t ≥ 0 and a t ∼ π ( ·| s t ) for t ≥ 1 . Moreover, the value function of π is defined as

in which s t +1 ∼ P ( ·| s t , a t ) and a t ∼ π ( ·| s t ) for t ≥ 0 . The Q-function and value function satisfy an equation, called the Bellman equation [6]:

A policy π ⋆ is called an optimal policy if it maximizes the value function for all states simultaneously. The optimal value function and optimal Q-function can be defined as

for any optimal policy π ⋆ . The optimal policy always exists and satisfies the Bellman optimality equation [30]:

2.2 Online Learning in an Infinite-Horizon MDP

We consider the online (single-trajectory) setting, in which the agent is permitted to execute a total of T steps sequentially in the MDP. More specifically, the agent starts from an arbitrary (and possibly adversarial) initial state s 1 . At each step t ∈ [ T ] , the agent at state s t computes policy π t , takes action a t based on π t ( ·| s t ) , receives reward r ( s t , a t ) , and transitions to state s t +1 in the following step. At the end of execution, the agent generates a trajectory ( s 1 , a 1 , r 1 , s 2 , a 2 , r 2 , · · · , s T , a T , r T ) , which amounts to T samples.

2.3 Problem: Regret Minimization

As a standard metric to evaluate the performance of the aforementioned agent over a finite number of T steps, the cumulative regret with respect to the sequence of stationary policies { π t } T t =1 learned by the algorithm is defined as follows:

Verbally, the regret measures the cumulative suboptimality between the optimal policy and the execution policy π t at each step throughout the T -step online interaction process. Naturally, one aims to minimize this regret by finding an algorithm whose regret scales optimally in T . This would require a strategic balance between exploration and exploitation, which can be difficult when sample size T is small.

Remark 1 . In the infinite-horizon setting, many prior works [12; 24] consider slightly different regret definitions with respect to non-stationary policies. Specifically, at each s t along the trajectory, this different regret metric compares the optimal value function V ⋆ ( s t ) against the expected cumulative reward of running the non-stationary policy { π k } ∞ k = t starting from s t . By doing this, it is effectively evaluating the cumulative reward difference between the stationary optimal policy and a non-stationary algorithm. While there exists no formal conversion between the regret defined in this way and the one in (3), it is expected to be smaller and thus more easily controlled than (3), because the execution policy π t improves over time. In addition, we can show our algorithm also achieves the same level of which satisfy

Algorithm 1: Q-SlowSwitch-Adv

1 Initialize: ∀ ( s, a ) , Q lazy ( s, a ) , Q ( s, a ) , Q UCB ( s, a ) , Q R ( s, a ) , Q M ( s, a ) ← 1 1 -γ ; N ( s, a ) ← 0 ; V ( s ) , V R ( s ) ← 1 1 -γ ; Q LCB ( s, a ) , θ ( s, a ) ← 0 ; D ← dict() ; µ ref ( s, a ) , σ ref ( s, a ) , µ adv ( s, a ) , σ adv ( s, a ) , B R ( s, a ) , δ R ( s, a ) ← 0 ; u switch ← False ; u ref ( s ) ← True ; H = ⌈ 2 ⌉ ; ι = log SAT .

regret under this different definition with an analysis specifically tailored to our algorithm, which is deferred to Appendix B. Furthermore, since the transition kernel in the infinite-horizon setting is invariant over time and the optimal policy itself is also stationary, it is more natural to compare the optimal policy to a stationary policy, e.g., the policy π t deployed by the algorithm at each step, as in (3). Before this work, this has also been recently studied in [49; 18].

Notation. Given any vector x ∈ R SA that represents a function x : S × A → R , we use x ( s, a ) to denote the entry corresponding to the state-action pair ( s, a ) . We also denote the probability transition vector at ( s, a ) with

that is, given any V ∈ R S , P s,a V = E s ′ ∼ P ( ·| s,a ) [ V ( s ′ )] . For two vectors x, y ∈ R SA , we override the notation x ≤ y to mean that x ( s, a ) ≤ y ( s, a ) in every dimension ( s, a ) .

3 Algorithm

In this section, we present our algorithm Q-SlowSwitch-Adv and some relevant discussion.

3.1 Review: Q-Learning with UCB and reference advantage

First, we make a brief review of the Q-learning with UCB method proposed in [15], referred to as UCB-Q hereafter, and its variance-reduced variant UCB-Q-Advantage , later introduced in [48]. The Q-function updates in Q-SlowSwitch-Adv are inspired by these two methods.

Q-learning with UCB The original Q-learning [41; 40] is a fixed-point iteration based on a stochastic approximation of the Bellman optimality equation (2). It uses a greedy policy with respect to its estimate of Q ⋆ , whose update rule can be summarized as:

Above, Q (resp. V ) is the running estimate of Q ⋆ (resp. V ⋆ ); η ∈ (0 , 1] is the (possibly varying) learning rate; ̂ P s,a V is a stochastic approximation of P s,a V (cf. (4)). Commonly, V ( s ′ ) is used for ̂ P s,a V in (5) as an unbiased estimate of P s,a V , when a sample of state transition from ( s, a ) , namely ( s, a, s ′ ) , is available.

However, as [15] point out, using (5) naïvely suffers from great regret suboptimality, for it rules out the state-action pairs with high value but few observations. To promote the exploration of such state-action pairs, UCB-Q appends (5) with an exploration bonus. Its update rule can be written as:

To encourage exploration, the bonus b ≥ 0 is designed to maintain an upper confidence bound (UCB) on ( ̂ P s,a -P s,a ) V , which in turn keeps Q UCB ( s, a ) as an 'optimistic' overestimate of Q ⋆ ( s, a ) .

Q-learning with UCB and reference advantage The regret guarantee for UCB-Q is still shy of being optimal. In order to attain optimality, one can turn to the celebrated idea of variance reduction [16; 23; 34; 37], which decomposes the stochastic approximation target into two parts: a low-variance reference estimated with batches of samples and a low-magnitude advantage estimated with every new sample. In this spirit, [48] introduce UCB-Q-Advantage based on UCB-Q and a reference-advantage decomposition. Specifically, given a reference V R that is maintained as an approximation for V ⋆ , the update rule of UCB-Q-Advantage reads:

This idea of (7) is used in subroutine update-q-reference () of our algorithm. Let us discuss it in more details:

Given a transition sample ( s, a, s ′ ) , we can take V ( s ′ ) -V R ( s ′ ) as an unbiased estimate of the advantage P s,a ( V -V R ) . The magnitude of V -V R is small when V and V R are close. This engenders smaller stochastic volatility, compared to ̂ P s,a V in (6) from UCB-Q .

The reference estimate ̂ PV R is a stochastic approximation of PV R . In our algorithm, the auxiliary estimate µ ref (cf. Line 13 of Algorithm 2) is used as the estimate for PV R . Specifically, µ ref ( s, a ) is a running mean of P s,a V R based on the samples from all past visitations of ( s, a ) . In contrast to the advantage, which is computed every time a new sample arrives, the reference is computed with a batch of samples and thus more stable. In sacrifice, the reference is only updated intermittently and not as up-to-date as the advantage.

The exploration bonus b R is computed from the auxiliary estimates in Line 8 and 9 to serve as an upper confidence bound on the aggregation of the aforementioned reference and advantage. Specifically, for each ( s, a ) , µ ref ( s, a ) and σ ref ( s, a ) are the running mean and 2 nd moment of the reference [ PV R ]( s, a ) respectively; µ adv ( s, a ) and σ adv ( s, a ) are the running mean and 2 nd moment of the advantage [ P ( V -V R )]( s, a ) respectively; B R ( s, a ) combines the empirical standard deviations of the reference and the advantage; δ R ( s, a ) is the temporal difference between B R ( s, a ) and its previous value. b R ( s, a ) can be computed from these estimates as a temporally-weighted average of B R ( s, a ) . Thanks to the low variability of the reference PV R , we can obtain a more accurate, milder overestimation in the upper confidence bound for faster overall convergence.

3.2 Review: Early settlement of reference value

Given the optimistic overestimates Q UCB and Q R , it is natural to design an update rule of our Q-function estimate as the minimum of the estimate itself and these two overestimates (Line 9 of Algorithm 1). This makes our Q-function estimate monotonically decrease without violating the optimistic principle Q ≥ Q ⋆ , which effectively enables us to lessen the overestimation in Q over time until it converges to Q ⋆ . In fact, this is precisely the update rule in UCB-Q-Advantage [48]. Nevertheless, we need to equip our algorithm with additional features to strive for regret optimality.

[23] introduced a way to update the reference with higher sample efficiency in the finite-horizon setting. As they noted, it is critical to update the reference V R in a smart fashion so as to balance the tradeoff between its synchronization with V and the volatility that results from too many stochastic updates. Concretely, the reference V R needs to be updated in a timely manner from V so that the magnitude of ̂ P s,a ( V -V R ) can be kept low as desired, but the updates cannot be too frequent either, because the stochasticity or variance in ̂ PV R ( s, a ) could be as high as that in ̂ P s,a V of (6) and thus lead to suboptimality if it is not carefully controlled. To resolve this dilemma, we can update V R until it becomes sufficiently close to V ⋆ and fix its value thereafter.

To this end, we maintain a 'pessimistic' underestimate Q LCB (resp. V LCB ) of Q ⋆ (resp. V ⋆ ) in the algorithm, which are computed from the lower confidence bound for Q ⋆ (resp. V ⋆ ). This can provide us with an upper bound on V R -V ⋆ , which will be used to determine when the update of the reference V R should be stopped.

In particular, the if-else block in Line 23 to 26 is designed to keep the reference V R synchronized with V for each state s respectively and terminate the update once

This can guarantee | V -V R | ≤ 6 throughout the execution of the algorithm. As a result, the standard deviation of ̂ P s,a ( V -V R ) is guaranteed to be O (1) , which can be O ( 1 1 -γ ) times smaller than the standard deviation of ̂ P s,a V in (2). This can lead to smaller 1 1 -γ factor in the final regret guarantee.

3.3 Adaptive low-switching greedy policy

Although these aforementioned designs from the finite-horizon literature help increase the accuracy of our estimate Q , they are still insufficient to attain regret optimality in the infinite-horizon setting. Since data collection takes place over a single trajectory with no reset, drastic changes in the execution policy can inflict a long-lasting volatility on the future trajectory and slow down the convergence. This is precisely the difficulty of the infinite-horizon setting over the finite-horizon one. The need to control the trajectory variability motivates us to design a novel adaptive switching technique.

Recall in UCB-Q and UCB-Q-Advantage , the execution policy is greedy with respect to the estimate Q , i.e., π t ( s t ) = arg max a Q ( s t , a ) . Every time Q gets updated, what the algorithm effectively does is to make an estimate of Q π t with the samples generated by π t . Such Q π t is only estimated and updated once before the execution policy is switched to π t +1 . This seems insufficient from a stochastic fixed-point iteration perspective, so we seek to update it more and learn each Q π t better before switching to a new policy.

To tackle this issue, we make the execution policy π t greedy to Q lazy , which is updated lazily yet adaptively in Q-SlowSwitch-Adv . Specifically, for every ( s, a ) , we use θ ( s, a ) (cf. Line 12 in Algorithm 2) to keep track of the cumulative difference between the current Q ( s, a ) and Q M ( s, a ) , the latter of which is defined to be the value of Q ( s, a ) last time Q lazy is updated immediately after visiting ( s, a ) . Whenever θ ( s, a ) exceeds 1 1 -γ , indicating Q lazy ( s, a ) and the execution policy has become outdated with respect to the current Q ( s, a ) , we reset θ ( s, a ) and set u switch to True , which will direct the algorithm to update the entire function Q lazy in the following step. update-q-lazy () updates Q lazy with the samples from D , which is a dictionary that serves as a buffer to store all the new sample transitions and their latest estimates since the last update of Q lazy .

In contrast, conventional low-switching algorithms update the execution policy on a predetermined, exponentially phased schedule [5; 48]. While trajectory stability is attained with these algorithms, as time goes on, it takes them exponentially longer to update policy, making them oblivious to recent large updates in the estimated Q-function. This would lead to suboptimal regret in the infinite-horizon setting, as continual choices of suboptimal actions will keep a lasting effect on future trajectory in the absence of trajectory reset. This issue is overcome in our algorithm by ignoring minor changes in function Q yet still being adaptive to substantial changes in any state-action pair.

4 Main Results

Our model-free algorithm Q-SlowSwitch-Adv can achieve optimal regret with short burn-in time. Its theoretical guarantee can be summarized in the following theorem.

Theorem 1. Choose any δ ∈ (0 , 1) . Suppose ι = log SAT δ and c b is chosen to be a sufficiently large universal constant in Algorithm 1. Then with probability at least 1 -δ , Algorithm 1 achieves

for an absolute constant C 0 > 0 .

To prove this theorem, we need to use a recursive error decomposition scheme different from the existing work. The stationary nature of the infinite-horizon setting gives rise to several error terms unique to the infinite-horizon setting, and our novel switching technique is crucial at controlling them optimally. The proof is provided in the full version [14]. Now let us highlight a few key properties of our algorithm.

Optimal regret with low burn-in. Q-SlowSwitch-Adv achieves optimal regret modulo some logarithmic factor as soon as the sample size T exceeds

This burn-in threshold is significantly lower than the S 3 A 2 poly(1 -γ ) threshold in [12] when SA ≫ 1 1 -γ . In other words, in the regime of (10), the regret of Q-SlowSwitch-Adv is guaranteed to satisfy

which matches the lower bound in Table 1.

Sample complexity. As a corollary of Theorem 1, it can be seen that Q-SlowSwitch-Adv attains ϵ -average regret (i.e. 1 T Regret( T ) ≤ ϵ for any fixed T ) with sample complexity

This is lower than the sample complexity of the model-free algorithm in [24], which is ˜ O ( SA (1 -γ ) 5 ϵ 2 ) .

Moreover, (12) holds true for any desired accuracy ϵ ∈ ( 0 , (1 -γ ) 13 SA ] . This is a broader range than the ones in [12; 49], which involve higher order of S and A and only allow their algorithms to attain their respective optimal sample complexity in the high-precision regime.

Space complexity. Q-SlowSwitch-Adv is a model-free algorithm that keeps a few estimates of the Q-function during execution, so its memory cost is as low as O ( SA ) . This is not improvable in the tabular setting, since it requires O ( SA ) units of space to store the optimal policy. In contrast, the model-based UCBVIγ in [12] stores an estimate of the probability transition kernel and thus incurs a higher memory cost of O ( S 2 A ) .

Computational complexity. The computational cost of Q-SlowSwitch-Adv is only O ( T ) . This is on the same order as reading samples along the T -length executed trajectory and is thus unimprovable. In comparison, our algorithm has a considerably lower computational cost than the one in [12], which requires O ( S 2 AT ) operations overall.

5 Discussion

This work has introduced a model-free algorithm that achieves optimal regret in infinite-horizon discounted MDPs, which reduces the space and computational complexity requirement for regret optimality in the existing work. It also achieves optimal sample efficiency with a short burn-in time compared to other algorithms, including [12; 49]. Moreover, our algorithm has demonstrated the importance of switching policies slowly in infinite-horizon MDPs and introduced a novel technique might be of additional interest to future work. While our burn-in threshold is considerably reduced with respect to the order of S and A , it still has nontrivial suboptimality in the effective horizon 1 1 -γ , which is a price we pay for using the reference-advantage technique. It is open for future work to investigate how to further improve the effective horizon factors in the burn-in cost.

References

Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory , pages 67-83. PMLR, 2020.

Peter Auer and Ronald Ortner. Online regret bounds for a new reinforcement learning algorithm. Proceedings 1st Austrian Cognitive Vision Workshop (ACVW 2005) , pages 35-42, 2005.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Mach. Learn. , 47(2-3):235-236, may 2002. ISSN 0885-6125. URL https: //doi.org/10.1023/A:1013689704352 .

Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Yu Bai, Tengyang Xie, Nan Jiang, and Yu-Xiang Wang. Provably efficient q-learning with low switching cost. Advances in Neural Information Processing Systems , 32, 2019.

Dimitri P. Bertsekas. Dynamic Programming and Optimal Control . Athena Scientific, 3rd edition, 2005.

Ronen I. Brafman and Moshe Tennenholtz. R-max - a general polynomial time algorithm for near-optimal reinforcement learning. J. Mach. Learn. Res. , 3(null):213-231, mar 2003. ISSN 1532-4435. doi: 10.1162/153244303765208377. URL https://doi.org/10.1162/ 153244303765208377 .

Kefan Dong, Yuanhao Wang, Xiaoyu Chen, and Liwei Wang. Q-learning with ucb exploration is sample efficient for infinite-horizon mdp. arXiv preprint arXiv:1901.09311 , 2019.

Simon S Du, Jianshu Chen, Lihong Li, Lin Xiao, and Dengyong Zhou. Stochastic variance reduction methods for policy evaluation. In International Conference on Machine Learning , pages 1049-1058. PMLR, 2017.

Minbo Gao, Tianle Xie, Simon S Du, and Lin F Yang. A provably efficient algorithm for linear markov decision process with low switching cost. arXiv preprint arXiv:2101.00494 , 2021.

Rong Ge, Zhize Li, Weiyao Wang, and Xiang Wang. Stabilized svrg: Simple variance reduction for nonconvex optimization. In Conference on learning theory , pages 1394-1448. PMLR, 2019.

Jiafan He, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for discounted mdps. Advances in Neural Information Processing Systems , 34:22288-22300, 2021.

Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. J. Mach. Learn. Res. , 11:1563-1600, Aug 2010. ISSN 1532-4435.

Xiang Ji and Gen Li. Regret-optimal model-free reinforcement learning for discounted mdps with short burn-in time. arXiv preprint arXiv:2305.15546 , 2023.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I. Jordan. Is Q-learning Provably Efficient? arXiv e-prints , art. arXiv:1807.03765, July 2018.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the 26th International Conference on Neural Information Processing Systems , NIPS'13, pages 315-323, Red Hook, NY, USA, 2013. Curran Associates Inc.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

Ian A Kash, Lev Reyzin, and Zishun Yu. Slowly changing adversarial bandit algorithms are provably efficient for discounted mdps. arXiv preprint arXiv:2205.09056 , 2022.

Michael Kearns and Satinder Singh. Finite-sample convergence rates for q-learning and indirect algorithms. In Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II , pages 996-1002, Cambridge, MA, USA, 1999. MIT Press.

Tor Lattimore and Marcus Hutter. Pac bounds for discounted mdps. In International Conference on Algorithmic Learning Theory , pages 320-334. Springer, 2012.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model. Advances in neural information processing systems , 33:12861-12872, 2020.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Sample complexity of asynchronous q-learning: Sharper analysis and variance reduction. Advances in neural information processing systems , 33:7031-7043, 2020.

Gen Li, Laixi Shi, Yuxin Chen, Yuantao Gu, and Yuejie Chi. Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning. Advances in Neural Information Processing Systems , 34:17762-17776, 2021.

Shuang Liu and Hao Su. Regret bounds for discounted mdps. arXiv preprint arXiv:2002.05138 , 2020.

Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variancereduced) policy gradient and natural policy gradient methods. Advances in Neural Information Processing Systems , 33:7624-7636, 2020.

Pierre Ménard, Omar Darwiche Domingues, Xuedong Shang, and Michal Valko. Ucb momentum q-learning: Correcting the bias without forgetting. In International Conference on Machine Learning , pages 7609-7618. PMLR, 2021.

Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takáˇ c. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International Conference on Machine Learning , pages 2613-2621. PMLR, 2017.

Aldo Pacchiano, Philip Ball, Jack Parker-Holder, Krzysztof Choromanski, and Stephen Roberts. On optimism in model-based reinforcement learning. arXiv preprint arXiv:2006.11911 , 2020.

Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International conference on machine learning , pages 4026-4035. PMLR, 2018.

Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., USA, 1st edition, 1994. ISBN 0471619779.

Dan Qiao, Ming Yin, Ming Min, and Yu-Xiang Wang. Sample-efficient reinforcement learning with loglog (t) switching cost. arXiv preprint arXiv:2202.06385 , 2022.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.13890 , 2022.

Aaron Sidford, Mengdi Wang, Xian Wu, Lin F Yang, and Yinyu Ye. Near-optimal time and sample complexities for solving discounted markov decision process with a generative model. arXiv preprint arXiv:1806.01492 , 2018.

Aaron Sidford, Mengdi Wang, Xian Wu, and Yinyu Ye. Variance reduced value iteration and faster algorithms for solving markov decision processes. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 770-787. SIAM, 2018.

Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. Pac model-free reinforcement learning. In Proceedings of the 23rd International Conference on Machine Learning , ICML '06, pages 881-888, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832.

Istvan Szita and Csaba Szepesvari. Model-based reinforcement learning with nearly tight exploration complexity bounds. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, pages 1031-1038, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.

Martin J Wainwright. Variance-reduced q -learning is minimax optimal. arXiv preprint arXiv:1906.04697 , 2019.

Mengdi Wang. Randomized linear programming solves the discounted markov decision problem in nearly-linear (sometimes sublinear) running time. arXiv preprint arXiv:1704.01869 , 2017.

Tianhao Wang, Dongruo Zhou, and Quanquan Gu. Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems , 34:13524-13536, 2021.

C. J. C. H. Watkins. Learning from Delayed Rewards . PhD thesis, King's College, Oxford, 1989.

Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning , 8(3):279-292, 1992. doi: 10.1007/BF00992698.

Lin Xiao and Tong Zhang. A proximal stochastic gradient method with progressive variance reduction. SIAM Journal on Optimization , 24(4):2057-2075, 2014.

Tengyu Xu, Zhe Wang, Yi Zhou, and Yingbin Liang. Reanalysis of variance reduced temporal difference learning. arXiv preprint arXiv:2001.01898 , 2020.

Yuling Yan, Gen Li, Yuxin Chen, and Jianqing Fan. The efficacy of pessimism in asynchronous q-learning. arXiv preprint arXiv:2203.07368 , 2022.

Kunhe Yang, Lin Yang, and Simon Du. Q-learning with logarithmic regret. In International Conference on Artificial Intelligence and Statistics , pages 1576-1584. PMLR, 2021.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems , 34:7677-7688, 2021.

Junyu Zhang, Chengzhuo Ni, Csaba Szepesvari, Mengdi Wang, et al. On the convergence and sample efficiency of variance-reduced policy gradient method. Advances in Neural Information Processing Systems , 34:2228-2240, 2021.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Almost optimal model-free reinforcement learningvia reference-advantage decomposition. Advances in Neural Information Processing Systems , 33:15198-15207, 2020.

Zihan Zhang, Yuan Zhou, and Xiangyang Ji. Model-free reinforcement learning: from clipped pseudo-regret to sample complexity. In International Conference on Machine Learning , pages 12653-12662. PMLR, 2021.

Zihan Zhang, Yuhang Jiang, Yuan Zhou, and Xiangyang Ji. Near-optimal regret bounds for multi-batch reinforcement learning. arXiv preprint arXiv:2210.08238 , 2022.

Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning , pages 12793-12802. PMLR, 2021.

A Related Work

In this section, we take a moment to discuss the related work beyond those in Table 1.

Regret analysis for online episodic RL In the online episodic setting, regret is the predominant choice of metric for demonstrating the sample efficiency of a method [45; 28; 13]. [4] was the first to introduce a model-based method that can achieve near-optimal regret guarantee, but the model-based nature of their method induces a high space complexity and burn-in time. On the other hand, model-free methods are proposed in [15; 5], which are motivated by Q-learning and thus enjoy a low space complexity. However, these methods cannot guarantee optimal regret. It was not until [48] that proposed the first model-free method with optimal regret guarantee UCB-Q-Advantage , but it incurs a large burn-in time of S 6 A 4 H 28 , where H is the horizon of the episodic MDP. In addition, [26] proposed UCB-M-Q , a Q-learning variant with momentum, which can achieve optimal regret with low burn-in time, but it requires the storage of all momentum bias and thus incurs high memory cost. Recently, [23] propose a Q-learning variant with variance reduction that achieves optimal regret with O ( SAH ) space complexity and SA poly( H ) burn-in threshold at the same time. Table 1 in [23] provides a more detailed comparison of related work from the online episodic RL literature.

Sample complexity for infinite-horizon RL In the infinite-horizon setting, there exist other sample efficiency metrics besides sample complexity of exploration. Initially, [19] considered the sample complexity needed to find an ϵ -approximate optimal policy. The same definition is also considered in [38; 34; 33; 21]. Later, [37] studied the sample complexity needed to find an ϵ -approximate optimal Q-function. Note that all of these works assume the generative setting. Indeed, a limitation of these aforementioned sample complexity definitions is that they only measure the performance of the final output policy and do not reflect the online regret during learning. Thus, existing works that study the online setting consider sample complexity of exploration and cumulative regret instead.

Variance reduction in RL The idea of variance reduction was first introduced to accelerate stochastic finite-sum optimization by [16], which is followed by a rich literature [42; 27; 11]. Later, for better sample efficiency in RL, it is applied to policy gradient methods [25; 47; 29] as well as value-based methods in various problems including generative setting RL [33; 34; 37], policy evaluation [9; 43], asynchronous Q-learning [22; 44] and offline RL [46; 32].

Low-switching algorithms Since our algorithm includes a novel feature that switches the execution policy slowly, we make a review of the low-switching approaches in RL. The idea of changing the execution policy infrequently during learning was first introduced by [3] as an approach to minimize regret in the multi-armed bandit problem. [5] adapted this idea to tabular RL and formalized the switching cost as a secondary metric that an algorithm can minimize. To reduce the number of policy switches and thus the switching cost, their algorithm updates the policy in geometrically longer intervals. Similar techniques can be found in [48], whose algorithm can achieve regret optimality while maintaining low switching cost. Later, [10; 39] introduced a new low-switching approach in linear MDPs by switching policies when the estimated covariance matrix gets a significant update. All these methods guarantee a O (log T ) switching cost. The switching cost guarantee was later improved to O (log log T ) by the algorithms proposed in [31; 50].

B A Discussion about Different Regret Metrics

As we have discussed earlier, multiple metrics exist in the existing literature for evaluating the online performance of an RL algorithm in the infinite-horizon setting. In fact, [12] and [24] both use a respectively different regret metric from the one in this work and [18]. An argument showing the equivalence between the regret metric in [24] and the one in [12] is discussed in Appendix A.2 of [12]. For this reason and the fact that the regret guarantee in [24] is not minimax-optimal, we focus on the relation between our theoretical guarantee in Theorem 1 and the regret metric from [12] in this section.

Recall that the goal of RL is to learn the optimal policy through online interactions with the environment (and there always exists a stationary optimal policy), so the regret metric we consider in this

work is

where π t is the stationary policy that the algorithm uses to take action a t at step t .

The non-stationary regret metric considered in [12] is

Here, V { π j } ∞ j = t ( s ) := E [∑ ∞ i =0 γ i r ( s i , π t + i ( s i )) | s 0 = s ] , which is the expected cumulative future reward of the non-stationary algorithm starting from time t .

In fact, similar difference can also be observed in the literature focused on sample complexity of exploration. Notably, the metric in [49] compares the optimal value function V ⋆ ( s t ) against V π t ( s t ) , while the metric in works such as [35; 8] compares against V { π j } ∞ j = t ( s t ) .

There is no formal equivalence between the two regret metrics in general, despite the intuition we have provided in Remark 1 that Regret NS ( T ) should be smaller than Regret( T ) for any improving algorithm. However, we can show that the specific algorithm Q-SlowSwitch-Adv (Algorithm 1) achieves ˜ O ( √ SAT (1 -γ ) 3 ) regret under the regret metric defined in [12], which matches the lower bound in [12] and is thus optimal for this metric as well.

Before the analysis, let us first define a notation. Let f and g be two real-valued functions that take X := ( S, A, γ, T, 1 δ ) as arguments. If there exists a universal constant C > 0 such that f ( X ) ≤ Cg ( X ) for any instantiation of X , we can denote this with the notation f ( X ) ≲ g ( X ) .

Let us define

̸

which is the set of time indices that the execution policy switches in the following step, and

which is the set of time indices that the execution policy switches in any of the following H steps. First, notice

where the second line holds when H ≳ log T 1 -γ , and the last line makes use of the Azuma-Hoeffding inequality.

Similarly, we have

since π t = π t + i for 0 ≤ i ≤ H and t / ∈ T H .

Putting (17) and (18) together leads to

by noticing that ∣ ∣ ∣ ∑ t ∈T H ( V π t ( s t ) -V { π j } ∞ j = t ( s t ) ) ∣ ∣ ∣ ≲ |T H | 1 -γ ≲ |T | log T (1 -γ ) 2 .

Since our algorithm is low-switching and |T | ≤ ˜ O ( SA (1 -γ ) 9 / 2 + ( SA ) 3 / 4 T 1 / 4 (1 -γ ) 5 / 4 ) , the difference between the two regret metrics (19) is dominated by the upper bound on the regret itself. Thus, given our result from Theorem 1 that Regret( T ) ≤ ˜ O ( √ SAT (1 -γ ) 3 ) , we can conclude Regret NS ( T ) ≤ ˜ O ( √ SAT (1 -γ ) 3 ) for our algorithm Q-SlowSwitch-Adv . In fact, this conversion holds as long as the algorithm's switching cost is dominated by the regret itself, e.g., when the switching cost is o ( √ T ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a nearly optimal model-free algorithm in discounted tabular MDPs which has a shorter burn-in time than previous works and provides valid theoretical guarantees.\n\nSTRENGTHS:\nThe paper is well motivated, trying to shorten the burn-in time and obtain a more sample-efficient model-free tabular MDP algorithm.\n\nThe writing is awesome. The detailed comparison with previous algorithms is highly appreciated.  \n\nThe proof seems correct with many previous techniques organically integrated together. I must say the author should be an expert on the related techniques and proof arguments.\n\nTo achieve a tighter burn-in time, it seems vital to update the reference advantage at an appropriate frequency and learn the Q-value of the current policy more accurately. To tackle the issue, the author proposes two new components. The first is the lazy update for the Q-value function which the greedy policy is generated; the other is an adaptive low-switching mechanism to determine what is the time to perform such an update. Both components might inspire future works.\n\nWEAKNESSES:\nIf I must point out some weaknesses, I think that the proof contains too many instances where the author didn’t include brackets when utilizing the summation symbol $\\sum$, such as in (46)-(48). \n\nThe frequency of these omissions is quite high, making it difficult to enumerate them here.  I think it is important that all brackets are appropriately included for clarity and correctness.\n\nI have some questions about the results and the theoretical analysis.\n\nQUESTIONS:\n1. The author discusses the impact of large state-action spaces on burn-in time several times. It is reasonable to expect that tabular MDPs with large state-action spaces are generally more challenging to learn, leading to longer burn-in times. However, the paper lacks a lower bound for the minimum burn-in time, although it does match the well-known lower bound on the dominant term (or variance term). It would be valuable for the author to address this gap and provide insights into the possible smallest burn-in time.\n\n2. Lemma 10 presents a rough estimate for $\\sum_{t=1}^T (V_{t-1}\\left(s_t\\right)-V^{\\pi_t}\\left(s_t\\right))$, which is subsequently utilized multiple times in the proof of Lemma 7 (e.g., in (153), (228), (229)). Instead of using this crude bound repeatedly, it seems to be more advantageous to bound the regret without this rough estimate. I mean to maintain the dependence of $\\sum_{t=1}^T (V_{t-1}\\left(s_t\\right)-V^{\\pi_t}\\left(s_t\\right))$ while analyzing intermediate terms in Lemma 4 and then to use Proposition 1 to disentangle this dependence at the end. By using this alternative argument, it may be possible to eliminate the need for the crude bound entirely. Additionally, it is worth investigating if this alternative argument can enhance the dependence of $(1-\\gamma)^{-1}$ on the burn-in time.\n\n3. It is notable that the regret of UCBVI-$\\gamma$ exhibits a more favorable dependence on $(1-\\gamma)^{-1}$ compared to the algorithm proposed in this work. It would greatly enhance the paper if the author could provide a discussion on the reasons behind this superiority. Understanding the factors contributing to better dependence would provide valuable insights into the comparative performance of the two algorithms and contribute to a deeper understanding of their respective strengths and weaknesses.\n\nLIMITATIONS:\nPlease see the questinos.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied infinite discounted tabular Markov decision process (MDP). The authors proposed a model-free algorithm: Q-SlowSwitch-Adv that achieves optimal regret with low burn-in cost, low space complexity and low computational cost. They introduce an innovative technique that changes the execution policy in a slow, adaptive manner, and employ variance reduction to enhance efficiency.\n\nSTRENGTHS:\n1. The major novelty of this work lies in the solution to achieve optimal regret using model-free algorithm for discounted tabular MDP.\n2. The property of the low burn-in cost seems interesting and I believe that the low computation cost and low space complexity could be good contributions, enhancing the feasibility of implementing reinforcement learning in real-world scenarios.\n3. The design of a slow, adaptive execution policy-switching technique is novel.\n\nWEAKNESSES:\n1. The paper lacks empirical evidence to substantiate the theoretical claims and the efficacy of the proposed algorithm, especially for the 'low' burn-in cost.\n2. The reason why the proposed algorithm can reduce the burn-in cost is not clear, please see the Question part for more details.\n\nQUESTIONS:\nCould the authors elaborate on the methodology used to reduce the burn-in cost? Specifically, is this reduction achieved through refined theoretical analysis, novel algorithm design, or a combination of both?\n\nLIMITATIONS:\n1. The paper's main limitation is the lack of empirical evidence to support the proposed method. Performance comparisons against classic algorithms would have made the theoretical claims more credible.\n\n2. Given the high level of complexity involved in real-world RL problems, the generalizability of the proposed method is not addressed. Therefore, it is unclear how the proposed algorithm will perform in more complex scenarios such as function approximation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies discounted infinite-horizon MDPs. The authors propose an online algorithm that attains optimal (i.e., least) regret with finite-sample performance guarantees.

STRENGTHS:
Optimal regret & finite-sample performance guarantees

WEAKNESSES:
Lack of numerical illustration, but not a deal-break.

QUESTIONS:
1. Some numerical experiments in the main body would be great.
2. Assumptions on the probability distribution for the algorithm to achieve the optimal regret and finite-sample performance guarantees should be made explicitly and crystal clear.

LIMITATIONS:
Lack of numerical illustration, but not a deal-break.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a version of Q-learning algorithm with Bernstein bonuses and control variate that allows or the correct (in terms of the dependence upon state and action space sizes $S$ and $A$) second-order term in the regret bound. This allows for an algorithm with the provably shorter burn-in period (again in terms of $S$ and $A$). This result for discounted MDPs is, to the best of my knowledge, novel.\n\nSTRENGTHS:\nThe paper presents a valuable contribution to the theory of model-free algorithms for solving MDPs with finite state- action spaces. The suggested idea of incorporating Berstein style bonuses with the UCB-advantage type of control variates is appealing and novel in the discounted MDP setting.\n\nWEAKNESSES:\nActually, I would also like to ask the authors why not considering submitting to a journal. The proof appendix is massive and it is obviously not possible to check all the technical details properly due to a very limited time for reviewing period.\n\nMinor issues and comments:\n1. The example concerning the Go game (lines 94-98) is spectacular, yet with $|S| = 3^{161}$ the linear dependence on $S$ is clearly as bad as $S^2$. I understand the author's willingness to explain the motivation beyond improving the dependence of the second-order term upon $S$ and $A#, yet I suggest to come up with a slightly less ambitious example;\n2. I would also suggest removing \"outstanding\" in line 108: it is clearly a matter of taste;\n\nQUESTIONS:\nI suggest the authors to add explicitly a paragraph when discussing the techniqus currently used to obtain the second-order term in the regret whch is optimal either in terms of episode length $H$ (resp. $1-\\gamma$ factor), or $S$ and $A$ factors. I would also suggest to add explicit comments on why the current technique in the paper is not suitable to get the second-order term, which is optimal also in terms of the $1-\\gamma$ factor. With this discussion added, I can increase my score by $1$ point.\n\nLIMITATIONS:\nThe paper is theoretical and I do not expect any negative societal impact of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies online learning in tabular infinite horizon MDPs, and proposes a model free algorithm that obtains (nearly) minimax optimal regret with space complexity $SA$, and such that the number of time steps needed to reach the minimax-optimality regime is $O(SA/(1-\\gamma)^{13})$.\nPrior art achieving minimax optimal regret required space complexity $S^2 A$ and $O(S^3A^2/(1-\\gamma)^{4})$ time steps to reach the optimal regime.\nIt should be noted the definition of regret used here is not the same as that of prior art.\n\nSTRENGTHS:\n* The paper proposes a new approach to learning infinite horizon MDPs.\n* The paper (perhaps) advances state of the art in online learning infinite horizon tabular MDPs.\n\nWEAKNESSES:\n- It is hard to say this work directly improves [14] because the objective considered is not the same. It would be helpful if a more formal discussion of this point was provided by the authors. For the same reason it is also unclear if the lower bound (also from [14]) applies. Given, this, the way the results are presented in the table are misleading.\n- The improvement is rather incremental, shaving off a factor of $O(SA)$ (at the cost of an extra $(1-\\gamma)^4$) from the lower order (non $T$ dependent) term. In addition, the algorithm is complicated and presentation does not make it particularly easy to understand.\n\nQUESTIONS:\n- Is there a more formal treatment of the two regret definitions? It would be helpful to make a more educated decision here; the improvement would have been marginal even if the objectives considered were the same, and when taking into account it may not even be an actual improvement --- the contribution seems questionable.\n- It seems like the approach given here borrowed most of the algorithm from the finite horizon case [54] and then employs a slow switching strategy to essentially reduce the infinite horizon problem to a finite horizon one, thereby gaining in the $SA$ factors but loosing in the horizon. This also seems to be the source of the different regret definition. The presentation does not make this entirely clear however, is this more or less the case?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}