KL-UCRL Revisited : Variance-Aware Regret Bound ∗

The problem of reinforcement learning in an unknown and discrete Markov Decision Process (MDP) under the average-reward criterion is considered, when the learner interacts with the system in a single stream of observations, starting from an initial state without any reset. We provide a novel analysis of the KL-Ucrl algorithm establishing a high-probability regret bound scaling as Õ (√ S ∑ s,a Vp(·|s,a)(b)T ) for this algorithm for ergodic MDPs, where S denotes the number of states and where Vp(·|s,a)(b) is the variance of the bias function of an optimal policy, with respect to the next-state distribution following action a in state s. The resulting bound improves upon the best previously known regret bound Õ(DS √ AT ) for that algorithm, where A and D respectively denote the maximum number of actions (per state) and the diameter of MDP. We finally compare the leading terms of the two bounds in some benchmark MDPs indicating that the derived bound can provide an order of magnitude improvement in some cases. Our analysis leverages novel variations of the transportation lemma combined with Kullback-Leibler concentration inequalities, that we believe to be of independent interest. Keywords: Undiscounted Reinforcement Learning, Markov Decision Processes, Concentration Inequalities, Regret Minimization, KL-UCRL 1. Introduction In this paper, we consider Reinforcement Learning (RL) in an unknown environment modeled by a discrete Markov Decision Process (MDP). The learner interacts with the system in a single stream of observations, starting from an initial state without any reset, and wishes to maximize the long term average-reward. More formally, let M = (S,A, ν, P ) denote an MDP where S and A respectively denote state-space and action-space (at any state), with respective cardinalities S and A. Furthermore, ν and P denote the reward function and the transition kernel, respectively. At time t = 1, the learner starts in some state s1 ∈ S. At each time step t ∈ N, she chooses one action a ∈ A in her current state s ∈ S based on her past decisions and observations. When executing action a in state s, she receives a random reward r drawn independently from distribution ν(s, a) with support [0, 1] and mean μ(s, a). The state then transits to a next state s′ ∈ S sampled with probability p(s′|s, a), and a new decision step begins. As the transition probabilities and reward functions are unknown, the learner needs to learn them by trying different actions and recording the realized rewards *. This paper is based on (Talebi and Maillard, 2018). . The authors contributed equally. The authors contributed equally. ©2018 M. Sadegh Talebi and Odalric-Ambrym Maillard. License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. and state transitions. We refer to (Sutton and Barto, 1998; Puterman, 2014) for background material on RL and MDPs. The performance of the learner can be quantified through the notion of regret, which compares the reward collected by the learner (or the algorithm) to that obtained by an oracle always following an optimal policy, where a policy π : S → A is a mapping from states to actions. Following (Jaksch et al., 2010), we consider the following definition of regret: