Regret minimization of Tabular Policy Gradient ∗

Reinforcement Learning (RL) learns to control a complex, unknown environment through interaction. RL algorithms were highly successfully applied to various domains. Policy gradient methods optimize directly a parameterized policy by computing gradients of the value function and updating the parameters. Policy gradient methods have been highly successful in many applications areas (such as playing GO [Silver et al. 2016] and robotics [Deisenroth et al. 2013]) and are among the most effective methods in Reinforcement Learning. The policy gradient theorem [Sutton et al. 1999] gives a method of computing the gradient as a function of the observable quantities in the MDP. Using Monte-Carlo methods and Reinforce Williams (1992) or actor-critic methodology give popular implementations of the policy gradient methodology. However, those fundamental results do not derive convergence bounds or performance guarantees. Our starting point is the work of Agarwal et al. (2020b) which studies the number of gradient steps required to reach an ε approximation of the optimal value function, assuming we receive exact gradients. The main contribution of our work is to avoid the assumption of exact gradients, and directly approximate the gradients from observations and in addition derive vanishing regret bounds. More specifically, this work focuses on finite horizon Markov Decision Process (MDP) with tabular policy parameterizations, i.e., there is a parameter per state-action pair. Our policy gradient algorithms approximate the policy gradient using episodes sampled from the MDP, i.e., we do not assume direct access to exact policy gradient. We analyze the algorithms with respect to the regret, i.e., the difference between the expected return of the optimal policy and the expected return of the policies chosen by the online algorithm (which is running a policy gradient algorithm). We stress that our focus is on understanding the performance of the widely used policy gradient methodology and not on deriving new algorithms or new performance guarantees. (See the related work section for a variety of regret minimization algorithms.) We consider two parameterized policy classes (similar to Agarwal et al. (2020b)). The first, direct parameterization, simply encodes the policy using a lookup table. The second, softmax, encodes a weight per state-action, and selects an action using a softmax function of the weights. At a high level, our algorithms work in phases, where during each phase they sample episodes using the current policy, and at the end of the phase compute an approximation of the policy gradient and update the policy. In the direct parameterization for every state-action pair s, a the parameter θs,a is the probability to choose action a in state s. In every phase we perform a policy gradient update of the parameters, and project the new parameters to the policy simplex. For the direct parameterization we show a regret bound of Õ(K 5 6H 7 6 |S| 23 |A| 43D),1 where K is the total number of episodes, S and A are the set of states and actions, H is the horizon, and we have an MDP parameter D which we will define later.