Title: Policy Optimization with $f$-Divergence Regularization
Primary area: reinforcement learning
Keywords: Reinforcement Learning, f-divergence, policy optimization

Abstract: Policy iteration is a common algorithm framework in reinforcement learning (RL) to find the optimal policy for a Markov decision process (MDP). To improve training stability and prevent catastrophic failure, researchers have developed several policy iteration algorithms based on the Kullback-Leibler (KL) divergence, such as the well-known trust region policy optimization (TRPO) and proximal policy optimization (PPO). However, these methods are limited to the KL divergence, which may not be the best choice for all environments. In this work, we generalize previous work using a more general form of divergence, the $f$-divergence, and design a new family of algorithms that can improve learning policy with theoretical improvement guarantees. Our method, $f$-divergence-regularized policy optimization ($f$RPO), can be applied to both online and offline RL settings. Empirical studies show that $f$RPO can outperform existing methods, including the commonly used KL divergence, on common benchmark problems in RL.

TL;DR: We develop an iterative policy optimization algorithm using f-divergence regularization, with a monotonic improvement guarantee and competitive results in both online and offline settings.