Title: Generative Actor Critic
Primary area: reinforcement learning
Keywords: Sequential Decision Making, Generative Model, Reinforcment Learninig

Abstract: Conventional Reinforcement Learning (RL) algorithms, typically focused on estimating or maximizing expected returns, face challenges when refining offline pretrained models with online experiences. This paper introduces Generative Actor Critic (GAC), a novel framework that decouples sequential decision-making by reframing policy evaluation as learning a generative model of the joint distribution over trajectories and returns, $p(\tau, y)$, and policy improvement as performing versatile inference on this learned model. To operationalize GAC, we introduce a specific instantiation based on a latent variable model that features continuous latent plan vectors. We develop novel inference strategies for both exploitation, by optimizing latent plans to maximize expected returns, and exploration, by sampling latent plans conditioned on dynamically adjusted target returns. Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods, even in absence of step-wise rewards.

TL;DR: Generative Actor Critic decouples decision-making by learning a generative model over trajectories and returns, then uses versatile inference on that model for policy improvement.