VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning.A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agents uncertainty about the environment.Computing a Bayes-optimal policy is however intractable for all but the smallest tasks.In this paper, we introduce variational Bayes-Adaptive Deep RL, a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection.In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty.We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.
VariBAD opens a path to tractable approximate Bayes-optimal exploration for deep RL using ideas from meta-learning, Bayesian RL, and approximate variational inference.This paper presents a new deep reinforcement learning method that can efficiently trade-off exploration and exploitation that combines meta-learning, variational inference, and bayesian RL.