Vision-Language Models as a Source of Rewards
Authors: name: Kate Baumli, name: Satinder Baveja, name: Feryal Behbahani, name: Harris Chan, name: Gheorghe Comanici, name: Sebastian Flennerhag, name: Maxime Gazeau, name: Kristian Holsheimer, name: Dan Horgan, name: Michael Laskin, name: Clare Lyle, name: Hussain Masoom, name: Kay McKinney, name: Volodymyr Mnih, name: Alexander Neitz, name: Fabio Pardo, name: Jack Parker-Holder, name: John Quan, name: Tim Rocktäschel, name: Himanshu Sahni, name: Tom Schaul, name: Yannick Schroecker, name: Stephen Spencer, name: Richie Steigerwald, name: Luyu Wang, name: Lei Zhang

Building generalist agents that can accomplish many goals in rich open-ended
environments is one of the research frontiers for reinforcement learning. A key
limiting factor for building generalist agents with RL has been the need for a
large number of reward functions for achieving different goals. We investigate
the feasibility of using off-the-shelf vision-language models, or VLMs, as
sources of rewards for reinforcement learning agents. We show how rewards for
visual achievement of a variety of language goals can be derived from the CLIP
family of models, and used to train RL agents that can achieve a variety of
language goals. We showcase this approach in two distinct visual domains and
present a scaling trend showing how larger VLMs lead to more accurate rewards
for visual goal achievement, which in turn produces more capable RL agents.