Vision-Language Models as a Source of Rewards
Authors: name: Kate Baumli; name: Satinder Baveja; name: Feryal Behbahani; name: Harris Chan; name: Gheorghe Comanici; name: Sebastian Flennerhag; name: Maxime Gazeau; name: Kristian Holsheimer; name: Dan Horgan; name: Michael Laskin; name: Clare Lyle; name: Hussain Masoom; name: Kay McKinney; name: Volodymyr Mnih; name: Alexander Neitz; name: Fabio Pardo; name: Jack Parker-Holder; name: John Quan; name: Tim Rocktäschel; name: Himanshu Sahni; name: Tom Schaul; name: Yannick Schroecker; name: Stephen Spencer; name: Richie Steigerwald; name: Luyu Wang; name: Lei Zhang
Published: 2023-12-14 18:06:17

Abstract: Building generalist agents that can accomplish many goals in rich open-ended
environments is one of the research frontiers for reinforcement learning. A key
limiting factor for building generalist agents with RL has been the need for a
large number of reward functions for achieving different goals. We investigate
the feasibility of using off-the-shelf vision-language models, or VLMs, as
sources of rewards for reinforcement learning agents. We show how rewards for
visual achievement of a variety of language goals can be derived from the CLIP
family of models, and used to train RL agents that can achieve a variety of
language goals. We showcase this approach in two distinct visual domains and
present a scaling trend showing how larger VLMs lead to more accurate rewards
for visual goal achievement, which in turn produces more capable RL agents.