Zero-shot Model-based Reinforcement Learning using Large Language Models
Authors: name: Abdelhakim Benechehab; name: Youssef Attia El Hili; name: Ambroise Odonnat; name: Oussama Zekri; name: Albert Thomas; name: Giuseppe Paolo; name: Maurizio Filippone; name: Ievgen Redko; name: Balázs Kégl
Published: 2024-10-15 15:46:53

Abstract: The emerging zero-shot capabilities of Large Language Models (LLMs) have led
to their applications in areas extending well beyond natural language
processing tasks. In reinforcement learning, while LLMs have been extensively
used in text-based environments, their integration with continuous state spaces
remains understudied. In this paper, we investigate how pre-trained LLMs can be
leveraged to predict in context the dynamics of continuous Markov decision
processes. We identify handling multivariate data and incorporating the control
signal as key challenges that limit the potential of LLMs' deployment in this
setup and propose Disentangled In-Context Learning (DICL) to address them. We
present proof-of-concept applications in two reinforcement learning settings:
model-based policy evaluation and data-augmented off-policy reinforcement
learning, supported by theoretical analysis of the proposed methods. Our
experiments further demonstrate that our approach produces well-calibrated
uncertainty estimates. We release the code at
https://github.com/abenechehab/dicl.