Title: Meta-RL Induces Exploration in Language Agents
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Model, Agent, Reinforcement Learning, Meta Learning

Abstract: Reinforcement learning (RL) has enabled the training of Large Language Model (LLM) agents to interact with the environment and to solve multi-turn longhorizon tasks. However, the RL-trained agents often struggle in tasks that require active exploration and fail to efficiently adapt from trial-and-error experiences. In this paper, we present LAMER, a general Meta-RL framework that enables LLM
agents to actively explore and learn from the environment feedback at test time. LAMER consists of two key components: (i) a cross-episode training framework to encourage exploration and long term rewards optimization; and (ii) in-context policy adaptation via reflection, allowing the agent to adapt their policy from task feedback signal without gradient update. Experiments across four different environments demonstrate that LAMER significantly improves performance over RL baselines, with more than 13% gains on Sokoban and more than 20% gains on MineSweeper and Webshop. It also generalizes better when evaluated on more challenging or previously unseen environments compared to the RL trained models. Overall, our results demonstrate that meta-reinforcement learning provides a principled approach to induce exploration in language agents, enabling more robust adaptation to novel environments through learned exploration strategies.