ICML 2025

Training a Generally Curious Agent
Fahim Tajwar * 1 Yiding Jiang * 1 Abitha Thankaraj 1 Sumaita Sadia Rahman 2 J Zico Kolter 1 Jeff Schneider 1
Russ Salakhutdinov 1
Abstract
Efficient exploration is essential for intelligent
systems interacting with their environment, but
existing language models often fall short in sce-
narios that require strategic information gathering.
In this paper, we present PAPRIKA, a fine-tuning
approach that enables language models to develop
general decision-making capabilities that are not
confined to particular environments. By training
on synthetic interaction data from different tasks
that require diverse strategies, PAPRIKA teaches
models to explore and adapt their behavior on
a new task based on environment feedback in-
context without more gradient updates. Experi-
mental results show that models fine-tuned with
PAPRIKA can effectively transfer their learned
decision-making capabilities to entirely unseen
tasks without additional training. Unlike tradi-
tional training, our approach’s primary bottleneck
lies in sampling useful interaction data instead
of model updates. To improve sample efficiency,
we propose a curriculum learning strategy that
prioritizes sampling trajectories from tasks with
high learning potential. These results suggest
a promising path towards AI systems that can
autonomously solve novel sequential decision-
making problems that require interactions with
the external world.
1. Introduction
Large language models (LLMs) are considered to be a
promising foundation for autonomous agents, systems capa-
ble of achieving goals independently with minimal human
supervision or intervention. A crucial requirement for such
systems is the ability to interact effectively with external en-
vironments and gather the information necessary to achieve
their objectives. This capability can be formalized as solving
*Equal contribution
1CMU 2North Carolina State University.
Correspondence to: Fahim Tajwar <ftajwar@cs.cmu.edu>.
Proce...