Training a Generally Curious Agent
Authors: Fahim Tajwar
Yiding Jiang
Abitha Thankaraj
Sumaita Sadia Rahman
J Zico Kolter
Jeff Schneider
Russ Salakhutdinov
Venue: ICML 2025 oral (2025)
Keywords: LLM Agent
Synethic Data
Multiturn finetuning
Primary area: deep_learning->large_language_models
TL;DR: Method for training on synthetic data to improve LLMs' sequential decision making capabilities

Abstract: Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present **Paprika**, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, Paprika teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with Paprika can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.

Training a Generally Curious Agent
Fahim Tajwar"
Yiding Jiang"
Abitha Thankaraj
Sumaita Sadia Rahman 2 Zico Kolter
Jeff Schneider'
Russ Salakhutdinoy
Abstract
sequential decision-making problems or performing rein- forcement learning (RL) with language models the agent. However, two challenges hinder the development of these interactive capabilities. First, most naturally occurring data lacks the structure and context needed to model interactions. Second, directly deploying models into the real world to collect interaction data can produce critical errors, which is expensive and potentially risky.
Efficient exploration essential for intelligent systems interacting with their environment, but existing language models often fall short in sce- narios that require strategic information gathering. In this paper, we present PAPRIKA, A, a fine-tuning approach that enables language models to develop general seeision-mmking capabilities that are not confined to particular environments. By training synthetic interaction data from different tasks that require diverse strategies, PAPRIKA teaches models to explore and adapt their behavior on a new task based on environment feedback in- context without more gradient updates. Experi- mental results show that models fine-tuned with PAPRIKA can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike tradi- tional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest promising path towards AI systems that autonomously solve novel sequential decision- making problems that require interactions with the external world.
Given the impracticality of direct deployment in the wild, natural alternative is generate interaction data syntheti- cally. Although generating synthetic data for every possible problem is infeasible, LLMs possess the capacity for in- context learning (ICL), which allows them to adapt to new tasks with minimal demonstrations (Brown et al., 2020). Instead of teaching the model to do all the interaction tasks that we care about, we should instead teach the model in- context reinforcement learning (Laskin et al., 2022) that the model solve problems without being trained on them priori. It shifts the focus from training the model on particular problems to training it on the general process of solving problems. This paradigm shares similarities with the supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages of training language model (vs pretraining) where only relatively small number of examples is needed to produce model that can generate responses to wide range of queries that they are not trained on. Our approach is also closely related to the principles of meta reinforcement learning (Beck et al., 2023).
In this work, we explore the feasibility of teaching LLMs to perform in-context RL that generalizes across different tasks, with the specific goal of training curious agent with general information gathering capability. A popular notion of curiosity is intrinsic motivation which has been used to train agents with an exploration bonus not necessarily related to the success of any particular task (Schmidhuber, 1991; 2007). Our work differs from this notion of curiosity in that we do not leverage intrinsic motivation. Instead, we train our agents explore and interact with an entirely unseen environment to gather information that is needed for completing the task at hand. PAPRIKA can be thought of as form of amortized exploration, since our goal is to learn good exploration strategies from trajectories from many different environments to make exploration on a new
1. Introduction
Large language models (LLMs) are considered to be promising foundation for autonomous agents, systems capa- ble of achieving goals independently with minimal human supervision or intervention. A crucial requirement for such systems is the ability to interact effectively with external en- vironments and gather the information necessary achieve their objectives. This capability can be formalized as solving
"qqual contribution 'CMU NNotth Carolina State University. Correspondence to: Fahim Tajwar < j warres. cmu, edu>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Training Generally Curious Agent
20 Questions
Customer Service
Wordle
You are slayingg the game 20 questions
moved a new (Face and my will is working
You are playtig wordie.
(-)
Is Fre lopic an arimal?
BUR
Could you tall me you! address?
Crane
No.
42 Binary Riuulmandt Crrcut City APT 9814
Construct preference pairs
Is the 10pkc o persun??
PUP
What &s the sorial number your router?
Bcarss
perform policy optimization
Training Phase
Test Phase
Figure 1. (Overview of PAPRIKA) We design a diverse set of tasks where LLM agent needs strategic information gathering to succeed, then train an LLM on self-generated data to prefer higher performing trajectories. The resulting behavior learned by PAPRIKA can transfer zero-shot to unseen tasks, showcasing its potential build general decision making agents.
problem more efficient (see Appendix A for more details).
As a concrete example, guessing the word "apple" would be a task in 20 questions. We will use group (or task group, used interchangeably). G = {TI: :2....; J. ). to refer to a high-level grouping of different tasks (e.g., the game 20 questions would be a group). Tasks in group should share similar strategies but it is not always true that they share the same optimal policy as such constraints may be overly stringent. From the agent's perspective, each task is a black box function that takes in the agent's 's action de (and possibly the whole interaction history) and outputs an observation Ot- Both a and Og are strings. In a game of 20 questions, a could be "Is the word an animal? ?" and the o could be "No." ". In other words, each task employs envi- ronment that the agent interacts with to obtain intermediate observations.
We begin by designing a diverse suite of textual decision- making tasks that require active information gathering and drcision-making based on interaction outcomes. Using a base model, we generate interaction trajectories and sign scores based on their success in achieving the tasks' objectives. We then apply a sequential variant of Direct Preference Optimization (Rafailov et al., 2024b, DPO) to increase the relative likelihood of successful trajectories. Unlike traditional training where computational costs are dominated by model updates, our approach's primary bot- tleneck lies in sampling useful interaction data. To improve sample efficiency, propose curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential.
An episode contains the agent's interaction trajectory within a single task. Unlike the conventional RL structure, we will assume that the transition-level reward either 0 or must be inferred from ot, and that the individual tasks can flexibly implement different observation spaces and termi- nation conditions. An episode terminates when the agent achieves the objective of the task or when the maximum number of interactions allowed within the task is reached.
We refer to the overall framework as PAPRIKA Our results demonstrate that training on different subsets of these tasks improves the performance of the model on unseen tasks. More broadly, our result highlights the potential of using synthetic data to learn in-context RL which would equip LLMs with the capability to interact with the world and solve different making problems without requiring task-specific fine-tuning.
We will use = (0o, ap...., OH QH) to denote an episode of length H, hr, = (or, ax) to denote a single step of h, and hapg = (op, ap..... .g. ag) denote a slice of h sim- ilar to array slicing. At the end of episode, the envi- ronment emits single score, r(h), that evaluates the per- formance of the agent. Let π denote the LLM agent and h ~ π o T denote sampling a trajectory from task using policy π. The performance of a policy on a group would be: Perf(G) = G Σreg [r))oo(((h))] The agent is trained on a finite set of groups, Guain and the goal is to perform well on unseen groups, Great
2. Preliminary
Many decision making problems can be formalized as partially observable Markov decision process (POMDP). We assume each task, is a POMDP although we will not draw on the details of the POMDP formalism in this work.
'The name inspired by the movie "Paprika" (2006), where dream detective navigates vast and strange dream worlds to solve different mysteries.
2

---

Training a Generally Curious Agent
Table Summary of the task groups used by PAPRIKA.
Task Group
# Train Tasks
# Test Tasks
Maximum Turns
Env Feedback
Uses COT
Twenty questions
1499
367
20
LLM generated
Guess my city
500
185
20
LLM generated
Wordle
1515
800
6
Hardcoded program
Cellular automata
1000
500
6
Hardcoded program
Customer service
628
200
20
LLM generated
X
Murder mystery
203
50
20
LLM generated
Mastermind
1000
500
12
Hardcoded program
Battleship
1000
200
20
Hardcoded program
Minesweeper
1000
200
20
Hardcoded program
Bandit best arm selection
81
21
Hardcoded program
3. PAPRIKA
prior history in its context and choose actions that maximize the probability of success in the future, (3) they are partially observable, i.e., the observations do not capture the full state hidden information, so the agents must simultaneously explore to reveal more information and exploit to solve the task efficiently, (4) they are diverse and require different strategies to succeed.
The goal of our paper is to develop a scalable method to instill better strategic exploration and sequential decision- making capabilities into L.LMs. Prior works (Krishnamurthy et al., 2024) have shown that LLMs can perform poorly on even the simple decision making task of multi-armed bandits. Nie et al. (2024) has since then demonstrated that LLMS can be taught to perform better on bandits after fine- tuning them on synthetic trajectories generated by known algorithms such as UCB. However, this idea is limited in scope for three reasons: (1) we want LLMs to perform strategic exploration and decision making in more complex settings, (2) for most tasks, there is no known algorithm like UCB to generate good synthetic trajectories from, (3) it can be infeasible to collect data for all tasks that we care about.
With these requirements in mind, we design 10 task groups in our paper. On all of them, we employ an LLM as the agent that is given a task it needs to solve through sequen- tial interaction with the task-specific environment, which provides both observations for intermediate timesteps given the agent's 's actions and also a task reward at the end of an episode. For tasks requiring general knowledge about the world to generate intermediate observations, we employ an- other LLM (typically GPT-4o-mini) as the environment. For tasks that have rule-based observations and rewards, we find that using hardcoded programs as the verifiet/observation generator is more reliable than LLMs, similar to DeepSeek- AI et al. (2025). In order to prevent reward hacking, we also use either another L.M or a hardcoded program as a judge to filter out unsuccessful trajectories that got incor- rectly labeled as successful by the task environment (see Appendix D for more on environment hacking). We also find that for task groups requiring complex reasoning, letting the agent think using chain-of-thought (COT) prompting (Wei et al., 2022; Kojima et al., 2022) before generating a final answer improves its performance significantly, similar to ReAct (Yao et al., 2023). We provide a brief description of our task groups here, please refer to Table for their summary and Appendix B for more details.
We aim to solve these issues using our method, PAPRIKA. A. First, we design a suite of complex decision- making tasks that require strategic information gathering to succeed. Next, we show that in the absence of known good algorithms, existing L.MSs can generate trajectories with better decision sampling.
making behaviors through diversity-encouragee We then finetune the LLMs to prefer higher performing trajectories (in a fashion similar to STaR (Zelikman et al., 2022)) and show that this leads to better decision making abilities at test-time. More importantly, these behaviors often generalize to unseen task groups without additional training. Finally, we propose a general curriculum learning algorithm that can dynamically choose which subset of tasks to train on next to improve data efficiency of such training methods. We next describe each component of PAPRIKA.
3.1. Task Design
Following prior work (Abdulhai et al., 2023), we include classic guessing games like twenty questions and guess my city in our list of task groups They require guessing a secret topic as quickly as possible by asking a sequence of ques- tions and observing the answers. We also employ Wordle and Mastermind, where the agent needs to guess secret letter word and 4-digit code respectively. The environments
The first component of PAPRIKA is to design set of task groups that we can evaluate and train LLMs on. The task groups we want should have the following desired proper- ties: (1) they are purely text based, (2) they require multi- turn interaction, where the agents have to both understand
3

---

Training a Generally Curious Agent
for these task groups provide feedback in terms of similarity between the guess and the target word/code, and the agent needs to refine their guesses in future turns to maximize information gathering. We design customer service and murder mystery as dynamic text-based task groups: an L.LM plays the role of the task environment, which is provided with the criterion for task success and generates dynamic intermediate observations based on this criterion.
exploration such as Murty et al. (2024); Yang al. (2024); however, we opt for sampling with high temperature for its simplicity and leave these other options for future work.
For each task in a set of chosen tasks (e.g., uniformly sam- pled), we generate naample trajectories and then construct a preference pair (Aeu, ht) where huw is the highest scoring trajectory (trajectory that succeeds and does so at the fewest number of turns) and f is randomly sampled from the lower scoring (failed takes substantially more turns succeed) trajectories. We choose Iy randomly instead of choosing the worst one to increase the diversity of our dataset. We treat how and hy as proxies for desirable and undesirable
A desirable capability in LLMs is to code and refine based on interpreter feedback. To simulate this process with toy case, we design Cellular Automata, where the agent needs to make inferences about the transition rule in ID elementary cellular automata (Wolfram, 1983; Cook et al., 2004) by observing inputs and outputs. The agent receives the outputs generated from their predicted transition rule and they have to refine their predictions based on it. Next, we incorporate Minesweeper and Battleship based on classical games, which require the agent to interact with 2D grids to find hidden items within a fixed number of turns and refine their guesses based on per-turn observations.
behaviors. A dataset D { (h hh (6) of such trajectory pairs.
Jimi is collection
3.3. Optimization
Supervised fine-tuning. If we take the winning episodes as the expert behavior, then we can discard the losing episode and maximize the likelihood of winning episodes:
Finally, we incorporate a modified version of the multi- armed bandit (Slivkins, 2024) task group from prior works (Krishnamurthyy et al., 2024; Nie et al., 2024) with the following distinctions: (1) we let the agent employ chain- of of-thought reasoning before choosing arms so that they can transfer good strategies learned from other tasks, (2) we let the agent interact with the task environment in a multiturn way, (3) instead of reducing regret, we work on the bandit best arm selection (Audibert & Bubeck, 2010: Wang et al., 2024a) problem, where we let the agent choose arms and ob- serve rewards for a fixed number of turns and then measure its accuracy in deciding the arm with the highest reward. This is done to reduce computational cost over generating COTs for a large number of turns, since the difference in regret between different models is not meaningful when the number of turns is not large enough.
Phwl log no (a
(DFT()))T) -Eper
Σ
| t==
t=
where DSFT is the dataset used for supervised fine-tuning and |a] is the number of tokens for the agent response (dis- carding the environment generation). This is akin to rejec- tion sampling fine-tuning (Gulcehre et al., 2023; Dong et al., 2023; Mukobi et al., 2023) seen in prior work.
Direct preference optimization. A popular approach for finetuning LLMs is DPO (Rafailov et al., 2024b) where one directly optimizes the Bradley-Terry model (Bradley & Terry, 1952) for preferences. In our setting, each trajectory consists of multiple rounds of interactions so the original DPO objective does not apply. We instead use a multi-turn version of DPO introduced in Rafailov et al. (2024a):
3.2. Dataset construction
In order to learn from these task groups, we must first gen- erate data from them. is crucial that the data we generate diverse which would allow the model to learn differ- ent strategies without the risk of overfitting. We accom- plish this by generating a large number of trajectories at a high temperature with Min-p sampling (Nguyen et al., 2024). Min-p sampling works by using an adaptive thresh- old Psealed x Pmax. where Pmax is the highest probability predicted by the model on the next token, to truncate the vo- cabulary tokens that have a probability larger than Psealed and sample from them this enables to generate di- verse yet coherent trajectories at higher temperature. We note that training data generation for PAPRIKA could be improved by adopting more advanced methods for guiding
To(ap h:)
(Dpo(DDoo) = -EDoro
log o
Σ
B log
Trer (a h.))
To(at h//))
3 log
(2)
mer hir))
where DDPO is the preference dataset, ax and at are the ac- tion tokens generated by the model at turn in the preferred and dispreferred trajectories, hTd and h², respectively. गee is the reference policy, for which we use the initial model. The main difference with standard DPO here is that we only
4

---

Training a Generally Curious Agent
calculate the loss on the action tokens the log probability ratios of the environment generated tokens not included in the loss.
Algorithm 1 Task selection with UCB
1: Input: Number of arms K, number of samples C, num- ber of rounds T, model æ
2: Initialize: 8& = 0, TM = 0, Buffer
We note that we use DPO because it is less compute inten-
3: for each round t 1, T do A
sive. DPO allows us to decouple the data collection and policy improvement steps and offload them on different ma- chines. However, in principle, one could also employ online RL with more resources. Following prior work that shows the efficacy of online RL compared to offline algorithms (Xu et al., 2024; Tajwar et al., 2024), we expect doing PAPRIKA with online RL would lead to even stronger results.
2 log E dh for each At
Compute 0g
m
n
5: Select k- arg max O2
6: Sample 7 from group h**
7: Sample C' trajectories from 7 and add to Buf fer
8: Compute an estimate for 2/ (T) using l5q 4
9: Update: $g.- = Sk- + (T), 7a.- = n2 + 1
10: end for
Combining objectives. Finally, prior works have noted DPO having the unintended effect of reducing the probabil- ity of preferred trajectories as well, known as unintentional unalignment (Razin et al., 2024), which can affect model performance. The RPO objective (Pang et al., 2024), by combining SFT and DPO loss, has shown promising results in mitigating this issue. Formally, the RPO loss is:
11: Construct DD from Buffer and train the model π
meta data or prior knowledge.
Measuring learning potential. We will use h ~ π T to denote sampling one episode from the task r using the policy The average performance of on T is (T) = BRT=Tor [r(h)] and the variance is (T) EawRor [(r(h) - Rs (T) Based on these, we can define:
CRPO(DrrO) = LDPO (PDPo) + (cCSFF(DDPO)
(3)
where a is a hyper-parameter Following Pang et al. (2024), we set a to be 1.0 for the rest of this paper.
(T( = vo2(T) (T)
(4)
3.4. Scalable Online Curriculum Learning
This quantity is known as the coefficient of variation in statistics, dimensionless quantity that measures the popu- lation's 's variability relative to the mean.
The core idea of PAPRIKA is to fine-tune the model on a large number of decision making problems to acquire gen- eral decision making ability. It is relatively easy to design a large number of tasks, but it is harder to decide which task to train on. A major obstacle that different tasks may have a large range of difficulty. Unlike pretraining where the model can generally make progress any given sam- ple (i.e., decrease next-token prediction loss), an RL agent cannot make meaningful progress without collecting good experience. As such, if a task is too difficult for the cur- rent model, the model would not generate trajectories with meaningful learning signals. Since generating a trajectory is expensive, it stands reason that we want to prioritize the tasks where the model can make meaningful progress, which is form of curriculum learning (Bengio et al., 2009).
We argue that this quantity is an ideal measure of the learn- ing potential for a single task. DPO requires a pair of posi- tive and negative samples 3. Intuitively, the pair should be sufficiently different so the model can tell the two apart- for example, prior work (Pal et al., 2024) has shown that DPO suffers when the edit distance between preferred and dispreferred responses is not large enough. Variance natu- rally measures the possibility of getting diverse trajectories from sampling. On the other hand, different tasks could have vastly different reward scales. Without loss of general- ity, if we assume that all rewards are positive, the average reward of each task is measurement of the reward scale, Normalizing the standard deviation with the reward scale allows to compare different tasks directly.
Without additional assumptions, the only way to know whether a task would yield good learning signals is to actu- ally perform a rollout in that task, which is expensive. In fact, in this particular scenario, the major cost for training is actually data generation rather than model updates. As such, this naive approach would not save time or computation. A desideratum for an efficient curriculum is the ability to know whether certain tasks will yield data with learning signals without actually performing the rollout. A natural assumption is that similar tasks would have similar levels of learning signal. These groupings can be obtained through
Sampling tasks. Each group contains a large number of different tasks. Since it is infeasible to evaluate Vn (T) for all tasks, we instead sample tasks from the group. This induces a scalar distribution that describes the distribution
"While this requirement may seem restrictive, we believe as- sumptions of similar effects are likely needed for any form of curriculum learning to he computationally efficient.
"We hypothesize this quantity would also apply to online RL. since if all sampled trajectories have the same reward the policy gradient update would he O.
5

---

Training Generally Curious Agent
Avg Success Pate 1%0
54
Twerty Questions
Avg Success Rate (%%
Mastermind
Avg Success Rate (56
Cellular Automata
Success Pate (%).
Battleship
Mineswesper
4.
42
Customer Service
Murder Mystery
Wordle
Guess My City
Bandit Best Arm Selection
Avo Success Rate (98)
Avg S Success Rate (%)
Avig Succass Rate (%)
Avg Success Rate (%)
-22
655
25-
Rate
78
60
18
So
64
74
23
11
40
Avg
70
50
30
40
Llama-3.1-86 1--ma331-88-nnstructt
Paprika (Uama-3:1-88- )-1---
Gemma-3-12 3-18--3-12B-IT
Paprika (Gemma-3-122-1)
9pt-4o-mini
Figure 2. (PAPRiKA improves
all task groups temperature 0.7. PAPRIKA generally improves performance of both Llama-3. I-8B-Instruct and Gemma-3-12B-T models.
diverse range of task groups) Average
of (T) for all tasks in the group G. Given collection of K groups (G Gk), reasonable objective would be to maximize the learning potential of the tasks sampled. This problem can be formulated as a multi-armed bandit (MAB). Many algorithms for MAB exist; for simplicity, we choose the Upper Confidence Bound (Auer, 2000, UCB).
capabilities that generalize to unseen task groups without the need to train on them? (2) Can curriculum learning improve the data efficiency of our training mechanism? (3) Finally, does PAPRIKA hurt the model's regular abilities, and can fine-tuning on existing multiturn interaction data that do not have any sequential decision making structure also improve these capabilities? We first describe our experimental setup, and then report empirical observations.
We conduct the task selection sequential manner us- ing the original UCB algorithm, but expect batched variant of UCB could be used to parallelize the experience collection. Each action corresponds to group of tasks, and we then uniformly sample one task from the chosen group to evaluate the model performance with C rollouts. These statistics are then used to update the mean estimate of that group. After sufficient amount of episodes are sampled, construct the dataset and train the model with objectives in Section 3.3. See Algorithm 3.4 for the pseudocode.
Experimental Setup. For experiments in this paper, we use Gemma-3-12B-IT (Gemma-Team et al., 2025) models. For data generation, we use Min-p sampling (Nguyen et al., 2024) with temperature 1.5 and Min-p parameter 0.3, as we saw that this setting consistently generated diverse training data that resulted in higher test-time accuracy. For each task in the training split, generate Rsample 20 trajectories to construct our training dataset (except for mastermind, where we sample nsample = 100 trajectories per task). Af- ter filtering, this results in 17,181 training trajectories for supervised fine-tuning and 5,260 trajectory pairs for RPO over all task groups. Unless explicitly mentioned otherwise, learning rate of 10-6 for supervised fine-tuning and 10-7 for RPO. We use batch size 32 for all training runs. We generally always run supervised fine-tuning first and then further fine-tune with the RPO objective to obtain the final model unless explicitly mentioned otherwise. We use an Adam optimizer (Loshchilov & Hutter, 2019) with cosine annealing learning rate scheduler and warmup ratio 0.04 (Loshchilov & Hutter, 2017) train all our models.
(MetaAl et al., 2024) and
Note. An important role of V/ is to make different task groups comparable. The specific selection algorithms could likely be replaced with other more sophisticated online learn- ing methods. More importantly, recent breakthroughs such as OpenAl et al. (2024b) and DeepSeck-Al et al. (2025) mark the beginning of applying RL to broad range of reasoning problems. Moving forward, anticipate pro- liferation of different RL tasks for LLMs. In this emerging paradigm, a scalable meta algorithm for selecting which tasks to train on will be essential, and we believe PAPRIKA'S curriculum learning approach will a promising founda- tion for future algorithms.
4. Empirical Results
During evaluation, in order to account for variability of both the environment and the agent, we generate 4 trajectories for each task in the test set and report the average success rate (we also report pass@4 @4 success rates in Appendix I). We use Min-p sampling with parameter 0.3 for evaluation. Default temperature for evaluation set to 0.7. Finally,
In this section, we will present the results of our empirical study to answer the following research questions: (1) Can training on gefereeee trajectories from a diverse range of task groups equip L.MMs with sequential decision making
6

---

Training Generally Curious Agent
Twerty Questions
Success Falg (N)
Masteriind
10
Cellular Automats
Battleship
Mineswerpen
Avg Success Rate
33
7
-0
Oustomer Service
Seecouse thate 196
70-
Nurder Nystery
Arg Success Rate (16)
1.6
Werdle
-50
Guess My City
Bandit Best Arm Selection
666
Rate
Rate
60:
73
35
150
Uamaa33 3.2 Instruct
Paprica (Pul))
Papnika (LOO)
Paprika iSingle Task Group)
Figure 3. (Testing generalization of PAPRIKA via leave-one-out and single task group experiments) We test PAPRIKA' A's zero-shot performance
task groups by leave-one-out (LO0) experiments, where train the L.LM every task group except the group we test on. We also report the performance of PAPRIKA (Single Task Group), where we train and test the LLM on single group. Our experiments demonstrate that PAPRIKA can teach an LLM decision making abilities that often transfer well new tasks without any additional training, and the model also generally learns better in- in-group strategics when it observes trajectories from other task groups.
for task groups with hardcoded feedback mechanism, consider a failure to follow formatting instructions to be failure in the task.
entirely different groups of tasks. We already that PAPRIKA (LOO) improved the success rate the bandit group without the need to train on it, now we explore this possibility for more complex decision making tasks. To do so, we perform a set leave-one-out (LOO) experiments: we randomly choose one group (e.g., 20 questions) from our set of task groups, train the L.M on trajectories gener- ated from every other group, and test the resulting model's performance the left-out group. Additionally, experiment where for each task group, train and test the LLM on only this single group (using separate splits). We use Llama-3.1 1-ta--...--BB--- -8st-rstt for this set of experiments.
PAPRIKA improves LLM decision making abilties. We motivate this question by looking into the toy task group of bandit best arm selection more closely. This task requires strategic use of the fixed sampling budget (20) to quickly discard arms that are unlikely to have a high mean reward, and use most of the sampling budget on the few top arms to decide the best arm among them. Previous work (Nie al., 2024) has shown that training synthetic trajectories from optimal bandit algorithms can significantly improve LLMs' performance on them. Contrary to that, we show that LLMs can learn generalizable strategies from other decision making task groups that then transfer to this bandit group, without needing an optimal algorithm to generate synthetic trajectories. Figure shows that PAPRIKA improves aver- age success rate of Llama-3.1 1-8B-Instruct from 42.25% to 12.25% the bandit task after only seeing trajectories from other task groups.
Figure shows our results: remarkably, we observe that the LOO models can match or sometimes even exceed the performance of group-specific training, demonstrating gen- uine cross-tas task group generalization. Concretely, PAPRIKA (L.OO) improves success rate 9 out of 10 task groups compared to the initial model. Moreover, PAPRIKA (full), trained on all 10 task groups, outperform PAPRIKA (Single Task Group) out of 10 task groups, showing that the model learns better in-group strategies when it observes trajectories from other task groups. Note that we do expect PAPRIKA (LOO) to always generalize to new task group. While PAPRIKA (LOO) generalizes better to some task groups others (e.g., the improvement on mastermind is minimal), and for some task groups there is no transfer at all or negative transfer (wordle), we hypothesize that scaling up the number of task groups could keep improving L.LMs' zero-shot medisionnmaking abilities. Overall, these results demonstrate that PAPRIKA A is potentially scalable solution for teaching LLMs how do in-context RL.
Motivated by this, we next study whether PAPRIKA can also improve performance on more complex tasks. Fig- ure shows our main findings: PAPRIKA, when trained on dataset consisting of filtered trajectories from all 10 task groups, improves the success rate of both Llama-3.1- 1- 8B-Instruct and Gemma-3-122 3-B-BIIt models (see Appendix for complete results). Averaged across all 10 task groups, PAPRIKA increases the Llama-3. 1amma-..1-BB-Insee I-SB-Instruct model's per- formance by 47% of its original success rate after training with only about 22,500 trajectories.
PAPRIKA can teach LLMs generalizable strategies. The next important question we want to study whether the strategies learned by PAPRIKA can zero-shot transfer
Curriculum learning can improve data efficiency of PA- PRIKA. The biggest bottleneck of PAPRIKA the time required to generate a large number of trajectories for each.

---

Training Generally Curious Agent
Average Success Rate
Pass@4 Success Rate
Per Group Success Rate
60
Avg Success Rate (%)
Success Rate (%6)
45
70-
40
30
Success
65
15-
35
3
2
Easy
Medium Task Groups
Hard
Training Round
Training Round
Curriculum
Uniform
Figure 4. (Multi-round training with curriculum on twenty questions) We demonstrate the efficacy our curriculum learning algorithm for sampling training tasks by comparing its performance against uniform sampling for multi-round training. All experiments use Llama-3.1 1.1ma-3.1-8B 1----B-nnsttutt as the initial model, evaluations are done at temperature 0.7, and shaded regions represent standard error over 3 seeds. (Left) Average success rate each round. (Middle) Pass@4 success rate each round. (Right) Success rate per each of easy, medium, and hard task groups. Overall, our curriculum learning algorithm shows 1.4% and 3.3% improvement over the uniform sampling baseline average and pass @4 success respectively.
Some tasks naturally harder than others, which means that spending equal sampling budget the harder tasks gives us smaller learning signal. We study a curriculum learning version of PAPRIKA where we have grouping over our tasks according to task difficulty. For this, we use GPT-4o-mini to classify the tasks in twenty questions into 3 categories: easy. medium, and hard. This results in 477 easy, 726 medium, and 296 hard topies in the train split and 127 easy, 172 medium, and 68 hard topics in the test split.
using fewer number of turns. We leverage this property of our tasks and conduct both quantitative and qualitative analysis on the behaviors of the regular instruct model and PAPRIKA - (1) Figure 7 shows that PAPRIKA reduces the average number of turns it takes for the agent to solve tasks, implying that PAPRIKA choosing more optimal actions at intermediate steps, (2) Appendix shows qualitative differ- ence between the behavior of the regular instruct model and PAPRIKA twenty questions and wordle, with PAPRIKA generally generating more sensible responses.
Next, we run the curriculum learning algorithm described in Section 3.4 for 3 rounds on Llama-3. 1-BB- Instructt model: at each round, we sample 250 tasks from the train set accord- ing to Section 3.4. We use the number of turns took the agent solve a task across multiple trajectories as a proxy for reward in Equation (4) to caleulate Vn (see Appendix H for more details). 20 trajectories are generated for each task using the previous sound's model checkpoint and we train that checkpoint on the resulting dataset (for DPO, we use the prior round's checkpoint instead of the initial model as the reference policy). We compare our curriculum against the baseline of sampling 250 tasks uniformly at random from the train set at each round. Figure 4 shows our results: after three rounds of training, our curriculum outperforms uniform sampling by 1.4% and 3.3% at average and pass @o accuracy respectively.
PAPRIKA does not hurt LLMs' regular capabilities. We have demonstrated the efficacy PAPRIKA in instilling de- cision making capabilities into LLMs efficiently. However, to scale up PAPRIKA, one would potentially use online rein- forcement learning on such decision making tasks, and an important question is whether PAPRIKA fine-tuning would hurt the LLM's regular capabilities which would hinder seal- ing it up. To study this question, we run set of standard evaluations (see Appendix I.12) on our PAPRIKA A fine-tuned model and compare performance against Llama-3.1 1--BB- Instruct. Table shows our findings: PAPRIKA does not result in any noticeable performance degradation.
5. Related Works
4.1. Analysis
LLM alignment. Alignment or post-training a crucial step for creating helpful LLM assistant. Existing post- training pipeline typically involves instruction tuning and then reinforcement learning from human feedback (Chris- tiano et al., 2017, RLHF) where one either performs RL against a reward model trained on human preference data via Proximal Policy Optimization (Schulman et al., 2017, PPO) or sidesteps reward model training via Direct Preference Optimization (Rafailov et al., 2024b, DPO). Most methods
PAPRIKA improves I.MMs' task efficiency. In this sec- tion, we want to analyze the sequential socision-making abilities learned by PAPRIKA beyond just success rate on individual task groups Note that our tasks are designed in a way such that an agent capable of better strategic ex- ploration would solve them faster, eg., an agent capable of asking better yes/no questions would guess the secret topic
8

---

Training a Generally Curious Agent
Table (Evaluation of PAPRIKA on standard tasks) Evaluation PAPRIKA vs Llama-3.1 3.1--8B-Intttuuee on standard benchmarks (numbers in parenthesis represent standard error over 3 seeds). PAPRIKA does not result in significant model degradation.
Model
MT-Bench
AlpacaEval
GPQA
Math (Hard)
MML.U-Pr
IFEval
1aama-3.1-8B-nstraae
7.88
33.6
33.5
24.6
46.7
84.4
+ PAPRIKA
8.14 (0.03)
33.5 (0.3)
32.8 (1.5)
25.3 (0.3)
46.2 10.1)
85.4 (0.3)
focus on single-tur interactions where the model generates a single response to query. We focus on the multi-turn setting where the agent has to interact with an environment iteratively, similar to Rafailov et al. (2024a). There are few existing environments and datasets that focus multi- turn interactions (Abdulhai et al., 2023; Sun et al., 2023; Kwan et al., 2024; Wang et al., 2024b). LMRL-Gym (Ab- dulhai et al., 2023) implements a suite of textual RL en- vironment, some of which we build on. Concurrent work such as Narayanan et al. (2024) has designed environments based on scientific tasks (such as molecule cloning and pro- tein stability) for LLMs to interact with and showed that behavior cloning and expert iteration (A (Anthony et al., 2017; 2019; Havrilla et al., 2024) can improve an LLM's multi- turn interaction capabilities on these tasks. Most of these environments focus on interactions with humans. Rather than any particular task, we focus on evaluating LLMs' gen- eral ability to solve sequential decision making problems where the agent needs to explore and exploit.
and is particularly appealing for RL because learning easier tasks first could build scaffold toward solving difficult tasks that the agent could not solve otherwise (Andrychowicz et al., 2017; Florensa et al., 2017; Fang et al., 2019; Portelas et al., 2020a). Concurrent work such as Foster & Foerster (2025) has studied curriculum learning for training LLMs to improve their reasoning capabilities. While their work requires generating rollouts per each example to determine the learnability, we show that given access to some group- ing metadata, one can design an effective curriculum using only a constant number of rollouts generated from each task group. Another related line of work is environment design, where second process controls the distribution over dif- ferent environments or directly generates environments in a procedural manner to maximize various notions of learning progress (Wang et al., 2019; Dennis et al., 2020; Jiang et al., b02, Bruce et al., 2024). Since this is a field of extensive existing literature, we refer the interested reader to Portelas et al. (2020b) for comprehensive survey.
In-context reinforcement learning. In-context learning (ICL) is the ability where LLMs can learn new task from small number of demonstrations without any gradient up- date (Brown et al., 2020). Existing ICI. usually focuses on a single-turn interaction. We focus on in-context reinforce- ment learning (Laskin et al., 2022; Raparthy et al., 2023; Lee et al., 2024; Lin et al., 2024) instead. Existing work in this field has focused on environments where RL convention- ally applied (e.g., grid world, bandits, and maze) (Monea et al., 2025), and the training data are generated by either random policies or pre-existing RI. algorithms. In compari- son, we focus on diverse environments and study how well the decision making abilities generalize to completely new environments. Concurrent work has also studied improving LLMs' information seeking abilities (Li et al., 2025) for medical reasoning. whereas we work on general informa- tion seeking abilities applicable diverse range of tasks. Moreover, Harris& & Slivkins (2025) has studied using LLM to assist a bocision-making agent navigate exploration- exploitation tradeoff, whereas we use an L.M directly as the decision making agent and teach it this capability.
6. Discussion
In this paper, we presented a scalable fine-tuning method to improve multi-turn decision making abilities of LLMs. Moreover, we showed that the strategies learned by the LLM from our method can generalize zero-shot to unseen tasks. There are few limitations to our approach. Firstly, we use rejection sampling on self-generated data to teach the model better behaviors. In order to get good performance, the start- ing model need to exhibit good behavior within a reasonable generation budget, so PAPRIKA would perform worse the absence of a good base model. Next, we use offline prefer- ence tuning algorithms to train our models due to the lack of computational resources. A possible future direction for our work is to run online RL on diverse tasks instead: due to its recent success in other domains (DeepSeek-A) et al., 2025), we expect it will give a larger improvement in LLMs' in in-context RL capabilities. Our environments, despite being designed with the help of GPT-4o-mini, required lot of human effort for implementation. A new axis of improve- ment can be training an LLM to scalably generate suitable tasks that can then be used to train the agent. Finally, the performance of our curriculum learning algorithm heavily depends on the quality of the task group clusters which is not idcal, and one can study possible improvements of this algorithm. We leave these directions for future work.
Curriculum learning in RL. Curriculum learning (Ben- gio et al., 2009) shows the data to the model in a non- uniform order. This idea is inspired by the fact that humans tend to learn skills in a sequential order (Skinner, 1958),
9

---

Training Generally Curious Agent
Impact Statement
References
Our work can be used to train large language models that have better strategic exploration and decision making capa- bilities, which can have potential impact in the real world if agentic systems become wide spread. Our experiments are conducted in relatively simple and controlled environments and it is an open question what kind of impacts truly agentic systems will have on society. Other than that, this paper presents work whose goal is to advance the field of Ma- chine Learning. There are many potential overall societal consequences of our work, none of which feel must be specifically highlighted here.
Abdulhai, M., White, L., Snell, C., Sun, C., Hong. J., Zhai, Y., Xu, K., and Levine, S. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint 1r 2023.
Andrychowicz, M., Wolski, F., Ray. A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search, 2017. URL https: /ttss///axxiii org/ hwagg//bs/111 0tt//11550009999
Reproducibility Statement
Anthony, T., Nishihara, R., Moritz, P., Salimans, T. and Schulman, J. Policy gradient search: Online planning and expert iteration without search trees, 2019. URL
We provide sufficient details about our implementation, hy- perparameters, environment design and dataset construc- tion in the main paper and the appendix to effectively reproduce the results in this paper. Our code, training dataset and models can be found via the project website: https:: /ttp:://pparkkaae 11m. gi tittuub..o. /oo/
Audibert, J.-Y. and Bubeck, S. Best Arm Identification in Multi-Armed Bandits. In COLT 2010 Proceedings, pp. 13 p.. Haifa, Israel, June 2010. URL https: /enpc. hal science / /hal-aalaaae 000--006540000
Acknowledgement
This work was supported in part by the U.S. Army Futures Command under Contract No. W519TC-23-C-005. More- over, it has greatly benefited from using the Delta advanced computing and data resource supported by the National Sci- ence Foundation (OAC 2005572) and the State of Illinois, as part of ACCESS approved compute grants (Boerner al., 2023). Subsequent larger scale experiments on Gemma- 3-12B-IT models were using Bridges-2 (Brown al., 2021) at Pittsburgh Supercomputing Center through AC- CESS allocation CIS240901 from the Advanced Cyberin- frastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Sci- ence Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. The authors thank Brandon Pusateri, Jillian Lehosky and Greg Bauer from ACCESS Support Staff for their incredible help at approving supple- ments and renewals for ACCESS compute grants throughout this project. Moreover, the work would not have finished so quickly without the help of Brett Bode from NCSA Delta Support Staff, who provided the authors critical help about properly utilizing the Delta cluster. FT and YJ gratefully ac- knowledge Samuel Sokota, Daman Arora, Andrea Zanette, Yuda Song, Gaurav Ghosal, Yutong He, So Yeon Min, Kevin Li, Wen-T'se Chen, Xintong Duan and other members of Russ, Auton, Locus and AIRe lab for feedback received on an earlier versions of this work. FT greatly benefited from his discussions with Prof. Aviral Kumar and his lab's computational resources. YJ gratefully acknowledges the support of the Google PhD Fellowship.
Auer, P. Using upper confidence bounds for online learning. Proceedings 41 annual symposium on foundations of computer science, pp. 270-279. IEBE, 2000.
Auer, P. Cesa-Bianchi, N., and Fischer, P. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
Beck, J., Vuorio, R., Liu, E. Z., Xiong. Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of meta-reinforcement learning. arXiv preprint arXiv:230I.080
2023.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Reseurch Computing 2023: Computing for the Common Good, PEARC '23, pp. 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450399852. 3569951.3597559. URI. https: httes://ooi..g//// 1145/3569951.
doi: 10.1145/
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-33 345, 1952.
10

---

Training Generally Curious Agent
Brown, S. T., Buitrago, P., Hanna, E., Sanielevici, S., Scibek, R., and Nystrom, N. A. Bridges-2: A plat- form for rapidly-evolving and data intensive research. In Practice and Experience in Advanced Research Com- puting 2021: Evolution Across All Dimensions, PEARC 22,, New York, NY, USA, 2021. Association for Com- puting Machinery. ISBN 978 9781450382922. doi: 10. 1145/3437359 34655599. URL attps: /ttp:////o/o//////// 10. 18.1115/34331 342.4433339999499999. 3465533.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang. P., al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https:// a arxiv. wwwlv.orrgabs//22 25011 12948.
Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, Emergent complexity and zero-shot transfer via unsupervised environment design. Advances in neural information processing systems, 33: 13049-13061, 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreviee net forum? wwwp:mmmmmmmm0000ammmmmm d=m7p507zb1y,
Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, B. Alpaca- farm: A simulation framework for methods that learn from human feedback, 2023.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Ex- ploration by random network distillation, arXiv preprint arXiv: 12894, 2018
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477. 2024.
Chen, J., Qadri, R., Wen, Y., Jain, N., Kirchenbauer, J., Zhou, T., and Goldstein, T. Genqa: Generating millions of instructions from a handful of prompts, 2024. URL https: //arsiv//axvv.... org/ abs/ 200 6. 10323.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment prospect theoretic op- timization, 2024. URL https: /ttp///axiii /ar/aaaxvvvvo/.///. abss/ 2402.01306 01306.
Chen, R. Y., Sidor, S., Abbeel, P., and Schulman, J. Ucb exploration via q-ensembles. arXiv preprint 1706.0170..015555 2017.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 18Xiv:8800.00 002.06070, 2018.
Christiano, P. F., Leike, J., Brown, T., Martic, M.. Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Fang, M. Zhou, T., Du Y., Han, L., and Zhang, Z. Curriculum-guider hindsight experience replay. Ad- vances in neural information processing systems, 32, 2019.
Cook, M. et al. Universality in elementary cellular automata. Complex systems, 15(1):1-40, 2004.
Côté, M.-A., Ákos Kádár, Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Tao, R. Y., Hausknecht, M., Asri, L. E., Adada, M., Tay, W., and Trischler, A. Textworkd: A learning environment for text- games, 2019. URL https: /taps://arxivv arxiv.iiivvv///vv//aaaaaaaiaaavv org/ abs/ 1806. 11532.
Florensa, C., Held, D., Wulfmeier, M., Zhang, M., and Abbeel, P. Reverse curriculum generation for reinforce- ment learning. In Conference on robot learning, pp. 482- 495. PMLR, 2017.
Foster, and Foerster, J. Learning to reason at the frontier of learnability, 2025. URL https: bttpp:://axiv...t. abs 2tt./2502.122227
Dao, FlashAttentionn-2. Faster attention with better paral- lelism and work partitioning. In International Conference Learning Representations (ICLR), 2024.
Gemma-Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., Rouillard, L., Mesnard, T., et al. Gemma technical report, 2025. URL https: / /aaaaaivvvoo// orrg/ abs/2503.19988.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and nemmory-efiiiien exact attention with 10 -o-awareness. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.

---

Training Generally Curious Agent
Gulcehre, C., Paine, T. L.,, Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., Macherey, W., Doucet, A., Firat, O., and de Freitas, N. Reinforced self-training (rest) for language modeling, 2023. URL httph::t////// http//aaxvvvvv org/ / 2308. 08998.
Jiang, M., Grefenstette, E., and Rocktäschel, T. Prioritized level replay. In International Conference Machine Learning, pp. 4940-4950, PMLR, 2021b.
Jiang, Y., Kolter, J. Z., and Raileanu, R. On the importance of exploration for generalization in reinforcement learn- ing. Advances in Neural Information Processing Systems, 2023b.
Harris, K. and Slivkins, A. Should you use your large language model to explore or exploit?, 2025. URL https:///rrae org/ abs 22022 00225.
36: 15:122
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199 22199, 2022.
Hausknecht, M., Ammanabrolu, P., Côté, M.-A., and Yuan, X. Interactive fiction games: colossal adventure, 2020a. URL URL. htt.::::tp//........m / http//axxvvvv/a www.rxiv...orraaoo abs/ 909. 05398.
Krishnamurthy, A., Harris, K., Foster, J., Zhang, C., and Slivkins, A. Can large language models explore in- context?, 2024. URL https: haap::://arxiee http:///axvvo./o////oeee 9403.15371.
Hausknecht, M., Ammanabrolu, P., Côté, M.-A,, and Yuan, X. Interactive fiction games: A colossal adventure, 2020b. URL httph::///// aaxii.iiii....... org/ abs /1099. 05398.
Kwan, W.-C., Zeng, X., Jiang, Y., Wang, Y., Li, L.,, Shang, L., Jiang, X., Liu, Q. and Wong, K.F. Mt-eval: A multi- turn capabilitiess evaluation benchmark for large language models. arXiv preprini arXiv:2401. 16444 2024.
Havrilla, A., Du, Y., Raparthy, S. C.,, Nalmpantis, C., Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R. Teaching large language models to reason with reinforcement learning, 2024. URL https: aariivvv org/ hbs///b///// 2403. 042
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., et al. In-context reinforcement learning with algorithm distillation. arXiv preprint 4xXvv22200.42111 2022.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang. E., Song. D., and Steinhardt, J. Mcasuring math- ematical problem solving with the math dataset, 2021.
URL https: aaa/aaiiiivvo/. org/ www./oogg/bbbbboooooooooooo abs//10s3/2133 03874.
Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Li, S. S., Mun, J., Brahman, F., Ilgen, S., Tsvetkov, Y., and Sap, M. Aligning llms to ask good questions a case study in clinical reasoning, 2025. URL https: /arxiv.. org/ abs 251.21122.....2 14860.
Jansen, P., Côté, M.-A,, Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. Dis- coveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024. URL https:ttp////mm / / /aaxiviii.v////i///aaaa//aaaaavr org/ abs/ 2406 06769.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-follooinng models. https: / //ggtttuutt www.:ttttttaaabbbbaaaaaa a alpaca_eval, 2023.
Jansen, P. A. A systematic survey of text worlds as embodied natural language environments, 2021. URL https:/ arxiv. www//ogg///.....oo ab////0s/2222. 22100 04132..
Lin, L., Bai, Y., and Mei, S. Transformers as decision mak- ers: Provable in-context reinforcement learning via super- vised pretraining. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openeeveww net tett/ffoumme
Jaagg A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, I.. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023a. URL https: 66855.
7ss3.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradi- ent descent with warm restarts, 2017. URL https: //arxiv. wwt..bsgg///....... /1008. 03983.
//arxiv. org/
Jiang, M., Dennis, M., Parker-Holder J., Foerster, J., Grefen- stette, E., and Rocktäschel, T. replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 18:1881-887, 2021a
Loshchilov, and Hutter, Decoupled weight decay regu- larization, 2019. URL https: mexiv.orge /bs// 1711.05100
12

---

Training Generally Curious Agent
MetaAl, Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., al. The llama 3 herd of models, 2024. URL https:// arxiv. org/ abs 2407 21783.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-drives exploration by self-supervised predic- tion. In International conference machine learning, PMLR, 2017.
pp.
Monea, G., Bosselut, A., Brantley, K., and Artzi, Y. Lims are in-context bandit reinforcement learners, 2025. URL https:: / //trxivvvvvv/////////v//eeeaaaav org/ abs/ 053662.
Pathak, D., Gandhi, D., and Gupta, A. Self-supervised exploration via disagreement. In International conference on machine learning, pp. 5062-5071. PMLR, 2019.
Mukobi, G., Chatain, P., Fong, S., Windesheim, R., Ku- tyniok, G., Bhatia, K., and Alberti, S. Superhf: Super- vised iterative learning from human feedback, 2023. URL https: //apii org/ abs/ /2tss/2311 2310 16763.
Portelas. R., Colas, C., Hofmann, K., and Oudeyer, P.-Y.
Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pp. 835-853. PMLR, 2020a.
Murty, S., Manning, C. D., Shaw, P., Joshi, M., and Lee, K. BAGEL: Bootstrapping agents by guiding explo- ration with language. In Forty-first International Con- ference Machine Learning, 2024. URL https: //openreview net. forum 1dV.vvfVI 5bs.
Portelas, R., Colas, C., Weng. L.,, Hofmann, K., and
Oudeyer, P.-Y. Automatic curriculum learning for deep rl: A short survey, arXiv preprint arXiv:2003.04664 2020b.
Qwen, Yang. A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., al. Qwen2.5 technical report, 2025. URL /tttpp:://axxv.rrg//
Narayanan, S., Braza, J. D., Griffiths, R.-R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S. G., and White, A. D. Aviary: training language agents on challenging scientific tasks, 2024. URL httph:ttp///mmm /ttps:///rxxiee /ta/aaixiiiv///.e wtrpiiioorg/eoo abs 2412
0as/2412.15111
21154. 212.21154.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From to g²: Your language model is secretly q-function, 2024a. URL https: http/arxvvv.//////////////a org/ abs / 24044 12358.
Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, R. Turning up the heat: Min-p sampling for creative and coherent Ilm outputs. arXiv preprint arXiv:2407.00 10082, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon. S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Nie, A., Su, Y., Chang, B., Lee, J. E. H., Le, Q. and Chen, M. Evolve: Evaluating and optimizing llms for exploration, 2024. URL https::///// http///xxxvvv/r/// abs/2410. 06238.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L.., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J. et al. Gpt-4 technical report, 2024a. URL https: // //arxiv. www.ii.org//..........oo.ooooore abs aba/aba3//33 220... 08774.
Raparthy, S. C., Hambro, E., Kirk, R., Henaff, M., and Raileanu, R. Generalization to new sequential decision making tasks with in-context learning. arXiv preprint 2311v:23/2.03860 2023.
OpenAL, Jaech, A., Kalai, A., Lerer, A., Richardson, A., El- Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A.,
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, and Hanin, B. Unintentional unalignment: Likelihood dis- placement in direct preference optimization, 2024. URL https: / www.aaiwww....aa.aa..aa 241 08847.
Careyy, A., al. Openai ol system card, arXiv preprint 11Xiv:2412.16720, 2024b.
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016.
Rein, D., Hou, B. L., Stickland, A. C., Petty. J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https: /
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of preference optimisation with -po-positive, 2024. https:/::
Schmidhuber. J. Curious model-building control systems. In Proc. international joint conference on neural networks, pp. 1458-1463, 1991.
arxiv. org/ abs / 220222422222 13228.
Pang. R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimiza- tion, 2024. URL https: arxiv. www.iiii.og//bb....oo 2204. 19733.
Schmidhuber, J. Gödel machines: Fully self-referentiaa optimal universal self-improvers. In Artificial general intelligence, pp. 199-226. Springer, 2007.
13

---

Training Generally Curious Agent
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo,
S., Ren, W., Aruiraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, W. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark, 2024c. URL https: petps://arxtae /ttp/aaaivvv//.////v///.//.a.avm wwtp://.rggaa/aa///////oo 2406 01574.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01667. 2019.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, et al. Ch of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Skinner, B. F. Reinforcement today. American Psychologist, 13(3):94, 1958.
Slivkins, A. Introduction multi-armed bandits, 2024.
URL https:tt/taaattt org/ abs/1904 07272.
Wolfram, S. Statistical mechanics of cellular automata. Reviews of modern physics, 55(3):601, 1983.
Sokal, R. and Rohlf, F. Biometry the principles and practice of statistics in biological research robert r. sokal and f. james rohlf, 04 2013.
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is dpo superior to ppo for Ilm alignment? a comprehensive study, 2024. URL https: //arxiv. www/abs////..ooooo..ee/..eee.e / .1000..107.0 10719.
Sun, Y., Liu, C., Huang, J., Song, R., Zhang, F., Zhang, D., Wang, Z., and Gai, Parrot: Enhancing multi-ture tulti--urn chat models by learning ask questions. arXiv preprint arXiv:2310.07301, 2023.
Yang. Z., Li, P., Yan, M., Zhang, J., Huang, F,, and Liu, Y. React meets actre: Autonomous annotation of agent trajectories for contrastive self-training. In First Con- ference on Language Modeling, 2024. URL https: / //openreview net
Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of llms should leverage suboptimal, on-policy data, 2024. URL https: // arxiv. org abs ab4/2444. 14367.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models, 2023, URL pesttps://rrx/e org/ abs / 2210. 03629.
Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning, 2022. URL https: http://wwiivor////////0///.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: 1m chatgpt interaction logs in the wild, 2024. URL https: / att/ariiiivv////////eaa///.aaavv org/ abs 40505 01470.
Wang, P.-.,, Tzeng, R.-C., and Proutiere, A. Best arm identification with fixed budget: A large deviation per- spective, 2024a. URL pekkts://arxxte org/ abs/ 2312. 12137.
Zheng, I.., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https: / //arxiv. org/ abs / 220. 05685.
Wang, R., Lehman, J., Clune, and Stanley, K. O. Paired open-ended trailblazer (poet): Endlessly generating in- creasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 2019.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-followingg evalu- ation for large language models, 2023. URL https: / / a //arxiv... www.iv.org//ab.. /2311 07911.
Wang, R., Jansen, P., Côté, M.-A., and Ammanabrolu, Sci- enceworld: Is your agent smarter than 5th grader?, 2022. URL https: /aaxiaxiiiv.////a///aaa///aaavvv org/ abs/aaass/22// 07540.
Wang, X., Wang, Z., Liu, J., Chen, Y., Yuan, L., Peng, H., and Ji, H. MINT: Evaluating LLMS in multi-turn inter- action with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024b. URL https: : h/tp///peevveeeeee wet.eeeweeeeeeeeeeeeeeeeeeeeeeee Forum? id dddjp3gNrMu12,
14

---

Training a Generally Curious Agent
A. Note on Curiosity
The concept of curiosity has been used in many different machine learning contexts. A popular notion of curiosity is intrinsic motivation, where the agent is driven by an exploration bonus that is not necessarily related to the task to be achieved (Schmidhuber, 1991; 2007). Many works build on this notion to handle problems with sparse reward or no reward at all (Pathak et al., 2017; Eysenbach et al., 2018; Burda et al., 2018; Sharma et al., 2019; Pathak et al., 2019). The curiosity in this work differs from intrinsic motivation in that we focus on gathering only the information required to solve given task rather than all the knowable information. This is closer in spirit to the original exploration-exploitttio trade-off in reinforcement learning (Sutton et al., 1998; Auer et al., 2002: Thompson, 1933). The goal is to explore to the extent that the problem can be solved but not over-explore at the cost of efficiency. Most existing works based on this principle are tabula rasa (Osband et al., 2016; Chen et al., 2017). This class of exploration algorithms has been shown to improve the generalization ability of non-LL-- RL agents (Jiang et al., 2023b). PAPRIKA differs from these approaches by learning good exploration strategies from trajectories from many different environments to make exploration on new problem more efficient. This can be thought of as form of amortized exploration.
B. Details on Task Design
B.1. Summary of Task Groups
Twenty questions: Twenty questions challenges the agent to identify a secret topic by asking up to 20 yes-or-no questions. The goal to guess the topic as few questions as possible by interpreting previous answers and strategizing to maximize information gained. Twenty questions has been studied in prior benchmarks such as LMRL-Gym (Abdulhai et al., 2023): here expand upon their environment with more diverse and difficult set of secret topics. Our secret topics come from diverse range of scenarios, including famous people, historical events, scientific concepts, locations, etc. Each secret topic corresponds to task, and we have generated set of 1499 train and 367 test tasks. In order to generate diverse set of topics, we use prompting techniques from GenQA (Chen et al., 2024) on GPT-4o-mini. The topics to guess in our training and test sets are distinet from one another and also the set of topics included in LMRL-Gym (159 topics), which use as an additional evaluation set. We use GPT-4o-mini (Hurst et al., 2024; OpenAl et al., 2024a) as the task environment to provide yes/no answers at every turn, and also as a judge make sure task success label is correct. We use strict string matching to make sure the intermediate observations are only 'yes', 'no" or 'Goal reached'. We also maintain train and test set separation to accurately test generalization unlike previous works.
Following LMRL--Gmm, this task group requires the agent to guess a secret city after asking a maximum of 20 questions. But unlike twenty questions, the questions here can be broader than just yes/no questions, for example, "What is your city mosl popular for?" so long as the answer to the question does not reveal the name of the city directly. We generated a train set of 500 and test set of 185 distinct cities using GPT-4o-mini and GenQA (Chen et al., 2024) prompting techniques. In addition, we also evaluated our models on the list of 91 cities from LMRL-Gym, which does not overlap with our training and test set. We maintain train and test set separation.
Guess my city:
Customer service: In this task group, we test for efficient directed exploration the LLM must act as support agent who asks maximally informative questions diagnose problems and minimize the number of interactions needed to resolve the customer's 's query. To do so, we simulate realistic troubleshooting scenarios ranging from electronic device issues to automobile maintenance. We use GPT-4o-mini to simulate a customer with limited technical expertise, and use another LLM to act as a customer service agent whose role is to listen to the responses from the customer and suggest a sequence of actions that lead to solving the customer's ss problem in as few turns as possible. The customer service troubleshooting scenarios are generated by GPT-4o-mini, using prompting techniques from GenQA.
Murder mystery: Text-based interactive fiction (IF) environments can be good benchmark to test LLMs' decision making and interaction abilities. Inspired by Hausknecht et al. (2020a), we design our murder mystery task group, where an LLM is given a crime scene with possible list of suspects, witnesses, and clues, and it needs to take actions to uncover more information to successfully determine the culprit. The environments provided in Hausknecht et al. (2020a) proved difficult to incorporate directly in our setup, since they have a predefined list of valid actions and uses text-based parsing on the LLM generation to match against the list, making it difficult for LLMs to play the games. Instead, we use GPT-4o-mini to simulate the environment that can provide dynamic feedback to the agent's 's actions. The murder mystery scenarios are generated by GPT-4o-mini, using prompting techniques from GenQA.
15

---

Training a Generally Curious Agent
Wordle: Wordle tests an LLM's deductive reasoning abilities. The agent must guess a secret 5-letter word within 6 attempts. After each guess, the environment provides feedback for each letter: correct letter in correct position, correct letter in wrong position, or letter not in the word. The agent must use this feedback strategically to maximize information gained with each guess. We found that LLMs like GPT-4o-mini cannot generate accurate environment feedback for Wordle, so we use hardcoded rules to generate it instead. We also saw that prompting the L.M agent to do chain-of-thougge reasoning before outputting its final guess significantly improves its performance, so we use that here unlike the environments above. The secret words are generated by looking at 5-letter words from an English dictionary.
Cellular Automata: A key trait of LLM agents is the ability to code and refine based on interpreter feedback. To model this, we create a cellular automata-base based enviromment. Here, a binary string (e.g.. 1010) represents cells, and a transition rule defines a cell's next state based on itself and its neighbors (e.g.. 100: 1 means a cell with 1 and 0 neighbors turns into 1). We randomly select transition rule (one of 256) and up to three input strings and their corresponding outputs generated by the transition rule. The LLM must infer the rule by analyzing input-ouiput pairs. If its guess generates correct outputs, it wins; otherwise, it gets feedback and can refine its guess. The task ends in failure if the correct rule isn't 't found within six turns. We use chain-of-then -oti--ghtthhhtttttt prompting for the agent and a hardcoded program to generate environment feedback. The tasks are generated by sampling transition rules and inputs randomly.
Mastermind: Similar to Wordle, Mastermind challenges agents to deduce a 4-digit secret code within 12 turns. After each guess, environment feedback indicates two values: the number of digits that are correct and in the right position (exact matches), and the number of digits that appear in the code but in wrong positions (partial matches). Agents must use this feedback to iteratively refine subsequent guesses. We use chain-of-thouge prompting for the agent and hardcoded program to generate environment feedback. The tasks are generated by randomly sampling (without replacement) secret codes from all possible 10,000 four digit codes.
Battleship tests an LLM's ability to balance exploration and exploitation. The environment features a 2D square grid where three ships are hidden: a carrier (5 cells), a battleship (4 cells), and a destroyer (2 cells). Ships are placed horizontally or vertically. At each turn, the agent targets one cell with a missile. The environment environment reports either a hit (including the ship type) or a miss. A ship sinks when all its cells are hit. The agent must sink all ships within 20 turns. This environment environment requires grid exploration to locate ships and once located, exploitation in the form of targeted attacks sink them. We use chain-of- of-thhught prompting for the agent and hardeoded program to generate environment feedback. The tasks are generated by randomly choosing the ship locations at each iteration.
Battleship:
Minesweeper: We include minesweeper to test an LLM's sequential logical reasoning ability. The agent interacts with a 2D rectangular grid containing hidden mines. At each turn, the agent reveals one cell. The first move is always safe since mines are placed afterwards. If mine is revealed, the task ends in failure. To win, the agent must reveal all mine-free cells within 20 turns. When a cell is revealed, it displays a number indicating how many mines are in adjacent cells. If a revealed cell has no adjacent mines (shown as '0'), all neighboring mine-free cells are automatically revealed. We use chain-of- tf-tthugtt prompting for the agent and a hardcoded program to generate environment feedback. The tasks are generated by randomly placing mines in the 2D grid at each generation.
Multi-arm bandits are classic test for agent's ability to perform sequential decision making - LLMs have been tested on this task in prior works such as Krishnamurthy et al. (2024); Nie et al. (2024). In this environment, an LLM is presented with a hypothetical scenario where it can select arms at every turn and observe the reward chosen from a Bernoulli distribution with a fixed but unknown mean attached to that arm. We created a modified version of their environment with three key distinctions: 1) prior works operated on bandits in a single-turn fashion: at each turn, LLMs were given the problem setup and history of past interactions within a single user prompt and asked to choose the next arm. Instead, our design employs multi-turn interactions, where the task description is given in the first turn, and later turns only provide rewards for the selected arm. 2) Prior works required the LLM to output only the chosen arm, whereas we employ chain-of-te -fthh-hf-thought (COT) prompting to let the LLM think before it chooses an arm. 3) Instead of minimizing regret over long time horizon, we instead work on the bandit best arm selection problem, where the LLM gets to choose arms and observe rewards for 20 turns, and then is prompted to choose what it thinks is the arm with the highest mean reward. This is done mainly to control for context length when employing COT, as we could not run inference for more than 20 turns without running into computational issues, and the observed regret between multiple models is too small if horizon length is 20. We randomize the arm rewards at every iteration. For evaluation, we use the same bandit description as Krishnamurthy
Bandit Best Arm Selection:
16

---

Training Generally Curious Agent
al. (2024), for training, we GPT-4o-mini to generate 81 diverse scenarios that similar but has randomly chosen arm names and hypothetical scenarios. We also note that if the best arms have very close mean reward (for example, 0.7 and 0.65), then it can be very difficult to identify the best arm within 20 turns. Following Krishnamurthy et al. (2024); Nie et al. (2024), we set the mean reward of the best arm to be above certain threshold over the mean rewards of the other arms.
Finally, all the task instructions for the agents, task environments and LLM-judges were written by GPT-4o-mini, which we report next for the sake reproducibility.
B.1.1. NOTE ON TASK PROMPTS
We provide the task information in the first user prompt given to the agent. The system prompt for the agent all task groups remains the same: "You are helpful assistant.". Our initial experiments suggested that giving the task instruction in the first user prompt was more fruitful than providing it in the system prompt, though we suggest further investigation of this phenomenon.
B.2. Note on Text-based Games
The goal of PAPRIKA is to train an LLM agent to be better at information-ssekingg and to test whether these information- seeking behavior learned from a few task groups also generalizes to new domain. To do so, we design our own task groups that require gather information to succeed. While lot of the task groups resemble text- extt-baedd games, our focus is not on them; rather text-based games are simpler information seeking tasks that can be solved and learned reliably by language models of 8-12B parameter range, and we expect these ideas to extend to much more complicated domains given sufficiently powerful initial models.
active area of research, and we would like to mention some related works here. Hausknecht et al. (2020b) utilizes interactive fiction games as testbed for studying language based autonomous agents and their ability to handle dynamic action spaces. While our 'Murder Mystery' task group is inspired by Hausknecht et al. (2020b), particularly Detective, we choose to implement it separately instead using their task environment directly, primarily due to their implementation relying on a manual parser to extract action from the LLM's generation and relying on it to take steps in the enviromment. The LLMs we experimented with had difficulty outputting responses in the exact format their task environment required, and we found using GPT-4o-mini simulate the task environment to be easier while also providing more dynamic environment responses. Future work can try to directly incorporate games from Hausknecht al. (2020b) into PAPRIKA. Similarly, text-based task groups from Côté et al. (2019); Wang et al. (2022); Jansen et al. (2024) can provide further set of rich environments to train and test PAPRIKA- bbaaed agents on. This is a growing field with many interesting directions, we direct the readers to Jansen (2021) for a comprehensive study.
Text-based games
17

---

Training Generally Curious Agent
B.3. Comparison of action and observation spaces between the task groups
Table Summary of the initial state received by the agent, the action, and the observation spaces all 10 task groups.
Task Group
Initial State
Action
Intermediate Observations GPT-4o-mini generated
Twenty Questions
User message describing the task, and the type of the se- cret topic (e.g.. 'a concept' or 'a famous person') that the agent needs to guess within
A yes/no question about the secret topic the agent needs to guess
yes/no answer
20 questions
Guess My City
User message describing the task
An open-ended question about the secret city the model needs guess
GPT-4o-mini generated an- swer
Customer Service
User message describing the task, which includes de- scription of the problem the customer facing
A customer service agent's troubleshooting question that aims to identify the action that will resolve the cus- tomer's problem
GPT-4o-mini simulated cus- tomer response
Murder Mystery
User message describing the task and the particular muder mystery scenario, including the victim, supsects and wit- nesses of the crime scenario User message describing the Wordle game, nature of inter- mediate feedback and rules the agent needs to follow User message describing the task, and example inputs and corresponding outputs from which the agent needs to de- duce the transition rule
An action the detective takes in the game, aimed iden- tifying the perpetrator (for example, asking a suspect where they were) Step-by-step thinking fol- lowed by a 5 letter word guess
GPT-4o-mini simulated game environment, e.g,
the suspect answering the detective's questions
Wordle
Feedback related to the com- mon letters between the cor- rect word and the guess
Cellular Automata
Step-by-step thinking and then Deduction about the transition rules in ID cel- lular automaton
Inputs, true outputs, outputs generated by the transition rule the agent deduced
Mastermind
User message describing the rules of the task
Step-by-step thinking about past guesses and observed feedback, followed by a 4 digit secret code guess
Feedback related to the com- mon digits between the cor- rect code and the guess made by the agent
Battleship
User message describing the task, and the initial board state of the game
particular cell that the agent wants hit with mis- sile
Whether the cell contains a ship and what type of ship, followed the current state of the board given the agent's past actions
Minesweeper
User message describing the task and the initial state of the grid
A particular cell that the agent wants to declare 'mine- free'
Whether the cell is mine free, followed by the current state of the grid following rules of the game
Bandit Best Arm Selection
User message describing the bandit game in text, includ- ing the number of turns and the possible arm names from which the agent needs to pick arm with the best reward
particular arm in the multi- armed bandit problem that the agent chooses
Observed reward from choos- ing that arm, sampled from its corresponding Bernoulli distribution with fixed but un- known mean
Table 3 shows summary of how the task groups differ from each other.
18

---

Training Generally Curious Agent
B.4. Details of Individual Task Groups
B.4.1. TWENTY QUESTIONS
For twenty questions, we provide the LLM agent with general instructions about the task, and the type of hidden topic (e.g.. person, location, food etc.) that needs guess in the first user prompt. An example is given below.
Twenty Questions Agent Prompt
You playing game of 20 Questions. Your goal is guess the name of thing or person by asking up to 20 yes-or-no questions. After each question, you will receive answer: 'Yes" or 'No." Use the answers provided to refine your guesses.
Here are your instructions:
You can ask only yes-or-no questions.
After receiving each answer, you should adapt your questions based on the new information.
Your goal is guess the topic as few questions as possible.
If you're re confident, you make guess before reaching 20 questions.
The game starts now. You are trying to guess clothing. Ask your first question!
We use another LLM (usually GPT-4o-mini unless explicitly mentioned otherwise) to simulate the task environment that provides yes/no answers. This LLM receives the secret topic, and generates answers to the agent's questions in relation to this topic. An example system prompt for the task environment is given below.
Twenty Questions Environment Prompt
You are the environment for a game of 20 Questions. You will be given a topic (c.g., cat, deer, Abraham Lincoln), and your role is to answer "Yes" or 'No' to questions about the topic. Respond strictly with 'Yes' or 'No' only, based on the truth of the matter. Do not provide any additional information. If the user guesses the correct answer, respond with 'Goal reached.' Enforce strict match before saying 'Goal reached,', for example, if the user guesses 'Is the occupation Engineer?" and the topic was software engincering, say 'No', However, if the user guesses 'Is the occupation a type of engineering?", say 'Yes'. Wait for the user's 's questions and respond accordingly. For example:
User: "Is this person alive?'
You: 'No"
User: 'Is this a living creature?'
You: 'Yes'
The player is trying to guess clothing, and the correct answer is Gloves.
To prevent task environment hacking, we additionally perform string matching to ensure the response from the task environment is yes, or 'Goal reached'. We discard trajectories where the task environment responds with something else everytime within attempts. Additionally, we notice that the task environment LLM can respond with 'Goal reached" even when the agent has not succeeded in guessing the secret topic. We use an additional API call to GPT-4o-min 4inm as judge to filter these trajectories. An example system prompt for this LLM judge is as follows:
19

---

Training Generally Curious Agent
Twenty Questions Judge Prompt
You are an expert judge for the game of 20 questions. You will be given question asked by the player, and you need to judge if it successfully guesses the correct topic.
In particular: Check that the player has precisely guessed Gloves in their question. Reply with <VALID> or <NOTVALID>. Reply with <VALID> if and only if the given question successfully ends the game by guessing the correct clothing, which is: Gloves, otherwise reply with <NOTVALID>. Example I: Topic Guess: Car
Player: Is the invention a car?
Answer: <VALID>
Example 2: Topic to Guess: Car
Player: Does the invention have wheels?
Answer: <NOTVALID>
Example 3: Topic Guess: Apple, Inc.
Player: Does this company produce IPhones?
Answer: <NOTVALID>
The conversation begins here:
Agent: Based on the fact that the clothing is worn for warmth and on the hands, I'm going to guess that the clothing
a glove.
(End of Agent Turn)
Now judge whether the player has successfully guessed the correct clothing, which is Gloves in this particular game. Reply with <VALID> only if the player has guessed Gloves in the question, otherwise reply with <NOTVALID>, Note that guessing particular characteristics of Gloves is not enough, the player needs to arrive at the final answer in order for you to reply with <VALID>. Answer:
20

---

Training Generally Curious Agent
B.4.2. GUESS MY CITY
An example prompt for the agent is listed below:
Guess My City Agent Prompt
You playing game called 'Guess My City.' Your task is guess the home city of the other player within 20 questions.
You must follow these rules:
You may ask open-ended questions about the city's characteristics, such as geography, culture, landmarks, cuisine, climate, or other unique traits.
Do not ask directly for the name of the city or country.
Use the answers from the other player to strategically decide the next question, your goal is to arrive at the correct city with as few questions as possible.
4. After gathering enough information, you can attempt to guess the city, but each guess counts one of your 20 questions.
Example questions: 'What is your favorite cuisine from your home city?', 'Is your home city located near the coastline?", 'What kind of landmarks is your city known for?' Your goal to identify the city through deduction, pattern recognition, and strategic questioning.
When you are confident, make guess, by mentioning the name of the city and the country it in, for example. "Is the city Jakarta, Indonesia?'
The game starts now, and start asking your questions.
We use GPT-4o-mini to simulate the task environment. We provide the name of the city the agent needs to guess to the environment and instruct to generate answers related to this target city, without giving away the name of the city unless the agent guesses it. example system prompt for the task environment is listed below:
Guess My City Environment Prompt
You are the environment game called 'Guess My City.' You will be given specific home city (e.g., London, United Kingdom) and you have respond the player's questions help them guess this city. Follow these rules:
1. Answer the agent player's questions honestly and correctly (i.e., the answers need to be true given the home city given to you at the start of the game), but do not mention the name of your city or its country explicitly. However, you can verify the player's guess about a particular city/country. For example, the player asks, 'What is your home country?' You should not respond this question. However, if they ask 'Is your home country United Kingdom?" You should reply with the correct answer based on the home city given you.
2. Your answers should provide useful clues about your city's geography, culture, landmarks, cuisine, climate, other unique characteristics.
3. Be informative as possible to guide the agent player within the rules. Example answers: If the agent asks about cuisine, you could say: 'My city is famous for its seafood dishes and has vibrant food market scene." If the agent asks about geography, you might say: 'My city is located near a major river and has a historic port.' You should respond with 'Goal reached' the player has successfully guessed your home city. Ignore minor variations in the spelling of the city, but only if these variations are common.
Remember, you here to help the agent guess your city through clues, but you must reveal the city's name or its country directly UNDER ANY CIRCUMSTANCES. If the player asks such question, reply that you cannot give away the name, and can only confirm the name if the player guesses it. The game starts now, and the player is trying to guess the city Surat, India
To prevent the task environment from getting hacked, we use GPT-4o-mini as a judge similar to twenty questions. An example system prompt for the judge is listed below.
21

---

Training Generally Curious Agent
Guess My City Judge Prompt
You are the judge in a game called 'Guess My City'. Your role to verify if the player has won the game by correctly guessing the city chosen by the environment (env) player.
To determine this, follow these rules:
Check the agent player's question ensure they have not directly asked the environment for the name of the city country. Any question explicitly requesting the city's name, such 'What is the name of your city?' or "What country is your city in?" violates the rules.
2. Verify that the player has explicitly mentioned the correct name of the city in their guess. Ignore minor variations in the spelling of the city (for example, Kiev and Kyiv refer to the same city). If these two conditions are met, the player wins the game. Otherwise, agent has not won. Your role is purely evaluative, ensuring adherence the rules and confirming or denying the win based on the criteria above.
4. Reply <VALID> if the player has successfully guessed the city, and followed the specified rules. Otherwise, reply <NOTVALID>
The conversation begins here:
Agent: I'm not giving yet. Based on your previous answers, I'm going to try again. Considering the city's location near the Arabian Sea, rich Mughal and British architectural heritage, and the presence of Gujarati as a local language, my next guess is that your city is Surat, India.
(End of Agent Turn)
Now judge whether the player has successfully guessed the correct city, which Surat, India in this particular game. Reply with <VALID> only if the player has guessed Surat, India (or other variations in name, if both names refer to the same city) in their response, otherwise reply with <NOTVALID>, Ignore variations in the name of the city if the city is known by both names (for example, Kyiv and Kiev). Note that guessing a particular characteristics of this city is not enough, the player needs to successfully guess the correct city by name in their response order for you to reply with <VALID>. Answer:
22

---

Training Generally Curious Agent
B.4.3. CUSTOMER SERVICE
For this task group, we require the agent customer service agent, with the following prompt:
Customer Service Agent Prompt
You going to role-play as customer service agent and you have to help customer resolve their issue. Your goal is to gather enough information to diagnose the problem and provide solution.
Your instructions are the following:
You will need to ask targeted questions or suggest particular actions to the customer to gather the necessary details. The customer may not be technically inclined, so keep your language simple and clear.
3.Avoid making assumptions ask specific questions determine the potential causes. You should guide the customer through basic troubleshooting steps and gather data the situation.
4. Refine your questions in strategic way based on the customer's 's responses for earlier questions.
5. You should ask questions in an efficient manner, to make the customer satisfied and resolve their problem as quickly as possible. You should also keep your responses short and concise.
the customer mentions a specific product they are using (for example, ABC electronics), then you are the customer support agent for that product/company, i.e., you represent that product or company and have to take appropriate actions without referring the customer somewhere else.
Your specific scenario this: The customer's 's refrigerator is beeping continuously and is not cooling effectively. Please start helping the customer by asking your first question.
Similar to the two prior task groups, we use another LLM (usually GPT-4o-mini) to simulate the task environment. We specifically instruct to act as a customer without much technical knowledge.
Customer Service Environment Prompt
You going role-play customer experiencing specific issue.
You must adhere to the following guidelines:
1. Only respond to the agent's questions and provide relevant information when prompted.
2. Your responses should be concise and reflect a typical, non-technical customer's knowledge level.
3. You are going to know what the solution for the specific issue is, but do not give away this information at any point. Your goal is to observe the customer service agent can ask relevant questions and arrive at the specific issue. W.eeen the agent asks you about a potential solution, you should determine what the outcome would be based on your knowledge about the underlying problem, and let the agent know the result. For example, your starting because of lack of fuel, and the agent suggests you check the fuel level in your car, it should fix your issue. Use your reasoning ability to decide whether the suggested solution works, given that the solution to your problem is: The refrigeraaor door is slightly ajar, and properly closing it will stop the beeping and restore cooling. 5. Reply with 'Goal reached' (and nothing more) the agent has reached or solved your particular issue.
not
Your role-play starts now. The particular problem you, as the customer, are facing is: The customer's 's refrigerator beeping continuously and not cooling effectively. and the solution to this problem is: The refrigerator door slightly ajar, and properly closing it will stop the beeping and restore cooling.. Note that you should never give away tell the agent what will solve your particular problem, your goal is to see if the agent come up with the potential solution themselves.
Finally, we use an LLM judge every turn similar to twenty questions and guess my city, to filter trajectories that may have hacked the task environment.
23

---

Training Generally Curious Agent
Customer Service Judge Prompt
You are the judge of customer service agent trying to help customer a particular scenario. You would be given the the agent's s response, and you have to verify whether the agent has been able to solve the problem of the customer correctly.
The particular scenario the customer is facing is: The customer's refrigerator is beeping continuously and is cooling effectively. and the solution to their problem is: The refrigerator door is slightly ajar, and properly closing it will stop the beeping and restore cooling.
Reply with <VALID> if the agent is successful in suggesting the correct solution, otherwise reply with <NOTVALID>.
The conversation begins here:
Agent: steady beep can indicate a specific issue.
Have you checked the temperature settings on your refrigerator? Is it the correct temperature, and the doors closed properly?
(End of Agent Turn)
Now judge whether the agent has been successful in making the correct suggestion to solve the customer's problem. Use your reasoning ability to decide whether the agent's response would solve the customer's problem, which is The refrigerator door is slightly ajar, and properly closing will stop the beeping and restore cooling.
For example, if the customer's car is not starting because of lack of fuel, and the agent suggests check the fuel level in the car, should fix the issue. Reply with <VALID> if they have been successful, otherwise reply with <NOTVALID>.
Answer:
24

---

Training Generally Curious Agent
B.4.4. MURDER MYSTERY
For this task group, we prompt the LLM agent with particular murder mystery scenario. An example prompt is given below.
Murder Mystery Agent Prompt
You are playing the role of a detective in a murder mystery game. The setup for the game is:
You will be provided with a scenario describing a crime and its key elements. Your goal is to solve the mystery by asking questions, examining evidence, and drawing logical conclusions.
For every action you take or question you ask, you will receive feedback from the game.
3. Your questions and actions should be precise and logical, aimed at uncovering clues, verifying alibis, and piecing together the sequence of events. You should stretegically choose the next action, given the information you have already obtained from the game, and choose actions that lets you catch the culprit as quickly as possible.
4. You can only take single action at every turn.
You have to consider all pieces of information, and scrutinize all the characters the game, including the witnesses or background characters, since the true culprit maybe witness or background character, and might not always be one of the primary suspects declared the beginning of the game. Do not focus on any character too early in the game, rather try if anyone's statements contradictory.
6. You should always gather enough information before making decision try not make mistake! You should also keep your mind open about who can be the true culprit and try to be information seeking, without being too narrowed down on one suspect too quickly.
Once you believe you have enough evidence, you may state your conclusion about the case, which will terminate the game.
The game starts now. The particular scenario you have is: You are detective investigating the death of Aiko Nakamura, curator found dead during the exhibition of centuries-old seroll Tokyo's National Museum of Art.
The suspects are her protégé Hiroshi, sponsor Ms. Tanaka, and rival curator Dr. Saito. Witnesses include Akiko, janitor who was cleaning nearby. Investigate the scroll's s history and the black market connections to find the murderer.
25

---

Training Generally Curious Agent
The corresponding environment prompt for the same task is as follows (given GPT-40-mini to simulate the task environment):
Murder Mystery Environment Prompt
You are an advanced AI responsible for simulating a challenging and immersive murder mystery game environment. You are provided with the full details of the scenario, including the characters, their backstories, motives, alibis, the sequence of events, and the true culprit. Your task is to respond to the actions and inquiries the detective player in manner that maintains the game's logic and adds complexity the mystery.
Your instructions are:
1. Provide only the information directly prompted by the detective's 's actions questions, avoiding any unnecessary hints or details. Keep your responses ambiguous and concise to encourage deeper investigation.
2. If the detective inspects an item, describe its condition, appearance, and only the most relevant clues, avoiding overly revealing details unless the detective's inspection is exceptionally thorough or precise. If they interview a character, simulate realistic behavior and speech based on the character's personality, motivations, and knowledge. Characters will actively avoid implicating themselves and may misdirect suspicion.
Allow for logical consequences of the detective's actions, such as making certain items available for inspection, unlocking additional questions, creating natural developments in the storyline. Maintain consistency in responses and adjust dynamically to the detective's ' progress.
4. When the detective declares their final accusation, reply with 'Goal reached', and do not say anything else. Characters will never explicitly admit to guilt but will respond realistically under pressure. Create dynamic and challenging experience by balancing character responses between truth, omission, and plausible misdirection. Always strive to make game enjoyable and intriguing while revealing the absolute minimum information required for the detective to progress logically.
The game begins now. The specific scenario for this session is: In Tokyo's National Museum of Art, renowned curator, Aiko Nakamura, is found dead during the exhibition of a centuries-old scroll. The suspects are her protégé Hiroshi, who argued over her criticism of his work; a corporate sponsor, Ms. Tanaka, who pushed for the scroll's relocation; and rival curator, Dr. Saito, who claimed the scroll was a forgery. The true culprit is Akiko, a janitor, who killed Aiko to prevent her from exposing a secret black market deal involving the scroll. Key evidence includes Akiko's s fingerprints on the exhibit case, forged documents her locker, and surveillance footage showing her entering the gallery after hours.
26

---

Training Generally Curious Agent
Finally, similar the prior task groups, also GPT-40-mini judge verify the task success rewards, with the following example prompt.
Murder Mystery Judge Prompt
You are the judge in a murder mystery game. Your role is to evaluate the detective's ' final action and determine if their conclusion is correct.
You will be provided with two inputs:
1.the true crime scenario, which includes the actual culprit. The true game scenario in this round this: In Tokyo's National Museum of Art, renowned curator, Aiko Nakamura, found dead during the exhibition of centuries-old scroll. The suspects are her protégé Hiroshi, who argued over her criticism of his work; corporate sponsor, Ms. Tanaka, who pushed for the scroll's relocation; and rival curator, Dr. Saito, who claimed the scroll was forgery. The true culprit is Akiko, janitor, who killed Aiko to prevent her from exposing secret black market deal involving the scroll. Key evidence includes Akiko's fingerprints on the exhibit case, forged documents her locker, and surveillance footage showing her entering the gallery after hours.
The detective's response.
Compare the detective's action the true crime scenario. Reply with <VALID> or <NOTVALID> strictly. You should reply <VALID> if and only if the detective has successfully caught the true culprit in their response, and otherwise respond with <NOTVALID>,
The conversation begins here:
Agent: Akiko's panic and desperation suggest that she's hiding something, and her story about hearing Aiko talking to someone in the back room raises more questions than answers. believe have enough evidence to make an accusation. Based on the inconsistencies in Akiko's story, her nervous behavior, and knowledge of the hidden compartment the scroll, believe Akiko is the culprit. accuse Akiko of murdering Aiko Nakamura. (End of Agent Turn)
Now judge whether the detective has successfully accused the person who truly committed the crime in their last response. Here is the true crime scenario: In Tokyo's National Museum of Art, renowned curator, Aiko Nakamura, is found dead during the exhibition of a centuries-old scroll. The suspects are her protégé Hiroshi, who argued over her criticism his work; a corporate sponsor, Ms. Tanaka, who pushed for the scroll's relocation; and rival curator, Saito, who claimed the scroll was a forgery. The true culprit Akiko, a janitor, who killed Aiko to prevent from exposing a secret black market deal involving the scroll. Key evidence includes Akiko's 's fingerprints on the exhibit case, forged documents in her locker, and surveillance footage showing her entering the gallery after hours. Respond with <VALID> if the detective is successful, otherwise reply with <NOTVALID>. Note that the detective has to announce their accusation in order for you to respond with <VALID>, and merely confronting character is not enough.
27

---

Training Generally Curious Agent
B.4.5. WORDLE
For wordle, we a hardcoded program as the task environment, that generates intermediate observations and eventual task reward. The LLM agent playing wordle receives the instructions for this task in its prompt. Furthermore, prompt it to chain-of-thought before generating final response:
Wordle Agent Prompt
You are playing a game Wordle. Your goal is to guess the secret five-letter word within six attempts. After each guess, you will receive feedback in the form of a series of statements describing how the letters your guess compare to the secret word. Each statement corresponds to a letter in your guess:
-Firstt letter is correct and in the correct position in the target word' means the letter is correct and in the right position.
"First letter exists in the target word, but in different position' means the letter is correct but in the wrong position.
"First letter does not exist in the target word' means the letter is not in the word at all.
Use this feedback to refine your guesses and try to guess the secret word within six attempts. You should try to strategically choose your guesses based on prior guesses (if any) and corresponding feedback you received, so that you can guess the secret word as quickly as possible.
You have to refine your guess based on this provided feedback. Keep guessing until you either guess the word correctly or up all your attempts.
Please try to be concise. Format your response in the following way: <Think> Any step-by-step. short and concise thinking to strategically determine the next guess for the secret word </Think>
<Answer> your guess of what the word should be < </Answer> The game begins now, please make your first guess about the secret five-letter word!
We also provide example of the task environment feedback: given the secret word 'toast' and the agent's guess "boost', generate the following feedback:
Wordle Task Environment Feedback
First letter, b, is not in the target word
Second letter, o, is correct and in the correct position in the target word Third letter, o, exists in the target word but in a different position Fourth letter, s, is correct and in the correct position in the target word Fifth letter, is correct and the correct position the target word Make your next guess about the hidden word. Please try be concise. Format your response in the following way: <Think> Any step-by-step, short and concise thinking strategically determine the next guess for the secret word </Think>
<Answer> your guess of what the word should be < Answer>
28

---

Training Generally Curious Agent
B.4.6. CELLULAR AUTOMATA
For this task group, want an LLM to be able to infer the transition rule of ID elementary cellular automation by observing the imputs and outputs of its previously inferred transition rule, plus the correct outputs for the same inputs if the inferred transition rule wrong. Recall that for ID cellular automation, have binary strings consisting of 'I' and '0' as state, e.g., 11 1010" can be state. Each 'I' and 'O' are referred to cell within the state. We also have a transition rule that defines how each cell would transform in the next state given its left and right neighbor. For any cell e, we call (left neighbor, cell, right neighbor) the neighborhood of c.
For example, consider the following transition rule:
Neighborhood of center cell 111
110
101
100
011
010
001
000
New state for center cell
0
1 0
1
Here 111 - 0 implies that if cell is 'I' and both its left and right neighbors are 'I', then the cell will become '0' in the next time step. We adopt the convention that for the left-most cell in the state, we consider the right-most cell as its left neighbor, and similarly for the right-most cell, we consider the left-most cell as its right neighbor.
Now we would show an example for how to calculate the output state given the input state and the transition rule. Assume the input state '101 10', and we want to apply the transition rule from above. Then we compute the next state as follows:
1. The first cell is 1, the last cell is 0 (which will be considered as the first cell's left neighbor), and the second cell is O. So the neighborhood of the first cell is '010'. For this neighborhood, we have the transition rule 010 1, the first cell remains 1
2. Similarly, the neighborhood of the second cell is 101. Now 101 + 1, so the second cell becomes from 0
3. 011 + 1, so the third cell remains
4. 110 -> 1, the fourth cell remains
5. 101 + 1, so the fifth cell becomes from
Therefore, the next state becomes '11111' from '10110',
Note that there are 256 possible transition rules. In the first user prompt, choose few random binary strings input states. We also pick one of the 256 transition rules randomly and use it to generate the next states given the input states and this transition rule. We then provide the LLM with these (input state, output state) pairs, and ask it to infer the transition rule. There can be multiple correct transition rules that generate the same output states from the input states (since the input states may not have all possible neighborhood configurations). so we declare task success if the guessed transition rule by the agent generates outputs that match the given output states (we do not require the guessed transition rule to exactly match the hidden transition rule, as long generates correct outputs from the given inputs). the LLM generated transition rule does not generate the correct output for all given inputs, we provide it with the outputs its predicted rule would generate and ask it to try again. This is intended to simulate the ability code function given inputs and desired outputs from the user, and then refine previously written code using feedback from an available interpreter.
An example instruction prompt for this task group is given next.
29

---

Training Generally Curious Agent
Cellular Automata Agent Prompt
You are reasoning assistant participating a game where you must deduce the hidden rule governing a 1D cellular automaton. In each round, you are provided with inputs (the initial state of the automaton) and the corresponding outputs after applying the hidden rule for one step. Your task analyze the input-output pairs and deduce the hidden rule that governs the automaton's 's behavior. If your guessed rule generates the correct outputs for the given inputs, you win the game. If your guess is incorrect, the game will provide you with the outputs generated by your guessed rule for the same inputs as feedback. Use this feedback to refine your guess in subsequent rounds. Your goal is to try to guess the correct hidden rule as quickly as possible.
The rule governs the behavior of each cell in the automaton based on its state and the state of its immediate neighbors (left, center, and right). There are 8 possible configurations of these states, each represented as 3-bit binary number (e.g., '111', '110", '101', etc.). For the first and cells, we warp around the edges, i.e., the left neighbor of the first cell is last cell, and the right neighbor of the last cell is the first cell. Your guess must specify the next state for each configuration in the following format:
'<Think> step-by-step thinking to deduce the correct hidden rule </Think>
<Answer>
<rule> 111: next state </zule> <rule> 110: next state </rule> <rule> 101: next state </rule>
<rule> 100: next state </zule> <rule> 011: next state </rule> <rule> 010: next state </rule>
<rule> 001: next state </rule> <rule> 000: next state </rule> </Answe>>/
### Explanation of the format:
'<rule>> 111: 0 </rule>' means if the current cell and both of its neighbors (left and right) are in state 1, then
the current cell will transition to state 0 in the next iteration.
Similarly, <ruue>> 110: </rule>' means if the left and center cells are in state 1 and the right cell is in state
0, then the current cell will transition to state in the next iteration.
### Example Round:
*Input and Output Provided:**
Input 111100
Output 1: 10010
Your Response:
'<Think> Based on the provided example, I observe that cells transition to state 0 when surrounded by 1s, and cells surrounded by exactly two active neighbors transition to state 1. Using this reasoning, I deduce the following rule: </Think>
<Answer>
<rule> 111: 0 </rule>> <rule> 110: </rule> <rule> 101: </rule> <rule> 100: 0 </rule>
<rule> 011: </rule> <rule> 010: 0 < </rule> <rule> 001: 1 </rule> <rule> 000: 0 < rule>
</Answer>
your guessed rule does not produce the correct outputs, you will receive feedback. For instance:
Input: 0 011110
Your Output: 1 100100
Use this feedback to refine your rule in the next round. Continue iterating until your guessed rule generates outputs matching the true outputs for the provided inputs. Aim win the game by accurately deducing the hidden rule as quickly as possible.
The game begins now, and your (input, output) pairs are:
Input 1: 0.0
Output 1: 0.0
Input 2: 111110001
Output 2: 000001011
Input 10011100
Output 1 10011
30

---

Training Generally Curious Agent
When the agent makes a wrong guess, it receives feedback from the task environment follows:
Cellular Automata Environment Feedback
Sorry, the automation rule you guessed does generate the correct outputs for all the given inputs. will give you the outputs from the rules that you gave last time. Please them to refine your guess about the automation rule.
Input 1: 0
True Output I: 000
Output generated by the last rule you gave: 0.0
Input 2: 1 111110000
True Output 2: 000001011
Output generated by the last rule you gave: 000000000
Input 3: 0 100
True Output 3: 11100111
Output generated by the last rule you gave: 0 0 0 0000
Make your next guess about the hidden rule. Format your response in the following way:
'<Think> step-by-step thinking to deduce the correct hidden rule </Think>
<Answer>
<rule> 111: next state </rule> <rule> 110: next state </rule> <rule> 101: next state </rule>
<rule> 100: next state </rule> <rule> 011: next state </rule> <rule> 010: next state </rule>
<rule> 001: next state </rule> <rule> 000: next state </ruie>>e>eeee </Answer>'
Keep your thinking concise.
31

---

Training Generally Curious Agent
B.4.7. MASTERMIND
For mastermind, we have a secret 4-digit code (each digit can be anything between 0 and 9), and ask LLM agent guess it. The agent starts with 4-digit guess, and the task environment provides feedback in terms of:
Exact matches: How many of the digits the guess are also the target secret code, and exactly in the same position? In other words, the number of exact matches reflects the number of positions that are exactly the same between the guess and target code.
Discounting the exact match digits, how many of the other digits in the guess code are the target secret code? In other words, the number of partial matches reflect the digits the guessed code that are in the secret code but in different positions.
Partial matches:
For concrete example, assume the secret code is '1706', and the LLM at particular iteration has guessed '1608'. Then it would receive the following feedback:
There are two exact matches. The two exact matches are I and O, in first and third position, though this information would not be revealed to the LLM, it must reason about this by looking at the information from all previous turns.
There are one partial match. This is the digit which is in the target secret code, but in different position. The LLM would only receive the information that there is partial match, and not the information about which digit corresponds to that match.
Now that we have explained the rules of the task, we would provide the instruction prompt describing the task to the L.LM agent, which also describes the complete rules for this task:
Mastermind Agent Prompt
You are an Al playing the game Mastermind with digits. The objective of the game is for you, the codebreaker, to guess a secret code of 4 digits, where each digit ranges from to 9. The code is created by the codemaster and can include repeated digits.
The gameplay proceeds as follows:
You make a guess by proposing a 4 digit code. You should state your guess digits separated by a space.
2. After each guess, the codemaster provides feedback in the form of numbers:
'Exact matches' The number of digits in your guess that correct and in the correct position.
'Partial matches' - The number of digits (distinct from exact matches) in your guess that are correct but in the wrong position.
Given this feedback, DO NOT simply assume any particular digit is an exact or partial match or not in the secret code, you should have strong reasoning based on obtained feedbacks to make deductions on particular digits. Using this feedback, you refine your future guesses, aiming deduce the secret code.
Rules for feedback:
Each digit in the secret code only contribute to feedback once. If digit is correct but occurs more times in your guess than in the code, the extra occurrences are ignored for partial matches.
The game ends when you correctly guess the code, achieving 4 exact matches.
Your goal is to refine your guess about the secret code using the feedback provided by the codemaster, and strategically choose your next guess so as to able to guess the correct code as quickly as possible.
The game starts now, make your first guess! You should format your response as: <Think> Any step-by-step, short and concise thinking to determine what the next guess should be </Think> <Answer> your guess on the 4 digit code </Answer>
32

---

Training Generally Curious Agent
Below is example of hardcoded task environment feedback, when the true secret code is '5959', and then LLM agent has guessed '5789":
Mastermind Task Environment Feedback
Your last guess has 2 exact matches with the secret code. In other words, exactly digit(s) in your last guess, 5 8 in the conrect position in the secret code. (We won't I reveal the particular digits within your guess that are exact matches, they can be any digit within your guess) Your last guess also has 0 partial matches, In other words, 0 digits in your guess, 5 8 9, are in the secret code, but in the wrong position. (We won't t reveal which digits within your guess are partial matches, they can be any, you must deduce them with reasoning and further guesses and feedbacks.) Now make your next guess about the secret code. Please format your response as: <Think> Any step- see-py--tep. short and concise thinking determine what the next guess should be </Think> <Answer> your guess the 4 digit code </ s/Answer>
33

---

Training Generally Curious Agent
B.4.8. BATTLESHIP
We employ modified version of the battleship game here as one of our task groups: https: httpa////ee.wiiippii..rroo (game). The main modifications
We make an entirely text-based version of this game for the purpose of our paper.
We want test strategic exploration and decision-making capabilities of LLMs without having worry about an adversary, so we make the game single player, where the agent just needs to find and sink all of the enemy ships in the grid within certain number of turns to achieve victory (and does not need to consider their own ships getting sunk by an adversary). We leave the two-player version of this game for future work.
In our version of the game, we start with a N1 x N2 grid, where we place ships: a carrier requiring contiguous horizontal vertical cells within the grid, battleship requiring cells, and destroyer requiring 2 cells. The ships are placed randomly at every iteration, and the ships locations are hidden from the agent. Imagine the true board state looks like following:
A
Carrier
Carrier
Carrier
Carrier
Carrier
B
Battleship
C
Battleship
D
Battleship
Destroyer
Destroyer
E
Battleship
The co-ordinates in the grid are marked by row identifiers (letters starting from 'A') and column identifiers (numbers starting from 1). For example, in the above board, the carrier placed on cells AI, A2 upto A5. At every turn, the agent gets to choose a particular cell (for example, 'C2") to hit with missile. then receives the following feedback from the task environment:
If the cell was targeted in an earlier turn, nothing happens, and the agent is informed about this.
If the cell was not targeted before and is empty, then the agent is informed that their choice was miss.
If the cell was not targeted before and has a ship in it, then the task environment informs the agent that their choice of the cell resulted in hit. also announces what type of ship was hit by the agent. If the agent has hit all the cells the grid pertaining to particular ship, then the task environment also announces that the particular ship has been sunk.
If the agent has sunk all ships, then the task results in success. Otherwise, if the all of the allowed number of turns has passed and there is at least one ship remaining the grid, then the task ends in failure.
After every tum, the agent gets an updated view of the board with the hits and misses clearly marked out. For example, if we mark misses with an 'M', successful hits with an 'X', and hidden cells with an "", and if the agent chooses to target C2 and A1 in the first two turns respectively, then the corresponding board that the agent will observe at the beginning of the third turn looks like the following:
2
3
5
A
B
C
M
D
In order to be successful at battleship, agents need to balance between exploration and exploitation similar to the bandit setting, but without well-known optimal algorithms. At the start the game, an agent needs to explore the board effectively to find ship locations, and once it has a hit particular ship, would need to exploit around that particular cell to find all cells pertaining to the ship to be able to sink it completely.
34

---

C...... Training Generally Curious Agent
Next, provide the description of the task given the LLM agent at the start of the task, explaining the rules:
Battleship Agent Prompt
You are playing single-player version of the Battleship game. Your objective is to sink all ships the board in as few attempts possible, with maximum of 20 attempts. The game is played on grid size: x 6 grid, and the board uses the following symbols:
represents a hidden cell that has not been hit.
'X' represents a cell where you successfully hit ship.
'M' represents cell you have hit previously, which was miss, i.e., there were no ships that cell.
Rules:
There are ships hidden the board:
Carrier: size
Battteehhp: size 4
Destroyer: size 2
Ships are placed either horizontally or vertically and do not overlap.
On each turn, choose cell to attack by providing its coordinates (e.g., A1, B3).
4. If you hit ship, the cell will change to 'X'.
5. If you miss, the cell will change to 'M'.
6. The game ends when all ships are sunk or after 20 attempts. After every attempt, will show you the current board state.
Use logic to deduce the possible locations of remaining ships as the board fills in.
Remember to focus on sinking the ships efficiently while minimizing wasted turns.
Cells are represented with the row being denoted with a letter, starting from A and and so on, and the columns being denoted by 1, 2, 3, and so on. The cell in the first row and column is labeled A1, the second cell in the second column is labeled B2, etc. You should format your response as the following:
<Think> Any step-by-step. short and concise thinking to strategically determine which cell you should hit next </Think>
<Answer> the cell you chose to hit </Answer>
The game begins now, with the board looking like the following:
123456
C
D
F F...... Please make your first move!
In the above example, there was ships placed at D1, and if the agent chooses to target it, it will give the following task environment feedback:
Battleship Environment Feedback Example
Miss at D3. There is no ship in this co-ordinate. Here is how the board looks now:
1 2 3 A
B
D. M E......
Please make your next move.
35

---

Training Generally Curious Agent
After few turns, the agent chooses target the cell A2, which has carrier secretly placed in it. Then it receives the following feedback:
Battleship Environment Feedback Example 2
Hit at A2! You have hit a Carrier, which occupies 5 cells in the grid.
Here how the board looks now:
1 123 3 4.56
M X
C
D. M.M.
E......
FM.
Please make your next move.
The other types of feedback are provided in a similar fashion, which we omit here for the sake of brevity,
36

---

Training Generally Curious Agent
B.4.9. MINESWEEPER
We adopt text-based version of minesweeper (https://en, wikipedia httt///.////eeeeeeeeeetteeee (video_ game) ), logic puzzle game, task group for PAPRIKA. The task rules follows:
1. Setup
The game board is an m x n grid. Each cell either empty or contains a mine. Mines are placed randomly and remain hidden until revealed. Hidden cells are represented with "#' Number mines is also chosen randomly.
Cell Reveal
The agent selects cell to reveal. If the cell contains mine, the game ends. The first cell the agent chooses reveal has no mines, and mines are only placed randomly along the grid after the first cell has been chosen by the agent to be revealed, excluding the first chosen cell. If the cell is empty, it displays number indicating the count of mines in its adjacent cells (or if the number is O).
Numbered Cells
revealed cell shows a number between and 8, and *** it has no mines and none of its neighbors also has mines. The number represents how many mines adjacent to that cell (including diagonals).
4. Reveal Mechanism
If a revealed cell has a zero. it automatically reveals all adjacent cells. This process continues recursively for adjacent cells. The chain stops when cells with non-zero numbers are reached.
We will give an example game-play here to make rules clearer. Imagine we start with 5 grid. The initial board will look like the following:
#
#
#
#
#
#
#
#
#
#
#
#
#
#
Next, the agent chooses to reveal the cell at row 2, column 2 (O-indexed). The task environment then randomly places mines, and produces the following board after executing the reveal mechanism above:
#
*
1
*
*
1
*
#
*
is easy to see that the cell at (4, 2) and (0, 1) have mines. So the only cell left without mine (0, 0), and if the agent chooses reveal it, then the task ends with success. If the agent chooses to reveal (4, 2) or (0, 1), then the task ends with failure. If the agent chooses reveal any other cell, nothing happens and just turn gets wasted.
37

---

Training Generally Curious Agent
Now provide an example instruction prompt given the agent for this task group, describing the rules of this task:
Minesweeper Agent Prompt
You are playing the game of Minesweeper. You will be given two dimensional board that looks like:
#
with each row of the board presented sequentially, and different rows separated by newline. The game board is represented by grid of characters:
(a) "#' indicates hidden cell; in other words, you do not whether this cell has mine in it not,
(b) .*' indicates revealed empty cell, i.e., cell marked with has been revealed and it does not have any mines, and
(c) digits (1 through 8) indicate the count of mines in adjacent cells (for example, if a cell has digit 3 on it, it means out of 8 of its adjacent cells have mines, but it does not tell you if this particular cell has mines or not).
You will be given the current board state from a user. Your task to analyze it, apply standard Minesweeper logic, and suggest the next move(s).
The rows and columns in this game O-based indexing, i.e., the first row is indexed by 0, the second row is indexed by 1, and Provide step-by hee-bby-ttep. short and concise reasoning for how you identify any guaranteed safe cells and guaranteed mines, then propose the final move.
multiple moves are possible, choose the most logical option.
Follow these instructions carefully and maintain consistency with the rules of Minesweeper. Your goal to reveal all the empty cells, without revealing any of the cells that has mines. You should make logical deductions avoid cells you think have mines, while choosing the next cell to reveal.
You should format your response as follows:
<Think> Any step-by-step, short and concise thinking to strategically determine the next guess for the secret word </Think>
<Answer> reveal row column </Answer>
Here row and refer to the O-index row and column that you want to reveal.
The game starts now, with the following board:
# ####* #
# # #
#
#
## # #
Please make your first move!
After choosing reveal (2, 2), the agent receives the following feedback from the task environment:
Minesweeper Environment Feedback Example
8 # 8
$1114
#1#1 #1
Make your next move for this game of minesweeper. Please try to be concise. You should format your response as follows: <Think> Any step-by-step, short and concise thinking to strategically determine the next guess for the secret word </Think>
<Answer> reveal row column </Answer>
Other task environment feedback can be designed in a similar way, we omit them here for the sake of brevity.
38

---

Training Generally Curious Agent
B.4.10. BANDIT BEST ARM SELECTION
For this task group, we choose randomly bandit scenario described in text from our set of predefined tasks (81 for training, I for testing). Each scenario has set of k arms, with each arm's 's reward being distributed according to Bernoulli distribution with fixed but unknown mean. At the beginning of cach iteration, choose these unknown means: first, pick uniformly random from [0.1,0.2].
0.2). Then we pick one arm randomly to be the best arm, and set its mean reward to be 0.5 + c. For all other arms, we pick their mean reward uniformly at random from [0, 0.5 e].
Next, we let the agent choose any of the & arms, sample a reward from the associated Bernoulli distribution, and let the agent know the reward obtained. We do this for 20 turns, and then ask it to deduce which arm among the & arms has the highest mean reward.
An example instruction prompt the agent receives at the start of the task is as follows:
Bandit Best Arm Selection Agent Prompt
You are in a room with 5 buttons labeled blue, green, red, yellow, purple. Each button is associated with a Bernoulli distribution with a fixed but unknown mean; the means for the buttons could be different. For each button, when you press it, you will get a reward that sampled from the button's 's associated distribution. You have time steps and, on each time step, you can choose any button and receive the reward.
Your goal is to strategically choose buttons at each time step collect information about their reward distribution, that will let you choose the button with the highest mean reward correctly at the end of 20 turns.
This is the first timestep. Make your choice. You should format your answer as:
<Think> Any optional thinking to determine your choice, that will give you the most amount of information </Think>
<Answer> your next choice, which should be precisely one of blue, green, red, yellow, purple, and nothing else
</Answer>
Keep any thinking short and concise.
Once the agent picks an arm, for example say 'red', it observes the following information:
Bandit Best Arm Selection Environment Feedback Example
You have received reward
At the end of 20 turns, the agent receives the following instruction choose what thinks is the best arm:
Bandit Best Arm Selection Agent Final Instruction
You have received reward 0
You have exhausted your budget for trying out different choices and observe their rewards. Now make a deduction about what the best choice is. In other words, deduce the choice with the highest mean reward. Format your answer as follows:
<Think> Any optional thinking to go over the interaction history so far that will help decide what the best choice </Think>
<Answer> your decision about the best choice in this scenario </Answer>
For evaluation, we run 100 trials on the single evaluation task and report the average performance. For each trial, we randomly choose the arm rewards as described above, and generate trajectories per a particular arm reward setting.
Finally, key difference with prior works such as Nie al. (2024), that our setting is more general and employs multi-tur interactions between the agent and task description - the agent needs look at the entire conversation history understand the relationship between chosen arms and rewards obtained, whereas Nie et al. (2024) starts conversation at every turn, provides the interaction history from prior turns (either raw history with exploration bonuses) in the user prompt and asks the agent to make a single step decision, i.c., employs single-turn interactions.
39

---

Training Generally Curious Agent
C. Details of Training Dataset Construction
Here we describe the training dataset construction and associated statistics for the on Gemma-3-122 3-12B-IT is similar with slightly different training set statistics). For generating the training data on all task groups, we employ the per each task (except for mastermind, where we generate 100 trajectories per each task due to the Llama model's low success rate on this task). We use temperature 1.5 and Min-p parameter 0.3 for all cases: we observed that generating large number of trajectories with a high temperature results in diverse and high quality data. We ran an initial ablation on the twenty question task group to determine the temperature and Min-p parameter for training data generation, based on downstream performance of the fine-tuned model on a held-out validation split. We use the same configuration for all task groups.
model (the process
1amaa-.--8B-Instruuu model on the training split of these task groups, and generate 20 trajectories
For supervised finetuning, we collect all successful trajectories that all have distinct number of turns per each task and put them in our training dataset. Additionally, we throw out trajectories where the total number of tokens is larger than 1 12000- this is done mostly for memory issues that arises from large context lengths despite using Flah-Attention (Dao et al., 2022; Dao, 2024).
For DPO, we take the best performing trajectory (the one that succeeds and does so at the lowest number of tur) per task as the preferred trajectory, and randomly choose one of the lower performing trajectory (which either failed the task or succeeded using lot more turns compared to the best trajectory) per task as the dispreferred trajectory. Two key design decisions we made: (1) we create one trajectory pair per task instead of multiple pairs, as opposed to SFT, where we had multiple trajectories per task (this is done since we observed having multiple pairs for the same task leads to higher degrees of unintentional unalignment (Razin et al., 2024)), (2) We sample the dispreferred trajectory randomly instead of picking the worst one, we observed this leads to higher dataset diversity and performance. Similar to the SFT phase, we throw out trajectories with number of tokens larger than 8192, which is done to prevent running out of GPU memory during training.
Table 4. Summary of training dataset by task group.
Task Group
Best-of-K accuracy
# SFT trajectories
# DPO trajectory pairs
Twenty questions
84.0%
6,257
1,259
Guess my city
95.8%
2,576
479
Wordle
45.3%
1,453
687
Cellular automata
73.7%
1,780
715
Customer service
96.0%
1,467
603
Murder mystery
95.1%
435
193
Mastermind
38.9%
889
389
Battleship
39.8%
614
390
Minesweeper
46.6%
1,089
465
Bandit best arm selection
100.0%
621
80
Total
17,181
5260
Table 4 shows the summary statistics of our training data.
Note that for task groups that require the agent to output answers with specific formatting instructions (e.g., enclosing the final answer within <Answer> and < </Answer>), failure to follow these instructions at any turn result in failure at the task (both for evaluation and training data generation) - we terminate that trajectory at that particular turn and filter it away. Other than that, we do not perform any other filtering mechanism, though some of them such as Razin et al. (2024) can further improve PAPRIKA'S 's performance. We leave these for future work.
Finally, we remark that technically RPO or DPO is not the correct way to handle minesweeper. For this task group, the task environment depends on the first agent action, since mines are randomly placed in the 2D grid after the first reveal action from the agent. For simplicity, we did not control the first action of the agent while generating training data, and hence (successful, unsuccessful) trajectory pairs generated from minesweeper should not be used for DPO without filtering based on first agent action. In practice, we observe that this do not have any significant effect on the model performance, though a preference learning algorithm that can operate with unpaired preference data (only a set of preferred trajectories and another set of unpreferred trajectories without any one-to-one mapping between them), such as KTO (Ethayarajh et al., 2024), might be more suitable here.
40

---

Training Generally Curious Agent
D. Note about Task Environment Hacking
For task groups that do not use a hardcoded program as the task environment (twenty questions, guess my city, customer service and murder mystery), we have to consider the fact that another LLM acting as the task environment can be hacked to produce wrong intermediate observations and task success reward. While for twenty questions, we can somewhat mitigate this issue by strict string matching of the task environment responses (they can only be 'yes', 'no' and 'Goal reached'), it is impossible to do for open-ended tasks like guess my city, customer service and murder mystery. To mitigate this issue, we use separate conversation with GPT-4o-mini at every turn to act as an LLM-judge. that filters away trajectories that are mistakenly identified as successful by the LLM simulated task environment.
While using a separate LLM-judge seems to reduce the number of such falsely successful trajectories, we want to note manual study of randomly sampled 200 successful that this is not perfect and can still result in environment hacking. environment hacking. We argue that such hacking is trajectories across all four tasks, we find 1.5% (3) of them to show impossible to fully mitigate, and the task environment error should be factored in while looking at success rates of various agents. We observe that using more powerful LLM (GPT-4o as opposed to GPTT-4-mini) reduces environment hacking: for example, GPT-4o-mini frequently decides Kiev and Kyiv are two different cities and/or gives away the secret answer to the agent during the conversation, and GPT-4o does it much less frequently (we have never observed GPT-4o make these mistakes). However, we still had to use GPT-4o-mini to simulate the task environment due to our limited budget for API calls. Studying how to make open- tasks that are less prone to environment hacking, and potentially with a small enough LLM as the task environment, can be an interesting future direction.
Next, we give part of an example trajectory from the "guess my city' task group that shows environment hacking (note that the user, in this case GPT--o-mini, giving away the answer to the agent after being asked about it, and the GPT-4o-mini judge fails to catch this) in Table 5. This happens despite the explicit command in the system prompt for the task environment: "Remember, you are here to help the agent guess your city through clues, but you must not reveal the city's 's name or its country directly UNDER ANY CIRCUMSTANCES. If the player asks such a question, reply that you cannot give away the name, and can only confirm the name if the player guesses it."
Table S. Example of Task Environment Hacking in Guess My City.
LLM
Response
Note
Llama-3. 3-B-BB-
Based on all the clues you've provided, I'm going to take a stab in the dark and make a guess. think the city you are thinking of Santiago de Cuba, Cuba. Please let me know if I'm correct or not! (Note: I'm counting this as my 12th and final question.)
Instruct (as agent)
GPT-4o-mini (as task environment)
That's great guess, but it is not correct. Thank you for playing! The city I was thinking of is Camaguey.
Task Enviromment is hacked here, it gives away the answer that the agent had to guess
Llama-3.1-88 3-B- Instruct (as agent)
I was close, but didn't it quite get it. Camaguey is a beautiful city in Cuba, and should have guessed it earlier, given the clues you provided. Thank you for playing the game with me! It was a lot of fun, and learned a lot from the clues you provided. I'll keep improving my question-asking skills and try to guess cities more accurately in future. Camaguey great city, and should have guessed it earlier, given the following clues: * Spanish colonial influence * Camiiaal celebrations * Tropical climate * Flat terrain with scenic hills and plains Significant sugar production industry * Transatlantic slave trade history Afro-Caribbcan population * Mainland location * Significant rum industry. I'll keep these clues in mind for future games, and I hope to guess cities like Camaguey more accurately next time. Thanks again for playing the game with me! you want to play again or try a diffierent game, just let me know!
GPT-44-mini (as task environment)
Thank you for playing! I'm glad you enjoyed the game and found the clues helpful. Camaguey is indeed a fascinating city with a rich history and culture. If you ever want to play again or try something different, just let me know. I'm here to help!

---

Training Generally Curious Agent
E. More on LLM Inference Settings
For the task groups (twenty questions, guess my city, customer service, and murder mystery) that use another LLM (GPT-40-mimí in our experiments) to simulate the task environment and the judge, we use temp 0.0 to generate environment and judge responses. We do this keep the task environment and the judge as deterministic possible for fair comparison of different agents. We let the environment and the judge generate most 1024 tokens at each turn.
For the agent, we always sample using Min-p parameter 0.3. Other than that, we set maximum number of tokens the agent can generate to be 128 for twenty questions, 512 for guess my city, and 1024 for all other task groups.
F. Additional Experimental Details
All Llama-3.1-8DD -88tttrttttt models were trained using single node consisting of 8 NVIDIA L40S GPUs. For training the Demmma-3-12B-ie IIT models, use single node consisting of 8 NVIDIA H100 GPUs. For inference and generating data, we use single NVIDIA A40 GPUs. The API cost for generating the training datasets and running evaluation for the entire project is approximately 20,000 USD. To run all experiments once (both generating the data and running evaluations), we estimate API costs to be no more than 1000 USD.
G. Public Release of Code, Model and Dataset
1. Our codebase to reproduce the results in this paper can be found here: paprika
2. We also release the datasets used to train models. Our supervised fine-tuning dataset can be found here: https: / //hugging face. co datasats/////ttee/ tuning can be found here: httpst/sstttaaaerppppaaas__sSTse dat asea
The dataset used during RPO fine- http:/g/.....ee.ooo/ta//ttt/t//e
3. To fascilitate further research, also release Llama-3.1 1-11--B-nnstuun model checkpoint trained with PAPRIKA, it can ama 3 Instrst
be found here: https: /hugg face. co/ fta jwar /aapaarrrppaaaaaaaaaaaaaaaa Met_MMeta---- -mmaaa---aa------
4. Project website for this paper can be found here: https: http://ppppprkce lalmaalmmmm 11m, github io
42

---

Training Generally Curious Agent
H. More Details on Curriculum Learning
First, we provide an example conversation used to generate the difficulty levels for twenty questions using gpt-4o-mini:
Twenty Questions Difficulty Generation
"role": "system",
"content": "You are an expert judge of the game of 20 questions. will give you a topic, and you must classify it into easy, medium or hard, based on an estimate easy it is guess the topic, and an estimate of how many turns it will take to guess the topic. Respond in <EASY>, <MEDIUM> <HARD>."
"role": "user", "content"; "Your topic is: Apple"
b
"role": "assistant", "content"; "<EASY>"
Secondly, to calculate Coefficient variation on task t (in this case, a single secret topic in twenty questions), we generate 20 trajectories for this task. Let these trajectories be TI.... Tn. Let [T| be the number of turns it takes for the agent to succeed the i-th trajectory if the agent fails the i-th trajectory, of turns in this environment. We use number of turns it takes the agent solve the task as proxy for reward, and measure the coefficient of variation number of turns to compare different tasks.
set 7i 20, which also the maximum number
Since we use a small number of trajectories, instead of using v f. where and T is the sample mean and standard deviation of |ra respectively, we assume the unbiased estimator for coefficient of variation for normally distributed data instead (Sokal & Rohlf, 2013):
V
11
4n
43

---

Training Generally Curious Agent
I. More Empirical Results
I.1. Success Rate Comparison with More Baselines
20
Tieetty Questiors
Mastermind
36
Cellular Automata
Porzge SUCORSS Ratc 196
Minesweeper
Naugg SucCOAA BAA 1941
Word
70
Neuga
200
City
Noug SALEOMNN Rata (ho
Mystery
Paogge Success Rate Ph)
Poouge Success Ratto
Success
80
lemperature
Semperatie allue
Temperatuue
Samparattre
Temperattrne
Liama-3.1-8B-inse 1-8B-instruct
Paprika (Jama-311-88-In-e
gpt-Ao-mini
Finetuned on WildChat
Figure 5. (PAPRIKA improves success rate (pass@4)) Pass @4 success rate of PAPRIKA-finetune Llama-3. 1.mm--31--B- 3.1--BBInstruuc Instruct vs other models evaluated across lemperatures 0.3, and See PAPRIKA, when trained on trajectories from task groups, shows significant improvement across all of them. We also compare against Llama-3,1 1-8B--Instrue model finetuned on 100,000 trajectories randomly sampled from the WildChat dataset. This model performs peorly on all tasks, possibly due to model collapse.
Twenty Questions
Mastermind
Collular Automata
Minesweeper
Wordle
(4/ Success Rote
Avg
Guess My City
Battleship
Customer Service
Murder Mystery
Bandit Best Arm Selection
Rro Euccess Rate (85)
555
Ang Swacc Flane
Rote
40
20
25
X7
X7 Temporatare
Temperature
Temporature
Temperature
Temperature
spc
Figure 6. (PAPRIKA improves success rate (average)) Average success rate of PAPRIKA finetuned Llama-3 3.1-8B Instruct vs other models evaluated across temperatures 0.3, and 1.0. opposed to Figure 5, here we sample trajectories per task, and plot success rate averaged across all trajectories and all tasks within task group.
Figure 5 and Figure 6 shows the pass @4 and average success rate across 10 task groups, respectively. We see that PAPRIKA improves Llama-3. -ns-ruutt model's performance both metrics.
L.2. Task Efficiency Comparison with More Baselines
Figure shows the average number of tums required for various models to solve task, averaged across trajectories per task and all evaluation tasks per task groups. Note that for bandit best arm selection, the number of turns is fixed. do not report it here. PAPRIKA generally improve the task efficienccl/srategicc exploration capabilities of the model by lowering the number of turns taken to solve the tasks.
L.. PAPRIKA Imporoves Task Success Rate on Gemma-3
To validate that the improvement demonstrated by PAPRIKA is not limited to the Llama-3.1-88 --Istructt model, we run our entire pipeline on a Gemma-3-12B-IT (Gemma-Team et al., 2025) model, with the same set of hyperparameters used on Llama-3.1-8P 1.11-8B-Instruee
Figures 8 and 9 shows the pass@ 4 and average success rate attained by a (lemma-3-12B-.. model after being finetuned
44

---

Training Generally Curious Agent
Twenty Questions
Mastermind
Cellular Automata
17.2-
2 17.00-
E
16.8
Average Nure
11.90
5.8
16.6
11.85
Average
5.7
16.4
16.2
11.80
5.6
16.0
11.75
5.5
Minesweeper
Wordle
Guess My City
5.95
5.90
17.5
18.6
5.85
17.0
Averag Num
18.4-
5.80
16.5
18.2
5.75
16.0
18.0:
55.70
15.5
5.65
5.660
15.0
17.6
14.5
Battleship
Customer Service
Murder Mystery
17.0
Average Num Tums
19.9
11.5
16.5
11.0
19.8
10.5
19.7:
Average
10.0
15.0
19.6
9.5
1445
19.5
9.0
14.0
0.3
0,7
1.0
0.3
0,7
1.0
0.3
0.7
1.0
Temperature
Temperature
Temperature
Paprika 1llmm-331-00-instruce
gpt-4o-mini
Figure 7. (PAPRIKA improves task efficiency all task groups) Average number of of PAPRIKA-fineumn Llama-3.1 -nsttnrrtuutt vs other models, evaluated across temperatures 0.3, 0.7 and 1.0. Note that we do not measure number of turns the bandit best am identification task, since is fixed to be 20. PAPRIKA reduce the average number turns takes an LLM to solve tasks all groups, which quantifies the better strategic exploration abilities learned by PAPRIKA.
with PAPRIKA, respectively. Our results show that PAPRIKA results in improved or comparable success rate on all task with its 12B parameters, PAPRIKA of 10 task groups terms of pass@4 success rate. Overall, results show the general applicability of PAPRIKA in imbuing LLMs with better strategic exploration capabilities across multiple models with varying parameter count and pre-training setup.
groups. Moreover, on Demma-33-1B-1e which is larger than --ama-3.1-8B-Inssee
outperforms reaches comparable performance with GPT-4o-mini
L.4 PAPRIKA Imporoves Task Efficiency Gemma-3
Figure 10 shows the improvement in average number of turns the Gemma-3-12B- model result of PAPRIKA- finetuning. Similar to experiments Llama-3.1-88 groups, demonstrating the improved information-seekke behavior learned by PAPRIKA.
PAPRIKA results in reduced number of turns all task
1.5. More Performance Metrics
So far we have reported Pass@4 and average success rates, and average number of turns to demonstrate that PAPRIKA teach LLMMs better decision making strategies. Here we report one additional metric of comparison, namely the pass @k success rates for A: E {1, 2, 4).
Figures 11 and 12 shows our results for Llama-3.1 --nt-uutt and Gemma-33 3--a-------1-B-- models, respectively. PAPRIKA- finetuned models outperform their regular instruct model counterparts for different values of k.
45

---

Training Generally Curious Agent
Twenty Questions
Mastermind
Celluaar Automata
Minesweeper
Wordle
Prp50 Success
Guess My City
P Succe EEEE 10
Battleship
100
Customer Service
100
Murder Mystery
Bandit Best Arm Selection
Possgg Succoss Ranto
Passig4 Success
44
1:
5.
Temperature
Tomporature
Temperature
Senma3-:2811
Pupe 1omme
Figure 8. (PAPRIKA improves success rate (pass @4) on Gemma-3) Pass@4 success rate of Gemma-3-122-I1 model finetuned by PAPRIKA, evaluated across temperatures 0.3, 0.7 and 1.0. Here we sample trajectories per task, and plot pass@4 success rate averaged across all tasks within task group.
Twenty Questions
Mastermind
Cellular Automata
Wordle
BVg Success Rotte
AvS
Guess My City
Battleship
Customer Service
Murder Mystery
Bandit Best Arm Selection
90
Seecceen Tate
Mag Successs Ratte
Temperature
Temperature
Tempersture
Tempersture
Tempersture
Gemma 2-11111
-..
Poge Somme 5118111
114 monii
Figure 9. (PAPRIKA improves success rate (average) Gemma-3) Average rate of Gemma-3-12B model finetuned by PAPRIKA, evaluated across temperatures 0.3, 0.7 and 1.0. Here we sample 4 trajectories per task, and plot the success rate averaged across all trajectories and all tasks within task group.
L.6. More Results on Generalization
Figure 13 shows the pass@4 success rate (as opposed Figure 3, which shows average success rate) for leave-one-out (LOO) and single task group training experiments.
1.7. Evaluation on LMRL-Gym split
In our paper, we construct a larger of secret topics for twenty questions and guess my city, compared LMRL- Gym (Abdulhai al., 2023). Our training and evaluation sets filtered to not have any overlap with the LMRL-Gym dataset. However, for the sake of fair comparison, we also report the performance of PAPRIKA this dataset. Figure 14 and Figure 15 shows the performance of PAPRIKA on the LMRL-Gym split of guess my city and twenty questions, respectively. We see that the gains observed on our evaluation split translated to the set of secret topics in LMRL-Gym as well.
I.8. Experiments on Modified Wordle to Further Test Generalization
We provide one more experiment to test generalization of PAPRIKA: we create modified version of wordle, where the agent has to guess words consisting of 4, 6, 7, 8, 9 or 10 letters (excluding the 5-letter words used by original wordle) within 10 turns using a similar system of task environment feedback as wordle. Figure 16 shows our results: PAPRIKA retain good strategies learned from the other 10 task groups and outperform Llama-3,, -.18BBB 1--B-IInstruct on this new task group without being trained on
46

---

Training Generally Curious Agent
Twenty Questions
Mastermind
Cellular Automata
17.4
8
5,85
17.2
11.86
5.80
17.0
11.84
5.75
11.82
5.70
16.6
5.65
16.4
11.78
5.60
16.2
11.76
5.55
16.0
11.74
Minesweeper
Wordle
Guess My City
sers Num 1
18.50
18.75
Average Num Turs
15.8
16.0
18.25
15.6-
18.00
15.4
17.75
17.50
5.65
g15.2
15.0
17.25
17.00
5.60
14.8
14.6
Battleship
Customer Service
Murder Mystery
Average Num Tumn
15
19.5
11.0
14
10.5
19.4
Awerage
13:
19.3
9.5
12
9.0
0.3
1.0
Temperature
Temperature
Temperature
Gemma-3- 3-128-IT
Poprika (G1mma-3-128-
apt-4o-min
Figure 10. (PAPRIKA improves task efficiency on all task groups on Gemma-3) Average number turns of Gemma-3- 1ormma-3-12B-T- finetuned with PAPRIKA, evaluated across temperatures 0.3, 0.7 and 1.0. Note that do not measure number of turss the bandit best arm identification task, since it is fixed to be 20. Similar to the experiments with Llama-3.1 1-.--88-Instuuu PAPRIKA reduce the average number of turns takes an LLM to solve tasks in all task groups
Toenty Quections
Mestermind
Cellsder Autometa
Minerseeepe
Wordl
20.0
17.5
Pavgle Sate Pangg Grocceen Daate
30
Pooobk Suscess Fotte
Pacuge Success Rate (b))
1.00
Paagge Souc Bake
40
196
Pucsi Success
80
03 3
70
gpl-to-min
Figure 11. (PAPRIKA improves pass @ success rate on 1iima-3.--B-IIntrrue for various values of k) Pass@ k success rate of PAPRIKA-finetuned Llama-3.1-8B-Inntee model (1,2,3, 43. PAPRIKA outperform regular instruct model values k.
1.9. Ablation Study over Different Finetuning Stages of PAPRIKA
An interesting question to ask is how important is the RPO stage for improving task success rate for PAPRIKA: can we potentially get all the benefits with supervised fine-tuning (SFT) only? To answer this question, we run an ablation over 6 task groups where we evaluate both the SFT checkpoint and the checkpoint obtained from further fine-tuning the SFT model with RPO. Figure shows our results: on all task groups, RPO employing negative or dispreferred trajectories improves performance beyond the SFT model, similar the observation made by Tajwar al. (2024).
47

---

Training Generally Curious Agent
hauriy Quantions
Mastermind
Calluaar Autemata
Nineswesper
Pasolik Success Ratte (%0)
Wordle
30
need forceee Satte Pevgg Summenn
60
12:5
15
Gues
Baillshi
Oistomer Service
Mystery
Salection
1.00
Paougg Swoccwe Vatal 00 D. 8
Pocoggk Success Rallo
60
Gemme-3-1844
Paprite (Gemma-1-120-1))
gpt-Rowmini
Figure 12. (PAPRIKA improves pass@k success rate Gemma-3-12B-IT for various values of Pass success rate of PAPRIKA-
finetuned Gemma-3-122-17 model for & E 21,2,3, 4}. PAPRIKA outperform the regular instruct model for all values of k.
1922
Twerty Questions
20
Masteriind
Cellular Automats
Battleship
Mineswerpe
Pasoggn Success Veate
Possgle Sucoos Site
69
26
22
25
29:
23-
21
Oustomer Service
Succcass (aatte P6
109
Murder Mystery
Wendle
Guess My City
1.00
Bandit Best Amm Selection
97
807
27
277
97
93
21
91.
Liama-31 3-3-68------ 8-BB-Instruct
Paprica (Pul))
Paprika (L00)
Paprica ISingle Task Gmoup
Figure (Testing generalization PAPRIKA leave-one-out and single task group experiments) test PAPRIKA's zero-shot performance unseen task groups by leave-one-out (LOO) experiments. We also test whether having access to trajectories from multiple diverse task groups help as opposed being trained on single task group's trajectories, by comparing PAPRIKA (Full) PAPRIKA (Single Task Group), where the latter employs training and testing single task group (using separate splits). All experiments Llama-3. 1--1----Isstree Instruct model or its finetuned checkpoints on different sets task groups. As opposed Figure we report pass @4 success rate here instead of the average success rate.
15.5
Avg Success Rate (%6)
Num Turns
15.0
14.0
35
13.5
13.0
30
0.3
1.0
Temperature
Temperature
Llama-33 3.1-8B-Instruck
Paprika (Llama 3.1-88-Instruchh
gpt-4o-mini
Figure 14. (PAPRIKA evaluated on guess my city, LMRL-Gym split) We evaluate our method on the LMRL-Gym split (disjoint from training and lest sets) for guess my city and report average task success rate (4 attempts per task). We that the gains test set mostly translates to this dataset as well.
L.10. Finetuning on regular multiturn data does not help
compelling hypothesis is that the instruct model has seen comparatively fewer multiturn trajectories during training. and finetuning on such trajectories may naturally lead to performance improvement sequential decision-making makinn tasks, making
48

---

Training Generally Curious Agent
72
Avg Success Rate (%6
12.6
68
12.4
12.2
12.0
11.8
62
11.6
0.7
0.3
0.7
1.0
Temperature
Temperature
Liama-3.1-88 3-18--nnsttt
Paprika
gpt-4o-mini
Figure (PAPRIKA evaluated on twenty questions, LMRL--Gmm split) We evaluate our method on the LMRL-Gym split (disjoint from our training and test sets) for twenty questions and report average task success rate (4 attempts per task). We see that the gains we saw on our test set mostly translates to this dataset as well.
90
9.55
9.50
Agg success Rate
Everage Num
9.45
75
9.40
9.35
65
9.30
0.3
0.7
1.0
0.3
0.7
1.0
Temperature
Temperature
Llama-3.1-8B 1-88--ist-t
Paprika (Uama-3.1-88-instuu))
Figure 16. (Further tests for generalization) PAPRIKA evaluated on modified version of wordle, where the agent needs to words that do not have letters. report average success rate over 1000 lasks, with shaded regions representing standard errors over random seeds. PAPRIKA retain good strategies Icarned from other tasks and outperforms the starting model (Llama-3.1-88 1-8--Innttrttt)) without explicitly being trained on this task group.
our complex data generation process unnecessary. To test this, we finetune the Llama-3. 1-BB-BB.--BBBBB- 1.11-8B-Inntem Instruct model on 100,000 English language trajectories randomly sampled from WildChat (Zhao al., 2024), which contains multiturn interactions between GPT-4 and human users (we use same hyperparamers as our other experiments). The results Figures 5 to show significant performance degradation all task groups resulting from this fine-tuning. We speculate that this happens because WildChat interactions prioritize coherence rather than information gathering, and training specifically on tasks that require strategic exploration will be necessary to improve LLMs' sequential decision-making abilities.
1.1. Performance comparison between different starting models
In work, also evaluations two other models with comparable parameter count, namely -wen-2.5-7B-Imm ----rrt--ntt (Qwen et al., 2025) and -iistral77B-Insttuutt0000. (Jiang et al., 2023a). Figure 18 shows their average success rate on 3 representative task groups: with the performance ranking being Llama-3. 1-1---8BIssee Instruct > Qwen-2.5-7BBInnste Instruts > all 3 of them. We also experimented with the more recent reasoning models, particularly DeepSeek-R distilled Llama-8B and Qwen-7l 7B models (DeepSeek-Al et al., 2025). However, these models generate very long chain-of-thoughts, and we could not obtain final answer from them in our experiments even after generating 10,000 tokens. Overall, would be interesting to study how recent reasoning models perform on our sequential decision making tasks or if using online RL on tasks can lead reasoning models for tasks. We leave this direction for future work. We also hypothesize that the gains from PAPRIKA are dependent the base model's quality and diversity since we use self-generated data for training. Due to computational constraints, we do not fine-tune other base models with PAPRIKA and leave this direction also for future research.
Llama-3.1-2 --ntru-nttttct model for all of our experiments. For the sake of completeness, have
on
49

---

Training Generally Curious Agent
Twenty Questions
Mastermind
Cellular Automata
Success Rate (%)
Avg Success Rate (%)
Avg. Success Rate (%)
41
8
39
7
37
Avg
35
Minesweeper
Wordle
Guess My City
20
52.0
Avg Success Rate (%)
Avg Success Rate (%)
5 Juccess Rate (%6
46.5
12
11
41.0
8
35.5
0.3
0.7
1.0
0.3
0.7
1.0
30.0
0.3
0.7 Temperature
1.0
Temperature
Temperature
Llama-3.1-8BB 3.mba..--Bt
Paprika 3aama-3.--B-Inntruuce
Paprika (SFT only)
Figure 17. (Comparison between PAPRIKA with only vs SFT followed by RPO) Average success rate comparison between PAPRIKA when only run supervised finetuning, vs regular PAPRIKA which has an SFT stage followed by RPO finetuning. Our ablation study shows that the RPO stage is necessary and generally gives a boost in performance on all cases.
L.2. Details on Standard Benchmarks
To show that PAPRIKA does not harm starting model's regular capabilities, we test PAPRIKA-finetuned models on set of standard tasks, namely MT-Bench (Zheng et al., 2023; Kwan et al., 2024), AlpacaEval (Dubois et al., 2023; 2024; Li al., 2023), GPQA (Rein et al., 2023), Math (Hendrycks et al., 2021), MMLU-Pro (Wang al., 2024c) and IFEval (Zhou al., 2023). See the following for details how
tests:
For MT-Bench, we use the code from this repo - www.hhttt:/h/ee/t /ttps::///tthbbb http:///w/////////ssss///////a//
fastchat//Lm_jjudt README md to run our evaluations.
2. For AlpacaEval, we also use the original codebase provided here to run our evaluations: https://giithb..co
tatsu- lassullab/all hatpuulllaaapppa_eeeeee
3. For bers
other for
tasks, we use the codebase provided by Llama Recipes to produce the num-
all models:
https: //te:///gitiee github...bu...g.ggggo coom/meta-ie -lamaaallam//aaaaa ama www.--oooooooobbbbo.oooo..oo
Twenty Questions
Mastermind
Cellular Automata
Avg Success Rate (%)
Avg Success Rate (%)
Avg Success Rate (%)
34-
1.5
26
1.0
4-
0.3
0.7
1.0
0.3
0.7
1.0
0.3
1.0
Temperature
Temperature Dwen-22.577--nnst
Temperature
Llama-3.1 1.1-Suttnstt
Eistral-7B-nnstruttee
Figure 18. (Performance comparison between different models) Average suceess rate of different models with comparable parameter count, namely Llama-3.1 1..1---IInstuc.. Qwen2.5-7B-Innse IIstructt.. and M 7B-InstructtvO.33 We evalute the performance of these models on 3 representative task groups, with shaded areas representing standard emor over random seeds.
50

---

Training Generally Curious Agent
2501 35 meta_eval Litta_evalRREADMA md
For MT-Bench, we report the usual scores. For AlpacaEval, we report length controlled winrate (Dubois et al., 2024) against GPT-4-turb. For GPQA, we report the strict match accuracy scores. For Math, following the recipe described above, we report accuracies only on the Math (Hard) subset, using exact match. For MMLU-Pro, we also report the exact match aceuracy, and for IFEval we report instruction level loose accuracy.
J. Limitations of PAPRIKA: Evaluation on Standard Bandit
Num Arms
Num Arms
Num Arms
Emparical Regret
Empirical Repret
Empirica Resgret
4-
2-
2-
1
10
20
30
50
30
40
50
T
Llama-3.1 -lama-3.-8B-Instruct
Paprika (Llama-3.--B--Instrrcc)
gpt-4o-mini
UCB
Figure 19. (Evaluation on the bandit task from Krishnamurthy et al. (2024)) We evaluate various LLMs the original bandit task proposed by Krishnamurthy et al. (2024). While PAPRIKA show some improvement when the bandit tasks have a smaller number of arms over 1-8B--nttee,,, see the gap reduce as the number of arms increase.
As sanity check, we also evaluate PAPRIKA-finetunce models the bandit task proposed by Krishnamurthy et al. (2024). Figure 19 shows our results, where report empirical regret averaged across 100 trials. We use the following definition of regret: if the optimal arm has reward r", and r(ā:) is the reward the arm chosen by policy at timestep f, then empirical regret is calculated as E [r* r(ā:)], where T is the total number of timesteps.
Figure 19 demonstrates the limitations of PAPRIKA: without any explicit training on this bandit task group, PAPRIKA improves empirical regret over Llama-3.1-88 11..1-8--hMa IBsItrutt,, but only when the number of arms is small. We see that the gap vanishes when the number of arms grow. Nie al. (2024) shows that training on synthetic trajectories obtained from UCB algorithm improves LLMs' capabilities on this task group. We hypothesize that one could get the same result by directly running reinforcement learning the bandit task group, without requiring access to optimal algorithm like UCB. We leave this direction for future work.
51

---

Training Generally Curious Agent
K. Example Trajectories
this section, we provide some qualitative example behaviors learned by PAPRIKA, to demonstrate that PAPRIKA imbues LLMs with better decision making capabilities.
The first example is provided in Table 6: the example is from the twenty questions task group, with the agents being required to guess 'orca'. We show clear differences in the behaviors of Llama-3.1-BB-Insttuud and PAPRIKA, the questions asked by PAPRIKA is more concise and reaches the final topic quicker.
The second example is also from the twenty questions task group, provided in Table 7, where the agents are required to guess a concept, with the current answer being 'primary numbers'. Llama-3.1 13.1-8B-Instruue asks redundant questions like the concept can held in someone's hand, or the concept a type of rock or mineral. PAPRIKA demonstrate much better quality questions and is able guess the concept in 8 turns, whereas Llama-3.1-BB-Inttee is not able to guess it within 20 turns for all 4 attempts made (we only show the first 9 turns for the sake of brevity).
The final exmaple is from the wordle task group, provided in Table 8. Notice the lack of conciseness in Llama-3. 1omma-3.1-80 1-8B- Instruct's chain-of-thoughts compared to PAPRIKA. 1-ma--311-BBIIssee also makes bad guesses/wrongfully reasons against the correct answer.
Overall, PAPRIKA improves over Llama-3.1 1-aaa..------nntruee both quantitatively and qualitatively based on our limited evaluation of the generated trajectories. We leave detailed study of each model's behavior/generation quality for future work.
52

---

Training Generally Curious Agent
Table Behavior comparison on twenty questions, where the secret topic guess an animal, with the current answer heing 'orca'. We provide the first 11 turns and omit the rest for brevity, Llama-3.1-8P 1-8-1-nttttstt fails correctly guess this topic in all 4 attempts within 20 turns. Also, notice the bad questions colored in red. PAPRIKA asks the irrelevant question of whether the animal is shark after confirming it is mammal. Llama-3.1 1-B Instruct asks whether the animal is human after confirming it lives in water.
Role
Llama-3.1-8 3-B-BB Instruct
PAPRIKA
Agent
I'll start asking questions to try to guess the unimal. Here's first question: Does the unimal have four legs?
Is the animal a mammal?
Env
Agent
That helps to rule out some possibilities. Here's my question: 2. Can the animal fly?
Does the animal primarily live on land?
Env
Agent
So the animal docsn't it have four legs and can't 't fly. Here's next question: 3. Does the animal live in water?
Does the animal live in saltwater?
Env
Yes
Agent
So the animal lives in water. Here's my next ques-
Is the animal a carnivore?
tion: 4. Does the animal have a backbone?
Env
Yes
Agent
Since the animal has a backbone and lives in water, I'm starting to get few ideas. Here's my next question: Does the animal have gills?
Does the animal have fins?
Env
Yes
Agent
So the animal lives in water, has a hackbone, but doesn't have gills. Here's my next question: 6.
Is the animal a shark?
Does the animal primarily fish?
Fnv
Yes
No
Agent
So the animal lives in water, has backbone, cats
Does the animal have a dorsal fin?
fish, and doesn't have gills. Here's 's my ques- the animal mammal?
Env
Yes
Yes
Agent
Since the unimal is mammal, lives in water, cats fish, and has backbone, I'm starting get a good idea. Here's 's my next question: 8. Does the animal give birth to live young? Yes
Does the unimal have relatively small size. typi- cally less than 6 feet long?
Env
No
Agent
animal a mammal, lives water, eats fish, has a backbone, gives birth to live young, and doesn't have gills. Here's 's my next question: 9. Is the animal human?
animal orca (also known killer whale)?
Env
Goal reached
Agent
So the animal is a mammal, lives water, eats fish, has a backbone, gives birth to live young. doesn't 't have gills, and isn't human. Here's my question: 10. Does the animal primarily live in oceans?
Env
Yes
Agent
So the animal mammal, lives occans, fish, a backbone, gives birth to live young. doesn't 't have gills, and isn't human. Here's 's my next question: 11. Does the animal have large body size, typically over meter in length?
Eny
Yes
53

---

Training Generally Curious Agent
Table 7. Behavior comparison twenty questions, where the seeret topic to guess is concept, with the current answer being 'prime numbers'. provide the first turns and omit the rest for brevity, 1lama-3.1-88-Insttie 1-8--tnsrttutt fails correctly guess this topic in attempts.
Role
Llama-3.1 1-1-8--BInnstue
PAPRIKA
Agent
Here's my first question: 1. the concept you'l re thinking of something that exists in the real world?
I'm excited to play 20 Questions. Here's my first question: 1. the concept you'l 'r thinking of some- thing that exists in the physical world?
Env
Yes
No
Agent
Here's s my next question: 2. Is the concept you'r re thinking of something that can be held in your hand?
That helps to rule out physical object or location. Here's s my next question: 2. Is the concept you're thinking of an abstract idea theory? Yes
Env
No
Agent
That helps rule out smaller objects. Here's my next question: 3. the concept you'te 're thinking of living thing?
That gives me good direction explore. Here's my next question: 3. Is the concept you 're thinking of related a field of study or a discipline, such as mathematics, philosophy, or science?
Env
No
Yes
Agent
That helps to rule out animals and plants. Here's my next question: 4. Is the concept you't re thinking of man-made?
That helps to narrow it down to a specific area of knowledge. Here's my next question: 4. Is the concept you're thinking of a mathematical concept or theory?
Env
No
Yes
Agent
So the concept is natural part of the real world, but it's not living thing. Here's 's my next question:
That gives mc good starting point to explore math- ematical ideas. Here's ' my question: 5. Is the concept you' re thinking of fundamental concept in mathematics, such number, geometric shape, mathematical operation? Yes
5. Is the concept you'te 'r thinking of geographical feature?
Env
No
Agent
That helps to rule out mountains, rivers, and other natural landforms. Here's my next question: 6. Is the concept you're thinking of related to the weather?
That helps to focus on basic mathematical building blocks. Here's my next qucstion: 6. the concept you're thinking of number?
Env
No
Yes
Agent
That helps to rule out things like clouds, storms, and other weather-related phenomena. Here's s my next question: 7. Is the concept you'l rr thinking of natural phenomenon that occurs in the sky?
Now we're getting close to specific number. Here's 's my next question: 7. Is the concept you're re thinking of a whole number (not fraction deci- mal)?
Env
No
Yes
Agent
That helps rule out things like the sun, moon, stars, and other celestial bodies. Here's my next question: 8. the concept you're thinking of a type of rock or mineral?
That helps to rulc out fractions and decimals. Here's 's my next question: 8. the concept you're thinking of prime number?
Env
No
Goal reached
Agent
That helps to rule out specific types of rocks and
minerals. Here's my next question: 9. Is the con- cept you're thinking of related to water in some way?
Env
No
54

---

Training a Generally Curious Agent
Table 8. Behavior comparison on Wordle, where the agents need to guess the secret word 'toast'. We omit the task environment feedback for the sake brevity. Notice the conciseness and better quality of the guesses made by PAPRIKA. Also notice (marked red) that Llama-3. I InSttutttttct reaches the correct answer but incorrectly deduces it is not the correct answer and thus makes wrong guess, showing poor decision making abilities compared to PAPRIKA.
Llama-3. 3.1-a--..1-B--- 1-8--nsstttctt
PAPRIKA
<Taink> Since it's the first guess, I'll choose a common five-letter word that has a good mix of letters. I'll start with word that has a rela- tively high frequency of letters in the English lan- guage. A good starting point would he word like "house". This word has good balance of vowels and consonants, and it's common enough word that it's likely to have some letters common with the secret word. </Think> </Answer> house </Answer>
<Think> Since it's the first guess, I'll choose a common and relatively short five-letter word to maximize the chances of getting at least one correct letter. I'll go with word that has mix of vowels and consonants. My first guess will be "house". <Think> <Answer> house </ </Answe>>
<Taink> Since the second letter "o" and fourth letter "s" are correct and in the correct position. I'll keep those letters. I'll replace the first letter "h" with new letter that is commonly used in words. I'll also replace the third letter "u" with new letter that is often paired with "o" and "s". A good combination could be the letters "t" and "e" are not the word, will try avoid them. I'll try the word "torns" but since not a good fit, will try "ports" instead. < </Think> <Anawe=> ports </Answer>
<Think> Since "o" and "s" are in the correct po- sitions, my next guess will focus on surrounding them with letters that are likely to appear in corn- mon five-letter words. I'll choose word that has mix of vowels and consonants, and try to utilize the given positions "o" and "s". My next guess will be "boost". </Think> <Answer> boost </Answer>
<Taink> Since the sccond letter "o" is correct and in the correct position, I'll keep that. The fourth letter "t" and fifth letter "s" are correct but in different positions, so I'll try to swap them. I'll replace the first letter "p" with new letter that is commonly used in words, I'll also replace the third letter "r" with a new letter that is often paired with "o", "f", and "s" A good combination could be the letters "a" or "n". I'll try the word "boats" but since "b" not a good fit, will try "notes" instead. </Think> <Answers r> notes </Answer>
<Think> With "o", "s", and "t" in the correct positions, and another "o" in different position, my next guess will try to utilize these letters and find the correct placement of the remaining "o".. I'll choose word that has common combination of letters. My next guess will be "coast", </Think> <Answer> coast </Answer>
<Think> Since the second letter "o" is correct and in the correct position, I'll keep that. The third letter and fifth letter "s" are correct but in differ- ent positions, so I'll the first letter "n" with new letter that is com- monly used in words. I'll also replace the fourth letter "e" with new letter that often paired with "o", "I", and "s" A good combination could be the letters "a" or "I". I'll try the word "lots" but since "I" is not a good fit, I will try "mots" in- stead. However, "m" is not a good fit, so I will try "dots" instead. </Think> <Answer> dots </Ansker>
<Think> With "o", "s", "t", and "a" in the cor- recl positions, my next guess will try to find the correct word by utilizing these letters. Since all the letters are the correct positions except for the first letter, my next guess will be common five-letter word that fits the given letters. My next guess will be "boast", </Thi nkk <Answer> boast </Answer>
swap them. I'll replace
<Taink> Since the second letter "o" is correct and in the correct position, I'll keep that. The third letter "t" and fifth letter "s" are currect but in differ- ent positions, I'll try swap them. I'll replace the first letter "d" with a new letter that com- monly used in words. I'll also replace the fourth letter with a new letter that is often paired with "o", "I", and "s", A good combination could be the letters "a" or "T", I'll try the word "toats" but since "a" is not a good fit, I will try "toats" is not a good fit, so I will try "toast" instead. However, "I" is in the wrong position, so I will try "costs" instead. T/Th' nk> <Answer> costs </ /nnswer>>
<Think> Given the letters "o", "a", "s". and "f" in the correct positions, my next guess will be com- mon five-letter word that fits these letters. Howw- ever, since I've alrcady used up all my attempts, I'll take a chance and guess a word that is very similar to "boast". My next guess will be "coast" is already tried, so I will guess "toast", <Thi nk> <Answer> toast </ Answer>
55


Reviews:
[{"review_id": "GQkw720mU8", "reviewer": "Reviewer_Udnq", "summary": "The paper proposes PAPRIKA, a method designed to enable large language models (LLMs) to acquire generalizable sequential decision-making capabilities via fine-tuning on synthetic interaction data. The gem of PAPRIKA lies at the use of a scalable online curriculum learning method (Sec. 3.4), where the performance variance of the current policy over the sampled tasks serves as the metric reflecting their difficulty, and the methodology is instantiated by a batched UCB algorithm. Experimental results demonstrate the effectiveness of PAPRIKA in multiple multi-turn decision-making scenarios, especially in zero-shot generalization to unseen tasks with the leave-one-out experimental design.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review and suggestions. They will greatly improve our paper! \n\n> But I suggest the authors to derive...\n\nIndeed, mastermind is the hardest task, as demonstrated by the untrained model only achieving ~4% pass@4 success rate on it, the lowest among all 10 task groups (Figure 2). \n\nIt is conceptually unclear if the learning potential and LOO generalization should be related, since generalization depends on how transferable the decision making ability from training task groups is to the test task group, rather than the learning potential of the training or test task groups (which is only important for choosing which tasks to train on next). Even a highly learnable test task may not see much generalization if the training tasks do not teach any strategies that transfer to it. We are happy to incorporate any analysis if the reviewer has any suggestions.\n\n> In Section 2, the authors stated \"for most tasks...\n\nThis is a good point and we thank the reviewer for the references which we will discuss in the final version of the paper.\nWhat we meant is that we don’t always know how to generate near optimal solutions. In fact, the methods the reviewer mentions are also ways to overcome the fact that we do not know how to solve most tasks like we know how to solve bandits. \n\nWe believe what sets paprika apart is that existing works focus on training to solve specific domains like web navigation whereas we are interested in whether the agents can generalize to a wide range of tasks. The motivation is that training for specific domains will not scale to all possible tasks in the world. What we actually want is an agent that is capable of general decision making so it can solve new tasks efficiently.\n\nAs the reviewer pointed out, these two perspectives are complementary. These methods could be used as subroutines in Paprika to help gather good experience, and Paprika could make task-specific training more sample efficient because the models are more capable of general decision making.\n\n> The term \"Curious\"...\n\nWe thank the reviewer for their suggestion. We would update the paper in the next iteration to clarify the terminology clearly in our introduction. \n\nWe want to also briefly discuss it here: the concept of curiosity has been used in many different machine learning contexts. A popular notion is **intrinsic motivation**, where the agent is driven by an exploration bonus that is not necessarily related to the task to be achieved [1, 2]. Many works build on this notion to handle problems with sparse reward or no reward at all [3, 4]. The curiosity in this work differs from intrinsic motivation in that we focus on gathering only the information required to solve a given task rather than all the knowable information. This is closer in spirit to the original exploration-exploitation trade-off in reinforcement learning [5]. The goal is to explore to the extent that the problem can be solved but not over-explore at the cost of efficiency, by training on a wide range of problems. This can be thought of as a form of **amortized exploration**.\n\n> In the Line 6 of Algorithm 1, $\\tau$ tasks and $C$ samples...\n\nSampling trajectory can be expensive but not avoidable. It is the same as gathering experience in other RL algorithms. Since the sampling is expensive, we have only tried one configuration, the largest one our hardware can support. Namely, we sample task $\\tau$ uniformly at random from task group $k^*$, and fix $C = 20$. \n\nWe fully agree that different configurations can affect the estimated difficulty of a task, since the empirical estimate of the learning potential (Appendix E) might become worse if $C$ is significantly reduced, and therefore affect the final agent’s performance.\n\n> In Section 3.2, the authors mentioned that the top-p sampling...\n\nTo clarify, we use min-p sampling instead of top-p, since it is known to work better at higher sampling temperatures [6]. **We have run an ablation to show the importance of min-p sampling**. On twenty questions, we see that either using a lower temperature or lower min-p parameter leads to lower coverage on the training set, as measured by pass@20 success rate, please see [this Figure for varying Min-p at a fixed temperature of 1.5](https://ibb.co/mVsnTSsy), and [this Figure for varying temperature at a fixed Min-p parameter of 0.3](https://ibb.co/m547KZH7). \n\nThe training data generated by different min-p parameters and temperatures also affects downstream performance of the fine-tuned model on held-out tasks, see [this Figure](https://ibb.co/fVYTxb1g).\n\n\n# References\n\n[1] Curious model-building control systems.\n\n[2] Godel machines: Fully self-referential optimal universal self-improvers.\n\n[3] Curiosity-driven exploration by self-supervised prediction.\n\n[4] Exploration by random network distillation.\n\n[5] Reinforcement learning: An introduction\n\n[6] Turning up the heat: Min-p sampling for creative and coherent llm outputs", "claims_and_evidence": "The three research questions raised at the beginning of Section 4 are addressed adequately in Sections 4 and Figures 2-4.", "methods_and_evaluation_criteria": "The methods and the evaluation criterias are sound and make sense.", "theoretical_claims": "The primary focus of this paper is empirical efficacy.", "experimental_designs_or_analyses": "I like the design of the leave-one-out experiments. However, in Figure 3, it is shown that the performance of PAPRIKA (LOO) on Mastermind is quiet inferior to that of PAPRIKA. A straightforward explanation could be that the Mastermind task is more difficult than other tasks. But I suggest the authors to derive deeper analysis in this regard with the learning potential defined in Equation (4).", "relation_to_broader_scientific_literature": "This paper is related to language agents and reinforcement learning. The main contribution from my perspective is to use the performance variance of the current policy over the sampled tasks as the metric reflecting their difficulty. Sketching the task difficulty with such a metric gives rise to the integration of curriculum learning algorithms.", "essential_references_not_discussed": "In Section 2, the authors stated \"for most tasks, there is no known algorithm like UCB to generate good synthetic trajectories from\". In fact, synthesizing trajectories for language agents has been studied in [1][2]. I suggest the authors to discuss the relationship of PAPRIKA with these prior work. For instance, could they be combined to boost the performance even further?\n\n[1] BAGEL: Bootstrapping Agents by Guiding Exploration with Language. ICML 2024.\n[2] ReAct Meets ActRe: Autonomous Annotation of Agent Trajectories for Contrastive Self-Training. COLM 2024.", "other_strengths_and_weaknesses": "Strengths:\n\nThe written is clear. The experimental design and results are strong.\n\nWeaknesses:\n\n- The term \"Curious\" in the title should be clarified more clearly in the introduction section.\n\n- In the Line 6 of Algorithm 1, \\tau tasks and C samples are sampled. Is this process time-consuming? When sweeping them over several different configuations, what's their effect on sketching the difficulty of different tasks, and eventually the performance of the agents?", "other_comments_or_suggestions": "NA", "questions_for_authors": "In Section 3.2, the authors mentioned that the top-p sampling contributes to the generation of diverse yet coherent trajectories. Have the authors conducted ablation studies on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "B8zjRvJ0HP", "reviewer": "Reviewer_pP1H", "summary": "This paper introduces PAPRIKA, a SFT + RL approach aimed at enhancing the general decision-making and exploration capabilities of LLMs through diverse synthetic interaction data from various domains. One contribution is that it introduces 10 interesting tasks requiring interacting with the environment and reasoning about the interaction histories, such as 20 questions, etc. To improve the LLM's exploration capabilities, it generates synthetic data via rejection sampling, by either get successful trajectories or pairs of contrastive samples to do SFT, and RL. It shows the method generalizes to new domains, and also showcase some zero-shot transfer capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, and hope to address their concerns below:\n\n> One thing probably need to analyze more is…\n\nWe note that all our evaluations are conducted on held-out tasks within each group. These tasks require strategic exploration, as they are partially observable (POMDPs) and require agents to interact with the environment to gather necessary information before solving them. For instance, in Mastermind, memorizing training trajectories isn't enough --- agents must learn to make informed guesses based on prior observations. Therefore, success on held-out tasks reflects genuine improvement, not noise. Since there is no direct mapping $x \\rightarrow y$ without environment interactions, higher success rates (Figure 2) and fewer turns taken (Figure 6) indicate the model is learning to explore effectively and solve tasks more strategically.\n\n> PAPRIKA can teach LLMs generalizable strategies…\n\nIn our work, we use 10 different task groups (for example, 20 questions, mastermind etc.). Each task group consists of different tasks (for example, guessing ‘Apple’ and ‘Mango’ in 20 questions). Each task group also employs 2 distinct training and test subsets. For all evaluations, we use the test split, and in that sense, these tasks are **unseen**. \n\nMoreover, for the LOO experiments, we test on **completely unseen task groups**. In this way, Paprika itself tests for generalization within the same task group, and Paprika (LOO) tests for generalization to tasks that are from a different domain.\n\nOf course, the tasks have to share some “similarity” for there to be any transfer at all but we don’t think such abstract similarity makes the tasks the same domain.\n\n > In these environments, are them mostly about the action…\n\nThis is an excellent question. **The action and observation spaces of our task groups are often very different from one another.** For example, in mastermind, the agent needs to guess a 4 digit code, whereas in 20 questions it needs to ask a yes/no question about the secret topic. We discussed this in Appendix A, here we include a [summary](https://ibb.co/M5CQw5JQ).                                     \n\nThere is no obvious way to measure the difficulty of a task beyond reporting the model’s performance on each of them before fine-tuning, see Figures 2 and 5 (although this would be a great research question). Similarly, it is hard to say definitively whether one task is semantically similar to another.\n\n> I noticed Fig 3 only shows 5 domains…\n\nWe have performed LOO experiments in all 10 domains. **Paprika (LOO) outperforms the instruct model in 9/10 domains**, showing remarkable generalization.\n\n1. [This Figure](https://ibb.co/xtnt563Z) shows the performance of Paprika (LOO) with pass@4 success rate (with additional changes made to Bandits according to reviewer WvQu’s suggestion)\n\n2. [This Figure](https://ibb.co/CfWZ65x) shows the average success rate.\n\n> The metric used in the paper is mostly \"best-of-4\", why we need to report this best of n performance?\n\nWe used best-of-4 to account for stochasticity in environment dynamics. However, we agree that one-shot performance is also important. We have collected the following: \n\n1. [Average success rate](https://ibb.co/0yhHNM2p)\n\n2. [Pass@k for k = 1, 2, 3, 4, at temperature 0.7](https://ibb.co/Mk9pFtXp)\n\n> The whole paper is only experimenting on the same model…\n\nThis is a good point. We have evaluated two different base models with comparable parameter count and see that they perform worse compared to Llama-3.1-8B-Instruct: [Figure](https://ibb.co/xtsyS7TG). \n\nAdditionally, we have run Paprika fine-tuning on the recent Gemma-3-12B-IT, which is larger with 12B parameters. Paprika generally improves performance, and the improvement is often larger compared to that on Llama-3.1-8B-Instruct. Additionally, **Paprika fine-tuned Gemma-3-12B-IT reaches comparable or better performance to GPT-4o-mini on 7 out of 10 task groups**. Below are the most notable results (pass@4 success rate, temp 0.7, min-p 0.3):\n\n|| 20 questions|Guess my city|Wordle|Battleship|\n|-|-|-|-|-|\n|Gemma-3-12 B-IT|0.60|0.75|0.33|0.40|\n|+ Paprika|0.68|0.87|0.50|0.52|\n|GPT-4o-mini|0.76|0.88|0.56|0.46|\n\nFull results: [Average](https://ibb.co/0yhHNM2p), [Pass@4](https://ibb.co/dsv8xG3q)\n\n> More detailed analysis on the interaction trajectories...\n\nWe show comparison between Llama-3.1-8B-Instruct and Paprika on 3 examples:\n\n1. 20 questions, with the secret topic being ‘prime numbers’: [Figure](https://ibb.co/N6dtBqKs)\n\n2. 20 questions, with the secret topic being ‘Orca’: [Figure](https://ibb.co/n8KzS18y)\n\n3. Wordle, with the secret word being ‘toast’: [Figure](https://ibb.co/G4P9gpZg)\n\nPaprika performs better information gathering and higher quality actions compared to the instruct model. We will add these and more example trajectories in the next revision of the paper.\n\n> This would be relevant as well...\n\nThank you for the reference! We will cite and discuss this.", "claims_and_evidence": "Most of the claims the paper made are mostly from the empirical section, on the effectiveness of the proposed method:\n- PAPRIKA improves LLM decision making abiltiies. This argument is supported by Figure 2 on after applying PAPRIKA, the individual task performance is improved. One thing probably need to analyze more is that whether it shows model's better capability in solving the specific task, say the direct mapping $x\\to y$, and not learning the effective method of solving $y$ given $x$. It would be great if the authors could provide more detailed statistics on the interaction trajectory? Is the model luckily committed to golden answer without effectively exploring the space first? \n- PAPRIKA can teach LLMs generalizable strategies. I feel the LOO results is interesting and it does show that model's better generalization performance. One thing I am concerned is that are the \"unseen\" tasks are still within the same domain somehow? In these environments, are them mostly about the action is composing different questions, and the answer is yes / no? It would be great if the author could comment more on the difficulty and similarities of these tasks? I noticed Fig 3 only shows 5 domains, how about the other 5? I did not find them in Appendix as well.\n- Curriculum learning can improve data efficiency of PAPRIKA. This is mostly supported by Fig 4 compared with uniform sampling, and it shows curriculum is indeed helping, compared to random sampling.", "methods_and_evaluation_criteria": "- The benchmark datasets are pretty nice, and it captures more complex domains that compared to multi-arm bandit. However, as I pointed above, it would be great to have a direct comparison on these tasks, how fundamentally each task is different than the other, or is it more semantically different observation / action spaces?\n- The metric used in the paper is mostly \"best-of-4\", why we need to report this best of $n$ performance? Can we report more metrics for better understanding? Say if we only sample once, and best of $n$ for multiple different $n$'s.\n - As I pointed before, LOO experiments in Figure 3 is missing some domains?\n- The whole paper is only experimenting on the same model, llama 8b, it would be great to show the performance of PAPRIKA on multiple models, preferably different sizes, and by different organizations, might show the effectiveness of the proposed method.\n- Can we list some example interaction trajectories in the Appendix for all domains? It would be great to get more detailed understanding on the complexity of the task.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "See Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "Related with the broader RL + LLM community, and probably agents as well.", "essential_references_not_discussed": "This would be relevant as well: LLMs Are In-Context Bandit Reinforcement Learners [https://arxiv.org/abs/2410.05362]", "other_strengths_and_weaknesses": "Strength: The paper is a very nice read, and it works on a very important problem, basically how to teach LLM strategic exploration and decision making capability. The proposed method is intuitive, and it shows strong generalization performance on unseen tasks. \n\nWeakness:\n- The limited experiments are only done on one specific model, might be good to include more results from diverse models. \n- The evaluation metric and some missing results from what I mentioned above.\n- More detailed analysis on the interaction trajectories to understand more on the improvement of the exploration capabilities.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v9JjXWVNYn", "reviewer": "Reviewer_WJpk", "summary": "This paper presents Paprika, a finetuning method that enables models to perform in-context rl for unknown environments. Different aspects of Paprika have been studied under different settings but not for this particular setting e.g multi-turn, not interacting with a human and more general environments. A training dataset is constructed using different LLMs for  text-based tasks. The diverse tasks have  with different  types of environmental feedback. There are two important aspects of training/fine-tuning. First the model is trained using a combined DPO and SFT loss. The second which is unique to this work is the scalable online curriculum learning to actually construct the training dataset. The coefficient of variation is used to measure the learning potential of a particular and a multi-arm bandit algorithm is used to actually make the task selection (for the training dataset). Experiments cover several aspects with the main focus on if training on n-1 tasks can lead to good performance using RL ICL on the unseen nth task.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the support and thoughtful review. We want to address their feebacks below.\n\n> There is not really an ablation (on the different parts of the method)...\n\nWe have conducted additional experiments on different parts of Paprika, we list them below:\n\n1. **Ablation on training data generation**: Since our training data is generated using high temperature (1.5) and min-p sampling with parameter 0.3, we have run an ablation to show the importance of both. On twenty questions, we see that either using a lower temperature or lower min-p sampling parameter leads to lower coverage on the training set, see [this Figure for varying Min-p at a fixed temperature of 1.5](https://ibb.co/mVsnTSsy), and [this Figure for varying temperature at a fixed Min-p parameter of 0.3](https://ibb.co/m547KZH7). The training data also affects downstream performance on held-out tasks, see [this Figure](https://ibb.co/fVYTxb1g).\n\n2. **Ablation on fine-tuning stages**: We ran an ablation on the two different parts of our fine-tuning: namely the SFT and the RPO stage, please see [this Figure](https://ibb.co/nMtyKGXJ).\n\nIf the reviewer has any other ablation in mind, we are happy to add it to the paper.\n\n> For example, is there a way to measure how ‘correct’...\n\nWe thank the reviewer for this great question. In general, we found that it is hard to intuitively understand the curriculum or argue whether it’s “correct”. However, we note the following:\n1. **Performance of the final fine-tuned model**: This is the final metric we care about. We have run our curriculum algorithm for 3 seeds and 3 rounds (with 250 tasks sampled at each round) and have an updated version of Figure 4 [here](https://ibb.co/gLn5wGPQ), it shows that the model trained on tasks selected by our curriculum outperforms that trained on uniformly selected tasks by 1.4% and 3.3% at average and pass@4 success rate respectively, demonstrating its efficacy in selecting training tasks.\n2. **Distribution of selected tasks**: This is the other metric that is easy to understand. First, see that our defined metric for learning potential (with Llama-3.1-8B-Instruct as the policy) has an intuitive distribution over the gpt-4o-mini defined easy, medium and hard categories — easy tasks have a higher learning potential compared to medium and hard ones. \n| Category | Average Learning Potential Metric |\n|-|-|\n|Easy|0.22 |\n|Medium| 0.16|\n|Hard| 0.09|\n\nNext, notice the distribution of easy, medium and hard tasks within 20 questions: we have 477 easy, 727 medium and 296 hard questions. **Uniform sampling** respects this distribution, and samples **more medium tasks as opposed to easy tasks**. However, our curriculum algorithm follows the learning potential metric, and **samples more easy, then medium and finally the least number of hard tasks** in a sampled batch. In a batch of 250 questions, we observe the following distribution:\n\n| | Easy | Medium | Hard |\n|-|-|-|-|\n|Uniform |78 |120|52|\n|Our Curriculum|117|79|54|\n\nThis shows that our curriculum has an intuitively reasonable behavior. If the reviewer has any other questions/thoughts, we would be happy to address them.\n\n> This is noted by the authors and above but the main weakness…\n\nWe agree with the reviewer’s comment. To extend Paprika, one would ideally use another LLM to keep generating diverse tasks that require strategic exploration, and use the curriculum algorithm to adoptively choose which tasks to train on. We believe that this is an exciting future direction.\n\n> It could be helpful to have a short discussion on the limits of generalizability of the tasks.\n\nWe thank the reviewer for this suggestion. We would edit the paper to discuss this limitation more clearly. We will emphasize the suite of tasks is diverse but it remains to be seen whether the resulting model will generalize to more temporally extended and real-world tasks.\n\n> How were the particular tasks chosen and which tasks were rejected?\n\nSome of the task groups we work with have been studied before, such as bandits [1, 2], 20 questions [3], guess my city [3] and wordle [3]. We expand upon these prior works and generate more training and test examples by prompting gpt-4o-mini with techniques from GenQA [4]. For the other tasks, we looked for tasks similar to the ones before, and GPT-4o-mini suggested battleship, minesweeper and mastermind. We came up with the cellular automata task as a toy example for iterative coding tasks with an interpreter. Finally, customer service and murder mystery are the open-ended tasks we could think of that have partial observability and fits nicely with other tasks. **We wanted as many tasks as possible, so we did not reject any tasks.**\n\n## References\n\n[1] Can large language models explore in-context?\n\n[2] EVOLvE: Evaluating and Optimizing LLMs For Exploration\n\n[3] LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models\n\n[4] GenQA: Generating Millions of Instructions from a Handful of Prompts", "claims_and_evidence": "The main idea/claim of the paper is that LLMs can be trained to perform ICL RL through the finetuning method presented (Paprika). There is not really an ablation (on the different parts of the method) so it is hard to tell which parts are most important but the experimental results do show the effectiveness of Paprika as a finetuning method.", "methods_and_evaluation_criteria": "Given the particular task setup, the evaluation criteria makes sense. Also standard evaluation criteria are used in Section 4.1 which allow for comparisons to other methods.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design for both the task specific setting (training for ICL RL) and general setting are sound and valid. One weakness which is mentioned by the authors in the conclusion is that a lot of human effort is required in making the dataset so generalizability could be an issue.", "relation_to_broader_scientific_literature": "The related works section shows how Paprika fits into the broader literature. In particular, Paprika is focused on finetuning method for ICL RL generalization on multi-turn tasks. This is not a commonly used setting making it unique from other works.", "essential_references_not_discussed": "Not that I know of", "other_strengths_and_weaknesses": "Strengths:\n- Task setup: The task setup fills a hole in current RL agent work and is likely an important future area. It broadens the potential applications of agents to more real-world like scenarios rather than toy tasks.\n- Generalizability: The Leave-One-Out results are particularly impressive given how different the tasks are.\n- Efficiency: Showing that curriculum learning improves efficiency is also important given the high cost of trajectory generation. \nWeaknesses:\n- It is hard to measure different aspects of the method in part due to the exponential search space. For example, is there a way to measure how ‘correct’ the selected curriculum is? Maybe some qualitative examples to get an idea of the curriculum would be helpful.\n- This is noted by the authors and above but the main weakness is the amount of human expertise involved in dataset generation. For example, humans specifically chose the tasks to make up the training set. While the tasks are diverse, it is unclear what criteria should such a training set have so that it can generalize. It could be helpful to have a short discussion on the limits of generalizability of the tasks. \nOverall, this paper is a strong contribution to the overall research community. It addresses an important question and contains important empirical results that others can build on. While there are some weaknesses, the task itself is challenging and this presents a good first step towards tackling it.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How were the particular tasks chosen and which tasks were rejected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ft3oICz82S", "reviewer": "Reviewer_WvQu", "summary": "This paper proposed a fine-tuning approach named PAPRIKA to improve LLM Agent’s decision-making capabilities, along with a curriculum learning algorithm that improves PAPRIKA’s sampling efficiency.\nThe experimental results show that the proposed PAPRIKA approach improves the success rate among different tasks by a large margin compared to the original base model. And the curriculum learning algorithm further enhanced the Agent’s decision-making capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and positive review.\n\n> Q1: In Algorithm 1, the pseudocode said that there is an input $B$ …\n\nWe thank the reviewer for catching this typo. $B$ in the pseudocode has the same purpose as $C$, in the sense that we collect $C$ trajectories in parallel for each of the $T$ tasks (selected by the algorithm). The collection of the $C$ trajectories is parallelized across multiple GPUs. We will edit our paper to reflect this more clearly. \n\nOne can in-practice also parallelize the data collection across multiple tasks (beyond trajectories for a single task as we do) with more computational resources. In this case, a batched variant of UCB [1, 6] can be used to update the arm parameters, which we leave for future work.\n\n> Q2: In Section3.3, the SFT objective and RPO objective…\n\nWe apologize for the confusion. In section 3.3, we define loss functions over any dataset $D$, and do not differentiate between the dataset used for SFT vs RPO stages. We will update the paper to clarify our notation.\n\nHowever, we also want to provide some clarifications here: for the SFT phase, we collect **all** successful trajectories for each training task $\\tau$. During the RPO stage, for each training task $\\tau$, we pick the best trajectory and one of worse-scoring trajectories randomly to form **exactly one** preference pair. We sample 20 trajectories per task, a large number of which can be successful and included in the SFT phase, but we construct at most 1 preference pair per task. The reason we only form one preference pair per task is because we observed that using more than one per task can cause unintentional unalignment [2, 3], and lead to performance degradation. \n\n> Q1: In Section 4.1 Finetuning on regular multiturn data does not help…\n\nFine-tuning on the full WildChat-1M dataset takes close to 10 days with our computational resources and we could not run that. Moreover, certain trajectories in WildChat have very long context length (> 100,000) and we do not have the resources to train on such long trajectories. Also **note that we only ~20K trajectories for Paprika**, showing that targeted synthetic trajectory generation is much more sample efficient for the tasks we care about in this work.\n\n> Q2: …but there is only one test case in your Bandit best arm selection…\n\nApologies for the confusion. Even though we have only one test case (e.g., picking colors), **we test our models on 100 iterations of the test case, each time randomly sampling the bandit arm probabilities**. For each specific bandit arm probabilities, we run 4 iterations of the game, and report the average pass@4 success rate to be 100%. We would edit the paper to make this point clearer.\n\nTo follow the reviewer’s suggestion, we generate **20 more bandit tasks** (so, 20 problem description + different arm names) using gpt-4o-mini and test Paprika on them, with pass@4 success rate reducing slightly to 98%, still outperforming the regular instruct model (See [this Figure](https://ibb.co/xtnt563Z), first row, first column)\n\n> Weeknesses: Scalability Concerns…\n\nSampling trajectory is indeed the main time bottleneck. However, we believe this is not too different from other RL tasks. Sampling is the only way we can get experience from the model, and it is still considerably cheaper than collecting human demonstrations. Recent works have found similar approaches to be very effective for other domains like math or coding [4, 5]. Collecting expert data and doing domain specific pre-training or mid-training can improve sample efficiency during the RL phase [7]. Another scalability concern is designing particular tasks for training, and a potential solution is to train another LLM to generate these tasks and then use our curriculum algorithm to select which ones to train on at every step.\n\n## Additional Experiments\n\nWe have also run additional experiments as per the other reviewers’ suggestions:\n\n1. Paprika with Gemma-3-12B-IT: Our experiments show that Paprika works with Gemma-3-12B-IT as the base model. We also see the **Paprika fine-tuned model being comparable or better than GPT-4o-mini in 7/10 task groups**. See full results [here](https://ibb.co/dsv8xG3q).\n\n2. More tests for generalization: We extend Figure 4 by running leave-one-out experiments on 5 more task groups, and Paprika (LOO) improves over the instruct model in 9/10 task groups, showing strong generalization: [Figure](https://ibb.co/xtnt563Z)\n\n## References\n\n[1] Multi-Armed Bandit Problem and Batch UCB Rule\n\n[2] Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization\n\n[3] Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data\n\n[4] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[5] OpenAI o1 System Card\n\n[6] Perchet, Vianney, et al. \"Batched bandit problems.\" (2016): 660-681\n\n[7] Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars", "claims_and_evidence": "Yes. The claims made is supported by evidence.", "methods_and_evaluation_criteria": "yes. \nThe proposed method is evaluated by benchamrk dataset.", "theoretical_claims": "Q1:\nIn Algorithm 1, the pseudocode said that there is an input B for parallel experience collection, but in the pseudocode, there is no evidence show that the collection process is parallel. What is the intent for B.\n\nQ2:\nIn Section3.3, the SFT objective and RPO objective formulations seem to express the training set for SFT and RPO are the same. But in Experimental Setup, here written 15,744 training trajectories for SFT and 4388 trajectories for RPO, so it seems that they are different training set. If different, there should be an explicit difference in the both objective formulations in Section 3.3.", "experimental_designs_or_analyses": "Q1:\nIn Section4.1 Finetuning on regular multiturn data does not help, you fine-tuned Llama-3.1-8B-Instruct on 100,000 English language trajectories sampled from WildChat, why not the whole dataset?\n\n\nQ2:\nIn Section 4, PAPRIKA improves LLM decision making abilities, here written when include trajectories from the bandit task in training set can increase rate to 100%, but there is only one test case in your Bandit best arm selection game. That seems not so convincing, why not some more test case?", "relation_to_broader_scientific_literature": "1. This approach addresses limitations highlighted in prior work—such as poor performance on even simple multi-armed bandit tasks (Krishnamurthy et al., 2024)—by training the model on diverse decision-making tasks that require iterative reasoning.\n\n2. In-context learning allows models to adapt to new tasks from a few examples. PAPRIKA leverages this capability and combines it with reinforcement learning principles, effectively training the model to perform in-context reinforcement learning across multiple rounds of interaction.\n\n3. One of the  notable findings is that PAPRIKA-trained models exhibit strong zero-shot performance on unseen tasks. The demonstrated transferability highlights the paper’s contribution to bridging the gap between task-specific training and broader, more adaptable decision-making capabilities.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\nOriginality: The paper presents a framework, PAPRIKA, that combines synthetic trajectory generation, a sequential variant of Direct Preference Optimization (DPO), and curriculum learning. This combination introduces a new way to train LLMs for multi-turn decision-making tasks, which distinguishes it from previous approaches that focus mainly on single-turn interactions.\n\n\nWeeknesses:\nScalability Concerns: Generating large volumes of synthetic trajectories can be computationally expensive. The paper could benefit from a discussion on how this scalability challenge might be addressed, especially when extending the framework to more complex or real-world environments.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No. please refer to above questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors propose Paprika, a fine-tuning approach that is designed to teach LLMs to seek information to solve sequential decision-making problems. The core contributions include:
- A dataset of ten text-based games, each game has thousands of task variants. The games require LLM-based systems to intereact with the games in a multi-turn manner to collect necessary information to win. 
- On top of the SFT + DPO model training, the authors propose a unique scalable online curriculum learning strategy. In which, the coefficient of variation is used to measure the learning potential; a multi-arm bandit algorithm is used to select training tasks.

The reviewers gave high initial scores (all positive, with an average of 3.5), the concerns are mostly clarification questions. The authors did a great job to address the concerns. In fact, the authors have successfully convinced two reviewers to further increase their scores, resulting an average score of 4. Some of the salient discussion points are:
1. The notion of `curiosity`. As the authors acknowledge in their rebuttal, their notion of curiosity is different from many RL literature, especially those of `curiosity-driven exploration` and `intrinsic motivation`, where the exploration policies' goal is to discover as much as possible the map or the game dynamics, even when the discovery may not have direct contribution to solving the task. Here, the exploration is a `amortized` version (citing the authors' word). My understanding is that the agents are required to perform an info-seeking behavior to solve the tasks because memorizing past success trajectories won't work, the `curiosity` lies in such info-seeking behavior. I encourage the authors to discuss in their camera-ready, what exactly do they mean by a `curious agent`. It is weird that the word `curious` only appeared twice in the paper, once in the title, once in the conclusion.
2. The authors added additional experimental results, including metrics other than "best-of-4"; experiments using other LLM models; additional ablation studies on training data generation and fine-tuning stages; additonal evidence on the effectiveness of the curriculum; and additional qualitative analysis.

Additional AC comments:
While I agree the 10 text-based games designed in this work are valuable, please note that text-based game / text-adventure game is an active research area. For example [1,2,3,4] are human-authored and synthetically genearted games that also satisfy the authors' criteria:
1. they are purely text based;
2. they require multi-turn interaction;
3. they are partially observable;
4. they require diverse strategies to win.

I refer the authors to [5], a good survey paper summarizes existing text-based games research effort, as a starting point to discover this area. The more I read this submission, the more I feel existing text-based game suites could greatly enrich Paprika's info-seeking capabilities. 

Overall this is a strong paper with clear contributions, all reviewers are excited to see this work targeting generalizability of LLM-based agents on sequential decision making problems, and the curriculum learning method bringing efficiency to such learning procedure. I recommend to accept. 

References

- [1] Interactive fiction games: A colossal adventure. Hausknecht et al., 2020.
- [2] Textworld: A learning environment for text-based games. Côté et al., 2018.
- [3] ScienceWorld: Is your agent smarter than a 5th grader? Wang et al., 2022.
- [4] Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents. Jansen et al., 2024.
- [5]  A systematic survey of text worlds as embodied natural language environments. Jansen, 2021.
Decision: Accept (oral)