Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
Authors: Sanjiban Choudhury
Paloma Sodhi
Venue: ICLR 2025 Poster (2025)
Keywords: learning from AI feedback
imitation learning
privileged information
Primary area: foundation or frontier models, including LLMs
TL;DR: An iterative fine-tuning framework where AI expert feedback with privileged information helps weaker models surpass stronger teachers without privileged test-time access

Abstract: While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state -- information available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time.
We evaluate LEAP on multiple decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed performance of strong teacher models (GPT-4o), and (3) allows weak models to self-improve using privileged versions of themselves. 
We provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with student’s realizability, which we empirically validate. Our code is available at \url{https://leap-llm.github.io}.

Published as a conference paper at ICLR 2025
BETТER THAN YOUR TEACHER: LLM AGENTS THAT LEARN FROM PRIVILEGED AI FEEDBACK
Sanjiban Choudhuryl: !,, Paloma Sodhi"
ABSTRACT
While large language models (LLMs) show impressive decision-making abilities, current methods lack mechanism for automatic self-improvement from errors during task execution. We propose LEAP, an iterative fine-tuning framework that continually improves L.M agents using feedback from Al expert teachers. Our key insight to equip the expert teachers with a privileged state information available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent's performance without access to privileged information at test time. We evaluate 1 LEAP on multiple decision- makinn benchmarks, including text- based games (ALFWorld), web navi- gation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed performance of strong teacher models (GPT-4o), and (3) allows weak models to self-improve using privileged versions of themselves. We provide a theoretical analysis showing that LEAP's success hinges on balancing privileged information with student's 's realizability, which we empiri- cally validate. Our code is available at https: /tt//leappp/p/ 11m github. io.
INTRODUCTION
Large language models (LLM) show impressive decision-making abilities across domains (Yao et al., 2022b; Jimenez et al., 2023; Sodhi et al., 2024). However, they struggle to learn and recover from errors encountered at test time (Madaan et al., 2024; Shinn et al., 2023). Prompting alone is insufficient, as it requires manually specifying exceptions and examples (Khot et al., 2022), which is difficult scale and leads to intractably long context lengths (Liu et al., 2024).
We address the problem of fine-tuning LLM agents to improve their decision decision-making using online interaction data. Imitation learning offers a sample-efficient way to train such models, where teacher corrects the student agent's 's actions (Ross et al., 2011). 1). However, finding an expert capable of both demonstrating the task and correcting mistakes from any state student visits extremely challenging (Chen et al., 2020). The expert must not only know how complete the task but also how to recover from the student's errors, which is often more complex than simply performing the original task (Walsman et al., 2022). For example, in a task like "Heat the mug and put it in the cabinet," the student places the wrong object in the microwave, the expert must first backtrack and remove the incorrect object before proceeding with the task.
Our key insight is to equip the expert teacher with access to a privileged state - information useful for solving the task but only available train time (Vapnik et al., 2015). This includes hidden states of a partially observable environment or intermediate subgoals. For a task "Heat the mug and put it in the cabinet", the privileged state reveals the mug's 's location and steps for heating it (Fig. 1). With this knowledge, the expert first clears the microwave and then teaches the student effective search strategies, like checking common mug locations. Over time, this enables the student to learn general strategies for solving the task and recovering from errors, ultimately outperforming the expert, even without access to the privileged state test time.
We propose Learning from Experts with Access to Privilege (LEAP), an iterative learning algorithm that fine-tunes L.LM agents using privileged Al expert feedback. As shown in Fig. LEAP begins
Coonell University, NY, USA. "Equal contribution. Correspondence to: Sanjiban <sanjibanccooomlllccooo Paloma <paloma.sodhie com>

---

Published as a conference paper at ICLR 2025
Task: Heat mug and put it in cabinet
LLM Expert Teacher x²
resson yy nnooo find fras. eg
reason
2+0
objective
You given trajactory containing clueese Lina, zeason, actioon geserated by studant You are teacher with accans to containing that hidden deon erodent Your goat io to
Student
or "Yuu piók mag
ILM Student
observation
Environment (6 (F.))
trajectory rollouts (A))
shat ressen it hare Tea mag."
"privillged
Agent Nj-1
Lapcore how the a budint anlewa the game ixprosting thate
SFT(U
Privsslged State N Exzential Objecte:
obserastion dng "-. you see nothing"
resson Fy nood find
mose etcieet to check pasce uheen magg are commsssyy found ahoinnt of outemubl action "go
fra,
(co beat
Morteefer for fo Optite '
Cricical Zocations:
IwQ
aholr [to reteeeve sugt siccoware (ca beat sual
Model Training (SFT/DPO)
corrections
Expert
o up from 37 rexson "1 nead
Oarget
"Lime Lup":
toriginal wollim"
Dataset 2
for may EESDARE aboelts??
"our and psatonns
action
"coerected sction":
Figure 1: LEAP overview. L.M student agent interacts with the environment, generating a reason-action trajectory (in orange) based on its policy Ti-1. An expert teacher, with privileged state available only during training, evaluates and corrects the trajectory (in green). These corrections update the lcarner's policy m, through SFT/DPO training. Updated policy m, is then rolled out at test time without access to privileged state.
with a student agent trained demonstration data (TT). During interactions, the student agent generates trajectories, and the privileged expert B) identifies key time steps where corrections are needed. The expert provides improved reason and action labels based on the privileged state, which are then assembled into a dataset used to update the student model (T, + T;-1). This process is repeated iteratively, progressively improving the student agent's 's performance.
Interestingly. we identify a key trade-off between leveraging privileged information and the student agent's 's realizability. While providing unrestricted privileged feedback allows for highly accurate corrections, these are often too complex for the student to act on effectively. Conversely, limiting feedback what the student can easily process makes it realizable but often suboptimal. Our theoretical and empirical analyses show that balancing these extremes is key, which we achieve by introducing a constrained privileged expert that provides optimal yet realizable feedback.
Our key contributions are:
1. A novel iterative learning framework LEAP that fine-tunes LLM agents using privileged expert feedback, balancing privileged information and realizability.
2. Theoretical and empirical analysis of the optimal trade-off between privileged information and agent realizability.
3. Experimental validation on diverse interactive decision-making benchmarks: AlfWorld (Shridhar et al., 2020b), WehShop (Yao et al., 2022a), and InterCode (Yang et al., 2024).
(a) LEAP consistently outperforms behavior cloning agents trained on gpt-4o or human demonstrations, showing significant improvements ALFWorld (65.7% 91.8%), Web- Shop (29.4- - 61.8), and InterCode Bash (60.3% + 71.8%).
(b) LEAP enables weaker models (e.g. 3-8B) surpass stronger models (gpt-4o), with improvements on ALFWorld (91.8% vs. 65.7%), WebShop (61.8 vs. 58.4), and InterCode Bash (71.8% vs. 71.3%).
(c) Ablations that show LLM agents can self-improve using their privileged versions as teachers (65.7% + 82.1 2)).%). explore the tradeoff between privileged information and realizability, and compare supervised finetuning with preference optimization.
2 PRELIMINARIES
Partially Observ- able Markov Decision Process (POMDP) (Kaelbling et al., 1998) because real-world environments often contain hidden incomplete information. The POMDP is defined by the tuple (S. A. O, T.r), where S represents the underlying state space of the environment, A the action space, O the observa- tion space, T : S A + S the transition dynamics, and S IR the reward function. The
Problem Formulation:
Decision-making LLMs. We frame decision-making
2

---

Published as a conference paper at ICLR 2025
true state st E S is not fully observable; instead, the agent receives partial observations or O each timestep. Hence it must rely on the history h2 E H. of observation observation-actions -cciions to choose actions.
II, which predict both reason Pr and an action da based on the agent's history h4. The reason pr is expressed in natural language to justify action dz. The history hy 01, -1,...,0 ot is the sequence of past observations and actions. The policy is (pt. (2) x (h+)- While reinforcement learning (RL) for POMDPs is already challenging due to the large history space H (Papadimitriou & Tsitsiklis, 1987), this is further exacerbated by the addition of reason space. which is infinite and unconstrained. This makes RL impractical to run for LLM agents.
LLM-based agents policies
Interactive Imitation Learning with DAGGER. Imitation learning (II.) offers promising alternative to RL, by allowing the agent to mimic an expert teacher policy π bypassing the need for complex exploration. The gold-standard for imitation learning are interactive methods like DAGGER (Ross et al., 2011 1) that query the teacher π for on-policy corrections on states h2 visited by the student. However, for POMDPs, computing such corrections is intractable and strictly more challenging than just solving the original task, often resulting in suboptimal feedback.
3 APPROACH
We present Learning from Experts with Access to Privilege (LEAF), framework for iteratively training an LLM agent through online AI feedback from a privileged expert (Fig. 1). Central to our approach is the concept of a privileged expert teacher, which leverages access to privileged state information to guide the student's ' recovery from errors. The goal to first train the agent to solve the problem using this privileged state, but then distill the policy into student that works from observations alone. Such frameworks have been used extensively in robotics for training robots in simulation and deploying in real world for applications like self-driving and legged locomotion (Sec. 5). We build on this paradigm, and propose novel extensions in the form of constrained privileged experts to adapt this for training LLM agents. Section 3.1 defines the privileged teacher, Section 3.2 details LEAP, and Section 3.3 presents the theoretical analysis of LEAP.
3.1 PRIVILEGED EXPERT TEACHER
We introduce a privileged expert teacher, policy that has access to privileged state information 8 which fully describes the environment at time t. This access transforms the expert's 's recision-makinn from POMDP into fully observable MDP making policy learning more tractable. Formally, the teacher policy takes the student's ' reason Pt. action at, history h,, and privileged state s and generates a corrected reason action (pe, āz) ~ π 5 (-lpe, at, h4, st).
The privileged information $; can come from multiple sources:
1. Simulator States: Many agentic domains train in simulated environments before real-world deployment, where the simulator exposes underlying states. For instance, in AlfWorld (Shridhar al., 2020a), agents observe only their location, but the simulator provides full item locations as privileged information during training.
2. Evaluation Criteria: Most domains have success metrics. In WebShop (Yao et al., 2022a), privi- leged information includes product attributes, options, and target prices, while in InterCode (Yang et al., 2024). it includes goal commands and unit tests.
3. Human Annotations: Privileged states can be extracted from human demonstrations, In AlfWorld, key subgoal details (e.g., "drawer contains book 2") come from demonstrations. Similarly, dialog agents leverage conversation-leww subgoal annotations, which easier to collect than dense demonstrations, making them a practical alternative.
crucial requirement for the privileged expert teacher that it must not reveal privileged information in its feedback to the learner. The prompt instructions specify that the teacher should provide general reason and action that avoid the direct use of privileged information. Using such information directly might solve tasks quickly but makes the corrected actions (pt. āz) unrealizable for the learner, who does not have access to the privileged state 8t, leading to a significant imitation gap and poor learner performance. In Section 3.3, we characterize the trade-off between using privileged information and student realizability. We show in Appendix. B that constrained privileged expert that effectively utilizes privileged information while remaining realizable for the student has the optimal performance. We approximate such an expert by designing prompts for the expert to instruct it to use privileged information, without revealing it. See Appendix E for prompts.
3

---

Published as a conference paper at ICLR 2025
Algorithm LEAP: Iterative Learning with Privileged Expert Teacher
Input: Privileged expert teacher x= Environment
Output: Learned policy и
1: Collect an initial set of demonstrations from expert teacher D + pi, pi)
2: Train BC To D using supervised fine-tuning (SFT)
3: for iteration i .., do
4: Roll out policy #i-1 in environment & to collect new data: D; t-
Pc, pt,
5: Compute privileged state s; for every datapoint in Dj
6:
Invoke privileged expert generate corrected reason and action: Pt, ūg n (.(ppes Ut, her 8t)
7: Augment dataset with corrected reason and action: D; <- {(he, Pt: Qt, Pr. ar)}
8:
Train policy иy using no-regret update: मं SFT (UD;) n; t DPO TT-1)
return Best π E (T1...., J on validation dataset
3.2 THE LEAP ALGORITHM
The LEAP algorithm iteratively trains the student policy т through online interactions with the environment & and corrections from privileged expert teacher π The process begins by collecting an initial set of demonstrations training a BC policy To on this data using supervised fine-tuning (SFT).
history, reason, and action from the teacher D < {h; Pł, aj }. and
At each iteration, the current learner policy æ, rolled out in the environment to generate new data D; < (he, Pt. at). For each timestep in D. LEAP computes the privileged state 8t by leveraging information available only during training. The privileged teacher is then invoked on trajectories where the agent fails, generating corrected reasoning and actions, Pr, ā. nh (.lpt, az fe,, 8t). These corrections are used to augment the dataset D,, The policy иN is subsequently updated using this augmented dataset through a no-regret learning method. This iterative process continues, refining the policy over multiple iterations until the best pest-performing policy is selected based on validation.
The update step in LEAP should be designed ensure no-regret learning, a property that guarantees the policy's ss cumulative performance will converge to that of the best possible policy in hindsight. The most common update is aggregate data UD,, D,, and train n, using supervised fine-tuning (SFT) on the aggregated dataset, as done in DAGGER (Ross et al., 2011). This is equivalent to Follow the Regularized Leader (FTRL), a no-regret update step. An alternate update to treat the problem as a preference optimization problem where the corrected reason action (p:, āz) is preferred over the student reason action (pr, Preference optimizers like Direct Policy Optimization (DPO) (Rafailov et al., 2024) or Kahneman- Tveesky Optimization (KTO) (Ethayarajh et al., 2024) optimize a preference loss while regularizing to the base policy, in this case the previous policy T;-1- This is equivalent to Online Mirror Descent (OMD) update, which is also a no-regret update. We use SFT by default but compare it against preference optimization as an ablation in the results.
3.3 ANALYSIS
We briefly touch on the theoretical guarantees of LEAP, and refer the reader to Appendix. B for full exposition. We define the realizability an expert policy and show how it affects LEAP performance. Definition 3.1 (Average Realizability Gap). The average realizability gap e(xT". T) between the privileged expert π and the best policy n² over time horizon T
F T) sup C
Σ Engbhrddyy (acls)) T" (ac(h))))
(1)
x
where [1-[h is the LI distance, di is the induced distribution over state and history. Theorem 3.2 (LEAP with Privileged Expert). Running N iterations of LEAP with privileged expert T yields at least one policy n such that the time-average performance + (Tr) is bounded as:
J(T))
J(M 1 J(mt) B) -H(#F)
(T +7())
(2)
performance
realizability
where T. is the privileged expert, H (TT is the recoverability coefficient of xF, e(T C.FFTT average realizability gap, and (NN is the average regret of the DAGGER update.
,T) is the
4

---

Published as a conference paper at ICLR 2025
The equation shows trade-off between performance of the privileged expert (first term) and the realizability of the privileged expert (second term), which we show how to optimize in Appendix. B.
4
EXPERIMENTS
4.1 OVERVIEW OF RESULTS
We evaluate LEAP across 3 decision decision-making domains: ALFWorld, text- based game, WebShop, web navigation environment, and Intercode, an interactive coding environment. Below is a summary:
1. LEAP improves upon base behavior cloning (BC) policy:
LEAP significantly improves the base BC policy To over successive fine-tuning iterations, showing gains on AlfWorld (65.7% + 91.8),, WehShop (29.4 - 61.8), and InterCode (60.3% - 71.8%). See Tables 1, 2, Fig. 3.
2. LEAP enables weaker student models surpass stronger teacher models: Over successive iterations, student models (e.g., Llama3-8B) outperform their stronger teachers (e.g.. GPT-4o), despite starting with lower or comparable performance. For example, on ALFWorld, the student model (Tg 91.8%) surpasses the teacher (a
75), with similar trends on WebShop (T2 : 61.8 vs. x 55.44) and InterCode (Tg : 71.8% vs. π 71.3%). See Tables 2, Fig. 3. 3. LEAP requires balance between using privileged information and generating realizable corrections: Experts using minimal privileged information (T : 73. 1%) generate more realizable actions, while those with excessive privileged feedback (N 5 5.22%) produce unrealizable corrections.
π. strikes perfect balance achieving optimal performance (Tg 91%). See Fig. 5. 4. LEAP enables LLM agents to self-improve by using their own privileged versions as experts: LLM agents can self-improve by using their privileged versions as teachers. For example, in ALFWorld. Llama3-8B bootstraps itself through iterations, improving from 65.7% 82.1% and finally to 91 91.8%. See Fig. 6.
5. LEAP can be trained using both SFT or preference optimization: Both SFT and preference optimization improve over the base BC policy (TO)), with SFT delivering stronger gains (68.6% - 96.4%) compared to DPO (68.6% + 74.3%) and KTO (68.6% + 87.1%) See Table 3.
4.2 DOMAIN 1: TEXT-BASED GAMES
Setup. ALFWorld (Shridhar et al., 2020b) is a text-based game where agents complete household tasks (c.g.. "heat mug and put in cabinet")) by navigating and interacting via text commands. Success requires planning subgoals, tracking progress, and efficiently searching for objects. Each task has 30 timesteps, spanning six categories, with training set of 3,257 games and two evaluation sets: 139 in-distribution and 134 out-of -iittii-iitiittiion games. We compare against prior works and extend ReAct (Yao et al., 2022b) with stronger models (gpt-4o', Claude", Gemini and general instruction prompt. Our approach fine-tunes Llama3-8B (Dubey et al., 2024) with LoRA (Hu et al., 2021), using gpt-4o 4o as a privileged expert teacher for 4 iterations. Privileged information includes essential objects, their locations, and optimal actions. See Appendix D.2, E. for details and prompts.
Results. Table shows that LEAP significantly outperforms all baselines, with the best policy achieving 91.8% success rate performance gain (65.7% + 91.0%), leading to a student policy Tj that surpasses the teacher gpt-4o with higher success rate (91.8% 65.7%) and lower actions (11.3 < 20.2). Note that performance improvements are not monotonic over iterations of LEAP, (65.7% + 91.0% + 83.6% 91.8%), however, all policies are better than the BC policy no. The regression from क to Ty are mainly from HEAT and LOOK tasks, which gets corrected immediately in the next iteration т3- Finally. we note that LEAP policies generalize well from in-distribution to diitr--disttiuution tasks with only a slight dip in performance for the final policy Ā3 (94.2% + 91.8%).
tasks. Iteration 1 of LEAP has the biggest
Fig. (a) shows an example of LEAP training and testing. A typical failure mode of To is that searches for items inefficiently. e.g. looking at shelves or drawers or cabinets one by one until time runs out. The expert teacher leverages the privileged state of where objects are to show the student how to search efficiently without revealing the object location, e.g. it's more efficient to search desks where cellphones are likely to be than drawers sequentially one after another. Training on this correction data, I generalizes this reasoning to new test-time tasks, e.g. putting a watch in safe. Fig. 2 (b) shows that while To inefficiently searches shelves, Tj searches likely locations for the watch, first the sidetable and then the dresser to solve the task. The main performance gains of LEAP over
hutp:///laaforrmoppennt wtwpaaa....o/occ///oooeeeeeeeela
5

---

Published as a conference paper at ICLR 2025
fasks
PICK takk
tasks
tosks
CooL
Loox tacks
tasks
Rauct
Fact]
Bauch
Rauct
Isuct
Raust
tauct
Nouat
BUTLER [1]
35.0
50.0
74.0
83.0
91.0
65.0
BeAust fow-shot [2]
57.0
65.0
39.0
83.0
76.0
55.0
24.0
Autogon apt-3,5 [3]
77.0
Eapel gpt-3.3
59.0
lexten gpt-5
88.0
75.0
90.3
91.3
90.5
88.9
94.1
Adehlanner gpt
100.0
100.0
gpt
91.7
$2.4
Rolct apt-42-mini
33.3
14,3
29,4
Relct clasct-b.3--omme
76.1
19.0
95.8
61.3
60.9
81.0
$8.9
76.5
Relet clauda-3,3-
16.4
27:1
33.3
9:7
8.7
9.5
38.9
0.00
ReAst Demintie 13-
19.4
26.3
41.7
12.9
13.0
19.0
16.7
11.8
LEAP C1ana3-m
0.7
29.3
0.0
0.0
0.0
4,8
0.0
0.0
LRAP 3. 88
65.7
18.6
66.7
74.2
73.9
66.7
66.0
35.3
LEAD S1ana3-88
83.3
LEAP 510.0-1mmmma3. 3-88
LEAP 21ama3--8
57.5
90.5
Method
tusks
PICK tasks
CLEAN tasks
HEAT tasks
COOL tasks
LooK tasks
PICK tasks
Nsuof
#wot
Raucf
Sauctt
Asucf
Rauct
Sauct
Sourt
ROT/PR [1]
40.0
76.0
60.0
KoAlt. gpt-to
51,3
22.9
91.4
33.3
31.2
12.0
84.6
66.7
ReAct opt-fo-mini
40.0
26.1
11.1
12.5
4,0
53.8
20,8
20.8
LEAF Llama3-88 BB
2.9
29.5
8.5
0.0
0.0
0.0
7.7
0.0
LEAE Liama3-3B
68.6
16.9
88.6
56.2
80.0
76.9
50.0
LEAF 112 m
96.4
10.5
LEAD 3- Rg
90.0
11.4
87.5
92.3
83.3
LEAF
L. Llama3-38 R3
94.2
10.3
100.0
96.3
100.0
$7.5
1000
83.3
Table 1: AlfWorld Evaluation. (top) 136 out-of-distribution games and (bottom) 140 in-distribution (max 30
actions). Baseline comparisons include [1] BUTLER (Shridhar et al., 2020a), [2] ReAct few-shot (Yao al., 2022b), [3] Autogen (Shridhar et al., 2020a), [4] ExpeL (Zhao et al., 2024). Note [5] Re flexion (Shinn et al., 2023) and [6] AdaPl Addallanne lanner (Sun et al., 2024) make multiple attempts on the same test task, while we do not. We also add our own REACT instruction prompt with different models. LEAP with an 8B model across iterations (T1, T2, T3)) outperforms the stronger teacher ReAct got-4o.
Zask: Find tae ocllphones
Liarer
Student Ko
riloout (2)4))
conrectionn (,)
Teacher 8
a00225 callchoees
Foord.
dankal
desks
SPONND Beccernially
desk3
Dataset generation at
Zack: Pot wacch on cafe
Lave water.
st to on to
Iritisl student
aldes hite
Ag rillout
asontable dons EGC. Noze BECOS bdoes coccalas wwwonble LEARRA
Conrected student Aj rollout
atdecabinl
solet
(b) Student policy test time
Figure 2: ALFWorld Training and Testing for LEAP. (a) Training: Student policy rolled out on training task to generate reason and actions, e.g. it fails cell phone because inefficiently searches all drawers. Expert teacher F uses privileged information (cellphone deskl) to generate general corrected reason actions that don't 't reveal privileged information (cellphoncs commonly found in desks, more efficicnt to scarch desks than drawers). (b) Testing: To fails to find a watch as it inefficently explores shelves one by one. Ti learns a more efficient exploration policy, prioritizing areas like sidetables and dressers, solving the task quickly.
gpt-4o come from such efficient exploration. gpt-4o searches likely locations, however, its prior from internet data and isn't necessarily aligned with the statisties of the ALFWorld environment. In contrast, LEAP leverages privileged feedback to learn this. See Appendix D for more results.
4.3 DOMAIN 2: WEB AGENTS
Setup. WebShop (Yao et al., 2022a) is online shopping environment where agents fulfill user requests via search and click. Performance is based on attribute, option, product type, and price matches, with partial credit for partial matches. Each task has 30 timesteps, with 12,086 training and 500 test tasks. We compare against ReAct (Yao et al., 2022b) (GPT-40. GPT-4o-mini and the IL baseline (Yao et al., 2022a), which trains separate search and click models. Our approach fine-tunes
6

---

Published as a conference paper at ICLR 2025
gpt-lo goi-ko-mini
Method
sooref
taot
[1]
59.9
50
Option
20
ReAct gpt-4o
58.4
8.0
25
ReAct gpi-4o-mini
51.0
11.3
ReAct claude-3-sonnet
44.4
11.4
ReAct gemini-flash
46.3
8.6
LEAP Ierationa
LEAP Iterations
LEAP 3-bb
1.2
29.9
LEAP Llama3-Bb ioo
29.4
21.1
-50
50
Type
Price
LEAP Llama3-Bb m
51.8
16.7
LEAP 3-Bb ie
61.8
11.6
LEAP L-ama3-8b ma
60.7
11.9
11o
FTL
112
123
1Tg
TT:
122
113
LEAP Iterations
LEAP Iterations
(+)
(b)
Figure 3: WebShop Evaluation. (a) Overall scoret and #act. on 500 test tasks (max 30 actions). (b) Performance of LEAP over iterations on different score components. Baseline comparisons include [1] IL (Yao et al., 2022a) and our ReAct instruction prompt with different models. LEAP with an 8B model across iterations (T2, Ta) outperforms the stronger teacher ReAct qpt -0.
Data point
Task need gtuten foes vegotartan esciced peppaced bacso cunce tpack of 21, and price Lover than 52.00 dollace
Dask zu0king
thes 23.2 El
Stodest rollost An by
Intiaal shudent
Conected student
... resson saed check use pesturer
Porolout
rolkut
commiti action slack [nacur Teachez correction Pe-dy
actADo
swazchiggauten Liue
action do vegetavinn lgack of
cnarhhigtater free
eugularids anoked pappered
pappered
flad thei
inntesntion. actioo By clock[bsck to aearch]
obearvation my Sooked Sason
obsesvation Uy Dappored Becon
[butron] The features provide
reeson have selactes Uhe
Data point
more inforration about the product, Sut page
back
ounce pack ot pappeced becin,
sull not paok
Teak Sanothez Lupk?
Student rolloot No
dide's
action elloklouck to
1eapoe P seuiil back Lae previcus
ireestant dessild.
action sy clickts poey
Uhe oriterias actson
pask instructionts recuizerent of sat of action oy clickge
match
Teacher oscrection Ps-5y
cesson should consinue
is BerS Brotsten trefricleney. should
navigacing
ESa nexe page to Find aoca sesiend action 2g click/nast >]
jpack 211
Exis item. actlon 9,. olicxiouy non)
acsion clook(newt >]
Action vg. olicklou novl
(a) Dalaset generation at train time
(b) Student policy at test time
Figure 4: WebShop Training and Testing for LEAP. (a) Training: Teacher policy generates corrections on student rollout backtrack when the product does not fit criteria or to commit product when it does. (b) Evaluation: no fails to solve task since it continues to search page after page even after discovering a good product. Wi learns when backtrack and when to commit to a product to solve the task in time.
gpt-4o Llama3-8B with LoRA, using gpt 4o as a privileged expert teacher. Privileged information includes product attributes, options, price constraints, and example product. See Appendix D.3, E.2 for details.
Results. Fig. 3 (a) shows that LEAP outperforms all baselines, with the best policy (T2) achieving 61.4 score. Iteration has the biggest performance gain (29.4 - 51.8) followed by iteration (51.8 + 61.8). This leads to a student policy Ty that surpasses the teacher GPT-4o with a higher score (61.8 58.4). Similar ALFWorld, while performance improvements are not monotonic, both T2, F3 remain above the teacher gpt 4o performance. Fig. (b) shows how LEAP improves on all components of the score. The biggest improvements between LEAP and gpt-4o in OPT component (42.4> 22.1), indicating gpt 4o fails to select a product with the right option often.
Fig.4 (a) shows LEAP's ' correction of mo and πı using training data. The teacher leverages privileged information (c.g., product attributes options) to guide the student in recognizing when to backtrack on mismatched items and when to commit to a suitable product. Training on this correction data produces policy π2, which balances backtracking and committing effectively. Fig.4 (b) highlights To's 's failure to commit, endlessly cycling through pages, whereas T2 successfully backtracks and selects suitable item complete the task.
4.4 DOMAIN 3: INTERACTIVE CODING
Setup. Intercode Bash (Yang al., 2024) is an interactive coding environment where agents solve filesystem tasks (e.g.. "Find all text files and write their names to a single file") using Bash. Tasks are drawn from four NL2Bash (Lin et al., 2018) datasets; we train on the first two and test the next two, with a 10-step time limit. We compare against ReAct (Yao et al., 2022b) using gpt-4o and gpt-40-m ni, well as the top three entries on the Intercode leaderboard -T--Trr TrryAga baselines
7

---

Published as a conference paper at ICLR 2025
all tasks
pick tasks
clean tasks
heat tasks
cool tasks
look tasks
pick tasks
100%
81.00%
20
Number of actions
80%
%6 Success
60%
15
40%
20%
10
11.00
0%
x1
#3
Level of privileged information
Level privileged information
(a)
Daak Pi.. Lvo katillw in cabinet
Privileged Lonstiona's
("eesertial objecta"i
(kattie 2), stevmssrnns (bactle
"lettle
ket:le 25, feritioal
Correction at 10% privllgee state]
X
Conrection from
Correction x 110% privleged stale)
Teacher 1LEA2)
Deachez consection
Deachaz correction
zeason A
Do beçin the peazch
coason Jo need to uxplecn the Lonations that Fiee more
resson A
ELeat
Sbees
efficienn ent by, shoold start by
1o
look For the
yeing to places abare
have Luu.a...
corntartope.
action to sabiinl
reason iy Bravers are less 1.kely to costain bettle. I67a zare
reason Drances are unlikely to costain kattlen. Io's battes
reason
Should Nead straight
contains
actsoon pahlnet
action
Action
(b)
Figure S: Privileged Information vs Realizability. (a) Performance policies trained with experts with varying levels of privileged information on ALFWorld, peaking for expert n.. (b) Examples of corrections from experts R F. F कं generates realizable reason action but predicts wrong action. R predicts correction action, but produces unrealizable reason action that contains privileged information. x." strikes perfect balance.
(Yang et al., 2024) with gpt-4, gpt-3. 3. 5, and CodeL 34B-INST (Roziere et al., 2023). Our approach fine-tunes Llama-3. --BBB and Llama-3.1 1-0BBBBBBBBBBB--- with LoRA for two iterations, using gpt-4o as a privileged expert teacher. Privileged information consists of the ground truth Bash command.
Table 2 shows that LEAP trained with 70B policy outperforms all baselines, with pol- icy गi achieving 71.8% success rate to match the gpt-4o expert teacher (71.8% > 71.3%). We find that on this coding benchmark, the gpt-4o ex- pert is a very strong baseline. LEAP trained with SB models improve over the BC policy, but falls short of the gpt-4o expert teacher. These results show that if the expert teacher is already strong baseline, even after leveraging privileged information, it might be hard for the student to outperform the teacher.
Results.
Method
taucot
#aot
qpt-4
48.5*
gpt 3.5
46.5*
Jama-3amm.34 [I]
36.0*
RoAct gpt-4o React qpt apt-4o-min
71.3
3.8
43.4
6.0
LEAP amm-- 3-1 To
41.4
6.6
LEAP Llama 3.1 8b
49.9
5.5
4.5 WHAT IS THE TRADE-OFF BETWEEN PRIVILEGED MATION AND REALIZABILITY?
LEAP 1.1ama-3. 3.1-70b to
60.3
4.8
LEAP Llama 3.1 70b aq
71.8
5.1
We study the trade-off between how much privileged state the expert vs. the realizability of their corrections on ALFworld. We create 5 different privileged experts : = by varying the prompt to instruct them use increasing amounts of privileged state, e.g. for J it says "Use this information sparingly for your correction." but for 7.. says "feel free include information from the privileged state". We run iteration of LEAP to generate updated π ....; 5², Fig. 5 (a) shows that success rate initially rises from π x3 but then falls off sharply till 2², Num actions has a similar trend. This validates the hypothesis that there is indeed much worse than 11 showing that it's more important to be realizable than to provide optimal yet unrealizable corrections. Fig. (b) shows the corrections from nf n T Th generates reasoning that is perfectly realizable to the student, but since it has no privileged state the actions are not optimal. In contrast, n5 reason, being unrealizable to the student. T strikes a perfect balance where it offers the correct action, but offers reason that is very much realizable, i.e. kettles are likely to be in stoveburners.
Table 2:
Intercode Bash Evaluation. Over- all success rate and actions test data (max 10 actions). Baselines from [1] (Yang et al., 2024) evaluated on "(train + test) data.
optimal tradeoff. Interestingly, ms
E predicts the correct action, but blatantly reveals the privileged information in the
We run another experiment to examine the effect of the teacher providing on- on-policy corrections on the states visited by the student, detailed in the Appendix C. Our hypothesis is that I. LEAP works due to
8

---

Published as a conference paper at ICLR 2025
Host tokk: PRouutt
Rick tasks (Mour
80
80
60
9
-0
40
20
25
X1
LOAP tussict
LEAP taston
LSAP Bwrd 999
Cool taoks Moau
1000
(Roauc +
1.00
Rcx tadk (saru +
60
30
20
80
90
70
20
47
F3
1PAP ENTRON
[A]
[b]
Figure 6: Self Correction. LEAP self self-improves Llama3-88 8B model that is used as both student and teacher. (a) Overall success rate and num actions. (b) Category- wise breakdown of success rate.
these two key factors: (a) the LLM teacher effectively balances the use of privileged information and (b) the LLM teacher provides on-policy corrections.
4.6 CAN LEAP BE USED TO SELF-CORRECT STUDENT?
We test the hypothesis that LEAP should enable model self-improve by using its privileged version as the expert. On ALFWorld, we fine-tune Llama3-88 8B initializing with the BC policy To and running 2 iterations of self improveereeentt.. Fig. 6 shows that LEAP is able to significantly self-improve policies over iterations (65.7% 82.1% + 91.8%). with the final policy matching the best policy when using GPT-4o as a teacher. We see uniform improvements across all categories, with some improving from To π and others from 1 T2- We conclude that for some environments, like ALFworld, the privileged state does the heavy lifting, ie. performance improvements happen at a similar rate using strong teacher vs. using the model itself as teacher. We note that this need not hold for all environments, e.g. for some a strong teacher is required to extract the optimal reason and action from the privileged state. This effect is seen WebShop in Appendix C.2 where while self-correction improves performance, it is unable to outperform the gpt-4o baseline.
4.7 How DOES SFT COMPARE TO PREFERENCE OPTIMIZATION?
We study the effect of different update methods for LEAP: SFT vs Preference Optimization. Our initial hypothesis is that KL regularized preference optimization should also result performance improvement without needing ag- gregate datasets. We run 1 iteration of DPO with different regularization values. Table. shows that while DPO improves upon the base To policy, the improvement is far smaller than SFT for both 3 = 0.1, 0.01. One explanation is that SFT is far more aggressive in using the correction data compared to preference optimization, i.e., SFT deems the corrected reason action to be better than any alterna- tive. To test this, we used a different preference optimizer KTO, that operates with unpaired preference data and set weight ÅD = 1.0, AD = 0.O, effectively emulating SFT-like behavior. KTO with these settings has significantly high performance of 88.1% % close to SFT (91.0%).
Method
Saucc
#act.
LEAD
68.6
16.9
in-dist.
LEAB D20 (8 0.11 d
74.3
15.7
LEAP DPD (8:001))
74.3
15.6
LEAP KTO (Au 0.00 m
87.1
11.6
LEAP S21
96.4
10.5
out-dist
LEA2 To
65.7
18.6
DPO (..1)
71.6
17.9
LEA2 DPO (8 0.011 I
70.9
17.7
(Ap m
88.1
LEA2 SE
91.0
11.9
Table 3: SET vs Preference: LEAP with SFT vs DPO vs KTO on ALFWorld.
5 RELATED WORK
Imitation Learning and Privileged Information. A powerful paradigm in machine learning is leveraging privileged information (Vapnik et al., 2015)-datl available only during training and inaccessible at test time. This concept has been transformative in robotics, where simulated expert policy with full state supervises a learner that only perceives sensor observations, such as navigation (Zhang et al., 2016; Uppal et al., 2024), self-driving (Chen et al., 2020), manipulation (Chen et al., 2023a; Hu et al., 2024), and legged locomotion (Lee et al., 2020; Kumar et al., 2021).
A fundamental challenge in utilizing privileged information is the realizability gap where actions suggested by an expert may be infeasible for the learner to predict. This gap often leads to spurious correlations, manifesting the "latching effect" where learners repetitively predict the same action,

---

Published as a conference paper at ICLR 2025
commonly observed in autonomous driving (Muller et al., 2005; Kuefler et al., 2017; Bansal et al., 2018; Codevilla et al., 2019) and language models (Bengio et al., 2015; Holtzman et al., 2019; Wang & Sennrich, 2020; Ortega et al., 2021). While off off-policy methods (Wen et al., 2020; 2022) have been proposed, (Swamy et al., 2022) show that online interaction with the environment is necessary.
Interactive imitation learning methods such as DAGGER (Ross et al., 2011), where experts provide corrective actions during the learner's 's rollouts, both theoretically and empirically lead to very effective policies in this regime (Choudhury al., 2018). These methods operate under the assumption of asymptotic realizability (Swamy et al., 2022), where the learner's s policy can eventually match the expert's 's actions as the episode progresses. However, when this assumption fails-ofter due to the partial observability of the task or model for L.LM agents, as demonstrated by our experiments (Sec. 4.5). While recent approaches resort to exploration or RI. (Walsman et al., 2022; Tennenholtz et al., 2021; Nguyen et al., 2022; Weihs et al., 2021), these are intractable to extend to LLM agents. Our work provides a practical recipe for LLM agents that optimally balances the trade-off between using privileged information and maintaining realizability, and further shows how such an expert can be used to enable policy self-improvement.
performance ensues, particularly
Learning from AI Feedback. A scalable method for aligning language model using AI feedback (Lee et al., 2023). Recent works have explored different types of feedback, from self- sefereenedted corrections to those provided by external Al models environments (Pan et al., 2023). One body of work focuses on self-correction, where LLMs refine their responses based on feedback from their own outputs (Bai al., 2022; Madaan et al., 2024). This occur either at train-time, where LLMs are fine-tuned to generate corrected responses (Bai et al., 2022; Ganguli et al., 2023), or at test-time, where the model improves its response interactively (Madaan et al., 2024). However, while simple in principle, Huang et al. (2024) shows that without external sources of feedback, LLMs often struggle to accurately judge the correctness of their reasoning, limiting the effectiveness of purely self- geferaeedd corrections.
Another cluster of work explores feedback from external sources. Some approaches use feedback from strong models like GPT-4 acting as judges (Zheng et al., 2023) or critics trained on high-quality curated data ( (Wang et al., 2023; Paul et al., 2023). Similarly, An et al. (2024) and Guo et al. (2024) use GPT-4 to provide corrections and preference feedback. Welleck et al. (2023); Qu et al. (2024) trains LLMs to self-correct from errors. Others leverage environmental feedback or external tools to guide improvements. Chen et al. (2023b); Olausson et al. (2023) utilize execution feedback to improve code generation, and (Gou et al., 2023) employs external tools. Notably, Reflexion (Shinn et al., 2023) leverages feedback from external environments to generate self-reflections for agents, although assumes multiple attempts in the same environment, which our approach does not. Our work introduces the use of privileged information in decision-making tasks for I.I. agents, a technique not yet fully explored in the literature. This also enables LLM agents self-improve using privileged information, setting it apart from existing self-improvement methods that rely on iterative solution generation and evaluation (Zelikman et al., 2022; Pang et al., 2024; Hosseini et al., 2024).
LIMITATIONS
We proposed LEAP, iterative fine-tuning framework that improves LLM agent performance using privileged AI feedback, enabling weaker models to surpass stronger expert teachers and allowing agents to self-improve. However, LEAP has notable limitations. First, generating interaction rollouts is time-consuming, particularly in complex environments requiring multi-step reasoning. Each rollout requires simulating the environment and generating complete trajectories, that becomes computation- ally expensive across multiple fine-tuning iterations. Second, corrective feedback from the expert often results in lengthy output tokens. As trajectory lengths grow, balancing the effectiveness of feedback with its verbosity becomes challenging. Exploring methods to distill and summarize expert feedback at critical points an interesting direction for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the NSF FRR (#2327973). NSF RI (#2312956), and ONR Young Investigator Award. Sanjiban Choudhury supported in part by the Google Faculty Research Award and the OpenAl Superalignment Grant.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we have made several efforts, as outlined below. Detailed information and resources are provided in the main paper, appendix, and supplementary materials:
Open source code. We open source the code including the Python package implementing LEAP, along with all prompts, configuration files, and training scripts necessary reproduce our results. 2. Experimental details. We provide descriptions of our experiments in the main paper (Section 4), with additional details the setup, hyperparameters, prompts found in Appendix D, E. We use open-source datasets and models in our experiments, with citations and links.
3. Analysis. The key results our theoretical analysis are presented in the main paper (Section 3.3), with full derivations, proofs, and assumptions detailed in Appendix B.
REFERENCES
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian MaanGuang Lou, and Weizhu Chen. Learning from mistakes makes LLM better reasoner. arXiv preprint arXiv:2310.20689, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeuret:: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint arXiv:1812.03077 2018.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances neural information processing systems, 28, 2015.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference Robot Learning, pp. 66-75. PMLR, 2020.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): cadc9244, 2023a.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023b.
Sanjiban Choudhury, Mohak Bhardwaj. Sankalp Arora, Ashish Kapoor, Gireeja Ranade, Sebastian Scherer, and Debadeepta Dey. Data-driven planning via imitation learning. The International Journal of Robotics Research, 37(13-14) 2018.
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9329-9338, 2019.
Abhimanyu Dubey, Abhinay Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The laaaa herd of models. arXiv preprint arXiv:2407.21783, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
11

---

Published as a conference paper at ICLR 2025
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self- coffectrrettonn in large language models. arXiv preprint XX0v:22302.04459, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint 113 2023.
Shangmin Guo, Biao Zhang. Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinnrres, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 2024.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:/904.009:
2019.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self- teught reasoners. arXiv preprint o7X44 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09683 09685, 2021.
Edward Hu, James Springer, Oleh Rybkin, and Dinesh Jayaraman. Privileged sensing scaffolds reinforcement learning. arXiv preprint arXiv:2405. 1eet22455.14855 14853, 2024.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprini arXiv:2310.017999 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
Alex Kuefler, Jeremy Morton, Tim Wheeler, and Mykel Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE intelligent vehicles symposium (IV), pp. 204-211 IEEE, 2017.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprini arXiv:2/07.04034 2021.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint 2.X0022.009.00267, 2023.
Joonho Lee, Jemin Hwangbo. Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xabbc5986, 2020.
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D Ernst. NI2bash: A corpus and semantic parser for natural language interface to the linux operating system. arXiv preprint 1802.089777 2018
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
12

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. Advances in neural information processing systems, 18, 2005.
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. arXiv preprint arXiv:2211.0199/, 2022.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306. 09896, 2023.
Pedro A Ortega, Markus Kunesch, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, Jonas Degrave, Bilal Piot, Julien Perolat, al. Shaking the foundations: delusions in sequence models for interaction and control. arXiv preprint arXiv:2 10. 10819, 2021.
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathami, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.00 1388, 188, 2023.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint urXiv:2404. 19733, 2024.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of markov decision processes. Mathematics of operations research, 12(3):441-4500 1987.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304. 0104,,10 2023.
Yuxiao Qu Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint urXiv:2407./82199 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Stéphane Ross, Geoffrey Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Artificial Intelligence and Statistics (AISTATS), 2011.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement (202), arXiv preprint cs.A1/2303.11366 2023.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proc. IEEE Int. Conf. Computer Vision and Pattern Recognition, 2020a.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766 2020b.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv:2209.15189 2022.
arXiv preprint
Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked Ilm policies for web actions. In Conference Language Modeling (COLM), 2024.
13

---

Published as a conference paper at ICLR 2025
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. Advances in Neural Information Processing Systems, 36, 2024.
Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learning with unobserved contexts. Advances in Neural Information Processing Systems, 35:17665-17676 2022.
Guy Tennenholtz, Assaf Hallak, Gal Dalal, Shie Mannor, Gal Chechik, and Uri Shalit. On co- variate shift of latent confounders in imitation and reinforcement learning. arXiv preprint 1eXiv:211000552 2021.
Shagun Uppal, Ananye Agarwal, Haoyu Xiong, Kenneth Shaw, and Deepak Pathak. Spin: Simul- taneous perception interaction and navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18133 18142, 2024.
Vladimir Vapnik, Rauf Izmailov, et Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res., 16(1):2023-2049, 2015.
Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Representations, 2022.
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. arXiv preprini arXiv:2005.03:: 03644, 2020.
Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O' Brien, Ramakanth Pasumuru, Jane Dwivedi- Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04533 2023.
Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances Neural Information Processing Systems, 34:19134-1911 2021.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. The Eleventh International Conference on Learming Representations, 2023.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems, 15:2564-25755 2020.
Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, and Yang Gao. Fighting fire with fire: avoiding dnn shortcuts through priming. In International Conference on Machine Learning, pp. 23723-23750. PMLR, 2022.
John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36, 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems (NeurIPS), 2022a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022b.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, Advances in Neural Information Processing Systems, 15:15576--5588, 2022.
Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search. In 2016 IEEE international conference robotics and automation (ICRA), pp. 528-535. IEEE, 2016.
14

---

Published as a conference paper at ICLR 2025
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Lim agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19632-19642, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 06:46595-466233 2023.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
A Broader Impacts
17
B Analysis
17
B.1 Trade-off between realizability and privileged information
17
B.2 Practical approximation for constrained privileged expert
21
Additional Ablations
21
LEAP vs SFT with Privileged Information
21
LEAF self-correction on Webshop
................................
22
C33 LEAP training on all data
22
D Experimental details
22
D.1 Hyper-parameters
22
D.2 ALFWorld
................................
24
WebShop
24
D.4 Intercode
25
E Prompts
25
E.1 AlfWorld
26
E.2 WebShop..
29
E.3 InterCode
34
16

---

Published as a conference paper at ICLR 2025
A BROADER IMPACTS
Technological Impact. By iteratively fine-tuning LLMs through privileged expert feedback, LEAP can significantly enhance Al performance across diverse applications, such as virtual assistants and dialogue agents handling complex decision-maie making tasks. This approach allows LLMs to autonomously refine their decision-making. reducing the need for human intervention, streamlining automation, and improving efficiency in both consumer and enterprise environments.
Additionally, LEAP 's ability to enable weaker models to outperform stronger ones democratizes Al access, making state-of-the-art models more accessible to organizations with limited computational resources. This also opens new possibilities for improving model generalization in settings where perfect information is unavailable. However, the reliance on privileged feedback during training underscores the importance of further research into model transparency and interpretability. As these models grow more autonomous, ensuring their decision-making remains understandable to users will be critical for building trust and fostering widespread adoption.
Societal Impact. Equipping LLMs with iterative infrooeppoeeettt capabilities unlocks range of societal benefits. In sectors such as education, customer service, and software development, these Al systems can assist professionals in decision-making, automate routine tasks, and enhance overall productivity. By continuously improving their performance, these models can reduce the cognitive load on workers, allowing them to focus on more complex, creative, and value-added activities.
Despite these advantages, there are ethical concerns that must be addressed. As L.LMs become capable of self- ceff-improvement, ensuring their trustworthiness and accountability becomes critical. Without proper safeguards, the autonomous refinement AI could lead to unpredictable behaviors, making it harder to ensure alignment with human values. Furthermore, there is risk that self- improving Als could be misused for malicious purposes, such as automating harmful tasks or spreading disinformation. To mitigate these risks, strict ethical guidelines and robust oversight is necessary to ensure deployment of such systems serves societal good and prevents misuse.
B ANALYSIS
B. TRADE-OFF BETWEEN REALIZABILITY AND PRIVILEGED INFORMATION
We now analyze the performance of a policy imitating an expert with access to privileged information. The performance depends both on the performance of the expert policy and the realizability gap between the learner and the expert. We then derive a constrained privileged expert to better trade off these two terms and provide a practical approximation of this constrained expert. For notational simplicity, we assume that a, represents both reason and action. We follow the derivation in (Swamy et al., 2022) but simplify for our setting.
Let π e be the student policy that we are training. We assume the class of policies II is rich enough contain all history-based policies. Let J(T) be the performance, i.e. the cumulative reward of learner policy that selects actions based on the history 4; = π (h+). Let J(Tg) be the performance of the expert policy that selects actions based the privileged state a; n= (s.). Let 7 ~ n be privileged rollout which is the sequence of observation, actions, and privileged state upon executing the policy. We represent the rollout as T = [(h,, 8t, a.)). Note that such a privileged rollout is only accessible at train time.
We begin by defining the average imitation gap between an expert policy π and learner π. Definition B.1 (Average Imitaiton Gap). The average imitation gap of learner π over horizon T:
AIG ( T) = + (J(mZ) J(z)) E Eraph
+ Hr(er.a))
E +r((
(3)
We next define the realizability gap between the privileged expert and the best policy in policy class as the LI distance (or equivalently Variatiii (TV) distance) between the two policies on the distribution of states and histories induced by any policy.
17

---

Published as a conference paper at ICLR 2025
Definition B.2 (Average Realizability Gap). The average realizabilityy gap over time between the privileged expert π E and the best policy in the policy class x² is:
T
<(T E T == sup T
Bs,
( - x- (ael4))))
(4)
1=1
We next define the recoverability of the expert. Let A (8, = Q= (s, - V* (s) be the advantage of state action (s, a) with respect the expert policy , A high negative advantage means that the mistaken action at a state results in a high value difference from the action the expert advantage.
would have taken. Recoverability is the worst
Definition B.3 (Recoverability Coefficient of the Privileged Expert). The recoverability of the expert policy n is the max advantage of the expert policy
H (T* max 5;a
||AT (s,
We are now ready to state the main theorem of LEAP. Theorem B.4 (LEAP with Privileged Expert). Running N iterations of LEAP with privileged expert nh yields at least one policy a such that
T J( 2 1 ( -0--H((=F) (c(TF,T))
(6)
where TF is the privileged expert, (TT)) is the recoverability coefficient of xF, E average realizability gap, and (N) is the average regret of the DAGGER update.
T) is the
The first term is the performance of the privileged expert and the second term is the average real- izability of the privileged expert. The more privileged information the expert policy N leverages the better (xx and the lower recoverability coefficient H (**)). However, the realizability gap e( E T) grows larger.
Proof. The proof relies on two important results.
The first is the Performance Difference Lemma (PDL) (Kakade & Langford, 2002) which states that performance difference between any two policies can be expressed as the sum of advantages.
T
J(m) - J(T')
E
Σ asa
Ar (st, (a)(s))))))))
where s; ~ dī is the induced state distribution by æ, and AT (st, = Q (or, ax) - V= V=((s) is the advantage w.r.t. x'.
To adapt the equality to our setting where two policies operate on different input spaces, we note that the equality different input spaces, e.g. state 8t history h2 joint state- sate-history st, h4. We express in terms the joint state-history s,, h2 space and write the inequality as:
(Tx) - J(
Es he
Σ A (st, (e)ss
(From (7))
Σ Ard he ac)T(he)))
(π depends only on hy)
T
= Σ he
Σ A (s,, (a)ht))
( depends only on st)
1=1
as Σ ATT (sc, (aclss) = =0)
Σ A (st, (T(a(hi) - x =
(8)
18

---

Published as a conference paper at ICLR 2025
Rearranging, get the following form of PDL which use in the main proof:
J(x²) - J(T)
Σ bacdh
Σ A (si, ar) ( & (aels)) क ((allee)
(9)
The second result we use us from interactive imitation learning DAGGER (Ross et al., 201 that reduces imitation learning to no-regret online learning. DAGGER shows that with π as the expert teacher guarantees that after N iterations, will find at least one policy
Es (. π = (.|s)]l I Es | (T(Aa) - (1|s]]1 + +NN)
(10)
where y(N) is the average regret, and d is the time average distribution of states and history induced by policy
We now prove the main result
1
ATG 1)
(J(PB)-J(
T
T
Bag maf
Σ AT (s1, at) (T - (((aa))((((aa))))
(From (9)
T
T
<
"|AT
(..-)loo
E En hr -d ||= - (.1se)l
(Holder's Inequality)
T
tm1
1
$
((TT))
Ba
( (11h)) -x (.18s)l
T
S (((
E he 1| (-1be) - 7 (-1s))l + ((N)
From (10)
s <((T))
(c(T , (7)++ +((
Rearranging, we get
1 J(t) 2 [( -H (c(T", +)++(N))
To optimize the performance bound, we introduce constrained privileged expert that provides actions similar the privileged expert but constrained be close to realizable.
Definition B.5 (Constrained Privileged Expert). Given state and history h. constrained privileged expert minimizes KI. distance to the privileged expert while being in a KL ball of the non-privileged expert.
F (-|st; he) arg min KL (F(11%, he
s.t.
KL ho)ls, (((b)) S (11)
We use the fact that we have a rich policy class II that includes the non priviieiieee expert x = (.[h), to show that the constrained privileged expert bounds the average realizability gap
T
el <
E (T) = sup
Es op 1| (arlst) - x"' (arlb.)))
T
sup
Es, hu d7
2KL( (atls
T
(12)
1=1
since n7 (.(b)) II
sup
T
< 28
from (11)
19

---

Published as a conference paper at ICLR 2025
Theorem B.6 (LEAP with Constrained Privileged Expert). Running N iterations of LEAP with constrained privileged expert oi yields at least one policy т such that
))
1 J(a)) 2 1 [(
{ (1a3) J(a)) +H((x))
25
+x(N))
(13)
from privileged expert
realizability
where T the privileged expert, H (xx)) ) the recoverubility coefficient of 5, 8 the KL constrain, ((N) is the average regret of the DAGGER update.
The first term is the performance gap between the privileged and the constrained privileged expert. The second term is the realizability of the constrained privileged expert. As â 0, the realizability gap goes down while the performance gap and recoverability coefficient go up. There is a critical value of that achieves optimal tradeoff.
Proof. The proof is similar to Theorem B.4 but with the modification that n depends on both state and history.
The PDL in this case can be expressed as
T
(( = Σ
Es herd
Σ A (st, (i, az) (aelse,))) m(arlhe)) a,
(14)
DAGGER (Ross et al., 2011) with To as the expert teacher guarantees that after N iterations, it will find at least one policy
Bg || Th (-1s,))11 S Bagtond || = M)1 + +NN)
(15)
where (NV is the average regret.
AIG(T, 7) T ( ((T))
T (J(#B)- - -((x))) + 7 [ -
T (J(TF)- ((rf)
1
1 Σ Esg Nardd Σ
A (st, ae) (Tf(os)s,, hi) T(aclhe))
(From (14))
T
T
$ T (J(TB) (( + ||A
(lo,,))llo
E,, E (1T((11e) - Th (-1se, h4)l1
(Holder's Inequality)
T
T (J(TB)- -(( + H(m
E
En [[T((1)))) - (-1se, ht)lla
T
1 (J(TB)- I(*8)) H(=B)
$ T
E by ||= (-(ht) ( he)l12 + ((N)
From (15)
s T (J(*D) -(( + H(=5) (c(=T5 F. T) + +NN))
< 7 (J(T5) - + H(mT) (v26 + +
From (12)
Rearranging, we get
T J(T) 2 [(TB) ( -((F))-((T3)) - H(*f) (V26 + +(22))
20

---

Published as a conference paper at ICLR 2025
B.2 PRACTICAL APPROXIMATION FOR CONSTRAINED PRIVILEGED EXPERT
Writing out the Lagrangian of the constrained privileged expert (Def. B.5) and solving for the primal
get:
F (p. als,h) ((s,
2 (0, (p))))))) (p,
(16)
where À is the Lagrange multiplier.
For fixed multiplier X, this closed-form expression suggests simple penalty-method approximation to constrained privileged expert:
1. Sample responses from both π (p, a, s) and x = alppp, h) to create R = {ø', a' d
2. Reweigh the responses in R according to
log Tog="(p. a, h) Alog x (p,
(17)
3. Sample from the reweighed distribution.
Since the majority of our experiments involve weighh-weights models (like gpt 4o), we are unable access the logits of the expert. Instead, we design LEAP prompts approximate the constrained privileged expert by providing instructions to not reveal the privileged information in the reasoning, and provide general reason and actions that should be predictable given only history h4.
C ADDITIONAL ABLATIONS
C.I LEAP VS SFT WITH PRIVILEGED INFORMATION
To investigate whether LEAP's performance gains primarily arise from the use privileged infor- mation, we introduce new baseline, SFT-privileged. This baseline collects demonstrations from privileged teacher that has access to the same privileged information as LEAP and trains student policy using Supervised Fine-Tuning (SFT). This follows the context distillation approach Snell et al. (2022), where privileged information is incorporated into the teacher's 's context and distilled into the student policy. To ensure fair comparison, we use the same prompt structure as LEAP, preventing the teacher from explicitly revealing privileged information to the student.
Key Observations: (1) The SFT-
privileged baseline achieves a success rate of 92.5% which is significantly lower than LEAP (Iter O: 65.7%, Iter 1: 91.0%). (2) When successful, SFT- privileged requires fewer actions (8.4) compared LEAP (Iter 0: 12.7, Iter 1: 10.1).
Model
$suct
fact.)
tact succe
LEAP SPT SPT-privileged
42.5
20.8
8.4
LEAP LEAP mo LEAP LEAP m
18.6
12.7
91.0
11.9
10.1
Table 4: Performance Comparison: Success rate and action effi- ciency across models.
Explanation: At training time, the privileged teacher always provides successful demonstrations, offering general reasoning (e.g., "Cellphones are commonly found on desks") and using privileged information locate objects and complete tasks. The SFT-privileged student is trained exclusively on these successful demonstrations. At test time, if the student's reasoning is correct, successfully mimics the teacher with an optimal action count (8.4). Having been trained only on successful demonstrations, it lacks the ability to handle failure cases. For example, it might hallucinate incorrect objects or repeatedly visit the same location. This inability to recover leads to significantly lower success rate (42.5%)
The improved performance LEAP driven not just by privileged information but by two key components: (1) On-policy corrections: On-policy corrections enable the student to recover from errors, as demonstrated by the significant improvement over the SFT-privileged baseline (91.0% 42.5%). (2) Realizable corrections: Realizable corrections, where the teacher avoids revealing privileged information, also play a critical role LEAP's performance. Ablations show a substantial drop when realizable corrections removed 1..0% 5.2%).
Summary:
21

---

Published as a conference paper at ICLR 2025
Model
Scoref
#act
x_att
r_option
r_type
r r_price
LEAP iter 0
29.4
21.1
28.0
25.8
35.0
36.8
LEAP iter 36.8
18.7
36.6
33.0
44.2
45.2
LEAP iter 2 39.8
18.4
39.9
31.7
48.1
51.0
Table 5: Self-correction on WebShop: LEAP self-improves Llama3-8B model, where the teacher is privileged version of the student.
Model
$succt
#act.
Pick
Clean
Beat
Cool
Look
Pick 2
LEAP ALFWORLD only iterd
65.7
18.6
66.7
74.2
73.9
66.7
66.7
35.3
LEAP ALEWORLD only iterl
91.0
11.9
83.3
90.3
91.3
95.2
94.4
94.1
LEAP COMBINED iter0
64.2
18.4
62.5
80.6
69.6
61.9
55.6
41.2
LEAP COMBINED iterl
91.0
11.9
87.5
90.3
82.6
95.2
100.0
94.1
Table 6: ALFWorld Results: Success rates and per-task performance across different iterations.
Model
Scoret
fact|
r_att
r_option
r_type
p_ricee
LEAP WEBSHOP only iter0
29.4
21.1
28.0
25.8
35.0
36.8
LEAP WEBSHOP only iterl
51.8
16.7
60.0
16.1
73.4
78.4
LEAP COMBINED izer0
21.3
22.6
16.6
29.7
38.4
21.5
LEAP COMBINED iserl
37.9
42.9
17.1
52.2
56.8
Table 7: WebShop Results: Performance comparison across iterations for WebShop tasks.
C.2 LEAP SELF-CORRECTION ON WEBSHOP
On WebShop, we observe that self-correction consistently improves performance across iterations, increasing from 29.4 to 36.8 and to 39.8 (Table 5). However, the performance gains lower when using a Llama-8B teacher compared to GPT-4o teacher. This in line with our hypothesis that for environments like Webshop, where the privileged information is not rich as ALFworld, the performance improvements do depend on the strength of the teacher to provide meaningful corrections.
C.3 LEAP TRAINING ON ALL DATA
We run experiment where we trained a single model combined datasets (called COMBINED). To do so, we downsample ALFWorld dataset to be the same order as Webshop. We run LEAP for two iterations and present results in Tables 6, 7 comparing to models trained individual benchmarks.
We make the following observations: (a) LEAP shows similar magnitude of improvement when trained on all datasets compared to when trained on individual datasets (on ALFWorld 64.2 -> 91.0, on Webshop 21.3 -> 37.9). COMBINED model matches ALFWorld performance compared to model trained on ALFWORLD only. (b) COMBINED model scores lower on Webshop compared to Webshop only. We attribute this mismatch in dataset size after downsampling. However, the magnitude of improvement LEAP the same for both models.
D EXPERIMENTAL DETAILS
D.1 HYPER-PARAMETERS
D.1.1 TRAINING PARAMETERS
Tables 8 and 9 contain hyperparameters for SFT training and DPO/KTO training using LoRA for the different datasets. All training rums were on machines with either 2 or 4 RTX A6000 GPUs, each with 48 GB of memory per GPU.
22

---

Published as a conference paper at ICLR 2025
Dataset
AlfWorld
WebShop
InterCode
Model
Llama3-8B
Llama3-8B
Llama3-8B
Llamaa-70B
Batch size
64
16
16
Max seq length
8000
6000
6000
Max epochs
5
Learning rate
3e-5
3c-5
3c-5
LoRA
64
64
64
LoRA r
128
128
128
LoRA dropout
0.05
0.05
0.05
Optimizer
Adam cosine
A cosine
AdamW
LR scheduler
cosine
Table 8: Hyperparameters for SFT training using LoRA
Method
DPO
KTO
Model
Llama3-8B
Llama3-8B
Batch size
32
64
Max seq length
8000
8000
Max prompt length
6000
6000
Max epochs
Regularization
0.01
0.01
Learning rale
5c-7
5c-7
LoRA
64
64
LoRA r
128
128
LoRA dropout
0.05
0.05
Optimizer
AdamW cosine
AdamW cosine
LR scheduler
Table 9: Hyperparameters for preference training (DPO, KTO) using LoRA
1.1.2 EVALUATION PARAMETERS
For inference using I.lama, ama 3 3-8 8b and 1.1 3--700,,
temperature setting of 0.3 and maximum token length of 256. The following code snippet shows the generate () function used:
outputs model .gnerrae(
attention htwpattioonmmssstookniiieee inputs ("attention I )
max he___e___ooe toe_nookkn-
eos_token_idd-1
tokeni2ee2.co__toke...
},
temperature-0.33
pad mad_token_id-sel www.:/te....oooo.ooo..... Baa
Listing 1: Generation function with hyperparameters for Llama3 models
For all OpenAl model calls, including tasks such as generating expert teacher corrections, labeling reasoning for actions, or running prompt baselines like ReAct, we use the API call with the following hyperparameters. See Appendix E for the detailed prompts.
model: got-4o
that_completion = client chat completiins.crettt.......o. messages mode1="
23

---

Published as a conference paper at ICLR 2025
temperature=0.3 top_p=l, n=1
response chat_completion.choo.ccc choices m
www.age. content
model: opt-4o-mini
chat_completiine = chaent.chat.... www..ecoeoooeiio..rtetee.et.
messages=nnssagges,
modeL="gott<ommnnn,,
temperature top_p=1, n=1
response chat cett_ccommetiion.ciiii... mmm.maaaeeeem.maaaaeee...
www.aag..ecc.........m.......... content
Listing Hyperparameters for OpenAl GPT-4o and GPT-4o-mini models
D.2 ALFWORLD
ALFWorld (Shridhar et al., 2020b) is text-based game built on the ALFRED embodied benchmark (Shridhar et al., 2020a). Each task has a high-level objective, c.g. "heat mug and put it in cabinet" which the agent must complete by navigating and interacting with virtual household through text commands, e.g. go shelf pick up mug 2, etc. It is a POMDP where the hidden state the location of all objects, and the observations reveal only the objects the agent can see. To solve it, the agent must plan subgoals, track progress, and explore efficiently aspect of ALFWorld is the requirement to infer the likely locations of household objects (e.g. mugs are more likely to be on shelves or cabinets), which the agent can learn from interaction with the environment. There are 6 categories of tasks and two datasets: 139 in-distribution games and 134 out-of-distribution games. Time horizon is limited to 30.
time runs out. An interesting
We compare against ReAct (Yao et al., 2022b) with different base models and prompts. The original ReAct paper (Yao et al., 2022b) used text-davinci-002 with task-dependent in-context examples. We ReAct wtug got-4o 4o and gpt 4o -inni and instruction prompt that is shared by all our methods. We also compare against BUTLER (Shridhar et al., 2020b), a seq2seq BERT model trained using DAGGER.
We LoRA fine-tune Jlama-3-8B model using got-4o as the expert teacher for 4 iterations. We use observation-action demonstrations on the training dataset of 3257 gamesShridhar et al. (2020b), which we annotate with gpt 4o to generate reasons and use this to train the BC policy no. We then roll out policy and generate corrections from the privileged expert on unsuccessful rollouts. The correction data is then aggregated with the existing dataset and used to train policy using SFT.
The privileged state in ALFworld consists of
1. Task: What is the overall task
2. Essential Objects: Enumerate all objects that are necessary to solve the game.
3. Critical Locations: List the important locations involved game solution.
4. Optimal Action Sequence: Suggest optimal sequence of actions that could theoretically minimize the steps needed to complete the game, based on the log provided. Be sure to copy the action verbatim.
D.3 WEBSHOP
WebShop (Yao et al., 2022a) is an online shopping environment consisting of 1.18 million real- world products and 12, 586 human-provided instructions. Each task is a shopping request from a user ("I need gluten-free bacon, 4 oz pack of 2, priced under $50") that the agent has to solve through
1 Since we look at the single trial setting, it rules out approaches like Reflexion (Shinn al., 2023) where agents require multiple trials the game.
24

---

Published as a conference paper at ICLR 2025
set of interactions (search["gluten free bacon"] or selecting options such as "4 oz pack of 2"). The metrics are the overall score, and 4 components of the score: whether the attributes match (ATT), whether the options match (OPT), whether the product types match (TyPE) and whether the price matches (PRICE). We evaluate on 500 test instructions, and limit time horizon to 30.
We compare against ReAct. (Yao al., 2022b) with different base models like GPT-4o and GPT- 4o-mini. We use the same instruction prompt as in LEAP. We also compare against IL. baseline in WebShop (Yao al., 2022a) trained using BC. These baselines train different selection models, one for selecting search queries from a list of options and one for clicking from a list of options. This constraint search space simplifies the problem.
We LoRA fine-tune Llama-3-88 model using GPT4-o as the expert teacher for 4 iterations. We initialize To using observation-action demonstrations from Yao et al. (2022a) which we annotate with GPT4-o to generate reasons. We roll-out policies on the training dataset of 12086 tasks, and generate corrections from the privileged expert on unsuccessful rollouts. For every game in the training dataset, we construct privileged information that contains: the attributes the product must satisfy, the price constraint, the option that has to selected, and an example product that would satisfy all criteria. The correction data is then aggregated with the existing dataset and used to train policy using SFT.
The privileged state in ALFworld consists of
1. Task: What is the user instruction
2. Attributes: What are the attributes the product must have
3. Price: What price constraints must the product satisfy.
4. Example Product: Name of a product that satisfies all criteria.
D.4 INTERCODE
Intercode Yang al. (2024) is interactive coding environment, where agent interacts with compiler/ineerreeeer to execute its code and submit further refinements. We use the Bash environment, where typical task is "Find all text files and write their names single file.", and the agent issues commands like s, find create interactive datasets. We the first 2 for training, and the next 2 for testing. We limit the time horizon to 10. We compare against ReAct Yao et al. (2022b) with gpt 4o and gpt 4o-mini. We also compare against the top 3 entries on the Intercode leaderboard, which are the TryAgain baselines from Yang et al. (2024) with gpt-4, gpt-33 5 and CodeL.lama-34B Gama-34B-INS") -ISTT respectively. Note the numbers are directly from the leaderboard which evaluates on all (train+testt data and, while our numbers are only on test.
to solve the task. Intercode builds on the NL2Bash (Lin et al., 2018) to
E PROMPTS
We define the following three prompts across all datasets:
1. Expert corrections template: This prompt is used by the LLM expert teacher to provide corrections on the student agent's trajectories. These corrections offer improved reasoning and actions that balance the trade-off between utilizing the privileged state (accessible only to the expert during training) and ensuring the corrections are realizable by the student agent.
2. Student agent template: This prompt is used during test time by the Llama3 student agent models to generate reason and action trajectories that achieve the task objective. It is also used for prompting baselines, such GPT-4 ReAct, which generate reasoning and action trajectories to achieve the task objectives.
3. Autolabel reasons template: For datasets that contain human demonstrations with action labels (like AlfWorld, WebShop), this prompt is used to generate reasoning annotations for actions that the human took. Generating reasons is necessary the student agent needs to be trained to produce both reason and actions in its output trajectories.
For prompts used in different ablation studies, please refer the accompanying code.
25

---

Published as a conference paper at ICLR 2025
E.1 ALFWORLD
E.1.1 EXPERT CORRECTIONS TEMPLATE
{8 if system 8
You are given trajectory containing observations, reason and act ions generated by student agent solving text world game. You are teacher who has access information sufficient to solve the game but hidden from the student. Your goal is to improve how the student solves the game by improving their reason and actien at every timestep.
"privileged state" that contains secret.
888 Input
You will be provided with a JSON file that logs the student agent's s observation, candidate_astions reasen and action at every timestep while playing text-based game. The student fails to solve the task within the time horizon of the game.
The structure is an array of objects, containing the following at each
timestep:
timestep: Index of the current timestep
observation The observation provided the student agent candidate_acti The set of allowed actions
reason: The reason generated by the student agent to justify their action
action: The action taken by the student agent
You will also be provided the privileged state that contains hidden information that specifies how to sclve the task.
### Task
Analyze the student trajectory and summarize the mistakes it
making when trying to solve the game
Refer to your privileged state to know how the game can be solved
Generate IMPROVED reason and action for the student at every timestep to guide them towards the goal
Base your improved reasons solely on the student's s historical observations and actions up to each timestep
NOT include any information. from your privileged state in the improved reasons, the student dees not have
those Offer GENERAL principles or hints your improved reasons that explain why the student should prefer your suggested action over their original action. This would help the student generalizo better.
When generating improved reason, action at timestep t, assume that the student has followed their original trajectory up until timestep t. Copy over the original observation at timestep t from the student trajectory.
When generating improved action at timestep make the action available in the aindidate_actionn at timestep t. Don't select an action that is not available.
Important: Provide GENERAL principles or hints in your improved reasons that explain why the student should prefer your suggested action over their original action.
If you know from your privileged state that an object is in a different lccation from where the student is exploring, use common sense rationale not directly instruct the student to go to the desired location, as they do not have access to the privileged information you possess
guide the student
For example, if the atudent is exploring the wrong area, instead of stating the object in a di fferent location, suggest general
26

---

Published as a conference paper at ICLR 2025
because ", ".. this item often found such places", etc By following these steps, you help the student understand the logic
principle like, "It might useful to explore
behind the actions without revealing privileged information
Also note that you cannot carry more than one item at time
### Cutput
The output is a JSON containing a summary and a trajectory with the same Length as the input student trajectory as tollows: jjonon
"summary"; your aummary of the mistakes the student is making, "trajectory";
"timestep": Index of the current timestep,
"oriigina__dbservariioo" The original observation made by studont agent at timestop t (copy as is from student trajectory), "original_roason:: The original reason generated by the student at timestep t (copy "originallactionn: The original action the student tock at timestep t (copy as is from student trajectory), "corrected_rease"" should generate,
is from student trajectory),
The corrected reason that the student
irorrected_actinn The corrected action that the student should take (chosen from 1ist candidate actions at timestep t) }
18 endif (8 not system $
The student trajectory is below: ([student_trajectorry)
The privileged state for the task is below: spratil
Provide the 44 Output in the JSON format specified above. [8 endif uun
E.1.2 STUDENT AGENT TEMPLATE
18 if mode "input" $1
You are an intelligent assistart named ALFRED in text-based interactive game called TextNorld. Your objective is to complete the given tasks by reasoning through the information provided and taking appropriate actions.
Your task is the following: ([task))
Below the history previous observations and actions: {f Abserattoo..cciion.eeee.. }
Given the hi history of previous observat on and action above, reminder that your task is:
27

---

Published as a conference paper at ICLR 2025
[[task]]
You are given as input the current observation and the list of possible candidate_actions:
"observation": [[oseervttion],, rcandidate_actione ( ((candidate_actioons)
Your goal generate the action
this time step (chosen from candidate_acte ceciiatteeactnsssseeeeee along with the reason for taking the action
Please follow these general instructions:
You MUST choose action from the list of sendidate_actiona
* If "obsorvation": "Nothing happons." ", it is because you chose an invalid action not from the list of candidate_acttons in the previous timestep.
Oftentimes the task requires you interact with objects not present in your observation. You must search the environment to lecate the objective.
Consuit the history of previous observations and actions to what actions you have tried already NOT repeat the same action as the last action ietervationaationhiieee
to not repeat your actions. your It's going to yield the same result. Make sure action is VERBATIM copied from the list of candidate_actions.
You need to generate response in the following format. Please issue only a single action at a time.
REASON:
Rationale for what action to take next based the task and previous history. In your reason, consult candidate_actions to precisely state VERBATIM which action you will do.
ACTION:
The action to be taken, chosen ONLY from candi _action
[& elif mode 'output'
REASON:
[I reason 11
ACTION:
{[ action }}
[8 endif
E.1.3 AUTOLABEL REASONS TEMPLATE
{& if system $
You given JSON file containing observations and actions generated by an expert agent solving text world games, Your goal to generate the reasoning process of the agent to justify each of their actions.
888 Input
The input is a JSON array of objects, each containing: "observation'; The state observed by the agent at specific timestep
'action': The action taken by the agent in response the observation.
Example input format: 'sjoon
28

---

Published as a conference paper at ICLR 2025
"observation"; "You see locked door.", "action": "Use koy on door."
b
]
...
### Output
The output should be JSON array of objects, each containing:
"observation" The state observed by the agent at specific timestep
'reason': The zeasoning behind the action taken by the agent. 'action': The action taken by the agent in response to the observation.
Example output format: json
"observation", "You see a locked door.",
"reason": "The door is locked and I have key that might open
it.",
"action": "Use key door."
b
]
**# Requirements
1. **Causal Constraintse=: Ensure the reasoning satisfies causal constraints. Use only the observatiens up to the current timestep justify the action. not future information. BBrevity and Relevance**: The reasoning should bricf and relevant, using the observations up until that timestep to justify the action.
3. Sequence Consistencyax Knsure the output maintains the same sequence of observations and actions in the input.
18 endif 8)
{8 if not system $
The input JSON file below: ((input))
Provide the output in the format specified above. (8 endif
E.2 WEBSHOP
E.2. EXPERT CORRECTIONS TEMPLATE
{ system
You are given a trajectory containing observations, reason and actions generated by a student agent solving a shopping task by searching for items in website and clicking en links. You are teacher who has information sufficient student. Your goal is to improve how the student solves the task by improving their reason and action at every timestep.
"privileged state" that contains secret hidden from the
solve the task
#88 Input
29

---

Published as a conference paper at ICLR 2025
You will be provided with JSON file that logs the stucent agent's observation, candidate_actions reascn and action at every timestep while solving a shopping task. The student fails to solve the task within the time horizon of the task.
The structure is array of objects, containing the following each timestep:
timestep: Index of the current timestep
observation: The observation provided to the student agent iantidate_actions The set of allowed actions
reason: The reason generated by the student agent justify their action
action: The action taken by the student agent
You will also be provided the privileged state that contains hidden information that specifies how to solve the task.
### Task
Analyze the student trajectory and summarizo the main mistakes it is making when trying solve the task
Refer to your privileged state to know how the task can be solved
* Correct select set of timesteps where you know the student was definitely doing a wrong action
Generate IMPROVED reason and action for the student every timestep to guide them towards the goal
Base your improved reasons solely on the student's s historical
observations and actions up to each timestep
NOT include any information. from your privileged state in the those
improved reasons, the student dces not have
Offer GENERAL principles or hints your improved reasons that explain why the student should prefer your suggested action over their original action. This would help the student generalize better.
When generating improved reason, action at timestep t, assume that the student has fol lowlo their original trajectory up until timestep t. Copy over the original cbservation at timestep t from the student trajectory.
when generating ng improved action at timestep t, make sure available in the candidate candidate_actions actions at timestep t. Don't select an action that is not available. If candidate actions has only click actions, you can only choose click action from the list. If candidate candidate_actionn _actiooss has only search[<search query>], you must search by generating the search keywords on your own.
action
Important:
(1) Provide GENERAL principles or hints in your improved reasons that explain why the student should prefer your suggested action their original action.
If you know from your privileged state that the desirea product is different from the one the student is considering, use connon sense rationale guide the student
rectll instruct the student to search the ground truth product, as they da not have access to the privileged information you possess
By following these steps, help the student understand the logic behind the actions without revealing privileged information
(2) Put corrections sparingly.
Be strategic about which timesteps you want to correct. If you think an action is good enough reasonable, don't bether and action.
correcting. Just copy
the original
(3) Correct indecisive behavior of the student agent Often times, you will see the student agent not finishing the task and insteag continually browsing through items.
30

---

Published as a conference paper at ICLR 2025
Provide corrections key moments help it resolve indecisiveness. It's s better purchase suboptimal item than not finish the task within the time horizon.
The student gets partial points for matching a subset of the criteria in the instruction. They get 0 peints for not finishing the task the time herizon (max timesteps in student trajectory)
(4) Considerations for search.
Specifying prices in search does not work. Other specitications are fine.
Only correct egregious failures search queries, e.g. searching for specificn ficatiin not mentioned in the instruction. Otherwise search usually does not require any correction.
(5) Considerations for click. When suggesting corrected action,, make sure this action exists in the list of candidate_action at that timestop.
### Output
The output is JSON containing summary and trajectory with the same length as the input student trajectory as follows: "ijson
"summary"; your sumnary of the mistakes the student is making, "trajectory";
"timestep"; Index of the current timestep, "is_corrected"; True/False depending whether this timestep corrected or not
"corrected, r The corrected reason that the student should generate. If is_corrected=Falls copy over original reason.
The corrected action that the student should take (chosen from list candidate_action at timestep t) s__oorrected-Palse copy over criginal action. click ontion, make sure corrected_action belongs te 1ist of candi dade_actttttion at the corresponding timestep in input student_trajectory
(corrected_action)
}
{8 endif $)
[8 if not system
The student trajectory is below: [Istudent_trajecttor]]
The privileged state for the task is below: ([privillged_state))
Provide the ++# Output in the JSON format specified above. 18 endif 8)
E.2.2 STUDENT AGENT TEMPLATE
(8 if mode == input' $)
You are an intelligent shopping agent. Your objective is to solve shopping tasks searching for items in website, clicking 1inks until you solve the task.
31

---

Published as a conference paper at ICLR 2025
Below the history previous observations and actions: {f osservation }}
You are given input the current observation that contains
instructions for the task and contents of the current webpage. You are also given list of possible candidate_aattioss
"observation"; [lobservation)), sotaiidate_cctionn [|candidate actions
Your goal is to generate the action to take at this time step along with the reason for taking the acticn.
Please follow these general instructions:
* You MUST choose action from the list of aandiaat_acctions If candidatc_actions has only click acticns, you can only choese a click action from the list. If candidate_actions has only search search query>l, you must search by generating the search keywords on your own.
when choosing a search action, you MUST follow the format search[< search query>]
Choose search keywords that informative but not too specific too broac. Examples of search:
search [6 foot coaxial cable, pack of 31
search [satin brass frosted hallway light fixture]
when choosing click action, make sure action is VERBATIM copied from the list of seciiidtt__ctiionn Examples of click: click[satin brass frosted] click [buy now]
Often times the task requires webpage. You must search the webpages locate the object and verify indeed matches the specifications in the instructions. not, go back and refine your search.
to find objects not in the current
Consult the history of previous observations and actions to see what actions you have tried already action the last action in your
to not repeat your actions.
repeat the observation action history. It's going to yield the same result. When choosing click action, make sure action is VERBATIM copied from the 1ist of candidate actions.
Important: You must try to finish the task and click buy quickly as pessible. It's better to purchase suboptimal item than not finish the task within the time horizon. You get partial points for matching a subset of the criteria in the instruction. You get points for not finishing the task in the time horizon ( typically 20 timesteps).
You need to generate a response in the following format. Please issue only single action at time.
REASON:
Rationale for what action to next based on the task and previous history. In your reason, consult candidate_acction to precisely state VERBATIM which action you will do.
ACTION:
The action to be taken, choser ONLY from candidate_ actions 'output' $)
I8 elif
REASON:
[[ reason 11
ACTION:
[[ action }]
($ endif $)
32

---

Published as a conference paper at ICLR 2025
E.2.3 AUTOLABEL REASONS TEMPLATE
{8 if system 8)
You are given trajectory containing obscrvations and actions generated by an agent solving web shopping task. Your goal to generate the reasoning process that the agent must have been thinking at each timestep to justify each of their actions.
#*# Input
You will be provided with a JSON file that logs the agent' s observation and action at every timestep while solving the web shop task.
The structure is array of objects, containing the follewing each
timestep:
timestep: Index of the current timestep
observation: The instruction the agent has to solve and the current webpage observed by the agent at a specific timestep.
action: The action taken by the agent.
### Task
Analyze the agent trajectory
Generate
for the agent every timestep. The reason should be in first person, e,g, "I should
Hase your generated reason solely on the historical observations and actions to each timestep
that explain why * For example, if the agent is backtracking by clicking on previous, explain why that is the case
Offer GENERAL principles or hints in your the agent tock the action
NOT include any information from future obsorvations the agent does not know the future at that time Ensure the output maintains the same sequence actions as in the input.
In your REASON, be sure to provide detailed justification for the action. If candidate_action
has only click actions, specity which option from the list are you cheosing as your action.
Often times, the task requires you to select both product and specific options for that product. Be sure to justify why you are selecting particular option for product by referring to the task in your observation.
Often times, the agent settles for an option even though may not be exactly optimal. This is because the agent has fixed time horizon within which they must click buy now, Ensure that your reason raticnalizes why it's good finish the task clicking buy now.
#*# Output
The output is trajectory represented JSON array of objects with the length the input trajectory follows: jjon
"tinestep"; Index of the current timestep, "reason": reason to justify the action given observation and action up until the current timestep "action": The action taken by the agent (copy as is from input trajectory) b
18 endif $)
[$ if not system $)
The input trajectory balow:
33

---

Published as a conference paper at ICLR 2025
((input])
Provide the 448 Output in the JSON format specified above. ($ endif $
E.3 INTERCODE
E.3.1 EXPERT CORRECTIONS TEMPLATE
I8 if system 8
You given trajectory containing observations, reason and actions generated by student agent interacting with MySQL Database system using sql queries to answer a question. You are a teacher who has access to a "privileged state" that contains secret information sufficient to solve the task but is hidden from the student. Your goal is improve how the student solves the task by improving their reason and action at every timestep.
#*# Input
You will be provided with JSON file that logs the student agent's observation reason and action at every timestep while solving a shopping task. The student fails to sclve the task within the time horizon of the task.
The structure is array of objects, containing the following each
timestep:
timestep: Index of the current timestep
observation: The observation provided to the student agent
reason: The reason generated by the student agent to justify their action
action:
The action taken by the student agent
You will also be provided the privileged state. The privileged state contains example of optimal SQL command to solve the task and the expected observation that solves the task. Note that it the observation that the student gets evaluated on. Multiple possible actions can lead to that observation.
### Task
Analyze the student trajectory and summarize the main mistakes the student is making when trying to solve the task. Refer to your privileged state to understand how the task can be solved optimally.
Correct seleet set of timesteps where the student's s action clearly wrong.
* Generate <*IMPROVED REASONS** and **ACTIONS** for the student at each timestep to guide them towards the geal. The improved reason should framed as if it is the student's new reasoning, e.g., should ...".
The improved reason and action are intended to replace the student's original reason and action at that timestep. Do NOT refer to the original incorrect reason and acticn when generating improved reason and action.
Base your improved reasons solely on the student's historical observations and actions up to that timestep.
Do NOT include any information. from your privileged state in the improved reasons, the student does not have information.
that
offer general principles or hints in your improved reasons that explain why the student should prefer your suggested action over their original action. This would help the student generalize better.
34

---

Published as a conference paper at ICLR 2025
When generating improved reason and action timestep 't', assume the student followed their original trajectory up until timestep t'.
Important:
(1) Provide v*GENERAL principles hints** in your improved reasons. you know from your privilaged state that a particular set of tables should be queried to solve the task, suggest reasons and actions that guide the student to first discover those tables.
not directly instruct the student to output the privileged action, as they do not have access to privileged information.
By following these steps, you help the student understand the logic behind the actions without revealing privileged information.
(2) Apply corrections sparingly.
Be strategic about which timesteps you choose to correct.
### Output
The output is JSON containing summary and trajectory with the same length as the input student trajectory as follows: "ijson
"summary": your sumnary the mistakes the student is making, "trajectory";
"tinestep"; Index of the current timestep, "original_reason"; The original reason generated by the student
at timestep (copy as is from student trajectory), "original_action"" The original action the student took at timestep t (copy as is from student trajectory), "is_corrected"; True/False depending on whether this timestep is corrected or not,
"corrected__eason":: The corrected reason that the student should generate at timestep t. is_correctad-Pale leave blank "corrected_act The corrected action that the student should take at timestep t. If is cs_correcte--aaeee leave blank
}
{8 endif $)
[8 if not system
The student trajectory is below: [Istudent
The privileged state for the task is below: ([privileged_state))
Provide the ++# Output in the JSON format specified above. 18 endif 8)
E.3.2 STUDENT AGENT TEMPLATE
(8 if system 8
You are given trajectory containing observations, reason and act generated by student agent interacting with MySQI, Database system using sql queries to answer question. You teacher
35

---

Published as a conference paper at ICLR 2025
who has information sufficient solve the task student. Your geal is to improve how the student solves the task by improving their reason and action at every timestep.
"privileged state" that contains secret hidden from the
#*# Input
You will be provided with JSON file that logs the student agent's observation, reason and action at every timestep while solving a shopping task. The student fails to solve the task within the time horizon of the task.
The structure is an array of objects, containing the following at each
timestep:
timestep: Index of the current timestep
observation: The observation provided the student agent
reason: The reason generated by the student agent to justify their action
action:
The action taken by the student agent
You will also be provided the privileged state. The privileged state contains an example of an optimal SQL command to solve the task and the expected observation that solves the task. Note that it is the observation that the student gets evaluated on. Multiple possible actions Lead to that opservation.
H0 Task
Analyze the student trajectory and summarize the main mistakes the student is making when trying to sclve the task.
Refer to your privileged state understand how the task be solved optimally.
Correct select set of timesteps where the student's s action is clearly wrong.
Generate ><IMROVVDD REASONS** and *CCTONNS** for the student at each timestep to guide them towards the goal, The improved reason should be framed as if should ...".
the student's s new reasoning e.g.,
The improved reason and action are intended to replace the student/s original reason and action that timestep. NOT refer the original incorrect reason and action when generating improved reason and action.
Base your improved reasons solely the student's historical abservations and actions up to that timestep.
NOT include any information from your privileged state the improved reasons, as the student does not have access to that information.
Offer general principles or hints in your improved reasons that explain why the student should prefer your suggested action their original action. This would help the student generalize better.
When generating an improved reason and action at timestep 't', assume the student followed their original trajectory up until tirestep
Important:
(1) Provide G-GENERAL principles or hints** in your improved reasons. you know from your privileged state that particular tables should queried to solve the task, suggest reasons and actions that guide the student to first discover those tables.
Do not directly instruct the student to output the privileged action, as they do not have access to privileged information. By folloxing these steps, you help the student understand the logic behind the actions without. revealing privileged information.
(2) Apply corrections sparingly. strategic about which timesteps you choose correct.
36

---

Published as a conference paper at ICLR 2025
808 Output
The output is JSON containing summary and a trajectory with the same length the input student trajectory as follows: jjono
"summary": your sumnary of the mistakes the student is making, "trajectory";
"tinestep": Index of the current timestep, "originaiireason"; The original reason generated by the student at timestep t (copy as is from student trajectory), leligginl_acction The original action the student took at
timestep (cepy as is from student trajectory), "ssccoreeced": True/False depending on whether this timestep is corrected or not,
"corrocted_roason The corrected reason that the student should generate at timestep t. If ls_corrected-Fale, lcave blank "corrected_actione The corrected action that the student should take at timestep t. is_corrected-Palse leave blank
[3 endif $)
{3 if not system 8)
The student trajectory is bolow: [[student_traapcterrll
The privileged state for the task is below: ((privileged_state))
Provide the H+ Output in the JSON format specified above. 18 endif
37


Reviews:
[{"review_id": "XqZc4ZLVNQ", "reviewer": "Reviewer_EhvG", "summary": "The paper proposes LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. The key insight is to equip the expert teachers with a privileged state - additional information available during training but hidden at test time. This allows weak experts to provide guidance, improving the student agent's performance. LEAP is evaluated on diverse decision-making benchmarks, including text-based games, web navigation, and interactive coding, and shows that it outperforms state-of-the-art baselines, enables weak student models to exceed the performance of strong teacher models, and allows weak models to self-improve using privileged versions of themselves.", "questions": "**Q1:** In the proof of theorem B4, there is a step using the performance difference lemma, but I did not see the connection here. Could you provide the derivation of how you get the inequality?\n\n**Q2:** In the contribution section, Webshop is reported as achieving 61.8%. However, the metric used in the experiment is the score, which is usually much higher than the success rate. Could you clarify what metrics you are actually evaluating?\n\n**Q3:** Could you provide the self-correct results (using Llama3-8B as expert teacher) for WebShop and Intercode?\n\nThe overall quality of the paper is good, but there are still some weaknesses and questions that need to be addressed. I will reassess the score based on the authors' clarifications.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed method looks valid and outperforms the baseline (ReAct).\n* Evaluate three diverse environments: ALFWorld, WebShop, and Intercode.\n* Provide detailed experiment and theoretical analysis of the trade-off between realizability and privileged information.\n* The proof appears to be correct, and the prompts are included in the Appendix.", "weaknesses": "* The analysis of the costs associated with iterative fine-tuning and calling LLM experts was not included. While the authors assert that their method is not comparable to Reflexion [1], Reflexion appears to be a more cost-effective approach for enhancing the performance of LLM agents with minimal trials.\n* It seems somewhat inequitable to compare this method with those that do not utilize privileged information, as such information is typically not available in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jQbIep3YiS", "reviewer": "Reviewer_E725", "summary": "This is paper is about automatic self-improvement from errors during task execution. They proposed LEAT in which LLM agents as students use feedback from expert teachers to improve itself. The teachers hav access to a privileged state —information available during training. So then even weak teachers are effective. They evaluate that on including text-based games, web navigation, and interactive coding. The results show that it outperforms state-of-the art baselines, enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves.", "questions": "- How to provide those privilege state in more complicated tasks?\n- Trade-off between using privileged information and student realizability is not clear! \n- Compute privileged state s_t for every datapoint in Di in the algorithm is not clear. That is the limitation part of the algorithm. \n- how?  For each time step in Di, LEAP computes the privileged state s_t by leveraging information available only during training\n- What is the performance of LEAP vs ReAct but using the same privilege state?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- well written.", "weaknesses": "- Lack of good reference! Relevant references are missed in introduction! It needs to provide proper reference for each claim!\n- The issue of experts is how to prepare knowledge for the experts, how to distinguish between different experts. This is not clear in this work and it needs manual set up which is not applicable.\n- They are very similar works in the literature but with different names instead of experts! In that way the innovation of the work is very limited. \n- No Fair comparison at all!! The work did not consider fair baselines! There are a lot of up to date methods that works better than ReAct and BUTLER! There are a lot of works from self-correctness that they do the same things. \n- In general the contribution of the work is very limited! The teachers should know privilege knowledge however it is not always available! Even with that still prompting and few shot are needed which makes it the same as other prompting works or self correction works but here the authors just used different terms.  And it is very trivial that there should be some improvement vs e.g. react since it is literary using extra information of the environment and more over it uses a suggestion from a larger LLM with that extra information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2GuZX9rAaA", "reviewer": "Reviewer_nN6Y", "summary": "The paper seeks to address LLM Agents' difficulties when it comes from recovering from errors during task execution. \n\nThey begin by framing decision-making as a Partially Observable Markov Decision Process (POMDP). \nGiven this framing, they approach the issue by proposing LEAP, a DAGGER-like iterative training framework in which an expert teacher is equipped with \"privileged state\" corrects the faulty student agent trajectories during training. \n\nHere, the \"privileged state\" is simply a representation of the environment state that contains more information than what is available to the student. \n\nAfter outlining the problem and LEAP, the authors briefly examine their setup, analytically showing that there is a tradeoff between the performance and realisability of the privileged expert teacher.\n\nThe paper then moves on to the experiments, where the authors test their framework against a number of baselines across a variety of domains (text, web agents, and coding), demonstrating improvements in performance. Their experiments also show the possibility of student agents surpassing their teachers and of self-improvement when using a privileged version of themselves as the teacher. The authors also experimentally explore the tradeoff between how much privileged information to use and how realisable the corrections are. Finally, an ablation on the fine-tuning mechanism in LEAP demonstrates that SFT outperforms DPO and KTO in this setting.", "questions": "My main question is already mentioned above under \"weaknesses\". If we are allowed to show privileged information to the teacher, which then uses this to correct the student, why not directly show the privileged information to the student, especially given the student-teacher relationship is not adversarial? Which real-world scenario is this setup replicating?\n\nUPDATE 25/11/2024: This question and some of the weaknesses have been addressed in the rebuttal, so I have updated my review score from a 5 to a 6.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper provides a solid theoretical grounding in both the main text and appendix, especially around the analysis on the trade-off between privileged information and the agent's realizability.\n- I appreciated the variety and number of domains considered experimentally, which showed the wide applicability of the method.\n- The ablations on the fine-tuning methods, across SFT, DPO and KTO were greatly appreciated, as it was otherwise unclear which of these to pick from\n- Beyond the general experimental results, I found the questions considered relatively original: whether an agent can bootstrap to self-improvement, whether it can outperform its teacher, and what tradeoffs are faced in terms of amount of privileged information and correction realizability. These were all quite interesting results to read.\n- The paper explicitly acknowledges some its limitations, which is commendable and contributes to the paper's transparency and reliability in my view.", "weaknesses": "- I found the author's choice of framing decision making as a POMDP somewhat arbitrary and under-justified. There is plenty of valid work that studies decision making using fully observable MDP's, and the connection to the real world is a bit forced\n- Aside from the above, the main weakness in my view is that the authors never address/consider/justify the absence of the very obvious alternative: if you are able to train using privileged information anyway, why not simply show this information directly to the student rather than through the teacher? If the privileged information is accessible (by the teacher), this is not really a POMDP anymore really. What analogous real world scenario is this setup trying to address? It appears somewhat ill-defined from what i can tell.\n- Somewhat related to the above point, but I found some of the results somewhat obvious and not particularly novel: of course an agent which (indirectly) has additional privileged information will do better than one without such information. It is similarly not surprising that the student eventually outperforms its teacher, given that only the student is trained and is probably comparatively overfitting to the objectively. \n- I found section 3.3. rather unclear and perhaps underdeveloped in the main text. For instance, \"realizability\" on its own is never clearly defined. I would have devoted more of the paper to this analysis, perhaps sacrificing one of the domains from 4.1-4.4 to the appendix if space was necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hedVtOaPgZ", "reviewer": "Reviewer_Yn4v", "summary": "The authors propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Notably, they equip the expert teachers with a privileged state. They evaluate LEAP on several decision-making benchmarks, and their experiments show that LEAP enables weak student models to exceed the performance of strong teacher models.", "questions": "1. Have the author tried to perform preference optimization after SFT? Will this lead to a even higher score?\n\n2. The authors admit that generating interaction rollouts is time-consuming. How long does it typically take in terms of magnitude—hours, days, or even longer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel methods: There have been a huge amount of research papers discussing about LLM self-improvement. However, the idea of equipping the expert teachers with a privileged state is relatively novel.\n\n2. Good experimental results: LEAP significantly outperforms all baselines, even the stronger teachers like GPT-4o.\n\n3. Diverse experiments: The authors conduct various experiments across several fundamentally different benchmarks.", "weaknesses": "1. Privileged states can be hard to get in real world environments (though this may not fit in the scope of this work).\n\n2. Training an agent for each benchmark seperately is specific. The results would be much more promising if the authors demonstrate the LLM can be finetuned on all benchmarks together and still achieve good results. \n\n3. Some minor typos: For example, in line 24, GPT4-o should be written as GPT-4o. In line 127, $\\rho_t^*,a_t^*$ should be $(\\rho_t^*,a_t^*)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)