Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

## Abstract

Abstract While large language models (LLMs) show impressive decision-making abilities, current methods lack a mechanism for automatic self-improvement from errors during task execution. We propose LEAP , an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Our key insight is to equip the expert teachers with a privileged state —information available during training but hidden at test time. This allows even weak experts to provide precise guidance, significantly improving the student agent’s performance without access to privileged information at test time.
We evaluate LEAP on diverse decision-making benchmarks, including text-based games (ALFWorld), web navigation (WebShop), and interactive coding (Intercode Bash). Our experiments show that LEAP (1) outperforms behavior cloning and ReAct baselines (2) enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves. We also provide a theoretical analysis showing that LEAP ’s success hinges on balancing privileged information with the student’s realizability, which we empirically validate. Our code is available at https://leap-llm.github.io .

## 

## Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

## Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

### 1 Introduction

Large language models (LLM) show impressive decision-making abilities across domains (Yao et al., ; Jimenez et al., ; Sodhi et al., ). However, they struggle to learn and recover from errors encountered at test time. Prompting alone is insufficient, as it requires manually specifying exceptions, which is difficult to scale and leads to intractably long context lengths.

We address the problem of fine-tuning LLM agents to improve their decision-making using online interaction data. Imitation learning offers a sample-efficient way to train such models, where a teacher corrects the student agent’s actions. However, finding an expert capable of both demonstrating the task and correcting mistakes from any state the student visits is extremely challenging. The expert must not only know how to complete the task but also how to recover from the student’s errors, which is often more complex than simply performing the original task. For example, in a task like "Heat the mug and put it in the cabinet," if the student places the wrong object in the microwave, the expert must first backtrack and remove the incorrect object before proceeding with the task.

*Our key insight is to equip the expert teacher with access to a privileged state* — additional information unavailable to the student during test time, but crucial to improve the expert’s ability to offer corrective actions at train time. This may include hidden states of a partially observable environment or intermediate subgoals. For the task "Heat the mug and put it in the cabinet", the privileged state reveals the mug’s location and necessary steps for heating it (Fig. 1). With this extra knowledge, the expert can first clear the microwave of any incorrect items and then teach the student effective search strategies, like checking common mug locations. Over time, this enables the student to not only learn how to perform tasks correctly but also how to recover from mistakes, ultimately allowing the student to outperform the expert, even without access to the privileged state at test time.

We propose Learning from Experts with Access to Privilege (LEAP), an iterative learning algorithm that fine-tunes LLM agents using privileged AI expert feedback. As shown in Fig. 1, LEAP begins with a student agent trained on demonstration data ($\pi_{0}$). During interactions, the student agent generates trajectories, and the privileged expert ($\pi^{E}$) identifies key time steps where corrections are needed. The expert provides improved reason and action labels based on the privileged state, which are then assembled into a dataset used to update the student model ($\pi_{i}\leftarrow\pi_{i-1}$). This process is repeated iteratively, progressively improving the student agent’s performance.

Interestingly, we identify a key *trade-off between leveraging privileged information and the student agent’s realizability*. While providing unrestricted privileged feedback allows for highly accurate corrections, these are often too complex for the student to act on effectively. Conversely, limiting feedback to what the student can easily process makes it realizable but often suboptimal. Our theoretical and empirical analyses show that balancing these extremes is key, which we achieve by introducing a constrained privileged expert that provides optimal yet realizable feedback.

Figure: Figure 1: LEAP overview. LLM student agent interacts with the environment, generating a reason-action trajectory (in orange) based on its policy $\pi_{i-1}$. An expert teacher, with privileged state available only during training, evaluates and corrects the trajectory (in green). These corrections update the learner’s policy to $\pi_{i}$ through SFT/DPO training. Updated policy $\pi_{i}$ is then rolled out at test time without access to privileged state.
Refer to caption: x1.png

Our key contributions are:

- 1.
A novel iterative learning framework LEAP that fine-tunes LLM agents using privileged expert feedback, balancing privileged information and realizability.
- 2.
Theoretical and empirical analysis of the optimal trade-off between privileged information and agent realizability.
- 3.
Experimental validation on diverse interactive decision-making benchmarks: AlfWorld (Shridhar et al., ), WebShop (Yao et al., ), and InterCode (Yang et al., ).
(a)
LEAP consistently outperforms behavior cloning agents trained on gpt-4o or human demonstrations, showing significant improvements on ALFWorld ($65.7\%\rightarrow 91.8\%$), WebShop ($29.4\%\rightarrow 61.8\%$), and InterCode Bash ($60.3\%\rightarrow 71.8\%$).
(b)
LEAP enables weaker models (e.g. Llama3-8B) to surpass stronger models (gpt-4o), with improvements on ALFWorld ($91.8\%$ vs. $65.7\%$), WebShop ($61.8\%$ vs. $58.4\%$), and InterCode Bash ($71.8\%$ vs. $71.3\%$).
(c)
Ablations that show LLM agents can self-improve using their privileged versions as teachers ($65.7\%\rightarrow 82.1\%$), explore the tradeoff between privileged information and realizability, and compare supervised finetuning with preference optimization.

### 2 Preliminaries

Problem Formulation: Decision-making LLMs.
We frame decision-making as a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., ) because real-world environments often contain hidden or incomplete information. The POMDP is defined by the tuple $\langle\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},r\rangle$, where $\mathcal{S}$ represents the underlying state space of the environment, $\mathcal{A}$ the action space, $\mathcal{O}$ the observation space, $\mathcal{T}:\mathcal{S}\times\mathcal{A}\to\mathcal{S}$ the transition dynamics, and $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ the reward function. The true state $s_{t}\in\mathcal{S}$ is not fully observable; instead, the agent receives partial observations $o_{t}\in\mathcal{O}$ at each timestep. Hence it must rely on the history $h_{t}\in\mathcal{H}$ of observation-actions to choose actions.

LLM-based agents are policies $\pi\in\Pi$, which predict both a *reason* $\rho_{t}$ and an action $a_{t}$ based on the agent’s history $h_{t}$. The reason $\rho_{t}$ is expressed in natural language to justify action $a_{t}$.
The history $h_{t}={o_{1},a_{1},\dots,o_{t}}$ is the sequence of past observations and actions. The policy is expressed as:

$$ $(\rho_{t},a_{t})=\pi(h_{t})$ (1) $$

While reinforcement learning (RL) for POMDPs is already challenging due to the large history space $\mathcal{H}$ (Papadimitriou & Tsitsiklis, ), this is further exacerbated by the addition of reason space, which is infinite and unconstrained. This makes RL impractical to run for LLM agents.

Interactive Imitation Learning with DAgger. Imitation learning (IL) offers a promising alternative to RL, by allowing the agent to mimic an expert teacher policy $\pi^{E}$ bypassing the need for complex exploration. The gold-standard for imitation learning are interactive methods like DAgger (Ross et al., ) that query the teacher $\pi^{E}$ for feedback on states $h_{t}$ visited by the student. At each DAgger iteration $i$, the student policy $\pi_{i}$ is rolled out to collect histories $h_{t}$, and the teacher provides corrective reasons and actions $(\rho^{*}_{t},a^{*}_{t})$ on those histories $h_{t}$. The new student $\pi_{i+1}$ then minimizes the cumulative loss $\ell(.)$ on all the data up until the current iteration $i$:

$$ $\pi_{i+1}=\min_{\pi}\quad\sum_{j=0}^{i}\mathbb{E}_{h_{t}\sim\pi_{j},\rho^{*}_{ t},a^{*}_{t}\sim\pi^{E}(h_{t})}\left[\ell((\rho^{*}_{t},a^{*}_{t}),\pi(h_{t})) \right],$ (2) $$

DAgger guarantees that as iterations $N\rightarrow\infty$, it will find the best policy in policy class $\Pi$.

### 3 Approach

We present Learning from Experts with Access to Privilege (LEAP), a framework for iteratively training an LLM agent through online AI feedback from a privileged expert (Fig. 1).
Central to our approach is the concept of a *privileged expert teacher*, which leverages access to privileged state information to guide the student’s recovery from errors. Section 3.1 defines the privileged teacher, Section 3.2 details LEAP, and Section 3.3 presents the theoretical analysis of LEAP.

#### 3.1 Privileged Expert Teacher

An expert teacher is a policy that can be queried with student history $h_{t}$ of observations and actions, along with the student’s predicted reason action $(\rho_{t},a_{t})$ to generate a corrected reason action $(\tilde{\rho}_{t},\tilde{a}_{t})\sim\pi^{E}(\cdot|{r}_{t},{a}_{t},h_{t})$. For POMDPs, computing such corrections is intractable and strictly more challenging than just solving the original task, often resulting in suboptimal feedback.

We define a *privileged expert teacher* as a teacher that has access to privileged state information, $s_{t}$, which fully describes the environment at a given time. This access transforms the teacher’s decision-making into a fully observable Markov Decision Process (MDP), making it significantly more tractable. We formally define the privileged expert teacher as:

$$ $(\tilde{\rho}_{t},\tilde{a}_{t})\sim\pi^{E}(\cdot|{\rho}_{t},{a}_{t},h_{t},s_{ t})$ (3) $$

The privileged state $s_{t}$ can either be explicitly provided or is implicit in privileged text information provided to the expert. For instance, in Alfworld (Shridhar et al., ), it includes the exact location of objects and the goal necessary for the agent to navigate effectively. In WebShop (Yao et al., ), it involves the specific name of the target product and its attributes, aiding precise search queries or recommendations. Similarly, in InterCode (Yang et al., ), the privileged state consists of the ground truth code and any unit tests the code must pass, providing a clear benchmark for the agent to generate or correct code. Note that the privileged information is available only at train time.

A crucial requirement for the privileged expert teacher is that *it must not reveal privileged information in its feedback to the learner.* The prompt instructions specify that the teacher should provide general reason and action that avoid the direct use of privileged information. Using such information directly might solve tasks quickly but makes the corrected actions $(\tilde{\rho}_{t},\tilde{a}_{t})$ unrealizable for the learner, who does not have access to the privileged state $s_{t}$, leading to a significant imitation gap and poor learner performance. In Section 3.3, we characterize the trade-off between using privileged information and student realizability. We show in Appendix. B that a constrained privileged expert that effectively utilizes privileged information while remaining realizable for the student has the optimal performance. We approximate such an expert by designing prompts for the expert to instruct it to use privileged information, without revealing it. See Appendix D for prompts.

#### 3.2 The LEAP Algorithm

Figure: Algorithm 1 LEAP: Iterative Learning with Privileged Expert Teacher

The LEAP algorithm iteratively trains the student policy $\pi$ through online interactions with the environment $\mathcal{E}$ and corrections from a privileged expert teacher $\pi^{E}$. The process begins by collecting an initial set of demonstrations of history, reason, and action from the teacher $\mathcal{D}\leftarrow\{h^{*}_{t},\rho^{*}_{t},a^{*}_{t}\}$, and training a BC policy $\pi_{0}$ on this data using supervised fine-tuning (SFT).

At each iteration, the current learner policy $\pi_{i}$ is rolled out in the environment to generate new data $\mathcal{D}_{i}\leftarrow{(h_{t},\rho_{t},a_{t})}$. For each timestep in $\mathcal{D}_{i}$, LEAP computes the privileged state $s_{t}$ by leveraging information available only during training. The privileged teacher is then invoked on trajectories where the agent fails, generating corrected reasoning and actions, $\tilde{\rho}_{t},\tilde{a}_{t}\sim\pi^{E}(.|\rho_{t},a_{t},h_{t},s_{t})$. These corrections are used to augment the dataset $\mathcal{D}_{i}$. The policy $\pi_{i}$ is subsequently updated using this augmented dataset through a no-regret learning method. This iterative process continues, refining the policy over multiple iterations until the best-performing policy is selected based on validation.

The update step in LEAP should be designed to ensure no-regret learning, a property that guarantees the policy’s cumulative performance will converge to that of the best possible policy in hindsight. The most common update is to aggregate data $\bigcup\mathcal{D}_{i}$, and train $\pi_{i}$ using supervised fine-tuning (SFT) on the aggregated dataset, as done in DAgger (Ross et al., ). This is equivalent to Follow the Regularized Leader (FTRL), a no-regret update step. An alternate update is to treat the problem as a preference optimization problem where the corrected reason action $(\tilde{\rho}_{t},\tilde{a}_{t})$ is preferred over the student reason action $(\rho_{t},a_{t})$. Preference optimizers like Direct Policy Optimization (DPO) (Rafailov et al., ) or Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., ) optimize a preference loss while regularizing to the base policy, in this case the previous policy $\pi_{i-1}$. This is equivalent to an Online Mirror Descent (OMD) update, which is also a no-regret update. We use SFT by default but compare it against preference optimization as an ablation in the results.

#### 3.3 Analysis

We briefly touch on the theoretical guarantees of LEAP, and refer the reader to Appendix. B for a full exposition. We define the realizability of an expert policy and show how it affects LEAP performance.

###### Definition 3.1 (Average Realizability Gap) .

The average realizability gap $epsilon(\pi^{E},T)$ between the privileged expert $\pi^{E}$ and the best policy $\pi^{\star}$ over time horizon $T$ is:

$$ $\epsilon(\pi^{E},T):=\sup_{\pi}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t }\sim d^{\pi}_{t}}||\pi^{E}(a_{t}|s_{t})-\pi^{\star}(a_{t}|h_{t}))||_{1}$ (4) $$

where $||.||_{1}$ is the L1 distance, $d^{\pi}_{t}$ is the induced distribution over state and history of *any policy $\pi$*.

###### Theorem 3.2 ( LEAP with Privileged Expert) .

Running $N$ iterations of LEAP with privileged expert $\pi^{E}$ yields at least one policy $\pi$ such that the time-average performance $\frac{1}{T}J(\pi)$ is bounded as:

$$ $\frac{1}{T}J(\pi)\geq\frac{1}{T}J(\pi^{E})-H(\pi^{E})\left(\epsilon(\pi^{E},T) +\gamma(N)\right)$ (5) $$

where $\pi^{E}$ is the privileged expert, $H(\pi^{E})$ is the recoverability coefficient of $\pi^{E}$, $\epsilon(\pi^{E},T)$ is the average realizability gap, and $\gamma(N)$ is the average regret of the DAgger update.

The equation shows a trade-off between performance of the privileged expert (first term) and the realizability of the privileged expert (second term), which we show how to optimize in
Appendix. B.

###### Definition 3.1 (Average Realizability Gap) .

###### Theorem 3.2 ( LEAP with Privileged Expert) .

### 4 Experiments

#### 4.1 Overview of Results

We evaluate LEAP across $3$ decision-making domains: ALFWorld, a text-based game, WebShop, a web navigation environment, and Intercode, an interactive coding environment. Below is a summary:

- 1.
LEAP improves upon base behavior cloning (BC) policy: LEAP significantly improves the base BC policy $\pi_{0}$ over successive fine-tuning iterations, showing gains on AlfWorld ($65.7\%\rightarrow 91.8\%$), WebShop ($29.4\%\rightarrow 61.8\%$), and InterCode ($60.3\%\rightarrow 71.8\%$). See Tables 1, 2, Fig. 3.
- 2.
LEAP enables weaker student models to surpass stronger teacher models: Over successive iterations, student models (e.g., Llama3-8B) outperform their stronger teachers (e.g., GPT-4o), despite starting with lower or comparable performance. For example, on ALFWorld, the student model ($\pi_{3}:91.8\%$) surpasses the teacher ($\pi^{E}:65.7\%$), with similar trends on WebShop ($\pi_{2}:61.8\%$ vs. $\pi^{E}:58.4\%$) and InterCode ($\pi_{3}:71.8\%$ vs. $\pi^{E}:71.3\%$). See Tables 1, 2, Fig. 3.
- 3.
LEAP requires a balance between using privileged information and generating realizable corrections: Experts using minimal privileged information ($\pi^{E}_{1}:73.1\%$) generate more realizable actions, while those with excessive privileged feedback ($\pi^{E}_{5}:5.22\%$) produce unrealizable corrections. $\pi^{E}_{3}$ strikes a perfect balance achieving optimal performance ($\pi^{E}_{3}:91\%$). See Fig. 5.
- 4.
LEAP enables LLM agents to self-improve by using their own privileged versions as experts: LLM agents can self-improve by using their privileged versions as teachers. For example, in ALFWorld, Llama3-8B bootstraps itself through iterations, improving from $65.7\%$ to $82.1\%$ and finally to $91.8\%$. See Fig. 6.
- 5.
LEAP can be trained using both SFT or Preference Optimization: Both SFT and preference optimization improve over the base BC policy ($\pi_{0}$), with SFT delivering stronger gains ($68.6\%\rightarrow 96.4\%$) compared to DPO ($68.6\%\rightarrow 74.3\%$) and KTO ($68.6\%\rightarrow 87.1\%$). See Table 3.

#### 4.2 Domain 1: Text-based games

Setup. ALFWorld (Shridhar et al., ) is a text-based game where each task has a high-level objective, e.g. “heat mug and put it in cabinet” which the agent must complete by navigating and interacting with a virtual household through text commands, e.g. go to shelf 1, pick up mug 2, etc. To solve it, the agent must plan subgoals, track progress, and search for objects efficiently (e.g. mugs are more likely to be on shelves or cabinets). Each task has $30$ timesteps. There are $6$ categories of tasks, training dataset of $3257$ games and two evaluation datasets: $139$ in-distribution games and $134$ out-of-distribution games. We compare against a prior work BUTLER (Shridhar et al., ) and ReAct (Yao et al., ); the original ReAct paper used text-davinci-002 with task-dependent in-context examples. We add ReAct with stronger models such as gpt-4o (^1^11https://platform.openai.com/docs/models), claude(^2^22https://docs.anthropic.com/en/docs/about-claude/models), and gemini(^3^33https://ai.google.dev/gemini-api/docs/models/gemini), and a general instruction prompt. For LEAP, we LoRA (Hu et al., ) fine-tune Llama3-8B (Dubey et al., ) model using gpt-4o as the privileged expert teacher for $4$ iterations. The privileged information for each game contains the essential objects to solve the task, object locations, and the optimal actions. See Appendix. C.2, D.1 for full details and prompts.

**Table 1: AlfWorld Evaluation. (top) $136$ out-of-distribution games and (bottom) 140 in-distribution (max $30$ actions). Baseline comparisons include [1] BUTLER (Shridhar et al., ), [2] ReAct few-shot (Yao et al., ), and our own ReAct instruction prompt with different models. LEAP with an 8B model across iterations ($\pi_{1},\pi_{2},\pi_{3}$) outperforms the stronger teacher ReAct gpt-4o.**
| Method | All tasks | Pick tasks | Clean tasks | Heat tasks | Cool tasks | Look tasks | Pick 2 tasks |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| %suc$\uparrow$ | #act$\downarrow$ | %suc$\uparrow$ | %suc$\uparrow$ | %suc$\uparrow$ | %suc$\uparrow$ | %suc$\uparrow$ | %suc$\uparrow$ |  |
| BUTLER [1] | 35.0 | - | 50.0 | 74.0 | 83.0 | 91.0 | 39.0 | 65.0 |
| ReAct few-shot [2] | 57.0 | - | 65.0 | 39.0 | 83.0 | 76.0 | 55.0 | 24.0 |
| ReAct gpt-4o | 65.7 | 20.2 | 91.7 | 35.5 | 56.5 | 52.4 | 100.0 | 76.5 |
| ReAct gpt-4o-mini | 29.9 | 25.5 | 33.3 | 25.8 | 17.4 | 14.3 | 66.7 | 29.4 |
| ReAct claude-3.5-sonnet | 76.1 | 19.0 | 95.8 | 61.3 | 60.9 | 81.0 | 88.9 | 76.5 |
| ReAct claude-3.5-haiku | 16.4 | 27.2 | 33.3 | 9.7 | 8.7 | 9.5 | 38.9 | 0.0 |
| ReAct gemini-1.5-flash | 19.4 | 26.3 | 41.7 | 12.9 | 13.0 | 19.0 | 16.7 | 11.8 |
| LEAP Llama3-8B | 0.7 | 29.8 | 0.0 | 0.0 | 0.0 | 4.8 | 0.0 | 0.0 |
| LEAP Llama3-8B $\pi_{0}$ | 65.7 | 18.6 | 66.7 | 74.2 | 73.9 | 66.7 | 66.0 | 35.3 |
| LEAP Llama3-8B $\pi_{1}$ | 91.0 | 11.9 | 83.3 | 90.3 | 91.3 | 95.2 | 94.4 | 94.1 |
| LEAP Llama3-8B $\pi_{2}$ | 83.6 | 13.1 | 87.5 | 90.3 | 73.9 | 95.2 | 66.7 | 82.4 |
| LEAP Llama3-8B $\pi_{3}$ | 91.8 | 11.3 | 87.5 | 93.5 | 91.3 | 90.5 | 94.4 | 94.1 |

Figure: Figure 2: ALFWorld Training and Testing for LEAP. (a) Training: Student policy $\pi_{0}$ rolled out on training task to generate reason and actions, e.g. it fails to cell phone because it inefficiently searches all drawers. Expert teacher $\pi^{E}$ uses privileged information (cellphone in desk1) to generate general corrected reason actions that don’t reveal privileged information (cellphones commonly found in desks, more efficient to search desks than drawers). (b) Testing: $\pi_{0}$ fails to find a watch as it inefficently explores shelves one by one. $\pi_{1}$ learns a more efficient exploration policy, prioritizing areas like sidetables and dressers, solving the task quickly.
Refer to caption: x2.png

Results.
Table 1 shows that LEAP significantly outperforms all baselines, with the best policy achieving $91.8\%$ success rate on out-of-distribution tasks. Iteration $1$ of LEAP has the biggest performance gain $(65.7\%\rightarrow 91.0\%)$, leading to a student policy $\pi_{1}$ that surpasses the teacher gpt-4o with higher success rate $(91.8\%>65.7\%)$ and lower actions $(11.3<20.2)$. Note that performance improvements are not monotonic over iterations of LEAP, $(65.7\%\rightarrow 91.0\%\rightarrow 83.6\%\rightarrow 91.8\%)$, however, all policies are better than the BC policy $\pi_{0}$. The regression from $\pi_{1}$ to $\pi_{2}$ are mainly from Heat and Look tasks, which gets corrected immediately in the next iteration $\pi_{3}$. Finally. we note that LEAP policies generalize well from in-distribution to out-of-distribution tasks with only a slight dip in performance for the final policy $\pi_{3}$ $(94.2\%\rightarrow 91.8\%)$.

Fig. 2 (a) shows an example of LEAP training and testing. A typical failure mode of $\pi_{0}$ is that it searches for items inefficiently, e.g. looking at shelves or drawers or cabinets one by one until time runs out. The expert teacher leverages the privileged state of where objects are to show the student how to search efficiently without revealing the object location, e.g. it’s more efficient to search desks where cellphones are likely to be than drawers sequentially one after another. Training on this correction data, $\pi_{1}$ *generalizes this reasoning to new test-time tasks*, e.g. putting a watch in a safe. Fig. 2 (b) shows that while $\pi_{0}$ inefficiently searches shelves, $\pi_{1}$ searches likely locations for the watch, first the sidetable and then the dresser to solve the task. The main performance gains of LEAP over gpt-4o come from such efficient exploration. gpt-4o searches likely locations, however, its prior is from internet data and isn’t necessarily aligned with the statistics of the ALFWorld environment. In contrast, LEAP leverages privileged feedback to learn this. See Appendix C for more results.

#### 4.3 Domain 2: Web agents

Figure: Figure 3: WebShop Evaluation. (a) Overall score$\uparrow$ and #act$\downarrow$ on $500$ test tasks (max $30$ actions). (b) Performance of LEAP over iterations on $4$ different score components. Baseline comparisons include [1] IL (Yao et al., ) and our ReAct instruction prompt with different models. LEAP with an 8B model across iterations ($\pi_{2},\pi_{3}$) outperforms the stronger teacher ReAct gpt-4o.
Refer to caption: x3.png

Setup. WebShop (Yao et al., ) is an online shopping environment where each task is a shopping request from a user (“I need a gluten-free bacon, 4 oz pack of 2, priced under $50”) that the agent has to solve through web interactions (search[“gluten-free bacon”] or click options). Success is determined by an overall score with $4$ components: whether attributes match (Att), whether options match (Opt), whether product types match (Type) and whether price matches (Price). Each task has $30$ timesteps. The training dataset has $12086$ tasks, the test dataset has $500$ tasks. We compare against ReAct (Yao et al., ) with different base models like GPT-4o and GPT-4o-mini. We also compare against IL baseline in WebShop (Yao et al., ), which trains two models, one for search (from a list of keywords) and one for clicking, thus simplifying the problem.
We LoRA fine-tune Llama3-8B model using gpt-4o as the privileged expert teacher for 4 iterations. The privileged information contains product attributes, product option, price constraint, and an example product that would satisfy all criteria. See Appendix. C.3, D.2 for full details and prompts.

Figure: Figure 4: WebShop Training and Testing for LEAP. (a) Training: Teacher policy generates corrections on student rollout to backtrack when the product does not fit criteria or to commit to a product when it does. (b) Evaluation: $\pi_{0}$ fails to solve the task since it continues to search page after page even after discovering a good product. $\pi_{1}$ learns when to backtrack and when to commit to a product to solve the task in time.
Refer to caption: x4.png

Results.
Fig. 3 (a) shows that LEAP outperforms all baselines, with the best policy ($\pi_{2}$) achieving $61.4$ score. Iteration $1$ has the biggest performance gain $(29.4\rightarrow 51.8)$ followed by iteration 2 $(51.8\rightarrow 61.8)$. This leads to a student policy $\pi_{2}$ that surpasses the teacher GPT-4o with a higher score $(61.8>58.4)$. Similar to ALFWorld, while performance improvements are not monotonic, both $\pi_{2},\pi_{3}$ remain above the teacher gpt-4o performance. Fig. 3 (b) shows how LEAP improves on all $4$ components of the score. The biggest improvements between LEAP and gpt-4o is in Opt component $(42.4>22.1)$, indicating gpt-4o fails to select a product with the right option often.

Fig. 4 (a) shows an example LEAP correction of $\pi_{0}$ and $\pi_{1}$ from the training data. The teacher leverages the privileged state such as the attributes or options of the product to teach the student to recognize when an item does not fit the description and the agent should backtrack. It also teaches the student when to commit to a product if it satisfies all criteria rather than being non-committal and searching for options. Training on this correction data yields a policy $\pi_{2}$ that balances correctly backtracking vs committing to a product. Fig. 4 (b) shows that $\pi_{0}$ fails to commit to a product needlessly cycling through pages till time runs out. In contrast, $\pi_{2}$ correctly backtracks and then selects a second item that solves the task.

#### 4.4 Domain 3: Interactive Coding

Setup. Intercode Bash (Yang et al., ) is an interactive coding environment, where the agent receives a task that involves operations on a filesystem, e.g. “Find all text files and write their names to a single file”, and must interact with Bash to solve it and submit a final command.

**Table 2: Intercode Bash Evaluation. Overall success rate and actions on test data (max $10$ actions). Baselines from [1] (Yang et al., ) evaluated on ^⋆(train + test) data.**
| Method | %succ$\uparrow$ | #act$\downarrow$ |
| --- | --- | --- |
| gpt-4 [1] | $48.5^{\star}$ | - |
| gpt-3.5 [1] | $46.5^{\star}$ | - |
| CodeLlama-34B-INST [1] | $36.0^{\star}$ | - |
| ReAct gpt-4o | 71.3 | 3.8 |
| ReAct gpt-4o-mini | 43.4 | 6.0 |
| LEAP Llama-3.1-8b $\pi_{0}$ | 41.4 | 6.6 |
| LEAP Llama-3.1-8b $\pi_{1}$ | 49.9 | 5.5 |
| LEAP Llama-3.1-70b $\pi_{0}$ | 60.3 | 4.8 |
| LEAP Llama-3.1-70b $\pi_{1}$ | 71.8 | 5.1 |

Intercode uses NL2Bash (Lin et al., ) to create $4$ datasets. We use the first $2$ for training, and the next $2$ for testing. We limit time horizon to $10$. We compare against ReAct (Yao et al., ) with gpt-4o and gpt-4o-mini.
We also compare against the top 3 entries on the Intercode leaderboard, which are the TryAgain baselines from (Yang et al., ) with gpt-4, gpt-3.5 and CodeLlama-34B-INST (Roziere et al., ) respectively. We LoRA fine-tune both Llama-3.1-8B model and Llama-3.1-70B model, each for $2$ iterations, using gpt-4o as the privileged expert teacher. The privileged information is the ground truth Bash command.

Results. Table 2 shows that LEAP trained with $70$B policy outperforms all baselines, with policy $\pi_{1}$ achieving $71.8\%$ success rate to match the gpt-4o expert teacher $(71.8\%>71.3\%)$. We find that on this coding benchmark, the gpt-4o expert is a very strong baseline. LEAP trained with $8$B models improve over the BC policy, but falls short of the gpt-4o expert teacher. These results show that if the expert teacher is already a strong baseline, even after leveraging privileged information, it might be hard for the student to outperform the teacher.

#### 4.5 What is the trade-off between Privileged Information and Realizability?

Figure: Figure 5: Privileged Information vs Realizability. (a) Performance of $5$ policies trained with experts with varying levels of privileged information on ALFWorld, peaking for expert $\pi^{E}_{3}$. (b) Examples of corrections from experts $\pi^{E}_{1},\pi^{E}_{3},\pi^{E}_{5}$. $\pi^{E}_{1}$ generates realizable reason action but predicts wrong action. $\pi^{E}_{5}$ predicts correction action, but produces unrealizable reason action that contains privileged information. $\pi^{E}_{3}$ strikes a perfect balance.
Refer to caption: x5.png

We study the tradeoff between how much privileged state the expert uses vs the realizability of their corrections on ALFworld. We create $5$ different privileged experts $\pi_{E}^{1},\dots,\pi_{E}^{5}$ by varing the prompt to instruct them to use increasing amounts of privileged state, e.g. for $\pi_{E}^{2}$ it says “Use this information sparingly for your correction,” but for $\pi_{E}^{4}$ it says “feel free to include information from the privileged state”. We run 1 iteration of LEAP to generate updated $\pi^{1},\dots,\pi^{5}$. Fig. 5 (a) shows that success rate initially rises from $\pi^{1}$ to $\pi^{3}$, but then falls off sharply till $\pi^{5}$. Num actions has a similar trend. This validates the hypothesis that there is indeed an optimal tradeoff. Interestingly, $\pi_{5}$ is much worse than $\pi_{1}$, showing that it’s more important to be realizable than to provide optimal yet unrealizable corrections. Fig. 5 (b) shows the corrections from $\pi^{E}_{1},\pi^{E}_{3},\pi^{E}_{5}$. $\pi^{E}_{1}$ generates reasoning that is perfectly realizable to the student, but since it has no privileged state the actions are not optimal. In contrast, $\pi^{E}_{5}$ predicts the correct action, but blatantly reveals the privileged information in the reason, being unrealizable to the student. $\pi^{E}_{3}$ strikes a perfect balance where it offers the correct action, but offers a reason that is very much realizable, i.e. kettles are likely to be in stoveburners.

#### 4.6 Can LEAP be used to self-correct a student?

Figure: Figure 6: Self Correction. LEAP self-improves a Llama3-8B model that is used as both student and teacher. (a) Overall success rate and num actions. (b) Category-wise breakdown of success rate.
Refer to caption: x6.png

We test the hypothesis that LEAP should enable a model to self-improve by using its privileged version as the expert. On ALFWorld, we fine-tune Llama3-8B initializing with the BC policy $\pi_{0}$ and running $2$ iterations of self-improvement. Fig. 6 shows that LEAP is able to significantly self-improve policies over iterations $(65.7\%\rightarrow 82.1\%\rightarrow 91.8\%)$, with the final policy matching the best policy when using GPT-4o as a teacher. We see uniform improvements across all categories, with some being improved more from $\pi_{0}\rightarrow\pi_{1}$ and others from $\pi_{1}\rightarrow\pi_{2}$. We conclude that for some environments, like ALFworld, the privileged state does the heavy lifting, i.e. performance improvements happen at a similar rate to using a strong teacher vs using the model itself as a teacher. We note that this need not hold for all environments, e.g. for some a strong teacher is required to extract the optimal reason and action from the privileged state. Additionally, if the base policy $\pi_{0}$ is not strong, there maybe not lift off for the model and rate of improvement would be limited.

#### 4.7 How does SFT compare to Preference Optimization?

**Table 3: SFT vs Preference: LEAP with SFT vs DPO vs KTO on ALFWorld.**
|  | Method | %succ$\uparrow$ | #act$\downarrow$ |
| --- | --- | --- | --- |
| in-dist. | LEAP $\pi_{0}$ | 68.6 | 16.9 |
| LEAP DPO ($\beta:0.1$) $\pi_{1}$ | 74.3 | 15.7 |  |
| LEAP DPO ($\beta:0.01$) $\pi_{1}$ | 74.3 | 15.6 |  |
| LEAP KTO ($\lambda_{U}:0.0$) $\pi_{1}$ | 87.1 | 11.6 |  |
|  | LEAP SFT $\pi_{1}$ | 96.4 | 10.5 |
| out-dist. | LEAP $\pi_{0}$ | 65.7 | 18.6 |
| LEAP DPO ($\beta:0.1$) $\pi_{1}$ | 71.6 | 17.9 |  |
| LEAP DPO ($\beta:0.01$) $\pi_{1}$ | 70.9 | 17.7 |  |
| LEAP KTO ($\lambda_{U}:0.0$) $\pi_{1}$ | 88.1 | 11.7 |  |
|  | LEAP SFT $\pi_{1}$ | 91.0 | 11.9 |

We study the effect of different update methods for LEAP: SFT vs Preference Optimization. Our initial hypothesis is that KL regularized preference optimization should also result in performance improvement without needing to aggregate datasets. We run $1$ iteration of DPO with different regularization $\beta$ values. Table. 3 shows that while DPO improves upon the base $\pi_{0}$ policy, the improvement is far smaller than SFT for both $\beta=0.1,0.01$. One explanation is that SFT is far more aggressive in using the correction data compared to preference optimization, i.e., SFT deems the corrected reason action to be better than any alternative. To test this, we used a different preference optimizer KTO, that operates with unpaired preference data and set weight $\lambda_{D}=1.0,\lambda_{D}=0.0$, effectively emulating SFT-like behavior. KTO with these settings has a significantly high performance of $88.1\%$ close to SFT ($91.0\%$).

### 5 Related Work

Imitation Learning and Privileged Information.
A powerful paradigm in machine learning is leveraging privileged information (Vapnik et al., )—data available only during training and inaccessible at test time. This concept has been transformative in robotics, where a simulated expert policy with full state supervises a learner that only perceives sensor observations, such as navigation (Zhang et al., ; Uppal et al., ), self-driving (Chen et al., ), manipulation (Chen et al., ; Hu et al., ), and legged locomotion (Lee et al., ; Kumar et al., ).

A fundamental challenge in utilizing privileged information is the realizability gap—where actions suggested by an expert may be infeasible for the learner to predict. This gap often leads to spurious correlations, manifesting as the “latching effect” where learners repetitively predict the same action, commonly observed in autonomous driving (Muller et al., ; Kuefler et al., ; Bansal et al., ; Codevilla et al., ) and language models (Bengio et al., ; Holtzman et al., ; Wang & Sennrich, ; Ortega et al., ). While off-policy methods (Wen et al., ; ) have been proposed, (Swamy et al., ) show that online interaction with the environment is necessary.

Interactive imitation learning methods such as DAgger (Ross et al., ), where experts provide corrective actions during the learner’s rollouts, both theoretically and empirically lead to very effective policies in this regime (Choudhury et al., ). These methods operate under the assumption of asymptotic realizability (Swamy et al., ), where the learner’s policy can eventually match the expert’s actions as the episode progresses. However, when this assumption fails—often due to the partial observability of the task or model capacity—suboptimal performance ensues, particularly for LLM agents, as demonstrated by our experiments (Sec. 4.5). While recent approaches resort to exploration or RL (Walsman et al., ; Tennenholtz et al., ; Nguyen et al., ; Weihs et al., ), these are intractable to extend to LLM agents. Our work provides a practical recipe for LLM agents that optimally balances the trade-off between using privileged information and maintaining realizability, and further shows how such an expert can be used to enable policy self-improvement.

Learning from AI Feedback. A scalable method for aligning language model using AI feedback (Lee et al., ). Recent works have explored different types of feedback, from self-generated corrections to those provided by external AI models or environments (Pan et al., ).
One body of work focuses on self-correction, where LLMs refine their responses based on feedback from their own outputs (Bai et al., ; Madaan et al., ). This can occur either at train-time, where LLMs are fine-tuned to generate corrected responses (Bai et al., ; Ganguli et al., ), or at test-time, where the model improves its response interactively (Madaan et al., ). However, while simple in principle, (Huang et al., ) shows that without external sources of feedback, LLMs often struggle to accurately judge the correctness of their reasoning, limiting the effectiveness of purely self-generated corrections.

Another cluster of work explores feedback from external sources. Some approaches use feedback from strong models like GPT-4 acting as judges (Zheng et al., ) or critics trained on high-quality curated data (Wang et al., ; Paul et al., ). Others leverage environmental feedback or external tools to guide improvements. (Welleck et al., ) uses environmental feedback to train a corrector model, (Chen et al., ; Olausson et al., ) utilize execution feedback to improve code generation, and (Gou et al., ) employs external tools. Notably, Reflexion (Shinn et al., ) leverages feedback from external environments to generate self-reflections for agents, although it assumes multiple attempts in the same environment, which our approach does not. Our work introduces the use of privileged information in decision-making tasks for LLM agents, a technique not yet fully explored in the literature. This also enables LLM agents to self-improve using privileged information, setting it apart from existing self-improvement methods that rely on iterative solution generation and evaluation (Zelikman et al., ; Pang et al., ; Hosseini et al., ).

### 6 Limitations

We proposed LEAP, an iterative fine-tuning framework that improves LLM agent performance using privileged AI feedback, enabling weaker models to surpass stronger expert teachers and allowing agents to self-improve. However, LEAP has notable limitations. First, generating interaction rollouts is time-consuming, particularly in complex environments requiring multi-step reasoning. Each rollout requires simulating the environment and generating complete trajectories, that becomes computationally expensive across multiple fine-tuning iterations. Second, corrective feedback from the expert often results in lengthy output tokens. As trajectory lengths grow, balancing the effectiveness of feedback with its verbosity becomes challenging. Exploring methods to distill and summarize expert feedback at critical points is an interesting direction for future work.

### Reproducibility Statement

To ensure the reproducibility of our findings, we have made several efforts, as outlined below. Detailed information and resources are provided in the main paper, appendix, and supplementary materials:

- 1.
Open source code. We open source the LEAP code that includes the Python package implementing LEAP, along with all prompts, configuration files, and training scripts necessary to reproduce our results.
- 2.
Experimental details. We provide detailed descriptions of our experiments in the main paper (Section 4), with additional details on the setup, hyperparameters, and prompts found in Appendix C, D. We use open-source datasets and models in our experiments, with citations and links provided in the main paper.
- 3.
Analysis. The key results of our theoretical analysis are presented in the main paper (Section 3.3), with full derivations, proofs, and assumptions detailed in Appendix B.

## Appendix

### Appendix A Broader Impacts

###### Technological Impact.

By iteratively fine-tuning LLMs through privileged expert feedback, LEAP can significantly enhance AI performance across diverse applications, such as virtual assistants and dialogue agents handling complex decision-making tasks. This approach allows LLMs to autonomously refine their decision-making, reducing the need for human intervention, streamlining automation, and improving efficiency in both consumer and enterprise environments.

Additionally, LEAP ’s ability to enable weaker models to outperform stronger ones democratizes AI access, making state-of-the-art models more accessible to organizations with limited computational resources. This also opens new possibilities for improving model generalization in settings where perfect information is unavailable. However, the reliance on privileged feedback during training underscores the importance of further research into model transparency and interpretability. As these models grow more autonomous, ensuring their decision-making remains understandable to users will be critical for building trust and fostering widespread adoption.

###### Societal Impact.

Equipping LLMs with iterative self-improvement capabilities unlocks a range of societal benefits. In sectors such as education, customer service, and software development, these AI systems can assist professionals in decision-making, automate routine tasks, and enhance overall productivity. By continuously improving their performance, these models can reduce the cognitive load on workers, allowing them to focus on more complex, creative, and value-added activities.

Despite these advantages, there are ethical concerns that must be addressed. As LLMs become capable of self-improvement, ensuring their trustworthiness and accountability becomes critical. Without proper safeguards, the autonomous refinement of AI could lead to unpredictable behaviors, making it harder to ensure alignment with human values. Furthermore, there is a risk that self-improving AIs could be misused for malicious purposes, such as automating harmful tasks or spreading disinformation. To mitigate these risks, strict ethical guidelines and robust oversight is necessary to ensure deployment of such systems serves societal good and prevents misuse.

### Appendix B Analysis

#### B.1 Trade-off between realizability and privileged information

We now analyze the performance of a policy imitating an expert with access to privileged information. The performance depends both on the performance of the expert policy and the realizability gap between the learner and the expert. We then derive a constrained privileged expert to better trade off these two terms and provide a practical approximation of this constrained expert. For notational simplicity, we assume that $a_{t}$ represents both reason and action. We follow the derivation in (Swamy et al., ) but simplify for our setting.

Let $\pi\in\Pi$ be the student policy that we are training. We assume the class of policies $\Pi$ is rich enough to contain all history-based policies.
Let $J(\pi)$ be the performance, i.e. the cumulative reward of a learner policy that selects actions based on the history $a_{t}=\pi(h_{t})$. Let $J(\pi_{E})$ be the performance of the expert policy that selects actions based on the privileged state $a_{t}=\pi^{E}(s_{t})$. Let $\tau\sim\pi$ be a *privileged rollout* which is the sequence of observation, actions, and privileged state upon executing the policy. We represent the rollout as $\tau=\{(h_{t},s_{t},a_{t})\}$. Note that such a privileged rollout is only accessible at train time.

We begin by defining the average imitation gap between an expert policy $\pi^{E}$ and learner $\pi$.

###### Definition B.1 (Average Imitaiton Gap) .

The average imitation gap of a learner $\pi$ over a horizon $T$:

$$ $\mathrm{AIG}(\pi,T):=\frac{1}{T}\left(J(\pi^{E})-J(\pi)\right)=\mathbb{E}_{ \tau\sim\pi^{E}}\left[\frac{1}{T}r(s_{t},a_{t})\right]-\mathbb{E}_{\tau\sim\pi }\left[\frac{1}{T}r(s_{t},a_{t})\right]$ (6) $$

We next define the realizability gap between the privileged expert and the best policy in policy class as the L1 distance (or equivalently Total-Variation (TV) distance) between the two policies on the distribution of states and histories induced by *any policy*.

###### Definition B.2 (Average Realizability Gap) .

The average realizability gap over time between the privileged expert $\pi^{E}$ and the best policy in the policy class $\pi^{\star}$ is:

$$ $\epsilon(\pi^{E},T):=\sup_{\pi}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t }\sim d^{\pi}_{t}}||\pi^{E}(a_{t}|s_{t})-\pi^{\star}(a_{t}|h_{t}))||_{1}$ (7) $$

We next define the recoverability of the expert. Let $A^{\pi^{E}}(s,a)=Q^{\pi^{E}}(s,a)-V^{\pi^{E}}(s)$ be the advantage of a state action $(s,a)$ with respect to the expert policy $\pi^{E}$. A high negative advantage means that the mistaken action at a state results in a high value difference from the action the expert would have taken. Recoverability is the worst case advantage.

###### Definition B.3 (Recoverability Coefficient of the Privileged Expert) .

The recoverability of the expert policy $\pi^{E}$ is the max advantage of the expert policy

$$ $H(\pi^{E}):=\max_{s,a}||A^{\pi^{E}}(s,a)||$ (8) $$

We are now ready to state the main theorem of LEAP.

###### Theorem B.4 ( LEAP with Privileged Expert) .

Running $N$ iterations of LEAP with privileged expert $\pi^{E}$ yields at least one policy $\pi$ such that

$$ $\frac{1}{T}J(\pi)\geq\frac{1}{T}J(\pi^{E})-H(\pi^{E})\left(\epsilon(\pi^{E},T) +\gamma(N)\right)$ (9) $$

where $\pi^{E}$ is the privileged expert, $H(\pi^{E})$ is the recoverability coefficient of $\pi^{E}$, $\epsilon(\pi^{E},T)$ is the average realizability gap, and $\gamma(N)$ is the average regret of the DAgger update.

The first term is the performance of the privileged expert and the second term is the average realizability of the privileged expert. The more privileged information the expert policy $\pi^{E}$ leverages the better $J(\pi^{E})$ and the lower recoverability coefficient $H(\pi^{E})$. However, the realizability gap $\epsilon(\pi^{E},T)$ grows larger.

###### Proof.

DAgger (Ross et al., ) with $\pi^{E}$ as the expert teacher guarantees that after $N$ iterations, it will find at least one policy

$$ $\mathbb{E}_{s,h\sim d^{\pi}}||\pi(.|h)-\pi^{E}(.|s,h)||_{1}\leq\mathbb{E}_{s,h \sim d^{\pi}}||\pi^{\star}(.|h)-\pi^{E}(.|s,h)||_{1}+\gamma(N)$ (10) $$

where $\gamma(N)$ is the average regret, and $d^{\pi}$ is the time average distribution of states and history induced by policy $\pi$.

$$ $\displaystyle\mathrm{AIG}(\pi,T)$ $\displaystyle=\frac{1}{T}\left(J(\pi^{E})-J(\pi)\right)$ $\displaystyle\leq\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t}\sim d^{\pi}_ {t}}\left[\sum_{a}A^{\pi^{E}}(s_{t},a_{t})\left(\pi^{E}(a|s_{t})-\pi(a|h_{t}) \right)\right]$ (PDL) $\displaystyle\leq||A^{\pi^{E}}(.,.)||_{\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb {E}_{s_{t},h_{t}\sim d^{\pi}_{t}}||\pi(.|h_{t})-\pi^{E}(.|s_{t},h_{t})||_{1}$ (Holder’s Inequality) $\displaystyle\leq H(\pi^{E})\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t} \sim d^{\pi}_{t}}||\pi(.|h_{t})-\pi^{E}(.|s_{t},h_{t})||_{1}$ $\displaystyle\leq H(\pi^{E})\left(\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h _{t}\sim d^{\pi}_{t}}||\pi^{\star}(.|h_{t})-\pi^{E}(.|s_{t},h_{t})||_{1}+ \gamma(N)\right)$ From ( 10 ) $\displaystyle\leq H(\pi^{E})\left(\epsilon(\pi^{E},T)+\gamma(N)\right)$ $$

Rearranging, we get

$$ $\frac{1}{T}J(\pi)\geq\frac{1}{T}J(\pi^{E})-H(\pi^{E})\left(\epsilon(\pi^{E},T) +\gamma(N)\right)$ $$

∎

To optimize the performance bound, we introduce a constrained privileged expert that provides actions similar to the privileged expert but is constrained to be close to realizable.

###### Definition B.5 (Constrained Privileged Expert) .

Given a state $s_{t}$ and history $h_{t}$, a constrained privileged expert minimizes KL distance to the privileged expert while being in a $\delta$ KL ball of the non-privileged expert.

$$ $\pi^{E}_{\delta}(.|s_{t},h_{t}):=\arg\min_{\tilde{\pi}}\mathrm{KL}\left(\tilde {\pi}(.|s_{t},h_{t})||\pi^{E}(.|s_{t})\right)\quad\mathrm{s.t.}\quad\mathrm{KL }\left(\tilde{\pi}(.|s_{t},h_{t})||\pi^{E}(.|h_{t})\right)\leq\delta$ (11) $$

We use the fact that we have a rich policy class $\Pi$ that includes the non-privileged expert $\pi^{E}(.|h)$, to show that the constrained privileged expert bounds the average realizability gap

$$ $\displaystyle\epsilon(\pi^{E}_{\delta},T)$ $\displaystyle=\sup_{\pi}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t}\sim d ^{\pi}_{t}}||\pi^{E}_{\delta}(a_{t}|s_{t})-\pi^{\star}(a_{t}|h_{t}))||_{1}$ (12) $\displaystyle\leq\sup_{\pi}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t} \sim d^{\pi}_{t}}\sqrt{2\mathrm{KL}(\pi^{E}_{\delta}(a_{t}|s_{t})||\pi^{\star} (a_{t}|h_{t}))}$ $\displaystyle\leq\sup_{\pi}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t} \sim d^{\pi}_{t}}\sqrt{2\mathrm{KL}(\pi^{E}_{\delta}(a_{t}|s_{t})||\pi^{E}(a_{ t}|h_{t}))}$ since $\pi^{E}(.|h_{t})\in\Pi$ $\displaystyle\leq\sqrt{2\delta}$ from ( 11 ) $$

###### Theorem B.6 ( LEAP with Constrained Privileged Expert) .

Running $N$ iterations of LEAP with constrained privileged expert $\pi^{E}_{\delta}$ yields at least one policy $\pi$ such that

$$ $\frac{1}{T}J(\pi)\geq\frac{1}{T}J(\pi^{E})-\left(\frac{1}{T}\underbrace{\left( J(\pi^{E})-J(\pi^{E}_{\delta})\right)}_{\textup{gap from privileged expert}}+H (\pi^{E}_{\delta})\left(\underbrace{\sqrt{2\delta}}_{\textup{realizability}}+ \gamma(N)\right)\right)$ (13) $$

where $\pi^{E}$ is the privileged expert, $H(\pi^{E}_{\delta})$ is the recoverability coefficient of $\pi^{E}_{\delta}$, $\delta$ is the KL constraint, $\gamma(N)$ is the average regret of the DAgger update.

The first term is the performance gap between the privileged and the constrained privileged expert. The second term is the realizability of the constrained privileged expert. As $\delta\rightarrow 0$, the realizability gap goes down while the performance gap and recoverability coefficient go up. There is a critical value of $\delta$ that achieves an optimal tradeoff.

###### Proof.

DAgger (Ross et al., ) with $\pi^{E}_{\delta}$ as the expert teacher guarantees that after $N$ iterations, it will find at least one policy

$$ $\mathbb{E}_{s,h\sim d^{\pi}}||\pi(.|h)-\pi^{E}_{\delta}(.|s,h)||_{1}\leq \mathbb{E}_{s,h\sim d^{\pi}}||\pi^{\star}(.|h)-\pi^{E}_{\delta}(.|s,h)||_{1}+ \gamma(N)$ (14) $$

where $\gamma(N)$ is the average regret.

$$ $\displaystyle\mathrm{AIG}(\pi,T)=\frac{1}{T}\left(J(\pi^{E})-J(\pi)\right)$ $\displaystyle=\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+\frac{1}{ T}\left(J(\pi_{\delta}^{E})-J(\pi)\right)$ $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+\frac{ 1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t}\sim d^{\pi}_{t}}\left[\sum_{a}A^{ \pi^{E}_{\delta}}(s_{t},a_{t})\left(\pi^{E}_{\delta}(a|s_{t})-\pi(a|h_{t}) \right)\right]$ (PDL) $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+||A^{ \pi^{E}_{\delta}}(.,.)||_{\infty}\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_ {t}\sim d^{\pi}_{t}}||\pi(.|h_{t})-\pi^{E}_{\delta}(.|s_{t},h_{t})||_{1}$ (Holder’s Inequality) $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+H(\pi^ {E}_{\delta})\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t}\sim d^{\pi}_{t}} ||\pi(.|h_{t})-\pi^{E}_{\delta}(.|s_{t},h_{t})||_{1}$ $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+H(\pi^ {E}_{\delta})\left(\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{s_{t},h_{t}\sim d^{\pi }_{t}}||\pi^{\star}(.|h_{t})-\pi^{E}_{\delta}(.|s_{t},h_{t})||_{1}+\gamma(N)\right)$ From ( 14 ) $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+H(\pi^ {E}_{\delta})\left(\epsilon(\pi^{E}_{\delta},T)+\gamma(N)\right)$ $\displaystyle\leq\frac{1}{T}\left(J(\pi^{E})-J(\pi^{E}_{\delta})\right)+H(\pi^ {E}_{\delta})\left(\sqrt{2\delta}+\gamma(N)\right)$ From ( 12 ) $$

Rearranging, we get

$$ $\frac{1}{T}J(\pi)\geq\frac{1}{T}J(\pi^{E})-\left(\frac{1}{T}\left(J(\pi^{E})-J (\pi^{E}_{\delta})\right)+H(\pi^{E}_{\delta})\left(\sqrt{2\delta}+\gamma(N) \right)\right)$ $$

∎

###### Definition B.1 (Average Imitaiton Gap) .

###### Definition B.2 (Average Realizability Gap) .

###### Definition B.3 (Recoverability Coefficient of the Privileged Expert) .

###### Theorem B.4 ( LEAP with Privileged Expert) .

###### Proof.

###### Definition B.5 (Constrained Privileged Expert) .

###### Theorem B.6 ( LEAP with Constrained Privileged Expert) .

###### Proof.

#### B.2 Practical approximation for constrained privileged expert

Writing out the Lagrangian of the constrained privileged expert (Def. B.5) and solving for the primal we get:

$$ $\pi^{E}(\rho,a|s,h)=\frac{1}{Z(s,h)}\pi^{E}(\rho,a_{t}|s_{t})\left(\pi^{E}( \rho,a_{t}|h_{t})\right)^{\lambda}$ (15) $$

where $\lambda$ is the Lagrange multiplier.

For a fixed multiplier $\lambda$, this closed-form expression suggests a simple penalty-method approximation to constrained privileged expert:

- 1.
Sample $N$ responses from both $\pi^{E}(\tilde{\rho},\tilde{a}|{\rm\rho},{a},h,s)$ and $\pi^{E}(\tilde{\rho},\tilde{a}|{\rm\rho},{a},h)$ to create $\mathcal{R}=\{\tilde{\rho}^{i},\tilde{a}^{i}\}_{i=1}^{N}$
- 2.
Reweigh the responses in $\mathcal{R}$ according to
$\log\pi^{E}(\tilde{\rho},\tilde{a}|{\rho},{a},h,s)+\lambda\log\pi^{E}(\tilde{
\rho},\tilde{a}|{\rho},{a},h)$
(16)
- 3.
Sample from the reweighed distribution.

Since the majority of our experiments involve closed-weights models (like gpt-4o), we are unable to access the logits of the expert. Instead, we design LEAP prompts to approximate the constrained privileged expert by providing instructions to not reveal the privileged information in the reasoning, and provide general reason and actions that should be predictable given only history $h_{t}$.

### Appendix C Experimental details

#### C.1 Hyper-parameters

##### C.1.1 Training parameters

Tables 4 and 5 contain hyperparameters for SFT training and DPO/KTO training using LoRA for the different datasets. All training runs were on machines with either 2 or 4 RTX A6000 GPUs, each with 48 GB of memory per GPU.

**Table 4: Hyperparameters for SFT training using LoRA**
| Dataset | AlfWorld | WebShop | InterCode |
| --- | --- | --- | --- |
| Model | Llama3-8B | Llama3-8B | Llama3-8B / Llama3-70B |
| Batch size | 64 | 16 | 16 |
| Max seq length | 8000 | 6000 | 6000 |
| Max epochs | 1 | 1 | 5 |
| Learning rate | 3e-5 | 3e-5 | 3e-5 |
| LoRA $\alpha$ | 64 | 64 | 64 |
| LoRA $r$ | 128 | 128 | 128 |
| LoRA dropout | 0.05 | 0.05 | 0.05 |
| Optimizer | AdamW | AdamW | AdamW |
| LR scheduler | cosine | cosine | cosine |

**Table 5: Hyperparameters for preference training (DPO, KTO) using LoRA**
| Method | DPO | KTO |
| --- | --- | --- |
| Model | Llama3-8B | Llama3-8B |
| Batch size | 32 | 64 |
| Max seq length | 8000 | 8000 |
| Max prompt length | 6000 | 6000 |
| Max epochs | 1 | 1 |
| Regularization $\beta$ | 0.01 | 0.01 |
| Learning rate | 5e-7 | 5e-7 |
| LoRA $\alpha$ | 64 | 64 |
| LoRA $r$ | 128 | 128 |
| LoRA dropout | 0.05 | 0.05 |
| Optimizer | AdamW | AdamW |
| LR scheduler | cosine | cosine |

##### C.1.2 Evaluation parameters

For inference using Llama3-8b and Llama3-70b, we use a temperature setting of 0.3 and a maximum token length of 256. The following code snippet shows the .generate() function used:

Figure: Listing 1: Generation function with hyperparameters for Llama3 models

For all OpenAI model calls, including tasks such as generating expert teacher corrections, labeling reasoning for actions, or running prompt baselines like ReAct, we use the API call with the following hyperparameters. See Appendix D for the detailed prompts.

Figure: Listing 2: Hyperparameters for OpenAI GPT-4o and GPT-4o-mini models

#### C.2 ALFWorld

ALFWorld (Shridhar et al., ) is a text-based game built on the ALFRED embodied benchmark (Shridhar et al., ). Each task has a high-level objective, e.g. “heat mug and put it in cabinet” which the agent must complete by navigating and interacting with a virtual household through text commands, e.g. go to shelf 1, pick up mug 2, etc. It is a POMDP where the hidden state is the location of all objects, and the observations reveal only the objects the agent can see. To solve it, the agent must plan subgoals, track progress, and explore efficiently till time runs out. An interesting aspect of ALFWorld is the requirement to infer the likely locations of household objects (e.g. mugs are more likely to be on shelves or cabinets), which the agent can learn from interaction with the environment. There are $6$ categories of tasks and two datasets: $139$ in-distribution games and 134 out-of-distribution games. Time horizon is limited to $30$.

We compare against ReAct (Yao et al., ) with different base models and prompts. The original ReAct paper (Yao et al., ) used text-davinci-002 with task-dependent in-context examples. We run ReAct wtug gpt-4o and gpt-4o-mini and an instruction prompt that is shared by all our methods. We also compare against BUTLER (Shridhar et al., ), a seq2seq BERT model trained using DAGGER.(^4^44Since we look at the single trial setting, it rules out approaches like Reflexion (Shinn et al., ) where agents require multiple trials on the same game.)

We LoRA fine-tune Llama-3-8B model using gpt-4o as the expert teacher for $4$ iterations. We use observation-action demonstrations on the training dataset of $3257$ gamesShridhar et al. (), which we annotate with gpt-4o to generate reasons and use this to train the BC policy $\pi_{0}$. We then roll out a policy and generate corrections from the privileged expert on unsuccessful rollouts. The correction data is then aggregated with the existing dataset and used to train a policy using SFT.

The privileged state in ALFworld consists of

- 1.
Task: What is the overall task
- 2.
Essential Objects: Enumerate all objects that are necessary to solve the game.
- 3.
Critical Locations: List the important locations involved in the game solution.
- 4.
Optimal Action Sequence: Suggest an optimal sequence of actions that could theoretically minimize the steps needed to complete the game, based on the log provided. Be sure to copy the action verbatim.

#### C.3 WebShop

WebShop (Yao et al., ) is an online shopping environment consisting of $1.18$ million real-world products and $12,586$ human-provided instructions. Each task is a shopping request from a user (“I need a gluten-free bacon, 4 oz pack of 2, priced under $50”) that the agent has to solve through a set of interactions (search[“gluten-free bacon”] or selecting options such as "4 oz pack of 2"). The metrics are the overall score, and 4 components of the score: whether the attributes match (Att), whether the options match (Opt), whether the product types match (Type) and whether the price matches (Price). We evaluate on $500$ test instructions, and limit time horizon to $30$.

We compare against ReAct (Yao et al., ) with different base models like GPT-4o and GPT-4o-mini. We use the same instruction prompt as in LEAP. We also compare against IL baseline in WebShop (Yao et al., ) trained using BC. These baselines train two different selection models, one for selecting search queries from a list of options and one for clicking from a list of options. This constraint search space simplifies the problem.

We LoRA fine-tune Llama-3-8B model using GPT4-o as the expert teacher for 4 iterations. We initialize $\pi_{0}$ using observation-action demonstrations from Yao et al. () which we annotate with GPT4-o to generate reasons. We roll-out policies on the training dataset of $12086$ tasks, and generate corrections from the privileged expert on unsuccessful rollouts. For every game in the training dataset, we construct privileged information that contains: the attributes the product must satisfy, the price constraint, the option that has to be selected, and an example product that would satisfy all criteria. The correction data is then aggregated with the existing dataset and used to train a policy using SFT.

The privileged state in ALFworld consists of

- 1.
Task: What is the user instruction
- 2.
Attributes: What are the attributes the product must have
- 3.
Price: What price constraints must the product satisfy.
- 4.
Example Product: Name of a product that satisfies all criteria.

#### C.4 Intercode

Intercode Yang et al. () is an interactive coding environment, where an agent interacts with a compiler/interpreter to execute its code and submit further refinements. We use the Bash environment, where a typical task is “Find all text files and write their names to a single file.”, and the agent issues commands like ls, find .. to solve the task. Intercode builds on the NL2Bash (Lin et al., ) to create $4$ interactive datasets. We use the first $2$ for training, and the next $2$ for testing. We limit the time horizon to $10$. We compare against ReAct Yao et al. () with gpt-4o and gpt-4o-mini.
We also compare against the top 3 entries on the Intercode leaderboard, which are the TryAgain baselines from Yang et al. () with gpt-4, gpt-3.5 and CodeLlama-34B-INST respectively. Note the numbers are directly from the leaderboard which evaluates on all (train+test) data and, while our numbers are only on test.

### Appendix D Prompts

We define the following three prompts across all datasets:

- 1.
Expert corrections template: This prompt is used by the LLM expert teacher to provide corrections on the student agent’s trajectories. These corrections offer improved reasoning and actions that balance the trade-off between utilizing the privileged state (accessible only to the expert during training) and ensuring the corrections are realizable by the student agent.
- 2.
Student agent template: This prompt is used during test time by the Llama3 student agent models to generate reason and action trajectories that achieve the task objective. It is also used for prompting baselines, such as GPT-4 or ReAct, which generate reasoning and action trajectories to achieve the task objectives.
- 3.
Autolabel reasons template: For datasets that contain human demonstrations with action labels (like AlfWorld, WebShop), this prompt is used to generate reasoning annotations for actions that the human took. Generating reasons is necessary as the student agent needs to be trained to produce both reason and actions in its output trajectories.

For prompts used in different ablation studies, please refer to the accompanying code.

#### D.1 AlfWorld

##### D.1.1 Expert corrections template

##### D.1.2 Student agent template

##### D.1.3 Autolabel reasons template

#### D.2 WebShop

##### D.2.1 Expert corrections template

##### D.2.2 Student agent template

##### D.2.3 Autolabel reasons template

#### D.3 InterCode

##### D.3.1 Expert corrections template

##### D.3.2 Student agent template

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: **Q1:** In the proof of theorem B4, there is a step using the performance difference lemma, but I did not see the connection here. Could you provide the derivation of how you get the inequality?

**Q2:** In the contribution section, Webshop is reported as achieving 61.8%. However, the metric used in the experiment is the score, which is usually much higher than the success rate. Could you clarify what metrics you are actually evaluating?

**Q3:** Could you provide the self-correct results (using Llama3-8B as expert teacher) for WebShop and Intercode?

The overall quality of the paper is good, but there are still some weaknesses and questions that need to be addressed. I will reassess the score based on the authors' clarifications.
soundness: 3
strengths: * The proposed method looks valid and outperforms the baseline (ReAct).
* Evaluate three diverse environments: ALFWorld, WebShop, and Intercode.
* Provide detailed experiment and theoretical analysis of the trade-off between realizability and privileged information.
* The proof appears to be correct, and the prompts are included in the Appendix.
summary: The paper proposes LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. The key insight is to equip the expert teachers with a privileged state - additional information available during training but hidden at test time. This allows weak experts to provide guidance, improving the student agent's performance. LEAP is evaluated on diverse decision-making benchmarks, including text-based games, web navigation, and interactive coding, and shows that it outperforms state-of-the-art baselines, enables weak student models to exceed the performance of strong teacher models, and allows weak models to self-improve using privileged versions of themselves.
weaknesses: * The analysis of the costs associated with iterative fine-tuning and calling LLM experts was not included. While the authors assert that their method is not comparable to Reflexion [1], Reflexion appears to be a more cost-effective approach for enhancing the performance of LLM agents with minimal trials.
* It seems somewhat inequitable to compare this method with those that do not utilize privileged information, as such information is typically not available in real-world scenarios.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - How to provide those privilege state in more complicated tasks?
- Trade-off between using privileged information and student realizability is not clear! 
- Compute privileged state s_t for every datapoint in Di in the algorithm is not clear. That is the limitation part of the algorithm. 
- how?  For each time step in Di, LEAP computes the privileged state s_t by leveraging information available only during training
- What is the performance of LEAP vs ReAct but using the same privilege state?
soundness: 3
strengths: - well written.
summary: This is paper is about automatic self-improvement from errors during task execution. They proposed LEAT in which LLM agents as students use feedback from expert teachers to improve itself. The teachers hav access to a privileged state —information available during training. So then even weak teachers are effective. They evaluate that on including text-based games, web navigation, and interactive coding. The results show that it outperforms state-of-the art baselines, enables weak student models (e.g., Llama3-8B) to exceed the performance of strong teacher models (GPT4-o), and (3) allows weak models to self-improve using privileged versions of themselves.
weaknesses: - Lack of good reference! Relevant references are missed in introduction! It needs to provide proper reference for each claim!
- The issue of experts is how to prepare knowledge for the experts, how to distinguish between different experts. This is not clear in this work and it needs manual set up which is not applicable.
- They are very similar works in the literature but with different names instead of experts! In that way the innovation of the work is very limited. 
- No Fair comparison at all!! The work did not consider fair baselines! There are a lot of up to date methods that works better than ReAct and BUTLER! There are a lot of works from self-correctness that they do the same things. 
- In general the contribution of the work is very limited! The teachers should know privilege knowledge however it is not always available! Even with that still prompting and few shot are needed which makes it the same as other prompting works or self correction works but here the authors just used different terms.  And it is very trivial that there should be some improvement vs e.g. react since it is literary using extra information of the environment and more over it uses a suggestion from a larger LLM with that extra information.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: My main question is already mentioned above under "weaknesses". If we are allowed to show privileged information to the teacher, which then uses this to correct the student, why not directly show the privileged information to the student, especially given the student-teacher relationship is not adversarial? Which real-world scenario is this setup replicating?

UPDATE 25/11/2024: This question and some of the weaknesses have been addressed in the rebuttal, so I have updated my review score from a 5 to a 6.
soundness: 2
strengths: - The paper provides a solid theoretical grounding in both the main text and appendix, especially around the analysis on the trade-off between privileged information and the agent's realizability.
- I appreciated the variety and number of domains considered experimentally, which showed the wide applicability of the method.
- The ablations on the fine-tuning methods, across SFT, DPO and KTO were greatly appreciated, as it was otherwise unclear which of these to pick from
- Beyond the general experimental results, I found the questions considered relatively original: whether an agent can bootstrap to self-improvement, whether it can outperform its teacher, and what tradeoffs are faced in terms of amount of privileged information and correction realizability. These were all quite interesting results to read.
- The paper explicitly acknowledges some its limitations, which is commendable and contributes to the paper's transparency and reliability in my view.
summary: The paper seeks to address LLM Agents' difficulties when it comes from recovering from errors during task execution. 

They begin by framing decision-making as a Partially Observable Markov Decision Process (POMDP). 
Given this framing, they approach the issue by proposing LEAP, a DAGGER-like iterative training framework in which an expert teacher is equipped with "privileged state" corrects the faulty student agent trajectories during training. 

Here, the "privileged state" is simply a representation of the environment state that contains more information than what is available to the student. 

After outlining the problem and LEAP, the authors briefly examine their setup, analytically showing that there is a tradeoff between the performance and realisability of the privileged expert teacher.

The paper then moves on to the experiments, where the authors test their framework against a number of baselines across a variety of domains (text, web agents, and coding), demonstrating improvements in performance. Their experiments also show the possibility of student agents surpassing their teachers and of self-improvement when using a privileged version of themselves as the teacher. The authors also experimentally explore the tradeoff between how much privileged information to use and how realisable the corrections are. Finally, an ablation on the fine-tuning mechanism in LEAP demonstrates that SFT outperforms DPO and KTO in this setting.
weaknesses: - I found the author's choice of framing decision making as a POMDP somewhat arbitrary and under-justified. There is plenty of valid work that studies decision making using fully observable MDP's, and the connection to the real world is a bit forced
- Aside from the above, the main weakness in my view is that the authors never address/consider/justify the absence of the very obvious alternative: if you are able to train using privileged information anyway, why not simply show this information directly to the student rather than through the teacher? If the privileged information is accessible (by the teacher), this is not really a POMDP anymore really. What analogous real world scenario is this setup trying to address? It appears somewhat ill-defined from what i can tell.
- Somewhat related to the above point, but I found some of the results somewhat obvious and not particularly novel: of course an agent which (indirectly) has additional privileged information will do better than one without such information. It is similarly not surprising that the student eventually outperforms its teacher, given that only the student is trained and is probably comparatively overfitting to the objectively. 
- I found section 3.3. rather unclear and perhaps underdeveloped in the main text. For instance, "realizability" on its own is never clearly defined. I would have devoted more of the paper to this analysis, perhaps sacrificing one of the domains from 4.1-4.4 to the appendix if space was necessary.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have the author tried to perform preference optimization after SFT? Will this lead to a even higher score?

2. The authors admit that generating interaction rollouts is time-consuming. How long does it typically take in terms of magnitude—hours, days, or even longer?
soundness: 3
strengths: 1. Novel methods: There have been a huge amount of research papers discussing about LLM self-improvement. However, the idea of equipping the expert teachers with a privileged state is relatively novel.

2. Good experimental results: LEAP significantly outperforms all baselines, even the stronger teachers like GPT-4o.

3. Diverse experiments: The authors conduct various experiments across several fundamentally different benchmarks.
summary: The authors propose LEAP, an iterative fine-tuning framework that continually improves LLM agents using feedback from AI expert teachers. Notably, they equip the expert teachers with a privileged state. They evaluate LEAP on several decision-making benchmarks, and their experiments show that LEAP enables weak student models to exceed the performance of strong teacher models.
weaknesses: 1. Privileged states can be hard to get in real world environments (though this may not fit in the scope of this work).

2. Training an agent for each benchmark seperately is specific. The results would be much more promising if the authors demonstrate the LLM can be finetuned on all benchmarks together and still achieve good results. 

3. Some minor typos: For example, in line 24, GPT4-o should be written as GPT-4o. In line 127, $\rho_t^*,a_t^*$ should be $(\rho_t^*,a_t^*)$.

Meta-review: This paper proposes an interesting framework to improve LLM agents by utilizing a LLM teacher with access to privileged information (the ground truth states that may be hidden in practice) to correct the reasoning and action trace of the student model, and iteratively fine-tune on such corrected traces. This iterative process can obtain student models outperforming the teacher model. While acquiring privileged information can be hard for certain applications like search agent in practice, the idea and observations in this paper looks promising and can be applied once the privileged information can be obtained.

Decision: Accept (Poster)