Fine-tuning with RAG for Improving LLM Learning of New Skills

## Abstract

Abstract Large language model (LLM) agents deployed for multi-step tasks frequently fail in predictable ways: attempting actions with unmet preconditions, issuing redundant commands, or mishandling environment constraints. While retrieval-augmented generation (RAG) can improve performance by providing runtime guidance, it requires maintaining external knowledge databases and adds computational overhead at every deployment. We propose a simple pipeline that converts inference-time retrieval into learned competence through distillation. Our approach: (1) extracts compact, reusable hints from agent failures, (2) uses these hints to generate improved teacher trajectories via one-shot retrieval at episode start, and (3) trains student models on these trajectories with hint strings removed, forcing internalization rather than memorization. Across two interactive benchmarks, ALFWorld (household tasks) and WebShop (online shopping), distilled students consistently outperform baseline agents, achieving up to 91% success on ALFWorld (vs. 79% for baselines) and improving WebShop scores to 72 (vs. 61 for baselines), while using 10-60% fewer tokens than retrieval-augmented teachers depending on the environment. The approach generalizes across model scales (7B/14B parameters) and agent architectures (ReAct/StateAct), demonstrating that retrieval benefits can be effectively internalized through targeted fine-tuning without permanent runtime dependencies.

## 1 Introduction

Large language models are increasingly deployed as agents that interact with environments to complete multi-step tasks. Success requires not just generating plausible text but maintaining goals across extended interactions, managing state and preconditions, and recovering from errors. Despite advances in prompting strategies, LLM agents still exhibit systematic failures in interactive environments.

Prior work has explored multiple approaches to improve agent performance. Structured prompting methods like ReAct (Yao et al., 2023b) and StateAct (Rozanov & Rei, 2025) provide scaffolding for reasoning and state tracking. Self-reflection approaches such as Reflexion (Shinn et al., 2023) enable learning from mistakes across multiple attempts. Retrieval-augmented methods (Lewis et al., 2021; Zhao et al., 2024; Fu et al., 2024) inject external knowledge to guide decisions. Fine-tuning approaches (Chen et al., 2023; Zhang et al., 2025) directly modify model parameters using expert demonstrations. Each approach involves trade-offs: retrieval adds token cost and deployment complexity, fine-tuning risks overfitting and requires substantial data, while self-reflection assumes multiple attempts are feasible.

We propose a pipeline that combines the benefits of retrieval and fine-tuning while avoiding their individual limitations. Our key insight is that retrieval-augmented generation need not remain a permanent runtime dependency. Instead, it can serve as a source of improved training supervision that gets internalized into model parameters. Specifically, we: (1) run base agents to collect failures, (2) extract generalizable hints from these failures, (3) use hints to generate better teacher trajectories, and (4) distill these trajectories into students that no longer need hints.

Our contributions are:

- •
Failure-driven hint extraction: A method to automatically generate typed, reusable guidance from agent mistakes without expert supervision, using GPT-4o to diagnose failures and propose corrective rules
- •
One-shot retrieval distillation: A training approach where hints are retrieved once at episode start during teacher data generation, then removed during student training to force internalization
- •
Efficiency analysis: Demonstration that distilled models dominate the accuracy-efficiency frontier, achieving highest task success with lowest token usage

## 2 Related Work

### 2.1 Prompting-Based Agents

Chain-of-thought prompting (Wei et al., 2022) revealed that explicit reasoning steps improve complex task performance. ReAct (Yao et al., 2023b) extended this to interactive settings by interleaving ”Thought” and ”Action” tokens, achieving strong results on ALFWorld and WebShop without training. StateAct (Rozanov & Rei, 2025) augmented ReAct with explicit state representations (goal, inventory, location) that ground reasoning over long horizons, improving success rates by 10-30% across environments. THREAD (Schroeder et al., 2025) introduced recursive reasoning through dynamically spawned sub-threads, allowing agents to allocate computation to complex subtasks. These methods demonstrate impressive zero-shot capabilities but remain limited by the parametric knowledge of the model.

### 2.2 Learning from Experience

Several approaches enable agents to learn without gradient updates. Reflexion (Shinn et al., 2023) generates self-critiques after failures and accumulates lessons across multiple attempts on the same tasks, reaching 91% on HumanEval through iterative refinement. However, this requires ground-truth feedback and multiple trials per task. ExpeL (Zhao et al., 2024) extracts insights from training trajectories and retrieves them alongside demonstrations at test time, improving systematically as experience grows. AutoGuide (Fu et al., 2024) learns state-conditioned guidelines that are retrieved dynamically based on current context. AutoManual (Chen et al., 2024) builds comprehensive domain manuals through exploration. While effective, these methods create permanent dependencies on external knowledge stores and retrieval mechanisms.

### 2.3 Fine-Tuning and Distillation

FireAct (Chen et al., 2023) demonstrated that fine-tuning on just 500 GPT-4 trajectories can improve a 7B model’s success rate by 77% on tool-use tasks. Parameter-efficient fine-tuning has been widely studied (Hu et al., 2021; Dettmers et al., 2023; Han et al., 2024), enabling adaptation of large backbones with small memory and compute cost. Prompt distillation (Kujanpää et al., 2025) showed that complex prompts can be compressed into model weights by training on outputs generated with the prompt but without providing it as input. Chain-of-thought distillation (Hsieh et al., 2023) and self-training approaches (Zelikman et al., 2022) transfer reasoning traces into smaller models. Classical model distillation approaches such as DistilBERT (Sanh et al., 2020) compress large transformers into lighter students, but operate in a static supervised setting rather than interactive environments. Our work differs from these approaches: unlike FireAct which requires expensive GPT-4 supervision, we generate teacher data from the model’s own failures augmented with self-extracted hints. We distill dynamically retrieved, failure-specific guidance rather than standard prompt distillation which compresses static prompts.

### 2.4 Positioning Our Approach

Unlike multi-attempt approaches, we learn from a single training run, and we use retrieval only during training, not as a permanent fixture attached to the model. We also evaluate using a single run on the test set. We automatically generate supervision from the failures of the agent rather than requiring expert demonstrations. This positions our method as a practical path to improving deployed agents without runtime complexity.

## 3 Methodology

### 3.1 Problem Setting

We model tasks as partially observable MDPs $(\mathcal{S},\mathcal{A},T,R,\Omega,O)$ with latent state $s_{t}\!\in\!\mathcal{S}$, observation $o_{t}\!\in\!\Omega$ drawn from $O(\cdot\mid s_{t})$, action $a_{t}\!\in\!\mathcal{A}$, transition $T(s_{t+1}\mid s_{t},a_{t})$, and reward $r_{t}\!=\!R(s_{t},a_{t})$.

We define different agent policies:

- •
Base policy: $\pi_{\theta}(a_{t}\mid o_{\leq t})$ conditions only on the observation history.
- •
RAG policy: $\pi_{\theta}^{\text{RAG}}(a_{t}\mid o_{\leq t},H_{0})$ where $H_{0}=\{h_{1},\ldots,h_{k}\}$ are hints retrieved once at $t=0$.
- •
SFT policy: $\pi_{\phi}^{\text{SFT}}(a_{t}\mid o_{\leq t})$ with parameters $\phi$ fine-tuned on successful trajectories from $\pi_{\theta}$ (without retrieval).
- •
Distilled policy: $\pi_{\phi}(a_{t}\mid o_{\leq t})$ with parameters $\phi$ trained on successful trajectories from $\pi_{\theta}^{\text{RAG}}$, with hint strings removed to enforce internalization.

For ReAct agents, we augment the action space to include reasoning: $a_{t}=(c_{t}^{\text{thought}},c_{t}^{\text{action}})$ where $c_{t}^{\text{thought}}$ is internal reasoning and $c_{t}^{\text{action}}\in\mathcal{A}$ is the environment action. StateAct further augments with explicit state: $a_{t}=(c_{t}^{\text{state}},c_{t}^{\text{thought}},c_{t}^{\text{action}})$. We evaluate on two complementary benchmarks:

ALFWorld (Shridhar et al., 2021): A text-based household environment with 6 task types (Pick & Place, Examine in Light, Clean & Place, Heat & Place, Cool & Place, Pick Two & Place) requiring multi-step object manipulation. Agents navigate rooms, operate containers, and transform objects using commands like goto, take, open, clean.

WebShop (Yao et al., 2023a): An online shopping environment with 1.18M real products where agents must find and purchase items matching natural language specifications. Agents navigate search results, compare products, and select options using actions like search[query], click[element]. For WebShop specifically, we omit the $c_{t}^{\text{thought}}$ component in ReAct & StateAct, consistent with prior findings that this leads to stronger prompt-only performance in that environment (Rozanov & Rei, 2025).

### 3.2 Pipeline Overview

Our training pipeline (Figure [1](#S3.F1)) consists of four stages:

Stage A - Base Agent Rollouts: We deploy base agents (ReAct or StateAct) across the training split, collecting both successful and failed trajectories. Successes form the baseline supervised fine-tuning (SFT) dataset; failures serve as input for hint extraction.

Stage B - Self-Hint Extraction: For each failed trajectory $\tau$, we construct a complete failure example containing the task instruction, initial observation, full action sequence, and resulting observations. We then prompt GPT-4o to generate 1-4 imperative hints that diagnose the failure and provide generalizable guidance. Hints use placeholders ({object}, {container}) and are typed by task category for retrieval. Some examples of extracted hints:

- •
Ensure the {container} is open before attempting to place the {object} inside.
- •
Verify inventory capacity before attempting to take additional items.
- •
Use a systematic search pattern to avoid missing {object} in {location}

Figure: Figure 1: Training Pipeline. Stage A represents the initial base agent run. Stage B is the hint extraction using failures of the agent. Stage C is running the RAG agent. Stage D is training the models based on regular supervised fine-tuning (SFT) and our distillation method.
Refer to caption: /html/2510.01375/assets/figures/Pipeline2.png

Stage C - Teacher Data Generation:
Given instruction $g$ and initial observation $o_{0}$, we determine the task category $c(g)$ (explicitly provided in ALFWorld or inferred from the instruction in WebShop) and retrieve:
$H_{0}\;=\;\operatorname*{arg\,top\text{-}k}_{h\in\mathcal{H}_{c(g)}}s_{\psi}(h\,;\,g,o_{0}),$

where $s_{\psi}$ is an LLM re-ranking score computed by a quantized Qwen-2.5 7B (Qwen et al., 2025) model that evaluates candidate hints given the task instruction $g$ and initial observation $o_{0}$. We inject $H_{0}$ once at $t{=}0$ and then run the teacher $\pi_{\theta}^{\text{RAG}}(a_{t}\mid o_{\leq t},H_{0})$ for the remainder of the episode.

The addition of this one-time hint block equips the agent with corrective priors distilled from earlier failures (from the *training set*). By running the teacher policy in this retrieval-augmented mode, we collect a new set of successful trajectories that represent improved behaviors relative to the baseline. Importantly, only *successful* episodes are retained at this stage, since the purpose of the teacher is to provide high-quality demonstrations for distillation.

Stage D - Dataset Construction and Training: Datasets are constructed for supervised fine-tuning (SFT) from the collected trajectories. Two complementary sets are defined:

- 1.
Base Dataset (SFT): $\mathcal{D}_{\text{SFT}}$ composed of successful agent training trajectories produced by the base agent from Stage A, representing the baseline policy without retrieval.
- 2.
RAG Dataset (Distillation): $\mathcal{D}_{\text{Distillation}}$ composed of successful agent+RAG trajectories produced by the teacher from Stage C, with the hint strings removed from the serialized text. This forces the student model to internalize the hints implicitly, rather than memorizing them.

We then train low-rank adapters (LoRA) (Hu et al., 2021) on teacher trajectories with hint strings removed from inputs. This forces students to learn behaviors that substitute for explicit hints rather than memorizing the text.

Together, these four stages form a closed feedback loop: failures of the base agent are mined for corrective hints, which in turn guide the teacher policy, yielding stronger demonstrations for student training. This pipeline operationalizes the idea of self-improvement without reliance on external experts or handcrafted rules.

### 3.3 Hint Extraction Details

To reduce reliance on hand-engineered rules, we induce compact, reusable hints directly from the failures of our own base agents. Our extraction process maps failed rollouts to compact hints, which are partitioned by task type or product category. The goal of this process is to distill actionable advice that can be reused across similar tasks, thereby improving robustness without requiring privileged supervision. We design prompts for GPT-4o that enforce constraints on hint length, form, and structure: hints must be imperative, use placeholders for generality, and emphasize preconditions and sequencing rather than surface descriptions. Outputs are required to be strict JSON for programmatic parsing.

Extracted hints are then normalized, deduplicated using fuzzy matching, and grouped by task type (ALFWorld) or product category (WebShop), enabling efficient retrieval within relevant partitions. Deduplication uses fuzzy string matching with a Levenshtein distance threshold of 0.85, chosen empirically to catch near-duplicates (e.g., word variations) while preserving semantically distinct hints. The prompt is shown in Appendix [A.1](#A1.SS1).

### 3.4 Retrieval Mechanism

At $t{=}0$, we retrieve a fixed set of $k$ hints $H_{0}$ from the category-specific bank, conditioned on the task instruction $g$ and initial observation $o_{0}$. No additional retrieval occurs during the episode, the agent must rely on this static set of hints for the remainder of the trajectory. This design both limits token overhead and reflects a realistic use case where guidance is provided only once at the outset rather than continuously. The retriever: (i) determines the task category from the instruction (explicitly provided in ALFWorld, inferred in WebShop), (ii) scores candidate hints within the relevant partition using an LLM re-ranking step (Sun et al., 2023; Qin et al., 2024) (with a quantized Qwen-2.5 7B model), and (iii) selects the top-$k$ hints to form the hint block. The block is injected immediately after the task description and before the few-shot examples. The full agent prompt is provided in Appendix [A.3](#A1.SS3).

Figure: Figure 2: Hint distillation process. We remove the hint block and few-shot examples, since these are constant per task and provide no useful training signal. The end result is a trained adapter. During inference, we combine the adapter with the base agent to get our trained student.
Refer to caption: /html/2510.01375/assets/figures/Hint_Distillation.png

### 3.5 Training Configuration

We construct two datasets from successful agent rollouts:
(i) $\mathcal{D}_{\text{SFT}}$ from the baseline agent and
(ii) $\mathcal{D}_{\text{Distillation}}$ from the retrieval-augmented teacher with all hint strings and few-shot demonstrations removed from the trajectory $\tau$.
Since few-shot demonstrations are fixed across tasks and do not reflect the agent’s actual decision process, we exclude them to prevent the student from memorizing constant scaffolds rather than learning the underlying behaviors.

We train our student on $\mathcal{D}_{\text{Distillation}}$, and as an ablation, we also train it on $\mathcal{D}_{\text{SFT}}$ to compare between baseline supervised fine-tuning with our distillation method. We fine-tune low-rank adapters (LoRA) inserted into attention and MLP projections of the backbone while freezing all base weights. Let $x_{1:N}$ denote the tokens of a serialized teacher trajectory $\tau$. The training objective is full-sequence next-token cross-entropy:

$$ $\min_{\phi}\;\mathbb{E}_{x\sim\mathcal{D}}\left[-\frac{1}{N}\sum_{i=1}^{N}\log\pi_{\phi}(x_{i}\mid x_{<i})\right],$ (1) $$

where $\mathcal{D}\in\{\mathcal{D}_{\text{Distillation}},\mathcal{D}_{\text{SFT}}\}$ and $\pi_{\phi}$ is the student with parameters $\phi$. This is equivalent to maximum likelihood training on teacher demonstrations and can be interpreted as distillation from a retrieval-augmented teacher.

As we perform fine-tuning with LoRA on top of frozen backbones, we only optimize adapter parameters attached to attention and MLP projections while keeping base weights fixed. This preserves general language competence and yields stable updates with a small memory footprint.

Each episode is serialized as a single sequence (system + instruction + initial observation followed by step turns). For Distillation, we remove the hint block from the input to prevent copying. This allows us to internalize the hints into the model weights, ultimately letting the agent reproduce teacher behavior without hints at inference (as seen in Figure [2](#S3.F2)). Training minimizes the token-level cross-entropy in Eq. [1](#S3.E1). We apply token-level label smoothing for WebShop to mitigate overconfidence on its shorter traces. ALFWorld has a step limit of 50, so its traces are much longer than WebShop’s 15 step limit.

We obtain a student policy $\pi_{\phi}$, where $\phi$ denotes the backbone parameters $\theta$ (that remain frozen throughout fine-tuning) together with the low-rank adapter $\Delta(\phi)$ applied on top. In other words, $\phi$ is $\theta$ with LoRA adapters integrated.
$\pi_{\phi}(a_{t}|o_{\leq t})\approx\pi_{\theta}^{\text{RAG}}(a_{t}|o_{\leq t},H_{0})$

## 4 Experimental Setup

### 4.1 Datasets and Evaluation

We use the standard train/test partitions and fixed step/context budgets summarized in Table [1](#S4.T1).

**Table 1: Datasets and evaluation budgets. Note: Success in WebShop does not account for partial fulfillment, only scores of 100 are considered successful.**
| Env | #Episodes | Max Steps | Obs. | Actions | Success |
| --- | --- | --- | --- | --- | --- |
| ALFWorld | 1200 Train/134 Test | 50 | text | verb–object | goal reached |
| WebShop | 1200 Train/100 Test | 15 | text/HTML | search/click | score $=100$ |

We use the standard splits provided by the environment. ALFWorld provides a train set of size 3553, and an unseen test set of 134 samples. We use the first 1200 samples from the train set in our experiments, and the full 134 unseen test set for evaluation. The unseen set consists of new room layouts and object-item combinations different from those in the train set.

For WebShop, we take the first 1200 examples out of the original train set of 10,587 as our training set. We do the same with the test set, we take the first 100 examples out of 500 total. The reason behind picking the first 100 was to cap the compute costs. We provide episode IDs in the trials to assist in reproducibility, all subsets are seeded and reproducible. The details of the splits and distribution of task categories are shown in Table [2](#S4.T2).

To isolate the effect of retrieval, we construct two training sources. (i) SFT uses successful trajectories from the base agent (without hints), yielding a hint‑free policy. (ii) Distillation uses successful trajectories produced by the retrieval‑augmented teacher (Agent+RAG), but we strip the hint strings from the input serialization so the student learns the induced behavior rather than relying on hints. For WebShop we retain high‑scoring episodes ($\text{score}\geq 67$) to ensure a sufficient, clean training set.

**Table 2: ALFWorld & WebShop data split distribution and categories**
| ALFWorld | WebShop |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Task Category | Train set | Test set | Product Category | Train set | Test set |
| Cool & Place | 159 | 21 | Beauty | 262 | 24 |
| Clean & Place | 248 | 31 | Electronics | 219 | 19 |
| Examine in Light | 104 | 18 | Fashion | 251 | 23 |
| Heat & Place | 152 | 23 | Food | 239 | 20 |
| Pick & Place | 258 | 24 | Furniture | 229 | 14 |
| PickTwo & Place | 279 | 17 |  |  |  |
| Total | 1200 | 134 | Total | 1200 | 100 |

### 4.2 Baselines and Variants

We evaluate four policy families under matched decoding and step budgets, holding prompt scaffolds fixed within each agent family so that differences reflect retrieval and distillation rather than prompt engineering.

- 1.
Prompt-only baselines (Base). ReAct interleaves Thought and Action with no external guidance. StateAct augments the prompt with Goal/State notes. We use both to assess whether our process transfers across prompting styles (for WebShop, we omit ”Thought” tokens for both agents).
- 2.
Retrieval at inference (RAG). ReAct+RAG and StateAct+RAG receive a fixed advisory block of the top-$k{=}3$ hints injected once at $t{=}0$. Hints are advisory and must yield to the current observation in case of conflict.
- 3.
Supervised fine-tuning (SFT). Trained on successful prompt-only Agent training set trajectories without hints and evaluated without retrieval.
- 4.
Distillation. Trained on successful Agent+RAG trajectories with hint strings removed, and evaluated without retrieval to test whether the student internalizes the guidance.

All policies are run with Qwen-2.5 14B Instruct and Qwen-2.5 7B Instruct backbones under identical decoding settings, step caps, and seeds across comparisons.

### 4.3 Implementation Details

For hint induction, we used GPT-4o to analyze all failed trajectories, yielding 760 ReAct and 650 StateAct hints in ALFWorld, and 756 ReAct and 831 StateAct hints in WebShop after deduplication.

When calculating token cost, we also include the quantized 7B retriever and the injected hint block (100 tokens for k=3 on average).

To ensure data quality, we filtered out trajectories with more than two invalid actions or repeated no-ops, preventing the student from imitating incorrect behaviors.

### 4.4 Training Protocols

We train student models using a QLoRA-style (Dettmers et al., 2023) setup: the backbone is quantized to 4‑bit, while LoRA adapters are trained in 16-bit precision (bf16). We use Unsloth (Han et al., 2023) for adapter loading and low‑precision execution, and TRL’s SFTTrainer (von Werra et al., 2020) for supervised fine‑tuning. Environment‑specific hyperparameters appear in Table [3](#S4.T3).

**Table 3: Training hyperparameters. LR is the learning rate, Seq. len is the maximum input sequence length during training, LoRA rank is the dimension of the low-rank adapters, and LoRA $\alpha$ is the scaling factor applied to them.**
| Environment | LR | Seq. len | LoRA rank | LoRA $\alpha$ | Dropout | Weight decay |
| --- | --- | --- | --- | --- | --- | --- |
| ALFWorld | 2e-4 | 1024 | 64 | 128 | 0.10 | 0.01 |
| WebShop | 2e-4 | 1024 | 16 | 32 | 0.20 | 0.05 |

For optimization, we use 8‑bit AdamW (Loshchilov & Hutter, 2019) with a linear schedule, batch size 2 and 4 gradient‑accumulation steps (effective batch size 8), a single training epoch, and 10% warm‑up. WebShop uses token‑level label smoothing ($\varepsilon{=}0.1$) to reduce overconfidence on short traces, ALFWorld uses $\varepsilon{=}0$. All experiments run on a single NVIDIA A100 (80 GB) with a seed of 42.

## 5 Results

### 5.1 Main Performance Results

Table [4](#S5.T4) shows performance across methods and model scales. Distilled students achieve highest success without retrieval, validating our core hypothesis.

In ALFWorld, distillation exceeds RAG at 14B (91.04% vs. 82.09%) and improves over both base and SFT; at 7B it recovers most of the retrieval gains without requiring hints. In WebShop, distillation matches or exceeds RAG while delivering higher scores with comparable or lower token cost.

**Table 4: Performance of Qwen-2.5 14B Instruct and Qwen-2.5 7B Instruct models averaged across both ReAct and StateAct agents. See Appendix [B.1](#A2.SS1) for individual agent performance.**
| Model | Method | ALFWorld Success | WebShop Success | WebShop Score |
| --- | --- | --- | --- | --- |
| 14B | Base | 79.85% | 38.5% | 60.87 |
|  | Base+RAG | 82.09% | 43.5% | 67.08 |
|  | SFT | 85.45% | 43.0% | 72.09 |
|  | Distilled (ours) | 91.04% | 43.5% | 72.40 |
| 7B | Base | 26.49% | 13.0% | 28.12 |
|  | Base+RAG | 71.27% | 8.5% | 18.46 |
|  | SFT | 62.69% | 22.0% | 54.38 |
|  | Distilled (ours) | 73.88% | 22.5% | 61.04 |

7B models benefit dramatically from hints (26.5% $\rightarrow$ 71.3%) in ALFWorld, but struggle to use them effectively at inference in WebShop. Hints push the small model towards the wrong attributes or over‑constrain the search for the smaller models, the 14B model’s higher capacity lets it better interpret hint semantics. Training offers a better solution, Distillation and SFT greatly improve performance compared to their incredibly low bases. The higher capacity lets the model interpret hint semantics and exploit them at test time better than the smaller models. Distillation compresses capability and, in WebShop‑7B, matches a larger base (Distilled WebShop-7B 61.04 vs Base WebShop-14B 60.87). Across scales, the pattern is consistent: distillation recovers most of RAG’s gains without permanent retrieval, with the largest efficiency advantage observed in WebShop.

### 5.2 Efficiency Analysis

Table [5](#S5.T5) compares token usage and step counts, revealing that distilled models achieve superior performance with lower resource consumption. Distilled models use 10% fewer in ALFWorld and  47% fewer in WebShop vs. base, and  17–61% fewer vs. RAG, while taking fewer steps to complete tasks. This efficiency stems from cleaner action sequences learned from hint-improved teacher demonstrations.

**Table 5: Efficiency and effectiveness of agents across environments using Qwen-2.5 14B Instruct. Metrics are averaged per episode. Success is used for ALFWorld, Score for WebShop. Lower is better for Tokens and Steps. See Appendix [B.2](#A2.SS2) for 7B efficiency results.**
| Environment | Method | Tokens/Episode $\downarrow$ | Steps/Episode $\downarrow$ | Success/Score $\uparrow$ |
| --- | --- | --- | --- | --- |
| ALFWorld | Base | 50.13k | 18.94 | 79.85% |
|  | RAG | 53.97k | 18.69 | 82.09% |
|  | SFT | 50.36k | 19.38 | 85.45% |
|  | Distilled (ours) | 44.82k | 16.68 | 91.04% |
| WebShop | Base | 7.99k | 7.16 | 60.87 |
|  | RAG | 11.05k | 6.34 | 67.08 |
|  | SFT | 4.29k | 5.00 | 72.09 |
|  | Distilled (ours) | 4.27k | 4.98 | 72.40 |

As seen in Figure [3](#S5.F3), across both environments, the trend is consistent. Retrieval helps, but its token overhead shifts methods to the right, while distillation recovers most of the retrieval gains at a much lower cost. In ALFWorld, distillation dominates the trade-off. It matches or exceeds RAG’s accuracy at a fraction of the tokens, achieving the highest success rates ($\sim$90-93%). In WebShop, the most efficient frontier is formed by both SFT and distillation, achieving the best scores ($\sim$71–73) with the lowest average token counts per episode (below 6k). RAG improves over the base agent, but remains less token-efficient. The shaded ellipses highlight that these behaviors are regime-level, not just single points: base runs generally cluster at low cost/low accuracy, RAG clusters at higher cost with high performance, and distillation clusters at low cost/highest performance, especially in ALFWorld.

Figure: (a) ALFWorld
Refer to caption: /html/2510.01375/assets/figures/alfworld_score_vs_tokens.png

## 6 Limitations

This study has several limitations. First, hint generation relies on repeated GPT-4o API calls, making cost dependent on the number of failures in the training data and potentially limiting scalability in larger environments. Second, retrieval is restricted to a single one-shot process at $t=0$, which cannot adapt to mid-episode surprises and may be less effective in long-horizon or highly stochastic tasks. Third, all reported results are based on single-seed evaluation and therefore represent point estimates; while consistent trends are observed, multi-seed experiments are required to quantify variance and assess robustness. Finally, our evaluation is limited to the training environments (ALFWorld and WebShop), without testing cross-domain transfer. Understanding how well distilled agents generalize to novel settings remains an open question.

## 7 Conclusion

We presented a method for converting retrieval-augmented generation from a runtime necessity into a training-time teacher. By extracting hints from failures, using them to generate better trajectories, and distilling with hints removed, we produce agents that internalize guidance while eliminating deployment overhead. Distilled students achieve 91% success on ALFWorld (vs 79% baseline) and score 72.4 on WebShop (vs 60.9 baseline), while using fewer tokens than any alternative approach.

The method is simple, requires no expert supervision, and generalizes across model scales and agent architectures. Results suggest many augmentation strategies currently treated as runtime requirements might better serve as training-time supervision. Future work should explore dynamic retrieval triggers, trajectory-level objectives for long-horizon tasks, and cross-environment transfer to test whether distilled competencies truly generalize beyond their training distribution.

#### Reproducibility Statement

We have taken several steps to ensure reproducibility. A link to our anonymized source code, including training and evaluation scripts, is provided here: [https://anonymous.4open.science/r/anonymized-submission-iclr/README.md](https://anonymous.4open.science/r/anonymized-submission-iclr/README.md).
Detailed prompts are described in Appendix [A](#A1).
All experiments are run with fixed seeds. The processed hint banks and student training datasets used to generate the reported results are included in the source code. Together, these resources should enable full replication of the experiments and verification of our claims.

## References

- Chen et al. (2023)
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.
Fireact: Toward language agent fine-tuning, 2023.
URL [https://arxiv.org/abs/2310.05915](https://arxiv.org/abs/2310.05915).
- Chen et al. (2024)
Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He.
Automanual: Constructing instruction manuals by llm agents via interactive environmental learning, 2024.
URL [https://arxiv.org/abs/2405.16247](https://arxiv.org/abs/2405.16247).
- Dettmers et al. (2023)
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
Qlora: Efficient finetuning of quantized llms.
*arXiv preprint arXiv:2305.14314*, 2023.
- Fu et al. (2024)
Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Honglak Lee.
Autoguide: Automated generation and selection of context-aware guidelines for large language model agents, 2024.
URL [https://arxiv.org/abs/2403.08978](https://arxiv.org/abs/2403.08978).
- Han et al. (2023)
Daniel Han, Michael Han, and Unsloth team.
Unsloth, 2023.
URL [http://github.com/unslothai/unsloth](http://github.com/unslothai/unsloth).
- Han et al. (2024)
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang.
Parameter-efficient fine-tuning for large models: A comprehensive survey, 2024.
URL [https://arxiv.org/abs/2403.14608](https://arxiv.org/abs/2403.14608).
- Hsieh et al. (2023)
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.
Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes, 2023.
URL [https://arxiv.org/abs/2305.02301](https://arxiv.org/abs/2305.02301).
- Hu et al. (2021)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models, 2021.
URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Kujanpää et al. (2025)
Kalle Kujanpää, Pekka Marttinen, Harri Valpola, and Alexander Ilin.
Efficient knowledge injection in llms via self-distillation, 2025.
URL [https://arxiv.org/abs/2412.14964](https://arxiv.org/abs/2412.14964).
- Lewis et al. (2021)
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela.
Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
URL [https://arxiv.org/abs/2005.11401](https://arxiv.org/abs/2005.11401).
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization, 2019.
URL [https://arxiv.org/abs/1711.05101](https://arxiv.org/abs/1711.05101).
- Qin et al. (2024)
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky.
Large language models are effective text rankers with pairwise ranking prompting, 2024.
URL [https://arxiv.org/abs/2306.17563](https://arxiv.org/abs/2306.17563).
- Qwen et al. (2025)
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 Technical Report, January 2025.
URL [http://arxiv.org/abs/2412.15115](http://arxiv.org/abs/2412.15115).
arXiv:2412.15115 [cs].
- Rozanov & Rei (2025)
Nikolai Rozanov and Marek Rei.
Stateact: Enhancing llm base agents via self-prompting and state-tracking, 2025.
URL [https://arxiv.org/abs/2410.02810](https://arxiv.org/abs/2410.02810).
- Sanh et al. (2020)
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
URL [https://arxiv.org/abs/1910.01108](https://arxiv.org/abs/1910.01108).
- Schroeder et al. (2025)
Philip Schroeder, Nathaniel Morgan, Hongyin Luo, and James Glass.
Thread: Thinking deeper with recursive spawning, 2025.
URL [https://arxiv.org/abs/2405.17402](https://arxiv.org/abs/2405.17402).
- Shinn et al. (2023)
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning, 2023.
URL [https://arxiv.org/abs/2303.11366](https://arxiv.org/abs/2303.11366).
- Shridhar et al. (2021)
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning.
In *Proceedings of the International Conference on Learning Representations (ICLR)*, 2021.
URL [https://arxiv.org/abs/2010.03768](https://arxiv.org/abs/2010.03768).
- Sun et al. (2023)
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren.
Is ChatGPT good at search? investigating large language models as re-ranking agents.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 14918–14937, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.923.
URL [https://aclanthology.org/2023.emnlp-main.923/](https://aclanthology.org/2023.emnlp-main.923/).
- von Werra et al. (2020)
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec.
Trl: Transformer reinforcement learning.
[https://github.com/huggingface/trl](https://github.com/huggingface/trl), 2020.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.
Chain of thought prompting elicits reasoning in large language models.
*CoRR*, abs/2201.11903, 2022.
URL [https://arxiv.org/abs/2201.11903](https://arxiv.org/abs/2201.11903).
- Yao et al. (2023a)
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.
Webshop: Towards scalable real-world web interaction with grounded language agents, 2023a.
URL [https://arxiv.org/abs/2207.01206](https://arxiv.org/abs/2207.01206).
- Yao et al. (2023b)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023b.
URL [https://arxiv.org/abs/2210.03629](https://arxiv.org/abs/2210.03629).
- Zelikman et al. (2022)
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman.
Star: Bootstrapping reasoning with reasoning, 2022.
URL [https://arxiv.org/abs/2203.14465](https://arxiv.org/abs/2203.14465).
- Zhang et al. (2025)
Yunxiao Zhang, Guanming Xiong, Haochen Li, and Wen Zhao.
Edge: Efficient data selection for llm agents via guideline effectiveness, 2025.
URL [https://arxiv.org/abs/2502.12494](https://arxiv.org/abs/2502.12494).
- Zhao et al. (2024)
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang.
Expel: Llm agents are experiential learners, 2024.
URL [https://arxiv.org/abs/2308.10144](https://arxiv.org/abs/2308.10144).

## Appendix

## Appendix A Prompts

LLMs can be sensitive to prompt design, and the exact wording often plays a critical role in performance.
To ensure full transparency and reproducibility, we include here the complete prompts used in our experiments.
This appendix documents the prompts used for hint generation and agent evaluation.

### A.1 Hint Generation

The prompts used with GPT-4o to generate runtime hints are given below. For ALFWorld:

The prompt for WebShop is identical, it is given below:

### A.2 Hint Retrieval Prompt

Below are the prompts used in retrieval, they were designed to also account for usage mid-episode, but we ended up not using that functionality as it would increase token usage.

### A.3 Agent Prompt

Below are the prompts used for the main agent. It provides the agent with its task and few-shot examples, and includes a hint block (if using RAG). If evaluating without hints, the lines are removed from ”Here are some hints:” until the separators ”=============”.

## Appendix B Extended Results and Ablations

### B.1 Per-Agent Results

While the main text reports averages (Table [4](#S5.T4)), here we report per‑agent results.

Tables [6](#A2.T6) and [7](#A2.T7) break down performance by agent family.
For 14B models, both ReAct and StateAct benefit from distillation: ReAct shows the largest jump in ALFWorld (77.6% $\rightarrow$ 92.5%), while StateAct achieves the strongest WebShop performance (score 72.8–72.5).
For 7B models, the same trend holds: ReAct+Distillation lifts WebShop score from 44.0 to 57.1, while StateAct+Distillation reaches 79.1% success on ALFWorld and 65.0 WebShop score, substantially outperforming its baselines.
These per-agent breakdowns confirm that distillation is effective across prompting styles, with complementary strengths: ReAct gains most in ALFWorld, while StateAct consistently excels in WebShop.

**Table 6: Performance of Qwen-2.5 14B Instruct agents across methods. ALFWorld reports Success (%), WebShop reports Success (%) and Score.**
| Method (14B) | ALFWorld Success | WebShop Success | WebShop Score |
| --- | --- | --- | --- |
| ReAct | 77.61 | 37.00 | 61.61 |
| +RAG | 79.10 | 43.00 | 66.97 |
| +SFT | 86.57 | 42.00 | 71.40 |
| +Distillation (ours) | 92.54 | 43.00 | 72.32 |
| StateAct | 82.09 | 40.00 | 60.12 |
| +RAG | 85.07 | 44.00 | 67.18 |
| +SFT | 84.33 | 44.00 | 72.78 |
| +Distillation (ours) | 89.55 | 44.00 | 72.49 |

**Table 7: Performance of Qwen-2.5 7B Instruct agents across methods. ALFWorld reports Success (%), WebShop reports Success (%) and Score.**
| Method (7B) | ALFWorld Success | WebShop Success | WebShop Score |
| --- | --- | --- | --- |
| ReAct | 11.19 | 19.00 | 44.05 |
| +RAG | 73.13 | 9.00 | 28.80 |
| +SFT | 50.00 | 22.00 | 44.25 |
| +Distillation (ours) | 68.66 | 31.00 | 57.11 |
| StateAct | 41.79 | 5.00 | 12.18 |
| +RAG | 69.40 | 4.00 | 8.12 |
| +SFT | 75.37 | 38.00 | 64.51 |
| +Distillation (ours) | 79.10 | 39.00 | 64.97 |

### B.2 7B Efficiency Results

Table [8](#A2.T8) compares token usage and step counts for 7B models.
As with 14B (Table [5](#S5.T5)), distillation provides the strongest balance of effectiveness and efficiency.
In ALFWorld, distilled students reach 68.7% success, far above the base (11.2%) and close to RAG (73.1%), while avoiding the heavy runtime dependence on retrieval.
In WebShop, the advantage is even clearer: distillation improves the score to 57.1 while using fewer tokens than all other methods.
Taken together, these results show that distillation allows 7B models to internalize most of the retrieval gains, achieving accuracy competitive with larger or retrieval-augmented baselines while being substantially more efficient.

**Table 8: Efficiency and effectiveness of agents across environments using Qwen-2.5 7B Instruct. Metrics are averaged per episode. Success is used for ALFWorld, Score for WebShop. Lower is better for Tokens and Steps.**
| Environment | Method | Tokens/Episode $\downarrow$ | Steps/Episode $\downarrow$ | Success/Score $\uparrow$ |
| --- | --- | --- | --- | --- |
| ALFWorld | Base | 48.71k | 22.26 | 11.19% |
|  | RAG | 62.74k | 27.16 | 73.13% |
|  | SFT | 62.08k | 25.14 | 50.00% |
|  | Distilled (ours) | 67.62k | 27.86 | 68.66% |
| WebShop | Base | 9.36k | 7.89 | 44.05 |
|  | RAG | 22.02k | 10.07 | 28.80 |
|  | SFT | 8.18k | 8.50 | 44.25 |
|  | Distilled (ours) | 5.95k | 7.15 | 57.11 |

### B.3 Retrieval Depth (k) Ablations

In our experiments, we initially set k=3 as a pragmatic default. Small k is preferred in retrieval-augmented setups because it balances performance against prompt length. To validate this choice, we swept k$\in\{1,3,6,9\}$ on both ALFWorld and WebShop with Qwen-14B and report success, score, steps, and token cost (Tables [9](#A2.T9) & [10](#A2.T10)).

**Table 9: Different retrieval k on ALFWorld with Qwen-2.5 14B Instruct. Results are averaged across ReAct and StateAct agents.**
| Top-k | Success Rate (%) | Avg. Steps | Avg. Tokens |
| --- | --- | --- | --- |
| 1 | 83.96 | 19.11 | 52.02k |
| 3 (ours) | 82.09 | 18.69 | 53.97k |
| 6 | 84.33 | 18.13 | 50.95k |
| 9 | 76.87 | 19.27 | 57.26k |

For ALFWorld, performance is stable across k$\in\{1,3,6\}$, with success rates in the 82–84% range and token cost between 51k–54k. A sharp drop appears at k=9, where success falls to 76.9% and token usage inflates to over 57k tokens. This confirms that large hint blocks become counterproductive by increasing prompt length without improving guidance.

In WebShop, retrieval depth also shows diminishing returns. The best balance is reached at k=3 (43.5% success, score 67.1), which ties or slightly improves on k=1 while maintaining similar token cost. Larger k values ($6$ and $9$) lead to higher token usage (12–13k) and reduced scores ($\approx$61), showing that excessive hints can over-constrain search.

**Table 10: Different retrieval k on WebShop with Qwen-2.5 14B Instruct. Results are averaged across ReAct and StateAct agents.**
| Top-k | Success Rate (%) | Score | Avg. Steps | Avg. Tokens |
| --- | --- | --- | --- | --- |
| 1 | 40.50 | 64.73 | 6.45 | 11.04k |
| 3 (ours) | 43.50 | 67.08 | 6.34 | 11.05k |
| 6 | 40.00 | 61.80 | 6.94 | 12.75k |
| 9 | 40.50 | 61.12 | 7.16 | 13.12k |

We therefore keep k=3 as a uniform default. This beats k=1 in WebShop, remains competitive with k=1 and 6 in ALFWorld, and avoids the instability of k=9 in both environments. If we were to tune per domain, a simple policy would be to use more hints (k=6) for structure-heavy tasks (ALFWorld), and use fewer hints (k=3) for noisy, attribute-heavy tasks (WebShop).

## Appendix C The Use of Large Language Models (LLMs)

We acknowledge using ChatGPT (GPT-5) provided by OpenAI ([https://chatgpt.com](https://chatgpt.com)) to help draft, edit text, and to produce LaTeX code templates. All outputs were reviewed, fact-checked, and substantially edited by us. We confirm that the work presented is our own.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. **Hint Generator Robustness**
   How sensitive is your pipeline to the quality of the hint generator?

   * Have you tested weaker or smaller LLMs (e.g., GPT-3.5, Claude, or open-source models like Mistral-7B) for hint extraction?
   * If performance degrades, what specific aspects, i.e., hint conciseness, structure, or accuracy, most affect downstream learning?
     *A convincing ablation here would clarify whether the pipeline is viable without expensive closed-source LLMs.*

2. **Cross-Domain Generalization**
   Can the distilled agent transfer knowledge from one environment to another?

   * For instance, could a model trained on ALFWorld hints generalize to unseen household task variations or WebShop-like reasoning tasks without additional RAG?
   * Do the hints encode reusable “task primitives” or are they too domain-specific?
     *Understanding the degree of reusability of the learned hints would help assess the true scope of this method.*

3. **Statistical Rigor and Variance Reporting**
   Are the reported results averaged over multiple seeds?

   * Given that retrieval and LLM outputs introduce randomness, have you measured performance variance across runs?
   * If not, could the observed improvements (e.g., 91% vs. 82%) be within expected variance margins?
     *Providing mean ± std metrics would significantly improve credibility.*

4. **Comparison to RAFT and Retrieval-Based Fine-Tuning Methods**
   Have you considered comparing directly to RAFT or hybridizing with it?

   * For instance, pretraining the backbone with RAFT, then applying your hint-distillation pipeline to internalize common retrieval patterns?
   * Do you expect the two methods to be additive, or would RAFT’s retrieval-conditioned fine-tuning conflict with your retrieval-free student training?
     *A small-scale RAFT baseline or hybrid experiment would clarify the relationship and strengthen your positioning.*

5. **One-Shot Retrieval Limitation**
   Why restrict hint retrieval to a single step at episode start?

   * Did you attempt multi-step retrieval or adaptive hint updates mid-trajectory?
   * If not, is this primarily a cost/complexity trade-off or a deliberate design choice to enforce self-sufficiency?
     *Clarifying this would help understand whether the framework can extend to more dynamic, long-horizon tasks.*

6. **Overfitting and Hint Style Bias**
   The student model is trained on teacher trajectories with hints removed—but could it still overfit to stylistic patterns introduced by the RAG teacher?

   * Have you tested robustness under paraphrased or intentionally noisy hints?
   * Do hints produced by different LLMs (with different tone or structure) affect final student performance?
     *This would clarify whether the distilled competence reflects genuine abstraction rather than surface imitation.*

7. **Scalability of Failure Mining in Real-World Systems**
   Your pipeline assumes access to structured failure trajectories and outcome signals.

   * How feasible is it to apply this approach to free-form, user-facing LLM interactions where “failure” is subjective or weakly labeled?
   * Could implicit feedback (e.g., user corrections or re-prompts) serve as proxy failures for hint extraction?
     *Discussing this would extend the method’s relevance beyond simulator benchmarks.*

8. **Long-Term Learning and Continual Adaptation**
   Once a student is distilled, can the pipeline be repeated iteratively as new failures appear?

   * Does performance plateau after one distillation round, or can further RAG-assisted hint extraction continue improving the student?
     *Evidence of continual learning potential would make the approach even more compelling.*

9. **Compute and Efficiency Accounting**
   Could you share concrete compute/token usage breakdowns for each stage (failure mining, hint extraction, RAG teacher runs, student fine-tuning)?

   * This would help readers balance efficiency gains at inference against additional training overhead.

10. **Interpretability of the Learned Knowledge**
    Since hints are typed and structured, have you tried aligning the student’s internal representations with these hint categories post-training?

     * For example, probing whether the model has implicitly learned “open-container” or “check-object-state” concepts.
     *Such analysis could offer a unique window into how RAG-derived supervision shapes internal reasoning.*
soundness: 4
strengths: 1. **Conceptual Novelty (Turning RAG into a Training-Time Teacher)**
   The paper introduces a genuinely fresh perspective on retrieval-augmented learning. Instead of relying on RAG *at inference*, it reinterprets RAG as a *supervision mechanism* using retrieved, failure-derived hints to generate improved demonstrations and distilling them into the model. This effectively *converts external retrieval into internal reasoning skill*, a direction that feels both original and timely.

2. **Failure-Driven Knowledge Extraction Pipeline**
   The automatic hint extraction from failed trajectories (via GPT-4o) is one of the paper’s strongest contributions. By converting raw failure logs into short, typed hints with placeholders, the method operationalizes the idea of *learning from one’s own mistakes*. This makes the system data-efficient and highly interpretable, i.e., each hint represents a distilled form of experiential correction.

3. **Practical and Deployment-Friendly Design**
   The approach strikes an excellent balance between performance and efficiency. The distilled student models retain the accuracy of RAG agents while dropping retrieval costs and context overhead. In practice, this means fewer API calls, lower latency, and more predictable runtime behavior which are key deployment benefits often ignored in academic RAG work.

4. **Empirical Results Support the Hypothesis**
   On both **ALFWorld** and **WebShop**, the results clearly show that the distilled agents outperform or match the RAG baseline while using fewer tokens. The ablations (e.g., number of retrieved hints (k), LoRA scale) are sensible and validate design choices. The efficiency frontier plots strongly convey that the gains are not just anecdotal but structural.

5. **Methodological Clarity and Reproducibility**
   Despite involving multiple moving parts (hint extraction, re-ranking, teacher generation, and LoRA distillation), the paper presents its pipeline in a clean, reproducible manner. Each stage is modular, making it easy to replicate or adapt for other domains.

6. **Complementarity with RAFT and Other RAG–Fine-Tuning Hybrids**
   From a broader research standpoint, this work complements approaches like **RAFT**. While RAFT fine-tunes models to use retrieval *during inference*, this paper focuses on distilling retrieval’s benefits *into the model weights*. This complementarity opens exciting ground for hybrid pipelines, i.e., RAFT for retrieval robustness; this is for retrieval-free competence.

7. **High Potential for Generalization Beyond Current Benchmarks**
   The approach is architecture-agnostic and task-flexible. Any domain with observable failures (reasoning, code, robotics, or web agents) could feasibly benefit from failure-driven hint distillation. It’s a general recipe for turning external supervision into internal skill formation.
summary: This paper proposes a novel approach to *retrieval-augmented fine-tuning* where Retrieval-Augmented Generation (RAG) is not used merely at inference but as a **training-time teacher**. The authors introduce a pipeline that leverages **failure-driven hint extraction** and **hint-based RAG distillation** to improve the downstream agent’s reasoning and efficiency without requiring retrieval at runtime.

The method operates in three main stages:

1. **Failure Mining and Hint Extraction** – The base agent’s failed trajectories are analyzed using GPT-4o to produce short, typed *hints* (e.g., “try opening the `{container}` before placing `{object}`”), generalizing the error into reusable advice.
2. **Hint Retrieval and RAG Teacher Generation** – At training time, a small set of relevant hints ((k \leq 3)) is retrieved once per episode and injected into the teacher prompt. The RAG teacher then produces improved, successful trajectories.
3. **Hint-Removed Distillation** – The student model (fine-tuned with LoRA) is trained on the teacher’s improved trajectories, but with hints *removed*, encouraging it to internalize the corrections rather than rely on retrieval tokens.

Empirical results on **ALFWorld** and **WebShop** demonstrate that the distilled students achieve comparable or superior success rates to RAG-based agents, e.g., up to *91% success* on ALFWorld while reducing inference token usage and removing retrieval dependency entirely.

The core insight is that *retrieval can serve as a teacher rather than a crutch*: by extracting and reusing failure-derived knowledge during training, models can **inherit RAG’s strengths while maintaining fine-tuned efficiency**.

This perspective is complementary to methods such as **RAFT (Retrieval-Augmented Fine-Tuning)**, which enhance RAG performance at inference time; in contrast, this work uses RAG supervision to **eliminate the need for retrieval at deployment**.
weaknesses: 1. **Dependence on Expensive Hint Generation (GPT-4o)**
   The hint extraction process relies heavily on GPT-4o to produce concise, structured hints from failed trajectories. While conceptually elegant, this introduces a non-trivial *cost and dependency barrier*. It’s unclear how well the pipeline performs if the hint generator is weaker (e.g., Claude, GPT-3.5, or an open-source model). An ablation on hint-source quality would be crucial to validate the approach’s robustness and accessibility.

2. **Limited Domain Scope and Missing Generalization Evidence**
   Evaluation is restricted to **ALFWorld** and **WebShop**, both of which are text-based and relatively synthetic. There’s no evidence of transfer to other environments or domains (e.g., tool-use, programming, or reasoning tasks). Without cross-domain or out-of-distribution experiments, it’s difficult to gauge whether the distilled behaviors generalize or merely specialize to the training distribution.

3. **Single-Seed Results and Lack of Statistical Rigor**
   The reported results appear to rely on single-seed runs. Given the stochasticity of both retrieval and LLM rollouts, variance across seeds could be substantial. Multi-seed evaluation and confidence intervals would strengthen empirical reliability.

4. **One-Shot Retrieval Constraint**
   The design retrieves hints only *once* at episode start. While this simplifies the setup, it limits adaptivity in longer-horizon or dynamic tasks where new context emerges mid-trajectory. This constraint could lead to brittle performance in environments requiring sequential replanning or dynamic retrieval updates.

5. **Potential Overfitting to Hint Styles**
   Even though hints are removed during student fine-tuning, the student may still internalize surface-level templates or stylistic priors specific to the GPT-4o hint format. The absence of tests on noisy, paraphrased, or conflicting hints leaves open the question of whether the distilled competence reflects *true abstraction* or *pattern mimicry*.

6. **Missing Comparison to RAFT or Other RAG-Fine-Tuning Baselines**
   The paper positions itself as complementary to RAFT (Retrieval-Augmented Fine-Tuning), yet no direct empirical comparison is provided. A controlled RAFT baseline or a hybrid experiment where RAFT pretraining precedes hint distillation would help clarify relative strengths. Without this, the distinction remains conceptual rather than quantitative.

7. **Scalability and Practicality of the Failure Mining Loop**
   The method assumes the availability of structured failure logs and deterministic access to environment feedback. This is feasible in ALFWorld-like simulators but less practical in noisy real-world pipelines (e.g., user-facing chat agents). Some discussion or lightweight simulation of such conditions would make the method’s applicability clearer.

8. **Lack of Long-Term Behavior Evaluation**
   The experiments measure immediate task success, but not whether the distilled policies preserve *stability* or *general reasoning improvements* over multiple episodes or domains. A longitudinal or continual-learning evaluation would have given deeper evidence that the model truly internalized RAG-derived knowledge.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Please outline the benefits of using an LLM for hint reranking and retrieval.

2. Please provide additional information regarding the quality of the hints generated by the LLM.

3. Please include evaluations on additional challenging interactive benchmarks, such as WebArena [1].

[1] Zhou, Shuyan et al. “WebArena: A Realistic Web Environment for Building Autonomous Agents.” arXiv abs/2307.13854 (2023).
soundness: 2
strengths: The strengths of this paper are listed below:
1. This work successfully demonstrates that the hints generated with the help of hints lead to better performance.
2. The distilled model demonstrates better token efficiency.
summary: Despite their strong performance on reasoning benchmarks, large language models (LLMs) tend to perform poorly on multi-step tasks without preconditioning on task actions and environment state spaces. LLMs augmented with retrieval-augmented generation (RAG) have demonstrated strong performance but require maintaining external knowledge bases and incur additional computational cost at runtime. In this work, the authors propose a framework that generates improved reasoning trajectories from a teacher model using RAG for hint selection. These generated trajectories are then used to enhance the performance of a student model on interactive reasoning benchmarks.
weaknesses: The weaknesses of this work are as follows:

1. Supervised Fine-Tuning (SFT) is known to cause overfitting to the training domain; hence, the observed performance gains may not generalize well.

2. The performance of RAG remains low on both benchmarks. It would be valuable to explore how these failure cases could be leveraged in RL training to learn more generalizable reasoning patterns.

3. The hints are generated using API calls to an external LLM, which raises questions about the quality and reliability of the generated hints.

4. An LLM is also used for reranking the retrieved hints, but it is unclear why a more traditional and cost-effective retrieval pipeline was not employed instead.

5. For the 14B model, the performance improvement is evident only on the ALFWorld benchmark.

6. The approach is evaluated on only two benchmarks; extensive experiments on additional datasets are necessary to validate its general applicability.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. why the distilled student model performs better than the teacher model (RAG)? Are there any hyper-parameter optimization included?
2. why use LoRA instead of full-finetuing?
soundness: 2
strengths: 1. Clear motivation: The paper tackles a real limitation of RAG—its runtime cost and dependency on external databases—by proposing a training-time alternative that internalizes retrieval benefits.

2. Strong empirical results: Distilled models achieve 91% success on ALFWorld (vs. 79% baseline) and 72.4 score on WebShop (vs. 60.9 baseline) while using 10–60% fewer tokens.

3. Generalization across setups: Demonstrated effectiveness for both 7B and 14B models and two agent architectures (ReAct, StateAct).
summary: This paper introduces a method to **distill retrieval-augmented generation (RAG)** into fine-tuned LLMs, eliminating runtime retrieval while retaining its benefits. By extracting corrective hints from agent failures and using them once during training to generate improved teacher trajectories, the model learns to internalize guidance instead of depending on external retrieval. Tested on ALFWorld and WebShop, the distilled models achieve **notably higher success (up to 91%) and efficiency (10–60% fewer tokens)** than baselines. The approach is **simple, effective, and generalizes across architectures**, though scalability and evaluation breadth remain limited.
weaknesses: * **Limited novelty:** While well-executed, the core idea—distilling knowledge from prior retrieval or failures—is conceptually close to existing self-improvement and distillation frameworks (e.g., FireAct, Reflexion). The contribution lies more in engineering and integration than in fundamental innovation.
* **Narrow experimental scope:** All experiments are conducted only on **Qwen-2.5 models (7B/14B)**, raising concerns about generality across architectures (e.g., Llama, Mistral etc).
* **Dependence on GPT-4o for hint extraction:** The pipeline still relies on a powerful external model to generate failure-driven hints, which may limit scalability and reproducibility.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Does the quantized Qwen-2.5 7B provide reliable hint ranking? Is there any ablation study on the effect of hint ranking or retrieval quality?
2. Have the authors tested whether the distilled model maintains its advantage when evaluated on unseen environments or different task distributions? Could the retrieved hints unintentionally leak task-specific answers?
soundness: 3
strengths: Overall, the main message that retrieval can be treated as a tempoaray training teacher rather than a permanent runtime component is interesting and useful. The paper is clearly written and the automatic pipeline seems practical and deployment-friendly.
summary: This paper reframes retrieval-augmented generation (RAG) as a training-time teacher rather than a runtime dependency, which is an interesting idea. The authors propose a four-stage pipeline, and the resulting “distilled” agents internalize the retrieval benefits and achieve better performance while using fewer tokens per episode.
weaknesses: 1. Generalization: The main concern lies in the generalization of the findings. As the authors acknowledge, they evaluate only on two text-based environments (ALFWorld and WebShop) using Qwen-2.5 7B/14B as the base models and single-seed runs. Although the reported results are consistent, it would be more convincing to include evaluations on more diverse benchmarks and multiple random seeds.
2. Re-ranking biase: The use of a quantized Qwen-2.5 7B model for scoring hint relevance may introduce bias, as it belongs to the same model family as the student and teacher agents. It would be preferable to use a different model family or an unquantized version for re-ranking to reduce potential bias.
3. Training costs: The training process relies on repeated GPT-4o calls for hint generation and the quantized Qwen-2.5 7B for hint scoring. This setup likely increases training costs and may slow down the process due to waiting for external feedback.


Decision: Reject