Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations
Authors: Zichao Li
Gang Wu
Zichao Wang
Ruiyi Zhang
Wanrong Zhu
Ryan A. Rossi
Vlad I Morariu
Jihyung Kil
Venue: ICLR 2026 Poster (2026)
Keywords: hindsight learning
agentic LLM
LLM
post training
RL
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a sample-efficient post-training method for LLM agents that turns their trajectories into successful demonstrations the agents use to learn and improve.

Abstract: Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.

Published as a conference paper at ICLR 2026
SPINNING STRAW INTO GOLD: RELABELING LLM AGENT TRAJECTORIES IN HINDSIGHT FOR SUCCESS- FUL DEMONSTRATIONS
Zichao Li Gang Wu'; Zichao Wang", Ruiyi Zhang?, Wanrong Zhu?, Ryan A. Rossi", Vlad I. Morariu", Jihyung Kil?
Mila, McGill University Adobe Research
zichao. 1i Semall.ae wew.emmmmmam.l........m........m wwgill.cca. jkil@adobe. com
ABSTRACT
Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-languaa goals the agent actually achieved. HSL. then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post- training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.
INTRODUCTION
Large language model (LLM) agents extend foundation models (Bommasani et al., 2021) by enabling them to operate within interactive environments, including but not limited to autonomous web browsing and form completion (Zheng et al., 2024), tool-augmented question answering (Zhuang et al., 2023), software engineering (Jimenez et al., 2023), strategic decision-making in simulators and games (Fan et al., 2022), and high-level control of embodied robots (Li et al., 2024). Such agents increasingly important because they bridge the gap between raw language models and practical, interactive intelligence. However, building effective LLM agents remains a challenging task. In most tasks, the dynamics of the underlying high-dimensional environment are complex and hidden, and the observations available to the agent reveal only partial information about it. Even so, the agent should plan over long horizons and choose actions whose effects uncertain.
variety of methods have been explored to train LLM agents, including supervised fine-tuning (SFT), direct preference optimization (DPO) (Rafailov al., 2023), and other reinforcement learning (RL. techniques (Schulman et al., 2017; Liu et al., 2025). Among them, SFT is the most widely used, but it relies heavily on expert demonstrations, which are costly collect and often lack diversity. More importantly, it does not fully leverage the trajectories generated by the agent itself. In contrast, RL can, in principle, exploit such data through trial and error. However, requires discovering high-return trajectories before learning can proceed, which is challenging in long- horizon tasks with sparse rewards. The difficulty is further intensified when the agent's 's interaction with the environment is restricted, for example, by safety or privacy concerns in embodied or web agents (Tur et al., 2025).
The intuition behind this work, inspired by hindsight experience replay (Andrychowicz et al., 2017) in goal-conditioned RI, is that an LLM agent may accomplish unintended but valid tasks regardless of whether completes the instructed one. Take Figure/ I as a conceptual example. When instructed to reach the blue flag, the agent might miss the target entirely and instead ar- rive at the red flag. Alternatively, it might reach the yellow flag before eventually reaching the
Work done during an intersship Adobe Research.

---

Published as a conference paper at ICLR 2026
blue one. Therefore, it is feasible to relabel the agent's trajectories with the unintended goals, such as the red or yellow flags, and then extract such unintended achievements from the agent's experience, turning them into successful demonstrations that can be consolidated for learning and performance gains. We assume that the relabeling task is easier than solving the origi- nal task, it be performed after the full trajectory is collected and does not require pre- dicting environment dynamics, which are often among the hardest aspects of LLM agent tasks. Instead, the problem reduces inferring which tasks were actually accomplished by reasoning over the observations in hindsight, a perception and reasoning challenge that LLMs are well- suited to handle.
Go to the blue flag
In view of this, we propose Hindsight Supervised Learning (HSL) (Figure 2). HSL. uses an auxiliary LLM relabel agent- generated trajectories with the goals actually achieved by the agent and fine-tunes these relabeled examples with SFT. Since the original trajectories are generated under different instrue- tions, they may be suboptimal for the relabeled goals. We thus introduce additional learning strategies, irrelevant-acho masking and rela- beled demonstration reweighting. These strate- gies help agents learn more effectively to follow natural language instructions and reproduce suc- cessful behaviors.
Figure I: Toy example. Given the instruction "Go to the blue flag". the agent may instead reach the red flag or visit the yellow flag along the way. While these trajectories are incorrect or not optimal for the original instruction, they can be relabeled as successful demonstrations for the goals the agent actually achieved.
In sum, our contributions are threefold:
We propose a novel post-training method for LLM agents, Hindsight Supervised Learning (HSL), which iteratively mines successful demonstrations by relabeling agent trajectories and fine-tunes the agent on the relabeled data.
Based on operational insights from our theoretical analysis, we introduce two simple yet effective learning methods, irrelevant-action actioon masking and sample reweighting, which are further validated through ablation studies.
Our experiments on ALFWorld (Shridhar et al., 2020), PlanCraft (Dagan al., 2024) and WebShop (Yao et al., 2022) demonstrate that (1) HSL is compatible with existing post- training methods such as SFT and DPO, yielding notable improvements (e.g., by 8% 32% success rate in ALFWorld), (2) HSL is sample-efficient For instance, with less than quarter of ground-trun truth demonstrations, outperforms bascline methods trained on the full dataset.
RELATED WORK
2.1 LLM AGENTS
Early language agents were developed for artificial text-based interactive environments (Chevalier- Boisvert et al., 2018), where agents follow natural language instructions to change states and achieve goals. This shift in focus transformed NLP from static prediction to sequential decision-making. With the rapid advancement of language models, I.LM agents now operate more realistic and complex domains. These include embodied agents (Li et al., 2024); GUI agents (Yao al., 2022; Zhou et al., 2024; Xie et al., 2024); and code agents (Jimenez et al., 2023).
Post-training methods for these agents follow two main approaches. Supervised fine-tuning (SFT) on demonstrations is effective but costly and limited by coverage. Preference- and RL-based training, such PPO (Schulman et al., 2017; Hu et al., 2025), DPO (Rafailov al., 2023; Song et al., 2024), aims to improve agents with weaker supervision, yet often struggles under sparse, delayed feedback and long horizons. To mitigate sparsity, recent approaches synthesize feedback using heuristics or
2

---

Published as a conference paper at ICLR 2026
learned judges (Da al., 2025) and conduct intensive interactions with the environment to obtain denser intermediate rewards (Xiong et al., 2024).
By contrast, our work proposes alternative approach: maximizing the number of successful demonstrations through agent experiences. Nevertheless, this approach is complementary to existing learning algorithms, and we demonstrate that it can provide additive improvement when combined with different post-training methods.
Another line of research synthesizes large-scatl aarge-ccale training data with heuristics (Xu et al., 2024), some of which also involve generating a label conditioned on agent trajectories (Murty et al., 2024; Sun et al., 2024). While these methods assume single goal or instruction per trajectory, we propose to mine all achieved goals within the trajectory. More importantly, these data synthesis methods generate training data using a different model from the target LLM agent, which is fine-tuned with the synthesized dataset fixed. By contrast, we continually refresh the relabeled buffer with the same target agent. This on-policy, continuously updated supervision aligns the hindsight distribution with the agent's evolving occupancy, improves coverage where the expert policy has support, and mitigates the drift induced by fixed synthetic datasets, As shown in Section 3.4 and Section 4, this design both tightens the expert-agent gap in analysis and translates to consistent empirical gains over fixed-data synthesis baselines.
2.2 GOAL RELABELING FOR GOAL-CONDITIONED RL
The idea of goal relabeling first proposed in hindsight experience replay (HER) for robotic tasks (Andrychowicz et al., 2017). HER relabels episodes with goals that are actually achieved, enabling efficient learning from sparse, binary rewards. Since then, HER has become workhorse for multi-goal RI. and robotics. Several extensions refine how experiences are selected and weighted, such as countering bias in relabeled experiences by applying more aggressive hindsight rewards (Lanka & Wu, 2018), scheduling replay toward experiences closer to actual goals while maintaining diver- sity (Fang et al., 2019), relabeling trajectories explicitly assemble success from multiple failures (Fang et al., 2018). Later, Cideron et al. (2020) extends this idea to ianguage-condtioone an instruction generator to map the reached states to text-based instructions. While this approach. based on HER, trains the agent on the relabeled experiences with the original offline RL objective, GCSL (Ghosh et al., 2019) instead applies supervised learning to relabeled successes. Nonetheless, complementary robotics line learns from play data by retrospectively specifying goals within unlabeled telcoperation and training goal-conditioned policies (Lynch et al., 2020; Cui et al., 2022).
agents, learning
Our work differs from prior goal-conditioned RL research using goal relabeling techniques in several ways. First, most existing methods assume the agent has direct access to the whole state of the enviromment. In contrast, we address a more complex setting where the agent only receives partial observations, so goal relabeling must be inferred from multiple key observations. Second, methods such as Cideron al. (2020); Ghosh al. (2019) typically relabel only the final goal of failed trajectories. In contrast, we relabel all goals achieved along the trajectory, even when the instructed goal is eventually reached. Third, while HER and its variants train agents using offline RL, we apply SFT and propose two additional techniques to drive the agent towards optimal policy. Finally, our approach focuses on enhancing LLM agents by leveraging the reasoning abilities of LLM for the relabeling process.
More recently, Zhang al. (2023) proposed rewriting queries better align with LLM -eneeeeeedtedd answers for BigBench reasoning tasks (Srivastava et al., 2023), such as logical deduction and word sorting. Although the high-level idea is related, our work focuses on agentic POMDP tasks and introduces distinct techniques.
METHODS
In this section, we present Hindsight Supervised Learning (HSL), new post-training method for LLM agents. We begin with the background and problem formulation, then outline the three main stages of HSL: trajectory collection, relabeling trajectories with auxiliary LLM produce successful demonstrations. and learning from these relabeled demonstrations. We then explain the relabeling process in detail, the core component of HSL, which consists of two steps: goal identification and
3

---

Relabeling process. We auxiliary LLM M to revisit each collected trajectory and identify the actual K goals (K 2 0), represented as instructions {I'...., .L)). that the agent successfully achieved. From these, extract pairs of relabeled instructions and their corresponding trajectories Published as a conference paper at ICLR 2026
Original objective
Hindsight objective
You are in the niddle af room, Looking quickly around cabinet cabinet cabinet cabine:
You are goal-affarenc Decide the agent trajectory.
achioved
Your goal cool same cup and put cabinet
Training Data
Relabeled Data
Action: to mirowave
Achieved goals: (8]
Trajectory
Obs rettrvation: The microwave 1 closed
Relabeling
Relebeder
Action: take mug from coffeemachine
Achioved goals: (P)
Obs ruation: You pick the from the coffeemachine
Goal
Action:
fridge
Achioved goals:
LLM Agent
Observation:
fridge
Figure 2: Left: Overview of the existing training pipeline with HSL.. Right: Example of the relabeling process. The relabeler assigns new goals to the agent's trajectory based on the agent's achievements.
action relevance labeling. Finally, we present two additional learning techniques that further optimize the agent, relevance-baie based loss masking and relabeled demonstration reweighting.
3.1 BACKGROUND
We model an LLM agent as operating in partially observable Markov decision process (POMDP). At each step I, the environment produces an observation o, that reveals only part the underlying state 8t- The transition from the high high-dimensional s, 8t+1 induced by action a, is assumed be unknown. Conditioned on the natural language instruction T, the trajectory history Tt-1 {(o,, a,) >==1 and the new observation Or, the agent seleets an action a₂ that maximizes its learned conditional distribution To(a, (T--1, ot. 1). An reward function Ry (#T, (()) measures how well the final state achieves the goal specified by J. We also assume access to a general textual description of the goal space g. The task is to find the best agent that maximize the expected reward: max E, [R Typically, the agent is trained using offline ground-truth demonstrations. A simple and straightforward way of learning from demonstrations is supervised fine-tuning (SFT). In addition, one may use RL methods such as PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024).
3.2 FRAMEWORK OF HINDSIGHT SUPERVISED LEARNING
In general, our proposed HSL framework adds parallel branch with three key steps: trajectory collection, relabeling, and learning. This branch can be combined with arbitrary existing training pipelines, including SFT and DPO. We describe each step below.
Trajectory collection. At each training step, we sample goals expressed in natural- instructions I Dusin from the training set. The agent then rolls out by following I through interaction with the environment. Specifically, the agent selects an action 0₂ To (a/ oc. 7) and receives the next observation 0t+1- This process continues until the task terminates the maximum step limit is reached. We then collect trajectories (Tr, including the final observation o", for relabeling.
(I', r')x & as successful demonstrations. The relabeling process is described in detail in Section 3.3. The resulting demonstrations are stored in a dynamic buffer D' of size N.
Learning from relabeled demonstrations. We improve the LLM agents with the relabeled demon- strations. At each optimization step, we randomly sample a batch of relabeled demonstrations from D' and calculate C/SL To further improve agent optimality, we propose two techniques: irrelevant action masking and demonstration reweighting. We detail both of them and the concrete loss function CHSL. Section 3.4.
This HSL training pipeline on the relabeled demonstrations D' can be combined with a wide range of existing post-training methods, including SFT and DPO, D Notably, the relabeling process
4

---

Published as a conference paper at ICLR 2026
avoids reliance any ground-trite truth actions or reward signal, allowing the relabeling model to operate in an almost unsupervised manner.
3.3 RELABELING FOR SUCCESSFUL DEMONSTRATION IN HINDSIGHT
Next, we provide detailed explanation of the relabeling process, which is the core component of HSL. It consists of two steps: goal identification and action relevance labeling.
Goal identification. While is hard foresee which goals will be achieved after an action de- reasoning from the resulting observation oc+1 makes the task more manageable. Typically, oc+1 reveals whether ae had any effect on the environment and, if so, what that effect was. One can also infer the long-term impact of a, by examining subsequent observations.
We therefore employ an external LLM M to revisit each trajectory and infer running list of achieved goals represented as instructions T' I, ...., l'K along the trajectory. Although M could be trained or adapted for this task, we instead rely on its zero-shot reasoning ability and prompt to infer the achieved goals in one pass: [....., 1 Mins( (Tr,o"), where inst denotes the system prompt (provided in Appendix C) describing the space of valid goals, and o* is the final observation.
Figure 2 (right) illustrates concrete example: the agent has missed the original goal cool some cup and put in cabinet, but achieved an uninstructed one put a mug in a fridge. In some cases, the agent may have multiple achievements during the trial, which can be independent of one another. the agent fails to achieve any meaningful goal in ty, we discard the trajectory. We then extract each pair of relabeled goal and its corresponding trajectory segment (Ts(E)) (1) successful demonstration, where S(1) is the step where I is newly accomplished.
Action relevance labeling, Since the trajectory Ts(rg) is originally collected for instruction Z, some actions in Ts( may be irrelevant to achieving the relabeled goal l'k. For each pair (Ts(r)), Tg) we reuse M to infer a label (uEI)...((I()) for each action au in Ts(r))) indicating its relevance to Te (k: Mretevance (Ts()), (s(R)+1, Tk),, where relevance is the system prompt shown in Appendix C. The inferred #1:S(k) is subsequently used to train the LLM agents with the relabeled demonstrations.
While existing hindsight generation in RL mostly relabels a single goal on the failed trajec- tory (Cideron et al., 2020; Ghosh et al., 2019), our relabeling model M identifies all valid instructions achieved in т. This design motivated by two reasons. First, even successful trajectories, the agent may accomplish additional tasks at intermediate steps. We find that the LLM agent benefits from learning these uninstructed tasks. Second, HSL is less dependent on explicit reward signals, which are often noisy difficult obtain in LLM agent tasks such web-based agents.
At the end of relabeling, we extract each triplet of (TS(K)) Igg (1:)(1) and append it to the relabeled demonstration buffer D'.
3.4 THEORETICAL ANALYSIS AND LEARNING TECHNIQUES
We would like to analyze how learning from the relabeled demonstrations bridges the gap between the LLM agent and the optimal (expert) policy n² We define the hindsight expert induced by agent trajectories and the relabeling model M as:
Th (a T, o, I') = Pr [1]+1 = a M-l =T, 9//+1 = o, = = [1|
(1)
recall that SM (1') denotes that the first timestep I' is achieved. Accordingly, we define KE-H H as the occupancy coverage ratio of x² and TH, and ōg as the discrepancy between R² and nH:
Pr (T, 0, 1) Pan (T (0);) 8F - [[] 1,,0 1) - n (,1,,,,
A max (T,0.)
KE-H H1
where Par is the occupancy measure of over I).
Our target is to minimize the following discrepancy between agent ny and optimal policy -":
A E(r,o,)) [|] T,O, 1) m o, D]]]
5

---

Published as a conference paper at ICLR 2026
Theorem (Upper Bound of Expert-Agent Discrepancy). Under the setup above,
(0) I 1)
(- (T,)0,, O T,0, (T,01 + Cr dg.
(2)
+C
where CT -
Appendix D provides the detailed proof. The key takeaways of Theorem are as follows. The upper bound on the discrepancy between mee and x* decreases when we apply SFT to both ground-truth demonstrations (first term) and relabeled demonstrations (second term). The gap shrinks further as the occupancy coverage and the optimality of тh improve. practice, this can be achieved by using a stronger relabeling model and, more importantly, by continually updating the relabeled buffer D' so that F is constructed on- on-policy. As the agent succeeds more often, PTи concentrates its mass where Pr- has mass. We also prove a corollary in Appendix E showing that SFT on relabeled demonstrations strictly tightens the expert-agent aggett discrepancy (Equation (2)). Guided by these insights, we train on relabeled demonstrations with SFT and introduce two simple, effective learning techniques.
Demonstration reweighting. Some instructions are inherently casier and thus appear dispropor- tionately in D', which may bias the agent toward solving only trivial tasks, leading bad coverage R. To mitigate this issue, we sample demonstrations E D' with a weight wy that prioritizes learning from trajectories that solve more difficult tasks more optimally. Concretely, wa is calculated by: T )a rid, where na is the number of actions associated with the relevance label e, 1, and Ta is the total number of actions in d. The ratio T. reflects how optimally the task is solved, while nd serves as proxy for task difficulty. The hyperparameter a balances these two factors.
Irrelevant action masking. Because the original trajectories are collected under instructions different from the relabeled I', naively applying SFT all actions in each trajectory may imitate a sub-optimal hindsight expert with large ōg. Therefore, we mask out the loss for actions labeled irrelevant (2+ 0). And hence, the training objective on relabeled demonstrations is defined
T
cp' E. (T.
Σ
24 log Po(aelt-
T
w/a where P(d') EeED wa Suggested by Theorem further incorporate the SFT loss ground-truth demonstrations Duran using mixture weight X, and thus the resulting learning objective is:
1 T Σ log Po Pa (:- 0t,
C -C (11-))
Similar many existing methods of post-training LLM agents (Song al., 2024), HSL requires ground-truth demonstrations for stable learning. However, as we will show later in the experiment, HSL sample efficient and achieves performance that matches or exceeds baseline methods while using much less D
4 EXPERIMENT
4.1 SETUP
Tasks and Evaluation Metrics. We evaluate on three well-adopted agentic benchmarks with different levels of task diversity. ALFWorld (Shridhar et al., 2020) is an embodied agent benchmark which the LLM agent navigates rooms and completes household tasks to satisfy a natural language goal (e.g.. "put some vase in safe"). The evaluation set consists of a split (new tasks within scenes present in the training set) and an unseen split (rooms or layouts absent from the training set). A task is considered successful only if all goal conditions report the average Success Rate. PlanCraft (Dagan et al., 2024) is a Minecraft crafting benchmark that tests agent's planning in simplified crafting UI with text-only observations. Given natural
met. Following Song al. (2024),

---

Published as a conference paper at ICLR 2026
language goal item, initial inventory and the recipe of the goal item, the agent must craft the target by moving items between inventory slots and the crafting grid and by smelting items as needed. We exclude the impossible subset and report the average Success Rate. WebShop (Yao et al., 2022) is a web agent benchmark in which an agent follows a natural language instruction to navigate a simulated e-commerce site and purchase product. Each episode ends upon purchase and returns dense reward r E [0, 1J based on the type matching, the coverage of the requested attributes/options, and the price constraint. We report Task Score (100x average reward). ALFWorld features longer horizons (T = 40), diverse set of valid goals, and multiple achievable goals per episode. In contrast, PlanCraft and WebShop each consist of a single task type with shorter horizons (T 30 and T 10, respectively). Moreover, each trajectory in WebShop can achieve only one goal. Therefore, we expect HSL to yield larger gains on ALFWorld, while PlanCraft and WebShop primarily test its robustness in narrower task space.
Implementation. We employ Llama-3.2-1B BB (Dubey et al., 2024) agent model and Llama- 3.3-70B (Dubey et al., 2024) as a relabeling model. We set X = 0.3 and a = 0.8, and maintain the relabeled dataset D' queue of size 100. After each optimization step, HSL collects and relabels b = 18 trajectories, which are appended to D' while the oldest entries are removed once the limit is reached. Additional implementation details are provided in Appendix F.
Baselines. We evaluate HSL as an add-on to SFT and Exploration- based Trajectory Optimization (ETO) (Song et al., 2024). SFT fine-tunes the agent model on ground-truth demonstrations from the original datasets, while ETO applies DPO (Rafailov et al., 2023) to agent tasks, using ground- truth demos preferred samples and agent-generated failures dispreferred samples. We also include SELFIMIT (Shi et al., 2023), which fine-tunes the agent on its own successful trajectories. To demonstrate that the gains stem from our relabeling approach rather than merely using powerful external LLM, we include baseline methods that also use Llama-3, 3.3-7 Concretely, REACT (Yao SEREAVIORCIOOS
al., 2023) directly applies Llama-3.3-70B for reasoning and action selection. uses -Llama-3.3-700 to synthesize demonstrations and then fine-tunes the agent on the union of ground-truth and synthetic data, following Zeng et al. (2024). BAGEL (Murty et al., 2024) is an offline data synthesis method that generates trajectories with the agent and uses Llama-3.3-70B to label and filter them; subsequently, the agent is fine-tuned on the synthesized and ground-trut data.
4.2 MAIN RESULTS
Table 1 presents the performance of our proposed HSL and prior related methods across all three benchmarks. HSL improves both SFT and DPO significantly and consistently with much larger gains on ALFWorld, which involves longer tasks and a larger set of valid goals. In addition, the improvement on WebShop smaller than that on PlanCraft, likely because WebShop allows only one achievable goal per trajectory, whereas L.M agents can craft multiple items within a single task in PlanCraft. Although REACT and BELAVIORCL.ONE use the external LLM (L.ama-3.3-70BB to enhance the agent, they perform substantially worse than SFT+HSL, validating our claim that predicting actions for an agentic task itself is more challenging than relabeling. BAGEL also improves SFT on ALFWorld but still underperforms compared to SFT+HSL, showing the importance of continuously updating the relabeled demonstrations for the target agent during training. This finding echoes analysis in Section 3.4, which suggests that the relabeling process benefit from the evolved agent. Finally, SELFIMIT fails to improve upon SFT, underscoring the importance of mining all successful demonstrations, even for uninstructed goals. To further verify the robustness of our results, we rerun all fine-tuning experiments on ALFWorld and WebShop using different random seeds and report the results in Appendix G.
Sample efficiency. To assess the sample efficiency of HSL, we evaluate how the number of ground- truth demonstrations affects the performance of HSL and the post-training baselines on ALFWorld and WebShop. As shown in Figure 3, adding HSL to either SFT and DPO consistently and substantially increases success rates ALFWorld, given the same ground-truth data budget as the SFT and DPO baselines. Notably, the improvement is particularly larger in the Unseen split, where HSL nearly doubles SFT 800 demonstrations and doubles DPO at 1,600 demonstrations. This again highlights that HSL facilitates generalization. More importantly, HSL, which uses less than one quarter of the ground-truth data, surpasses SFT-only DPO-only models trained on the full dataset.
7

---

Published as a conference paper at ICLR 2026
ALFWorld unseen
Method
PlanCraft
WehShop
REACT (Yao et al., 2023)
33.57
20,90
59.17
48.37
BEHAVIORCLOnE (Zeng et al., 2024)
83.57
88.81
64.38
65.19
BAGEL (Murty et al., 2024)
84.29
91.79
69.17
62.18
SELFIMIT (Shi al., 2023)
84.29
76.87
56.25
58.37
SFT
82.14
78.36
70,00
63.81
DPO (Song et al., 2024)
85.71
82.84
71.25
69.54
SFT+HSL (Ours)
93.57
97.76
75.12
66.97
DPO+HSL (Ours)
92.86
94.78
75.42
70.52
Table 1: Performance on ALFWorld, PlanCraft, and WebShop.
ALrWorld (Seen)
ALF World (Unseen)
WebShop
94.5%
40.%
100%
98.4%
00%
R00%
857%
78.5
62.5
Rowen
7N.0%
10.0%
65.5
645
70%
47.
70%
60%-
635
00.4%
9 D00-401
DED
3FT-HSL DPOHIBL
300
1600
Full (-3200)
800
1600
Full
1000
Fvll
(-200))
(-1800)
Ghound-taal dememetnaiinn
Ground-trul demarratraiuua
Goaad--nuth danemiintions
Figure 3: Sample efficiency of HSL with different post-training methods.
For example, DPO+HSL reaches 92.5% with just 800 ground-truth demonstrations on ALFWorld (Unseen), whereas DPO attains only 82.8% even with more than 3,200 ones. On WebShop. the improvements are smaller but follow trends similar to those in ALFWorld across data sizes and baselines. This suggests that HSL is particularly effective for open-ended tasks with diverse goal types, such as ALFWorld, where agents are more likely to "accidentally" accomplish uninstructed tasks and therefore benefit more from relabeling.
4.3 ABLATION STUDIES
We conduct ablation studies to quantify the contribution each component and technique in HSL. In particular, evaluate DELLABEIFFAIURRE which only relabels the final achieved goal failed trajectories and resembles existing hindsight generation methods the RL. literature (Cideron et al., 2020; Ghosh al., 2019); UNIWEIGHT. which samples relabeled demonstrations uniformly without any reweighting; and NOMASK, which does not apply the action-irrelevant masking.
Figure 4 presents all variants, including the complete model SFT+HSL trained with different numbers of ground truth demonstrations on ALFWorld. Removing any component reduces performance, though the drop from removing demonstration reweighting (UN(WEIGHT) smaller in the Seen split. In the Unseen split, UNIWEIGHT is markedly worse, which suggests that increasing expert-hindsight occupancy coverage in Theorem facilitates generalization. With the fewest ground truth demon- strations, NOMASK shows the most significant decline, likely because a weaker base agent executes many back and forth actions within trial, for example repeatedly visiting a receptacle, which are not helpful for the relabeled goal. This highlights the need for stronger hindsight expert who proposes more optimal actions. RELABEL FAILUEE is consistently and substantially worse than the full model. In particular, it does not benefit from additional ground truth demonstrations, mainly because more potent base agent trained with more demonstrations produces fewer failed trajectories. verifies our decision to leverage all the intermediate goals that have been achieved.
8

---

Published as a conference paper at ICLR 2026
Seen
Unseen
13.0%
AL.%
97.4%
90%
X0/7
85.1%
50.1%
Suices nate
50%
80%
70%
72.4%
67.%%
60%
60%
2.5%%
Relasthaber
SFT ISSL
500
1600
Full (-3200)
800
1600
Fill
(-3200)
Ground trath Cemonnsrattonss
Ground touth demonstrations
Figure 4: Ablation studies on ALFWorld.
AI World Goal Type Distribution and HSI. Performance (#groond-rmth=a80o)
40
Eval
Goal Percentage (%6
10
Success
0.00
0,20
0.40
1.00
put
clean
heat
cool
examine
put-two
Goal Type
Figure 5: Top: Distribution of the goal types relabeled and ground-truth demonstrations. Bottom: Success rate of SFT+HSL across different goal types.
4.4 ANALYSIS: UNDERSTANDING THE EFFECTIVENESS AND LIMITATIONS OF HSL
To gain a deeper understanding of the relabeled demonstrations and how learning from them helps, conduct quantitative and qualitative analyses on ALFWorld, which includes diverse goal types. Because the model trained on the full dataset achieves near-perfect success rates across all tasks in ALFWorld, we focus on SFT+HSL trained with only 800 ground-truth demonstrations. First, we compare the distribution of goal types in the relabeled demonstra- tions with that of the ground-truth data. As shown in Figure 5, the most augmented goal in the relabeled set is put, and the agent correspondingly achieves perfect performance put. For clean, heat, and cool, the pro- portions in the relabeled data close those in the ground truth, and the agent achieves success rates above 80%. In trast, exami ne and put-te two are under- represented in the relabeled demos. While put and put- www appear at similar rates the ground truth, the relabeled demos contain roughly twice as many put as pwt-two. Moreover, examine accounts for less than 5% of the relabeled set. This pattern likely arises because put a sub- task of pwt-two, and examine is the most tongetailed goal in the training data (under 10%), such that the agent rarely
1.0
100.0%
100.0% 100.0%
81.8%
71.8%
721.7%
Accuracy
0.6
0.2
put
put-two
Goal Type
Figure 6: Accuracy of relabeling per goal type.
9

---

Published as a conference paper at ICLR 2026
Example Relabeled Demonstrations (Cornec)
Exumple Relabrled Demonstrations (Incurrect)
Relabeled Goal: put
clean cloth in bailet
Relabeled Goal: put opal Lonato in Fridge
Stap
Step
Action: Lake cloth From countertop observation: You pick the cloth 2 toon the countertop 2-
Action: Lake sonata from counteztop observation: You pick up the tonato trom the countert
Step 5
Step 31
Action:
cloth sinkbasin the sinkbetin
Action:
observation:
using
cridge open. you see a place 2, and potato 3, Relevant to Goal: True
Relevant to Goal: True
Step 71che lest stop)
step (2hh stcp)
Action: put cloth ia/ca toilet 1 observation: You put the cloth in/on the toilet
Action: cool tomato with tringe 1 observation: You cool the torato 1 using fridgs
Two examples (correct and incorrect) of relabeled demonstrations in ALFWorld. Each shows agent trajectories with the corresponding outputs (relabeled goals and action relevance) by relabeler. In the right example, the relabeler makes mistake: the agent cools tomato but never places in the fridge, so only a subtask the relabeled goal satisfied.
Figure 7:
achieves it by chance. As result, the agent trained with the relabeled demos records its low- est success on examine ne at 54.85%, and its performance put-two lags notably behind put.
We further assess the quality of the relabeled data. We randomly sample 200 relabeled demonstrations and manually verify their labels. Of these, 180 are correct, yielding accuracy of 90%, which helps explain the strong performance of the HSL-trained LLM agent. Figure shows the accuracy for each goal type. Notably, the relabeling is performed in zero-shot setting, yet is substantially more accurate than predicting actions with in-context examples (REACT), This observation further supports our hypothesis that relabeling trajectories in hindsight is easier than executing the task itself. Two illustrative cases are shown in Figure 7.
Takeaway
Taken together with our theoretical (Section 3.4) and empirical results, these findings indicate that HSL. narrows the gap between the agent and the expert policy in sample-efficient manner by transforming noisy agent trajectories into high-quality demonstrations from a hindsight expert. Nevertheless, the magnitude of the improvement depends on the hindsight expert's optimality and coverage, as well as task characteristics such as diversity and the number of tasks per trajectory.
CONCLUSION
We present Hindsight Supervised Learning (HSL), a sample-efficient learning framework that relabels agent trajectories with goals actually achieved and learns from these relabeled demonstrations. By incorporating irrelevant-action masking and reweighting, HSI. enhances both the coverage and quality of relabeled data. Comprehensive experiments on three agentic benchmarks, ALFWorld, PlanCraft and WebShop, show that HSL consistently boosts SFT and DPO while reducing reliance on trouh demonstrations, with larger gains long- horihor tasks with diverse goal spaces. These findings show that hindsight relabeling is an efficient way to leverage LLMs' reasoning capabilities while avoiding the need to model environment dynamics. As future work, plan to extend this framework to more open-ended environments and multimodal agents. Nevertheless. our work still has limitations, which discuss in Appendix B.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
We thank Tong Sun for discussions, as well as the anonymous reviewers and ACs for their feedback and comments.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv e-prints, pp. arXiv-2108. 2021.
Maxime Chevalier-ooisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv:1810.08272. 2018.
Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pictquin. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 225-232. IEFE, 2020.
Zichen Jeff Cui, Yibin Wang, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:22 10.0007777 2022.
Jeff Da, Clinton Wang, Xiang Deng, Yuntao Ma, Nikhil Barhate, and Sean Hendryx. Agent- rlvr: Training software engineering agents via guidance and environment rewards, 2025. URL https://arxiv/ org/ wbs./////////aaoooaaoooaaasaaaee 2506. 11425.
Gautier Dagan, Frank Keller, and Alex Lascarides. Plancraft: an evaluation dataset for planning with IIm agents. arXiv preprint arXiv:2412.21033 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang, Dher: Hindsight experience replay for dynamic goals. In International Conference on Learning Representations, 2018.
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-uuided hindsight experience replay. Advances in neural information processing systems, 32, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019.
Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: survey on milm-based agents for computer, phone and browser use. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7436-7465, 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
11

---

Published as a conference paper at ICLR 2026
Sameera Lanka and Tianfu Wu. Archer: Aggressive rewards to counter bias in hindsight experience 2018.
replay. arXiv preprint
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking Ilms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang. Min Peng, and Wotao Yin. Symagent: A neural- symbol-symbolic self-learning agent framework for complex reasoning over knowledge graphs. In Proceedings of the ACM Web Conference 2025, pp. 98-108, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17Xiv:1711.05100 2017.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113-1132. Pmlr, 2020.
Shikhar Murty, Christopher D Manning, Peter Shaw, Mandar Joshi, and Kenton Lee. Bagel: Boot- strapping agents by guiding exploration with language. In International Conference on Machine Learning, pp. 36894-36910. PMLR, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 11553728-53741, 2023.
Stephane Ross and J Andrew Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010.
Stephane Ross, Geoffrey Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reason- ing in open language models. arXiv preprint 202.::240,.03300, 2024.
Zijing Shi, Yunqiu Xu, Meng Fang, and Ling Chen. Self-imitation learning for action generation in text-based games. In EACL 2023-17th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, 2023.
Mohit Shridhar, Xingdi Yuan, Alex-dle Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766 2020.
Yifan Song. Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration- trajectory optimization of Imm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 7584- 7600, 2024.
Aarohi Srivastava, Abhinay Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazary, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone. Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew Kyle Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh
12

---

Published as a conference paper at ICLR 2026
Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakay, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argucta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burchh Christopher Waites, Christian Voigt. Christo- pher D Manning. Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Ho Lee, Dylan Schrader, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolà, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong. Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germän Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch- Lopez. Gregor Betz, Guy Gur-Ari. Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hamnanch Hajishirzi, Harsh Mehta. Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang. Liam Dugan, Lianhui Qin, Lidia Contreras-Ochance
Louis-Phillipe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Ciiveros-Colón, Luke Metz, Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramirez-Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michat Swedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski. Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ra- mon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian. Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Ham- dan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammie Debnath,
Evans,
13

---

Published as a conference paper at ICLR 2026
Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephamie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tat- sunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Pad- makumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Sophie Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, 7achary Seid, Zhuoye Zhao, Zijian Wang, 7ijie J. Wang. Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL ftttps://openreeee
net/ forum? f d
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.197235 2024.
Ada Define Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Span- dana Gella, Karolina Staáczak, and Siva Reddy. Safearena: Evaluating the safety of autonomous web agents. arXiv preprint arXiv:2503. 04957, 2025.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao. Toh J Hua, Zhoujun Cheng. Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks real computer environments. Advances Neural Information Processing Systems, 37:52040-52088 52094, 2024.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch every step! Ilm agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1556-1572. 2024.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang. Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. arXiv preprint arXiv:2412.09603 2024.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744-20757, 2022.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting language models. In International Conference on Learning Representations (ICLR), 2023.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lss. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3053-3000 5533-377,200 2024.
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E. Gonzalez. The wisdom of hindsight makes language models better instruction followers. In Proceedings of the 40th International Conference Machine Learning, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web PMLR,
agent, if grounded. In International Conference Machine Learning, pp. 2024.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tranyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2026
Yuchen Zhuang, Yue Yu, Kuan Wang. Haotian Sun, and Chao Zhang. Toolqa: A dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 36: 50117-50143, 2023.
15

---

Published as a conference paper at ICLR 2026
A LANGUAGE MODEL USAGE STATEMENT
During the preparation of the manuscript, we have used LLM to polish the draft and fix grammar issues. LLM did not participate in the ideation of the project, such as problem formulation and methodology.
B LIMITATIONS
Limitations of the Proposed Method. Our method, in its current form, uses an external LLM for zero-shot relabeling without fine-tuning, so it depends on the reasoning ability of powerful models. This design lets us leverage state-of-the-art proprictary LLMs, and our experiments show that using them for relabeling yields markedly better final performance than asking the same models to solve the tasks directly. For more complex tasks, however, the relabeling model may require further adaptation and optimization. Jointly learning the relabeling model and the LLM agent is left for future work.
Second, we label each action in a trajectory as either relevant or not to the relabeled goal, which assumes a black-and-whic notion of relevance. In complex settings action can be relevant yet suboptimal for the goal. For example, purchasing multiple items one by one is less efficient than placing them all in cart and paying once. A more fine-grained notion of relevance may therefore be needed, which would entail fine-tuning the relabeling model in role akin to reward or value function.
Third, as shown in the experiments, although our demonstration reweighting improves agent per- formance and many goal types are well represented in the relabeled demonstrations, the resulting distribution does not fully match the goal-type distribution in the original dataset. Our method chiefly exploits exploration in a sample-efficient way, so further work needed increase the diversity of exploration.
Fourth, while HSL is sample-efficient and reduces reliance on ground truth demonstrations, unavoidably increases training compute because it uses external relabeling model. The inference cost stays the same. This mirrors a broader trend in LLM agents: investing more compute during training (GPU hours and LLM tokens) improve LLM agent performance (Lightman al., 2023; Song et al., 2024). We believe that this a valid contribution since collecting human demonstrations for agent tasks is costly and hard to scale.
Last, although the relabeling process and learning from relabeled demonstrations are unsupervised or self-supervised, HSL still requires ground truth demonstrations to stabilize training by aligning the agent's support with the expert's. s. This limitation is shared by many other post-training methods. Future work should develop stronger exploration strategies that enable HSL without any ground-truth demonstrations.
Limitations of Experiments. We use Llama-3 as the base LLM agent and evaluate on three standard agentic benchmarks, ALFWorld, PlanCraft and WebShop. Although conduct comprehensive ablations and analyses, we do not cover stronger base LLMs or additional agentic tasks. It is mainly due the limit of time and computational resource as well as lack of lusk-specific ground-trute demonstrations. For many GUI-agent benchmarks, for instance, either an interactive environment or ground-trute truth demonstrations are missing. When they are available, the environments are often resource-intensive to set up.
Second, as discussed in the experiments, the gains from HSL are smaller for short-horizon tasks with narrow goal space. We expect larger benefits in more open-ended settings with diverse goals, where LLM agents must adapt to broader task distributions.
Third, the tasks adopted in experiments contains predefined goal spaces, enabling the relabeling model to determine whether there is any valid goal achieved and what it is. This setting is common in existing LLM agent benchmarks, but it may not hold in open-world world environments. Future work can explore relabeling without a predefined goal space, such as deriving goal space from a set of user instructions based heuristics.
Last, most of existing LLM agent benchmarks, such as the ones adopted in our experiments, assumes all goals are equally valuable and safe. However, in complex real-world scenarios, the unintended
16

---

Published as a conference paper at ICLR 2026
goals may have low value for the users even be malicious. To address these issues, future work can score the utility of unintended goals by comparing it with the user instructions, and adopt safeguard models to filter out or penalize the malicious goals.
C PROMPTS FOR RELABELING PROCESS
Figure and Figure 12 show the goal relabeling prompts for ALFWorld and WebShop, respectively. Figure 9 and Figure 13 are the action relevance labeling prompts for ALFWorld and WebShop. Because the crafted item in PlanCraft can be read directly from the returned text-based observation, we do not use the relabeling LLM to infer the crafted item. Instead, we use to infer another part of the input: the recipe used to craft the goal item. Figure 10 presents the corresponding prompt. Figure 11 is the action relevance labeling prompt for PlanCraft.
D PROOF OF THEOREM
Preliminaries and notation. For simplicity of notation, we let v; = (T--1, 0z). a policy x specifies conditional action distribution x(a xr, 7). We total variation distance (V(p,)) / blpp æ/l and KL-divergence (xll)) = plog q to measure distance between action distributions. We assume prior distribution p()) over instructions and a finite horizon T (the maximum number of steps in each episode). The hinite-horizon occupancy (joint measure over (æ, I)) of is
1 1 Σ T
Pa(æ, I) - p(T)
Pr[a.
(5)
The hindsight expert induced by relabeled goal I' and the trajectories generated by LLM agent ти is
E1=1 PI It = x, a: a. (T' =
TH(a x., 1')
(6)
PTou -.=xx (!') -
where SM (I') is the first time step at which I' becomes newly satisfied according the relabeling model M. We assume every (z, I) pair that the optimal expert policy can visit with nonzero probability is also covered by the hindsight expert, and define the expert-hindsight occupancy coverage ratio
Pu- (x, I) (x.1) PTu (x,
KE-H A max
E (1)
The agent-expert discrepancy
TV( x, 12,1,, x' (1112111) 12,1)]
E
(8)
A finite-horizon simulation inequality and the constant Cr. We measure occupancy mismatch using the following inequality, which a finite-horizon version of standard error propagation bounds in imitation learning.
Lemma 1 (finite-horizon simulation inequality). For any policies and measurable g X x [0, 1],
Ep [9] [ell < Cr De [TV( (TVeit xx 1) x, n)). Cr 2(T
Proof sketch. Let (- denote the law of X, under w. A standard coupling/perturbation argument (cf. the finite-horizon analyses in Ross & Bagnell (2010); Ross et al. (2011)) yields, for each t > 1,
TV D2 1, <
Ex Ex. TV((-- In
X., P( Xx,
Averaging over t = 1,... T, multiplying by p(I) and symmetrizing gives equation 9 with Cy 2(T - 1))
17

---

Published as a conference paper at ICLR 2026
Proof of Theorem 1. Let I) & TV(To(: x, x. x. I)) E [0, 1]. Add and subtract
pp [[]:
A = pf + (Eprp [f] - Bpp lf))
(10)
Applying Lemma 1 to g f. (w, = (To, x²), (0) S Pp. + CT Ep (V
(11)
Use the triangle inequality,
(T x²) S TV(To, Tn) + "V(Hn,*');
(12)
and denote åg - (TV '²' Then
(0) If] Ep (TTTTTT)) + Cr
(13)
Change measure using coverage equation 7:
pp. (TV,,, Th)] & KE- Pog [V((To:
(14)
Apply Pinsker inequality and Jensen's inequality:
E p.. [f] = Ep.. (IV(TT) S / (
(15)
E, P-y TV To, mF) < V Er [RII(TTN|TTO]]]
(16)
Combining equation 13 equation 16 yields:
A()) S V Ba. (TL 2. To(- 22, [)]
(17)
+ CT CTEE-HY H B(r. (Prt,)) (TM (- x. I) no( 2x, [)] + dr
Finally, v Y=+ + 1
1 for 2 converts the first term to a linear KI., and the additive constant
(independent of 0) can be absorbed, giving the bound in Theorem 1:
(0) < ( (xL(TT' ( [TT)] + Cr KEEH V Be (PK((T((TTT) + Cr dg. (18)
E COROLLARY TO THEOREM 1: HSL STRICTLY TIGHTENS THE DISCREPANCY BOUND
Let
(E( = (x DRI( (- xr 1) TT( I [))],
and
LH (e |KL[TTH( (: xr I) me(: x. [)]]
and define
Cr KE V 2
B(0) & (E( + 7 Lu Cr Ōg. By Theorem (Eq. equation 2), for all 0 we have A()) B(0).
linear surrogate for training. The square-root term admits standard Young foung-type relaxation: for any e > 0,
22 E
WC CH S
+
CH-
(19)
2s
2
Hence minimizing (0) + 3LH with B c/2 minimizes additive upper bound on B(0) up to 9-independent constant.
Corollary (Strict dominance at equal expert fit). Fix 01, O2 with Lx(01) (o) If (01) L.r then B(1)) < B(0z) and consequently
<
(();) < B(O)) < B(02).
In words: among policies that fit the expert demonstrations equally well, the one that additionally fits
the relabeled demonstrations achieves a strictly tighter upper bound on the expert-aged agent discrepancy.
Proof. The map u + Vu is strictly increasing on [0, 20), so B(O,) - BB(2) alV / (o)( O.
18

---

Published as a conference paper at ICLR 2026
F IMPLEMENTATION DETAILS
We employed Llama3.2-1b (Dubey et al., 2024) as the agent model and Llama3.3-706 2 as the relabeling model. Following Song et al. (2024), adopted the same hyperparameters, setting the learning rate to 2 x 10-s, the batch size to 32, and using the AdamW W optimizer (Loshchilov & Hutter, 2017). All tine-tuning runs lasted for epochs. Because HSL naturally incorporates SFT, we first trained Llama3.2 2-1b on ground- demonstrations with SFT, then fine-tuned it with the objective Equation (4) to obtain SFT+HSL. For DPO+HSI,, we followed the same approach of ETO for combining SFT and DPO objectives (Song et al., 2024): we first fine-tuned the agent with HSI., then continued fine-tuning with DPO, All fine-tuning experiments on ALFWorldd and WebShop were run three times with random seeds 42, 123, and 2026. Experiments seed 42. Each experiment used cight NVIDIA A A100 GPUs and finished within one day. During each HSL training step, rollout collection, relabeling, and loss computation with optimization took average 11.53, 62.03, and 24.03 seconds, respectively.
PlanCraft were run with random
G RESULTS ACROSS MULTIPLE RANDOM SEEDS
Table 2 reports results of all fine-tuning experiments across three random seeds: 42, 123 and 2026.
World
Method
WebShop
seen
unscen
BEHAVIORCLOnE (Zeng et ul., 2024)
80.72 +:2477
79.85 48.31
61.68 4:2.9
BAGEL (Murty et ul., 2024)
88.09 +3.93
87.06
+:486
63.46
#1.13
SELFIMIT (Shi al., 2023)
69.05 +5.02
57.46
11010
62.77 +0.25
SFT
79.52 +3.38
75.87 +4.31
62.61 #1.04
DPO (Song al., 2024)
84.52
81.09
69.27 1.040
SFT+HSL (Ours)
93.57 1:0 00
96.27
12.11
66.48
4:049
DPO+HSL (Ours)
92.86
1:0.00
95.27
10.43
70.28
1:0.22
Table 2: Performance on ALFWorld and WebShop, averaged across multiple random seeds.
"htrps://huugiinggfccee co/meta 1ama / 1. aamaaannnn.m 3.22 1-- Instruct
hhttpss//huggiigggfaas cote..oo/mttt coommta-- amma 1.mam-3.3 m..33.3-- 708- Instruct
19

---

Published as a conference paper at ICLR 2026
Goal inference prompt for ALFWorld
You are goal-inference assistant for AlfWorld. Given sequence of Actions and Observations, track the agent's Location and Inventory after each step, then derive and record any goals from the templates below that have been completed. A trajectory may achieve multiple goals nonc.
1. After each Action/Obserration pair:
(I) update the agent's 's Location and Inventory. Invalid actions (e.g., using or dropping object the agent doesn't i have) leave both unchanged. You should determine if the action has any effect based on given Ohservation!
(2) Then check whether any of the goal templates have been satisfied by the agent's actions up to that point. When a goal is achieved, add it to the running list of Reached goal values and keep that list for subsequent steps.
(3) Do summarise or skip any steps, even the observation identical to previous ones.
2. Hide all object IDs; refer to objects and receptacles only by their type names (c.g. "mug", "knife", "drawer"), never by numeric alphanumerie identifiers.
3. Inventory format: list each inventory item by type, repeating names for duplicates (e.g. [mug. knife, knife]).
4. At the end, output Final goal: followed by the list of all goals achicved (e.g. [goalA, goalB)). If no goals were achieved, set Final goal: a brief description the agent's behaviour.
Allowed goal templates (with their intended behaviours):
put [ohject] in [receptacle] put some [object] on [receptacle] - Pick Place: agent must find an object of the desired type, pick it up, find the correct location to place it, and put it down there.
look [object] under the [lamp]/ / examine [object] with the [lamp] Examine Light: the agent must find an object of the desired type, locate and turn on a light source with the desired object in-hand
put clean [object] [receptacle] / clean some [object] and put [receptacle] Clean Place: the agent must find an object the desired type, pick it up. go to a sink or a basin, wash the object by turning on the faucet, then find the correct location to place it. and put it down there.
put hot [object] in [receptacle] / heat some [object] and put in [receptaclc] Heat & Place: the agent must find an object of the desired type, pick up. go to a microwave, heat the object turning on the microwave, then find the correct location to place it, and put it down there.
put cool [object] in [receptacle] / cool some [object] and put in [receptacle] Cool & Place: agent must find an object the desired type, pick it up, go to a fridge, put the object inside the fridge and cool it, then find the correct location to place it, and put it down there.
put [object] in [receptacle] / find two [object] and put them in [receptacle] Pick Two & Place: the agent must find an object the desired type, pick up, find correct location place it, put it down there, then look for another object of the desired type, pick it up, return to previous location, and put it down there with the other object.
Output format (exactly): Return a single JSON list. Each element of the should JSON object with the following structure for each step:
"step": <number>,
"act o "<action>", "observation"; "cobservation>",
"reasoning"; "canalyze whether the action has any effect based on the given observation, is yes what is that>", "Location": "<location>",
inventory;; ["<item>", "<item>", ...l, reached_goals": ["<goalA>", "<goalB>", ...]
Figure 8: Goal inference prompt for ALFWorld
20

---

Published as a conference paper at ICLR 2026
Action relevance labeling prompt for ALFWorld
You are step-relevance classifier for AlfWorld. Given goal and sequence actions, observations, with location and inventory derived by model, decide for each step whether it necessary to achieving the goal. A step is "relevant" if it is a necessary prerequisite or directly advances toward the goal; actions that involve the wrong objects, revisit unrelated locations, or otherwise do not help achieve the goal are "irrelevant". Some goals may require exploration the early stage find the relevant objects, and intermediate tasks such as heating, cooling, cleaning, examining, or finding an object. For each step. provide brief chain thought to explain how you judged the step relevant or irrelevant. Do not summarise or skip any steps, even if the observation is identical to previous ones.
Output format (exactly): Retum single JSON array. For each step, output object with these fields:
stepp: <number>,
"action": "sprovided action>",
"observation": sprovided observation>", "Location"; "sprovided location>",
" "Laventory"; ["<item>", 1,
"reasoning"; "<analyze the effect and function of the action, then analyze whether it's necessary to achieving the goal>" "is
"yes" | "no",
Figure 9: Action relevance labeling prompt for ALFWorld
Goal inference prompt for PlanCraft
You are recipe-inference assistant for MineCraft. Given crafted item, its recipes and sequence of actions, the change of slots (inventory, crafting grid, output) effects of the action, derive which recipe is actually used for crafting the item.
Slot legend:
[0] the output slot where the crafted result appears.
[A1]-[33] are the 3 crafting grid (rows A/B/C, columns 1-3).
[T]]-[T36] regular inventory slots used for storage.
Task:
Compare the actions and items in crafting grid in the last several steps the given recipes. For shapeless items, ignore coordinates, match only the multiset of required ingredients and counts, with no extras. For shaped items. match the relative pattern to translation; coordinates shown just valid placement, and do not rotate/mirror unless explicitly allowed. Smelting: crafting-giid locations are irrelevant urelevant-ideniffy from the action sequence and match the smelt input(s) + output.
If there only available recipe, copy that directly.
If there are more than one recipes align with the actions, select the first one.
Output format (exactly):
Return exactly one ISON object:
"reasoning": "canalyze how each recipe fits the action and items in crafting grid>", "used boseddrecipeiide _d":: "crecipe id>", resc__e "<copy the recipe heres"
Figure 10: Goal inference prompt for PlanCraft
21

---

Published as a conference paper at ICLR 2026
Action relevance labeling prompt PlanCraft
You are step-relevance classifier for Minecraft. Given crafted item, the chosen recipe, and sequence of actions with slot deltas (inventory, crafting grid, output), decide for cach step whether the action is NECESSARY to craft that item. Slot legend:
[0] = output slot (crafted result appears here; nothing can moved into [0]).
[A]]-[C3] 3x3 crafting grid.
[[]]-[136] = inventory.
Location rules:
Shapeless:
ignore coordinates; match only the multisct of required ingredients and counts (no
extras).
Shaped: match the relative pattern up translation (no rotate/mirror unless stated).
Smelting: grid coordinates are irrelevant; match input-toutput.
Decision procedure (deterministic):
Identify the successful craft: the first step where the target's count increases (or appears in [O]); include any subsequent move that transfers the target from [0] to inventory as NECESSARY.
Back-preee proca-prpaaate dependencies from craft: Mark craft/smelt step itself as NECESSARY. For each grid cell whose contents were consumed by that craft, mark as NECESSARY the last move that placed that consumed item into that cell (only the last placement before consumption). Recursively mark NECESSARY any earlier steps that produced intermediates (craft/smelt moves from [0]) which were later consumed (directly or further intermediates) this chain. Unblockers are NECESSARY: a move that removes/rolocates a wrong or extra item from a cell required by the final layout so that the required item can occupy it. The mistaken placement itself is NOT necessary.
Everything else NOT necessary: redundant placements beyond needed counts, inventory shuffles not used by the chain, unrelated crafts/smelts, no-ops, attempts to move/smelt into [0], or actions that don't 't match the observed slot delta.
Task: For EVERY step, output label according to the procedure above. Output format (exactly): Return SINGLE JSON list for each step. Each element:
[[
"step": <number>,
"action": "<provided action>".
"slot_upe "slot_update": "ederive update of inventory, crafting ggidd, output slots>", "reasoning": "<explain the step's effect and why it is or isn't in the dependency chain>", "is relevant"; "yes"! "no"
}...
Figure 11: Action relevance labeling prompt for PlanCraft
22

---

Published as a conference paper at ICLR 2026
Goal inference prompt for WebShop
You are goal-inference assistant for Webshop. Given a full trajectory with Actions (search or click) and Observations (page text, system message), infer what user's intended and also succesfully purchased:
product (str) generic product type; ignore brand/manufaccture and DON'T copy the full title. Prefer the head noun from category/title
attributes (list) short descriptive phrases from title/eecrrition; not clickable (e.g., ["portable", "mid-century style"]). NOT brand.
options clicked (dict) the literal option texts the agent clicked this product, in click order (e.g-, <size> <color> <quantity>)- Do not invent labels or pairs; just copy the clicked option strings. quantity (strinull) the chosen quantity it was explicitly clicked: otherwise 1.
price (number) the price of the selected product/variant
Derive procedure:
1. Extract the exact "query" string(s) from search actions.
2. Derive 'selected' (extracting 'product", 'attributes', 'options', 'quantity", and "price") from clicks
+ final product page. Note: clicking an non-existing product select nothing!
3. 'selected price': copy the exact per-item price number shown on the final product page after the last option click. If it's range, copy the upper bound.
4. "query satisfaction' (compare 'query' 'selected'). verify that all requirements in the "query" are satisfied by 'selected'; Spot any contradiction.
5. Derive 'purchase success' based on purchase completion: purchase success = true only if Obser- vations confirm terminal purchase action took effect.
Output format (exactly): Return a single JSON array For each step, output an object with these fields:
query': <extract the exact queries from search act actions>, selected'; <product typel attributes... options_clicked... quantity price>,
peplecced_pprce"" <number>> "reasonin': <brief analysis of whether ALL query requirements are satisfied and any contradictionsy, suurry_satisfaction True False "purchase_sucooe success' True False,
Figure 12: Goal inference prompt for WebShop
23

---

Published as a conference paper at ICLR 2026
Action relevance labeling prompt for WebShop
You step-relevance classifier for WehShop. Input:
'target intention' (ISON): the shopping intention.
"taaeettoy":: ordered steps; each step
'Action": the web action by user.
"Ohservation": the observsation returned by the web server after the action.
'current intention' (ISON): intention inferred up to this step.
Decide per step (in hindsight) whether or not: (1) Did 'current intention' change vs the previous step? Summarize the delta; else none. (2) If no change: was the action needed for the eventual purchase path? Needed removing it would break what actually led to purchase. Not needed = no observable effect, dead ends later abandoned, toggles undone before use, unrelated clicks, no-ops.
Judge ONLY from observation inserratio-cooffrmeed effects the provided state. Do skip steps.
Rules
Use only Observation-confimmes effects and the provided state.
Judge every step; don't 't skip.
'relevance' = yos iff (1) intention changed or (2) action was needed.
Output format (exactly): Return a single JSON array; one object per step:
"step": <number>,
Daction" "<exact provided action>", " iittentioo_delltal" "<concise diff 'none">", "needed for _or_ppurchas"; "cexplain whether the action necessary to the purchase behavior>", "relevance"; "yes" "no"
Figure 13: Action relevance labeling prompt for WebShop
24


Reviews:
[{"review_id": "3vbaKPmTAG", "reviewer": "Reviewer_PFk9", "summary": "This paper introduces Hindsight Supervised Learning (HSL), a post-training method for LLM agents. It leverages the unintended but successful goals achieved during an agent’s rollouts by relabeling trajectories with goals that were actually accomplished. These relabeled examples are then used for fine-tuning, with two key techniques: irrelevant-action masking and demonstration reweighting. Experiments on ALFWorld and WebShop benchmarks show HSL improves both supervised fine-tuning and DPO, achieving higher success rates with fewer expert demonstrations.", "questions": "Please see my comments and clarification questions in the Weaknesses section above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper provides somewhat strong empirical validation, showing consistent performance gains and higher sample efficiency across multiple benchmarks.\n- There is a theoretical property offered although it is difficult to parse due to presentation issues. I still appreciate the attempt to include a theoretical result, which is rare to see in papers with large foundation models.", "weaknesses": "Below are my comments in the order of appearance in the paper.\n\n1- In addition to hindsight relabeling, the paper should also discuss learning from play data in robotics. In that line of research, it is common to define goals in play data and label the trajectories with those goals.\n2- Around line 177-178, I was confused by why the paper uses both a goal state and a language instruction. I understand that it is desirable to have the function $\\delta$ because it will be useful for detecting the agent reaching other goals, but then, maybe goal state is all the paper needs? Why also have an instruction? Having both seems redundant from an RL perspective.\n3- In line 195, I believe the statement $K \\leq T$ implicitly assumes the goals are mutually exclusive so the agent cannot achieve multiple goals at the same time. Perhaps this should be clarified earlier, because in reality an agent can achieve both \"picking up a fruit\" and \"picking up a banana\" at the same time, for example (or \"closing the drawer\" and \"putting the mug in the drawer\", etc.)\n4- \"improve agent optimality\" is a weird phrase. If something is optimal, it cannot be improved by definition.\n5- There is a system prompt that describes the space of valid goals. So the paper needs a definition for each goal. Perhaps something like \"this set of states achieves this goal\". But if it is a mapping, there would be no need for an LLM. So I assume, what is needed is a natural language description of each goal. This should be part of the problem statement.\n6- In the theoretical analysis section, there is this variable $h$ in Equation 1, but it is not defined anywhere. This makes it very difficult to follow the theory. I assume it is history. But then, how can $\\tau_{t-1}$ for different values of $t$ can be equal to $h$? The former has varying length depending on $t$ whereas the latter is fixed. Does this mean the probability term inside the summation is nonzero only for a single $t$? If so, writing it as a summation is unnecessarily complex. Perhaps this is not the case, and there is something I miss about $h$. Since it is not defined, I do not know what it can be.\n7- The conclusion section should discuss when (for what tasks) this method can and cannot be applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FGCih79J7", "reviewer": "Reviewer_MGUs", "summary": "The authors introduce an approach Hindsight supervised learning which trains an LLM agent on relabeled data. A relabeler LLM is used to relabel the LLM agent's trajectories with the goals that the LLM actually achieved. The authors provide results showing that by doing so, the agent is able to outperform baseline approaches.", "questions": "Could you please discuss the goal space and how this might be defined in environments that don't have clearly predefined goals\n\nCould you discuss how incorrect labels might dealt with and how they impact downstream performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very easy to read and provides a thorough theoretical and experimental analysis. The results are strong and very promising.", "weaknesses": "The relabeler’s output space is manually constrained with environment-specific templates. It would be helpful to discuss how HSL would scale to domains without predefined goals or how to automatically infer such goal spaces.\n\nThe method assumes that any achieved goal is beneficial to learn from but in some situations unintended achievements may not align with user intent or task utility. It would be useful to discuss ways to filter or weight relabeled goals based on relevance to user-defined objectives.\n\nIt would be useful to discuss the distinction between successful but unintended goals and partial progress toward intended goals. The method seems to treat both similarly, but their learning value may differ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mt9MRz0TNq", "reviewer": "Reviewer_jNmR", "summary": "The paper proposes Hindsight Supervised Learning (HSL) for (text-only) LLM agents in closed-world environments (i.e., with a structured schema of available goals): in addition to training the policy on expert demonstrations, train on on-policy rollouts labeled on the fly by an auxiliary, highly capable LLM, which relabels rollouts with all goals actually achieved, as judged by the auxiliary LLM. Relabeled trajectories are kept in a FIFO queue that is continually updated, and enter training alongside pre-existing expert demonstrations. Theoretical results provide some motivation for their approach; experiments on ALFWorld and WebShop show meaningful gains on the former and smaller gains on the latter. The paper does not establish results for open-world goal spaces.", "questions": "I found this paper interesting. Below are some questions/requests that I think that, if addressed, would substantially improve the quality of the paper.\n\n- Seeds/variance: How many seeds per result? Seems only one. Please report mean±std (≥5) for Table 1 and all curves.\n\n- Transparency: Will you release code, relabeling prompts, and the relabeled data?\n\n- Relabeler sensitivity & cost: How does performance change with 3B/8B/70B relabelers? Report relabeling tokens, wall-clock, and total GPU hours per setting.\n\n- Label validation at scale: can you label a substantially larger set of trajectories (e.g. 200) and/or add automatic predicate checks (where supported) to strengthen the statistical significance, and report confusion matrices per goal type?\n\n- Ablation deconfounding: Please re-run a variant of RELABELFAILURE that (i) relabels all trajectories (successes included) and (ii) fixes the relabeled-sample budget across variants to disentangle algorithmic effect from data volume.\n\n- Theory diagnostics: Any proxy measurements over training for coverage (e.g., diversity/goal-type coverage of hindsight set) and hindsight-expert quality (environment predicate agreement) to support the claimed mechanisms?\n\nthank you!", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "Practical, simple idea that composes known ingredients (hindsight relabeling + SFT) into a new, tidy, reusable pipeline for language agents.\n\nClear empirical signal on ALFWorld; sample-efficiency curves indicate strong wins at lower demo budgets. Notably, on ALFWorld Unseen the method reaches ≈92.5% with 800 demos vs ≈82.8% for DPO with >3,200, indicating real demo-efficiency.\n\nImplementation details (masking + reweighting + on-policy refresh) are intuitive and ablations suggest each adds value.\n\nTheory is tidy (imitation-style discrepancy bound) and aligns with the empirical recipe—even if mostly motivational.", "weaknesses": "The paper has potential to be a nice contribution but has serious methodological shortcomings in its current presentation:\n\n- No variance / seeds (critical). Table results and curves lack seeds, error bars, or confidence intervals. Delta could be within variance. A single cherry-picked seed is an absolute non-starter to get the paper accepted. (Addressing this point alone would change my score from 2 to 4; see questions below).\n\n- Code / reproducibility (critical). Prompts are provided, but there’s no released code or relabeled dataset for third-party checking (and no determinism details). Uploading assets as a ZIP to OpenReview (or to some CDN that is anonymized) for inspection is a totally reasonable academic practice (especially if there are manually labeled results on the paper) that the authors should be expected to satisfy.\n\n- Cost/efficiency underreported (important). The paper states fine-tuning wall-clock but does not clarify whether this includes the compute cost for continual relabeling with a large model, nor report the latter separately. Without this, “data efficiency” is unclear relative to real cost.\n\n- Label-quality evidence is thin (important). The relabeler is validated with a very small human spot-check of only 50 trajectories.\n\n- Heavy dependency on a powerful annotator (important). Relabeling uses a ~70B model for a 1B agent. There’s no sensitivity to smaller relabelers; conclusions may depend on annotator strength.\n\n- The framing of \"almost unsupervised\" could be misread (important). The method requires ground-truth demonstrations and mixes them in every training step. It does not replace demos; the current evidence shows materially improved use of demos, certainly not demo-free learning. A modification of language here could be in order. The paper also does not compare clearly the compute cost of their method vs that of their baseline (i.e. using expert demonstrations only); not discussing trade-offs between / scaling laws of compute and dataset size limits the reach of their conclusions.\n\n- Ablation confounds in RELABELFAILURE (preferable). Because that variant relabels only failed trajectories, it gets less data as the base improves. This confounds “algorithm” with “data volume.”\n\n- Theory–practice gap (preferable). The bound depends on coverage/optimality constants that are not measured nor bounded experimentally; masking/reweighting are argued heuristically to help them, but this link isn’t empirically probed.\n\nThese evidence gaps drive my Soundness=1 and overall score. My take is that this paper is a valuable practical contribution with clear empirical signal on the right kind of tasks, but acceptance should be contingent on addressing the issues above; most notably, statistical significance. These are fixable and would substantially strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CMYvdwakmf", "reviewer": "Reviewer_8bBB", "summary": "This paper proposes a sample-efficient learning method that reuses agent trajectories via hindsight experience relabeling. The methods are benchmarked in embodied navigation tasks with good empirical performances. Theoretical performance bounds are provided for the proposed method as well.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper is easy to follow, and the introduced algorithms are simple to add on to existing models, making them widely usable. The baselines are chosen well and extensively, while the studies on sample efficiency, ablations, and analysis are also designed well. Empirical performances are promising for the proposed method. (especially section on ensuring the performance gain doesn't only come from using a more powerful LLM)", "weaknesses": "- None of the main results have any statistical significance or uncertain qualification. The tasks from ALFWorld and WebShop should ideally be reran multiple times for every algorithm in order to at least get some standard deviation in score or success rate.\n- The novelty of this work is rather limited, as hindsight experience relabeling is a widely used technique across robot learning and reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)