Weak-to-Strong Generalization with Failure Trajectories
Authors: Ruimeng Ye
Zihan Wang
Yang Xiao
Zinan Ling
Manling Li
Bo Hui
Venue: ICLR 2026 Poster (2026)
Keywords: Weak-to-Strong Generalization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Weak-to-Strong generalization (W2SG) is a new trend to elicit the full capabilities of a strong model with supervision from a weak model. While existing W2SG studies focus on simple tasks like binary classification, we extend this paradigm to complex interactive decision-making environments.  
Specifically, we fine-tune a strong model with trajectories of intermediate actions generated by a weak model. Motivated by the human learning process, we propose to generalize not only successful knowledge but also failed experiences so that the strong model can learn from the failed trajectories accumulated by weak models. To effectively and efficiently elicit the potential of strong agents, we further construct ``trajectory trees," a hierarchical representation that organizes weak model-generated action trajectories, coupled with Monte Carlo Tree Search (MCTS) to optimize the strong model. Through theoretical analysis, we provide formal guarantees for the effectiveness of our method in improving W2SG performance. Our empirical evaluations demonstrate substantial improvements in reasoning and decision-making capabilities across diverse task domains, validating the scalability and robustness of our proposed framework. Our code is available at: https://github.com/yeruimeng/TraTree.git.

Published as a conference paper at ICLR 2026
WEAK-TO-STRONG GENERALIZATION WITH FAILURE TRAJECTORIES
Ruimeng Ye', Zihan Wang', Yang Xiao', Zinan Ling', Laing,Mann Manling LI, Bo Hui University of Tulsa NNortwweteen University (ruy9945, bo-hui Nutull sa. edu
ABSTRACT
Weak-to-Strong generalization (W2SG) is new trend to elicit the full capabil- ities of strong model with supervision from a weak model. While existing W2SG studies focus simple tasks like binary classification, paradigm to complex interactive decision-making environments. Specifically, we fine-tune strong model with trajectories of intermediate actions generated by weak model. Motivated by the human learning process, we propose to general- ize not only successful knowledge but also failure experience that the strong model can learn from the failed trajectories accumulated by weak models. To effectively and efficiently elicit the potential of strong agents, we further con- struct "trajectory trees," hierarchical representation that organizes weak model- generated action trajectories, coupled with Monte Carlo Tree Search (MCTS) to optimize the strong model. Through theoretical analysis, we provide for- mal guarantees for the effectiveness of our method in improving W2SG perfor- mance. Our empirical evaluations demonstrate substantial improvements in rea- soning and decision- makiing capabilities across diverse task domains, validating the scalability and robustness of our proposed framework. Our code is available at: hhttp:: h/ cewiluu.coo//eerrimeee com//yeruiieeg//rarreeee..
extend this
INTRODUCTION
The advent of Large-scale Language Models (LLMs) has marked a significant advancement in wide range of tasks. Currently, the alignment and supervision of these models primarily rely on human feedback and fine-tuning paradigms such as RLHF Ouyang et al. (2022); Christiano et al. (2017) 2 Stiennon al. (2020);
: [Bai al. 2022). However, as the research interest LLMs continues growing and new capabilities of LLMs are being developed rapidly, it is believed that superintelli- gence (i.e., AI smarter than humans) could arrive within the next 10 years [urnns et al. (2023) This raises critical challenge, providing reliable supervision becomes increasingly difficult when superhuman models potentially surpass human-level intelligence across numerous domains Casper et al. (2023). Thus, how effectively supervise LLMSs that may exceed human capabilities: in com- plex tasks remains an open and pressing question.
This challenge has prompted researchers explore alternative supervision mechanisms. A partic- ularly promising paradigm is the Weak-to-Strong Generalization (W2SG) framework [Brns et al. 20223: ; Ye al. (2025), which utilizes weak models substitutes for human supervision, elicit- ing strong models to Tearn from the weak labels generated by these less capable models. While this approach has shown remarkable performance simple tasks such as binary classification, its application to complex scenarios such as reasoning remains largely unexplored. As the majority of existing alignment methods L.MSs leverage RLHF (reinforcement learning from human feedback) to align LLMs with human values (e.g., safety), we remark that the human values can be generalized from the weak model with the W2SG framework when reliable human supervision is unavailable. For example, if we fine-tune a weak model with SFT SFyang et al. 2022 to follow human intention in complex decision- decision-making, the challenge is how to generalize the intention with a weak supervisor and elicit the optimal policy in strong model. Just like humans supervising strong models, weak models carrying the human's 's intention to align the strong models. Such setup is called weak- to-strong learning in Burns al. (2023). In parallel, recent research has explored performance-baser

---

Published as a conference paper at ICLR 2026
Prumpt Mestare de koperattre of soalum chlovisle
Successful Trajectory
Prompt Minzzure the sesperature of sodiat oifortde
Look aroind
Focus themoometer
around
Pick up thermometer
Telepart to kitchen
Teleport buthroom
Weak Agent
Focts on the thermometer
Pick up thermooneter
Look around
Fallure Trajectory
Strong Agent
Pick up sodium chioridee
x
Prompt Mesiurr the iompenatare of zoolan ciloride
Pick up thermometer
Look around
Teleport to kischen
Focus on thermometer investory
Focus on sodium chlerids
Weak Agent
Focus on the solium
Trajectory Tree
Figure 1: Illustration of the trajectory tree construction and the weak-to-strong generalization with trajectories. The left part explores different trajectories with weak model. trajectory tree is constructed by merging the same action path. Nodes the tree represent different actions and performing different actions will lead to various subsequent paths. Then the trajectory tree used to elicit the ability of the strong model.
learning approaches such as Direct Preference Optimization (DPO) Rafailov et al. (2024), which en- ables agents to learn from trajectory pairs during exploration. However, these approaches still face limitations due to the binary nature of pairwise preferences, which fails to capture the rich informa- tion and structural relationships among multiple reasoning paths Amini et al. (024).
In this paper, first extend the W2SG problem complex decision-makinn tasks where the lution of an LLM agent trajectory of actions Li et al. (2022). Figure [ demonstrates our pro- posed W2SG workflow, where we use the weak LLM agent to explore diverse trajectories in the environment, obtaining feedback, and the stronger model can learn from both positive and nega- tive outcomes. Note that use the weak LLM agent to explore environments to generate diverse solution trajectories, since sub-optimal solutions may limit the generalizability. The benefits are threefold: (1) The strategy addresses the limitation of entirely relying on human expert trajectories by exploring solution trajectories with weak supervision model. It enables strong LLM agents to learn without human intervention (Section (.T). (2) The explored trajectories can elicit the strong model to explore the larger unknown solution landscape due to the limited ability of weak models (Section 3.2); (3) The generalized knowledge can be passed with a bootstrapping framework (i.e.. 1113-GPTT-GPPTT
We remark that both success and failure trajectories are important for W2SG. Similar to human learning from the failure experience summarized by ancestors, the failure trajectories elicit the strong model's ability to avoid the same failure (Section 4.2
The cornerstone of our innovation the trajectory trees, a hierarchical representation that funda- mentally extends beyond traditional linear Chains of Thought (CoT) Wei et al. (2022). While the Tree-of-Thoughts (ToT) approach Yao et al. (2023b) also explores multiple reasoning paths, our tra- jectory explicitly organize both successful and failed trajectories from weak models, thereby capturing richer hierarchical relationships among diverse solution paths. Instead of relying on single, linear reasoning paths, we explore and collect multiple solution trajectories from weak models and construct a comprchensive tree structure. Figure Tillustrates the construction of the trajectory tree. Different from DPO Rafailov et al. (2024) that guides learning through random contrastive prefer- ence data pairs (i.e., there is no overlap between two solution trajectories in a random preference pair), trajectory trees capture the global relationships and hierarchical structures among reasoning paths, providing more comprehensive and diverse training signals. For example, while the success trajectory (the purple path in Figure [ and the failure trajectory (the red path) share the same prefix actions, the very first different action across two paths could be the key to the success. Compared with random pairs, such structural differences can improve the efficacy W2SG. We traverse the trajectory tree and fine-tune the strong model with the preference of actions the tree structure. To
2

---

Published as a conference paper at ICLR 2026
further improve the performance and efficiency of W2SG, introduce Monte Carlo Tree Search (MCTS) Grill et al. 22020); Castellini et al. (2023) as the policy optimization algorithm motivated by the success of MCTS in board games Schrittwieser al. (2020). Optimized actions are selected by computing the cumulative reward and the node visit count. The optimization algorithms enable the strong model efficiently generalize from policies converging to optimality with small error probability. Specifically, the upper confidence bound applied to the trajectory trees is to deal with the exploration-exploiattiio dilemma. Lastly, we theoretically prove that the weak-to-strong model can surpass the strong model's ' performance trained on expert trajectories, even when learning from imperfect trajectories generated by the weak policy.
In the experiment, we demonstrate that weak, well-trained models can effectively produce auxiliary signals to guide the learning of strong models, establishing more scalable pathway for improving L.M agent performance. Our primary contributions include:
We investigate the feasibility of the weak-to-strong generalization of LLM agents in com- plex tasks where the solution is trajectory of actions. Our work addresses the limitation of existing W2SG works in an analogous setup.
We propose to construct trajectory trees to organize both success and failure trajectories ex- plored by weak model. Instead of relying on single reasoning paths or random contrastive pairs, the tree structure can capture the shared path between success trajectory and fail- ure trajectory. The divergence between the two paths vital for knowledge generalization.
To the best of the author's knowledge, this the first work that introduces MCTS in W2SG. We employ MCTS to capture hierarchical relationships between reasoning paths and pro- vide more detailed and complete representation compared traditional linear CoT ap- proaches. We also present a theoretical analysis of W2S.
Surprisingly, we find that the W2S model can even outperform the SFT strong model in the experiment. verifies the validity and effectiveness of our approach.
2 PRELIMINARY
LLM Agent Tasks Formulation Consider an LLM agent performing tasks formalized partially observable Markov decision process (POMDP) (U, S, A, O, T, R). These components represent: of available actions A; the domain of observable feedback O: a state transition mapping T : S x A + S; and an immediate reward function R: S x A [0, 1] quantifying performance for state-action pairs.
the space of possible instructions U: the complete state space S; the
For any instruction u E U, the LLM agent's policy ma, parameterized by 0, generates an action dj each step j sequentially according to aj (T(-1u [u 41, Oj...., aj-1, 0j-1) (where (01, (01,0)...) oI....) is the interaction history; for j 1, history contains u or an initial observation). This process yields trajectory:
.......,,,,.... a2-1: O On 0n), (u, a1, 01, ................................ ...
where n. is the number of actions. The probability of the agent's actions in given is:
no(e|u)= = II (oaaal lu, 01,01, a--11,0-11),
(2)
Ø=1
For each completed trajectory e, the environment assigns final scalar score G(e) E [0, 1]. which quantifies its overall quality or success rate on the task u. This G(e) serves as the primary feedback for the trajectory. The overall performance of policy Ty is its expected score, R(TO), defined as:
R(ma) Eu (G(e),
Here, Dy a distribution over the set of initial instructions U. R(TT) represents the policy's 's average task score and is our primary evaluation metric. Further details how G(e) is computed and how it relates to the immediate reward R and the expected score R(x,) can be found in Appendix E
Problem Setup Our work investigates weak-to-stron generalization (W2SG) in the context of LLM agents performing multi-step interactive tasks. We consider two types of models: a weak model (TT..) and strong model (T,). These models differ in their underlying capacity (e.g., parameter count,
3

---

Published as a conference paper at ICLR 2026
architecture), with и assumed to have larger capacity and potentially higher performance ceiling than Tw. We denote the base pre-trained models as Thise and xx= When tne-ttued on a dataset of expert-demonstrated ground truth trajectories Dex Dexpen using Supervised Fine-Tuning (SFT) Ouyang al. 2022), these models become =FST and T respectively. The model TF serves crucial baseline representing a strong model trained with high-quality expert supervision.
Our W2SG vs Prior Work
This framing extends the traditional Weak-to-Strong Generalization Burns et al. (2023), which often focuses on strong model learning from (potentially noisy) discrete labels pro- vided by weak supervisor. In contrast, our work applies W2SG to sequential decision- making tasks where the weak model T generates entire trajectories interaction. We propose that by structuring these explored experiences (via trajectory trees) and applying advanced optimization algorithms (like DPO informed by tree structure, or MCTS for path refinement),
more effectively distill knowledge from weak model explorations guide the strong model.
The open problem address is: supervision derived from the less capable ा. (in the form of its generated trajectories) used to elicit the full potential of тs potentially enabling to achieve performance comparable to exceeding ? This exploration aims to determine if W2SG holds for complex interactive tasks, evaluated by the policy's expected task score R(T).
3 METHODS
Figure 2 depicts the workflow of our method. Our first step is to generate diverse action trajectories with a fine-tuned weak model, = These trajectories are subsequently organized into a hierar- chical trajectory tree. This tree then serves as the foundation for two proposed W2SG fine-tuning algorithms designed to enhance the strong model's (F,) reasoning and decision-makin capabilities by learning from the structured success and failure experiences of the weak model.
3.1 TRAJECTORY EXPLORATION
buse using SFT on expert demonstration The initial step involves training the base weak LLM Twe dataset Dexpen {(X,, YO) Naspon obtain SFF The SFT objective the standard negative log- likelihood:
C (0w)
Noves [Yd
(8) log ne (w X1, Veri Ow),
(
Nexpert
Once SPT is obtained, is used explore the task environments and further generalize the human feedback to the strong model. For each instruction we prompt =SFT multiple times, varying sam- pling parameters (e.g., temperature, top-p) to generate diverse This diversity is crucial for constructing rich trajectory tree that captures wide range of behav- iors-including successes, failures, and suboptimal paths. The score G(ex) ( for each trajectory e is provided by the environment's objective success criteria. To further enhance exploration diversity, we can optionally refine the exploration policy using a KL-divergence penalty against previously generated trajectory distributions Song al. (2024a):
M trajectories (c1,..., ead )-
Lexplore (0 (0w)
(OF)(Ow) - A (L (-IX; Ow Ie
(5)
where exploore is reference policy from previous exploration phase. These collected diverse trajectories form the subsequent tree.
3.2 TRAJECTORY TREE CONSTRUCTION AND WEAK-TO-STRONE GENERALIZATION
The diverse trajectories collected from T SFT are organized into a unified trajectory tree T = (V,E), where is set of nodes and & represents directed edges. This tree is constructed iteratively by
4

---

Published as a conference paper at ICLR 2026
Trajectory Exploration
Coustruct Trajectory Iree
(cl) W2SG with Structural Contrastive Pairs
put the posato ant the miccowave
Moddi
Weak Model
(c2) W2SG with MCTS
Ob: There a fidge Agest: Open the frigg Ot: the fisege.... Agent lake the poiato
Roward: 1.0
Artion spact
Strong Model
UCBH!!
Bask Propegation
Figure 2: Illustration of the Weak-to-Strong framework. (a) Given an instruction, the weak LLM agent interacts with the environment collect both success and failure trajectories of actions. (b)
The explored trajectories are used to construct a trajectory tree by merging prefixes of actions. We propose two methods to supervise the strong model: (cl) DPO with structural contrastive failure-
success pairs of trajectories instead of random pairs; (c2) Fine-tune the strong model with Monte
Carlo Tree Search.
adding each explored trajectory, starting from a root node that signifies the initial instruction. Each node v E this tree signifies an agent's "execution step" (o,, th, f,, 0,), comprising the preceding environment observation o,, the agent's 's thought the and its subsequent action a,- Edges in the tree implicitly represent the sequential progression from one such execution step to the next. To foster a compact and generalizable structure, paths merged: when adding from a parent node, if an existing child node represents the same action Onew taken from seman- tically similar observation ou (where similarity between Og and Onew is determined using sentence embeddings and cosine similarity threshold {sim), that existing child node is reused and its visit count updated. Otherwise, new node for (Onew; thhnew (new) is created. Our merging strategy uses exact matches for actions; consequently, even minor phrasal variations in actions lead to distinct branches. The final enviromment- prrveiddd score G(e) for each original trajectory associated with its terminal execution step (node) in the tree, for example, by aggregation into total moellrewardd
step (Onew, Uhnew (new)
"good" trajectory tree, conducive to our W2SG methods, exhibits several key characteristics: (i) Diversity (Breadth), stemming from the varied explorations of TSST, thereby capturing a wide range of strategies and behaviors; (ii) Representativeness (Depth), with paths that are sufficiently long to represent complete near-complete attempts at solving the given tasks; and (iii) Informativeness, featuring clear divergence points where different actions taken from similar (merged) states lead to demonstrably varied outcomes, indicated by their aggregated G(e)) scores from the initial weak model explorations.
Based this trajectory tree, propose W2SG algorithms fine-tune the strong model: W2SG with action preferences Instead directly fine-tuning with DPO Rafailov et al. (2024) based random contrastive pairs in common DPO practice, propose model the structural difference between two paths on the tree. Preference pairs are formed from divergence points within the tree, where two different continuations o and from shared prefix path h lead distinct aggregated G((e) outcomes in the weak model's exploration. We define r+ (h, a+) and - (h, o emphasize these critical decision points. The strong model He is then fine-tuned on the resulting dataset D.. = {(T,t .T. D using the following DPO loss:
SFT) CTeeDDOO TE
Ber irr JNDw [log a (7+) Vn, (TT))) + 3 +. (T. SITT,
(6) where Tx. (7) denotes the implicit DPO score of trajectory т under the strong model n, (which being optimized), and n. serves the fixed KI. reference model during this DPO training
5

---

Published as a conference paper at ICLR 2026
phase. W2SG with MCTS While weak-to-strong generalization with the preference of actions is intuitive, the computation complexity of optimizing with all contrastive pairs on the tree is high due to the large action space a large-scale dataset. To reduce the cost and identify informative trajectory paths within this tree structure, we use MCTS offline to search the static trajectory tree and synthesize high-quality trajectory e* for SFT-based imitation by म. The MCTS process iteratively traverses the tree: child execution step nodes are selected from a parent node using UCB, which balances exploration with exploitation based node statisties (r)) (v)) These statistics are updated via backup of terminal G(e) scores from the original weak trajectories that define the traversed MCTS path. The UCB formula is:
UCB(e') (M
log Cm cAr(v") cx)("" + VV CM (t')
After MCTS iterations, an optimized path ex extracted by greedily selecting child nodes with the highest MCTS-refined average rewards (rM (v)/ (u)//c//// (c)) each step. The strong model n. then learns from a dataset De- of these e* paths via SFT:
{MCTS(FE)
1-
Σ DED
Ta (k) context," (k)),
(8)
D
3.3 ANALYSIS OF W2SG
We provide a theoretical analysis for our weak-to-strong generalization (W2SG) framework, focus- ing on the scenario where Direct Preference Optimization (DPO) is employed with preference pairs derived from the trajectory tree. Our aim is to explain why this method, despite learning from "im- perfect labels", enable a strong model # #TTseDPO to surpass its SFT-trained baseline (TSPT) and potentially recover a significant portion of the performance achieved by more heavily supervised or "ceiling" models. The policy performance metric is R(T) as defined in Section 2
Our analysis is grounded in the Bayesian interpretation of DPO Rafailov et al. (2024); Jones et al. 1998). We consider The T E II, (where II, is the hypothesis class for strong models):
reference policy, forming part of prior over the strong policy
logp(T,) -B (L (T, = +C,
The preference dataset D., (15 TK }
Np from the trajectory tree informs the likelihood, where The policy # TTeeDPP is then obtained by minimizing
p(r Tk n.)) (rx, (Tx) Tsr (Tx
the DPO objective LIeDDrO [T) T. SPT) defined Equation equation 6 from Section 3.2 Further
details on this Bayesian derivation are in Appendix
We introduce key assumptions for our analysis:
Assumption 1 (Strong Model Expressivity and Superior Policy). There exists policy X² E II,
such that:
R(=*)= R(TTFT) where R(T) denotes the expected score.
2. For each +",, E Dic we have T (T+) > rT-((T))
Assumption 2 (Weak Model Coverage and Tree Richness). The SFT-weak model T..... through diverse exploration, generates trajectories yielding a variety of outcomes, with a > 0 proportion being successful or high-quality.
Assumption (Tree-Derived Preference Informativeness)) Preference pairs (r+,77) from the tra- jectory tree-vvi shared prefixes and critical divergences identified from aggregated weak model G(c) scores are significantly more informative for DPO than unstructured, random pairs. Tree- derived pairs isolate key decision points, providing DPO with clearer, more targeted learning signals distilled from weak model experiences.
Assumption 2 concerns only coverage of the weak model's 's exploration and does not imply Assump- tion Informativeness requires that divergent branches reflect outcome-relevant differences, which does not follow from coverage alone. This distinction is essential for Theorem [[ Assumption for- malizes the informativeness of tree-derived preferences: reducing the DPO preference loss these

---

Published as a conference paper at ICLR 2026
structured pairs correlates with improvements in the true return. This property is essential for trig-
gering the loss loss- per-frmarfoeeee sensitivity bound in Eq. equation 13 whose derivation and justification discussed in Appendix G
Theorem (Performance Guarantee for W2SG via Tree-Guided DPO), #x #TreeDPO is the policy ob- tained by minimizing the TreeDPO loss over the random sampling of Dw:
R(#g ( R(x + (R(#*) R(xx(()
KL ( + log log((Np/bp) N6/
-
(10)
N₂
for some constant C > O. This implies that +TheeDro can outperform the SFT-strong baseline if the potential improvement margin R(T=)) R( ) is sufficiently large relative to third term (estimation error/complexity). which diminishes with Np. Full proof details are provided in Ap- pendix G The practical interpretation of Theorem is that TreeDPO is beneficial precisely when the trajectory tree generated by the weak model provides informative preference gaps along shared- prefix divergences. These structured preferences activate the loss- puss-performance sensitivity relation, enabling the strong model to improve. When weak-model exploration collapses the preference pairs carry no information, this relation does not activate, and TreeDPO naturally reduces to the SFT strong model without degrading its performance. This is exactly the failure-safe behavior guaranteed by the KL-regularized objective.
4
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
Datasets We evaluate our approach in three environments: WebShop Yao et al. (2022), virtual shopping platform for product search and purchase tasks; ScienceWorld Wang et al. (2022), an environment for conducting scientific experiments and analysis; and AIfWorld Shridhar et al. (2020). household task simulation environment. WebShop and ScienceWorle provide continuous rewards between 0 and 1 based on task completion quality, while AlfWorld uses binary rewards indicating successful task completion. For each task, we use Average Reward and Success Rate in the test set as the evaluation metrics. All experiments utilize Low-Rank Adaptation (LoRA) with rank r set as 64 and a set 128 for efficient training. During the SFT phase, employ the A optimizer with a batch size of 32 (with gradient accumulation) and a cosine learning rate of le 5. After SFT, instances to collect weak trajectories. For the ETO (Exploration
AlhWirrdsShridhar
the agent explores the training Trajectory Optimization, DPO- based baseline described later in Section 4.1 training phase, we maintain the batch size of 32 while setting the learning rate 2e 5. The DPO loss uses KL penalty coefficient& 3 = 0.1. We employ language models with different capacities to investigate weak-to- to-sonn generalization: Ilama and Qwen. By default, I.amma-7B serves as our weak model and 1-3a2-133B is our strong model. We also conduct experiments on the Llama3-8B. Llama2-70B and Qwen2.5. For details of the experimental setup, please refer to Appendix H
Baselines To verify the effectiveness of the proposed method in W2SG, we introduce the following baselines.: 1) Standard Supervised Fine-Tuning (SFT) uses expert trajectories to conduct imitation learning. SFT is applied to both the weak and strong models on expert demonstration dataset (Dexper) to obtain the SFT Weak Model (=S and the SFT Strong Model (T.FT). π SFT addi- tionally serves as the generator of initial trajectories for our tree construction, while T acts as a primary performance benchmark. 2) employ Best-of-N (N = 8) sampling, where wSPT gen- erates trajectories per task, and the one with the highest environment-prowwe provided score G(e) is selected, representing strong inference strategy. 3) ETO (Exploration Trajectory Optimiza- tion) Song et al. (2024b), DPO-based method, where TTSTT is further fine-tuned using preference establish nigh-performance
pairs derived from its own environmental explorations. Moreover, mark, Ceiling Model is trained by fine-tuning ग. via DPO, using preference pairs that designate trajectories from Despen as "preferred" over those generated by x SFT's
own explorations.
4.2 RESULTS AND ANALYSIS
Table shows the weak-to-ste so-t-nng generalization performance across three tasks. We can observe that the phenomenon of weak-to soak-to-strong generalization still holds for these interactive tasks. We use "W2SG" to represent the weak-to-strom performance.
7

---

Published as a conference paper at ICLR 2026
Method
WebShop
ScienceWorld
AlfWorld
Avg Reward
Success Rate
Avg Reward
Success Rate
Avg Reward
SFT Weak Model (
47.1
87.0
41.2
55.5
44.8
Llama-2-7b+SFFT)
W2SG with Tree DPO W2SG with MCTS (ours)
53.2 56.9 (2nd)
97.0
55.4 58.2 (Ist)
61.1 66.8 (1st)
56.0 57.5 (2nd)
(Ist)
SFT Strong Model Alama-2-13b+SFTT
51.0
94.0
53.6
59.2
51.5
SFT Strong Model
52.0
97.5
54.9
61.1
53.7
-lama-2-13b+ET)
SFT Strong Model
52.3
96.0
55.3
60.7
55.2
Best of
Ceiling Model
58.3 (1st)
96.5
56.9 (2nd)
63.5 (2nd)
59.0 (1st)
The average reward and success rate of different approaches three agent datasets. W2SG methods trained with weak model trajectories generalize better than strong SFT baselines across tasks, with MCTS-based W2SG achieving best performance under purely weak supervision.
Table I:
Base LLM
WebShop
SciWorld
AlfWorid
Llama3-8B+ SFT
60.8
59.5
59.7
Llama3-8B+ DPO
62.0
62.7
61.9
Llama3-8B+MCTS
65.3
67.9
65.7
Table 2: The average reward of weak-to-strong generalization to Llama3-8B.
Specifically, we can find that the fine-tuned strong model with trajectories generated by a weak model consistently outperforms the fine-tuned weak model. Compared to the SFT weak model, the W2SG with DPO improves a maximum 4.3%. in terms of the average reward. Compared with the ceiling model based expert trajectories, find imperfect trajectories unlock the potential of strong models, recovering to 39.4% of the ceiling model's performance under weak supervision. Notably, W2SG achieves these improvements without requiring additional Luman-annotate data, relying entirely on weak models, which is particularly important in resource- constrained scenarios.
effectively
We further verify the effectiveness of MCTS in W2SG. Surprisingly, by struc- turing the weak model's trajectories in tree format and using Monte Carlo Tree Search to combine optimal nodes generating training signals, the result- ing strong model even outperforms the SFT strong model. There is an aver- age reward improvement of 11.6% and 11.7% over the SFT strong model on Web- Shop and AlfWorld, respectively. More- over, on ScienceWorld tasks, it outper- forms the ceiling model trained with the ETO method. This demonstrates that the proposed method enhances weak-to- strong generalization performance. We compute the p-value to judge the significance of the im- provement over runs with random seeds. To justify that W2SG with TreeDPO significantly out- performs the SFT strong model, equals 0.0003 (null hypothesis: two population means are actually equal). By conventional criteria, this difference is considered be statistically significant. Similarly, the P value for SFT Strong and W2SG MCTS (null hypothesis: two population means are actually equal) is 0.0001. Therefore, the superiority of W2SG MCTS over the SFT strong model is also extremely statistically significant.
Figure 3: Three agents are compared: SFT, WTS, and MCTS. X shows the time steps while Y illustrates the scores. The task ID is shown above the plot.
t-test and compute the p-value. The two-tailed P value
We also verify the phenomenon of weak-to-strowe to-otrong generalization with different LLM families. Ta- ble shows the average reward of Llama3-8B models Supervised by -lama2- 13B. The result further
8

---

Published as a conference paper at ICLR 2026
verifies the feasibility of eliciting strong capabilities with weak supervision. Our work makes em- pirical progress on the challenge of aligning superhuman models.
Effectiveness of Parameters. As shown in Figure we assess agent efficiency in the Worll en- vironment by measuring how effectively solves tasks with minimal action steps. Each task broken down into fine-grained subgoals, and re- wards are incrementally granted as these subgoals are achieved. Table shows that when we use a higher 3 value in W2SG in the Science World, the performance was inferior compared to using a lower one, which shows lower & value helps the strong model maintain better knowledge transfer while avoiding overfitting to the reference policy. We further explore the impact of the number of trajectories used to construct the trajectory tree in W2SG. Specifically, we adjust the de- coding temperature and top-p sampling strat- egy of the LLM agent collect multiple tra- jectories generated by the weak model. By in- creasing the breadth of the trajectory tree, we
Method
Tree
B
Avg Reward
agent
SFT WTS
53.6
0.1
54.9
0.5
49.2
MCTS
5
57.6
6
58.2
7
54.9
Table 3: The effects of tree breadth and 3.
66
50
Pusrage Ramard
56.21
examined the effect of the number of collected trajectories on the results, ranging from three to ten trajectories. As shown in Figure we observe that increasing the number collected trajectories initially improves the model's per- formance but eventually leads to a decline. No- tably, the ScienceWorld task, the weak-to- strong model achieves performance surpassing the ceiling model when trained with six trajec- tories. However, this does not imply that simply adding more trajectories will consistently enhance generalization. For the AlfWorld task, when the number trajectories used construct the tree ex- ceeds seven, the average reward achieved by the W2S model begins to decrease and the phenomenon also observed in the WebShop task. While increasing the number of trajectories can improve per- formance, there is optimal range beyond which additional trajectories may not yield better results and could even negatively impact W2SG.
Deb5haa APhord Scllars
Number
Figure 4:
The performance changes with the in- crease of trajectory numbers.
Performance on Qwen. We additionally investigate the W2SG performance on the Qwen2.5 family. We used Qwen2.5-14B 14BB as the strong model and Qwen2.5-7B as the weak model. The results shown in Table[]] TreeDPO improves the Qwen2.5- 14B SFT baseline both WebShop and AlfWorld, despite using trajectories from significantly weaker model (Qwen2.5- 7B). This trend mirrors our Llama experiments and confirms that weak-to-strong generalization architecture-agnosicc and not specific the Llama family.
Base LLM
Method
Webshop
Alfworld
Qwen2.5-7B
SFT
65.5
58.2
Qwen2.5-
SFT
71.7
61.9
Owen2.5-14B
TreeDPO
72.3
62.7
Qwen2.5-14B
MCTS
76.1
65.7
Table 4: W2SG on Qwen.
Ablation study. To better isolate the contribution of the
Method
Avg Reward
tree structure, additionally conducted an ablation on Alfworld using unstructured preference pairs on Llama3- 8B, where two complete weak trajectories are randomly paired based solely on reward. Unlike tree-derived pairs, these pairs do not share a common prefix and therefore contain substantially higher noise. As shown in Table 5 the unstructured DPO variant improves slightly over the strong SFT baseline but is noticeably weaker than TreeDPO. This confirms that shared-prefit diver- gences provide much clearer and more stable training signals than arbitrary preference pairs.
SFT
59.7
Unstructured DPO
60.4
TreeDPO
61.9
MCTS
65.7
Table 5: Tree DPO random pairs

---

Published as a conference paper at ICLR 2026
Cost of trajectory exploration and MCTS Rollouts. We remark that Trajectory tree construction is inexpensive in practice because it only processes the weak model's rollouts and does not require additional model training. For example, on Webshop, the tree construction and 100 MCTS Rollouts take 0.41 seconds and 0.23 seconds, respectively. Both components scale approximately linearly with the action horizon and the number of states in the tree, since all operations are local expansions or pointer traversals. Vocabulary size does not directly affect tree construction or search, since these stages operate purely on already-sampled trajectories rather than enumerating language actions.
Quality of weak models. Intuitively, the quality of the weak model will have an impact on the explored trajectories. We conduct additional experiments on Llama models and the Webshop Task with two substantially different weak
Model
Webshop
SciWorld
AlfWorld
Llama2-7B
47.1
41.2
44.8
Llama3-8B
60.8
59.5
59.7
Table 6: The weakness of Llama2-7B
Method
Llama2-7B
Llama2-133
models: Llama2-7B and Llama2-13B, while keeping the strong model fixed as
Jama3-8B+SFF
60.8
60.8
Llama3-8B+Tres DPO
61.3
62.0
Llama3-8B First, as shown in Table 6
Llama3-8B+MCCS
63.7
65.3
Llama2-7 B's standalone performance far below that of Llama3-8B. There-
Table 7: W2SG using Llama2-7B 7B and Llama2- 13B
fore, the weak model is indeed under- performing. Second, the results in 7 show a clear and consistent trend: using the much weaker Llama2-7B as the weak model, the weak-to-strong update still produces small but stable im- provement over the strong SFT baseline and never leads negative transfer. Using the stronger Llama2-13B 13B leads to significantly larger gains. These findings confirm the monotonic relationship predicted by our theoretical analysis: weak models with low exploration capacity offer limited but non-negative signals, while stronger weak models provide more informative tree structures.
Case Study Figure illustrates the actions of different models on the WehShop task. The strong model trained using SFT fails to complete the task correctly as it repeats incorrect actions, leading to failure and a reward score of 0. While the W2S model trained using the DPO method successfully completes the task, it makes suboptimal choices, such selecting product that does not fully meet the requirements. As a result, the reward score is relatively low at 0.5. However, the W2S model trained using trajectory trees not only completes the task successfully but also performs more detailed search and makes more precise choices.
a natning
that
also that skin, price loser
Agril
comeettt the larsag
The agna clecci tratinng
Agmm:
Brink libeabi casch
Ala
Thenkki bkakl roacl
Ageat
The
woold
Ind from ilted desi
dbaaiic
1 broggrn kook cleald of the
procact
pracace
Otu Aoa ecteopa)
(ir The Nchtpert
1h thoghht thenk take chick the Thyy
ancal
Cmn11
Thoate:
Yay of
prodle
TDreage:: bdbale procust
Andies ciicih Nevd)
Sucrmm
Fullre
Salion 2010
Revane
Taltrs
Emooed
Figure 5: An example of weak- stat-no- generalization in the WebShop scenario. The 4 interactions from left to right are the trajectories generated by the SFT Weak Model, the SFT Strong Model, the W2S model trained with DPO, and the W2S Model trained with MCTS.
CONCLUSION
In this work, we present a Weak-to-Strong Generalization (W2SG) framework in LLM agents. Our experiment verifies the feasibility of W2SG in reasoning and decision-makin tasks. The theoretical analysis provides robust foundation for understanding how W2SG can achieve superior perfor- mance, even when learning from imperfect trajectories. This work opens new pathways for scaling up L.M agents' training without requiring additional human supervision.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402. 10577,, 2024.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint MrXiv:2204008860 2022.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gian- inazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682-17690, 2024. No. 16.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner. Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong general- ization: Eliciting strong capabilitiiss with weak supervision. arXiv preprint arXi::2312.09399, 2023.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint MrXivv2307.15217 2023.
Alberto Castellini, Federico Bianchi, Edoardo Zorzi, Thiago D. Simão, Alessandro Farinelli, and Matthijs T. J. Spaan. Scalable safe policy improvement via monte carlo tree search. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.). International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 3732-3756. PMLR, 2023. URL https:://proceeeennggs m hnmlla.mmm
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to- strong generalization. arXiv preprint arXiv:2405.15116 2024.
Hao Chen, Jindong Wang, Lei Feng, Xiang Li, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, and Bhiksha Raj. A general framework for learning from weak supervision. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024a.
httpttpps///peeeeeeeeeee net nett/Forrumid-777gcaaaaaa
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning con- verts weak language models to strong language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b.
URL httpss://opeereeieww wwtpfoormmid--OAHHTTMA
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, and Chenguang Wang. Agent in- structs large language models to be general zero-shot reasoners. In Forty-first International Con- ference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL hhttps://oppnrrr www.oppnrreiieeeeeeeerorueeeeeee nww./oorumidd-Maaae.. vxr6CV
Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization. In Proceedings of the 37th International Conference on Muchine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3769-3778. PMLR, 2020. URL httphhtpp:p://// httphttt//pccceednngg mlr. wttpssssl..prpss//ieeeemmmm http//v11///// /9//g//lll0aa... html
Bo Hui. Da Yan, Haiquan Chen, and Wei-Shinn Ku. Trajnet: A trajectory-based deep learning model for traffic prediction. In Feida Zhu, Beng Chin Ooi, and Chunyan Miao (eds.), KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event,

---

Published as a conference paper at ICLR 2026
Singapore, August 14-18, 2021, pp. 716-724. ACM, 2021. doi: 10.1145//44734884477333 URL https://doi.org// 11000.1155/// 1145/3447548. 3467236
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455-499 492, 1998.
Germain Kolossov, Andrea Montanari, and Pulkit Tandon. Towards a statistical theory data selection under weak supervision. In The Twelfth International Conference Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL
https://openrewview wtwpr//rreeiieeeeeeooouuuuuueeee heeee/ooruummd-chhffcgneeeeeee
Hunter Lang, David Sontag, and Aravindan Vijayaraghavan. Theoretical analysis of weak-to-strong generalization. arXiv preprint arXiv:2405. 16043, 2024.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:3 199-31212, 2022.
Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai- Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584, 2025.
Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. Macpo: Weak-to-strong alignment via multi-agent contrastive preference opti- mization. arXiv preprint 11X1v22410.07672, 2024.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual confer- ence on Computational learning theory, pp. 164-170, 1999.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances neural information processing systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Jitao Sang, Yuhang Wang. Jing Zhang, Yanxu Zhu, Chao Kong, Junhong Ye, Shuyu Wei, and Jinlin Xiao. Improving weak-to-strong generalization with scalable oversight and ensemble learning. arXiv preprint arXiv:2402.0066 2024.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nat. 588(7839):604-606 2020. doi: 10. 00.10088841156--00000001111 URL 1038/541586-0200 03051-4
Junhao Shi, Qinyuan Cheng, Zhaoye Fei, Yining Zheng, Qipeng Guo, and Xipeng Qiu. How to mitigate overfitting in weak-to-strong generalization? arXiv preprint arXiv:2503.0422 2025.
Changho Shin, John Cooper, and Frederic Sala. Weak-to-stron, generalization through the data- centric lens. arXiv preprint arXiv:2412.03388) 2024.
Mohit Shridhar, Xingdi Yuan, Alexanle Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht, Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint Xix::00000000663 2020.
Yifan Song, Da Yin, Xiang Yue. Jie Huang. Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration based trajectory optimization for LLM agents. CoRR, abss/2403.02502 2024a. doi: htR.:stts:://de org/10. 18550/arXing 2403.02507
12

---

Published as a conference paper at ICLR 2026
Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and emror: Exploration based trajectory optimization for Ilm agents. arXiv preprint arXiv:2403.02502, 2024b.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and An. Q Improving multi-step reasoning for Ilms with deliberative planning. arXiv preprint arXiv:2406. 14283, 2024.
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? arXiv preprint mrXiv:2203.07540 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Yang Xiao, Zijie Zhang, Yuchen Fang, Da Yan, Yang Zhou, Wei-Shinn Ku, and Bo Hui. Ad- vancing certified robustness of explanation via gradient quantization. Edoardo Serra and Francesca Spezzano (eds.), Proceedings of the 33rd ACM International Conference on Infor- mation and Knowledge Management, CIKM 2024, Boise, ID, USA, October 21-25, 2024, pp- 2596-26066 ACM, 2024. doi: 10. 00.1145/362777333776666 URL https: /00///11111 1145/ 3627673. 0627673.3679650
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http:t:://ppereeiiieee
net wwte/fooruee ffme/orrmmmdd-115022k
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405. 18357, 2024.
Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong. Yankai Lin, and Ji- Rong Wen. Super aliimee)aligmmmntt: Strong models may deceive weak models in weak-to-strong generalization. arXiv preprint urXiv:2406.1 1431, 2024a.
Yuqing Yang, Yan Ma. and Pengfei Liu.
Weak-to-strong
reasoning.
arXiv preprint
arXhv:2407.13647, 2024b.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Pro- cessing Systems, 35:20744-20757,2000 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1809-180111 18099-18822 1822, 2023b.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond ohain-of-thoughh effective graph-of-thougle reasoning in language models. arXiv preprini arXiv:2305. 2023c.
Ruimeng Ye, Yang Xiao, and Bo Hui. Weak-to-stron toeak-t--ttrong generalization beyond accuracy: a pilot study in safety, toxicity, and legal reasoning, 2025. http::htts:///axiv.oom
12621
13

---

Published as a conference paper at ICLR 2026
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 3477222,7677, 2024a.
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference op-
timization: Improving chain-of-thought reasoning in Ilms. arXiv preprint arXiv:2406.09136, 2024b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.009923 2023.
Zhi-Hus Hua Zhou. brief introduction weakly supervised learning. National science review, 5(1): 44-53, 2018.
A LLM USAGE
During the preparation of this paper, we used large language models (e.g., ChatGPT) writing assistants for language polishing and clarity improvement. The models were not involved idea generation, experimental design, result analysis. All scientific content and conclusions are the responsibility of the authors.
B REPRODUCIBILITY STATEMENT
We have made efforts ensure the reproducibility of our results. All datasets used are publicly accessible. Anonymous source code and scripts for reproducing our experiments will be made available in the supplementary materials. These resources should allow researchers to replicate our results and extend our framework to new settings.
C LIMITATIONS
As discussed in Burns et al. (203)), when the weak model can contain errors that are easy to learn, the strong model could learn to imitate those errors. The limitation also exists in our setting: when weak model is explicitly misaligned, the misalignment can also be generalized to the strong model. Due to the lack of malicious human-labeled dataset used for RLHF, we are unable investigate the vulnerability of the W2SG framework. The research of W2SG is still in its early stages, and there many gaps between W2SG and other domains. Attacking through W2SG is future work we plan to explore and strengthen our claim in various practical scenarios.
D RELATED WORK
Weakly Supervised Learning. Traditional supervised learning methodologics are undergoing sig- nificant transformation Zhou 2018); Burns et al. (202)). When it is impractical to obtain fully annotated datasets, researchers have proposed to leverage imperfect but readily available signals Lang et al. (2024); Hui et al. (2021). Weak supervision offers an alternative approach supervised learning Sang et al. (2024): Charikar et al. (2024), aiming to keep advanced artificial intelligence models aligned with human intentions Yant et al. (2024a). A framework has recently demonstrated significant potential in enhancing Large Language Models' (LLMs) reasoning capabilities Lyu et al. (2024). Progressive frameworks enable stronger models autonomously optimize their training data with crucial insights from weak supervisors Chen et al. (2024b ay: Kolossov et al. (2024). Re- cent advances have further solidified the Weak-to-Strong Generalizzaton (W2SG) paradigm as powerful framework for eliciting strong capabilities from imperfect supervision. Beyond empirical gains LLM reasoning and decision-making Yang et al. (2024b); Lyu et al. (2024), recent works have explored its theoretical foundations Lang et al. (2024); Charikar et al. (2024). robustness in sudervuper settings Shin et al. 2024): : Xiao et al. (2024), and methods to mitigate overfitting from noisy weak supervision in reasoning intensive environments through staged bootstrapping Shhi jet al. (2025). These studies reveal that carefully designed weak signals can outperform traditional expert-labeled supervision in eliciting complex behaviors.
14

---

Published as a conference paper at ICLR 2026
LLMs Agents. Large Language Models have evolved beyond text generation into interactive agents capable of complex reasoning and task execution WWii et al.] (2022); Crispino et al. (2024); Xie
et al. (202)). A fundamental breakthrough came with chain-of-
prompting and reasoning frameworks ZZhang et al. (0033; Xu et al. (2024), enabling agents to break down complex tasks into manageable steps. The field has since advanced beyond simple sequential reasoning, with searchers developing more sophisticated approaches such as Tree of Thoughts (ToT) Zhang et al. (2024b); Yao et al. (2023a) for exploring multiple reasoning paths simultaneously and Graph of Thoughts (Go'T) Yao et al. (202cc; ; Besta et al.) (2024) for handling complex problem-solving sce- narios. Other works have also utilized search and value estimation for LLM reasoning. For instance, ReST-MCTS* Zhang et al. (2024a) employs a reinforced self-training approach where Monte Carlo Tree Search (MCTS) dynamically explores and expands reasoning paths to collect high-quality tra- jectories and infer process rewards, aiming for iterative self-improvement of a model's policy and value functions. Our west-no generalization (W2SG) framework mainly focuses on offline MCTS static trajectory tree generated by weak model to synthesize data for fine-tuning sep- arate, stronger model. Further distinct are methods like Q" Wang et al. (2024) and QLASS Lin et al.) (2025), , which primarily focus on enhancing an LLM's performance at inference time. Q" learns Q-value model as heuristic to guide an A AI-like search during the LLM's decoding process without altering the LLM's parameters. Similarly, QLASS trains QNet to estimate step-wise Q-values from self-generater exploration trees, using this QNet to guide the agent's 's decision -akking during infer- ence. In contrast, our work leverages weak model explorations not for direct inference guidance, but as a source of supervision to explicitly fine-tune and transfer knowledge to more capable strong model. However, significant challenges remain in ensuring reliability and safety, as L.LM agents can exhibit inconsistent behavior. Recent work has explored multi-agent systems, where LLM agents collaborate to solve complex problems Zhang et al. (2023). The field continues to evolve rapidly, addressing challenges in agent alignment, knowledge grounding, and robustness. While Weak-to- strong generalization has been studied on reasoning ability Yang et al.] (2024b), these works set up the task classification problem, which is different from interactive tasks in our setting.
E DETAILS ON TRAJECTORY SCORE AND POLICY PERFORMANCE METRIC
Trajectory Score G(e): As defined in Section 2 G((e) is the final scalar score assigned by the environment to a completed trajectory e (see Equation equation [ for trajectory definition, using your See 2.1 label). While the POMDP formalism includes immediate reward function R S x A + j0. 1), for many complex interactive tasks such as those in WebShop or Science the most salient form of feedback is terminal reward that reflects the overall success of the entire episode. This G(e) E [o. 1] directly represents this terminal reward. Formally, if one considers R(s, a) be non-zero only at the final transition leading task completion or failure, then G(e) can be seen as the sum Σ R(s,, az) (using notation where sy is the state after az, or 8)-1 is the state before a,), which simplifies the terminal score if intermediate rewards are zero
Policy Performance R(Tx): The performance of a policy ne. R(TT), is its expected trajectory score, from Equation equation 3 (using your Sec 2.1 label): R(TT) De This expectation averages the trajectory scores G(e) over:
1. The distribution of initial instructions tasks Dú.
2. The distribution of trajectories ne(-lu) generated for each task u, accounting for any stochasticity in Te or the environment.
Thus, R(TT)) provides a robust measure of the policy's general effectiveness. is the central quantity our theoretical analysis concerns and corresponds to empirical metrics like average reward over test set.
F FURTHER DETAILS ON BAYESIAN INTERPRETATION OF DPO
The Direct Preference Optimization (DPO) objective, referenced in Section 3.3 as Equation equa- tion (defined in Section 3.2,, is derived from finding the maximum a posteriori (MAP) estimate for the strong policy n. given preference data D,d {(T T m and prior p(m.). The posterior is p(T, Dic)) x p(D n. )((r))) The prior (TT.) x exp KT. ( *FFT)) (from Eq. equa-
Nr
15

---

Published as a conference paper at ICLR 2026
tion 9 The likelihood p(D TT) Ih p( Tk xg), where individual preferences
P(Tk n.) (rx, (T T.. (Tx )). Minimizing the negative log-posterior directly yields the DPO loss function. This framework ensures the learned policy balances fidelity to the reference Tw with exploiting preference signals from Dw-
G DETAILED PROOF OUTLINE FOR THEOREM 1
This appendix provides (T.) be the empirical DPO preference loss term from Equation equation 6 Let Lp(T.) be its true expectation over P, the underlying distribution of tree-derived preference pairs.
detailed outline for the argument supporting Theorem 1. Let
Proof The DPO objective (Eq. equation 6) finds TreeDDOO that minimizes empirical loss on D subject to a KL. penalty. Let (T,,..,T)) be the empirical preference loss term and Lp(x,) be its true expectation over the distribution A of tree-derived preference pairs. By standard PAC-Bayesian arguments McAllester (1999), the true loss L (heebi))) can bounded. Since Rg minimizer of the regularized empirical loss, its regularized empirical loss is no worse than that of T (from Assumption T):
#'TreDDo is the
(ATT:DDO LDD., (i TroeDPO) 1.3KL ( Tre:DPO TT
S
(...("))) + 3 KL |T SFT)
(11)
With high probability (at least 1 - ou). concentration inequalities ensure that empirical losses close to true expected losses for relevant policies. Thus, we can relate the true loss of R, TreeDPO to that of ==
Lp I4 < (T( + (KL(T TT
SFT
Cap(II,,, 8 KL) + Log(NNp/b)) o
(12)
N,
The core step is to link this preference loss Lp(xs) to the policy's ss actual performance R(T,). As- sumption is critical here: informative, tree-derived preference pairs ensure that lower preference loss (i.e., better alignment with true outcome distinctions) correlates strongly with higher policy performance R(x,). We posit that for policies w close to x", there's relationship like
R(x*) - R(T) S C(Ip(m) Lp )),
(13)
for some sensitivity >
0. This implies that reducing the preference error towards the optimum Lp(=*) directly translates to improving R(r)) towards R(x*).
Combining the relationship equation we obtain:
13 with the bound on T.p (TTTsDDP)) from Equation equation 12
R R(# TrocDPO)
> - (( SFT)
(
Cap(lI,, 3 KL) + log log((N/66))) N₂
+0
(14)
- into Equation equation 14 direecly leads the form stated in Theorem 1 (Equation equation TO where the constant C' in the theorem encapsulates Ç and constants from the (-)) notation.
Substituting R((=)) R(=
1. Bounding the Empirical Loss of TTTeDDPOO As stated in Equation equation in the main text, by definition of # TeeeDO as the minimizer of the regularized empirical loss, for π from Assumption
LeD_. (a TreeDPO) + 3 KL TreeDO || SFT S
LD,, (mT) + (KL (I*
(15)
2. Generalization Bound (Concentration of Empirical Loss to True Loss) With high prob- ability (at least 1 - 5p). standard concentration inequalities (e.g., based on McAllester's PAC- Bayesian bounds McAllester (1999) uniform convergence arguments for hypothesis class IT.))
16

---

Published as a conference paper at ICLR 2026
Base LLM
WebShop
SeiWorld
AlfWorld
Llama2-708 70B+SFT
65.1
64.6
63.4
(lama2-70B+The DPO
67.4
65.7
Llama2770B-MCTS
70.4
69.1
68.7
Table The average reward of weak-to-strong generalization with Llama2-708
relate empirical losses to true expected losses. For any policy [I.: [Lp(x) LD. GenError(NVy oo, IIg), where:
GenError(AN,. do, 11,) cap([Ts SFF) + log((o///// Np/6o)
(16)
No
The term cap(Il,, ف. SFT) represents a capacity or complexity measure of the policy class II., poten- tially influenced by the KL regularization towards T This term often involves quantities like VC dimension or Rademacher complexity for the functions parameterized by Ts. Applying this to # TreeDPO and '², and combining with the result from Step 1, yields Equation equation 12 from the main text:
Lp (Tg
(T TreeDPO) S [p(==')) KL || (
cap(II,, SFT log( (N Np
(17)
+ The term [L(T*|| r under the prior The
SFT here effectively captures the complexity component relevant to achieving reflected in the final theorem (Eq. equation 10).
'er
Linking Preference Loss to Policy Performance (TT)) This step, as described in the main text around Equation equation [3] relies Assumption 3 The core idea is that for stul-structured, informative preferences (which tree-derived pairs are assumed to be), better satisfaction of these preferences (lower (T(z)) implies better real-world performance R(x). The sensitivity parameter >> in R(x") R(T) < (TLp(x) Lr(r")) formalizes this positive correlation. The specific value of Ç depends on how directly the preferences captured by Lp (which relate Ts, scores) translate to the overall task score R. A strong correlation is more likely when Tx, accurately reflects true utility differences.
4. Deriving the Final Performance Guarantee The derivation proceeds outlined in the main text, substituting the bound for L TTeeDPP)) (Equation equation 12 into the loss-to-performance relationship (Equation equation 13). and then expressing R(xx") in terms of (TTTT to arrive at the statement in Theorem [] (Equation equation 10p. The constant C consolidates Ç and other constants arising from the O(.) notation and the specific form of the concentration inequalities used.
H DATASETS
Additional Results on Llama-2-70B To further validate the scalability of our method, we ad- ditionally conduct experiments with a 70B-parameter strong model. As shown in Table 88] both TreeDPO and MCTS continue to improve over the SFT baseline, with MCTS achieving the best average rewards across three tasks. This confirms that our weak-to-strong framework generalizes effectively to larger-capacity models.
WebShop WebShop is large-scale simulated e-commerce environment designed to test Large Language Model agents. evaluates an agent's '' ability to understand natural language instructions, navigate web pages, and select purchase products. The platform challenges agents with tasks such as query formulation, acting on noisy webpage text, and strategic exploration to assess their decision-making and language understanding capabilities.
17

---

Published as a conference paper at ICLR 2026
SciieneWWorle ScrenceWorld is designed to test agents' ability to perform elementary science tasks, such as conducting experiments or reasoning about scientific concepts. It evaluates an agent's 's scientific reasoning by requiring them to combine procedural actions with scientific knowledge to complete tasks like testing conductivity, modeling Mendelian genetics, or observing changes in states of matter.
AlfWorld AlfWorld is cross-modal framework that combines text-based and visually embodied environments to train agents in abstract reasoning and task execution. It evaluates an agent's ability to transfer abstract policies learned in textual simulations to complex embodied tasks, such object manipulation and navigation, in visually diverse and dynamic settings.
Experimental Settings
Task evaluation differs across environments: WebShop and Science World implement average rewards from 0 to to evaluate the model performance, whereas AlfWorld adopts simple binary success/failure rate metrics. We defined the task score as 100 Reward avg.. which captures the average reward obtained across episodes Yao et al. (2022).
Table Iillustrates the statistics of each dataset:
Dataset
Train
Text-Seen
Text-Unseen
WebShop
1,824
200
World
1,483
194
211
AlfWorld
3.119
140
134
Table 9: Dataset statisties.
Following the evaluation environment set up in Song et al. (2024b), we construct corresponding environments over three tasks evaluate the LLM agent's performance. The original ScienceWort and AlfWorld evaluation set comprises both seen and unseen sets. The Unseen set,s including new task instances, measures the out-of-distribution generalization of the agents. We evaluate the model performance on these unseen sets.
18

---

Published as a conference paper at ICLR 2026
Prompts
WebShop Instruction Prompts
You are web shopping. I will give instructions about what do. You have to follow the instruc- tions. Every round I will give you observation and list of available actions, you have respond an action based the state and instruction. You can use search action if search is available. You click one of the buttons clickables. An action should of the following structure: search[keywords]
click[value]
the action is valid, perform nothing. Keywords in search up you, but the value in click must value
list of available actions. Remember that your keywords search should be carefully designed.
Your response should use the following format:
Thought: think..
Action: clomessom)
Science Instruction Prompts
environment, are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, green- house, art studio, hallway You should explore the environment and find the items you need to complete the experiment. You can teleport to any room in step. All containers in the environment have al- ready been opened, you directly get items from the containers. The available actions are: open OBJ: open a container close OBJ: close a container activate OBJ: activate a device
You are helpful assistant some scientific experiment in an environment.
deactivate OBJ: deactivate device
connect OBJ OBJ: connect electrical components
disconnect OBJ: disconnect electrical components
use OBJ [on OBJ]: use device/item
look around: describe the current room examine OBJ: describe object in detail
look OBJ: describe a container's contents
read OBJ: read a note or book
move OBJ to OBJ: move object to container pick up OBJ: move object to the inventory
pour OBJ into OBJ: pour a liquid into container
mix OBJ: chemically mix container
teleport to LOC: teleport to specific room
focus on OBJ: signal intent on a task object
wait: task action for 10 steps waitl: task action for step
Alfworld Instruction Prompts
Interact with household solve task. Imagine you intelligent agent in household envi- ronment and your target to perform actions to complete task goal. At beginning of your interactions, you will be given the detailed description accomplish. For each of your turn, you will be given the observation of the last turn. You should first think about the current condition and plan for your future actions, and then output your action in this turn. Your output must strictly follow this format:
the current environment and your goal to
Thought: your thoughts.
Action: your next action. The available actions are:
1. go to recep
task obj from recep
put obj in/on recep open recep
5. close recep
6. toggle obj recep clean ubj with recep
heat obj with recep
19

---

Published as a conference paper at ICLR 2026
9. cool obj with recep where obj and recep correspond objects and receptacles. After your cach turn, the environment will give you immediate feedback based which you plany- our next few steps. the envrionment output "Nothing happened",, that means the previous action is invalid and you should try more options. Your response should use the following format: Thought: iyour thoughts; Action: iyour next actiong
20


Reviews:
[{"review_id": "LSvbkPBeiS", "reviewer": "Reviewer_SFUW", "summary": "This paper proposes a novel framework for Weak-to-Strong Generalization (W2SG) in complex decision-making tasks using large language model (LLM) agents. The key idea is to fine-tune a strong model using both successful and failed action trajectories generated by a weaker model. To organize these trajectories, the authors introduce a \"trajectory tree\" structure that captures shared prefixes and divergence points between success and failure paths. Two learning strategies are proposed: (1) Tree-guided Direct Preference Optimization (Tree-DPO) based on structured contrastive pairs, and (2) Monte Carlo Tree Search (MCTS) to extract high-quality paths for imitation. The paper provides theoretical guarantees and empirical results across multiple environments, demonstrating that strong models trained via this method can outperform standard supervised fine-tuning baselines and approach expert-level performance using only weak supervision.", "questions": "a. Can the authors elaborate on how the method performs when the weak model is significantly weaker?\n\nb.  Could the authors quantify the computational cost of trajectory tree construction and MCTS? How scalable is the method to tasks with longer action horizons or larger vocabularies?\n\nc. Is there a possibility of iterated W2SG, e.g., bootstrapping progressively stronger models via self-distillation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "a. Introduces trajectory trees for organizing weak model trajectories and identifies divergence points to train strong models. It is a novel application of structured contrastive learning.\n\nb. Provides a formal performance guarantee for the Tree-DPO method, showing that weak supervision can outperform expert-labeled SFT under specific assumptions.\n\nc. Demonstrates strong performance gains across multiple complex environments using only weak model trajectories.", "weaknesses": "a. The approach relies on the weak model producing sufficiently diverse and informative trajectories. Performance may degrade if the weak model is too underperforming.\n\nb. The MCTS variant lacks a formal performance analysis (unlike Tree-DPO).\n\nc. While the trajectory tree representation is compelling, an ablation study comparing with unstructured contrastive pairs would help quantify its specific contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJXFBgrnJj", "reviewer": "Reviewer_jPwv", "summary": "The paper extends the Weak-to-Strong Generalization (W2SG) framework to decision-making tasks for LLM agents. Instead of relying on human-labeled data, the method allows a strong LLM to learn from full action trajectories generated by a weaker agent, reducing the need for human supervision. The authors represent these trajectories as hierarchical trees that merge shared prefixes between successful and failed trajectories. This structure identifies divergence points where decisions lead to different outcomes, enabling preference-pair sampling guided by the tree rather than random (as standard DPO).\n\nThe authors propose to use Offline Monte Carlo Tree Search (MCTS) to sample high-quality trajectories used to optimize the strong model. The paper also provides a theorem showing that, under certain assumptions, the policy minimizing the proposed TreeDPO objective can outperform models trained with supervised fine-tuning (SFT). Empirical results on three agent benchmarks, demonstrate that W2SG under weak supervision improves both average reward and success rate over SFT baselines, approaching or surpassing the performance of expert-trained models", "questions": "See weaknesses and also:\n\nQ1. Could you clarify which aspects of Theorem 1, are specific to the tree-based formulation? I can see the weak-to-strong policy relationship clearly but not the tree related.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The extension of the W2SG framework to sequential decision-making for LLM agents is timely and relevant, it offers a way to reduce the need of human supervision in interactive reasoning tasks.\n\nThe use of hierarchical trajectory trees is well-motivated and effectively captures shared paths between successful and failed rollouts, enabling more informative preference sampling than standard DPO.\n\nThe paper includes a theoretical result showing that optimizing the proposed TreeDPO objective can outperform supervised fine-tuning (SFT) under certain assumptions.\n\nIncorporating offline MCTS to extract high-quality trajectories strengthens the training signal for the strong model and proves effective across the three evaluated agent benchmarks.", "weaknesses": "1. The approach relies heavily on the quality and diversity of trajectories generated by the weak model; limited exploration could constrain the effectiveness of the strong model’s improvement.\n\n2. The theoretical result provides an interesting insight, but its practical implications are unclear. It seems that when the weak policy \\pi_w^\"sft\"  diverges substantially from the optimal policy \\pi^*, the SFT-trained strong model could perform better. The paper does not analyze conditions under which TreeDPO is beneficial or when weak learners/trajectories should be discarded. This could make the theorem more actionable. \n\n3. Evaluation is done only on three environments, and a single model family (Llama).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oM16RFF4lj", "reviewer": "Reviewer_yTwq", "summary": "This paper investigates whether the Weak-to-Strong generalization (W2SG) paradigm can be extended from binary classification to sequential decision-making environments. The core idea is to fine-tune a stronger model using trajectories sampled by a weaker model. In order to achieve this, the authors propose to construct _trajectory trees_ (which are game/decision trees applied to text generated by LLMs) from the weak model rollouts, and use this tree to select which paths should be used to tune the stronger model. They present two approaches: select preference pairs for DPO from divergence points in the tree (TreeDPO), or use MCTS to identify paths with strong performance for SFT (W2SG with MCTS). For TreeDPO, they prove a theorem that, under strict assumptions, guarantees performance improvement over a strong SFT baseline. Finally, the empirical results on WebShop, Science World, and AlfWorld suggest W2SG methods  can improve the strong model's performance. \n\n**Recommendation:**\\\nEven though this topic falls quite far outside my area of expertise, I am an expert on exploration in sequential decision-making environments and reinforcement learning.  With this perspective, I recommend to reject this paper. It appears to lack in its narrative and framing within existing literature: it is framed as an W2SG approach but seems to not be aligned in the overall objective. Furthermore, it is currently not possible to judge the significance of the empirical results.", "questions": "- How do the proposed approaches align with the core alignment objective of the W2SG paradigm?\n- How is the significance of the empirical results judged? How many seeds were used and how did you avoid accidental bias during seed selection? \n- Theorem 1: It appears to me, that this theorem proves that under the assumption that the underlying algorithm improves performance (Assumption 3), the algorithm will improve performance. But this seems not very surprising. How is this interpretation wrong, and/or in what way is Theorem 1 significant or novel? \n\n\n\n**Questions that did not heavily impact decision:**\n- Line 278: You introduce node statistics but I cannot find where these are defined?\n- Line 285: You state MCTS is used to create a dataset of paths, but it is unclear to me how the proposed approach can generate more than one path? Doesn't the result of the MCTS iterations produce only a single path?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "- The W2SG paradigm seems very interesting, and its extension to sequential decision-making environments, the central focus of this paper, is highly relevant. \n- The main approach, to construct game trees and uses those to select better fine-tuning candidates, is very interesting.", "weaknesses": "- The framing of the paper appears slightly confused. From my understanding, the goal of the W2SG paradigm seems to be around aligning strong models to the preferences and intentions of a weaker model (whilst ideally not losing performance). However, the approaches in this paper don't seem to share this goal. Instead, they seem solely focused on improving the strong agent's performance. As such, the proposed methods don't seem to be part of the core objective of the W2SG paradigm, but the paper _is_ framed in this way. Instead, the approaches are very reminiscent of the field of exploration and MCTS in reinforcement learning, where the objective is to use diverse, exploratory trajectories to improve the performance of a policy in a sequential decision making environment. However, the paper is not framed in this way, and seems to lack positioning within this particular field of research. Therefore, the framing of the paper seems to fit neither of these two paradigms in its current state. \n- It is not possible to judge the significance of the empirical results. There is no mention in the paper of how significance is tested, or how many/which seeds were used. As such, whether the proposed methods actually improve over any of the baselines is currently not possible to judge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)