The State of Reinforcement Finetuning for Transformer-based Agents
Authors: Shengchao Hu
Peng Wang
Guozheng Ma
Shi Fu
Li Shen
Ya Zhang
Dacheng Tao
Venue: ICLR 2026 Poster (2026)
Keywords: Finetuning
Meta-RL
Generative Agents
Primary area: reinforcement learning

Abstract: Reinforcement finetuning (RFT) has garnered significant attention in recent years, particularly for enhancing large reasoning models such as OpenAI o1 and Deepseek R1. The appeal of RFT largely stems from its ability to refine model knowledge, better align outputs with user intent, and address challenges associated with limited finetuning data. Despite these advantages, the application of RFT in large Transformer-based generative agents remains relatively underexplored. Although these agents are designed to address multiple tasks through large-scale autoregressive pretraining and share many properties with large reasoning models, current adaptation strategies predominantly rely on supervised finetuning (SFT). In this work, we conduct a systematic investigation of several RFT techniques across a variety of finetuning parameter configurations and meta-reinforcement learning (meta-RL) environments, employing few-shot offline datasets. We provide a comprehensive analysis of RFT algorithm performance under diverse experimental conditions and, based on our empirical findings, introduce a lightweight enhancement to existing RFT methods. This enhancement consistently improves outcomes by combining the strengths of both SFT and RFT. Our findings provide valuable insights for advancing the effectiveness of RFT approaches and broadening their applicability to meta-RL tasks with large Transformer-based generative agents, motivating further research in broader domains.

Published as a conference paper at ICLR 2026
THE STATE OF REINFORCEMENT FINETUNING FOR TRANSFORMER-BASE AGENTS
Shengchao Hu Hu! Peng Wang Guozheng Ma', Shi Fu', Li Shen', Ya Zhang' Dacheng Tao Shanghai Jiao Tong University, Nanyang Technological University, 3 Shenzhen Campus Sun Yat-sen University
(charles-hu, y aaaagagggg
) 8 jtu. edu. en; wangp3890mail 1 sysu. edu.eni (GUOZHENGO01, shi0l1] Be ntu. edu.
/mathshenli, dacheng. tao @gnail.
ABSTRACT
Reinforcement finetuning (RFT) has garnered significant attention in recent years, particularly for enhancing large reasoning models such as OpenAl ol and Deepseek RI. The appeal of RFT largely stems from its ability to refine model knowledge, better align outputs with user intent, and address challenges associated with lim- ited finetuning data. Despite these advantages. the application of RFT in large Transformer-bat -based agents remains relatively underexplored. Although these agents are designed to address multiple tasks through large-scale autoregressive pretrain- ing and share many properties with large reasoning models, current adaptation strategies predominantly rely supervised finetuning (SFT). In this work, we conduct a systematic investigation of several RFT techniques across a variety of finetuning parameter configurations and meta-reinforcemen learning (meta-RL) environments, employing few-shot offline datasets. We provide a comprehensive analysis of RFT algorithm performance under diverse experimental conditions and, based on our empirical findings, introduce a lightweight enhancement to existing RFT methods. This enhancement consistently improves outcomes by combining the strengths of both SFT and RFT. Our findings provide valuable insights for advancing the effectiveness RFT approaches and broadening their applicability to meta-RL. tasks with large Transformer-base agents, motivating further research in broader domains.
INTRODUCTION
Large Reasoning Models (LRMs), such as OpenAl o1 (Jaech et al. 2024) and DeepSeek R1 Guo al., (2025), represent the leading edge of artificial intelligence, characterized by advanced reasoning and extended deliberation capacities. A key development in these models is the adoption of reinforce- ment finetuning (Lambet et al., 22224: Team et al. 2025), which facilitates efficient adaptation to domain-specifir tasks with limited labeled data. Through iterative
in test-time scaling- RFT enhances reasoning ability, factual accuracy, and alignment with user intent and ethical standards(Kumar et al. 2025). Consequently, RFT has become integral to both the advancement and deployment of large-scale Al systems.
The paradigm of leveraging and transferring knowledge from large-scale pretrained models within LRMs has catalyzed substantial progress across a wide range of domains (Dosovitskiy et al.. 2020 Raffel 2020). Within reinforcement learning (RL), Transformer-based Agents (TAs) have demonstrated exceptional efficacy and generalization by formulating decision-making problems as sequence modeling tasks (Wen et al., 2022 Reed et al. (022). These agents employ Transformer- based autoregressive decoders, which facilitate muuli-modality, multi-tasking, and scalable general- purpose decision-making (Chen al. 202T Janner al., (02T). Notably, TAs exhibit robust generalization to novel tasks through zero-shot and few-shot trajectory conditioning, and share many properties with LRMs (Hu et al. 2024c] AAgawal et al. (003). However, despite these similarities. the application of RFT for adapting TAs to new tasks- tasks-particullrly in the context of meta-RL. RL-remains
"ooresponding author: Dacheng Tao
Thheee authors contributed equally this work.

---

Published as a conference paper at ICLR 2026
underexplored. Current approaches for TA adaptation predominantly rely on supervised finetuning (SFT) (Wang et 2024 Hu et al. 2026), which may limit generalization to new tasks. Meanwhile, RL-based finetuming methods have achieved notable success in non-RL domains, suggesting their potential advantages for RI. tasks, especially those with dense rewards (Schulman et al. 2017). These observations motivate our investigation into whether RL--based finetuning can outperform SFT for adapting TAs to novel RL tasks, and to systematically evaluate its effectiveness in this context.
In this paper, we present a systematic study of RFT for TAs, with a particular emphasis on two promi- nent finetuning paradigms: (1) full-model finetuning (FMT), which updates all model parameters, and (2) parameter-efficient finetuning (PET), which optimizes only a small subset of (often newly introduced) parameters, such as prompt tuning (Hu et al. 2026), low-rank adaptation (LoRA) al. 202T). ). and the decorator approach(Yuar Yuan et al. 2024). We further investigate a range of finetuning algorithms, including: (1) supervised methods such as SFT (Wang et al. 2024, Hu et al. 2026) and DPO (Rafailov et al. 2023), (2) online RI. algorithms such as GRPO (Guo al., 2025 Shao et al. 2024) and PPO (Schulman et al., 2017), and (3) offline RL methods such CQL (Kumar ct al. 2020a). Building on our experimental findings, we also propose a lightweight Q-guided Policy Optimization (QP) that integrates the advantages of SFT with RFT approaches. To this end, conduct extensive experiments to systematically evaluate the interplay between finetuning algorithms and finetuning parameter configurations, providing a comprehensive analysis of their effectiveness for adapting TAs to new RL tasks.
Additionally, we provide in-depth analyses of several key factors that may influence finetuning performance. Specifically, we examine the impact of (1) the quality of the finetuning dataset, (2) the quantity of available finetuning trajectories, (3) the prevalence of sparse versus dense reward signals, and (4) the scale of the pretrained model. By systematically evaluating these variables, we aim to elucidate how different finetuning algorithms and parameter adaptation strategies perform under varying conditions, thereby offering comprehensive insights into their robustness and generalization capabilities across diverse RI. scenarios. Our main contributions are summarized below:
We introduce and rigorously analyze the application of RFT to TAs in meta-RL RL settings, evaluating performance across diverse finetuning parameter configurations and environments.
We propose lightweight enhancement to existing RFT methods by integrating SFT with RFT- based policy updates, resulting in robust and consistent performance improvements.
We conduct extensive empirical studies across various tasks, systematically investigating factors affecting finetuning outcomes, including finetuning trajectory quality, finetuning dataset size, reward sparsity, and pretrained model scale, Our proposed method demonstrates significant improvements over strong SFT and RFT baselines in all settings.
BACKGROUND
2.1 TRANSFORMER SARESOD AGENT
The adoption of Transformer architectures (Vaswani al. 2017) in offline RL sequential modeling (SM) has garnered significant attention in recent years (Hu et al. (024c).. Insights from NLP demonstrate that Transformers, when pre-trained on large-scale datasets, exhibit remarkable sho-shot and zero-shot learning capabilities within prompt-based frameworks (Liu et al. (203b) Brown et al. 2020). Inspired by these findings, recent studies have sought develop sufficiently large agents capable of tackling diverse tasks (Reed et al. 2022, Wen et al. 2022). Many of these works leverage the architectural principles of the prompt- based Decision Transformer (DT) (Xu et al. 2022 Hu et al. 2026), which adapts prompt-based methodologies from NLP to the offline RL setting, thereby enabling few-shot generalization to novel tasks.
Unlike NLP, where prompts are typically textual and often formatted for blank-filling to adapt to various tasks, Prompt-DT introduces trajectory prompts. These prompts comprise tuples of state, action, and retur-to-go (s*,a",,, a", "**, offering explicit guidance to RL agents through small number of demonstration steps. Each element with a superscript * denotes its association with the trajectory prompt. Notably, the length of trajectory prompt is typically much shorter than the full task horizon, encapsulating only the essential information required for effective task identification, yet insufficient for complete imitation of the task.
2

---

Published as a conference paper at ICLR 2026
During pretraining with offline collected data D,, Prompt-DT utilizes Ti, each task Ti. Here, Tut history Ta. and formulated as:
input (T Ts,e) input for
innut consists of the K -step trajectory prompt 7, and the most recent K-step
imput (f7,1= 1 a ......... ....... K S K. ai 4,8 fit K+ Si,t - K+1 ai,t- 1ort-K44 K++ ................................ fice Si, ai,e).
The prediction head linked to state token s is designed to predict the corresponding action a. For continuous action spaces, the training objective aims minimize the mean-squared loss:
LDT E input Di;
Σ K+1
(ar, x ((;; Te.m))?
K
In contrast to LLM-based agent frameworks (Park et al. 2023), where the model interacts with the environment through natural-language actions, the TÄs considered in work operate in continuous- control domains: they consume structured state representations and generate continuous action sequences, and their performance is evaluated by task-specific reward signals.
2.2 REINFORCEMENT FINETUNING
With the advent of advanced reasoning models such as OpenAl's ol (Taech et al. 2024), research
on large reasoning models has increasingly focused on enhancing reasoning capabilities through
RL techniques. Recent studies have demonstrated improved performance in reasoning-intensiv
tasks, including mathematical problem-solving (Cai et al. 2024 Trung et al. 2024 Shao et al. 2024
Yang et al., 2024) and code generation (Hui et al. [2224] Jiao et al. 2024 Zhang et al. 2024a b). A
notable milestone is Deepseek-Be R1 RZ-Zero (Guo et al. 2025 which achieved robust reasoning abilities using RI. alone, entirely omitting the SFT stage. However, most RL-bbased reasoning research has been restricted to the language domain, with limited investigation into Transformer-basee agents in meta-RL settings, where traditional SFT remains the standard for finetuning (Zhao et al., (2022). To address this gap, our work systematically analyzes the efficacy of RFT for such agents.
3 MODEL FINETUNING OVERVIEW
3.1 FINETUNING ALGORITHMS
Given pretrained agent Ty and corresponding few-shot finetuning datasets P; for each task T--where PP:| < |D;| (the dataset used for pretraining strategies can be employed to further adapt the policy via reinforcement learning. For simplicity, although the transformer- based policy æ typically conditions full history, i.e., To (a (fa, S.,, a,--1). here denote it as TT (s). Note that finetuning need not update all parameters; further details are discussed in Section 3.2 For brevity, we use 0 denote the set of parameters subject to tuning in this context. Due to practical constraints, we could not include all prominent RFT algorithms in our experiments and therefore select representative subset of widely used methods.
Supervised Fine- Toni-Tuning (SFT) is the most straightforward way to adapt a pretrained model. It minimizes the mean squared error (MSE) between the predicted and ground-truth actions on the small finetuning dataset P:
LSFT (0) = (s,.))- |a |a- (aass))
(3)
Direct Preference Optimization (DPO) (Rafailov et al. 2023) reformulates reinforcement learning from human feedback (RLHF) direct optimization problem. For each state s, preference pairs (a, a) are constructed, with à representing a perturbed version of a. The policy is optimized to favor preferred actions while maintaining proximity reference model meets with 8 controlling the trade-off:
Ta(a))) Tret (a|s)
(a(s))
(0) -E( logo (glog
Blog
Teat (a|s)

---

Published as a conference paper at ICLR 2026
where a is the logistic function.
Group Relative Policy Optimization (GRPO) (Shao et al. 2024; Guo et al. 2025) compares groups
of candidate responses direetly, eliminating the need for additional critic model. Here we consider the trajectories with the same imitial state as group of candidate responses. The objective encourages higher probabilities for preferred actions, regularized with KL divergence to the reference policy:
(o) (x
(5)
K
To Ta(a"|s)
Ta(aass) Toaa ((**ss))
(min(
As. clip
c, +)4+)) 3DxL [re))
Tha (a))s))
(6)
T Tres (a" log DKL (
Tre (a 5.)
1,
(7)
nol (s)
nu (a (s))
where e and 3 are syper-parameters, and A& is the advantage, computed using a group of rewards (ri, corresponding to the outputs within each group:
mean (fr1, "er Tk])
A std ad((ri, P2. rk)
(8)
Proximal Policy Optimization (PPO) (Schulman et al. 2017) is a widely used policy gradient algorithm. It maximizes clipped surrogate objective ensure stable updates by constraining the policy within a trust region around the previous policy Told:
(0) = (s,
min (ro(als)A, clip (ra(aa)), c, -.
(9)
aas(a]s) the probability ratio between the current and old policies, and Å denotes the advantage estimate, which calculated by the learned critic network with generalized advantage estimation (Schulman al. 2015). Note that, PPO is an on-policy method, its effectiveness is limited in strictly offline settings, but the PPO loss can still be used for gradient computation.
where rals)) ma(ais)
Conservative Q-Learning (CQL) (Kumar al. 2020b) is designed for offline RL, mitigating overestimation by penalizing Q-values for out-of-distributio actions. The loss function combines a Bellman loss with a conservative regularization term:
1
(o) (ss.,,,,'')--
(Q)(s.a) - (r (o(ss
(10)
+ ar (Ba~Ty (s, - (s.
(11)
where y denotes the random policy and a the regularization strength. After learning the Q-function, policy improvement performed by updating the policy Ta maximize the expected Q-value, which can be formulated as:
(c) EE-p [Qo n (s))]
(12)
Here, the policy parameters 0 optimized select actions that maximize the value predicted by the learned Q-network, thereby iteratively improving the policy while remaining robust to the limitations of the offline dataset.
Q-guided Policy Optimization (QP, ours). In RFT methods such as PPO and CQL, a Q-network is often used to estimate the advantage function. However, in offline RL settings, Q-network estimates are often unreliable due to distributional shift, necessitating additional constraints or regularization to prevent divergence from the behavior policy. Conversely, supervised methods can maintain policy stability but are generally limited to in-distribution actions, which may lead to sub-optimal performance. Recent studies have demonstrated the potential extending Q-learning to Transformer- based agents for offline RL, achieving promising results (Hu et al.) 2024a). Building on this line
of research and the detailed results in Sections and 6 we propose Q-guided policy optimization (QP), which augments the objective of supervised methods by incorporating learnd Q Qeneework. Specifically, we augment standard supervised training (c.g.. SFT or DPO) with policy improvement term guided by the Q-network. This approach, denoted QP-SFT QP-DPO, aims to combine
4

---

Published as a conference paper at ICLR 2026
the distributional robustness of supervised methods with the policy improvement capabilities of RL, thereby enhancing both stability and performance in offline settings:
(QP- = (x.a))>
-T((aass) - a Qo(s, TT(s))
(13)
(or) = E (s.a.a)>p log
BBlgg
To (a(s)) B log To(a
Qu(s, mo(s))
(14)
(als)
Teet (āls)
where a controls the regularization strength and Qo is trained via a TD-loss with double Q- learning 2010):
)( Qm - Qo, (s.a))
(15)
where Qm = min Qes (s)
(16)
1=12
where 7 is the discount factor and denotes the predicted action output by the target model Tgr. The first term in Equations 13 14 corresponds to the standard imitation (SFT DPO) objective, anchoring the updated policy to the behavior distribution. The second term provides a value- based correction that encourages the policy select actions with higher predicted return under Qa- In this way, QP augments the stability of supervised learning with conservative value-guided policy improvement, enabling the policy to extrapolate beyond the behavior policy while mitigating the risk of distributional drift in offline settings.
3.2 FINETUNING PARAMETER CONFIGURATIONS
In the context of finetuning large Transformer-based agents for RI., various parameter-efficient and full-model adaptation strategies can be considered. Below, detail several approaches for finetuning different components of the agent, as illustrated in Figure 0 and further described in Section D
Prompt Tuning. This approach updates only the prompt parameters- -tpically initialized from sampled task trajectories while keeping the backbone model fixed (Hu et al., 2026 20256).
Prompt tuning enables efficient few-shot adaptation with minimal risk of overfitting.
Adaptor Tuning. Following Huang et al. (2024), this method inserts lightweight LoRA modules, typically into the MLP layers of the Transformer (Lawson & Qureshi 20244 Hu al. 2024e). Only the adaptor parameters are updated for each new task, allowing efficient and isolated adaptation without affecting the shared backbone.
Decorator Tuning. Inspired by residual policy learning (Yuan et al. 2024). this strategy introduces a residual policy Ties trained on top of frozen base policy Thase. The action taken is the of both policies: Toae (s) + Ties(s), enabling targeted adaptation while retaining prior knowledge.
Fullmodel Tuning. All parameters of the agent are finetuned jointly. While this maximizes adaptation capacity, it increases computational cost and overfitting risk in low-data regimes.
4 EXPERIMENTAL SETUP
Environments. We evaluate all proposed methods on two standard meta-RL benchmarks: (i) the multi-task MuJoCo locomotion suite (Ni et al. 2023 Todorov al. 2012), which includes the Cheetah-Dir, Cheetah-Veis and Ant-Dir; and (ii) the MetaWorld robotic manipulation platform (Yu al. 2020). comprising 50 distinct tasks. For the MuJoCo locomotion suite, we follow the protocol of Wang et al.] (2024) by randomly sampling tasks from the overall distribution and partitioning them into a training set (T and test set (Ttsst. In MetaWorld, 45 tasks are designated for pretraining, while the remaining 5 tasks are reserved for meta-testing and adaptation. To construct the datasets, we employ Soft Actor-Critic (Harroaa et al., 2018) to independently train single-task policies for each training task. We then generate types of offline datasets Medium and Expert-oon corresponding to different levels of policy proficiency. Further details regarding the environments and dataset construction provided in Appendix [
Training. We instantiate Prompt-DT as our TA backbone, implemented on top of the soun-source minGPT codebase. For each environment, we first pretrain a policy on the offline dataset D,
5

---

Published as a conference paper at ICLR 2026
Table I: Performance of different finetuning algorithms meta-RI. environments using expert-level offline datasets. Each method is evaluated with 50 finetuning trajectories, and results are averaged over three independent runs with different random seeds.
Zero-shot
SPT
DPO
GRPO
PPO
COL.
QP-DPO
QP-SFT
Prompt
AmDDi
258.61
326,52:xxoz
319, 05
323 323,07=
316 53
325 93
325, 325.10e 14
325.35
HallCChettshDir
600.
631
634.608.1.0
638 38
629. 12
628 52
645.24 646.24,aaa
639 26
Vel
-138.01
-111,57
113.79
-10966--2.0
-114.722000
111 g...2
-2.2.
MetaWorld
414.22
414.22g000
414.22_p00
414.22
414 22
414.22g9.00
414.2244.00
414.224u.00
Average
284.08
315.05
313.73
315.47
312,55
313.49
318.39
317.07
Adaptor
315.06 62
325,81 325.81_5..
.1.0
335.5 305.5718m
336.62,2.7
329.30 (19.336,..0
HallCChettahDir
600
631,06
637.39
632.83 23.37
633 242.3
636 23,.00
631 22.47
640.60,2.73
HallCCheesah
-138.01
-118.98
-115.92a359
-109.74:3.0
-113.51 10..5
-119.11a1.s
11133 22:3.47
-112 19
MetaWorld
414.21
480 (2222.0
485.63211.21
490.21 490.21
422.91
437.28
502.52
486.71
Average
284.08
331.69
333.38
333.24
327.70
321.63
339.46
335.87
Decorator
AmDDic
259.61
12
332.00_
317.25_..
306.06,1790
123 123.07
330.87 1.
HalfCheetthDDi
600.49
632.30
639 639.85.:411
629,88
636.
637
63V, 603.10
640.64;2.33
HolfChhettskVel
-138.01
118. 46
121.07
1188522
-1:7.13
109.65
20
-112.90p3.46
MetaWorid
414.21
452 17
407.78_5.44
113.52 5.36
468. 495.47
108 (0.m9
473.35_
478.03 -:4.0
Average
281.08
321.73
314.66
313.01
323.03
322,83
330.08
334.16
Fullmodee
AntDi
250.61
13
300.96
319.69
335.06 10
304.66 to
37..08
412.18 122.1_:66.0
HalfChhesthhi
600.49
634.61a122
643.79
630.08
634 82 42
627 027.78ga.u
650.84
642.62 642.0222..
HalfCheetaaVell
- -138.01
133.02
122. -122.11g
-119.87
-200.77
-117:2223:46
-121 291
-119,79
MetaWorid
414.21
441.05
424.85 197.8
471.02
468.97
434.59
$48.95_100
$53.36 _,3.00
Average
284.08
336.07
326.33
325.23
329.62
312.53
362,13
372.09
which aggregates trajectories from multiple distinct tasks. The detailed model architecture and hyperparameters are provided in Appendix C Subsequently, for each finetuning algorithm, we initialize from the same pretrained policy and apply the four finetuning parameter configurations described in Section 3.2, utilizing the same finetuning trajectory set P (PI = 50) and maintaining identical finetuning iterations to ensure fair comparison. Notably, the number of trajectories available for finetuning is substantially smaller than that used during pretraining (IP) < [D),, corresponding to only 0.1 1%1% of the pretraining data), thereby establishing a few-shot adaptation setting that more closely reflects practical, real-world scenarios. For each loss function, all methods are tuned via grid search over the corresponding hyperparameters to ensure that the reported results represent their best achievable performance. For each method, select and report the best- pess-peeforming checkpoint over training, which accounts for potentially different convergence dynamics while preserving fairness in the comparison.
5 THE STATE OF REINFOR FINETUNING
Here, we show the empirical results for reinforcement finetuning in the pretrained transformer-bart based agents in Table | and Figure 8 Additional experimental results, thcoretical support, and extended discussions are provided in Appendix B and Appendix [F]
The effect of finetuning parameters. We compare four finetuning parameter leagguurttinoPoonore (0.76KB), Adaptor (0. 19M)). Decorator (0.54MB), and Fullmodel (2.52MB) to investigate the impact of parameter count on finetuning efficacy. Experimental results across meta-RL environments indicate that finetuning performance is highly sensitive to the choice of parameter configurations. While Fullmodel finetuning-upde scores for both QP-DPO and QP-SFT, not uniformly optimal. Performance exhibits clear method preference: supervised approaches (SFT, DPO) generally benefit from full-model updates, whereas RL-based methods (e.g.. GRPO, CQL) often perform best under parameter-efficient schemes (Adapters, Decorators). This dichotomy reflects differences in gradient supervision and signal propagation across learning paradigms. In supervised settings, direct imitation of provided trajectories preferences is most effectively achieved through full-model finetuning, which facilitates comprehensive signal propagation across all network layers and typically yields stronger convergence and generalization (Mandlekar et al.] 2021). Conversely, RL finetuning operates inherently noisy, high variance reward signals, where update errors compound over extended horizons, causing full-model parameter updates exhibit unstable oscillatory behavior (Kumar al. 2022). In
the largest parameter set-achieves the highest average

---

Published as a conference paper at ICLR 2026
Aatter
Necweel
Medium Dataset
Export Dataset
10
Nicasteed Pertrante
08
0.0
Figure 1: Comparison of finetuning dataset quality across different finetuning algorithms in ml. settings, under a fixed finetuning budget of 50 trajectories. Performance is normalized such that the highest score for each setting is scaled to I.
such regimes, parameter-efficient finetuning restricts the update subspace, mitigating overfitting to stochastic rewards and improving stability and sample efficiency. Moreover, the results suggest that Prompt-based finetuning, while parameter-effiiient tends to underperform in high-variance or complex environments (e.g.. MetaWorld), likely due its limited expressive flexibility. In the Meta World setting, the pretrained agent possesses only limited prior knowledge about the unseen downstream tasks, and updating solely the prompt parameters does not provide sufficient capacity to substantially reshape the policy. Consequently, all finetuning objectives constrained to optimize the same narrowly scoped parameter subset, and thus tend to converge to similar performance levels.
The effect of finetuning algorithms. Empirical results across diverse meta-RL environments and parameter configurations consistently demonstrate the superiority of QP-based finetuning algorithms (QP-DPO, QP-SFT) over both supervised methods (SFT, DPO) and RL-based approaches. QP variants frequently achieve the highest or near-highes performance across various tasks and finetuning parameters, highlighting their robustness and broad applicabillty. By optimizing policies with respect to soft value estimates or preference-guided advantages, QP methods combine the stability and sample efficiency of supervised learning with the adaptability and asymptotic performance of RL finetuning, In contrast, pure RL methods such as PPO and CQL, while occasionally competitive, often exhibit instability and lower average performance. This is likely attributable to the lack of strong inductive priors from imitation- based objectives and the high variance in gradient estimates, particularly in few-shot demonstration regimes. Notably, in certain tasks such as Meta World with Adaptor finetuning parameter-the choice of finetuning algorithm can yield greater performance improvements than merely increasing finetuning parameter size. For example, with PPO, switching the finetuning parameter from Adaptor to Decorator results in modest 3% performance gain, whereas changing the algorithm to QP-DPO yields a 10% improvement. These findings underscore the critical importance of algorithm selection in meta-RL settings.
No single algorithm consistently achieves optimal performance across all finetuning parameter configurations and environments, underscoring the need for adaptive finetuning strategies. Moreover, the choice of finetuning algorithm has a substantial impact on overall performance, often comparable to or greater than the effect of finetuning parameter scale.
Takeaway:
DISCUSSION
Ablation on finetuning data quality and quantity. We We jointly analyze the effects of dataset quality (Medium vs. Expert) and the number of finetuning trajectories on seven algorithms (as shown in Figure [ and Figure 2). Two consistent patterns emerge. First, quality: QP-DPO and QP-SFT dominate across both medium- and expert-quality regimes. RL-based methods (e.g., GRPO, PPO) exhibit pronounced gains under expert data, suggesting stronger sensitivity to high-quality demonstrations and effective credit assignment when optimal trajectories are present. In contrast, supervised methods benefit relatively more from medium quallity datasets that contain a larger proportion of sub-optimal trajectories, providing stable imitation- lower bound on policy performance. Second, quantity: increasing the number of finetuning trajectories typically improves performance for all methods, underscoring the centrality of data availability for policy adaptation. Notably, QP-based methods
7

---

Published as a conference paper at ICLR 2026
IPO
OPO
AdtOr (Medsaen)
HoRChertahtlr (Modhan)
Noithherzkkeel (Modiar))
Resstterdd (Modkas))
Astovr Nutthwetttr
wuttchenawe
Teatay Suiter
Tripddar
Figure 2: Impact of the number of finetuning trajectories different algorithms acTOSs various environments, evaluated with both Medium and Expert datasets. Scores are averaged over multiple finetuning parameter configurations for each algorithm within each environment.
DPQ
GIPO
193
QP-PDD
OP-ST
Autor Itspart
sortChertabDir (Esper)
HafChertaivel (Esart
Hics (Espert
Thalcter Sunber
Teacary Nircer
Phlettrr sunce
Figure 3: Impact of the number of finetuning trajectories on the performance of various algorithms across multiple environments, assessed under both Medium and Expert dataset conditions in the sparse reward setting. Scores are computed as the average across multiple finetuning parameter configurations for each respective finetuning algorithm within the environment.
sustain superior results in low-data settings, while purely supervised approaches degrade more sharply when trajectories are scarce, indicating higher sample sensitivity.
Takeaway: (i) Supervised approaches yield stable, imitation-driven performance- when data quality is mixed and the number of trajectories exceeds moderate wchre- RL-based methods can surpass them given expert demonstrations and are comparatively ad- vantageous in low-data regimes; (ii) Increasing the number of finetuning trajectories does not universally lead to performance improvements, the effect depends on the choice of finetuning algorithm and environment; and (iii) Across regimes of both quality and quantity, QP-based methods (QP-DPO, QP-SFT) exhibit mostly improvement trends, showing the robustness.
Ablation study of the sparse reward setting, To assess the robustness learning algorithms under challenging feedback conditions, evaluate model performance in sparse reward environments, where agents receive only a cumulative reward at the final timestep mirroring recent RL. finetun- ing practices in LLM reasoning (Guo et al. 2025 Swamy et al. 2025). As shown in Figure 3 sparse rewards significantly widen performance disparities both across algorithms and data regimes. QP-DPO and QP-SIT exhibit greater robustness, maintaining stable performance with minimal degradation, which we attribute to their policy optimization mechanisms that effectively combine the strengths of SFT and RFT. In contrast, GRPO suffers from persistent underperformance. likely due to optimization challenges and insufficient sample efficiency in offline settings. On the other hand, we occasionally observe that RFT methods achieve higher performance under sparse rewards than under dense rewards. This does not contradict the usual benefits of dense rewards in interactive RL, but reflects a specific failure mode in our low-data offline regime. In the dense case, TD-style value and advantage estimates aggregate bootstrapped targets at every timestep; with very few trajectories and poor satteacction coverage, approximation errors of the learned value function at intermediate states repeatedly propagated backward, leading to severely biased advantages. In the sparse construction, we keep the same total return but concentrate at the terminal step, so the value target is dominated by single, well-defined terminal return. Under data scarcity, this makes the update
8

---

Published as a conference paper at ICLR 2026
Medium
Expert
140
0F:000
Sparse
Figure Performance comparison of various methods under both sparse and dense reward conditions. Each method is evaluated with 50 finetuning trajectories, and results are averaged across multiple environments and a range of finetuning parameter configurations.
more Monte Carlo-like, reduces sensitivity to value-function approximation error, and can yield relatively less noisy and more stable advantage estimates.
models trained on expert data con- sistently incur less performance loss than those trained on medium data, highlighting the value of optimal demonstrations when reward signals are limited. Moreover, while the relative trends between supervised and RL-based approaches are similar to those observed in dense reward settings, their magnitudes diverge. low-data sparse settings, RL.-based methods (e.g., PPO) demonstrate smaller improvements as the number of trajectories increases, indicating that the absence of dense feedback limits the benefits additional data and constrains policy improvement. Conversely, supervised approaches show more stable and monotonic gains, their objective is insensitive reward spar- sity. In high-quality settings, however, RFT methods continue to achieve comparable asymptotic performance, reflecting their capacity exploit optimal trajectories more effectively.
Data quality proves particularly critical in the sparse regime:
comparative summary across dense and sparse reward conditions (Figure (4) reinforces these observations. QP-based methods remain consistently superior and robust across feedback structures, with their performance margin over alternative approaches more pronounced in the sparse than in the dense setting. By contrast, GRPO attains competitive results primarily under dense rewards but degrades markedly under sparse supervision, decline plausibly attributable to inaccurate group- advantage estimation the absence of frequent intermediate feedback.
Takeaway: In sparse reward environments, (i) data quality plays pivotal role- expert trajecto- rics substantially mitigate performance degradation; (ii) RFT methods achieve higher asymptotic returns given high-quality data, reflecting their ability to exploit optimal demonstrations; (iii) supervised approaches provide more stable gains across data regimes, being less sensitive to reward sparsity; and (iv) QP-based methods demonstrate superior robustness across both dense and sparse settings, confirming their adaptability to different feedback structures.
Ablation study of scaling pretrained agents. We examine the effect of model scaling on downstream finetuning performance in the World benchmark by systematically comparing four finetuning parameter configurations under both Medium and Expert data regimes. In certain configurations, such as Adaptor and Fullmodel, the number of trainable parameters increases proportionally with the size of the pretrained agent, whereas in others, such as Prompt and Decorator, the parameter count remains constant regardless of model size. Additional details provided in Appendix F
Empirical results in FigureBJindicat that most finetuning strategies benefit from larger pretrained agent sizes, exhibiting consistent performance improvements model capacity increases. However, in the case Fullmodel finetuning, the substantial rise in trainable parameters associated with larger models- models-combinedd combined with limited finetuning even degradation. These findings underscore the necessity of balancing model capacity with the availability of high-quality supervision when developing scalable finetuning strategies.
results in performance plateaus or

---

Published as a conference paper at ICLR 2026
Prompt (Medium)
Prompt (Expert)
790
Adaptor (Medium)
Adaptor (Expert)
Decorator (Medium)
Decorator (Expert)
Fullmodel (Medium)
490
Fullmodel (Expert)
33500
0.50
CM 0,50M
0.504
12.40M
43.224
37.524
Brnturing Tevereders
Figure 5: Performance of QP-DPO under various finetuning parameter configurations across different pretrained agent sizes in the MetaWorld benchmark. For certain configurations (c.g., Adaptor, Fullmodel), the number of trainable parameters increases with the size of the pretrained agent, while for others, the parameter count remains constant.
Takeaway: Training instability and overfitting can arise when the parameter space is expanded without sufficient data, as the resulting optimization problem becomes increasingly challenging to solve effectively.
CONCLUSION
In this work, we investigate the state of RFT for TAs in meta-RL settings. From an empirical perspec- tive, we evaluated the performance of various RFT methods across range of finetuning parameter configurations within meta-RL environments. Furthermore, we systematically examined the impact the quality and quantity of finetuning data, reward sparsity, and pretrained
of key factors-included model scale-on on downstream performance. Additionally, we introduce lightweight extension that integrates SFT with RFT-based policy improvement. This approach mostly demonstrated superior results across all evaluated scenarios.
We envision this study as foundation for future research on RFT algorithms for TAs, particularly in real-world applications (e.g.. robotics, autonomous driving, and safety-critical domains) where online exploration is expensive or risky. In such settings, collecting small set of nigh-quality expert trajectories is often more practical than running large-scale online RL. By combining supervised imitation with value-guided policy improvement, QP-based methods help bridge offline RI., imitation learning, and LLM-style post-training for embodied and multi-modal agents, thereby providing principled and practical pathway toward robust TA adaptation in complex environments.
Limitation. Although this work investigates RFT for pretrained TAs, the largest models evaluated smaller than contemporary large language models
have up 40 million with hundreds of millions or billions of parameters. While sizable relative to traditional RL agents, this scale may not fully capture the behaviors of larger models. Future work incorporating larger agents may yield different insights into the effectiveness of RFT.
ACKNOWLEDGMENTS
Dr Tao's research is partially supported by NTU RSR and Start Up Grants. Li Shen is sup- ported by National Key R&D Projects (NO. 2024YFC3307100 100), NSFC Grant (No. 62576364), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO. JCYJ20221220114443000 CCF-DiDi GAIA Collaborative Research Funds (NO. CCF-DiDi GAIA 202508).
REFERENCES
Pranav Agarwal, Aamer Abdul Rahman, Pierre-Lue St-Charles, Simon JD Prince, and Samira Ebrahimi Kahou. Transformers reinforcement learning: a survey. arXiv preprint 007599230..05599 2023.
10

---

Published as a conference paper at ICLR 2026
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision maiiin?? arXiv preprint arXiv:2211.156577 2022a.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022b.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-con conditiond supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 15:1542-1553, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internim2 technical report. arXiv preprint arXiv:2403. /7297, 2024.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu, Offline reinforcement learning via high-fidelity generative behavior modeling, arXiv preprint arXiv:2209. 14548, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 14045-15050997 2021.
Juntao Dai, Taiye Chen, Yaodong Yang. Qian Zheng, and Gang Pan. Mitigating reward over- optimization in rlhf via behavior-supported regularization. arXiv preprint arXiv:2503.181300 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1I 11999, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1135-1155 PMLR, 2017.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:201322200111111 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Yunkai Gao, Rui Zhang, Jiaming Guo, Fan Wu, Yi, Shaohui Peng. Siming Lan, Ruizhi Chen, Zidong Du, Xing Hu, et al. Context shift reduction for offline reta-reeiforrement learning. Advances in Neural Information Processing Systems, 36:80024-80043, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprins arXiv:2501. 2025.
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1874 Pmlr, 2018.
Hado Hasselt. Double q-learning. Advances neural information processing systems, 23, 2010.
Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, et Dita: Scaling diffusion transformer for generalist lanoon-lanuag--actiio policy. arXiv preprint 2rXiv:2503.1955: 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685, 2021.
11

---

Published as a conference paper at ICLR 2026
Shengchao Hu, Ziqing Fan, Chaoqin Huang, Li Shen, Ya Zhang. Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. International Conference on Machine Learning, pp.
PMLR, 2024a.
Shengchao Hu, Ziqing Fan, Li Shen, Ya Zhang, Yanfeng Wang. and Dacheng Tao. Harmodt: Harmony multi-task decision transformer for offline reinforcement learning. In International Conference on Machine Learning, pp. 19182-19197. PMLR, 2024b.
Shengehao Hu, Li Shen, Ya Zhang. Yixin Chen, and Dacheng Tao. On transforming reinforcement learning with transformers: The development trajectory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):8580-855) 8599, 2024c.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. The Twelfth International Conference on Learning Representations, 2024d.
Shengehao Hu, Yuhang Zhou, Ziqing Fan, Jifeng Hu, Li Shen, Ya Zhang, and Dacheng Tao. Continual task learning through adaptive policy self-composition. arXiv preprint arXiv:2411.1136 2024e.
Shengchao Hu, Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer for offline reinforcement learning. Science China Information Sciences, 68(6))169101 2025a.
Shengchao Hu, Wanru Zhao, Weixiong Lin, Li Shen, Ya Zhang, and Dacheng Tao. Prompt tuning with diffuusion for few-shot pre-trained policy generalization. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pp. 2556-2558, 2025b.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Prompt tuning with preference ranking for few-shot pre-trained decision transformer. Science China Information Sciences, 69(1):112105 2026.
Kaixin Huang, Li Shen, Chen Zhao, Chun Yuan, and Dacheng Tao. Solving continual offline reinforcement learning with decision transformer. arXiv preprint arXiv:2411.08478. 2024.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1283 2021.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint (tti::2005009911 2022.
Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F Chen, Shafiq Joty, and Furu Wei. Preference optimization for reasoning with pseudo feedback. arXiv preprint 2.1::2411.16345 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint 10X0v:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-1199 2020a.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 13:1179-1191, 2020b.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.056// 2022.
12

---

Published as a conference paper at ICLR 2026
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Haae Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. arXiv preprint at arXiv:2502.21321, 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T, ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 1.1424, 2024.
Daniel Lawson and Ahmed H Qureshi. Merging decision transformers: Weight averaging for forming multi-task policies. In 2024 IEEE International Conference on Robotics and Aulomation (ICRA), IEEE, 2024.
pp. 12942-12948.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint wrXiv:2005.01643 2020.
Lanqing Li, Rui Yang, and Dijun Luo. Focal: Efficient fully-offine meta-reinnfrrcement learning via distance metric learning and behavior regularization, arXiv preprint arXiv:2010.0/112, 2020.
Lanqing Li, Hai Zhang, Xinyu Zhang, Shatong Zhu, Yang Yu, Junqiao Zhao, and Pheng-Am Heng. Towards an information theoretic framework of context-based offline meta-reinforcement learning. arXiv preprint arXiv:2402.02422) 2024.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-4491, 2023a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly adversarial regularizer. Advances in Neural Information Processing Systems, 37:1 13663-137717 2024.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03296 2021.
Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 2023.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.
Fei Ni, Jianye Hao, Yao Mu. Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pp. 26087-26105. PMLR, 2023.
Joon Sung Park, Joseph O' Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pp- 1-22, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5323 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 167,, 2020.
13

---

Published as a conference paper at ICLR 2026
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-policy mett-reinforremmet learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340. PMLR, 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv22005.06177 2022.
Thomas Schmied, Markus Hofmarcherr Fabian Paischer, Razvan Pascanu, and Sepp Hochreiter. Learning to modulate pre-trained models in rl. Advances in Neural Information Processing Systems, 36:38211 36:38231-38265 2023.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0240 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06447 2017.
Kajetan Schweighofer. Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-Zadeh. and Sepp Hochreiter. A dataset perspective on offline reinforcement learning. In Conference Lifelong Learming Agents, pp. 470-517. PMLR, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Luong Trung Xinbo Zhang, Zhanming Jie. Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601-7614, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.06199 2022.
Zhi Wang, Chunlin Chen, and Daoyi Dong. Lifelong incremental reinforeement learning with online bayesian inference. IEEE Transactions on Neural Networks and Learning Systems, 33(8): 4003-4016 2021.
Zhi Wang, Li Zhang, Wenhao Wu, Yuanheng Zhu, Dongbin Zhao, and Chunlin Chen. Meta-dt: Offline meta-rl conditional sequence modeling with world model disentanglement, Advances in Neural Information Processing Systems, 37:44845-44877 2024.
Ying Wen, Ziyu Wan, Ming Zhou, Shufang Hou, Zhe Cao, Chenyang Le, Jingxiao Chen, Zheng Tian, Weinan Zhang, and Jun Wang. On realization of intelligent decision-makinn in the real world: A foundation decision model perspective. arXiv preprint arXiv:22122 220012.22220 /2669, 2022.
14

---

Published as a conference paper at ICLR 2026
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631-24645. PMLR, 2022.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning.
Advances in neural information processing systems, 31, 2018.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer:
Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference Machine Learning, pp. 38989-39007. PMLR, 2023.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint 2rXiv:2409./2122 2024.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1994-1100 PMLR, 2020.
Haoqi Yuan and Zongqing Lu. Robust task representations for offline meta-reinforcement learning via contrastive learning. In International Conference Machine Learning, pp. 25747-25759. PMLR, 2022.
Xiu Yuan, Tongzhou Mu, Stone Tao, Yunhao Fang, Mengke Zhang, and Hao Su. Policy decorator: Model-agnostic online refinement for large policy model. arXiv preprint arXiv:2412. 13630, 2024.
Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. Codedpo: Aligming code models with self generated and verified source code. arXiv preprint arXiv:2410.056005 2024a.
Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. ol-coder: ol replication for coding, arXiv preprint arXiv:2412.00154, 2024b.
Mandi Zhao, Pieter Abbeel, and Stephen James. On the effectiveness of fine- ine-tuning versus meta- reinforcement learning. Advances in neural information processing systems, 35:26519-26531, 2022.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research, 22(289):1-39 2021.
15

---

Published as a conference paper at ICLR 2026
The Use of Large Language Models. In this work, exclusively employ large language models (LLMs) to refine the writing and presentation of our manuscript.
A RELATED WORK
Offline Reinforcement Learning. Offline RL trains policies solely from static datasets, without further environment interaction (Levine et al. 2020). This is especially valuable when interactions are costly or risky. However, offline RL faces challenges from distribution shift between the behavior
and learned policies, often leading to performance drops (Fujimoto et al. 2019). Approaches such as constrained or regularized dynamic programming help mitigate this issue (Fujimoto & Gu [
Kumar et al. 2020a, Kostrikov et al., 2021). Conditional sequence modeling offers a supervised
alternative, predicting actions from past state-action-
sequences and keeping the learned policy close to the data distribution (Chen et al.) 2021 Hu et al. 2025a 2024a] Yamagata et al., 2023
Hu et al. 2026; 2024d| Meng et al. 2023). This LLM-inspired paradigm enables scalable RL with
large data and compute. Diffusion models have also been adopted for offiine RL, using generative modeling techniques to represent policies or dynamics and achieving strong empirical results (Janner
et al. 2022 Ajay et al., 2022a Chen et al. 2022 Wang et al. 2022). Related work by Kumar et al. (2022) examines when offline RL outperforms behavioral cloning (supervised fine-tuning), addressing a question closely aligned in our motivation. Their focus is standard offline RL benchmarks and conditions under which policy improvement surpasses imitation, whereas our focus is met a pretrained, Transformer- baaddd based agent must adapt to unseen tasks using only few-shot, offline data. The two perspectives are therefore complementary- -urs emphasizes rapid cross-task adaptation under strict data constraints, while theirs characterizes the policy-improvement vs. imitation trade-off within a single-task offline regime.
Learning. Offline meta- RL seeks to generalize to tasks by training on a distribution of offline tasks (Gao et al.] 2023 Ni et al. 2023). Optimization-base
Offline
methods (Finn) ctt al. 2017 Xu al. 2218)) Mitchell | 2021) 1) and context-based approaches (Rakelly al.
2019 Zintgraf et al. [021]] et al. 2020 Yuan & Lu 2022 Gao et al. 2023 Li et al. 20244 have
both been explored, typically relying temporal difference learning. However, these methods often face instability due to the "deadly triad" and depend on hand-crafted heuristics to remain
within the offline dataset distribution (Brandfonbrener et al. 2022 Waang et al. 2021] Ajay al.
2022b). From the perspective of finetuming, [Schweighofer et al. (2022) study dataset quality metrics in offline RL, quantifying trajectory returns and state-action coverage of behavior policies and relating these statistics to the performance of classical offline RI. algorithms trained from scratch. Schmied et al. ( propose the Learning-to-MM ModMulllaee (1.2M) framework, which explores parameter- efficient finetuning and prompt-based tuning for continual-task RL in benchmarks such as Meta World, DMControl, and Continual-
with the goal of mitigating catastrophic forgetting while adapting pretrained models to new tasks. In this work, we focus on RFT for pretrained transformer-basee agents in renforeeinor learning tasks. This approach contrasts with previous studies that primarily seek to modify model architectures introduce novel optimization methods.
B ENVIRONMENTS AND DATASETS
In this section, we show details of evaluation environments over variety of testbeds, as well as the offline dataset collection process conducted on these environments.
Following established practice offline meta-RL (Yuan & Lu 2022 Mitchell et al. 2021 Wang et al. 2024), we adopt two classical benchmarks: multi-task MuJoCo control (Ni et al. 2023 Todorov et al. 2012 and Meta World YYu et al., 2020). All methods are evaluated on the following environments:
MuJoCo Multi-Task Control: We use Cheetah- Vel, Cheetah-Dir, and Ant-Dir, where task variations arise from differing reward functions.
- Cheetah- Vel requires a planar cheetah to achieve target velocities sampled uniformly from U|00075, 3.0], with rewards based on proximity to the goal velocity.
Cheetah- Dir and Ant-Dir involve controlling a cheetah or quadruped ant to move in specific directions, with rewards proportional the cosine similarity between the velocity and goal
-
16

---

Published as a conference paper at ICLR 2026
Table 2: TQ and SACo of finetuning datasets with 50 trajectories per task.
Enviromments
Quality
TQ
SACp
AniDir
Expert
1.00
0.24
AntDir
Medium
HalfCheetthhDr
Expert
0.99
0.27
HallChhetthhDir
Medium
0.12
HaltChhctthhVll
Expert
0.99
0.50
HalfCheetahNel
Medium
0.21
0.09
MetaWorld
Expert
1.00
0.04
MetaWorid
Medium
0.34
0.06
direction (sampled from 010 2n] for Ant-Dir and limited to forward/backkwrd for Cheetah- Dir). Each episode is limited to 200 steps.
Meta World: MetaWorke comprises 50 diverse robotic manipulation tasks with shared dynamies, where a Sawyer robot interacts with objects of varying shapes and mechanisms. Tasks differ in state and reward structures, requiring the robot to achieve different goals using a 4-dimensional action space (3D end-effector position and gripper control). Performance is measured by the average success rate across all tasks.
For both MuJoCo and MetaWorld, we 45 tasks for training and held-out tasks for evaluation.
For the MuJoCo benchmark, we generate offline datasets by training separate policy for each task using the Soft Actor-Critic (SAC) algorithm (Haarnoja al. 2018). Policy checkpoints are periodically saved to produce different dataset types:
Medium: Trajectories generated by a policy achieving between one-third and one-half of expert
performance.
Expert: Trajectories generated by the final, converged expert policy.
We collect 100 trajectories per task for each dataset type. For MuJoCo, this yields a pretraining corpus of 4,500 trajectories, whereas finetuning is performed with only 50 of the pretraining data catt--plccing the evaluation firmly in a few-shot adaptation regime.
1.1%
For MetaWorld, we follow the procedure of Hu et al. (2024b), training task-specific SAC policies to convergence. Offline datasets are constructed by sampling million transitions per task from the SAC replay buffer, capturing the training process up to convergence. We consider two dataset compositions:
Consists of the first 50% trajectories (50 million transitions), representing early-stage learning with reduced expert-level behavior.
Medium:
Consists of 100 million transitions per task, spanning from random initialization to expert-level performance.
Expert:
For fair comparison between data qualities, we subsample 1,000 expert trajectories to match the medium dataset size. The resulting pretraining corpus contains 45,000 trajectories, while finetuning again uses only 50 eejectories-aaproxiinae an even more stringent few-shot adaptation regime.
0.1% of the pretraining data-thus corresponding to
We report the TQ and SACo metrics proposed by Schweighhoer et al. (2022) to quantitatively characterize the offline finetuning datasets used in our experiments. For each benchmark environment, we compute and average TQ and SACo over the 5 meta-test tasks, using 50 trajectories per task under both Expert and Medium demonstration regimes. For metric normalization, we use the full collected dataset (Expert Medium) as the reference buffer (e.g.. 200 trajectories for HalfCheetahDir and 2000 trajectories for MetaWorld), In contrast to Schweighofer et al. (2022), we do not sweep random seeds when constructing the datasets; instead, we report single-seed metric values. As summarized Table Expert datasets consistently lie in a high-TT TQ regime (TQ z 1.0) across all environments, whereas Medium datasets exhibit substantially lower TQ (TQ 0.21-0.53). SACo values are generally low due to the limited number of finetuning trajectories, with particularly low coverage observed in the Meta World benchmarks.
17

---

Published as a conference paper at ICLR 2026
Upcated tnjectery
Outppe
Caunal Transforme
Cauual
Causall Transform
Tranaformer
Pollcy
Causal
Input
Ispat
Input
Transtomer based Agents
Prompt Tuning
Adaptor Tuning
Decorator Tuning
Fullmodel Turiig
Figure 6: Finetuning parameter configurations. The left panel depicts the Transformer- based Agents, while the four panels on the right illustrate the configurations for Prompt Tuning, Adapter Tuning, Decorator Tuning, and Fullmodel Tuning, respectively.
C IMPLEMENTATION DETAILS
We build our policy as Transformer-based model, which based on minGPT We vary the number of Transformer layers, hidden dimensions, and attention heads depending on the scale of the pretrained (from (3, 128, 1) to (48, 256, 16)) model.
Pretrained Agents.
open-source code |
Q-networks. For finetuning algorithms that require acte-nction value estimation, instantiate the Q-function as a three-layer multilayer perceptron with Mish activations and hidden width of 256 units per hidden layer.
Compute setup. All finetuning experiments are conducted on a single NVIDIA RTX 4090 GPU. For pretraining, we use four RTX 4090 GPUs in parallel to accelerate training as the scale of the pre-trained agents increases.
D FINETUNING PARAMETER CONFIGURATIONS
In Section 3.2
we introduce four distinct finetuning parameter configurations. Here, we elaborate on these approaches in greater detail, as illustrated in Figure 6
Prompt Tuning, Prompt tuning involves updating only a small set of prompt parameters, which are typically initialized from sampled task trajectories P, while keeping the underlying model backbone fixed (Hu et al.) 20266] 2025b). This approach enables highly efficient adaptation to new tasks, particularly in few-shot learning scenarios, with minimal risk of overfitting due to the limited number of trainable parameters. Prompt tuning is especially advantagcous when computational resources or labeled data are scarce.
Adapter Tuning. Following the methodology of Huang et al. (2024). adapter tuning inserts lightweight parameter-efficient modules such as LoRA modules primarily into the MLP lay- of the Transformer architecture (Lawson Qureshi 2024 Hu et al. 2024e). The number and placement of these adapters are typically determined by the structure of the backbone model, e.g.. one adapter per MLP layer. During finetuning, only the adapter parameters task, leaving the shared backbone parameters unchanged. This configuration allows for efficient, isolated task adaptation and facilitates continual learning without catastrophic forgetting.
updated for each new
Decorator Tuning. Decorator tuning draws inspiration from residual policy learning Yuan et al. 2024). this approach, a residual policy wes trained on top frozen base policy Toase- The agent's 's action at each state is computed as the sum of the outputs from both policies: Thase (s)
(s). This setup enables targeted adaptation to new tasks while preserving the knowledge encoded in the base policy, thereby supporting both stability and flexibility in policy improvement.
Fullmodel Tuning. Fullmodel tuning entails updating parameters of the agent jointly during finetuning, Although this approach offers the greatest capacity for adaptation and optimization, also carries a significantly higher risk of overfitting, particularly low-data regimes. Additionally, full-model tuning substantially increases computational requirements, as all model weights are subject to optimization.
https:://gtthb..com/ http:://gtthubb.m//arrrrt/tyrrr
18

---

Published as a conference paper at ICLR 2026
E THEORETICAL SUPPORT
We provide theoretical insights to support the observed performance gains of QP-SFT over standard SFT based on Hu et al. (2024a). Specifically, we derive performance guarantee under a well-defined of assumptions in the sequential decision-making setting.
Consider a Markov Decision Process (MDP) M = (S, A, R, Ho), where is the state space, A is the action space, defines the reward function, and S + [O, 1] is the initial state distribution. We now state the following lemma.
A + [0, 1] is the state transition probability function, R. S A R
Lemma E.1 (Alignment with respect to the conditioning function (Brandfonbrener et al. 2022 Consider an MDP, behavior 3 and conditioning function f". Let (TT Er 9 where g'(r)= Σ Tt. Assume the following:
1. Return coverage: Polg (T) (si()s)) 2 arr for all initial states $t.
2. Near determinism: P(r & R(s,,a a) or s' + T(s, ass,a))s S at all 8, for some functions
T and R. Note that this does not constrain the stochasticity of the initial state.
3. Consistency of f (s) f'((') + for all
Then
arr ( + +3) H??
J" - J'(m) se
(17)
where ı is derived from Equation 3 nt is the optimal policy, and H is the horizon length of episode. Moreover, there exist problems where the bound is tight up to constant factors.
Corollary E.2. If gr > -, O. and (*1s (**(s)) for all initial states st, then J J"(""" J'(T).
Remark E.3. The behavior policy 3 specifies the data- aata-generating distribution of the collected dataset, while the conditioning function f" corresponds to the return-to-go token f used in Transformer- based agents. The lemma and corollary imply that, in nearly deterministic environments, and given appropriate conditioning and sufficient data coverage (i.c., the behavior distribution places support on near-optimal trajectories), SFT can recover policies that are near-optimal.
Based on Lemma E.I we now give the following Theorem:
Theorem E.4. Consider an MDP with binary rewards and costs, behavior policy 3, and conditioning function f". Let (") Ei= H. is the horizon length of episode. Assume the following:
1. Return coverage: Polg (r) (s)))(s) 2 arr for all initial states S1.
2. Near determinism: P(r + R(s, or ss + T(s, a) S e all s, a for some functions
T and R.
3. Consistency of J": (s) +r for all
For timestep i, the probabilities of selecting actions with maximum reward satisfy:
Reward Selection: P(P; P or. Vi) 1 5,, where Py and P are probabilities under the respectively. With probability at least (1 -8,):
policies updated by Equation and Equation
1 111_00 +3)72 Hor
B Erag (g (r)) E (g (7)
el arr
where à is derived from Equation 13
Note this can be exactly enforced (as in prior work) by augmenting the state space to include the cumulative reward observed far.
19

---

Published as a conference paper at ICLR 2026
Proof. We begin with:
E - [T)] (="
(18)
= E, Er [g"(T) - Er (g (T)] ] + E, Nn (or [g"(r] - (E
(19)
(x²" - -J)((T) J (IT) E [g (T)
(20)
S 1 +3 H12 + (m) EE (g" (r)].
(21)
af"
Next, for the second term in Equation
J J'(TT) E (T)]
(22)
= E, Ey (g" (T)] En 1g (T)
(23)
Erox
(re) - -E
(re)
(24)
t=1
1=1
Es
(P .T)) - Es Σ (P(-re) r.),
(25)
(26)
where Pf and P7. represent the probabilities of selecting the maximum-rewart actions under policies derived from Equation Bl and Equation [3 respectively. Since rewards are binary and by the condition
P(P: - P. 2 Gr, V 2 6r, we have:
H
E
PT r4) Esg Σ (Pr .r)
(27)
H
Es
>
IPP: - -P()r
(28)
H
$ Egg
(-o,).0
rr
(29)
1=1
S -H
(30)
Substituting Equation 30 into Equation
21
we get:
E E ' [g"(T) - En (T))
(31)
S c afr
+3 H² Hor
(32)
Corollary E.S. Iffr 0,E O, and (T) Vr* for all initial states s1, then J' (Tr" J'(()== = J' under P.- = P. and a, = 0.
Remark E.6. Because the corollary's s conditions are stringent and seldom satisfied practice, informative to compare against Lemma E.I]] Relative to the standard SFT objective in Eq. our framework (QP-SFT) achieves an additive improvement of H o,, thereby tightening the objective and yielding superior policies compared with SFT baselines.
MORE DISCUSSION
Ablation study of hyperparameter sensitivity. We present a comprehensive analysis of the impact of the hyperparameter the performance of two algorithms, QP-DPO and QF-SFT across four continuous control environments Figure 7 Experiments are conducted under both Medium and Expert data regimes, evaluating five values of a (0.001, 0.01, 0.1, 1, and 2).
20

---

Published as a conference paper at ICLR 2026
ArDDv
holChestan
Vedulfoold
2013
1000
411
441
-10
Figure 7:
Performance of QP-DPO and QP-SFT algorithms across various tasks under different values of the hyperparameter a.
Table Performance of QP-DPO under various finetuning parameter configurations across different pretrained agent sizes in the MetaWorld benchmark. For certain configurations (e.g.. Adaptor, Fullmodel), the number of trainable parameters increases with the size of the pretrained agent, while for others, the parameter count remains constant. The table reports both the finetuning parameter count and corresponding performance, providing a comprehensive comparison of algorithmic scalability and efficiency.
Medium
Size
9.16KB
9.16KB
9.16KB
9.16KB
9.16KB
Prompt
410.88 +:
419.78 1.3
368.83 + 2.5
391.68 4 5.0
675.39 +
Size
0.5M
1MB
2MB
4MB
6MB 675.39
Adaptor
372.81
460.43 3.3
298.20
5.2
593.51 + 3.5
Size
0,54M
0,54M
0.54M
0,54M
0,54M
Decorator
449.68
476.23
277.14
24.4
500.00
23.2
672.58
Size
6.61M
12.66M
49,32M
97,52M
145,73Mb
Fullmodel
504.42
694.87 + 20.3
375.
34.2
650.33 23.5
750.41 28.7
Expert
Size
9.16KB
9,16KB
9.16KB
9.16KB
9.16KB
Prompt
410.88
419.78 1.1
1.7
675.39
Size
0.5M
IMB 481.93
2MB 349.31
4MB
6MB
Adaptor
505.46
392.53
4.2
675.39
Size
0.54M
0.54M
0.54M
0.54M
0.54M
Decorator
488.06 + 10:4
475.68 +: 14.5
278.57 = 13.5
588.15 14.7
672.75 16.2
Size
6.61
12.66M
49.32M
97.52M
145.73Mh
Fullmodel
504.49 + 20.5
539.49 + 23.1
393. 38 25.2
732.14 = 26.4
101.17-255
Our findings reveal several notable trends regarding the sensitivity of QP-DPO and QP-SFT to the hyperparameter a. In the HalfCheetahDir and HalfCheetah Vel environments, both algorithms demonstrate stable performance across the evaluated range of a, indicating relative insensitivity to this byperparameter. Nevertheless, at higher values of a (e.g., a = 2), QP-SFT exhibits a modest decline performance, suggesting that excessive weighting may hinder effective policy optimization. In contrast, the AntDir environment displays greater sensitivity to a. For QP-DPO, higher a values (0.1-1) correspond improved performance, whereas QP-SFT experiences marked performance deterioration as a increases, particularly when utilizing expert datasets. Within the MetaWorld benchmark, QP-DPO achieves optimal performance at QP-SFT attains its best results at 0.1. Both algorithms exhibit reduced stability at extreme values of a (i.e.. 0.01 and a 2). indicating that moderate values promote more stable and effective learning in complex, multi-task scenarios. Overall, moderate values of a (specifically,
under the Medium data regime, while
0.1 to = 1) yield more robust and consistent performance across the majority of environments and data regimes. In contrast, very small (a 0.001) or very large (a 2) values tend to introduce instability or lead to underfitting and overfitting effects.
Ablation study on scaling pretrained agents. As discussed in Section|]) we present figure illus- trating the performance implications of scaling pretrained agents. To provide a more comprehensive understanding, Table Joffers detailed quantitative analysis of these effects. The pretrained agents in our study are instantiated as Transformer based agents, with model scale primarily controlled by varying the number of attention layers and attention heads. This approach allows us systematically
21

---

Published as a conference paper at ICLR 2026
Table 4: Runtime comparison across pretrained agent scales and finetuning algorithms.
Runtime Comparison Across Pretrained Agent Scales QQP Algorithm)
Pretrained ugents stale
5.83M
14.61M
65.32M
129.52M
193.73M
Prompe
143.13
236.29 min
323.47 min
592.63 min
665. 86
Adaptor
162.16 min
260.91 min
367.86 min
699.91 min
745.77 min
Decorator
158.43
241.62 min
min
601.83 min
666.47
Fullmodel
147.3 min
238.11 min
327.34 min
602.61 min
670.85 min
Runtime Comparison Across Tunnng Algorithms (Fixed Model Size)
Algorithms
SFT
DPO
GRPO
PPO
CQL
QP
Prompt
130.55 min
136.30 min
144.10 min
143.80 min
144.53 min
143.13 min
Adaptor
139.91
144.31
min
min
162.16
Decorator
141.36 min
147.62 min
150.29 min
152.01 min
157.46 min
158.43 min
Follmodel
135.14
138.35
142.70 min
min
145.38
147.39
investigate the impact of model size downstream performance. As shown in Table increasing the size of the pretrained agents generally leads to improved performance across a variety of finetuning parameter configurations and finetuning dataset qualities. However, for Fullmodel finetuning, the substantial increase in trainable parameters introduced by larger architectures, when coupled with limited finetuning data, can result in performance saturation highlight the importance balancing model capacity with the availability of high-quality supervision when designing scalable finetuning strategies. Careful consideration of this trade-off is crucial for maximizing performance gains while mitigating the risk of overfitting or inefficiency.
degradation. These results
A Ablation study of computational cost. We compare relative wall-clock time across algorithms and pretrained agent scales ( Because absolute runtime is hardware-dependen,, our focus on relative comparisons across algorithms and pretrained agent scales under a fixed setup. Two factors primarily drive runtime differences:
(i) the size of the pretrained model, which governs forward/backward compute; and (ii) the complexity of the optimization objective, which varies modestly across finetuning strategies but remains of the same order of magnitude. Empirically, our method attains substantially higher performance at runtime comparable to strong baselines, indicating favorable efficiency at similar computational cost.
Ablation study of main results. To complement Table Figure & presents a task-specific visualiza- tion that elucidates the relative performance across methodologies through normalized performance metrics. Error bars convey variability across random seeds, and per-task rankings make effect sizes more apparent than in tabular form. Across finetuming parameter configurations and environments, our proposed methods consistently deliver competitive performance, typically ranking within the top half of methods on most tasks.
Ablation study on more complex tasks. To further assess the scalability of our approach to larger models and more challenging environments, we conduct additional experiments using the LIBERO simulation benchmark (Liu et al. (023a). LIBERO is designed to evaluate embodied agents on tasks requiring complex reasoning, spatial understanding, and long-horizon cerision-making. It comprises four task suites:
LIBERO-Spatial Focuses on spatial relations and object positioning
LIBERO-Object. Involves reasoning over diverse object categories
LIBERO-Goal. Requires understanding and generalizing goal semantics
LIBERO-Long (LIBERO-10) Comprises extended, sequential tasks that demand multi-step reasoning and planning
For these experiments, we adopt Dita (Hou et al.) 2025). transformer-basee agent with 334M parameters (221M trainable). We finetune Dita for 40,000 steps using both standard SFT and our proposed QP-SIT approach. The model is trained on 8x NVIDIA RTX 4090 GPUs over three days. As shown in Table QP-SFT consistently outperforms SET across all task suites, with the largest gains observed in long-horizon settings- wweeeggss-hrrreee structured policy improvement is most critical. These results demonstrate that QP-SFT scales effectively to larger models and more complex embodied environments.
22

---

Published as a conference paper at ICLR 2026
SFT
DPO
GRPO
PPC
CQL
QP-DPO
QP-SIT
Franut
AVADN!
HullChostitor
Haftheotahee
3.0
82
AndD
HallCheebao
HaalCheebaWee
Futtnedel
3.6
1.
92 8.6
3.8 62
34
2.4
3.6
1.0
Figure 8: Normalized performance different finetuning algorithms on meta-RL. environments using expert-level offline datasets. Each method is evaluated with 50 finetuning trajectories, and results averaged over three independent runs with different random seeds.
Table 5: Performance comparison of SFT and QP-SFT on the four LIBERO benchmark suites. QP-SFT yields consistent improvements, particularly on long horizhoiizon tasks.
Spatial
Object
Goal
Long
SFT QP-SFT
84.2%
96.3% 95.4%
85.4% 87.8%
63.8%
89.0%
67.8%
Table 6: Ablation comparing the original QP-SFT update rule and max-saapling variant. While max-sampling may reduce value underestimation, it did not yield consistent performance gains in the limited-data regime.
QP-SFT (original)
QP-SFT (max)
AniDir HalfChoctahDir
412.18 :: 16.48
405.22 +: 20.30
642.62 3.96
644.37 + 6.2
HalfChectahlel
-119.79 0.26
-120.74 2.32
MetaWorld
553.36 35.35
350.89 4: 38.22
Ablation study the Q-update rule. We further investigate alternative variant of the QP objective that samples multiple candidate actions and takes the maximum predicted value, instead of relying single predicted action. Specifically, compare the original update formulation in
23

---

Published as a conference paper at ICLR 2026
Equation 16 with the max-sampling variant:
E(s.as"'=// 6m - Qの (s,
(33)
where Q. + Qei (s' a))
(34)
where multiple actions are sampled for the maximization operator. This variant can in principle mitigate value underestimation, but it increases both variance and computational cost. In our limited- data setting, as shown Table it did not yield consistent improvements.
Motivation for studying RFT in offline meta-RL. Although RFT is primarily used in online RL settings, we highlight two key reasons why evaluating RFT in the offline meta- ml regime both meaningful and practically important:
Practical motivation. In many TA applications-suce as robotics, embodied control, and safety-critical domains-onlin interaction expensive, slow, or risky, making RtHF-style interactive finetuning infeasible. At the same time, TAs employ the same autoregressive Transformer backbone LLMs, raising the natural question of whether RFT RFT-highly effective in RLHF can also improve adaptation from static trajectory datasets beyond what SFT provides. Prior work these domains has focused almost exclusively on SFT (Zhao) et al. 2022); the role of RFT in offline meta-RL. remains largely unexplored. Our study fills this gap by evaluating whether RFT-style updates yield meaningful benefits under realistic data constraints.
Theoretical Justification. RFT objectives such as PPO-, DPO-, and GRPO-style losses can be interpreted as regularized policy improvement schemes that do not intrinsically require online interaction, provided that a suitable advantage, value, preference signal is available, In our QP formulation, we compute these signals through learned Qo a). yielding KL-regularized policy improvement step aligned with recent work connecting preference- based updates and constrained policy optimization (Liu et al. 2024
Dai et al.
2025).
Thus, our goal is to test whether these RFT-style regularized updates can meaningfully improve offline policy adaptation of pretrained TAs compared to pure SFT, under realistic data constraints.
24


Reviews:
[{"review_id": "wgcEZZpYJt", "reviewer": "Reviewer_RG77", "summary": "This paper investigates how to fine-tune a Decision Transformer-style generative agent on a new task with RL in a few-shot way, as existing works mainly investigate supervised fine-tuning of such a generative agent. They compare several existing methods and a newly proposed method called QP which adds an additional term of maximizing the Q-value of the policy to either SFT or DPO. They conduct extensive experimental analysis on Mujoco and MetaWorld to investigate how different fine-tuning algorithm choices and adaptation methods influence the fine-tuning performance, which provide useful insights for future work on RL fine-tuning of generative agents.", "questions": "1. Which checkpoint's performance is reported for each method? The last one after training for a fixed amount of steps or the one with the best evaluation performance?\n2. Why not do \"real\" PPO learning? Is it because you want to do few-shot adaptation?\n3. The CQL regularization term in equation 11 seems to be inverse?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly motivated with a research gap, i.e., RL fine-tuning of generative agents, to fill. \n2. The paper thoroughly investigate many possible RL ways for fine-tuning, and different ways parameter-efficient fine-tuning methods. \n3. The experiments extensively investigate many problem setting and algorithm choices that may influence the performance of RL fine-tuning.", "weaknesses": "1. My main concern is with the significance of novelty of this paper. It's more like a benchmarking paper instead of proposing a new idea. The QP method proposed by the authors is more like a direct extension of the QT (Hu et al. 2024) algorithm to a multi-task setting, which is limited in originality. \n2. As discussed by the authors, this paper only considers moderate-size models on relatively simple benchmarks like Mujoco locomotion and MetaWorld. Whether the lessons learned from these benchmarking results can be extended to larger-scale models on more realistic tasks or not remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejSQPjlUg8", "reviewer": "Reviewer_CvWD", "summary": "This paper presents a comprehensive study on Reinforcement Fine-Tuning (RFT) for Transformer-based Generative Agents (TGAs) within meta-reinforcement learning (meta-RL) settings. The authors explore various finetuning algorithms, parameter configurations, and their interplay in adapting TGAs to new tasks. They propose a lightweight enhancement combining Supervised Fine-Tuning (SFT) and RFT, providing empirical evidence of its effectiveness across multiple meta-RL environments. The study’s results show that RFT can improve performance, especially in few-shot and sparse reward settings, and the proposed QP-based methods outperform traditional approaches like SFT.", "questions": "1.The paper mentions a “...lightweight improvement DP...” that integrates the advantages of SFT and RFT. However, the term \"DP\" is not clearly defined.\n\n2.Can the authors provide a more detailed theoretical explanation for why RFT should benefit TGAs in meta-RL, beyond analogy with non-RL models?\n\n3.How does the QP method combine RL and SFT? What are its key advantages and potential challenges in practical applications?\n\n4.Given the increasing size of modern language models, how scalable is the proposed RFT method to models with billions of parameters, and what computational challenges might arise?\n\n5.Could the authors discuss the broader impact of their RFT method on meta-RL and real-world applications, and highlight any potential limitations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper systematically evaluates different RFT methods, providing valuable insights into the trade-offs between various finetuning strategies.\n\n2.The introduction of a lightweight enhancement that combines SFT and RFT is a notable innovation, and the proposed QP-based finetuning methods offer a promising direction for improving model performance in meta-RL settings.\n\n3.The authors conduct extensive experiments across multiple environments (MuJoCo, MetaWorld) with varying dataset qualities and sizes, demonstrating the robustness of their methods.\n\n4.The research addresses a practical challenge in adapting large pre-trained models to real-world tasks with limited data, an important topic given the increasing use of Transformer-based agents.", "weaknesses": "1.While the paper highlights the success of RFT in non-RL domains, the motivation for applying RFT to TGAs in meta-RL environments lacks a strong theoretical justification. The analogy to non-RL models feels speculative, without clearly explaining the structural or optimization similarities that would make RFT effective in RL settings. A more principled explanation would strengthen the rationale.\n\n2.The introduction of QP (Q-guided Policy Optimization) is promising, but it requires more clarification. The explanation of how QP combines RL with SFT is not immediately clear, and the potential advantages and challenges of QP are not adequately addressed. A more concise and focused summary would help readers better understand its benefits.\n\n3.The paper focuses on models with up to 40M parameters but does not address the applicability or efficiency of RFT in larger models, such as those with billions of parameters, which are common in current large language models. A discussion on scalability would provide a more complete picture of the method's practical limitations.\n\n4.The conclusion summarizes the contributions well but could benefit from explicitly discussing the broader implications of the proposed RFT method for meta-RL and real-world applications. Additionally, acknowledging potential limitations, such as the method's dependency on smaller model scales, would offer a more balanced view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JT1agf0fE3", "reviewer": "Reviewer_EYuE", "summary": "This work attempts to provide an earnest and comprehensive comparison view on the state of reinforcement fine-tuning (RFT) compared to supervised fine-tuning (SFT) of RL agents. The authors identify several RFT and SFT methods commonly used in the literature for RFT used for LLMs and apply them to RL agents on different environments (Metaworld, Mujoco) in the few-shot scenario to sparse and dense reward settings. Moreover based on the current observation of methods the authors propose a new method that combines RFT and SFT. Across a wide range of experiments the authors demonstrate the benefits of their newly proposed method, which is usually among the best performing methods. Finally, the authors try to provide useful takeaways beneficial for practitioners.", "questions": "- Eq 15: why is the expectation only over (s,a) tuples? why can we not just as in regular double Q-learning use (s,a,s’) tuples for training? \n - Eq 16: would it make a difference if you sampled multiple actions from the policy and take the max instead of only using $\\hat{a}$? I know $\\pi_\\theta$ already maximizes the q-value, but it might help wich counteracting approximation errors.\n - Line 261 mentions that the fine-tuning dataset is much smaller than the pretraining dataset, it would be helpful to provide actual numbers here. While Table 1 mentions 50 finetuning trajectories, this should be made more explicit in the text and not just in the caption.\n - Table 1: Why is the performance on MetaWorld for Prompt tuning equal across all methods? \n - Line 320 mentions that PPO and CQL lack inductive priors, what does that mean?\n - Line 323 mentions that switching from Adapters to Decorator for PPO yields 3% improvement, however the average score of Adapters for PPO is larger than the one of Decorator, am I missing something?\n - The final reached reward for the sparse setting (Figure 3) is around the same (sometimes even higher) than for the dense setting, why? What is the delay in the reward? For the same number of update steps I would expect massive differences if there was a substantial delay in the reward as shown in [1], which does not seem to be the case for e.g. PPO or CQL. \n - Any intuition as to why PPO trained on expert data is better in sparse reward settings than in dense reward settings?\n   \n [1] RUDDER: Return Decomposition for Delayed Rewards, Arjona-Medina et al., NeurIPS 2019", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper attempts to provide an earnest and comprehensive comparison of SFT vs RFT on RL tasks.\n- Plenty of experiments for different fine-tuning methods (SFT vs RFT)\n- The proposed method seems to generally perform pretty well on the different settings.", "weaknesses": "**Inconsistent results and takeaways**\n\nI like the approach of the authors to try to break down plenty of results into key takeaways, however I found multiple contradictions to the key takeaways, some of which are listed below. This is the main factor why I am currently leaning towards rejection at the moment, if they can be resolved I'd consider increasing my rating.\n\nThe paragraph on line 312 states the superiority of QP algorithms and their robustness and broad applicability. Taking a closer look at Table 1, however, the gap to competitors is not large and there is usually a competitor that lies within one standard deviation of QP (e.g. DPO vs QP-SFT for Adapters and Decorators). This means that if a test for statistical significance was performed, the result would be that there is no statistically significant difference. Therefore I recommend to tone down the claiming on superiority and robustness of QP.\n\nIn line 368 states that increasing the number of trajectories consistently improves performance for all methods, however looking at e.g. PPO this is not the case. Similarly performance on DPO for expert is sometimes worse with 100 trajectories than with 50 for expert data. Furthermore, Figure 1 is utterly confusing to me, why is there a distinction in dataset quality for different methods? Does that mean the different methods were trained on different fine-tuning data? Or does it just show the same scores as Figure 2, but normalized? If the latter is the case, please correct the caption because at the moment it states \"Comparison of finetuning dataset quality\" which led to my confusion. Furthermore, if I interpreted it correctly and they essentially show the same information, one of those figures could be moved to appendix to accomodate space for further interpretation/results.\n\nAnother example are takeaways in line 372: \"Supervised approaches yield stable, imitation-driven performance—particularly\nwhen data quality is mixed\", however SFT is one of the worst methods on HalfCheetah (Medium). Yet another example: \"(ii) regardless of the algorithmic family, more finetuning trajectories monotonically enhance performance\", however CQL on Metaworld (Expert) gets worse with more trajectories.\n\n**Novelty**\n\nOne important aspect of the contributions of this work is the investigation of the data quality for fine-tuning, however this has been investigated in prior work already [1]. In particular, this prior work introduces metrics quantifying trajectory quality and state-action coverage and their influence on performance for several offline RL algorithms. It would be helpful to clarify the distinction to this work.\n\nFurthermore, [2] also provides a comprehensive comparison between different parameter-efficient fine-tuning approaches for single-task fine-tuning on Metaworld and DMControl, including a wider variety of PEFT methods than this work. Though they only investigate SFT and no RL fine-tuning. It would be helpful to outline the difference to those works concretely.\n\n[1] A dataset perspective on offline reinforcement learning, Schweighofer et al., CoLLAs 2022\n\n[2] Learning to Modulate pre-trained Models in RL, Schmied et al., NeurIPS 2023\n\n**Conceptual framing**\n\nThe authors mention several times throughout the paper that they investigate a metaRL setup, however to me it seems it is more like an actual transfer learning/ multitask fine-tuning setup as in [1]. There are also no conventional metaRL algorithms compared which makes me wonder whether the \"metaRL\" framing is the correct terminology. Furthermore the terminology of \"generative agents\" is rather confusing to me, as this terminology is usually employed for LLM-based agentic frameworks [2]. Also the focus on \"transformer-based\" is not necessary as all the tested algorithms are agnostic to the architecture.\n\n[1] Learning to Modulate pre-trained Models in RL, Schmied et al., NeurIPS 2023\n\n[2] Generative Agents: Interactive Simulacra of Human Behavior, Park et al., UIST 2023\n\n\n\n**Choice of methods**\n\nIt might be interesting looking into [1] for a LoRA-variant that has been shown to significantly improve upon LoRA based single-task finetuning on the MetaWorld tasks in [2]. This could potentially affect the ranking of the methods in the experiments.\n\n[1] Parameter Efficient Fine-tuning via Explained Variance Adaptation, Paischer et al., ENLSP workshop at NeurIPS 2024\n\n[2] Learning to Modulate pre-trained Models in RL, Schmied et al., NeurIPS 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nSCOTfWpTd", "reviewer": "Reviewer_RqYc", "summary": "The paper “The State of Reinforcement Finetuning for Transformer-based Generative Agents” aims to study how reinforcement finetuning (RFT) techniques, commonly used for aligning large language models such as through RLHF, can be applied to Transformer-based “generative agents.” It positions itself as a comprehensive evaluation of RFT methods in offline reinforcement learning settings, integrating variants such as reward modeling and policy-gradient finetuning. The authors claim that RFT can enhance sample efficiency and reasoning ability in pre-trained Transformer agents and present a benchmark framework comparing different RFT configurations. However, the paper’s objectives remain vague—it is unclear whether it introduces a new algorithm, proposes a benchmark, or provides theoretical insights—and the distinction between “Transformer-based generative agents” and standard LLMs or RL agents is not well defined.", "questions": "- Could the authors clarify the main objective of the paper — is it intended to serve as a benchmarking effort, a new RFT algorithm, or a conceptual analysis of reinforcement finetuning for Transformer-based agents? Clear positioning would help readers understand how to interpret the experimental results and contributions.\n\n- How do the authors define a “Transformer-based generative agent” in contrast to standard LLMs fine-tuned with RLHF or RFT? Are these agents expected to operate in simulated environments, handle sequential decision-making, or simply produce text-conditioned reasoning? A more precise definition would make the scope of the study much clearer.\n\n- Why were offline RL tasks chosen as the evaluation domain for RFT, which is traditionally an online or human-feedback-based training process? Are there theoretical or practical motivations for believing that RFT can meaningfully improve offline RL policy learning?\n\n- What base model and pretraining setup were used for finetuning? Without information about the backbone architecture, scale, or initial performance, it is difficult to interpret whether the observed improvements come from RFT or model capacity.\n\n- Could the authors provide stronger experimental baselines or ablation studies (e.g., comparing against standard offline RL algorithms such as CQL, IQL, or decision-transformer-based approaches) to substantiate the empirical claims?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "The paper touches on a important topic. The motivation to unify reinforcement-based alignment with multi-task or offline RL adaptation reflects an original attempt to connect two active research directions. The authors make an effort to survey existing RFT approaches and to frame them within a consistent experimental setting, which could, in principle, contribute to clarifying the landscape of RL-based post-training methods. The writing is generally clear at a surface level, and the paper demonstrates awareness of recent developments in RLHF and RFT, attempting to position itself at the intersection of language modeling, reinforcement learning, and meta-adaptation. Overall, while the execution is weak, the underlying idea of systematically analyzing reinforcement-style finetuning for large Transformer-based agents shows conceptual ambition and topical relevance.", "weaknesses": "The main weakness of this paper lies in its lack of conceptual clarity and methodological grounding. It is unclear whether the paper’s goal is to propose a benchmark, a new algorithm, or an analytical study—and this ambiguity undermines its overall contribution. The term “Transformer-based generative agent” is used extensively without a precise definition or boundary relative to standard large language models, prompting confusion about what specific architecture or behavior distinguishes these agents from common RLHF-trained LLMs.\n\nFrom a technical standpoint, the paper’s use of offline reinforcement learning tasks is poorly justified. RFT and RLHF are inherently online or preference-based alignment methods, not baselines for static offline RL datasets. The decision to evaluate RFT in this context suggests a fundamental misunderstanding of the underlying paradigms. Furthermore, the experimental setup is under-specified: there is no mention of which base model was used, what datasets were employed, or how the finetuning protocols were configured. Without this information, the reported results lack reproducibility and interpretability.\n\nThe evaluation design is also inadequate—the authors omit strong baselines such as CQL, IQL, or other modern offline RL methods, and they provide no ablations or sensitivity studies to validate their claims. The comparison across RFT variants remains superficial and largely descriptive, without clear analysis of why certain methods perform differently. Finally, the paper’s writing gives the impression of a survey-like overview rather than a focused, hypothesis-driven study; key references are discussed only at a surface level (e.g., RLHF and GRPO), and theoretical or empirical novelty is minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)