DEAS: DEtached value learning with Action Sequence for Scalable Offline RL
Authors: Changyeon Kim
Haeone Lee
Younggyo Seo
Kimin Lee
Yuke Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Offline Reinforcement Learning
Vision Language Action Model
Primary area: applications to robotics, autonomy, planning

Abstract: Offline reinforcement learning (RL) presents an attractive paradigm for training intelligent agents without expensive online interactions. However, current approaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions, enabling reduction of the effective planning horizon by considering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in-distribution actions that achieve high returns in the offline dataset. We demonstrate that DEAS consistently outperforms baselines on complex, long-horizon tasks from OGBench and can be applied to enhance the performance of large-scale Vision-Language-Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.

Published as a conference paper at ICLR 2026
DEAS: DETACHED VALUE LEARNING WITH
ACTION SEQUENCE FOR SCALABLE OFFLINE RL
Changyeon Kim 1+ Haeone Lee'
Younggyo Seo Kimin Lee Yuke Zh4
KAIST "UC Berkeley "The University of Texas at Austin "NVIDIA
ABSTRACT
Offline reinforcement learning (RL) presents an attractive paradigm for train- ing intelligent agents without expensive online interactions. However, current ap- proaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions, enabling reduction the effective planning horizon by con- sidering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in- distribution actions that achieve high returns the offline dataset. We demon- complex, long-horizon
strate that DEAS consistently outperforms baselines tasks from OGBench and can be applied to enhance the performance of large- scale Lasion-Langaag-Acttinn Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.
INTRODUCTION
Offline reinforcement learning (RL) (Lange et al., 2012; Levine et al., 2020) enables learning from static datasets without incurring online data collection risks, while circumventing the need for ex- pensive expert demonstrations. However, existing methods primarily focus on short-horizon tasks with dense rewards (Yu et al., 2020; Fu et al., 2020; Gulcehre et al., 2020; Mandlekar et al., 2021) and struggle to scale to complex long-horizon scenarios. Recent attempts using large-scale architec- tures (Kumar et al., 2023a;b; Chebotar et al., 2023; Springenberg et al., 2024) show promise, but their effectiveness on complex tasks remains unexplored.
To address the need for long- horizon evaluation, recent work (Park et al., 2025a;b) has proposed challenging benchmarks for complex offline RL and demonstrated that reducing the effective plan- ning horizon (i.e., shortening the time span over which the agent must plan) in both value and policy learning via ne-teep TD updates with high n values and hierarchical policies is essential. However, these approaches rely on goal-conditioned RI. with explicit, expert-provided goals, which are often unavailable in practice. For instance, high n values in n-step TD updates introduce increased bias and bootstrap еигог in standard RL without explicit goal information (Tsitsiklis & Van Roy, 1996; Kearns & Singh, 2000; Sutton & Barto, 2018).
These limitations underscore the need for alternative approaches horizon reduction (reducing the planning horizon) that work without explicit goal conditioning. One promising direction is leverag- ing action sequences, which have shown success in behavior cloning (Pomerleau, 1988) for captur- ing noisy, remporall distributions in expert demonstrations (Chi et al., 2023; Zhao et al., 2023). However, existing attempts to use action sequences for RI. remain insufficient for achieving robust horizon reduction. Q-chunking (Li et al., 2025b) has explored the use of action sequences for RL, demonstrating their potential for temporally consistent exploration. However, introducing action sequences to standard actor-critic frameworks causes severe value overestimation (Seo & Abbeel, 2025) due to actors maximizing over potentially erroneous critic estimates with widely spanned action spaces. This problem is exacerbated in offline RL where distribution shift creates extrapola- tion errors (Kumar et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020). While CQN-AS (Seo
Work done while visiting The University of Texas at Austin. Project page: https: http:://changrro..eiteeee site//// site/deas I Equal advising.

---

Published as a conference paper at ICLR 2026
Previous Methods
Su
Critic
(500: (0:1-1)
Target Criti
Policy
Qs Oraan
+ Rea-t
0H:2H-1
π
DEAS (Ours)
Critic
S0
V
Mou/111
Sp
Target
Value
SH
+ 71 Ross
Figure I: Overview, DEAS an offline RL framework that learns from action sequences instead of single actions. Unlike previous methods that couple actor-critic training, our key insight is to train the critic separately from the policy (detached value learning) using action sequences, which enables stable learning while avoiding value overestimation. We further enhance stability by combining distributional RL objectives and using dual discount factors, which leads to additional improvement.
& Abbeel, 2025) proposes a value-only approach to avoid this issue, it introduces discretization er- rors that limit performance in complex tasks and cannot leverage expressive policy classes (Wang et al., 2023; Hansen-Estruch et al., 2023; Park et al., 2025c). For this reason, our research aims to develop methods that can leverage action sequences for horizon reduction while avoiding value overestimation and maintaining compatibility with expressive policy architectures.
Our approach We present DEtached value learning with Action Sequence (DEAS), an offline RL framework that leverages action sequences for scalable value lcarning complex tasks. Our method treats consecutive action timesteps as inputs to the value function to provide more expressive information than single-step actions. This design provides principled horizon reduction analogous to n-step TD updates with temporally extended actions, while action sequences offer richer infor- mation than single-step actions without requiring explicit goal conditioning. To address the value overestimation challenges inherent in learning value functions with action sequences in offline RL settings, we employ detached value learning (Kostrikov al., 2022) that decouples critic training from the actor, biasing value estimates toward high-return actions present in the offline dataset. This method is appealing as can be applied to any expressive policy architectures including large-scale Lision-Language--
models (VLAs) without the hazard of value overestimation. Additionally, we propose to incorporate distributional RL (Farebrother et al., 2024) in value learning to mitigate instability from accumulated bias in multi-step returns.
We validate DEAS through comprehensive experiments on challenging long-horizon tasks from OGBench (Park al., 2025a), where standard offline RL methods struggle to achieve meaningful success rates. Our method consistently outperforms all baselines, demonstrating its effectiveness on complex tasks. Additionally, we show that DEAS can be used to improve the performance of VLAs (Bjorck et al., 2025) in hard tasks from RoboCasa Kitchen (Nasiriany et al., 2024) and real- world manipulation tasks, which significantly improves performance compared to policies trained solely on expert demonstrations. These results demonstrate DEAS's practical applicability and po- tential for scaling offline RL to real-worke scenarios.
Contributions We highlight the key contributions of our paper below:
We present DEAS: DEtached value learning with Action Sequence, a simple yet effective of- fline RL method that leverages action sequences for training crities and employs detached value learning with classification loss for stable training.
We demonstrate that DEAS significantly outperforms baselines on complex, Long-horizon tasks across 30 diverse scenarios in OGBench (Park al., 2025a).
2

---

Published as a conference paper at ICLR 2026
We demonstrate that DEAS can enhance the performance of large-scale VLAs, achieving perior results on complex tasks from RoboCasa Kitchen (Nasiriany et al., 2024) and real-world manipulation tasks compared to policies trained solely expert demonstrations.
RELATED WORK
Offline reinforcement learning Offline RL focuses on learning policies from fixed datasets with- out further environment interaction (Levine et al., 2020). The primary challenge lies the distri- butional shift between the behavior policy and the learned policy, which can result in value over- estimation and suboptimal performance. Previous work has proposed various approaches including weighted regression (Peng al., 2019; Nair al., 2020; Wang al., 2020), conservative regu- larization (Kumar et al., 2020), behavioral regularization (Fujimoto et al., 2019; Fujimoto & Gu, 2021: Tarasov et al., 2023; Park et al.. 2025c), and in in-sample distribution maximization (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023). Our method builds upon in-sample distribution max- imization approaches, particularly IQL. (Kostrikov et al., 2022), extending them to handle action sequences while maintaining stability by removing the critic update based on the actor's output. Furthermore, our method has the advantage of being adaptable to any policy extraction method for the final policy, making more flexible and practical.
BC/RL with action sequence Adopting action sequence has been actively investigated in both im- itation learning and RI.. Behavior cloning advances show that predicting action sequences captures temporal dependencies from expert demonstrations that single-step actions miss (Chi et al., 2023; Zhao al., 2023; Black al., 2025; Bjorck et al., 2025; Intelligence al., 2025). Several works have introduced action sequences into RL (Li et al., 2024; Tian et al., 2025), with Q-Chunking (Li al., 2025b) demonstrating incorporation into actor-critic frameworks in offline-to-online RL with- out policy class constraints. However, this approach faces fundamental challenges: expanded action spaces increase value overestimation risk, particularly in offline settings with limited data cover- age (Kumar et al., 2019), yet this issue remains unaddressed. CQN-AS (Seo & Abbeel, 2025) cir- cumvents this by removing the actor entirely, but introduces accumulating discretization errors that severely limit performance in complex tasks and prevent use of expressive policy classes (Wang et al., 2023; Park et al., 2025c). Our approach uniquely combines both paradigms: we leverage hori- reduction from action sequences while addressing value overestimation through detached value learning, enabling stable training with any policy architecture.
3 PRELIMINARIES
Problem formulation We consider a Markov Decision Process (MDP) (Sutton & Barto, 2018) M (S,A. p, R. po. 7),, where S is the state space, A is the action space, R(s, a) S x A - R is the reward function, p(8'|s (s, a) S x A > A(S) is the transition function, is the initial state distribution, and is the discount factor. In this paper, we focus offline reinforcement learning.
where we have access only to a static dataset D = {r') containing N trajectories of fixed length H, where each trajectory 7' (sg. ao, To: $H aH, represents a sequence of states, actions, and rewards. The dataset is collected using a data collection policy TD S ((A), which may be unknown or suboptimal. Unlike online RI., we cannot interact with the environment during training. The objective is to learn a policy S - ((A) that maximizes the expected sum of discounted rewards Ep,... R. 12 7² R[ss,, az)] using only this fixed dataset.
Temporal-diiferenn (TD) learning Modern value-based RL approaches usually update a state- action value function Q S x A R to approximate the maximum discounted cumulative reward by bootstrapping from a successor state:
Q(s., 04) Q1) + a + max (()-1,aa)) Q(st, as). d'EA
This one-step update progressively aligns Q with its Bellman target using sampled transitions. For reducing the bootstraping bias problem in long- morgooorzoon tasks, multi-step TD (Tsitsiklis & Van Roy, 1996; Kearns& Singh, 2000; Sutton & Barto, 2018) extends this idea by accumulating rewards several steps before bootstrapping while retaining the same basic structure.
over
3

---

Published as a conference paper at ICLR 2026
Implicit Q Learning (IQL) (Kostrikov et al., 2022) Instead of regularizing the critic with the actor output, IQL approximates the optimal critic to be maximized only the region of action distributions present in the offline dataset with an in-sample expectile regression. Given param- eterized critic ((s,, Oz; 0), target critic Q(+e, ae; 0), and value network ((s;; the objective for value learning defined
Cv(v) ) [g( ar [(s5))))]
(0) Re(eaae04 [R(s), 0:) ) 7 -(, 0))
(2)
where (5(u) | - -1(u 0)(u² is the expectile loss with expectile parameter r E [0, 1]. By using > 0.5, Equation (1) applies the higher weight on positive errors than negative errors, so that V
approximates upper expectile of the in-distribution TD targets. This objective allows V and Q to be approximated only within the in-distribution region of actions.
4 METHOD
We propose DEtached value learning with Action Sequence (DEAS), offline RL method that models action sequences for scalable value learning. Our approach consists of two key components: (1) critic (st,, ay; ()) that estimates expected returns for H-step action sequence a, 0t:t+-111 from state St under the data collection policy πD, and (2) flexible policy update mechanism ap- plicable to any policy T(a;: St. (t, Φ) that outputs H-step action sequences. Section 4. describes how we extend temporal with action sequences, while Section 4.2 introduces how DEAS enables stable training through detached value learning. distributional RL, and dual discount factors. We provide pseudocode in Algorithm 1 and additional details in Appendix B.
4.1 TD LEARNING OVER ACTION SEQUENCES
Complex tasks require coordinated action sequences where each action's effectiveness depends on its context within the sequence. For instance, in OGBench puzzle e or cube tasks, success depends on planning through multiple intermediate steps and maintaining consistent actions over extended periods. These temporal dependencies and hidden sub-tasks are not captured by current state repre- sentations, making it challenging for agents learn effective policies. The challenge becomes greater in goal-free settings, where agents must discover these sequential patterns from offline data without explicit goal instructions.
To address this challenge, we formalize temporally extended decisions by treating each fixed-length H-step action sequence as a single decision unit. Given the underlying MDP M,, consider an state-action value function Q(s, a) over H-step sequence of actions a, := (a,, a.+1...... H++H-1 H-1) E A" Executing a, from state S, means applying the primitive actions in order, collecting the dis- counted return
H-1
R(s a, 7) =
Σ k=
7 R(S ac+k) St, a
(3)
and transitioning to successor state 8t+ Our TD updates are then performed directly on Q(s,, ar) using R(st,a,, a,, 7) as the multi-step target, i.e., standard TD learning applied to sequence actions in A This formulation is equivalent to considering decision-making at every H-th time step with temporally extended actions.
For updating ((s, a), we can use the following TD learning objective that extends standard Q- learning (Bradtke & Duff, 1994; Sutton al., 1999):
Q(s;, + Q(s,, ad + a Riss, ar, m)) + " max Q(5(+H, - Q( a:)
deA"
where Y1 is used to construct the H-step return within sequence, while 72 discounts across sequence-level decision points. This formulation aggregates rewards over H steps and propagates value estimates across temporally extended transitions, achieving horizon reduction similar to n- step TD learning (Park et al., 2025b) while guarantecing unbiased value estimates for the H-step rewards (Li et al., 2025b).
4

---

Published as a conference paper at ICLR 2026
Algorithm DEAS
Required: Offline dataset D. Support range return Vmin. Vmax, number of bins m, discount factor "/:
Initialize parameters t) 0, 0
while not converged do
Sample batch {st., ac, Rt.c H- Sc4.)) } from
Compute the discounted return of H H-step action sequence R(st, a) using Equation (3)
Compute (ss, a; 0) and (s; 1) using Equation (6)
Update Network
Update (s; v)) to minimize Equation (7) with Q(s, a: 0) and V(s: v)
Update Q Network
Update (s, a; 0) to minimize Equation (8)
D. Update Actor Network Update π (s; Φ) with any type of policy extraction algorithms (c.g., BoN, DPG, AWR, etc.)
Update 0 (1 3) 0 3-0
return (s)
4.2 DEAS: DETACHED VALUE LEARNING WITH ACTION SEQUENCE
Detached value learning for handling action sequence Action sequences introduce challenges for value function approximation, as the expanded action space makes it harder for the critic to esti- mate Q-values accurately. Meanwhile, the actor can exploit regions where the critic makes prediction errors, leading to value overestimation and unstable learning (Seo & Abbeel, 2025). To address this, we adopt detached value learning (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023) that decouples actor and critic training, introducing critic ( az; and value ((s;; networks following IQL (Kostrikov et al., 2022):
(v( 3e [5(Q(s), ac; [(+G+)))
(Q ( (R(s), ar, m1) 72 H V(S+H: a;;
(5)
This approach steers the critic toward high-return action sequences in the offline dataset without the potential exploiting critic approximation errors, preventing value overestimation and enabling stable value learning even with longer action sequences. In Appendix E, provide the theoretical proof of extending action sequences in detached value learning without loss of generality.
Distributional RL for enhanced stability Even with detached value learning, the cumulative reward term R could introduce significant variance when H is large. To enhance stability, we extend our framework with distributional RL (Bellemare et al., 2017; Farebrother et al., 2024), modeling both critic and value networks as categorical distributions over fixed support [Vmin, Viaxx discretized into bins:
(s,, 0) = E (8, (ss, a; = Σ 01 0) 00:00)55. 62, a=(s, = m Di=1 cli(s,a0))
(6)
where Pú denotes the predicted probabilities for location 2,. V(s; 1) is computed similarly, by con- ditioning only the state To address scale differences between regression and classification objectives, we maintain IQL's weighting scheme but replace regression with classification- learning:
Cv(w) B( 0c. Σ Di(Se, ax; 0) log (1
T 0t 1-T
Q(s (() otherwise.
(Q( = -B(en.ar )NDD
Σ
ps log (i
(8)
5

---

Published as a conference paper at ICLR 2026
Figure 2: Simulation task examples. We study DEAS on 30 different tasks from OGBench (Park et al., 2025a) and 4 challenging manipulation tasks from RoboCasa Kitchen (Nasiriany et al., 2024).
For target probabilities p., we adopt the truncated normal distribution with mean Bellman tar- get (TV) +) = Σ 7fr444 + 7 (884 v)) and standard deviation o = 0.75 (Umar
Umin /m), inspired by Farebrother et al. (2024).
Dual discount factors To further enhance stability and expressiveness in value estimation, we employ two separate discount factors: 7 for rewards within H-step action sequences and 72 for the summation across sequence-level decision points. This dual-discounting scheme enables the value function to appropriately weigh immediate and future returns, mitigating issues such as value explosion or collapse that can arise from improper return scaling. In our experiments, we observe that decreasing "I and increasing 72 lead to more stable training and are critical for performance improvement, especially as the action sequence lengthens (see Section 5.4 for supporting results).
Compatible policy methods For obtaining final policy x <(8; Φ), our framework is compatible with variety of policy extraction strategies (Park al., 2024), including weighted behavior cloning (Peng et al., 2019), deterministic policy gradient (DPG) (Fujimoto & Gu, 2021), best-of- sampling (Chen et al., 2023), and flow-matching approaches (Park et al., 2025c). Since value function training does not require querying the policy, it can be performed independently. and the policy can be updated separately. To demonstrate this, we illustrate the effectiveness of our method using various policy extraction methods in our experiments.
5 EXPERIMENTS
We first validate the effectiveness of DEAS through extensive experiments on various complex, long- horizon tasks in OGBench (Park et al., 2025a). Additionally, to prove that DEAS can be naturally plugged into large-scale VLAs for practical applications, we evaluate DEAS by fine-tuning GROOT N1.5 (NVIDIA, 2025) using offline RI. methods on 4 hard tasks from RoboCasa Kitchen (Nasiriany al., 2024) and also conduct real-world experiments with Franka Emika Research 3 Robot Arm. See Figure 2 and Figure 4 for task examples used in our experiments.
5.1
OGBENCH EXPERIMENTS
Setup We evaluate manipulation environments from OGBench (Park et al., 2025a), each with subtasks. We use datasets ranging from IM to 100M transitions based on task difficulty. While OGBench is originally designed for offline goal-conditioned RL, ('singletask') for reward-mane maximiiin offline RL. For fair comparison, all methods use identical MLP architectures for actor networks and adopt the same policy extraction approach as FQL (Park et al., 2025c), except for CQN-AS, which uses value function networks as the actor itself through discretization. Action sequence length H is set for scene and puzzle tasks, and 4 for cube tasks, with n = H is used for n-step FQL. More details about the experimental setup can be found in Appendix B.
its single-task variants
Baselines We compare against FQL (Park et al., 2025c), a state-of-the-arr offline RL method using one-step distillation between flow matching models with different denoising steps, and n- step FQL (Sutton Barto, 2018), which extends FQL with n-step TD updates for horizon reduc- tion (Park et al., 2025b). While increasing n increases bias in standard offline RI,, DEAS explic- itly models action sequences while maintaining horizon reduction benefits. We also consider Q- Chunking (QC) (Li et al., 2025b), which uses action chunking for actor-critic training while keeping the interaction between actor and critic, while DEAS uses detached value learning. For fair compar- achieve higher performance than
ison with ours, extensively tune QC-FQL hyperparameters the original paper. Lastly, CQN-AS (Seo & Abbeel, 2025), a value-basee RL method with action sequence utilizing multi-level crities with iterative discretization, is included as baseline.

---

Published as a conference paper at ICLR 2026
Table 1: Offline RL results in task categories from OGBench (Park et al., 2025a). We report the success rate (%) and 95% stratified bootstrap confidence interval over 4 runs. Bold indicates the values or above 95% of the best performance. Please refer Table 9 for the full results.
Task Category
#Data
FQL
N-step FQL
QC-FQL
CQN-AS
DEAS
scene-play-ssingletak (5 tasks)
50 +3
36
73
1 1+1
76
suee-doubber-pllyy-snnleeee (5 taska)
1M
14 +2
4 +2
41 +s
2 +1
48 =2
puzzle-SkS-pllayssinlllatt (5 tasks)
44 =a
36 =3
62 a7
0 =0
91 =3
cube-trriple-play--tttteeterr (5 taske)
10 1.
23 12
83
0 1o
82
10M
puzzlle-kdd-play---igllaaa--- (5 tanks)
32 2:
19 ab
69 19
0 a0
82 2:6
cube qube-quadruple pla.eapppee-payys-sneeee.eeee (5 tasks)
100M
17 11
36 110
45
0 1.
64
cube-dauble
rube triple
cube quadr.ppe
100
SEAS((uu))
100
SE4S (Burt)
100
(Dunt)
Sucress (N6
80
Success (%))
80
Sucrecs Ratn (1))
80
TOL
60
40
20
20
20
10
1.0
Dataset Size (M)
Dataset Size (M)
Dataset Size (M)
Figure 3: Agent performance across varying dataset sizes on three representative OGBench (Park al., 2025a) tasks, evaluated by success rate (%). Solid lines indicate the mean, while shaded areas denote the stratified bootstrap confidence intervals over 4 independent runs.
Quantitative results As shown in Table 1, DEAS consistently achieves the best performance across all 6 task categories with various dataset sizes. Comparing FQL and N-step FQL. we ob- serve that simply increasing the n-step mostly leads to performance degradation due to bias in standard offline RI., while our detached value learning approach enables stable training with ac- tion sequences. Notably, DEAS matches or outperforms QC-FQL across all tasks, demonstrating the effectiveness of our stable value learning in addressing offline RL instability, The method shows particularly strong performance on tasks requiring long-horizon reasoning like puzzle and the most challenging tasks (i.e., cube quadruple), where the benefits of using action sequences most pronounced. CQN-AS shows significantly lower performance, likely due to its direct application of strong BC regularization the value function in the presence of predominantly suboptimal data, along with cumulative errors from iterative discretizations that reduce action precision.
Scaling analysis To further validate the scalability of DEAS, we conduct scaling analysis on three representative OGBench tasks with varying dataset sizes. As shown in Figure 3, DEAS consis- tently outperforms all baselines across all dataset sizes, achieving the highest success rates in every environment. The method demonstrates robust scaling across different dataset sizes, maintaining consistent performance gains even with larger datasets. This superior performance validates ap- proach of explicitly modeling action sequences while effectively leveraging suboptimal data through detached value learning and stable multi-step training.
5.2
VLA EXPERIMENTS
To validate the practical applicability of DEAS, we demonstrate its effectiveness with large-scale VLAs (Black et al., 2025; Bjorck et al., 2025; NVIDIA, 2025). These models, trained on internet- scale diverse datasets with billion-scale parameters, predict much longer action sequences and are widely used in roboties applications. However, deploying these models typically requires fine-tuning on task-specific data, which often necessitates collecting expensive expert demonstrations. We de- sign our experiments to validate whether DEAS can improve VLA performance by effectively uti- lizing suboptimal demonstrations alongside limited expert data, potentially reducing the required amount of costly expert demonstrations, See Appendix B.2 for more details.
5.2.1 ROBOCASA KITCHEN EXPERIMENTS
Setup We employ GROOT NI.5 (NVIDIA, 2025) and To (Black et al., 2025) as the backbone VLA. First, we fine-tune the VLA using 100 expert demonstrations from all 24 RoboCasa Kitchen

---

Published as a conference paper at ICLR 2026
Table RoboCasa Kitchen evaluation results. We fine-tune VLAs on 24 RoboCasa Kitchen tasks using 100 expert demonstrations per task. For selected tasks, we collect 300 rollouts and apply of- fline ILRRL algorithms. Success rates (%) on 50 episodes, aggregated with seeds. PnPC2M denotes PPPCConterToMicrowany and PnPM2C denotes PuPMiccrowaveooCoutee Bold and underline indicate best and runner-up results, respectively.
Result from Bjorck et al. (2025)
Reproduced performance
Models
CoffeeSetupHug
PnPC2M
PnPM2C
TurnOffStove
Avg.
Base model GROOT NI
15.7
4.4
GROOT N1.5t W
4.7
21.3
7.3
14.7
12.0
20.0
10.0
8.0
12.3
Imitation learning
GROOT N1.5 Filtered BC
14.7
25.3
19.3
18.5
Tu Filtered BC
30.7
16.7
14.7
10.0
18.0
Offline
GROOT N1.5 IQL
23.3
30.0
14.7
12.7
20.2
GROOT N1.5 QC
16.0
28.7
14.7
10.7
17.5
GROOT N1.5 + DEAS (Ours)
28.7
36.0
18.0
18.0
25.2
To DEAS (Ours)
37.3
15.3
19.3
15.3
21.8
tasks to verify that we achieve performance similar the original GROOT NI (Bjorck et al., 2025). From these tasks, we select tasks with the lowest success rates their respective categories for our offline IL/RL experiments. We then collect 300 rollouts for each task from the resulting policy and apply various offline IL/RL methods. For RL methods, we ine-tune the base policy using behavior cloning both expert demonstrations and the rollout dataset and the model training critic functions when necessary. For policy extraction, we adopt best-of-N sampling (Chen al., 2023; Nakamoto al., 2024), where sample multiple outputs from the policy and select the action sequence with the highest Q-value. We set H = 16 in GROOT N1.5 and H 50 in To for all methods match the original action chunk size used for each model.
actor for
Baselines We compare against several baselines across both imitation learning and reinforcement learning paradigms. For imitation learning, we consider Filtered BC, which fine-tunes the base pol- icy using both expert demonstrations and successful episodes from the rollout data (Oh et al., 2018) 8). For reinforcement learning, we evaluate IQL, a value-based method that operates on single actions without requiring policy outputs. For determining action sequence in IQL, we use the very first ac- tion in the sequence for value estimation. Lastly, we consider QC, which employs action chunking for critic training but relies predicted action sequenees from VLA for the critic update.
Results As shown in Table 2, DEAS achieves the highest success rates in 3 out of tasks, with the remaining task also showing improved performance compared the base model. While filtered BC improves performance with simple approaches, our approach further improves performance by effectively leveraging suboptimal data. While single-step IQL also demonstrates effectiveness, it yields smaller performance gains across all tasks than our approach, due to its lack of action- sequence understanding. QC shows limited improvement compared to BC-based approaches, high- lighting the advantage of our detached value learning with action sequences. A similar pattern holds for the To backbone: DEAS yields clear improvements over the base model across all tasks, even with significantly longer action sequence (50) than the GR0OT N1.5 backbone. This confirms that our method reliably enhances VLA policies regardless of backbone and action- sequenee length.
5.2.2 REAL-WORLD EXPERIMENTS
Setup We further investigate the effectiveness of DEAS in real-world tasks using Franka Emika Research 3 Robot Arm. Inspired by RoboCasa Kitchen, we design pick-and-place tasks from the countertop to the bottom cabinet, with three different objects: peach, milka, and hichew (see Fig- ure 4). For each task, we collect 5 demonstrations, fine-tune GROOT N1.5, collect 25 rollouts, and apply various offline IL/RL methods. We evaluate using 20 rollouts per task from different initial points and use the same baselines in the RoboCasa Kitchen experiments. Success rates cal- culated based on partial success scoring (0-1 scale) that considers subtask completion, with detailed evaluation methodology provided in Section B.2.2.
8

---

Published as a conference paper at ICLR 2026
Figure 4: Real-world tasks. We conduct mick-and-place tasks from the countertop the bottom cabinet with peach, milka, and hi hich.w.
Table Real-world evaluation results. We report the partial success rate (%, over 20 trials per task) on 3 tasks from 5 initial points. Bold and underline indicate best and runner-up results, respectively.
Models
peach
milka
hichew
Avg.
Base model
GROOT N1.5
62.0
45.0
85.0
64.0
Imitation learning
Filtered BC
76.3
25.0
92.5
64.6
Offline RL
+IQL
82.5
37.5
78.8
66.3
QC
58.8
15.0
45.0
39.6
DEAS (Ours)
86.3
53.8
95.0
78.4
Results In Table 3, DEAS achieves the highest success rates across all three pick-and-place tasks compared to baselines. The method shows consistent improvements, particularly on challenging objects like milka (a deformable object) where other approaches struggle. Notably, QC shows de- graded performance compared to the base model, likely due to its instability when using action sequences with relatively small datasets, while method shows stable improvement limited data. These results demonstrate that our detached value learning approach can be effectively applied to real-world robotic tasks and remains stable regardless of the dataset size.
with
5.3 QUANTITATIVE ANALYSES
Alignment to actual returns To evaluate whether a critic with action sequences is properly aligned with actual returns, we analyze value cali- bration between the critic values and the true discounted returns. The main intuition of this experiment is that properly trained critic should main- tain a monotonic relationship be- tween predicted values and actual returns on unseen state-action distri-
Figure 5: Value calibration curve comparing predicted critic values to Monte-Carlo returns on held-out trajectories in 4xz (left) and sube-quadruple (right).
Prodicted a (rormalzee)
2.0
Prediced Q (ncrmalized)
0.a.
0.6
0.6
0.4
0.8
02-
02
0.5
0/2
0/4 0.s
1.0
butions. First, we compute critic pre-
MC Return (ormalized
Return (normaiized)
dictions Q(.,, a.) and Monte Carlo returns G(s ay from transitions of 5000 unseen trajectories. We then partition transitions into bins by Q values and plot the average & within each bin. The deviation from the diagonal y quantifies overestimation (above) underestimation (below). As shown in Figure 5, DEAS exhibits a calibration curve consistently closer to the diagonal than QC-FQL, indicating reduced overestimation and improved value alignment. These findings support the claim that DEAS's detached value learning framework with distributional objectives provides more reliable value estimation than baselines that incorporate actor-critic interactions in extended, high-dimensional action spaces.
Robustness to data qualities To validate the robustness DEAS across dataset qualities, we con- struct mixed datasets by combining play and noisy datasets provided by OGBench, where noisy datasets contain more suboptimal transitions, at different ratios. We measure success rates and ana- lyze calibration curves of DEAS and QC-FQL in Figure 6. DEAS outperforms QC-FQL across all data regimes, demonstrating robust performance under varying data quality. Additionally, DEAS's calibration curve remains consistently closer the diagonal across all regimes, which further sup- ports the effectiveness of our method.
5.4 ABLATION STUDIES
We investigate the effect of hyperparameters and various components of DEAS by running experi- ments OGBench 4uzzle-4x4 task.

---

Published as a conference paper at ICLR 2026
Sucocsa tare 7h1
Dcria
Irdicco O losamnal 225-0
medicedd Q Insrmneton
1.2
trediceed (nommllzss
1.0
GCFO.
20
0.4
0.
0.0
64
1.5
16
222
06
(a) Success Rate (%)
(b) 50% noiey 50%
(c) play 70% noisy 30%
(d) play 90% noisy 10%
Figure 6: Success rate (%) (a) and value calibration curve (b,c,d) across different mixtures on cube- quadruple task. DEAS exhibits not only higher task performance but also an improved value cali- bration closer to the diagonal across all data regimes.
Actor
512 x
512 x
25 #5
#3
256 x 4
Critic
256 x
Value
69 #7
SR
TQI.
HL.G
75
5R
0.8
71
0.959
72
87 :44
SR
512 x 4
75 18
512 x
256 x
88 1:4
63 26
0.9
0.999
512
1024
256 x
91 +44
88
0.99
0.999
81 4-5
16
512 x
51 1:4
512 x
512 x
50 4:4
0.999
0.999
80 4:8
1024
(b) Critic size
(c) Objectives
(d) ?1 and
(a) Action sequence
Table 4: Ablation studies. We investigate the effect of (a) action sequence length H, (b) critic and value model size, (c) training objectives, and (d) separate discount factors Y1 and %2. SR denotes success rate (%) and default settings are highlighted in gray... Bold indicates values at or above 95% of the best performance.
Effect of action sequence length Table 4a investigates the impact of action sequence length on performance. When using single-step or two-step action (H 1, 2), DEAS fails to achieve meaning- ful performance, confirming the necessity of action sequences for long-horizon tasks. Performance improves with longer sequences, but when the sequence length becomes longer than 8, it requires proportionally larger actor networks to handle the increased action dimensions, suggesting trade- off between sequence length and computational efficiency.
Effect of network size Table 4b analyzes the sensitivity to network sizes. For the critic network, observe that increasing capacity initially improves performance by better approximating the value function. For the value function, we find that the network needs sufficient capacity to cap- ture the complexity of action sequence values, but excessive capacity without proper regularization causes instability in value estimation, leading to performance degradation.
Effect of training objective In Table 4c, compare different training objectives for value estima- tion. We found that using only distributional RL (HLG) (Farebrother et al., 2024) or only standard regression (IQL.) shows limited performance. However, combining detached value learning with distributional estimation significantly improves results, suggesting both components are crucial for stable training with action sequences.
Effeet of dual discount factors Lastly, we examine the effect dual discount factors on learning dynamics in Table 4d. Proper tuning of 7 (the discount factor for action sequences) is essential for performance, as controls the temporal horizon for value estimation within sequences. In this paper, 71 0.9 for all experiments.
CONCLUSION
We introduced DEAS, simple but effective offline RL method that exploits action sequences to learn efficiently on complex tasks. DEAS provides a practical recipe for modeling temporally ex- tended actions while avoiding value overestimation via detached value learning and distributional networks, yielding a principled reduction of the effective planning horizon that is critical in long- horizon settings. Empirically. it consistently outperforms strong baselines challenging OGBench tasks and scales to large VLAs across simulation and real-world experiments, demonstrating its potential to bring offline RI. closer real-world applications.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide full hyperparameter and implementation details in Section and Section B. In addition, facilitate reproduction, we release an open-source implementation on the project website.
ACKNOWLEDGMENTS
CY thanks Jachyun Nam, Juyong Lee, Yonghoon Dong, and anonymous reviewers for providing helpful feedback and suggestions for improving our paper. CY also thanks Angeline S. Kim for as- sistance with improving the paper's expression and visualization. This work was partly supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded Artificial Intelligence Graduate School Pro-
by the Korea government (MSIT) gram (ann(AAK (AIST); 2SS220240509229, Global AI Frontier Lab) and the National Research Foundation of Korea (NRF) grant funded by the Korea govermmen(MSTT) (RS-2024-0041488)
REFERENCES
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference Machine Learning, 2017.
Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.
Johan Bjorek, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox. Fengyuan Hu, Spencer Huang, et al. Gr0Ot nl: An open foundation model for gener- alist humanoid robots. arXiv preprint arXiv:2503. 14734, 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: vision- vision-language language-action flow model for general robot control. In Robotics: Science and Systems, 2025.
Steven Bradtke and Michael Duff. Reinforcement learning methods for continuous-time markov decision problems. In Conference on Neural Information Processing Systems, 1994.
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, al. Q-transformer: Scalable offline reinforcement learning via autoregressive f-functions. In Conference on Robot Learning, 2023.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelit generative behavior modeling. In International Conference on Learning Repre- sentations, 2023.
Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao. Con- rft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.088 054500 2025a.
Zengjue Chen, Runliang Niu, He Kong, and Qi Wang. Tgrpo: Fine-tuning iision-lagguag--action model via trajectory wise group relative policy optimization. arXiv preprint wxXiv:2506.084400 2025b.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. International Journal of Robotics Research, 2023.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Ir- pan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RI.. In International Conference on Machine Learning. 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219, 2020.

---

Published as a conference paper at ICLR 2026
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach offline reinforcement learning. In Conference on Neural Information Processing Systems, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2019.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL. without entropy. In International Conference on Learning Representations, 2023.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Conference on Neural Information Processing Systems, 2020.
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen. Improving Assionlanguaag-acttin model with online reinforcement learning. arXiv preprint arXiv22501.1666, 2025.
Philippe Mansen-Estruchh Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606. 08415, 2016.
Donghhi Huang, Zhirui Fang. Tianle Zhang, Yihang Li, Lin Zhao, and Chunhe Xia. Co-rft: Effi- cient fine-tuning of vision-language-actinn models through chunked offline reinforcement learn- ing. arXiv preprint arXiv:2508.022 022/9, 2025.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky, Wu.s: acction
vision- sasion-lnngage-acce action model with open- open-world generalization. arXiv preprint arXiv:2504.16054 2025.
Michael Kearns and Satinder Singh. Bias-variance error bounds for temporal difference updates. In Conference on Learning Theory, 2000.
Diederik P Kingma. Adam: A method for stochastic optimization. In International Conference
Learning Representations, 2015.
Iyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations, 2022.
Tejas Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Conference on Neural Information Processing Systems, 2016.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Conference on Neural Information Processing Systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Conference on Neural Information Processing Systems, 2020.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. In International Conference on Learning Representations, 2023a.
12

---

Published as a conference paper at ICLR 2026
Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials. In Robotics: Science and Systems, 2023b.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning: State-of-the-rr, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann. Top-erl: Transformer- based off- policy episodic reinforcement learning. arXiv preprint arXiv:2410.09536, 2024.
Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuckai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674 2025a.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. arXiv preprint (tV.:250.00769 2025b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer-
ence on Learning Representations, 2019.
Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.03299; 2021.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In Conference Robot Learning, 2023.
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy agmostic rl: Offline rl and online rl fine-tuning of any class and backbone. arXiv preprint arXiv:2412.00685, 2024.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Conference Neural Information Processing Systems, 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.0933: 09359, 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark. Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- tuning. In Conference on Neural Information Processing Systems, 2023.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. In Conference on Robot Learning, 2024.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Roboties: Science and Systems, 2024.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample efficient continuous control. In Con- ference on Neural Information Processing Systems, 2024.
NVIDIA. GrOOt nl.5 15:5: An improved open foundation model for generalist humanoid robots. https: June 2025, Accessed: 2025-09-09.
//research ch, nvidia. www.coo///aa/c http///aabgggeagrrttoonlll/
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference Machine Learning, 2018.
13

---

Published as a conference paper at ICLR 2026
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? In Conference on Neural Information Processing Systems, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In International Conference on Learning Representations, 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. In Conference on Neural Information Process- ing Systems, 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning, 2025c.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintege-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Conference Neural Information Processing Systems, 1988.
Younggyo Seo and Pieter Abbeel. Coarse-to- to-fee q-network with action sequence for data-efficient robot learning. In Conference Neural Information Processing Systems, 2025.
Jost Tobias Springenberg. Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Maria Elisabeth Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning, 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999.
Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl. Ript-vla: Interactive post-training for vision-languagg-actionn models. arXiv preprint arXiv:2505. 17016, 2025.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. In Conference on Neural Information Process- ing Systems, 2023.
Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-base soft actor-critic with n-step returns. arXiv preprint arXiv:2503.03660, 2025.
John Tsitsiklis and Benjamin Van Roy. Analysis of lemporal-difffereece learning with function approximation. In Conference on Neural Information Processing Systems, 1996.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hicrarchical reinforcement learning. In International Conference on Machine Learning, 2017.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In International Conference on Learning Representa- tions, 2023.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. In Conference on Neural Information Processing Systems, 2020.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang. Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline RI. with no OOD actions: In-sample learning via implicit value regulariza- tion. In International Conference on Learning Representations, 2023.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Metd-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2020.
14

---

Published as a conference paper at ICLR 2026
Hongyin Zhang, Zifeng Zhuang, Han Zhao, Pengxiang Ding, Hongchao Lu, and Donglin Wang- Reinbot: Amplifying robot visual-language manipulation with reinforcement learning. arXiv preprint arXiv:2505.07395, 2025.
Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. Grape: Generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19300 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning ane-grained bimanual manipulation with low-cost hardware, In Robotics: Science and Systems, 2023.
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND FUTURE WORK
While DEAS demonstrates significant improvements over existing offline RL methods, several lim- itations and opportunities for future research remain. First, our current approach uses fixed action sequence lengths across different tasks, whereas optimal sequence lengths vary significantly depend- ing on task complexity. One natural direction is to extend DEAS to even longer horizons, potentially with explicit hierarchical or option-based sequence generation. The other to investigate adaptive mechanisms that can dynamically adjust action sequence lengths based on task requirements, po- tentially by adopting hierarchical policies (Kulkarni et al., 2016; Vezhnevets et al., 2017; Nachum et al., 2018), which would be an intriguing research direction.
Second, while DEAS shows promising results on individual tasks, scaling to large-scale unified value functions remains critical challenge for real-world deployment. DEAS currently trains re- ward models on 3-4 tasks simultaneously, but practical applications require learning from hundreds or thousands of diverse tasks. Future research should focus developing scalable architectures and training procedures that can handle massive multi-task datasets while maintaining sample efficiency and avoiding catastrophic forgetting.
Third, our method relies on distributional RL with fixed support ranges (Vmin= Vmax) and discretiza- tion parameters, which can significantly impact performance. The sensitivity to these hyperparame- ters limits the method's s robustness across different domains and reward scales. Future work should develop more robust frameworks that can automatically adapt to different reward distributions or provide principled ways to set these parameters.
B IMPLEMENTATION AND TRAINING DETAILS
B. OGBENCH EXPERIMENTS
Tasks We evaluate our method on 6 URS Robot Arm manipulation environments from OG- Bench (Park et al., 2025a), each with subtasks. All tasks are state-based, and goal-free setup. For each task, the observation space consists of the proprioceptive state of the URS Robot Arm, and low-dim state vector informing the target object state and position. The action space consists of the cartesian position of URS robot arm, gripper yaw, and gripper open/close. For substituting condicondi environment to standard function, we use the simple semi-sparse reward function, which is defined the negative number of uncompleted subtasks in the current state, following Park et al. (2025a). For all tasks, the maximum episode length is set to 1000.
Implementation details We implement our method top of the open-source implementation of FQL (Park et al., 2025c) 1. Unless otherwise mentioned, we largely follow the training/evaluation setup and network architecture from Park al. (2025c) and Park et al. (2025b). For training value network, we use the smaller size network compared to critic network for all experiments, which shows the best performance, and we use the doubled size of network for the critic network. For cube experiments, we use BRO (Nauman et al., 2024) for additional regularization between relatively small range of returns value function training. For selecting Vmin and Vmax for distributional RL, we use two procedures: 1) data-ceniric: compute return distribution from the dataset and select I 1% and 99% quantiles with 20% padding, and 2) universal: compute theoretical bounds using reward range /'in: ]maa] horizon L, action sequence length H, and discount factors 21, 72. In this case, the theoretical bounds are:
1 1- 1f 1 - K
Vmin = Tmin
(9)
1- 72 71 1 1 7/ 2 T'ax
Vmax
(10)
72 1 M
where ?1 and 72 denote discount factors for inner-sequence and across-sequence-level decision, respectively.
16

---

Published as a conference paper at ICLR 2026
Training and evaluation For the training dataset, dataset released by Park et al. (2025a) 2, where the dataset is collected by open-loop. non-Markovian scripted policies with temporally correlated noise. As 100M dataset consists of 100 separate files with 1M transitions for each, we use the first 10 files sorted by name for 10M dataset. We train our method and baselines for 1M (IM data) / 2.5M (10M/100M data) gradient steps. For selecting BC coefficient for policy extraction, we first normalize the Q loss as in Fujimoto & Gu (2021) and sweep the value from {0.1, 1, 10} and choose the best one for each task and baseline, except cube - double, where we follow the hyperparameter used in Li et al. (2025b). For eval- uation, we report the average success rates across the last three evaluation epochs (800K, 900K, 1M for IM dataset, 2.3M, 2.4M, 2.5M for 10M/100M dataset) following Park et al. (2025c) and Park et al. (2025b). For checking additional hyperparameters used our experiments, please refer to Section B.3.
the open-sourced IM/100M play
Baselines For reporting results from FQL and n-step FQL, we use the implementation from Park et al. (2025c). For Q-Chunking. we re-implement the code from Li et al. (2025b) in our codebase. We found that simply increasing discount factor 7y leads to significant performance improvement for Q-Chunking, we use the discount factor be same with 722 for value function training. For implementing CQN-AS, we use the original implementation released by the authors from Sco & Abbeel (2025)* and integrate OGBench related codes on top of the codebase. Originally, CQN-AS is designed to apply auxiliary BC loss only on expert demonstrations, but considering the dataset distribution of OGBench tasks with nearly no success rollouts, we modify the BC loss on the subop- timal data well (Fujimoto & Gu, 2021; Park et al., 2025c; 2024), where no significant difference with the original implementation. As the reward scale for OGBench is highly different according to the domain, we normalize the reward scale to be in [-1, 0]. and use vmin and V "max as 200 and O, respectively. For levels and bins, we use (level) and (bins) for all experiments.
Computing hardware For all OGBench experiments, we use single NVIDIA RTX 3090 GPU with 24GB VRAM and it takes about 2 hours for training the small model (used for IM dataset) and about 8 hours for training the large model (used for 10M/100M dataset).
B.2 VLA EXPERIMENTS
Computing hardware For all VLA experiments, we use NVIDIA A A100 SOGB GPUs. Fine-tuning GROOT NI.5 takes about 4 hours for 100 expert demonstrations and successful rollouts. For training DEAS and baselines, it takes about 10 hours with the same data,
use a larger batch size.
VLA fine-tuning We implement our method and baselines on top of the open-source implementa- tion of GROOT N1.5 (NVIDIA, 2025) 5, As our code is based on an earlier version of GROOT N1.5, we conduct experiments without introducing future tokens to the action expert modules. For fine- tuning GROOT N1.5, we use a batch size of 32 and train for 30K (RoboCasa Kitchen) 10K (Real Robot) steps using AdamW (Loshchilov & Hutter, 2019) optimizer with learning rate 1 x 10-4 and cosine annealing schedule.
B.2.1 ROBOCASA KITCHEN EXPERIMENTS
Task RoboCasa Kitchen (Nasiriany et al., 2024) is simulation environment with mobile manipulator attached to a Franka Panda robot arm in household kitchen environments. Among 24 atomic tasks provided by the environment, we select 4 challenging tasks (CoffeeSetupMug, PnPMi MaaMiccroaveTooounteer relatively
crowave, PnPMicr MaPMiccrovaeeootoov and delicate manipulation with small grasping part, which is demonstrated by the low success rate of the base model. For perception, camera images from 3 different view- points (left front, right front, wrist), proprioceptive states including position/velocities of joint/base, and natural language instructions, are provided. For reward function, we use the pre-defined success detector in the environment, and use the sparse reward function where the reward is 1 the task is
that require
17

---

Published as a conference paper at ICLR 2026
Franka Hand
Franka Research 3 Robot Arm
Camera Setup
Visual Observation
Realsense D435 Front Camera
Realsense D435 Wrist Camera
Front
Wrist
Figure 7: Real-robot platform.
completed, and ( otherwise. Following previous works (Kumar et al., 2023b; Mark et al., 2024), we utilized the practical heuristic annotating the last = transitions every successful trajectory with reward of +1.
Implementation details As input for the value function, we first use the proprioceptive states from the robot, including joint position/angle, base position/oriennation for the mobile manipulator. To provide information target objects to the value function, utilize the encoded representation of three different camera views and task instructions from the VLM backbone. For the value/critic network architecture,
the same hyperparameters those used for the 100M dataset experi- ments. For optimizing value and critic function, we use the expectile parameter as 0.7, 71 = 0.9, 72 0.99, and universal support type for distributional RL, for all experiments. For selecting action candidates with the value function, we first sample N = 10 candidates from the policy. For select- ing final actions, we try either 1) greedy sampling with highest Q-value or 2) inspired by Nakamoto et al. (2024), sampling the action from categorical distribution obtained by temperature controlled softmax Q-values: a, Softmax Q(s..a)) the best result for each task.
Q(s),N)) with temperature
and report
Training and evaluation For expert demonstrations, we randomly sample 100 expert demonstra- tions using the publicly available dataset generated by MimicGen (Mandlekar et al., 2023). For training DEAS and baselines, we use a batch size of 64 and train for 30K steps using Adam opti- mizer with learning rate of 3 x 10 For collecting rollouts, we use randomized environments using the object instance set A. For each task, we evaluate the model performance across trials on five distinct evaluation scenes with 3 different evaluation seeds, totaling 150 rollouts. To test generalization capabilities, we evaluate the policy only on unseen object instances.
B.2.2 REAL ROBOT EXPERIMENTS
Hardware platform We use Franka Research 3, 7-DoF robotic arm, for our experiments. For visual perception, we utilize the dual camera with Intel RealSense D435i: camera attached to column next to the robot base to provide global view, and wrist-mounted camera for close- range view. Teleoperated demonstrations are collected using an Oculus Quest 2, and we log time- synchronized RGB images, joint states, and gripper width for data collection. Demonstrations recorded at 15 Hz. See Figure for visual examples.
Task We evaluate the model performance on bick-and-plae tasks from the countertop to the bot- tom cabinet, with three different objects: peach, mi mikka, and hi hichew. Each object has different
18

---

Published as a conference paper at ICLR 2026
Figure 8: Initialization points used for pick-and- place tasks.
properties: peach is rigid object with a relatively larger size that is easy to occlude, milka is deformable object with a relatively smaller size that is easy to deform, and hichew is hard object requiring precise grasping due
small width. For collecting demonstrations, we use different initialization points (center, top, bottom, left, right) and collect one demonstration for each position (see Figure for the initialization points used in our experiments). For accurate value function es- timation, we manually label the reward function for each task. Specifically, we split the task into 4 stages: 1) moving to the countertop. 2) picking up the object, 3) moving to the target position, and 4) placing the object. For each stage, label the reward function if the task is completed, and 0 otherwise, and we set the reward function as the negative number of uncompleted stages following Park al. (2025a).
Implementation details Unless otherwise mentioned, we follow the same implementation details as the RoboCasa Kitchen experiments. For selecting final actions, we use 50 candidates from the policy and use the same procedure for selecting the final action as in the RoboCasa Kitchen experiments.
B.3 HYPERPARAMETERS
We list the hyperparameters used in our OGBench experiments in Tables 5 and 6. For the BC coef- ficient used for policy extraction, please refer Table 7.
Table 5: DEAS hyperparameters OGBench experiments.
Hyperparameter
Value
Gradient steps
1M (IM dataset), 2.5M (10M/100M dataset)
Optimizer
Adam (Kingma, 2015)
Learning
0.0003
Batch size
256 (1M dataset), 1024 (10M/100M dataset)
Actor MLP size
[512, 512, 512, 512] (IM dataset)
1024, 1024, 1024, 1024] (10M/100M dataset)
Critic MI.P size
2256, 256, 256, 256 (1M dataset)
512, 512, 512, 512) (10M/100M datasel)
Value MLP size
128, 128, 128, 128
(IM dataset)
256,, 256, 256, 256] (10M/100M dataset)
Nonlinearity
GELU (Hendrycks & Gimpel, 2016)
Layer normalization
True
Target network update rate
0.005
Discount factor 7i
0.9
Discount factor ?22
0.995 (cube), 0.999 (scene, (scene,puzzle)
HL-Gaussian Atoms
HL-Gaussian
0.75
Hl.-Gaussian Support range type
data-centric (cube), universal (acene, puzzle)
Flow steps
10
Critic ensemble size
2
Action sequence length H
4 (cube), 8 (scene, puzzle)
Expectile (DEAS)
0.9 (IM dataset), 0.95 (10M/100M dataset) min Q2) Table
Double Q aggregation
Policy extraction hyperparameters
19

---

Published as a conference paper at ICLR 2026
Table 6; Baseline hyperparameters for OGBench experiments.
Hyperparameter
Value
Critic MLP size
512, 512, 512, (1M datasct)
[1024, 1024, 1024, 1024] (10M/100M dataset)
Discoumt factor 7 (FQL, re-tep FQL)
0.99
Discoumt factor y (QC-FQL)
0.995 (cube), 0.999 (puzzle)
Horizon reduction factor
4 (cube), 8 (puzzle)
Policy extraction hyperparameters
Table 7
Levels (CQN-AS)
5
Bins (CQN-AS)
9
C51 Vmin. V Vmxx (CQN-AS)
-200,
Table 7: Policy extraction hyperparameters for OGBench experiments. Note that apply Q- Normalization (Fujimoto Gu, 2021) actor loss, except cube-double tasks.
Task
FQL
n-step FQL
QC-FQL
DEAS
scene
3
1
3
3
cube-double
300
100
300
300.0
puzzle-3x3
3
1
3
cube-triple
3
1
1
puzzle-4x4
3
1
3
quabruu
3
1
1
C EXTENDED RELATED WORK
Reinforcement learning with VLAs Recent efforts have applied RL to VLA training (Zhang et al., 2024; Chen et al., 2025a; Zhang et al., 2025; Guo et al., 2025; Tan et al., 2025; Chen et al., 2025b; Li et al., 2025a), but most focus on on-policy online RL, which requires expensive interac- tions and cannot reuse transitions. A key limitation is that existing methods functions Qss,a a) for value learning, despite modern VLAs being designed to predict action se- quences (Black al., 2025; Bjorck al., 2025; Intelligence al., 2025). This mismatch between single-step value learning and multi-step action prediction limits the effectiveness RL with VLAs. The most related work CO-RFT (Huang al., 2025), which applies chunked offline RL VLA training, but differs from our approach in three key aspects: (1) CO-RFT uses actor-critic meth- ods (Nakamoto al., 2023) with single-step value functions while DEAS uses detached value learning with action sequences, (2) CO-RFT relies on human telcoperated expert demonstrations while we use small expert sets with large suboptimal rollouts, and (3) CO-RFT requires sophisti- cated transformer architectures while DEAS achieves improvements with simple MI.P networks.
single-step value
D USE OF LARGE LANGUAGE MODELS
We acknowledge the use of large language models (LLMs) in preparing this manuscript. LLMs were employed solely to refine writing quality, including grammar correction, vocabulary suggestions, and typographical checks. All substantive ideas, analyses, and conclusions in this paper are entirely work of the authors.
20

---

Published as a conference paper at ICLR 2026
E PROOFS
In this section, we will show that DEAS leveraging action sequences can recover the optimal value function under the dataset support constraints following flows of Kostrikov al. (2022) with the primitive action a replaced by the sequence action o A
Lemma E.1 (Expectile limit and monotonicity; ef. Lemma of Kostrikov al. (2022)), Let X be a real- valued random variable with bounded support and let x² == sup( E Pr(X $ x) < 1) denote the supremum of its support. For (0, 1), let my be the T-expectile of X, i.e.
my = arg min EE - 1fx < (X mm)] mER
Then:
1. For any < T1 < T2 < 1, we have My, I Myz
2. lim my = x.
r-1
Let po | 8) denote the empirical behavior distribution over H-step action sequence a, = (a....., a++H-1) E AA extracted from the dataset. Given the multi-step return R(s,, ag. 71) and discount 72 across sequence-level decision points, define the optimal solutions of Equation and Equation (5) by
V, (s) argmin E Eo- (11) [L, -))]
(11)
oER
Q+ a) R(s, a. m) + # ( 8= 5, =o Lemma E.2 (Action-sequence analogue of Lemma 2). Let 0 < < < 1. Then, for all E S,
(12)
(s) < (s).
Proof. The proof Lemma Kostrikov al. (2022) uses (i) Lemma E.1 and (ii) the Bellman recursion for Q2 and V,. In setting, Equation (11) and Equation (12) play the role of the original value and Bellman definitions, with A in place of a E A. Replacing by o in the argument reproduces the same inequalities verbatim, yielding V,, (s) s (s.
(s) for all
We now define the dataset-comsttainned optimal value function Q* with action sequence a, as:
Q" (s, a) R(s, 7) 72
H max d'Eupppa ())
Q* d') 5, a. of
(13)
Corollary E.2.1 sequene analogue of Corollary 2.1). For any T E (0, 1) and any S, (s,
V+(s) S
max
oESupplan
where V, defined Equation (11) and Q* is given by Equation (13).
Proof. For each fixed the expectile (+(s is convex combination of the values Q+ for in the support of pn 8):
(s(s) = Σ wi
w; 2 O, Σ wi = 1, or E Supp(pa( s)).
Therefore
Vi(s) <
max oESupp(nn
(s)
Theorem E.3 (Action-sequence analogue Theorem 3). For all E S, (s,
lim oESupp(ant(0)
V=(a)
max
where Q" is the dataset-constrainet optimal sequence-action value function defined in Equa- tion (13).
21

---

Published as a conference paper at ICLR 2026
Proof. Lemma E.1 applies directly the scalar random variable (s, a) and implies that the т- expectile of this random variable converges to its supremum as 7 - 1. Combining this fact with Lemma E.2 and Corollary E.2.1 yields the claimed limit, exactly as in the proof of Theorem 3 in Kostrikov et al. (2022).
In summary, we show that the results of Section 4.4 Kostrikov al. (2022) hold without modi- fication when the single action is replaced by a fixed-length action sequence E A# Moreover, integrating IQL with action sequences and distributional RL is also theoretically sound, as estab- lished by the same arguments in Bellemare et al. (2017; 2023).
22

---

Published as a conference paper at ICLR 2026
F OGBENCH DATASET STATISTICS
Table 8: Dataset statistics for OGBench tasks.
Taak
Rate
Min
scone- cingletaast tackl-19
5.0 (6000000))
-5.0
0.0
-3.1733255
L0295831
-4823.0
-1.0
-1557.8765
986,0316
tas22-2
(40/1000)
-5.0
0.0
-3.163389
0.9956842
-4302.0
0.0
-1553.0287
971.3409
5.5 (66/1900))
-5.0
0.0
-3.155803
L0406192
-4765.0
0.0
-1551.890
985.3525
10.1 (101/1000)
-5.0
0.0
-3.091423
1000534
-4696.
0.0
-1494 2867
965.7743
0.700/)0000
-2.0
-1.949015
022175753
-20010
0.0
-974-9428
567,5881
(171000)
-2.0
8.0
-1,949817
2000.0
-976.5885
(17/100)
-2.0
0.0
-1.999555
0:2221605
-20000
0.0
-976
567. 5661
2.9 1000)
-2.0
0.0
-1,9738399
0.16801369
-2000.0
0.0
-958.5528
572,3737
6.5 (65/1000)
-9.0
0.0
-4 49132
1.9922006
-5459.0
0.0
-2349 (69
1317.5267
52 (52/1000)
-9.0
0.0
505488
L
-562.0
-1.0
-2552 3616
1321 4901
(55/1000)
-9.0
4,496664
1.58579
0.0
2212
1318.7554
(48/10000)
-9.0
0.0
1.9911112
-5630.0
-1.0
0.01 (1/10000)
-3.0
0.0
-,99222252
0.272244
3000.0
-1.0
-1663.3553
850.65326
0,03 (3/10000)
-3.0
0.0
-29227301
0,27177206
-50000
-1.0
-1463.7299
850.6014
0.02 (2/10000))
-3.0
0.0
-2.9219937
0,275608
3000.0
-1.0
-1462 3278
850.748 78963
0.04 (4/10000)
-3.0
0.0
-2.9230156
0.27888177
-30000
-1.0
-1463 4613
850.6792
(17100000
-3.0
8067
0,02 (2/10000)
-8.006288
2068
pazele-444pples
(2/18000)
-16.0
0.0
2.0014088
-2.0
7666
2351.26
ta
0.04 (3/10000)
-15.0
0.0
-7.99373
1,995151
10055.0
-1.0
8801
9917
0.04 (4/10000)
-16.0
0.0
-7,999333
1.991151
10055.0
-1.0
4001.889
2349. 9917
0.001 (1/100000)
-4.0
0.0
-3.001978
D. 0.30891102
-00000
-1.0
-1954 3688
11 1134,5515
cube seaadrppeeetnnlltattee
0,002 (2/000000
-4.0
0.0
-3. -38890014
0.3141 3057
-4000.0
-1.0
-1952.7515
1133,8800 88077
qube-quadrppe--a
0.001 (1/101000)
-4.0
0.2897 2844
-000.0
-19612894
1157.1301
cube
0.28862227
cube uube-qiadrapler wtg.le/siggglttaa--aeee.
(1210000)0000
-3.962122
0.18561256
0.0
11926.1152
23

---

Published as a conference paper at ICLR 2026
G FULL EXPERIMENTAL RESULTS
We include the full experimental results in OGBench experiments in Table 9.
Table 9: Full offline RL Results in 30 OGBench tasks. indicates the default task in each environ-
ment. We report the success rate (%) and 95% stratified bootstrap confidence interval over 4 runs.
Task
#Data
FQI.
N-step FQL
QC-FQL
CQN-AS
DEAS
arear-payysinnnatttakttaaka--
100
100 =0
99 =0
2
99 =1
+3
99
97 =1
IM
64
75
3 12
0
68
65
0 10
0 20
35
45
46
17
68
7
76
10
1 10
47
51
1M
11
40g6
0.
47 1/4
+
8
+1
44+3
57
97
100 :00
19
40
81
94
IM
15
14
50 111
20
91
35 #4
23
31 =:
0.
91 =3
=11
=0
96
::14
91 15
100 ::0
98 ::1
92 #2
95
10M
92
88
(
0.00
59
45
2 111
0 1o
74 114
:10
87
54
28
66 117
2:
92
24 =:
2 =1
80 #:10
=0:
42 =77
10M
28 +7
=22
=0
99 +1
88
22
312
G1
auu
89
79
92
97
63
100
cube cuueruuurrrplpppaaaaaattttttttte
100M
613
1:
33 17
62 19
cube hate--uudrrrllelpyyy-iiglltttttd
13 13
38 177
1.
31 117
0.20
:6
4:10
24


Reviews:
[{"review_id": "Dl9L4EB5QT", "reviewer": "Reviewer_FBjU", "summary": "This paper presents an offline RL framework that tackles the value-overestimation problem through detached value learning. To improve stability, it also integrates distributional RL and dual discount factors for intra- and inter-option transitions. Empirically, DEAS is shown to outperform baselines on 30 long-horizon OGBench tasks and to improve large Vision-Language-Action (VLA) systems on both RoboCasa simulations and real Franka robot manipulation tasks. The main contributions are: (1) framing sequence-level offline RL under SMDP Q-learning with detached critics, (2) a recipe (distributional RL + dual discounts) for stable learning with action sequences, and (3) demonstrating practical gains on long-horizon benchmarks and VLA applications.", "questions": "1. Do you have experiments integrating DEAS (action sequences + detached critic) with representative methods such as CQL? For example, does adding detached sequence-level critics to CQL further reduce overestimation or improve long-horizon performance? \n2. To address potential value miscalibration, have you evaluated DEAS across datasets with varying proportions of expert trajectories (e.g., 10%/30%/50% expert)? How does data quality influence (a) final performance, and (b) the calibration of predicted Q vs realized returns? If absent, please add such sensitivity analysis or discuss limitations clearly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experiments applying DEAS to large VLA models are a strong selling point: DEAS improves VLA performance using mixed-quality offline rollouts plus a small set of expert data, which is important for real-world robotics.\n2. The manuscript is structured logically and presents a well-targeted technical design.", "weaknesses": "1. My main concern is that the paper’s core components are largely integrations or adaptations of prior work, such as IQL-style detached learning, distributional RL methods, and option/sequence mechanisms similar to Q-Chunking.\n2. Detached value learning updates the critic using in-distribution high-return actions, but when the offline dataset contains predominantly suboptimal trajectories, this strategy may still yield biased or miscalibrated value estimates (under- or over-pessimism). The paper does not adequately verify whether the learned value function aligns with true returns across different data quality levels. I recommend that the authors analyze Q-error across bins of trajectory quality and conduct sensitivity studies using datasets with controlled expert-to-suboptimal ratios (e.g., 10%, 30%, 50% expert data).\n3. Several key equations are unnumbered and inconsistently referenced, which increases the difficulty of understanding Algorithm 1.\n4. Although the central motivation of DEAS is to alleviate value overestimation induced by action sequences, the experimental results only report task-level success metrics without providing direct analyses of overestimation (e.g., predicted Q-values vs. realized returns). Including such quantitative evidence would substantially strengthen the paper’s claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iXW1PFyYVo", "reviewer": "Reviewer_EcDt", "summary": "The paper addresses the challenge of scaling offline reinforcement learning to complex, long-horizon tasks. The authors identify that while using action sequences (instead of single actions) can reduce the effective planning horizon, it introduces severe value overestimation when combined with standard actor-critic frameworks.The core contribution is DEAS (DEtached value learning with Action Sequence), a method that learns a value function over $H$-step action sequences. To prevent value overestimation, DEAS \"detaches\" the critic and value function training from the actor. It does this by adopting an IQL-style expectile regression loss, which learns the value function by regressing only on the in-sample action sequences found in the offline dataset. This avoids querying the actor for out-of-distribution actions.The framework is stabilized by incorporating distributional RL and dual discount factors ($\\gamma_1$ for intra-option rewards, $\\gamma_2$ for inter-option values). The authors demonstrate that DEAS significantly outperforms existing single-step and action-sequence baselines on complex OGBench tasks and can be used to improve the performance of large-scale Vision-Language-Action (VLA) models in simulation and on a real robot.", "questions": "1.  Can the authors clarify the novelty of the method beyond the combination of IQL and action sequences? Is the primary contribution the empirical demonstration that this combination is effective, or is there a more fundamental insight that I have missed?\n2.  Regarding the scalability weakness: The paper notes the actor network must scale with $H$. This seems to be a major bottleneck. Have the authors investigated methods to decouple this than the entire $H$-step block at once?\n3.  Given that the duration $H$ is fixed and deterministic, why was the method presented as an SMDP rather than, more precisely, as a standard MDP with a temporally-extended action space $\\mathcal{O} = \\mathcal{A}^H$? I suggest authors give this paper a major re-write and detach concepts such as SMDPs and options out of scope.\n4.  Given the engineering success and strong empirical performance, I am happy to raise score if authros position their contribution from a more rigorous view.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  **Clear Problem Identification:** The paper clearly articulates a significant and practical problem: coupled actor-critic methods are unstable when the action space is $\\mathcal{A}^H$.\n2.  **Strong Empirical Results:** This is the paper's greatest strength. DEAS demonstrates state-of-the-art performance, with significant gains over baselines on the challenging OGBench tasks.\n3.  **Real-World & VLA Validation:** The authors successfully scale the method to large VLA models and demonstrate its effectiveness and stability on a real robot. The fact that the QC baseline *collapses* on the real-world task while DEAS improves performance is a very strong supporting data point.\n4.  **Effective Ablations:** The ablation studies in Table 4 are well-executed and convincingly demonstrate the necessity of each component of the method (e.g., $H > 2$, the combination of IQL and distributional RL).", "weaknesses": "1.  **Limited Conceptual Novelty:** As stated in the *Contribution* section, the method is a synthesis of IQL, fixed-duration options, and distributional RL. The contribution is an empirical and engineering one, not a fundamental algorithmic advance.\n2.  **Imprecise Technical Terminology:** The paper repeatedly invokes \"SMDP Q-learning\" and the \"options framework\", but the method itself is not a true SMDP. The use of a *fixed, deterministic* duration $H$ does not create an option induced Semi-Markov process. The system is, more accurately, a standard 1st-order MDP where the agent selects from a temporally-extended, high-dimensional action space $\\mathcal{O} = \\mathcal{A}^H$. This imprecise language is confusing and oversells the connection to the general options framework. From this point of view, it's only another variant of ordinary chunked action sequences and the performance boost whether come from temporal abstraction is skeptical.\n3. **Unjustified Optimality**: Detaching value functions and fixed horizon clearly alters the underlying decision process compares original SMDP. The convergence and optimality of DEAS is unjustified and not bounded.\n4.  **Poor Scalability with Horizon $H$:** The paper's own ablation study reveals a significant scalability limitation. The authors state that \"when the sequence length becomes longer than 8, it requires proportionally larger actor networks to handle the increased action dimensions\". This creates a direct and problematic coupling between the task horizon ($H$) and the required network capacity, which severely limits the method's applicability to *truly* long-horizon problems.\n5.  **Increased Hyperparameter Sensitivity:** The method adds several critical new hyperparameters that require tuning, including the sequence length $H$ (which Table 4a shows is task-dependent), and the dual discount factors $\\gamma_1$ and $\\gamma_2$, which the authors note are \"critical for stable training\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbfZIqett6", "reviewer": "Reviewer_uCyh", "summary": "The paper proposes DEAS, a method for offline reinforcement learning that models each decision as an action sequence rather than a single action. Building on the IQL framework, DEAS learns a critic consisting of $Q$ and $V$ networks while allowing an arbitrary policy architecture for action generation. During critic training, both the predicted and target values are treated as distributions and optimized via a cross-entropy loss. Experiments on continuous-control tasks, including both simulated and real-world robotic environments, demonstrate the effectiveness of the approach.", "questions": "1. How can the policy be effectively learned when each decision involves generating a high-dimensional action sequence? Does the paper propose any mechanism to mitigate the exponential growth of the action space?  \n2. Why do Equations (1) and (2) place the target distribution inside the logarithm term in the cross-entropy loss, and how should we interpret the missing parameter $\\theta$ in Equation (2)?\n3. How are the support bounds $v_{min}$ and $v_{max}$determined? These values appear to constrain the expressiveness of the critic network, while setting a larger range may make the learning process more difficult.\n\nMinor comments:\n1. The description of IQL in Section 3 is inaccurate. The paper states that “By using $\\tau > 0.5$, Equation 3 penalizes the overestimated value in out-of-distribution actions.” (By the way, there is no Equation 3.) However, since $u = Q - V$, where $Q$ is obtained from the target network and $V$ is the learnable network, the equation should instead penalize the **underestimated** values of the value network.  \n2. There is also no Equation (4.2), as referenced in Algorithm 1. Please verify and correct the numbering.  \n3. It is recommended to mention HLG again after Equation (2).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Effectively leverages action sequences to address sparse rewards and long-horizon credit assignment challenges.\n2. Appropriately builds on the IQL framework for stable critic learning under the offline setting.\n3. Demonstrates strong empirical performance on complex, long-horizon continuous-control tasks in both simulation and real-world environments.", "weaknesses": "1. The novelty of the paper is somewhat limited. It mainly combines existing ideas from action-sequence RL, IQL, and HLG.  \n2. The cross-entropy loss formulations in Equation (1) and Equation (2) are confusing. Typically, the predicted distribution appears inside the logarithm term and the target distribution outside. In contrast, both equations place the target inside the log term, which is unconventional. Moreover, in Equation (2), the role of parameter $\\theta$ on the right-hand side is unclear, as only $\\psi$ and $\\hat{\\theta}$ are explicitly shown.\n3. The paper provides limited discussion on how to learn the policy based on the trained critic. Since the policy outputs an entire sequence of actions per decision, the resulting product action space becomes extremely large, making policy learning challenging. However, the authors do not explain how this issue is addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ueKWOgOT6S", "reviewer": "Reviewer_dtkT", "summary": "This paper proposes a value learning framework that leverages action sequences for value learning to facilitate complex, long-horizon sequential decision making. Experimental results on OGBench, RoboCasa Kitchen, and real-world tasks demonstrate the effectiveness of the proposed method.", "questions": "Have you compared your method with other competitive baselines such as CQL [1]?\n\nReference:\n\n[1] Kumar et al. \"Conservative Q-Learning for Ofﬂine Reinforcement Learning\", NeurIPS, 2020.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the proposed method is well explained, and this paper is easy to follow.\n- The proposed DEtached value learning with Action Sequence (DEAS) framework that decouples critic training from the actor, encouraging the value function to prioritize high-return actions observed in the offline dataset.\n- The proposed approach outperforms baselines in various benchmarks and real-world tasks.", "weaknesses": "- No quantitative results are provided to demonstrate the prevention of value overestimation.\n- The VLA experiments are conducted with only a single baseline. It would be more convincing to include comparisons with other VLA methods, such as OpenVLA [1].\n- No sensitivity analysis is conducted on the hyperparameter $H$.\n\nReference:\n\n[1] Kim et al. \"OpenVLA: An Open-Source Vision-Language-Action Model\", arXiv preprint arXiv:2406.09246, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)