DEAS: DEtached value learning with Action Sequence for Scalable Offline RL

## Abstract

Abstract Offline reinforcement learning (RL) presents an attractive paradigm for training intelligent agents without expensive online interactions. However, current approaches still struggle with complex, long-horizon sequential decision making. In this work, we introduce DEtached value learning with Action Sequence (DEAS), a simple yet effective offline RL framework that leverages action sequences for value learning. These temporally extended actions provide richer information than single-step actions and can be interpreted through the options framework via semi-Markov decision process Q-learning, enabling reduction of the effective planning horizon by considering longer sequences at once. However, directly adopting such sequences in actor-critic algorithms introduces excessive value overestimation, which we address through detached value learning that steers value estimates toward in-distribution actions that achieve high return in the offline dataset. We demonstrate that DEAS consistently outperforms baselines on complex, long-horizon tasks from OGBench and can be applied to enhance the performance of large-scale Vision-Language-Action models that predict action sequences, significantly boosting performance in both RoboCasa Kitchen simulation tasks and real-world manipulation tasks.

## 1 Introduction

Offline reinforcement learning (RL) (Lange et al., 2012; Levine et al., 2020) enables learning from static datasets without incurring online data collection risks, while circumventing the need for expensive expert demonstrations. However, existing methods primarily focus on short-horizon tasks with dense rewards (Yu et al., 2020; Fu et al., 2020; Gulcehre et al., 2020; Mandlekar et al., 2021) and struggle to scale to complex long-horizon scenarios. Recent attempts using large-scale architectures (Kumar et al., 2023a; b; Chebotar et al., 2023; Springenberg et al., 2024) show promise, but their effectiveness on complex tasks remains unexplored.

To address the need for long-horizon evaluation, recent work (Park et al., 2025a; b) has proposed challenging benchmarks for complex offline RL and demonstrated that reducing the effective planning horizon (i.e., shortening the time span over which the agent must plan) in both value and policy learning via $n$-step TD updates with high $n$ values and hierarchical policies is essential. However, these approaches rely on goal-conditioned RL with explicit expert-provided goals, which are often unavailable in practice. For instance, high $n$ values in $n$-step TD updates introduce increased bias and bootstrap error in standard RL without explicit goal information (Tsitsiklis & Van Roy, 1996; Kearns & Singh, 2000; Sutton & Barto, 2018).

These limitations underscore the need for alternative approaches to horizon reduction (reducing the planning horizon)
that work without explicit goal conditioning. One promising direction is leveraging action sequences, which have shown
success in behavior cloning (Pomerleau, 1988) for capturing noisy, temporally-relevant distributions in
expert demonstrations (Chi et al., 2023; Zhao et al., 2023). However, existing attempts to use action sequences for RL remain insufficient for achieving robust horizon reduction. Q-chunking (Li et al., 2025b) has explored the use of action sequences for RL, demonstrating their potential for temporally consistent exploration. However, introducing action sequences to standard actor-critic frameworks causes severe value overestimation (Seo & Abbeel, 2025) due to actors maximizing over potentially erroneous critic estimates with widely spanned action spaces. This problem is exacerbated in offline RL where distribution shift creates extrapolation errors (Kumar et al., 2019; Fujimoto et al., 2019; Kumar et al., 2020). While CQN-AS (Seo & Abbeel, 2025) proposes a value-only approach to avoid this issue, it introduces discretization errors that limit performance in complex tasks and cannot leverage expressive policy classes (Wang et al., 2023; Hansen-Estruch et al., 2023; Park et al., 2025c). For this reason, our research aims to develop methods that can leverage action sequences for horizon reduction while avoiding value overestimation and maintaining compatibility with expressive policy architectures.

Figure: Figure 1: Overview. DEAS is an offline RL framework that learns from action sequences instead of single actions. Unlike previous methods that couple actor-critic training, our key insight is to train the critic separately from the policy (detached value learning) using action sequences, which enables stable learning while avoiding value overestimation. We further enhance stability by combining distributional RL objectives and using dual discount factors, which leads to additional improvement.
Refer to caption: x1.png

##### Our approach

We present DEtached value learning with Action Sequence (DEAS), an offline RL framework that leverages action sequences for scalable value learning in complex tasks. Our method treats consecutive action timesteps as inputs to the value function, implementing the simplest form of the options framework (Sutton et al., 1999; Stolle & Precup, 2002). This design provides principled horizon reduction analogous to $n$-step TD updates with temporally extended actions, while action sequences offer richer information than single-step actions without requiring explicit goal conditioning. To address the value overestimation challenges inherent in learning value functions with action sequences in offline RL settings, we employ detached value learning (Kostrikov et al., 2022) that decouples critic training from the actor, biasing value estimates toward high-return actions present in the offline dataset. This method is appealing as it can be applied to any expressive policy architectures including large-scale Vision-Language-Action models (VLAs) without the hazard of value overestimation. Additionally, we propose to incorporate distributional RL (Farebrother et al., 2024) in value learning to mitigate instability from accumulated bias in multi-step returns.

We validate DEAS through comprehensive experiments on challenging long-horizon tasks from OGBench (Park et al., 2025a), where standard offline RL methods struggle to achieve meaningful success rates. Our method consistently outperforms all baselines, demonstrating its effectiveness on complex tasks. Additionally, we show that DEAS can be used to improve the performance of VLAs (Bjorck et al., 2025) in hard tasks from RoboCasa Kitchen (Nasiriany et al., 2024) and real-world manipulation tasks, which significantly improves performance compared to policies trained solely on expert demonstrations. These results demonstrate DEAS’s practical applicability and potential for scaling offline RL to real-world scenarios.

##### Contributions

We highlight the key contributions of our paper below:

- •
We present DEAS: DEtached value learning with Action Sequence, a simple yet effective offline RL method that leverages action sequences for training critics and employs detached value learning with classification loss for stable training.
- •
We demonstrate that DEAS significantly outperforms baselines on complex, long-horizon tasks across 30 diverse scenarios in OGBench (Park et al., 2025a).
- •
We demonstrate that DEAS can enhance the performance of large-scale VLAs, achieving superior results on complex tasks from RoboCasa Kitchen (Nasiriany et al., 2024) and real-world manipulation tasks compared to policies trained solely on expert demonstrations.

## 2 Related Work

##### Offline reinforcement learning

Offline RL focuses on learning policies from fixed datasets without further environment interaction (Levine et al., 2020). The primary challenge lies in the distributional shift between the behavior policy and the learned policy, which can result in value overestimation and suboptimal performance. Previous work has proposed various approaches including weighted regression (Peng et al., 2019; Nair et al., 2020; Wang et al., 2020), conservative regularization (Kumar et al., 2020), behavioral regularization (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Tarasov et al., 2023; Park et al., 2025c), and in-sample distribution maximization (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023). Our method builds upon in-sample distribution maximization approaches, particularly IQL (Kostrikov et al., 2022), extending them to handle action sequences while maintaining stability by removing the critic update based on the actor’s output. Furthermore, our method has the advantage of being adaptable to any policy extraction method for the final policy, making it more flexible and practical.

##### BC/RL with action sequence

Adopting action sequence has been actively investigated in both imitation learning and RL. Behavior cloning advances show that predicting action sequences captures temporal dependencies from expert demonstrations that single-step actions miss (Chi et al., 2023; Zhao et al., 2023; Black et al., 2025; Bjorck et al., 2025; Intelligence et al., 2025). Several works have introduced action sequences into RL (Li et al., 2024; Tian et al., 2025), with Q-Chunking (Li et al., 2025b) demonstrating incorporation into actor-critic frameworks in offline-to-online RL without policy class constraints. However, this approach faces fundamental challenges: expanded action spaces increase value overestimation risk, particularly in offline settings with limited data coverage (Kumar et al., 2019), yet this issue remains unaddressed. CQN-AS (Seo & Abbeel, 2025) circumvents this by removing the actor entirely, but introduces accumulating discretization errors that severely limit performance in complex tasks and prevent use of expressive policy classes (Wang et al., 2023; Park et al., 2025c). Our approach uniquely combines both paradigms: we leverage horizon reduction from action sequences while addressing value overestimation through detached value learning, enabling stable training with any policy architecture.

## 3 Preliminaries

##### Problem formulation

We consider a Markov Decision Process (MDP) (Sutton & Barto, 2018) $\mathcal{M}=(\mathcal{S},\mathcal{A},p,R,\rho_{0},\gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $R(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function, $p(s^{\prime}|s,a):\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the transition function, $\rho_{0}$ is the initial state distribution, and $\gamma$ is the discount factor. In this paper, we focus on offline reinforcement learning, where we have access only to a static dataset $\mathcal{D}={\{\tau^{i}\}}_{i=0}^{N}$ containing $N$ trajectories of fixed length $H$, where each trajectory $\tau^{i}=(s_{0},a_{0},r_{0},\ldots,s_{H},a_{H},r_{H})$ represents a sequence of states, actions, and rewards. The dataset is collected using a data collection policy $\pi_{\mathcal{D}}:\mathcal{S}\rightarrow\Delta(\mathcal{A})$, which may be unknown or suboptimal. Unlike online RL, we cannot interact with the environment during training. The objective is to learn a policy $\pi:\mathcal{S}\rightarrow\Delta(\mathcal{A})$ that maximizes the expected sum of discounted rewards $\mathbb{E}_{\rho_{0},\pi,p}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right]$ using only this fixed dataset.

##### Options framework

To formalize the idea for flexible temporal abstractions in RL and MDP, a Markovian option $\omega\in\Omega$ is defined as a triplet $(\mathcal{I}_{\omega},\pi_{\omega},\beta_{\omega})$. $\mathcal{I}_{\omega}\subseteq\mathcal{S}$ is the initiation set, $\pi_{\omega}$ is an intra-option policy, and $\beta_{\omega}:\mathcal{S}\rightarrow[0,1]$ is the termination function.
For any MDP $\mathcal{M}$ and any Markovian option $\omega_{\mathcal{M}}$ defined on $\mathcal{M}$, a decision process that follows only the option can be configured as an Semi-Markovian Decision Process (SMDP), which guarantees the existence of a set of optimal policies, denoted as $\Pi^{*}_{\omega}$. For more detailed explanations and proofs, please refer to Sutton et al. (1999).

##### Implicit Q Learning (IQL) (Kostrikov et al., 2022 )

Instead of regularizing the critic with the actor output, IQL approximates the optimal critic to be maximized only in the region of action distributions present in the offline dataset with an in-sample expectile regression. Given a parameterized critic $Q(s_{t},a_{t};\theta)$, target critic $Q(s_{t},a_{t};\bar{\theta})$, and value network $V(s_{t};\psi)$, the objective for value learning is defined as:

$$ $\displaystyle\mathcal{L}_{V}(\psi)=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[L^{\tau}_{2}(\bar{Q}(s_{t},a_{t};\bar{\theta})-V(s_{t};\psi))\right]$ $\displaystyle\mathcal{L}_{Q}(\theta)=\mathbb{E}_{(s_{t},a_{t})\sim\mathcal{D}}\left[(R(s,a)+\gamma V(s_{t+1};\psi)-Q(s_{t},a_{t};\theta))^{2}\right]$ $$

where $L^{\tau}_{2}(u)=|\tau-\mathbbm{1}(u<0)|u^{2}$ is the expectile loss with expectile parameter $\tau\in[0,1]$. By using $\tau>0.5$, Equation [3](https://arxiv.org/html/2510.07730v1#S3.Ex1) penalizes the overestimated value in out-of-distribution actions, letting $V$ and $Q$ to be only approximated in the region of in-distribution actions.

## 4 Method

We propose DEtached value learning with Action Sequence (DEAS), an offline RL method that models action sequences for scalable learning. Our approach consists of two key components: (1) a critic function $Q(s_{t},o_{t};\theta)$ that estimates expected returns for the option (consisting of $H$-step action sequence) $o_{t}:=a_{t:t+H-1}$ from state $s_{t}$ under the data collection policy $\pi_{\mathcal{D}}$, and (2) a flexible policy update mechanism applicable to any policy $\pi(a_{t:t+H-1};s_{t},\phi)$ that outputs $H$-step action sequences. Section [4.1](https://arxiv.org/html/2510.07730v1#S4.SS1) describes how we integrate action sequences into SMDP Q-learning for horizon reduction, while Section [4.2](https://arxiv.org/html/2510.07730v1#S4.SS2) introduces how DEAS enables stable training through detached value learning, distributional RL, and dual discount factors. We provide pseudocode in Algorithm [1](https://arxiv.org/html/2510.07730v1#alg1) and implementation details in Appendix [A](https://arxiv.org/html/2510.07730v1#A1).

### 4.1 Options framework for action sequence RL

Complex tasks require coordinated action sequences where each action’s effectiveness depends on its context within the sequence. For instance, in OGBench $\mathtt{puzzle}$ or $\mathtt{cube}$ tasks, success depends on planning through multiple intermediate steps and maintaining consistent actions over extended periods. These temporal dependencies and hidden sub-tasks are not captured by current state representations, making it challenging for agents to learn effective policies. The challenge becomes even greater in goal-free settings, where agents must discover these sequential patterns from offline data without explicit goal instructions.

To address these challenges, we propose modeling consecutive action sequences as single decision units within the options framework. We treat each $H$-step action sequence $o_{t}:=a_{t:t+H-1}=\{a_{t},a_{t+1},\ldots,a_{t+H-1}\}$ as an option, which naturally induces a SMDP (Bradtke & Duff, 1994; Feinberg, 1994; Sutton et al., 1999; Baykal-Gürsoy & Gürsoy, 2010) that guarantees the existence of an optimal policy. Specifically, the option $\omega^{*}$ is defined as:

$$ $\displaystyle\omega^{*}=(\mathcal{I}_{\omega^{*}},\pi_{\omega^{*}},\beta_{\omega^{*}})=(\mathcal{S},\pi(o_{t}\mid s_{t}),\beta^{*}(s_{t},k))$ $\displaystyle\beta^{*}(s_{t},k)=\begin{cases}1&\text{if }k=H\\ 0&\text{otherwise}\end{cases}$ $$

where $k$ denotes the number of steps executed within the current option. This leads to a Q-learning update rule that extends standard Q-learning (Bradtke & Duff, 1994):

$$ $\displaystyle Q(s_{t},o_{t};\theta)\leftarrow\sum_{k=0}^{H-1}\gamma_{1}^{k}R(s_{t},a_{t+k})+\gamma_{2}^{H}\max_{o^{\prime}\in\mathcal{O}}Q(s_{t+H},o^{\prime};\theta)$ $$

where $\gamma_{1}$ and $\gamma_{2}$ are discount factors for intra-option and inter-option transitions, respectively. This formulation aggregates rewards over $H$ steps and propagates value estimates across temporally extended transitions, achieving horizon reduction similar to $n$-step TD learning (Park et al., 2025b).

Figure: Algorithm 1 DEAS

### 4.2 DEAS: DEtached value learning with Action Sequence

##### Detached value learning for handling action sequence

Action sequences introduce challenges for value function approximation, as the expanded action space makes it harder for the critic to estimate Q-values accurately. Meanwhile, the actor can exploit regions where the critic makes prediction errors, leading to value overestimation and unstable learning (Seo & Abbeel, 2025). To address this, we adopt detached value learning (Kostrikov et al., 2022; Xu et al., 2023; Garg et al., 2023) that decouples actor and critic training, introducing a critic $Q(s_{t},o_{t};\theta)$ and a value $V(s_{t};\psi)$ networks with the following losses following IQL (Kostrikov et al., 2022):

$$ $\displaystyle\mathcal{L}_{V}(\psi)=\mathbb{E}_{(s_{t},o_{t})\sim\mathcal{D}}\left[L^{\tau}_{2}(\bar{Q}(s_{t},o_{t};\bar{\theta})-V(s_{t};\psi))\right]$ $\displaystyle\mathcal{L}_{Q}(\theta)=\mathbb{E}_{(s_{t},o_{t})\sim\mathcal{D}}\left[(\hat{R}_{t:t+H-1}+\gamma_{2}^{H}V(s_{t+H};\psi)-Q(s_{t},o_{t};\theta))^{2}\right],$ $$

where $\hat{R}_{t:t+H-1}=\sum_{k=0}^{H-1}{\gamma_{1}}^{k}R(s_{t},a_{t+k})$ is the discounted return for the action sequence. This approach steers the critic toward high-return actions in the offline dataset without the potential of exploiting critic approximation errors, preventing value overestimation and enabling stable value learning even with longer action sequences.

##### Distributional RL for enhanced stability

Even with detached value learning, the cumulative reward term $\hat{R}_{t:t+H-1}$ could introduce significant variance when $H$ is large. To enhance stability, we extend our framework with distributional RL (Bellemare et al., 2017; Farebrother et al., 2024), modeling both critic and value networks as categorical distributions over fixed support $[\mathbf{v}_{\min},\mathbf{v}_{\max}]$ discretized into $m$ bins:

$$ $Q(s,o;\theta)=\mathbb{E}\left[Z(s,o;\theta)\right]\quad Z(s,o;\theta)=\sum_{i=1}^{m}\hat{p}_{i}(s,o;\theta)\cdot\delta_{z_{i}}\quad\hat{p}_{i}(s,o;\theta)=\frac{e^{l_{i}(s,o;\theta)}}{\sum_{i=1}^{m}e^{l_{i}(s,o;\theta)}},$ (1) $$

where $V(s;\psi)$ is computed similarly, by conditioning only on the state $s$.
To address scale differences between regression and classification objectives, we maintain IQL’s weighting scheme but replace regression with classification-based learning:

$$ $\begin{split}\mathcal{L}_{V}(\psi)&=\mathbb{E}_{(s_{t},o_{t})\sim\mathcal{D}}\left[\alpha_{t}\cdot\sum^{m}_{i=1}\hat{p}_{i}(s_{t};\psi)\log{\hat{p}_{i}(s_{t},o_{t};\bar{\theta})}\right]\\ \alpha_{t}&=\begin{cases}\tau&\text{if }\bar{Q}(s_{t},o_{t};\bar{\theta})\geq V(s_{t};\psi)\\ 1-\tau&\text{otherwise},\end{cases}\end{split}$ (2) $$

$$ $\displaystyle\mathcal{L}_{Q}(\theta)=\mathbb{E}_{(s_{t},a_{t:t+H-1},s_{t+H})\sim\mathcal{D}}\left[\sum^{m}_{i=1}p_{i}(s_{t};\psi)\log{\hat{p}_{i}(s_{t},a_{t:t+H-1};\hat{\theta})}\right].$ (3) $$

For target probabilities $p_{i}$, we adopt the truncated normal distribution with mean as Bellman target $(\mathcal{\hat{T}V})(s,a_{t:t+H-1})=\sum_{k=0}^{H-1}\gamma_{1}^{k}r_{t+k}+\gamma_{2}^{H}V(s_{t+H};\psi)$ and standard deviation $\sigma=0.75\cdot(v_{max}-v_{min}/m)$, inspired by Farebrother et al. (2024).

##### Dual discount factors

To further enhance stability and expressiveness in value estimation, we employ two separate discount factors: $\gamma_{1}$ for intra-option (within action sequence) rewards and $\gamma_{2}$ for inter-option (across action sequences) rewards. This dual-discounting scheme enables the value function to appropriately weigh immediate and future returns, mitigating issues such as value explosion or collapse that can arise from improper scaling of returns. In our experiments, we observe that decreasing the intra-option discount factor $\gamma_{1}$ and increasing the inter-option discount factor $\gamma_{2}$ leads to more stable training, and is critical for stable training, especially when the action sequence becomes longer (see Section [5.3](https://arxiv.org/html/2510.07730v1#S5.SS3) for the supporting results).

##### Compatible policy methods

For obtaining final policy $\pi(s;\phi)$, our framework is compatible with a variety of policy extraction strategies (Park et al., 2024), including weighted behavior cloning (Peng et al., 2019), deterministic policy gradient (DPG) (Fujimoto & Gu, 2021), best-of-N sampling (Chen et al., 2023), and flow-matching approaches (Park et al., 2025c). Since value function training does not require querying the policy, it can be performed independently, and the policy can be updated separately. To demonstrate this, we illustrate the effectiveness of our method using various policy extraction methods in our experiments.

Figure: Figure 2: Simulation task examples. We study DEAS on 30 different tasks from OGBench (Park et al., 2025a) and 4 challenging manipulation tasks from RoboCasa Kitchen (Nasiriany et al., 2024).
Refer to caption: figures/png/concatenated.png

## 5 Experiments

We first validate the effectiveness of DEAS through extensive experiments on various complex tasks in OGBench (Park et al., 2025a). Additionally, to prove that DEAS can be naturally plugged into large-scale VLAs for practical applications, we evaluate DEAS by fine-tuning GR00T N1.5 (NVIDIA, 2025) using offline RL methods on 4 hard tasks from RoboCasa Kitchen (Nasiriany et al., 2024) and also conduct real-world experiments with Franka Emika Research 3 Robot Arm. See Figure [2](https://arxiv.org/html/2510.07730v1#S4.F2) and Figure [5.2.2](https://arxiv.org/html/2510.07730v1#S5.SS2.SSS2.Px2) for task examples used in our experiments.

### 5.1 OGBench Experiments

##### Setup

We evaluate on 6 manipulation environments from OGBench (Park et al., 2025a), each with 5 subtasks. We use datasets ranging from 1M to 100M transitions based on task difficulty. While OGBench is originally designed for offline goal-conditioned RL, we use its single-task variants ($\tt{`-singletask^{\prime}}$) for reward-maximizing offline RL. For fair comparison, all methods use identical MLP architectures for actor networks and adopt the same policy extraction approach as FQL (Park et al., 2025c), except for CQN-AS, which uses value function networks as the actor itself through discretization. Action sequence length $H$ is set to 8 for $\tt{scene}$ and $\tt{puzzle}$ tasks, and 4 for $\tt{cube}$ tasks, with $n=H$ is used for $n$-step FQL. More details about the experimental setup can be found in Appendix [A.1](https://arxiv.org/html/2510.07730v1#A1.SS1).

##### Baselines

We compare against FQL (Park et al., 2025c), a state-of-the-art offline RL method using one-step distillation between flow matching models with different denoising steps, and $n$-step FQL (Sutton & Barto, 2018), which extends FQL with $n$-step TD updates for horizon reduction (Park et al., 2025b). While increasing $n$ increases bias in standard offline RL, DEAS explicitly models action sequences while maintaining horizon reduction benefits. We also consider Q-Chunking (QC) (Li et al., 2025b), which uses action chunking for actor-critic training while keeping the interaction between actor and critic, while DEAS uses detached value learning. For fair comparison with ours, we extensively tune QC-FQL hyperparameters to achieve higher performance than the original paper. Lastly, CQN-AS (Seo & Abbeel, 2025), a value-based RL method with action sequence utilizing multi-level critics with iterative discretization, is included as a baseline.

**Table 1: Offline RL results in 6 task categories from OGBench (Park et al., 2025a). We report the success rate (%) and 95% stratified bootstrap confidence interval over 4 runs. Bold indicates the values at or above 95% of the best performance. Please refer to Table [8](https://arxiv.org/html/2510.07730v1#A5.T8) for the full results.**
| Task Category | #Data | FQL | N-step FQL | QC-FQL | CQN-AS | \columncolorgreen!10DEAS |
| --- | --- | --- | --- | --- | --- | --- |
| scene-play-singletask ($\mathbf{5}$ tasks) | 1M | $50$ $\pm 3$ | $36$ $\pm 2$ | $\mathbf{73}$ $\pm 2$ | $1$ $\pm 1$ | \columncolorgreen!10$\mathbf{76}$ $\pm 2$ |
| cube-double-play-singletask ($\mathbf{5}$ tasks) | $14$ $\pm 2$ | $4$ $\pm 2$ | $41$ $\pm 3$ | $2$ $\pm 1$ | \columncolorgreen!10$\mathbf{48}$ $\pm 2$ |  |
| puzzle-3x3-play-singletask ($\mathbf{5}$ tasks) | $44$ $\pm 3$ | $36$ $\pm 3$ | $62$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{91}$ $\pm 3$ |  |
| cube-triple-play-singletask ($\mathbf{5}$ tasks) | 10M | $10$ $\pm 3$ | $23$ $\pm 2$ | $\mathbf{83}$ $\pm 4$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{82}$ $\pm 5$ |
| puzzle-4x4-play-singletask ($\mathbf{5}$ tasks) | $32$ $\pm 4$ | $19$ $\pm 5$ | $69$ $\pm 8$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{82}$ $\pm 6$ |  |
| cube-quadruple-play-singletask ($\mathbf{5}$ tasks) | 100M | $17$ $\pm 8$ | $36$ $\pm 10$ | $45$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{64}$ $\pm 8$ |

Figure: Figure 3: Agent performance across varying dataset sizes on three representative OGBench (Park et al., 2025a) tasks, evaluated by success rate (%). Solid lines indicate the mean, while shaded areas denote the stratified bootstrap confidence intervals over 4 independent runs.
Refer to caption: x2.png

##### Quantitative results

As shown in Table [1](https://arxiv.org/html/2510.07730v1#S5.T1), DEAS consistently achieves the best performance across all 6 task categories with various dataset sizes. Comparing FQL and N-step FQL, we observe that simply increasing the $n$-step mostly leads to performance degradation due to bias in standard offline RL, while our detached value learning approach enables stable training with action sequences. Notably, DEAS matches or outperforms QC-FQL across all tasks, demonstrating the effectiveness of our stable value learning in addressing offline RL instability. The method shows particularly strong performance on tasks requiring long-horizon reasoning like $\tt{puzzle}$ and the most challenging tasks (i.e., $\tt{cube-quadruple}$), where the benefits of using action sequences are most pronounced. CQN-AS shows significantly lower performance, likely due to its direct application of strong BC regularization on the value function in the presence of predominantly suboptimal data, along with cumulative errors from iterative discretizations that reduce action precision.

##### Scaling analysis

To further validate the scalability of DEAS, we conduct a scaling analysis on three representative OGBench tasks with varying dataset sizes. As shown in Figure [3](https://arxiv.org/html/2510.07730v1#S5.F3), DEAS consistently outperforms all baselines across all dataset sizes, achieving the highest success rates in every environment. The method demonstrates robust scaling across different dataset sizes, maintaining consistent performance gains even with larger datasets. This superior performance validates our approach of explicitly modeling action sequences while effectively leveraging suboptimal data through our detached value learning and stable multi-step training.

### 5.2 VLA Experiments

To validate the practical applicability of DEAS, we demonstrate its effectiveness with large-scale VLAs (Black et al., 2025; Bjorck et al., 2025; NVIDIA, 2025). These models, trained on internet-scale diverse datasets with billion-scale parameters, predict much longer action sequences and are widely used in robotics applications. However, deploying these models typically requires fine-tuning on task-specific data, which often necessitates collecting expensive expert demonstrations. We design our experiments to validate whether DEAS can improve VLA performance by effectively utilizing suboptimal demonstrations alongside limited expert data, potentially reducing the required amount of costly expert demonstrations. See Appendix [A.2](https://arxiv.org/html/2510.07730v1#A1.SS2) for more details.

#### 5.2.1 RoboCasa Kitchen Experiments

##### Setup

We employ GR00T N1.5 (NVIDIA, 2025) as the backbone VLA. First, we fine-tune the VLA using 100 expert demonstrations from all 24 RoboCasa Kitchen tasks to verify that we achieve performance similar to the original GR00T N1 (Bjorck et al., 2025). From these tasks, we select 4 tasks with the lowest success rates in their respective categories for our offline IL/RL experiments. We then collect 300 rollouts for each task from the resulting policy and apply various offline IL/RL methods. For RL methods, we fine-tune the base policy using behavior cloning on both expert demonstrations and the rollout dataset and use the model as an actor for training critic functions when necessary. For policy extraction, we adopt best-of-N sampling (Chen et al., 2023; Nakamoto et al., 2024), where we sample multiple outputs from the policy and select the action sequence with the highest Q-value. We set $H=16$ for all methods, matching GR00T N1.5’s action chunk size.

**Table 2: RoboCasa Kitchen evaluation results. We fine-tune GR00T N1.5 (NVIDIA, 2025) on 24 RoboCasa Kitchen tasks using 100 expert demonstrations per task. For 4 selected tasks, we collect 300 rollouts and apply offline IL/RL algorithms. Success rates (%) on 50 episodes, aggregated with 3 seeds. $\tt{PnPC2M}$ denotes ‘PnPCounterToMicrowave’ and $\tt{PnPM2C}$ denotes ‘PnPMicrowaveToCounter’. Bold and underline indicate best and runner-up results, respectively.**
| $\dagger$ Reproduced performance |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Models | CoffeeSetupMug | PnPC2M | PnPM2C | TurnOffStove | Avg. |
| *Base models* |  |  |  |  |  |
| GR00T N1^∗ | 0 $2.0$ | 0 $0.0$ | 0 $0.0$ | 0 $15.7$ | 0 $4.4$ |
| GR00T N1.5${\dagger}$ | 0 $4.7$ | 0 $21.3$ | 0 $7.3$ | 0 $14.7$ | 0 $12.0$ |
| \arrayrulecolorblack!40*Imitation learning* |  |  |  |  |  |
| + Filtered BC | 0 $14.7$ | 0 $25.3$ | 0 $\underline{14.7}$ | 0 19.3 | 0 $18.5$ |
| \arrayrulecolorblack!40*Offline RL* |  |  |  |  |  |
| + IQL | 0 $\underline{23.3}$ | 0 $\underline{30.0}$ | 0 $\underline{14.7}$ | 0 $12.7$ | 0 $\underline{20.2}$ |
| + QC | 0 $16.0$ | 0 $28.7$ | 0 $\underline{14.7}$ | 0 $10.7$ | 0 $17.5$ |
| \rowcolorgreen!10 + DEAS (Ours) | 0 $\mathbf{28.7}$ | 0 $\mathbf{36.0}$ | 0 $\mathbf{18.0}$ | 0 $\underline{18.0}$ | 0 $\mathbf{25.2}$ |

##### Baselines

We compare against several baselines across both imitation learning and reinforcement learning paradigms. For imitation learning, we consider Filtered BC, which fine-tunes the base policy using both expert demonstrations and successful episodes from the rollout data (Oh et al., 2018). For reinforcement learning, we evaluate IQL, a value-based method that operates on single actions without requiring policy outputs. For determining action sequence in IQL, we use the very first action in the sequence for value estimation. Lastly, we consider QC, which employs action chunking for critic training but relies on predicted action sequences from VLA for the critic update.

##### Results

As shown in Table [5.2.1](https://arxiv.org/html/2510.07730v1#S5.SS2.SSS1.Px1), DEAS achieves the highest success rates in 3 out of 4 tasks, with the remaining task also showing improved performance compared to the base model. While filtered BC improves performance with simple approaches, but our approach exhibits additional performance gains by effectively utilizing suboptimal data. While single-step IQL also demonstrates effectiveness, it shows smaller performance gains across all tasks compared to our approach, due to its lack of understanding of action sequences. QC shows limited improvement compared to BC-based approaches, highlighting the advantage of our detached value learning with action sequences.

#### 5.2.2 Real-world Experiments

##### Setup

We further investigate the effectiveness of DEAS in real-world tasks using Franka Emika Research 3 Robot Arm. Inspired by RoboCasa Kitchen, we design pick-and-place tasks from the countertop to the bottom cabinet, with three different objects: $\tt{peach}$, $\tt{milka}$, and $\tt{hichew}$ (see Figure [5.2.2](https://arxiv.org/html/2510.07730v1#S5.SS2.SSS2.Px2)). For each task, we collect 5 demonstrations, fine-tune GR00T N1.5, collect 25 rollouts, and apply various offline IL/RL methods. We evaluate using 20 rollouts per task from 5 different initial points and use the same baselines as in the RoboCasa Kitchen experiments. Success rates are calculated based on partial success scoring (0-1 scale) that considers subtask completion, with detailed evaluation methodology provided in [Section A.2.2](https://arxiv.org/html/2510.07730v1#A1.SS2.SSS2).

##### Results

In Table [3](https://arxiv.org/html/2510.07730v1#S5.T3), DEAS achieves the highest success rates across all three pick-and-place tasks compared to baselines. The method shows consistent improvements, particularly on challenging objects like $\tt{milka}$ (a deformable object) where other approaches struggle. Notably, QC shows degraded performance compared to the base model, likely due to its instability when using action sequences with relatively small datasets, while our method shows stable improvement even with limited data. These results demonstrate that our detached value learning approach can be effectively applied to real-world robotic tasks and remains stable regardless of the dataset size.

**Figure 4: Real-world tasks. We conduct pick-and-place tasks from the countertop to the bottom cabinet with $\tt{peach}$, $\tt{milka}$, and $\tt{hichew}$.**
| Models | peach | milka | hichew | Avg. |
| --- | --- | --- | --- | --- |
| *Base model* |  |  |  |  |
| GR00T N1.5 | 62.0 | 45.0 | 85.0 | 64.0 |
| \arrayrulecolorblack!40*Imitation learning* |  |  |  |  |
| + Filtered BC | 76.3 | 25.0 | 92.5 | 64.6 |
| \arrayrulecolorblack!40*Offline RL* |  |  |  |  |
| + IQL | 82.5 | 37.5 | 78.8 | 66.3 |
| + QC | 58.8 | 15.0 | 45.0 | 39.6 |
| \rowcolorgreen!10+ DEAS (Ours) | 86.3 | 53.8 | 95.0 | 78.4 |

### 5.3 Ablation Studies and Analyses

We investigate the effect of hyperparameters and various components of DEAS by running experiments on OGBench puzzle-4x4 task.

##### Effect of action sequence length

Table LABEL:table:effect_of_as investigates the impact of action sequence length on performance. When using single-step or two-step action ($H=1,2$), DEAS fails to achieve meaningful performance, confirming the necessity of action sequences for long-horizon tasks. Performance improves with longer sequences, but when the sequence length becomes longer than 8, it requires proportionally larger actor networks to handle the increased action dimensions, suggesting a trade-off between sequence length and computational efficiency.

##### Effect of network size

Table LABEL:table:effect_of_critic_size analyzes the sensitivity to network sizes. For the critic network, we observe that increasing capacity initially improves performance by better approximating the value function. For the value function, we find that the network needs sufficient capacity to capture the complexity of action sequence values, but excessive capacity without proper regularization causes instability in value estimation, leading to performance degradation.

##### Effect of training objective

In Table LABEL:table:objectives, we compare different training objectives for value estimation. We found that using only distributional RL (HLG) (Farebrother et al., 2024) or only standard regression (IQL) shows limited performance. However, combining detached value learning with distributional estimation significantly improves results, suggesting both components are crucial for stable training with action sequences.

##### Effect of dual discount factors

Lastly, we examine the effect of dual discount factors on learning dynamics in Table LABEL:table:discount_factor. Proper tuning of $\gamma_{1}$ (the discount factor for action sequences) is essential for performance, as it controls the temporal horizon for value estimation within sequences. In this paper, we use $\gamma_{1}=0.9$ for all experiments.

## 6 Conclusion

We present DEAS, a simple yet effective offline RL method that leverages action sequences for scalable learning in complex tasks. By modeling temporally extended actions through the options framework, DEAS achieves principled horizon reduction via SMDP Q-learning while addressing value overestimation through detached value learning. Our experiments demonstrate consistent improvements over baselines on challenging OGBench tasks and successful application to large-scale VLAs, showing the practical potential for scaling offline RL to real-world scenarios.

### Reproducibility statement

We provide full hyperparameter and implementation details in [Section 5](https://arxiv.org/html/2510.07730v1#S5) and [Appendix A](https://arxiv.org/html/2510.07730v1#A1).
In addition, to further facilitate the reproduction, we release the open-sourced implementation through the [project website](https://changyeon.site/deas).

#### Acknowledgments

CY thanks Jaehyun Nam, Juyong Lee, and anonymous reviewers for providing helpful feedback and suggestions for improving our paper. CY also thanks Angeline S. Kim for the assistance in improving the expression and the visualization of the paper.

## References

- Ajay et al. (2021)
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum.
{OPAL}: Offline primitive discovery for accelerating offline reinforcement learning.
In *International Conference on Learning Representations*, 2021.
- Bacon et al. (2017)
Pierre-Luc Bacon, Jean Harb, and Doina Precup.
The option-critic architecture.
In *AAAI Conference on Artificial Intelligence*, 2017.
- Baykal-Gürsoy & Gürsoy (2010)
Melike Baykal-Gürsoy and K Gürsoy.
Semi-markov decision processes.
*Wiley Encyclopedia of Operations Research and Management Sciences*, 2010.
- Bellemare et al. (2017)
Marc G Bellemare, Will Dabney, and Rémi Munos.
A distributional perspective on reinforcement learning.
In *International Conference on Machine Learning*, 2017.
- Bjorck et al. (2025)
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al.
Gr00t n1: An open foundation model for generalist humanoid robots.
*arXiv preprint arXiv:2503.14734*, 2025.
- Black et al. (2025)
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al.
$\pi_{0}$: A vision-language-action flow model for general robot control.
In *Robotics: Science and Systems*, 2025.
- Bradtke & Duff (1994)
Steven Bradtke and Michael Duff.
Reinforcement learning methods for continuous-time markov decision problems.
In *Conference on Neural Information Processing Systems*, 1994.
- Chebotar et al. (2023)
Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al.
Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions.
In *Conference on Robot Learning*, 2023.
- Chen et al. (2023)
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu.
Offline reinforcement learning via high-fidelity generative behavior modeling.
In *International Conference on Learning Representations*, 2023.
- Chen et al. (2025a)
Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao.
Conrft: A reinforced fine-tuning method for vla models via consistency policy.
*arXiv preprint arXiv:2502.05450*, 2025a.
- Chen et al. (2025b)
Zengjue Chen, Runliang Niu, He Kong, and Qi Wang.
Tgrpo: Fine-tuning vision-language-action model via trajectory-wise group relative policy optimization.
*arXiv preprint arXiv:2506.08440*, 2025b.
- Chi et al. (2023)
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song.
Diffusion policy: Visuomotor policy learning via action diffusion.
*International Journal of Robotics Research*, 2023.
- Farebrother et al. (2024)
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal.
Stop regressing: Training value functions via classification for scalable deep RL.
In *International Conference on Machine Learning*, 2024.
- Feinberg (1994)
Eugene A Feinberg.
Constrained semi-markov decision processes with average rewards.
*Zeitschrift für Operations Research*, 1994.
- Fu et al. (2020)
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.
D4rl: Datasets for deep data-driven reinforcement learning.
*arXiv preprint arXiv:2004.07219*, 2020.
- Fujimoto & Gu (2021)
Scott Fujimoto and Shixiang Shane Gu.
A minimalist approach to offline reinforcement learning.
In *Conference on Neural Information Processing Systems*, 2021.
- Fujimoto et al. (2019)
Scott Fujimoto, David Meger, and Doina Precup.
Off-policy deep reinforcement learning without exploration.
In *International Conference on Machine Learning*, 2019.
- Garg et al. (2023)
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon.
Extreme q-learning: Maxent RL without entropy.
In *International Conference on Learning Representations*, 2023.
- Gulcehre et al. (2020)
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al.
Rl unplugged: A suite of benchmarks for offline reinforcement learning.
*Conference on Neural Information Processing Systems*, 2020.
- Guo et al. (2025)
Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, and Jianyu Chen.
Improving vision-language-action model with online reinforcement learning.
*arXiv preprint arXiv:2501.16664*, 2025.
- Hansen-Estruch et al. (2023)
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine.
Idql: Implicit q-learning as an actor-critic method with diffusion policies.
*arXiv preprint arXiv:2304.10573*, 2023.
- Hendrycks & Gimpel (2016)
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (gelus).
*arXiv preprint arXiv:1606.08415*, 2016.
- Huang et al. (2025)
Dongchi Huang, Zhirui Fang, Tianle Zhang, Yihang Li, Lin Zhao, and Chunhe Xia.
Co-rft: Efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning.
*arXiv preprint arXiv:2508.02219*, 2025.
- Intelligence et al. (2025)
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky.
$\pi_{0.5}$: a vision-language-action model with open-world generalization.
*arXiv preprint arXiv:2504.16054*, 2025.
- Kearns & Singh (2000)
Michael J Kearns and Satinder Singh.
Bias-variance error bounds for temporal difference updates.
In *Conference on Learning Theory*, 2000.
- Kingma (2015)
Diederik P Kingma.
Adam: A method for stochastic optimization.
In *International Conference on Learning Representations*, 2015.
- Kostrikov et al. (2022)
Ilya Kostrikov, Ashvin Nair, and Sergey Levine.
Offline reinforcement learning with implicit q-learning.
In *International Conference on Learning Representations*, 2022.
- Kulkarni et al. (2016)
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum.
Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.
In *Conference on Neural Information Processing Systems*, 2016.
- Kumar et al. (2019)
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.
Stabilizing off-policy q-learning via bootstrapping error reduction.
In *Conference on Neural Information Processing Systems*, 2019.
- Kumar et al. (2020)
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.
Conservative q-learning for offline reinforcement learning.
In *Conference on Neural Information Processing Systems*, 2020.
- Kumar et al. (2023a)
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine.
Offline q-learning on diverse multi-task data both scales and generalizes.
In *International Conference on Learning Representations*, 2023a.
- Kumar et al. (2023b)
Aviral Kumar, Anikait Singh, Frederik Ebert, Mitsuhiko Nakamoto, Yanlai Yang, Chelsea Finn, and Sergey Levine.
Pre-training for robots: Offline rl enables learning new tasks from a handful of trials.
In *Robotics: Science and Systems*, 2023b.
- Lange et al. (2012)
Sascha Lange, Thomas Gabel, and Martin Riedmiller.
Batch reinforcement learning.
In *Reinforcement learning: State-of-the-art*, 2012.
- Levine et al. (2020)
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.
Offline reinforcement learning: Tutorial, review, and perspectives on open problems.
*arXiv preprint arXiv:2005.01643*, 2020.
- Li et al. (2024)
Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann.
Top-erl: Transformer-based off-policy episodic reinforcement learning.
*arXiv preprint arXiv:2410.09536*, 2024.
- Li et al. (2025a)
Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al.
Simplevla-rl: Scaling vla training via reinforcement learning.
*arXiv preprint arXiv:2509.09674*, 2025a.
- Li et al. (2025b)
Qiyang Li, Zhiyuan Zhou, and Sergey Levine.
Reinforcement learning with action chunking.
*arXiv preprint arXiv:2507.07969*, 2025b.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations*, 2019.
- Mandlekar et al. (2021)
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín.
What matters in learning from offline human demonstrations for robot manipulation.
*arXiv preprint arXiv:2108.03298*, 2021.
- Mandlekar et al. (2023)
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox.
Mimicgen: A data generation system for scalable robot learning using human demonstrations.
In *Conference on Robot Learning*, 2023.
- Nachum et al. (2018)
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine.
Data-efficient hierarchical reinforcement learning.
In *Conference on Neural Information Processing Systems*, 2018.
- Nair et al. (2020)
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.
Awac: Accelerating online reinforcement learning with offline datasets.
*arXiv preprint arXiv:2006.09359*, 2020.
- Nakamoto et al. (2023)
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine.
Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.
In *Conference on Neural Information Processing Systems*, 2023.
- Nakamoto et al. (2024)
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine.
Steering your generalists: Improving robotic foundation models via value guidance.
In *Conference on Robot Learning*, 2024.
- Nasiriany et al. (2024)
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu.
Robocasa: Large-scale simulation of everyday tasks for generalist robots.
In *Robotics: Science and Systems*, 2024.
- Nauman et al. (2024)
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan.
Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control.
In *Conference on Neural Information Processing Systems*, 2024.
- NVIDIA (2025)
NVIDIA.
Gr00t n1.5: An improved open foundation model for generalist humanoid robots.
[https://research.nvidia.com/labs/gear/gr00t-n1_5/](https://research.nvidia.com/labs/gear/gr00t-n1_5/), June 2025.
Accessed: 2025-09-09.
- Oh et al. (2018)
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee.
Self-imitation learning.
In *International Conference on Machine Learning*, 2018.
- Park et al. (2024)
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar.
Is value learning really the main bottleneck in offline rl?
In *Conference on Neural Information Processing Systems*, 2024.
- Park et al. (2025a)
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine.
OGBench: Benchmarking offline goal-conditioned RL.
In *International Conference on Learning Representations*, 2025a.
- Park et al. (2025b)
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine.
Horizon reduction makes rl scalable.
In *Conference on Neural Information Processing Systems*, 2025b.
- Park et al. (2025c)
Seohong Park, Qiyang Li, and Sergey Levine.
Flow q-learning.
In *International Conference on Machine Learning*, 2025c.
- Peng et al. (2019)
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine.
Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.
*arXiv preprint arXiv:1910.00177*, 2019.
- Pomerleau (1988)
Dean A Pomerleau.
Alvinn: An autonomous land vehicle in a neural network.
In *Conference on Neural Information Processing Systems*, 1988.
- Seo & Abbeel (2025)
Younggyo Seo and Pieter Abbeel.
Coarse-to-fine q-network with action sequence for data-efficient robot learning.
In *Conference on Neural Information Processing Systems*, 2025.
- Springenberg et al. (2024)
Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Maria Elisabeth Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin Riedmiller.
Offline actor-critic reinforcement learning scales to large models.
In *International Conference on Machine Learning*, 2024.
- Stolle & Precup (2002)
Martin Stolle and Doina Precup.
Learning options in reinforcement learning.
In *International Symposium on abstraction, reformulation, and approximation*, 2002.
- Sutton & Barto (2018)
Richard S Sutton and Andrew G Barto.
*Reinforcement learning: An introduction*.
MIT press, 2018.
- Sutton et al. (1999)
Richard S Sutton, Doina Precup, and Satinder Singh.
Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.
*Artificial intelligence*, 1999.
- Tan et al. (2025)
Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl.
Ript-vla: Interactive post-training for vision-language-action models.
*arXiv preprint arXiv:2505.17016*, 2025.
- Tarasov et al. (2023)
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov.
Revisiting the minimalist approach to offline reinforcement learning.
In *Conference on Neural Information Processing Systems*, 2023.
- Tian et al. (2025)
Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann.
Chunking the critic: A transformer-based soft actor-critic with n-step returns.
*arXiv preprint arXiv:2503.03660*, 2025.
- Tsitsiklis & Van Roy (1996)
John Tsitsiklis and Benjamin Van Roy.
Analysis of temporal-diffference learning with function approximation.
In *Conference on Neural Information Processing Systems*, 1996.
- Vezhnevets et al. (2017)
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu.
Feudal networks for hierarchical reinforcement learning.
In *International Conference on Machine Learning*, 2017.
- Wang et al. (2023)
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou.
Diffusion policies as an expressive policy class for offline reinforcement learning.
In *International Conference on Learning Representations*, 2023.
- Wang et al. (2020)
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al.
Critic regularized regression.
In *Conference on Neural Information Processing Systems*, 2020.
- Xu et al. (2023)
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan.
Offline RL with no OOD actions: In-sample learning via implicit value regularization.
In *International Conference on Learning Representations*, 2023.
- Yu et al. (2020)
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.
Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.
In *Conference on Robot Learning*, 2020.
- Zhang et al. (2025)
Hongyin Zhang, Zifeng Zhuang, Han Zhao, Pengxiang Ding, Hongchao Lu, and Donglin Wang.
Reinbot: Amplifying robot visual-language manipulation with reinforcement learning.
*arXiv preprint arXiv:2505.07395*, 2025.
- Zhang et al. (2024)
Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao.
Grape: Generalizing robot policy via preference alignment.
*arXiv preprint arXiv:2411.19309*, 2024.
- Zhao et al. (2023)
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.
Learning fine-grained bimanual manipulation with low-cost hardware.
In *Robotics: Science and Systems*, 2023.

## Appendix A Implementation and Training Details

### A.1 OGBench Experiments

##### Tasks

We evaluate our method on 6 UR5 Robot Arm manipulation environments from OGBench (Park et al., 2025a), each with 5 subtasks. All tasks are state-based, and goal-free setup. For each task, the observation space consists of the proprioceptive state of the UR5 Robot Arm, and low-dim state vector informing the target object state and position. The action space consists of the cartesian position of UR5 robot arm, gripper yaw, and gripper open/close. For substituting goal-conditioned environment to standard function, we use the simple semi-sparse reward function, which is defined as the negative number of uncompleted subtasks in the current state, following Park et al. (2025a). For all tasks, the maximum episode length is set to 1000.

##### Implementation details

We implement our method on top of the open-source implementation of FQL (Park et al., 2025c) (^1^11[https://github.com/seohongpark/fql](https://github.com/seohongpark/fql)). Unless otherwise mentioned, we largely follow the training/evaluation setup and network architecture from Park et al. (2025c) and Park et al. (2025b). For training value network, we use the smaller size network compared to critic network for all experiments, which shows the best performance, and we use the doubled size of network for the critic network. For $\tt{cube}$ experiments, we use BRO (Nauman et al., 2024) for additional regularization between relatively small range of returns in value function training. For selecting $\mathbf{v}_{\min}$ and $\mathbf{v}_{\max}$ for distributional RL, we use two procedures: 1) data-centric: compute return distribution from the dataset and select 1% and 99% quantiles with 20% padding, and 2) universal: compute theoretical bounds using reward range $[r_{\min},r_{\max}]$, horizon $L$, option length $H$, and discount factors $\gamma_{1},\gamma_{2}$. For SMDP with $K=L/H$ options, the theoretical bounds are:

$$ $\displaystyle\mathbf{v}_{\min}$ $\displaystyle=r_{\min}\frac{1-\gamma_{2}^{H}}{1-\gamma_{2}}\frac{1-\gamma_{1}^{K}}{1-\gamma_{1}}$ (4) $\displaystyle\mathbf{v}_{\max}$ $\displaystyle=r_{\max}\frac{1-\gamma_{2}^{H}}{1-\gamma_{2}}\frac{1-\gamma_{1}^{K}}{1-\gamma_{1}}$ (5) $$

where $\gamma_{1}$ and $\gamma_{2}$ denote option-level and action-level discount factors, respectively.

##### Training and evaluation

For the training dataset, we use the open-sourced 1M/100M $\tt{play}$ dataset released by Park et al. (2025a) (^2^22[https://github.com/seohongpark/ogbench](https://github.com/seohongpark/ogbench)), where the dataset is collected by open-loop, non-Markovian scripted policies with temporally correlated noise. As 100M dataset consists of 100 separate files with 1M transitions for each, we use the first 10 files sorted by name for 10M dataset. We train our method and baselines for 1M (1M data) / 2.5M (10M/100M data) gradient steps. For selecting BC coefficient $\alpha$ for policy extraction, we first normalize the Q loss as in Fujimoto & Gu (2021) and sweep the value from $\{0.1,0.3,1,3,10\}$ and choose the best one for each task and baseline, except $\mathtt{cube-double}$, where we follow the hyperparameter used in Li et al. (2025b). For evaluation, we report the average success rates across the last three evaluation epochs (800K, 900K, 1M for 1M dataset, 2.3M, 2.4M, 2.5M for 10M/100M dataset) following Park et al. (2025c) and Park et al. (2025b). For checking additional hyperparameters used in our experiments, please refer to [Section A.3](https://arxiv.org/html/2510.07730v1#A1.SS3).

##### Baselines

For reporting results from FQL and $n$-step FQL, we use the implementation from Park et al. (2025c). For Q-Chunking, we re-implement the code from Li et al. (2025b) (^3^33[https://github.com/ColinQiyangLi/qc](https://github.com/ColinQiyangLi/qc)) in our codebase. We found that simply increasing discount factor $\gamma$ leads to significant performance improvement for Q-Chunking, so we use the discount factor to be same with $\gamma_{2}$ for value function training. For implementing CQN-AS, we use the original implementation released by the authors from Seo & Abbeel (2025) (^4^44[https://github.com/younggyoseo/CQN-AS](https://github.com/younggyoseo/CQN-AS)) and integrate OGBench related codes on top of the codebase. Originally, CQN-AS is designed to apply auxiliary BC loss only on expert demonstrations, but considering the dataset distribution of OGBench tasks with nearly no success rollouts, we modify the BC loss on the suboptimal data as well (Fujimoto & Gu, 2021; Park et al., 2025c; 2024), where no significant difference with the original implementation. As the reward scale for OGBench is highly different according to the domain, we normalize the reward scale to be in $[-1,0]$, and use $\mathbf{v}_{\min}$ and $\mathbf{v}_{\max}$ as $-200$ and 0, respectively. For levels and bins, we use 5 (level) and 9 (bins) for all experiments.

##### Computing hardware

For all OGBench experiments, we use a single NVIDIA RTX 3090 GPU with 24GB VRAM and it takes about 2 hours for training the small model (used for 1M dataset) and about 8 hours for training the large model (used for 10M/100M dataset).

### A.2 VLA Experiments

##### Computing hardware

For all VLA experiments, we use NVIDIA A100 80GB GPUs. Fine-tuning GR00T N1.5 takes about 4 hours for 100 expert demonstrations and successful rollouts. For training DEAS and baselines, it takes about 10 hours with the same data, as we use a larger batch size.

##### VLA fine-tuning

We implement our method and baselines on top of the open-source implementation of GR00T N1.5 (NVIDIA, 2025) (^5^55[https://github.com/NVIDIA/Isaac-GR00T](https://github.com/NVIDIA/Isaac-GR00T)). As our code is based on an earlier version of GR00T N1.5, we conduct experiments without introducing future tokens to the action expert modules. For fine-tuning GR00T N1.5, we use a batch size of 32 and train for 30K (RoboCasa Kitchen) / 10K (Real Robot) steps using AdamW (Loshchilov & Hutter, 2019) optimizer with learning rate $1\times 10^{-4}$ and cosine annealing schedule.

#### A.2.1 RoboCasa Kitchen Experiments

##### Task

RoboCasa Kitchen (Nasiriany et al., 2024) is a simulation environment with a mobile manipulator attached to a Franka Panda robot arm in household kitchen environments. Among 24 atomic tasks provided by the environment, we select 4 challenging tasks ($\tt{CoffeeSetupMug}$, $\tt{PnPMicrowaveToCounter}$, $\tt{PnPMicrowaveToMicrowave}$, $\tt{PnPMicrowaveToStove}$) that require relatively long-horizon and delicate manipulation with small grasping part, which is demonstrated by the low success rate of the base model. For perception, camera images from 3 different viewpoints (left front, right front, wrist), proprioceptive states including position/velocities of joint/base, and natural language instructions, are provided. For reward function, we use the pre-defined success detector in the environment, and use the sparse reward function where the reward is $1$ if the task is completed, and 0 otherwise.

##### Implementation details

As an input for the value function, we first use the proprioceptive states from the robot, including joint position/angle, base position/orientation for the mobile manipulator. To provide information on target objects to the value function, we utilize the encoded representation of three different camera views and task instructions from the VLM backbone. For the value/critic network architecture, we use the same hyperparameters as those used for the 100M dataset experiments. For optimizing value and critic function, we use the expectile parameter $\tau$ as 0.7, and use $\gamma_{1}=0.9$, $\gamma_{2}=0.99$, universal support type for distributional RL, for all experiments. For selecting action candidates with the value function, we first sample $N=10$ candidates from the policy. For selecting final actions, we try either 1) greedy sampling with highest Q-value or 2) inspired by Nakamoto et al. (2024), sampling the action from a categorical distribution obtained by temperature controlled softmax over Q-values: $a_{t}\sim\text{Softmax}(\frac{Q(s_{t},a_{1})}{\beta},\ldots,\frac{Q(s_{t},a_{N})}{\beta})$ with temperature $\beta=1$ and report the best result for each task.

##### Training and evaluation

For expert demonstrations, we randomly sample 100 expert demonstrations using the publicly available dataset generated by MimicGen (Mandlekar et al., 2023). For training DEAS and baselines, we use a batch size of 64 and train for 30K steps using Adam optimizer with a learning rate of $3\times 10^{-4}$. For collecting rollouts, we use randomized environments using the object instance set $A$. For each task, we evaluate the model performance across 50 trials on five distinct evaluation scenes with 3 different evaluation seeds, totaling 150 rollouts. To test generalization capabilities, we evaluate the policy only on unseen object instances.

#### A.2.2 Real Robot Experiments

##### Hardware platform

We use Franka Research 3, a 7-DoF robotic arm, for our experiments. For visual perception, we utilize the dual camera with Intel RealSense D435i: a camera attached to the column next to the robot base to provide a global view, and a wrist-mounted camera for a close-range view. Teleoperated demonstrations are collected using an Oculus Quest 2, and we log time-synchronized RGB images, joint states, and gripper width for data collection. Demonstrations are recorded at 15 Hz. See [Figure 5](https://arxiv.org/html/2510.07730v1#A1.F5) for visual examples.

Figure: Figure 5: Real-robot platform.
Refer to caption: figures/png/franka_real_setup.png

##### Task

We evaluate the model performance on pick-and-place tasks from the countertop to the bottom cabinet, with three different objects: $\tt{peach}$, $\tt{milka}$, and $\tt{hichew}$. Each object has different properties: $\tt{peach}$ is a rigid object with a relatively larger size that is easy to occlude, $\tt{milka}$ is a deformable object with a relatively smaller size that is easy to deform, and $\tt{hichew}$ is a hard object requiring precise grasping due to its small width. For collecting demonstrations, we use different initialization points (center, top, bottom, left, right) and collect one demonstration for each position (see [Figure 6](https://arxiv.org/html/2510.07730v1#A1.F6) for the initialization points used in our experiments). For accurate value function estimation, we manually label the reward function for each task. Specifically, we split the task into 4 stages: 1) moving to the countertop, 2) picking up the object, 3) moving to the target position, and 4) placing the object. For each stage, we label the reward function as $1$ if the task is completed, and 0 otherwise, and we set the reward function as the negative number of uncompleted stages following Park et al. (2025a).

Figure: Figure 6: Initialization points used for pick-and-place tasks.
Refer to caption: x6.jpg

##### Implementation details

Unless otherwise mentioned, we follow the same implementation details as in the RoboCasa Kitchen experiments. For selecting final actions, we use $N=50$ candidates from the policy and use the same procedure for selecting the final action as in the RoboCasa Kitchen experiments.

### A.3 Hyperparameters

We list the hyperparameters used in our OGBench experiments in [Tables 5](https://arxiv.org/html/2510.07730v1#A1.T5) and [6](https://arxiv.org/html/2510.07730v1#A1.T6). For the BC coefficient $\alpha$ used for policy extraction, please refer to [Table 7](https://arxiv.org/html/2510.07730v1#A1.T7).

**Table 5: DEAS hyperparameters for OGBench experiments.**
| Hyperparameter | Value |
| --- | --- |
| Gradient steps | $1$M (1M dataset), $2.5$M (10M/100M dataset) |
| Optimizer | Adam (Kingma, 2015) |
| Learning rate | $0.0003$ |
| Batch size | $256$ (1M dataset), $1024$ (10M/100M dataset) |
| Actor MLP size | $[512,512,512,512]$ (1M dataset) |
|  | $[1024,1024,1024,1024]$ (10M/100M dataset) |
| Critic MLP size | $[256,256,256,256]$ (1M dataset) |
|  | $[512,512,512,512]$ (10M/100M dataset) |
| Value MLP size | $[128,128,128,128]$ (1M dataset) |
|  | $[256,256,256,256]$ (10M/100M dataset) |
| Nonlinearity | GELU (Hendrycks & Gimpel, 2016) |
| Layer normalization | True |
| Target network update rate | $0.005$ |
| Discount factor $\gamma_{1}$ | $0.9$ |
| Discount factor $\gamma_{2}$ | $0.995$ ($\tt{cube}$), $0.999$ ($\tt{scene},\tt{puzzle}$) |
| HL-Gaussian - Atoms | 101 |
| HL-Gaussian - $\sigma$ | 0.75 |
| HL-Gaussian - Support range type | data-centric ($\tt{cube}$), universal ($\tt{scene},\tt{puzzle}$) |
| Flow steps | $10$ |
| Critic ensemble size | $2$ |
| Action sequence length $H$ | 4 ($\tt{cube}$), 8 (scene, puzzle) |
| Expectile $\kappa$ (DEAS) | $0.9$ (1M dataset), $0.95$ (10M/100M dataset) |
| Double Q aggregation | $\min(Q_{1},Q_{2})$ |
| Policy extraction hyperparameters | [Table 7](https://arxiv.org/html/2510.07730v1#A1.T7) |

**Table 6: Baseline hyperparameters for OGBench experiments.**
| Hyperparameter | Value |
| --- | --- |
| Critic MLP size | $[512,512,512,512]$ (1M dataset) |
|  | $[1024,1024,1024,1024]$ (10M/100M dataset) |
| Discount factor $\gamma$ (FQL, $n$-step FQL) | $0.99$ |
| Discount factor $\gamma$ (QC-FQL) | $0.995$ (cube), $0.999$ (puzzle) |
| Horizon reduction factor $n$ | $4$ (cube), $8$ (puzzle) |
| Policy extraction hyperparameters | [Table 7](https://arxiv.org/html/2510.07730v1#A1.T7) |
| Levels (CQN-AS) | 5 |
| Bins (CQN-AS) | 9 |
| C51 - $\mathbf{v}_{\text{min}},\mathbf{v}_{\text{max}}$ (CQN-AS) | -200, 0 |

**Table 7: Policy extraction hyperparameters for OGBench experiments. Note that we apply Q-Normalization (Fujimoto & Gu, 2021) for actor loss, except cube-double tasks.**
| Task | FQL $\alpha$ | $n$-step FQL $\alpha$ | QC-FQL $\alpha$ | DEAS $\alpha$ |
| --- | --- | --- | --- | --- |
| scene | $3$ | $1$ | $3$ | $3$ |
| cube-double | $300$ | $100$ | $300$ | $300.0$ |
| puzzle-3x3 | $3$ | $1$ | $1$ | $3$ |
| cube-triple | $3$ | $1$ | $1$ | $1$ |
| puzzle-4x4 | $3$ | $1$ | $1$ | $3$ |
| cube-quadruple | $3$ | $1$ | $1$ | $1$ |

## Appendix B Extended Related Work

##### Hierarchical RL and Options Framework

Some Hierarchical RL works seek to address the challenges of long-horizon and sparse-reward tasks by reducing the
effective horizon through learning value functions that consume multi-step actions (Kulkarni et al., 2016; Vezhnevets et al., 2017; Nachum et al., 2018; Ajay et al., 2021), usually combined with bi-level architectures. Among them, Options
framework (Sutton et al., 1999; Stolle & Precup, 2002; Bacon et al., 2017) introduces formalization of higher-level actions
that persist for multiple time steps with variable initiation/termination conditions, effectively reducing the planning
horizon and facilitating more efficient learning. Our approach leverages the options perspective by treating action
sequences as primitive options, enabling horizon reduction and improved value propagation without task-specific knowledge,
explicit goal conditioning, or manual sub-task specification.

##### Reinforcement learning with VLAs

Recent efforts have applied RL to VLA training (Zhang et al., 2024; Chen et al., 2025a; Zhang et al., 2025; Guo et al., 2025; Tan et al., 2025; Chen et al., 2025b; Li et al., 2025a), but most focus on on-policy online RL, which requires expensive interactions and cannot reuse transitions. A key limitation is that existing methods use single-step value functions $Q(s,a)$ for value learning, despite modern VLAs being designed to predict action sequences (Black et al., 2025; Bjorck et al., 2025; Intelligence et al., 2025). This mismatch between single-step value learning and multi-step action prediction limits the effectiveness of RL with VLAs. The most related work is CO-RFT (Huang et al., 2025), which applies chunked offline RL to VLA training, but differs from our approach in three key aspects: (1) CO-RFT uses actor-critic methods (Nakamoto et al., 2023) with single-step value functions while DEAS uses detached value learning with action sequences, (2) CO-RFT relies on human teleoperated expert demonstrations while we use small expert sets with large suboptimal rollouts, and (3) CO-RFT requires sophisticated transformer architectures while DEAS achieves improvements with simple MLP networks.

## Appendix C Limitations and Future Work

While DEAS demonstrates significant improvements over existing offline RL methods, several limitations and opportunities for future research remain. First, our current approach uses fixed action sequence lengths across different tasks, but the optimal sequence length varies significantly depending on task complexity. Future work should investigate adaptive mechanisms that can dynamically adjust action sequence lengths based on task requirements, potentially through adopting hierarchical policies (Kulkarni et al., 2016; Vezhnevets et al., 2017; Nachum et al., 2018). Second, while DEAS shows promising results on individual tasks, scaling to large-scale unified value functions remains a critical challenge for real-world deployment. DEAS currently trains reward models on 3-4 tasks simultaneously, but practical applications require learning from hundreds or thousands of diverse tasks. Future research should focus on developing scalable architectures and training procedures that can handle massive multi-task datasets while maintaining sample efficiency and avoiding catastrophic forgetting. Third, our method relies on distributional RL with fixed support ranges ($\mathbf{v}_{\min}$, $\mathbf{v}_{\max}$) and discretization parameters, which can significantly impact performance. The sensitivity to these hyperparameters limits the method’s robustness across different domains and reward scales. Future work should develop more robust frameworks that can automatically adapt to different reward distributions or provide principled ways to set these parameters.

## Appendix D Use of Large Language Models

We acknowledge the use of large language models (LLMs) in preparing this manuscript. LLMs were
employed solely to refine writing quality, including grammar correction, vocabulary suggestions, and
typographical checks. All substantive ideas, analyses, and conclusions in this paper are entirely the
work of the authors

## Appendix E Full Experimental Results

We include the full experimental results in OGBench experiments in Table [8](https://arxiv.org/html/2510.07730v1#A5.T8).

**Table 8: Full offline RL Results in 30 OGBench tasks. ^∗ indicates the default task in each environment. We report the success rate (%) and 95% stratified bootstrap confidence interval over 4 runs.**
| Task | #Data | FQL | N-step FQL | QC-FQL | CQN-AS | \columncolorgreen!10DEAS |
| --- | --- | --- | --- | --- | --- | --- |
| scene-play-singletask-task1-v0 |  | $\mathbf{100}$ $\pm 0$ | $\mathbf{100}$ $\pm 0$ | $\mathbf{99}$ $\pm 0$ | $2$ $\pm 1$ | \columncolorgreen!10$\mathbf{99}$ $\pm 1$ |
| scene-play-singletask-task2-v0 |  | $50$ $\pm 7$ | $4$ $\pm 3$ | $\mathbf{99}$ $\pm 1$ | $1$ $\pm 1$ | \columncolorgreen!10$\mathbf{97}$ $\pm 1$ |
| scene-play-singletask-task3-v0 | 1M | $\mathbf{95}$ $\pm 2$ | $78$ $\pm 5$ | $64$ $\pm 8$ | 0 $\pm 0$ | \columncolorgreen!10$75$ $\pm 6$ |
| scene-play-singletask-task4-v0^∗ |  | $3$ $\pm 2$ | 0 $\pm 0$ | $\mathbf{68}$ $\pm 1$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{65}$ $\pm 5$ |
| scene-play-singletask-task5-v0 |  | 0 $\pm 0$ | 0 $\pm 0$ | $35$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{45}$ $\pm 6$ |
| cube-double-play-singletask-task1-v0 |  | 46 $\pm 4$ | 17 $\pm 3$ | 68 $\pm 4$ | $7$ $\pm 1$ | \columncolorgreen!10$\mathbf{76}$ $\pm 3$ |
| cube-double-play-singletask-task2-v0^∗ |  | $10$ $\pm 2$ | $1$ $\pm 0$ | $47$ $\pm 8$ | $1$ $\pm 1$ | \columncolorgreen!10$\mathbf{51}$ $\pm 8$ |
| cube-double-play-singletask-task3-v0 | 1M | 9 $\pm 2$ | 1 $\pm 1$ | 40$\pm 6$ | 0 $\pm 1$ | \columncolorgreen!10$\mathbf{47}$ $\pm 4$ |
| cube-double-play-singletask-task4-v0 |  | 1 $\pm 1$ | 0 $\pm 0$ | $\mathbf{8}$$\pm 1$ | $1$ $\pm 1$ | \columncolorgreen!10$\mathbf{8}$ $\pm 1$ |
| cube-double-play-singletask-task5-v0 |  | 2 $\pm 1$ | 3 $\pm 1$ | 44$\pm 3$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{57}$ $\pm 3$ |
| puzzle-3x3-play-singletask-task1-v0 |  | $\mathbf{100}$ $\pm 0$ | $89$ $\pm 3$ | $\mathbf{97}$ $\pm 1$ | $1$ $\pm 2$ | \columncolorgreen!10$\mathbf{100}$ $\pm 0$ |
| puzzle-3x3-play-singletask-task2-v0 |  | $19$ $\pm 4$ | $40$ $\pm 10$ | $81$ $\pm 12$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{94}$ $\pm 5$ |
| puzzle-3x3-play-singletask-task3-v0 | 1M | $15$ $\pm 2$ | $14$ $\pm 3$ | $50$ $\pm 11$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{91}$ $\pm 3$ |
| puzzle-3x3-play-singletask-task4-v0^∗ |  | $35$ $\pm 4$ | $23$ $\pm 3$ | $31$ $\pm 4$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{91}$ $\pm 3$ |
| puzzle-3x3-play-singletask-task5-v0 |  | $47$ $\pm 4$ | $13$ $\pm 3$ | 50 $\pm 11$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{96}$ $\pm 2$ |
| cube-triple-play-singletask-task1-v0 |  | 31 $\pm 14$ | 17 $\pm 5$ | $\mathbf{100}$ $\pm 0$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{98}$ $\pm 1$ |
| cube-triple-play-singletask-task2-v0^∗ |  | $9$ $\pm 3$ | $\mathbf{91}$ $\pm 4$ | $\mathbf{92}$ $\pm 2$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{95}$ $\pm 2$ |
| cube-triple-play-singletask-task3-v0 | 10M | $12$ $\pm 5$ | 0 $\pm 0$ | $\mathbf{92}$ $\pm 2$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{88}$ $\pm 3$ |
| cube-triple-play-singletask-task4-v0 |  | 0 $\pm 1$ | 0 $\pm 0$ | $\mathbf{59}$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!1045 $\pm 7$ |
| cube-triple-play-singletask-task5-v0 |  | $2$ $\pm 1$ | 0 $\pm 0$ | $74$ $\pm 4$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{87}$ $\pm 5$ |
| puzzle-4x4-play-singletask-task1-v0 |  | 54 $\pm 4$ | 28 $\pm 5$ | $66$ $\pm 17$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{92}$ $\pm 8$ |
| puzzle-4x4-play-singletask-task2-v0 |  | 24 $\pm 3$ | 2 $\pm 1$ | $\mathbf{80}$ $\pm 16$ | 0 $\pm 0$ | \columncolorgreen!10$42$ $\pm 7$ |
| puzzle-4x4-play-singletask-task3-v0 | 10M | 36 $\pm 4$ | 42 $\pm 7$ | $69$ $\pm 22$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{99}$ $\pm 1$ |
| puzzle-4x4-play-singletask-task4-v0^∗ |  | $22$ $\pm 2$ | $28$ $\pm 3$ | $70$ $\pm 17$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{88}$ $\pm 4$ |
| puzzle-4x4-play-singletask-task5-v0 |  | $22$ $\pm 4$ | $3$$\pm 2$ | $61$$\pm 19$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{89}$ $\pm 6$ |
| cube-quadruple-play-singletask-task1-v0 |  | $79$ $\pm 6$ | $70$ $\pm 9$ | $79$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{92}$ $\pm 5$ |
| cube-quadruple-play-singletask-task2-v0^∗ |  | 0 $\pm 0$ | $\mathbf{97}$ $\pm 2$ | $63$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{100}$ $\pm 0$ |
| cube-quadruple-play-singletask-task3-v0 | 100M | $6$$\pm 3$ | 1 $\pm 1$ | $33$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{62}$ $\pm 9$ |
| cube-quadruple-play-singletask-task4-v0 |  | 0 $\pm 0$ | $13$ $\pm 5$ | $\mathbf{38}$ $\pm 7$ | 0 $\pm 0$ | \columncolorgreen!1031 $\pm 7$ |
| cube-quadruple-play-singletask-task5-v0 |  | 0 $\pm 0$ | 0 $\pm 0$ | $12$ $\pm 6$ | 0 $\pm 0$ | \columncolorgreen!10$\mathbf{35}$ $\pm 10$ |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Do you have experiments integrating DEAS (action sequences + detached critic) with representative methods such as CQL? For example, does adding detached sequence-level critics to CQL further reduce overestimation or improve long-horizon performance? 
2. To address potential value miscalibration, have you evaluated DEAS across datasets with varying proportions of expert trajectories (e.g., 10%/30%/50% expert)? How does data quality influence (a) final performance, and (b) the calibration of predicted Q vs realized returns? If absent, please add such sensitivity analysis or discuss limitations clearly.
soundness: 3
strengths: 1. The experiments applying DEAS to large VLA models are a strong selling point: DEAS improves VLA performance using mixed-quality offline rollouts plus a small set of expert data, which is important for real-world robotics.
2. The manuscript is structured logically and presents a well-targeted technical design.
summary: This paper presents an offline RL framework that tackles the value-overestimation problem through detached value learning. To improve stability, it also integrates distributional RL and dual discount factors for intra- and inter-option transitions. Empirically, DEAS is shown to outperform baselines on 30 long-horizon OGBench tasks and to improve large Vision-Language-Action (VLA) systems on both RoboCasa simulations and real Franka robot manipulation tasks. The main contributions are: (1) framing sequence-level offline RL under SMDP Q-learning with detached critics, (2) a recipe (distributional RL + dual discounts) for stable learning with action sequences, and (3) demonstrating practical gains on long-horizon benchmarks and VLA applications.
weaknesses: 1. My main concern is that the paper’s core components are largely integrations or adaptations of prior work, such as IQL-style detached learning, distributional RL methods, and option/sequence mechanisms similar to Q-Chunking.
2. Detached value learning updates the critic using in-distribution high-return actions, but when the offline dataset contains predominantly suboptimal trajectories, this strategy may still yield biased or miscalibrated value estimates (under- or over-pessimism). The paper does not adequately verify whether the learned value function aligns with true returns across different data quality levels. I recommend that the authors analyze Q-error across bins of trajectory quality and conduct sensitivity studies using datasets with controlled expert-to-suboptimal ratios (e.g., 10%, 30%, 50% expert data).
3. Several key equations are unnumbered and inconsistently referenced, which increases the difficulty of understanding Algorithm 1.
4. Although the central motivation of DEAS is to alleviate value overestimation induced by action sequences, the experimental results only report task-level success metrics without providing direct analyses of overestimation (e.g., predicted Q-values vs. realized returns). Including such quantitative evidence would substantially strengthen the paper’s claims.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1.  Can the authors clarify the novelty of the method beyond the combination of IQL and action sequences? Is the primary contribution the empirical demonstration that this combination is effective, or is there a more fundamental insight that I have missed?
2.  Regarding the scalability weakness: The paper notes the actor network must scale with $H$. This seems to be a major bottleneck. Have the authors investigated methods to decouple this than the entire $H$-step block at once?
3.  Given that the duration $H$ is fixed and deterministic, why was the method presented as an SMDP rather than, more precisely, as a standard MDP with a temporally-extended action space $\mathcal{O} = \mathcal{A}^H$? I suggest authors give this paper a major re-write and detach concepts such as SMDPs and options out of scope.
4.  Given the engineering success and strong empirical performance, I am happy to raise score if authros position their contribution from a more rigorous view.
soundness: 3
strengths: 1.  **Clear Problem Identification:** The paper clearly articulates a significant and practical problem: coupled actor-critic methods are unstable when the action space is $\mathcal{A}^H$.
2.  **Strong Empirical Results:** This is the paper's greatest strength. DEAS demonstrates state-of-the-art performance, with significant gains over baselines on the challenging OGBench tasks.
3.  **Real-World & VLA Validation:** The authors successfully scale the method to large VLA models and demonstrate its effectiveness and stability on a real robot. The fact that the QC baseline *collapses* on the real-world task while DEAS improves performance is a very strong supporting data point.
4.  **Effective Ablations:** The ablation studies in Table 4 are well-executed and convincingly demonstrate the necessity of each component of the method (e.g., $H > 2$, the combination of IQL and distributional RL).
summary: The paper addresses the challenge of scaling offline reinforcement learning to complex, long-horizon tasks. The authors identify that while using action sequences (instead of single actions) can reduce the effective planning horizon, it introduces severe value overestimation when combined with standard actor-critic frameworks.The core contribution is DEAS (DEtached value learning with Action Sequence), a method that learns a value function over $H$-step action sequences. To prevent value overestimation, DEAS "detaches" the critic and value function training from the actor. It does this by adopting an IQL-style expectile regression loss, which learns the value function by regressing only on the in-sample action sequences found in the offline dataset. This avoids querying the actor for out-of-distribution actions.The framework is stabilized by incorporating distributional RL and dual discount factors ($\gamma_1$ for intra-option rewards, $\gamma_2$ for inter-option values). The authors demonstrate that DEAS significantly outperforms existing single-step and action-sequence baselines on complex OGBench tasks and can be used to improve the performance of large-scale Vision-Language-Action (VLA) models in simulation and on a real robot.
weaknesses: 1.  **Limited Conceptual Novelty:** As stated in the *Contribution* section, the method is a synthesis of IQL, fixed-duration options, and distributional RL. The contribution is an empirical and engineering one, not a fundamental algorithmic advance.
2.  **Imprecise Technical Terminology:** The paper repeatedly invokes "SMDP Q-learning" and the "options framework", but the method itself is not a true SMDP. The use of a *fixed, deterministic* duration $H$ does not create an option induced Semi-Markov process. The system is, more accurately, a standard 1st-order MDP where the agent selects from a temporally-extended, high-dimensional action space $\mathcal{O} = \mathcal{A}^H$. This imprecise language is confusing and oversells the connection to the general options framework. From this point of view, it's only another variant of ordinary chunked action sequences and the performance boost whether come from temporal abstraction is skeptical.
3. **Unjustified Optimality**: Detaching value functions and fixed horizon clearly alters the underlying decision process compares original SMDP. The convergence and optimality of DEAS is unjustified and not bounded.
4.  **Poor Scalability with Horizon $H$:** The paper's own ablation study reveals a significant scalability limitation. The authors state that "when the sequence length becomes longer than 8, it requires proportionally larger actor networks to handle the increased action dimensions". This creates a direct and problematic coupling between the task horizon ($H$) and the required network capacity, which severely limits the method's applicability to *truly* long-horizon problems.
5.  **Increased Hyperparameter Sensitivity:** The method adds several critical new hyperparameters that require tuning, including the sequence length $H$ (which Table 4a shows is task-dependent), and the dual discount factors $\gamma_1$ and $\gamma_2$, which the authors note are "critical for stable training".

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How can the policy be effectively learned when each decision involves generating a high-dimensional action sequence? Does the paper propose any mechanism to mitigate the exponential growth of the action space?  
2. Why do Equations (1) and (2) place the target distribution inside the logarithm term in the cross-entropy loss, and how should we interpret the missing parameter $\theta$ in Equation (2)?
3. How are the support bounds $v_{min}$ and $v_{max}$determined? These values appear to constrain the expressiveness of the critic network, while setting a larger range may make the learning process more difficult.

Minor comments:
1. The description of IQL in Section 3 is inaccurate. The paper states that “By using $\tau > 0.5$, Equation 3 penalizes the overestimated value in out-of-distribution actions.” (By the way, there is no Equation 3.) However, since $u = Q - V$, where $Q$ is obtained from the target network and $V$ is the learnable network, the equation should instead penalize the **underestimated** values of the value network.  
2. There is also no Equation (4.2), as referenced in Algorithm 1. Please verify and correct the numbering.  
3. It is recommended to mention HLG again after Equation (2).
soundness: 3
strengths: 1. Effectively leverages action sequences to address sparse rewards and long-horizon credit assignment challenges.
2. Appropriately builds on the IQL framework for stable critic learning under the offline setting.
3. Demonstrates strong empirical performance on complex, long-horizon continuous-control tasks in both simulation and real-world environments.
summary: The paper proposes DEAS, a method for offline reinforcement learning that models each decision as an action sequence rather than a single action. Building on the IQL framework, DEAS learns a critic consisting of $Q$ and $V$ networks while allowing an arbitrary policy architecture for action generation. During critic training, both the predicted and target values are treated as distributions and optimized via a cross-entropy loss. Experiments on continuous-control tasks, including both simulated and real-world robotic environments, demonstrate the effectiveness of the approach.
weaknesses: 1. The novelty of the paper is somewhat limited. It mainly combines existing ideas from action-sequence RL, IQL, and HLG.  
2. The cross-entropy loss formulations in Equation (1) and Equation (2) are confusing. Typically, the predicted distribution appears inside the logarithm term and the target distribution outside. In contrast, both equations place the target inside the log term, which is unconventional. Moreover, in Equation (2), the role of parameter $\theta$ on the right-hand side is unclear, as only $\psi$ and $\hat{\theta}$ are explicitly shown.
3. The paper provides limited discussion on how to learn the policy based on the trained critic. Since the policy outputs an entire sequence of actions per decision, the resulting product action space becomes extremely large, making policy learning challenging. However, the authors do not explain how this issue is addressed.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Have you compared your method with other competitive baselines such as CQL [1]?

Reference:

[1] Kumar et al. "Conservative Q-Learning for Ofﬂine Reinforcement Learning", NeurIPS, 2020.
soundness: 3
strengths: - The motivation of the proposed method is well explained, and this paper is easy to follow.
- The proposed DEtached value learning with Action Sequence (DEAS) framework that decouples critic training from the actor, encouraging the value function to prioritize high-return actions observed in the offline dataset.
- The proposed approach outperforms baselines in various benchmarks and real-world tasks.
summary: This paper proposes a value learning framework that leverages action sequences for value learning to facilitate complex, long-horizon sequential decision making. Experimental results on OGBench, RoboCasa Kitchen, and real-world tasks demonstrate the effectiveness of the proposed method.
weaknesses: - No quantitative results are provided to demonstrate the prevention of value overestimation.
- The VLA experiments are conducted with only a single baseline. It would be more convincing to include comparisons with other VLA methods, such as OpenVLA [1].
- No sensitivity analysis is conducted on the hyperparameter $H$.

Reference:

[1] Kim et al. "OpenVLA: An Open-Source Vision-Language-Action Model", arXiv preprint arXiv:2406.09246, 2024.


Decision: Accept (Poster)