Reinforcement Learning with Action Chunking
Authors: Qiyang Li
Zhiyuan Zhou
Sergey Levine
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
offline-to-online RL
exploration
Primary area: reinforcement_learning
TL;DR: We propose Q-chunking, a simple, effective offline-to-online RL method that uses action chunking to improve value propagation and exploration via temporally coherent actions.

Abstract: We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a *chunked* action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased $n$-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.

Reinforcement Learning with Action Chunking
Qiyang Li, Zhiyuan Zhou, Sergey Levine
UC Berkeley
{qcli, zhiyuiihiiunnaaa www..hh.....ieeeeee.e...e. .ew.eeeeeeeeeeeeeeeeeeeeee.eeeee edu
Abstract
We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, is obvious how the offline data should be utilized acquire good exploratory policy. Our key insight is that action chunking, technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in 'chunked" action space, enabling the agent to (I) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased n-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline- to-online methods on a range of long-howe oong-hoiioon, sparse-reward manipulation tasks.
Time Steps
111 012
t13
t14
ti
t16 to7 118 119
Aggregated (5 OGBench Domains)
1.0
BFN
Environment
0.8
FQL
Success Rate
(1) efficient exploration
RLPD
with temporally coherent actions
0.6
IFQL
Action
0.4
0.2
State
Q(O,
0.0
0.0
0.5
1.0
1.5
2.0
(2) efficient value backup "skipping" time stops
Steps (x10%)
Figure 1: Q-chunking uses action chunking to enable fast value backups and effective exploration with temporally coherent actions. left: overview of our approach: Q-chunking operates in temporally extended action space that allows for (1) efficient value backups and (2) effective exploration via temporally coherent actions; right: Our method (QC) first pre-trains on an offline dataset for 1M steps (grcy) and then updates with online data for another 1M steps (white). Our method achieves strong aggregated performance over five challenging long-horizoo sparse-reward domains in OGBench. Code: github. com/ wowinoiyangnggg/gi.eoeggaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Reinforcement learning (RL) holds the promise of solving any given task based only reward function. However, this simple and direct formulation of the RL problem is often impractical: in complex environments, exploring entirely from scratch to learn an effective policy can be prohibitively expensive, as it requires the agent to successfully solve the task through random chance before learning good policy. Indeed, even humans and animals rarely solve tasks entirely from scratch, instead leveraging prior knowledge and skills from past experience. Inspired by this, a number of recent works have sought to incorporate prior offline data into online RL exploration [28, 39, 85]. But this poses a new set of challenges: the distribution of offline data might not match the policy that the agent should follow online, introducing distributional shift, and it is not obvious how the offline data should be leveraged to acquire a good online exploratory policy.
In the adjacent field of imitation learning (IL), widely used approach in recent years has been to employ action chunking, where instead of training policies to predict a single action based on the state observation from prior data, the policy is instead trained to predict short sequence of future actions (an "action chunk") [90, 11]. While a complete explanation for the effectiveness of action chunking in IL remains an open question, its effectiveness can be at least partially ascribed to better handling of non-Markovian behavior in the offline data, essentially providing a more powerful tool for modeling the kinds of complex distributions that might occur in (for example) human-provien provided demonstrations or mixtures of different behaviors [90]. Action chunking has not been used widely in RL, perhaps because optimal policies in fully observed MDPs are Markovian [74], and therefore chunking may appear unnecessary.
We make the observation that, though we might desire final optimal Markovian policy, the explo- ration problem can be better tackled with non-Markovian and temporally extended skills, and that action chunking offers very simple and convenient recipe for obtaining this. Furthermore, action chunking provides a better way to leverage offline data (with a better handling of non-Markovian behavior in the data), and even improves the stability and efficiency of TD-based RL, by enabling unbiased n-step backups (where n matches the length of the chunk). Thus, in combination with pre- training on offline data, action chunking offers compelling and very simple way mitigate the exploration challenge in RL.
We present Q-learning with action chunking (or Q-chunking in short), a recipe for improving generic TD-based actor-critic RI. algorithms in the offline-to-online RI. setting (Figure 1). The key idea is to run RI. at an action sequence level - (1) the policy predicts sequence of actions for the next h steps and exccutes them one-by-one open loop, and (2) the critic takes in the current state and a sequence of actions and estimates the value of carrying out the whole sequence rather than single action. The benefits of operating RL on this extended action space are two-fold: (I) the policy can be optimized to generate temporally coherent actions by regularizing it towards some prior behavior data that exhibit such coherency, (2) the critic trained with a standard backkc loss is effectively performing n-step backups, with no off- off-colicy bias (that typically occurs in naïve n-step return methods), since the critic takes the full action sequence into account.
Our main contribution is QC, practical offline-toonllne RL. algorithm that is instantiated from our Q-chunking recipe, essentially running RL. with behavior regularization in the chunked action space. QC is simple to implement, requiring only training (1) an action chunking behavior policy using a standard flow-matching loss, and (2) a temporally extended critic with the standard TD-loss (Algorithm I). QC achieves strong performance on a range of six challenging long-horizon, sparse- reward domains, outperforming prior offline-to-online methods. Moreover, Q-chunking is a generic recipe that can be applied to existing offline-to-onlle tonlinne algorithms with minimal modification. In this work, we demonstrate one such instantiation by applying it to FQL [58], resulting in QC-FQL (Algorithm 2), which shows significant improvements over the original method.
2 Related Work
Offline-to-online reinforcement learning methods focus on leveraging prior offline data to accelerate reinforcement learning online [88, 71, 37, 1, 89, 7,51, 92, 39]. The simplest way to tackle offline- to-online RL is to use an existing offline RI. algorithm to first pretrain on the offline data and then use the same offline optimization objective to continue training online using growing dataset that

---

combines the original offline data and the replay buffer data [50, 36, 33, 77, 58, 2, 42, 37J. While straightforward, this naive approach often result in overly pessimistic that hinders exploration and consequently the online sample-efficiency. Several prior works have attempted to address this issue by adjusting the degree of pessimism online [92, 51, 42, 37, 82]. However, these approaches can be difficult to tune and sometimes stills fall short in online sample efficiency compared to a simple, well- regularized online RL algorithm learning from scratch on both offline data and online replay buffer data [7]. Our approach takes step towards improving the sample efficiency of offline-to-online RL methods via value backup acceleration and temporally coherent exploration.
Action chunking is technique popularized by roboticists for imitation learning (IL), where the policy predicts and executes a sequence of actions in an open-loop manner ("an action chunk") [90]. Action chunking has been shown to improve policy robustness [90, 23, 8], and handle non- non-Markovian behavior in offiine data [90]. Existing RL methods that incorporate action chunking typically focus on fine-tuning policy pre-trained with imitation learning [61]. Tian et al. [78] propose to learn critic on action chunks by integrating n-step returns with a transformer. However, their method only applies chunking to the critic, while still optimizing a single-step actor. Li et al. [38] also observe that learning a critic over short action chunks removes the off-policy bias in n-step return backups, leading to more stable and effective value learning. There are key differences between their work and ours - Li et al. [38] operate in the online episodic RL setting [84, 31] and use Gaussian policies to predict parameters of motion primitives (MP) [63, 54], which are then used to generate full action sequences at the beginning of each episode. In contrast, we operate in the conventional offline-to- online RL setting and leverage more expressive flow- www-maahiing--a bassed policies (as we find Gaussian policies to be ineffective as shown in Figure 2) to predict short action sequences directly in the raw action space. Seo and Abbeel [65] also train critics on action chunks and impose behavior cloning loss that aligns with the principles behind Q-chunking. The key distinction from our work lies in their use of multi-level, factorized critic architecture [66], which generates and refines action chunks from coarse to fine granularity via iterative discretization. At each level, the action space is discretized into bins, and the Q-function is modeled independently for each action dimension and timestep, conditioned on the whole action chunks predicted at the previous coarser level. While this factorized critic design enables tractable value-maximiza maximizing action sampling. it imposes strong structural assumptions on the action space, limiting policy expressiveness at each refinement level. In contrast, we make no such assumptions in our recipe which allows us to derive two general algorithms where both the critic and policy operate directly on action chunks without requiring factorization or iterative discretization/refineeee
Exploration with temporally coherent actions. Existing methods either rely on temporally corre- lated action noises [40] that are constructed through heuristics; hierarchically structured policies (see the next paragraph). which are often tricky to stabilize during online training; or pre-trained frozen skill policies [60, 85], which are not amendable for fine- fine-graing grained online fine-tuning. Our method uses single network to represent the policy to generate temporally extended action chunk and it is trained using single objective function that stable to optimize. There is also no frozen. pretrained components in our approach, ensuring its online fine-tuning flexibility.
Hierarchical reinforcement learning, options framework. Learning temporally extended actions have also been widely studied in the hierarchical reinforcement learning (HRL) literature [14, 16, 80, 13, 35, 81, 59, 62, 49, 3, 67, 60, 22, 87J. HRL methods typically train a space of low-level policies that can directly interact with the environment along with a high-level policy that selects among these low-level policies. These lowelevel policies can be hand-crafted [12], automatically discovered online [16, 35, 80, 81, 49], or pretrained using offline skill discovery methods [54, 47, 67, 3. 70, 60, 79. 52, 28, 20, 9, 57]. The options framework provides slightly more sophisticated and more powerful formulation, where the low-level policy is additionally associated with learnable initiation condition and termination condition that makes utilization of the low-level policy more flexible [75, 46, 10, 44, 68, 69, 32, 13, 72, 53, 19, 4, 30, 5, 6, 15]. A long-lasting challenge in HRL its bi-level optimization problem: when both level and high-level policies are updated during training, the high-level policies must optimize moving objective function, which can lead to instability [49]. To mitigate this, some methods keep the low-level policies frozen after initial pretraining [3, 60, 85] to improve stability during online training. Our approach is a special case of HRL where the low-level skill executes a sequence of actions open-loop. This design choice allows us to collapse the bi-level optimization problem into standard RL objective in temporally extended action space, while retaining many of the exploration benefits associated with HRL methods. While
3

---

prior work in HRL has also explored such idea [45, 17], they often leverage a discrete set of action sequences that is either heuristically extracted from the prior experience or given in advance. In contrast, we use an expressive flow-matching based policy to directly parameterize continuous space of action sequences, and are directly trained and fine-tuned online with RI..
Multi-step latent space planning and search is a technique commonly used in model-based RL methods where they use a learned model to optimize a short-horizon action sequence towards high- return trajectories [53, 64]. These approaches work by training a dynamics model on an encoded latent space, where the model takes in a latent state and an action to predict the next latent state and the associated reward value. This latent dynamics model, along with value network on the latent state, can then provide an estimate of the Q-value on- on-the-fly for any given action sequence starting from a given latent state by simply simulating the action sequence in the latent dynamics model. In contrast, we do not learn latent dynamics model and instead train Q-network to directly estimate the value of the action sequence. Lastly, these approaches operate in the purely online RI. setting whereas we focus on the offline-to-online RL setting.
3 Background
Offline-to-online RI. In this paper, we consider an infinite-horizon, fully observable Markov decision process (MDP), (S,A, p: T. T,r,7) where S is the state space, is the action space, T(s'ls, |s, a) S x A + ASS) is the transition kernel, r(s, a) : S x A + R is the reward function, p (S) is the initial state distribution and y E 10, is the discount factor. We also assume there is prior offline dataset D that consists of transitions rollouts {(s, a, r)) from M. The goal of offline-tooonline RL to find policy π (als)) 8 + (AA) that maximizes the expected discounted cumulative reward (or discounted return): 77(T)= == E -e (= sx) xc 7'r(st, dt)l. Oftentimes, offline-to- online RL algorithms operate in two distinct phases; an offline phase where policy pretrained on the offline data D and an online phase where the policy is further fine-tuned online with environment
interactions. Our approach follows the same regime.
Temporal difference and multi-step return. TD-baased RI. algorithms typically learn Qo a) to approximate the maximum expected discounted cumulative reward that policy can receive starting from state s and action a by using a temporal difference (TD) loss [74]:
L( 0.)
where V is an estimate of Q(se. a.) that is commonly chosen as Vi-step:
st == + 7 (1,1), a++1 Th (1((18++1),
(2)
and 8,, az, St+1. are sampled from some off- off-policy trajectories and Ø is a delayed version of f that does not allow the gradient to pass through for learning stability. When the TD error is minimized, the Qs converges to the expected discounted value of the policy π.. As the effective horizon Ā =T=12 1/(1 - 7) goes up, the learning slows down as the value only propagates 1 step backward (from 8(41 to 81). To speed-up long-horizon value backup, a common strategy is to sample a length-n trajectory segment, (St, at, St+1s ,044n-1, S(tn), and construct a n-step return from it [83, 74]:
(+n-1
Vn Snstep Σ
[ Qg(Setn; 0t4n), O2tn Tig (:18etn),
where again rs r(se, az). This value estimate of Q(st, a²) allows for n times speed-up in terms of the number of time steps that the value can propagate back across. This estimator is sometimes referred to the uncorrected n-step relurn estimator [18, 34] because it is biased when the data collection policy is different from the current policy Tg. Nevertheless, due to the implementation simplicity of n-step return, it has been commonly adopted in large-scale RL systems [48, 26, 29, 86).
4 Q-Chunking
In this section, we first describe two main design principles of Q-chunking: (1) Q-learning on a temporally extended action space (the space of chunks of actions), and (2) behavior constraint in this extended action space, followed by practical implementations of Q-chunking (QC, QC-FQL.) as effective bD-based oflline-to-olline RL algorithms.

---

4.1 Q-learning on temporally extended action space
The first design principle of Q-chunking is apply Q-learning on the temporally extended action space. Unlike normal 1-step TD-based actor-critic methods, which train Q-function (s,,, at) and policy (aa st), we instead train both the critic and the actor with a span of h consecutive actions:
Q- Q-Chunking Policy: (( = (a(a, a++1," Gcth-1184)
@- @-Chunking Critic: Qo(si, aca+h) == Qo(st, de, 0c-1:" -(4h-1)
In practice, this involves updating the critic and the actor on batches transitions consisting of random state 8t, an action sequence act+ followed by the state, and the state h steps into the future, 81th- Specifically, we train Qo with the following TD loss,
)
(Poler, amath)
L(0) = E2, Gr
- (o(ssth,,ctha4aa))) actha+ah)
(4)
with 4:4441 ~ To (-((4))))) and @ being the target network parameters that are often an exponential moving average of 0 [25].
The TD loss above shares striking similarity to the n-step return Equation 3 (with n matches h) but with a crucial difference - the Q-function used in the n-step return backup takes in only one action (at time step !) whereas our Q-function takes in the whole action sequence. The implication of this difference can be best explained after we write out the TD backup equations for standard 1-step TD, n-step return, and Q-chunking:
Q(s a1) + + Q((++1 0(+1)
(standard 1-step TD)
(5)
h-
Q(s), +
Σ
rw
aeth),
(n-Step return, h)
(6)
biased
b-
Q(sr, a(ath) x
(x + +1 Q arthet-an)
(Q-chunking)
(7)
unbinsed
For the standard 1-step TD, each backup step propagates the value back by only 1 time step. n-step return propagates the value back hx faster, but can suffer from a biased value estimation issue when S1:th and aua++ are off- oif-cyllcy [18]. This is because the discounted sum of the n-step rewards Te:+h from the dataset or replay buffer is no longer an unbiased estimate of the expected n-step rewards under the current policy π. Q-chunking value backup is similar to the n-step return where each step also propagates the value back by h time steps, but does not suffer from this biased estimation issue. Unlike n-step return where we are propagating the value to 1-step Q-function, Q-chunking backup propagates the value back to a h-step Q- Q-function that takes in the exact same actions that are taken to obtain the restep rewards Trat+hh. eliminating the biased value estimation. We formalize this argument Theorem A. As a result, Q-chunking value backup enjoys the value propagation speedup while maintaining an unbiased value estimate.
4.2 Behavior constraints for temporally coherent exploration
The second design principle of Q-chunking addresses the action incoherency issues by leveraging behavior constraint in the objective for the Tp:
L(tb)= - -,, D.an..Deeettat [Qo(se, au=+h)) s.t. D(T (ar.thlsss) n TT(at:44(ss) S (8)
where we denote T ra(amt+1/s as the behavior distribution in the offline data D, and D as some distance metric that measures how different the learned policy π deviates from T..
Intuitively, a behavior constraint on the temporally extended action sequence allows us to leverage temporally coherent action sequences in the offline dataset. This is particularly advantageous in
We use ats44 (or simply a.) to denote concatenation of consecutive actions: [a: ac+h-1]
R for notation convenience. This is similar for statth and Pectt.
5

---

QC-RLPD
RLPD-AC
RLPD
cube-double
cube-triple
cube-quadruple
1.0
1.0
1.0
Success Rate
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Online Steps (x106)
Naïvely using action chunking for online RL with Gaussian policies leads to poor performance. (1) RLPD runs online RL on both offline data and online replay buffer [7]. (2) RLPD-AC is the same algorithm RLPD but operates in temporally extended action space (action chunk size of 5). (3) QC-RLPD additionally uses bchavior cloning loss on the actor (4 seeds).
Figure 2:
the temporally extended action space compared to in the original action space because offline data often exhibit non Marrkvvaan structure (e.g.. from scripted policies [56], human tele-operators [43], noisy expert policies for sub-tasks [56, 21]) that cannot be well captured by Markovian behavior constraint. Temporally coherent actions are desirable for online exploration because they resemble temporally extended skills (e.g., moving in certain direction for navigation, jumping motions for going over obstacles) that help traverse the environment in a structured way rather than using random actions that often result in data that is localized near the initial states. Imposing behavior constraint for an action chunking policy is a very simple way to approximately extract skills without the need often necessitated by skill-based methods (see more discussion in Section 2). We observe that Q-chunking, with such behavior constraints, can explore with temporally coherent actions (see Section 5.3), mitigating the exploration challenge.
of training policy with bi-level structure
4.3 Practical implementations
key implementation challenge of Q-chunking to enforce good behavior constraint that captures the non-Markovian behavior at the action sequence level. One prerequisites of imposing good behavior constraint the ability the policy to capture the complex behavior distribution (e.g., with flow/diffusion policy). A Gaussian policy, default choice in online RL algorithms, does not suffice. Indeed, if we naively take an off-the-shelf online algorithm, RLPD [7] for example, and apply Q- chunking with behavior cloning loss, we find that it often performs poorly (Figure 2).
To enforce good behavior constraint, we start by using flow- fow-natching objective [41] train behavior cloning flow policy to capture the behavior distribution. The flow policy is parameterized by ctate-condiiioned velocity field prediction model f(s, z, u) S Rahh x [0, 1] r> Ah and we denote f& (-Is) as the action distribution that the flow policy parameterizes, which serves as an approximation of the true behavior distribution the offline data (fe RS TT). Now, we are ready to present our main method:
QC: Q-chunking with implicit KL behavior constraint. We consider a KL constraint on our policy through the learned behavior distribution:
DKL. [e(())) S
While it is possible include the KL as part of the loss, estimating the KL divergence or log probability for flow models practically challenging. Instead, value while imposing this KI. constraint implicitly altogether. Practically, this involves first sampling N action chunks from the learned behavior policy fe(:(s).
best-of-N sampling [73] to maximize Q-
{a), a², a²) fe(-ls),
and then picking the action chunk sample that maximizes the temporally extended Q-function:
a.' t arg max (a(' a?... . ((s,a)

---

It has been shown in prior work that best-of-N sampling admits a closed-form upper- bound on the KL divergence from the original distribution [27]:
- N
DKL(a" (a=])f)(-)s)) & log N
(10)
which approximately satisfies KI. constraint implicitly (Equation 9). Tuning the value of directly corresponds to the strength of the constraint.
Since we approximate the policy optimization (Equation 8) with the best-of-N sampling, we can completely avoid separately parameterizing a policy ти and only sample from the behavioral policy fe(+)se). In particular, we use the best-of-N sampling generate actions to both (1) interact with the environment, and (2) provide the action samples in the TD backup following Ghasemipour et al. [24]. As result, our algorithm has only one additional loss function:
h
of rete Q0 o )
L(0) = E
(o((sraa))
Σ Fml
(11)
S,NDD fa/ R- - fe(l-ee-))
where again a = arg maxae(aj Q(5,)(
While QC is simple and easy to implement, does come with some additional computational costs associated with best-of-N sampling. In experiments, experiment with two other variants (QC- FQL and QC-IFQL) that leverage cheaper off-the-shelf lifline/ofline-toonniinn RL methods (FQL and IFQL respectively [58]). We present the FQL-version below performs better empirically,
QC-FQL: Q-chunking with 2- Wasserstein distance behavior constraint. For this variant of our method, we leverage the optimal transport framework to impose a Wasserstein distance (W₂) constraint, again, through the learned behavior policy fe(-(s):
W2 fe(-ls)) S
(12)
Following FQL [58], we parameterize the policy Ty with a noise-conditioned action prediction model, (s,
RA4 + RAh, which directly outputs an action from Gaussian noise one network forward pass. This noise-conditioned policy is trained to maximize the Q-chunking critic Qa(s;, alc++)) while being regularized to be close to the behavioral cloning How-matching policy via BC loss that is shown to be upper-bound on the squared 2- Wasserstein distance [58]:
I(bb) = EAIND,
a [a||=* ( 11 - Q Map (8e, =))
(13)
2 2E,,- ( ((()) - Hah (s. [1)]
(14)
where z is the ODE solution from a 0 to = following d2 fc "", u)du (the initial value z" is sampled from the unit Gaussian). The real-valued hyperparameter a directly controls the magnitude of the distillation loss. Finally, the TD loss remains the same the previous section with the only difference in how we parameterize the policy:
of r - -ol---oooooooo.oeeteteeee 27
L(0) an
Qo(si, ai)
Σ
(15)
where again AA).
Offline-to-online RL considerations. Since both variants of our methods use behavior constraint (implicit KL for QC, explicit W for QC-FQL), we can also directly run them for offline RL pre- training. For both offline and online training, we use the same behavior constraint strength (e.g., N for QC and for QC-FQL). We use the same algorithm for both offline and online training, and the only difference whether there environment interactions. See Section C, Algorithm and Algorithm 2 for an overview of QC and QC-FQI. during online training.
5 Experimental Results
We conduct extensive experiments to analyze the empirical effectiveness our method on range of long- hong-horizon, sparse-reward domains. In particular, we going to answer the following questions:
7

---

acena-sparse
cube-daubla taska)
cuba-quadruple
(5
(5 taska)
taaks)
(S taska)
tasks)
R1.PD
100
94
98
41
67
From Scratch (1-seep TD)
R1.DD-AC
SUPE-GT
100
92
60
D
52
0- 20
0++ 59
'0 0
0
00
0-
12
Guussian (L-step TD)
BeBRAC
53 100
99
3-
T+
50
I'd
1FQL
98
07
08
93
11
89
18
38
27
Flew (1-step TD)
FQL
100
100
07
93
28
76
T 18
0++
S7
58
98 100
99
76
G 23
T+
12
51
IFQL-2
84 100
68
92
3- 29
0 0 0
81
44
Flow (0-step TD)
98 100
18 770
11 77
0 T
7-
57
27
67
B7N-n
68 -
07
-
i - B
-
216
0C-IPQL
100 100
83
88
13
78
0 0
T 16
36
59
O-chunking (Ours)
QC-POL.
63 100
84
99
39
100
53
77
38
0C
100
P4
52
Table 1: Summary table for OGBench offline-fo-online RL results. For each cell, we report the offline performance after 1M of training steps and then the online performance after 1M of additional online steps. The best method(s) for each column is highlighted bold and color. Q-chunking methods outperform all prior methods at end the online training. See full results Table (complete table) and Figure 10 (individual plot).
QC
BFN
FQL
RLPD
RLPD-AC
lift
square
1.00
00000
1.00
1.00
Googg
Success Rate
Bodgo
0.50
0.50
0,50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps (x10%)
Figure 3: Robomimic results. QC achieves strong performance across all three robomimic tasks. The first 1M
steps offline and the next 1M steps online with environment step per training step (5 seeds).
(Q1) How well do Q-chunking methods perform compared prior offine-to-online RL. methods?
(Q2) Why does action chunking helps online learning?
(Q3) How does chunk length, critic ensemble size, and update-to-data ratio affect performance?
5.1
Environments and Datasets
We consider six sparse reward robotic manipulation domains with tasks of varying difficulties. This includes domains (5 tasks each) from OGBench [55], scene- sparse, puzzle-3x3-a -x3-sparse. cube-double/ coooulletrrppe wtteeeerrpee/eeeeee periiuulqqudruppee and 3 tasks from robomimic [43]. For OGBench, we use the default play-style datasets except for cube qubdruue where we use the larger 100M-size dataset. For robomimic, use the nulti-human datasets. See more details in Appendix B.
5.2 Comparisons
We compare with prior methods that speedup value backup as well as the previous best offline-to- online RL methods. We include brief description of them below with more details in Section C.
BFN (fest-of-N) a baseline that we propose combine the expected-max Q operator [24] with an expressive behavior flow policy. BFN operates in the original action space and uses best-of-N
8

---

BFN
Temporal Coherency in EEF Positions
Avg. of |x] Xt
0.06
0.04
QC
BFN
0.04
0.06
0.08
Online Steps (x10%)
Figure 5: End-effector movements early in the training and temporal coherency analysis on cube-tripp triple-taskk, Left: QC covers more diverse set of states compared to BFN in the first 1000 environ- steps. Right: QC exhibits higher temporal coherency in end-effector compared BFN.
sampling to pick the action (out of N) that maximizes the current Q-value. This baseline is an ablation to isolate the benefit of Q-chunking in QC.
FQL [58] is recently proposed offline RL method that achieves strong offline and oflline-to-online RL performance. This baseline is an ablation isolate the benefit of Q-chunking in QC-FQL.
BFN-n/FQL-n. These baselines are the same as BFN/FQL but uses n-step backup with n 1 (Equation 3) instead of the standard 1-step TD backup. This baseline enjoys the benefits of value backup speedup, but does not use chunked critic or actor, and potentially suffer from the bias issue.
RLPD [7], RLPD-AC. RLPD is sample-efficiet RI. algorithm that treats offline data as additional off-policy data and learn from scratch online. RLPD-AC is the same RLPD but operates the temporally extended action space. Both of them do not use a behavior constraint.
We also compare with SUPE-GT [85], recently proposed skill-based that is designed for offline- to-online RL. The original method designed to deal with unlabeled dataset by learning a reward model with reward bonuses. We adapt it to setting by directly using the ground truth rewards.
Finally, we compare with additional baselines: IQL [33], ReBRAC [76], IFQL (a baseline imple- mented in Park et al. [58] that combines IQL. with rejection sampling policy extraction), and IFQL (IFQL. with n-step return backup).
5.3 How well does our method compare to prior offline-to-online RL methods?
We report the performance of all Q-chunking methods and the baselines in Table (OGBench) and selective (strong) subset in Figure 3 (robomimic). QC achieves compet- itive performance offline, often matching sometimes outperforming best prior methods. In the online phase, QC shows strong sample-efficienyy, especially on two hardest OGBench domains (cube-trij) triple/ ttripleqquarrpplo quadruple), where it outperforms all prior methods (especially on cube-quadruple) by large margin. In addition, OG- Bench domains, all Q-chunking methods (QC, QC-FQL, QC-IFQL) outperforms both their corresponding 1-step TD counterpart (BFN, FQL, IFQL) and their correspdon- ing n-step return counterpart (BFN-n, FQL-n, IFQL-n). A similar trend continues on robomimic tasks as shown in Figure 4, Across OGBench and robomimic, n-step return baselines, which do not use chunked critics or policies, perform significantly worse than Q-chunking methods.
robomimic (3 tasks)
1.0
0.8
0.6
Success
0.4
0.2
QC
- BFN - BFN
QC-
- FQL.n -a- FQL
0.0
0.0
1.0 Steps (x109)
1.5
2.0
Figure 4: n-step return ablations robomimic. Both Q-chunking methods con- sistently outperform their re-tep and 1-step TD counterparts (5 seeds).
5.4 Why does action chunking help exploration?
We hypothesize Section 4.2 that action chunking policy produce more temporally coherent actions and thus lead to better state coverage and exploration. In this section, we study to what degree that

---

Action Chunk Size
Critic Ensemble Size
Update-to-data Ratio
1.0
+8-
1.0
1.0
BTA CUTD-1111
QC(UTD=S)
BTN (UTD-
0.8
Success Rate
0.8
Success Rate
0.8
0.6
0.6
0.6
0.4
QC (K=10)
DC (K-22)
BFN (K=10)
0.0
0.0
0.0
1.0
1.5
2.0
1.0
1.5 Steps (x10%)
2.0
2.0
Steps (x10F)
Steps (x10f)
Figure 6: Sensitivity analysis: action chunk size (h), critic ensemble size (K), and update-to-data ratio (UTD), Left: QC-FQL with different i on all 5 cube- trripe tasks (5 seeds). QC-FQI with h = 1 is equivalent FQL. Center:
Increasing the ensemble size to K 10 improves performance of both QC and BFN on cube-tripp trippe-taasseee (5 seeds). Right: QC with UTD 5 on cube cube-triple -aakk (5 seeds). We report only the online phase results, as all methods achieve near-zen zero success rates during the offline phase.
holds empirically. We first visualize the end-effector movements early in the training for QC and BFN (Figure 5, left). BFN's trajectory contains many pauses (as indicated by a very big and dense cluster near the center of the visualization), especially when the end-effector is being lowered to pickup a cube, In contrast, QC has fewer pauses (fewer and shallower clusters) and a more diverse state coverage in the end-effector space. We include additional examples in Appendix E, Figure 8 and Figure 9. To get a quantitative measure of the temporal coherency in actions, we record the 3- D end-effector position throughout training every 5 time steps: {xggt .55 average L2 norm of the difference vector of two adjacent end-effector positions. This average norm would be small if there any pauses or jittery motions, making good proxy for measuring the temporal coherency in actions. As shown in Figure 5 (right), QC exhibits a higher action temporal coherency throughout training compared BFN. This suggests that Q-chunking improves temporal coherency in actions, which explains the improved sample-efticiency that Q-chunking brings.
... } and compute the
5.5 How does action chunk length, critic ensemble size, and UTD ratio affect performance?
In Figure 6 (left), we examine the performance of QC-FQL across different action chunk lengths (h E {1, 5, 10, 25, 50)) on the cube-triple domain. Increasing the chunk length helps up to h 10, after which the asymptotic performance starts drop. Although h 25 shows faster early learning, it fails to achieve the same performance as h = 10 at the end of online fine-tuning. An even larger chunk length (h 50) fails achieve any success. We suspect that overly large chunk sizes either hurt policy reactivity too much or make policy learning too difficult, as the network must predict a much longer action sequence at once. We use h in all our other experiments as h = 5 generally performs well and is cheaper to run. We also include the individual task breakdown in Figure 14 and additional ablation results on qube-quadruple in Figure 15. In Figure 6 (center), we study how the critic ensemble size affects the performance of our method. Using 10 critics improves both QC and BFN. We use K = our other experiments as it cheap to run. Using K 10 could potentially make Q-chunking perform much better on the benchmark tasks we consider. Finally, increasing the update-to-data ratio (UTD) does not improve the sample efficiency of QC (Figure 6, right).
Discussions
We demonstrate how action chunking can be integrated into an offline-to-online RL agent with simple recipe. Our approach speeds up value backup and explores more effectively online with temporally coherent actions. As result, outperforms prior offline-to-online methods on range of challenging long- long-horizon tasks. Our work serves as a step towards training non-Markovian policy for effective online exploration from prior offline data. Several challenges remain, opening promising directions for future research. First, our approach use fixed action chunk, but it is unclear how to choose this size other than task- speciffc hyperparameter tuning. A natural next step would be to develop mechanisms that automatically determine chunk boundaries. Second, action chunking represents only a limited subclass of Mar-Maaoo policies and may perform poorly in settings where a figu-fryy control feedback loop is essential. Developing practical techniques for training more general MarkMvakovian policies for online exploration would further improve the online sample efficiency of offline-to-online RL algorithms.
10

---

Acknowledgments
This research was partially supported by RAI, and ONR N00014-22-1-2773. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley. We would like thank Seohong Park for providing the code infrastructure and the 100M dataset for cube- sube-quadruple We would also like to thank Ameesh Shah, Chuer Pan, Oleg Rybkin, Andrew Wagenmaker, Seohong Park, Yifei Zhou, Fangchen Liu, William Chen for discussions on the method and feedback on the early draft of the paper. Finally, we would like to thank NeurIPS review commitee for thoughtful comments and suggestions.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 28971. Curran Associates, Inc., 2022.
[2] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in neural information processing systems, 15:28955-22897 2022.
[3] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Of- fline primitive discovery for accelerating offline reinforcement learning. International Con- ference on Learning Representations, 2021. URL ptttttas://ppereveeeii wttp:eiwwwwweeefownn/eeoee V69LC=JO1IN.
[4] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[5] Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In Interna- tional Conference on Learning Representations, 2019.
[6] Akhil Bagaria, Ben Abbatematteo, Omer Gottesman, Matt Corsaro, Sreehari Rammohan, and George Konidaris. Effectively learning initiation sets in hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[7] Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference Machine Learning, pages 1577-1594. PMLR, 2023.
[8] Homanga Bharadhwaj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4788 4795. IEEE, 2024.
[9] Boyuan Chen. Chuning Zhu. Pulkit Agrawal, Kaiqing Zhang, and Abhishek Gupta. Self- supervised reinforcement learning that transfers using random features. Advances in Neural Information Processing Systems, 36, 2024.
[10] Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems, 17, 2004.
[11] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783669944127666, 2023.
[12] Murtaza Dalal, Deepak Pathak and Russ R Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Systems, 34:21847-21859, 2021.
[13) Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 17(93):1-50, 2016.
11

---

[14] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems, 5, 1992.
[15] Anita de Mello Koch, Akhil Bagaria, Bingnan Huo, Zhiyuan Zhou, Cameron Allen, and George Konidaris. Learning transferable sub-goals by hypothesizing generalizing features. 2025.
[16] Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303, 2000.
[17] Ishan P Durugkar, Clemens Rosenbaum, Stefan Dembach, and Sridhar Mahadevan. Deep reinforcement learning with macro-actions. arXiv preprint arXiv: 1606.046615555 2016.
[18] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning, pages 3061-3071. PMLR, 2020.
[19] Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv:1703.0822 2017.
[20] Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot rein- forcement learning via functional reward encodings. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. edi- tors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 13927-13942. PMLR, 21-27 Jul 2024. URL https::/proceeee
mlr prwss/ httpe/////fff/2mmeeem html.
[21] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[22] Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems, 14:11553-11564 2021.
[23] Abraham George and Amir Barati Farimani. One act play: Single demonstration behavior cloning with action chunking transformers. arXiv preprint arXiv:2339. 10175, 2023.
[24] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pages 3682-3691. PMLR, 2021.
[25] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[26] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[27] Jacob Hilton. KI divergence of max-of-n, 2023. BtRLhttps://ww. http://wwwwwacbb.....o co. alk/bon_ka pdf.
[28] Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai. and Chongjie Zhang. Unsupervised behavior extraction via random intent priors. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:: http:///ppppeeeeeeee///ee//e
[29] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[30] Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32, 2019.
[31] Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems, 21, 2008.
12

---

[32] George Dimitri Konidaris. Autonomous robot skill acquisition. University of Massachusetts Amherst, 2011.
[33] Nya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. arXiv preprint arXiv:2110.06169, 2021.
[34] Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting peng's q (A) for modern reinforcement learning, In International Conference on Machine Learning, pages 5794-5804. PMLR, 2021.
[35] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[36] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179- 1191, 2020.
[37] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning, pages 1702-1712. PMLR, 2022.
[38] Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann. TOP-ERL: Transformer based off-policy episodic reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://opennrevie. net/forumiid-ee Nd-WWNWWN.NN.
[39] Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine Accelerating explo- ration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024.
[40] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 150.0097,,, 2015.
[41] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[42] Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303. 17396, 2023.
[43] Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298, 2021.
[44] Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction reinforcement learning via clustering. In Proceedings of the twenty-first international conference on Machine learning, page 71, 2004.
[45] Amy McGovemn and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. 1998.
[46] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut dynamic discovery of sub-goals in reinforcement learning. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pages 295-306. Springer, 2002.
[47] Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Tch, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint
2018.
[48] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937 PmLR, 2016.
13

---

[49] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[50] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 20Xx::2006609339, 2020.
[51] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark. Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36, 2024.
[52] Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning, 2022.
[53] Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. Advances in neural information processing systems, 30, 2017.
[54] Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems, 26, 2013.
[55] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. ArXiv, 2024.
[56] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-condi arXiv preprint arXiv:2410.20092, 2024.
[57] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert repre- sentations. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview seee://frumidd-hhheeee
[58] Seohong Park, Qiyang Li, and Sergey Levine. 0235532502.02538, 2025.
Flow Q-learning.
arXiv preprint
[59] Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog), 36(())1--13 2017.
[60] Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188 204. PMLR, 2021.
[61] Allen Z Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint +XXXx:2409000588 2024.
[62] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. International conference on machine learning, pages 4344- 4353. PMLR, 2018
[63] Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid roboties. In Adaptive motion of animals and machines, pages 261-280. Springer, 2006.
[64] Julian Schritwieser,, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Si- mon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):0446999 2020.
[65] Younggyo Seo and Pieter Abbeel. Reinforcement learning with action sequence for data- efficient robot learning. arXiv preprint arXiv:2411.12155 2024.
[66] Younggyo Seo, Jafar Uruç, and Stephen James. Continuous control with coarse-to-fine re- inforcement learning. In 8th Annual Conference on Robot Learning, 2024. URL https: / //openreview. (ttt/formmidddjjjBB8....3
[67] Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference.
In International Conference on Machine Learning, pages 8624-8633, PMLR, 2020.
14

---

[68] Özgür Şimşek and Andrew G Barto. Using relative novelty to identify useful temporal abstrac- tions in reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 95, 2004.
[69] Özgür Şimşek and Andrew G. Barto. Betweenness centrality as basis for forming skills. Workingpaper, University of Massachusetts Amherst, April 2007.
[70] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learn- ing Representations, 2021. URL
[71] Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RI:: Using both offline and online data can make RI. efficient. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevier net/forumiid-yan /oyy
[72] Aravind Srinivas, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv: 1605.0535, 2016.
[73] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
[74] Richard S Sutton, Andrew Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[75] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2): 181-211, 1999.
[76] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances Neural Information Processing Systems, 16:11522-11600, 2023.
[77] Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
[78] Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-basse soft actor-critic with n-step returns. arXiv preprint arXiv:2503.03660 2025.
[79] Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations, 2022.
[80] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems, 29, 2016.
[81] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pages 3540-3549, PMLR, 2017.
[82] Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get family: State-adaptive balances for offiine-t-onllne reinforcement learning. Advances in Neural Information Processing Systems, 36:47081 2023.
[83] Christopher John Comish Hellaby Watkins et al. Learning from delayed rewards. 1989.
[84] Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforce- ment learning for neurocontrol problems. Machine Learning, 13(2):259-284 1993.
[85] Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine, Leveraging skills from unlabeled prior data for efficient online exploration. arXiv preprint
2024.
15

---

[86] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
[87] Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. In International Conference on Learning Represen- tations, 2021. URL https: : //tp//ppeeeeeeeeeeeeeeeeeeeeeeee wwwpeeeieiiieenneeeeeeeeeeeeeeee wwtteeetruuu ?id= (eed=jjXKKkkaa
[88] Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27407, 2021.
[89] Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging tonllie reinforcement learning. The Eleventh International Conference on Learning Representations, 2023. URL fttts:://operevieee net/ -orumiid=-Y344455
[90] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[91] Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1372--1330, 2023.
[92] Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data, arXiv preprint arX1:2241..0776.. 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction are supported by the results in Section 4 and Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss some limitations of our method in Section 6. Our assumptions and problem statement is included in Section 3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[Yes]
Justification:
the proof of our theoretical result is available in Section A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide the algorithm in Algorithm and Algorithm 2, and imple- mentation details, including hyperparameter choices in Section C. We provide public repo (github. com/ (oliinlyangli/gee of our code that reproduces results in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Yes, have open-sourced our code @ttthuu..coo//o111nnnagegg///// which also includes instructions to download the data.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
Doglii/ggides/Codeeee
cwteimiisssnooliiiiii onPolicy icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cc/public//pee hwtpc//gguues/eeeeeeeeee ottp://SooSoisiisoooooool.....
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we provide the details on the design choices in Section 5 and the hyperparame- ter values well as the hyperparameter tuning process in Section C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results in the paper have confidence intervals, denoting 95% confidence interval with four random seeds (five for robomimic results). We estimate the confidence intervals with bootstrapped sampling-
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide additional information on computer resources in Section D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eurriii /terrps..p.pp// cc htcp/ppllcc/thccccteeeeeeeeee
Answer: [Yes]
Justification:
We adhere to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is on core reinforcement learning algorithm that has minimal socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: all code packages and datasets are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release code as an asset and provide detailed documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do use LLM as an important component of this work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //terr//s....uuee cc/Coonferences/222555 2222L/) for what should or should be described.
23

---

A Theoretical Justification
Proposition A.1 (Q-chunking performs unbiased n-step return backup). Let St, "t, 8t4n be trajectory segment generated by following data collection policy Tz(ae, .a+n (8x) (i.e., 814K ~ T(- 814k-1, (,4kk-1) Vk [l,...., n.) and Tt, Tt-l.". Vitn-1 be the reward received at each corresponding time step (i.e., Tt+k -(stt (c+k)). Let (*(s)) be the value for action chunking policy πt T(ae,".. ... at+n $t) starting from state st. Let Q (st, ae. ac+1, (c-n--1) be the Q-value for t starting from state St and executing the action sequence at, ac+1, ,0ttn-11
The n n-etep return estimate Vu-step= == Ei=
(
+ (sL+n) under the trajectory segment [rt'-"re]
distribution described above is unbiased for Q" (31 a.. -actn-1) as long as V (sttn) is unbiased for V (St+a).
Proof. From the definition of Q* (8t, Ge: 0441,"'' 0t+n-1), we can write as an expectation:
T+n
(a aes a1111-101 " (ettn-1) = (s.
Σ
?
r(ss, ar)
+ V (8(-n))
(16)
T+n-l
R mein-
hr re
+ V (824n)
(17)
E E Pe 1010110
"r + (824a)
(18)
E Vustee
(19)
The second line uses the fact that the rewards rt, T"t1,""" M++4-1 associated with the trajectory segment St, a,,-. St+n. The third line uses the fact that (ss+4) is unbiased for V (sttn). The fourth line uses the definition of Vn-step-
The implication of this theorem is that we use the n-step return backup for the chunked Q- function (s;,, at a.+1,-.. a(+n-1) (with a value estimate of V. +- Q(s,, a,, a++1, 4(4n-1)). it converges the ground truth Qavauue under the same policy (st, ag a+t1,'. 444x-1)- In contrast, using n-step return backup in the original action space does not converge to the correct Q-value (i.e., Q" (st, dt, 0141," (t)n-1)).
B Domain Details
See overview of the six domains use in our experiments in Figure 7. We also include the dataset size, episode length and the action dimension in Table 2. In the following sections, we describe each domain in details.
B.1 OGBench environments.
We consider five manipulation domains from OGBench [55] and take the publicly available single- task versions of in our experiments. For scene- sreneesparse and puzzle-3x3s sparse, we sparsify the reward function such that the reward values are -1 when the task is incomplete and when the task is completed. For tube-double/triplee wtwttppppp/prruppleeeeee the RL agent needs to command an UR-5 arm to pick and place two/three/four cubes to target locations. In particular, cube-triple trriple and cube-uueaee -uubdeuqaauppeee are extremely difficult to solve from offline data only, and often achieve zero success rate. The RL agent must explore efficiently online in these domains to solve the tasks. The cube-* domains provide a great test ground for sample-efficiency offline-to-online RL algorithms which we primarily focus on. For cube- cube-quadree quadruple, we use the 100M-size dataset. The dataset is too big to our CPU memory, so we periodically (after every 1000 gradient steps) load in 1M- size chunk of the dataset for offline training. For online training of RLPD, QC-RLPD, use the
24

---

Tasks
Dataset Size
Episode Length
Action Dimension (.A)
scene-sparse-
1M
750
puzzle puzzle-3x3-pper -parse---
1M
500
cube-double-s
1M
500
cube-triple-
3M
1000
Lube-quadrrple-1000-
100M
1000
lift can square
31 127
500
62 756
500
7
80731
500
Table 2: Domain metadata. Datasel size (number transitions), episode length, the action dimension. OGBench tasks, the action dimension 5 (a position, position, position, gripper yaw and gripper opening). For robomimic tasks, the action dimension is for square to control one arm (3 degree of freedoms (DoF) for translation 3 DoF for rotation, and one final DoF for the gripper opening).
same strategy where we load in a 1M-size chunk of the dataset the offline data and perform 50/50 sampling (e.g., 50% of the data comes from the 1M-chunk of the offline data, 50% the data comes from the online replay buffer). For online fine-tuning of QC-#, FQL, FQL-n, BFN, and BFN-n, we keep a fixed IM-size chunk of the offline dataset as the initialization of D and adds new data to D directly. The remaining 99M transitions in the offline data are not being used online. We now describe each of the five domains in details:
This domain involves drawer, a window, a cube and two button locks that control whether the drawer and the window can be opened. These tasks typically involve a sequence of actions. For example, scene-task2 requires the robotic arm to unlock both locks, move the drawer and the window to the desired position, and then lock both locks. scene-task4 requires the robotic arm to unlock the drawer, open the drawer, put the cube into the drawer, close the drawer. The reward binary: -1 the desired configuration is not yet reached and the desired configuration is reached (and the episode terminates).
scene-sparse:
puzzle-3x3 3xp-sprrs: This domain contains 3 x grid of buttons. Each button has two states represented by its color (blue or red). Pressing any button causes its color and the color of all its adjacent buttons flip (red >> blue and blue red). The goal is achieve pre-specified configuration of colors. puzzle-3xx exactly one button (the lop-left one) to be red. puzzle-3x3 - starts with four buttons (top- center, bottom--ceter, left-center, right-center) to be blue, and the goal is to turn all the buttons to be blue. The reward is binary: -1 the desired configuration is not yet reached and if the desired configuration is reached (and the episode terminates).
starts with all buttons to be blue, and the goal is to flip
cube-doubles htleeeddpleeeerrteeeeeeeee httpe/ttrrppee/uuuruuleeeeee quadruple: These three domains contain 2/3/4 cubes respectively. The tasks in the three domains all involve moving the cubes to their desired locations. The reward is -fwrong where Thwong is the number of the cubes that are at the wrong position. The episode terminates when all cubes are at the correct position (reward is 0).
B.2 Robomimic environments.
We use three challenging tasks from the robomimic domain [43]. We use the multi-human datasets that were collected by six human operators. Each dataset contains 300 successful trajectories. The three tasks are as described as follows.
lift: This task requires the robot arm pick small cube. This is the simplest task of the benchmark.
can: This task requires the robot arm to pick up a coke can and place in a smaller container bin.
This task requires the robot arm to pick square nut and place it on a rod. The nut is slightly bigger than the rod and requires the arm to move precisely to complete the task successfully.
square:
25

---

a) scene
b) puzzle een22ee-xx3 e-3x3
c) sube-double
d) cube- cube-triple triple
e) quae-que
f) lift
g) can
h) square
Figure 7: We experiment on several challenging long-horizon, sparse-reward domains. See detailed task description for each domain in Appendix B. The rendered images of the robomimic tasks above are taken from Mandlekar et al. [43].
All of the three robomimic tasks use binary task completion rewards where the agent receives -1 reward when the task not completed and 0 reward when the task is completed.
C Implementation Details
In this section, we provide more implementation details on both our Q-chunking methods (QC, QC-FQL.) and baselines used in our experiments. For IFQL,, ReBRAC, IQL., we directly the implementation from Park et al. [58].
Algorithm QC
Algorithm 2 QC-FQL
Input: Behavior policy and critic: Se(aetta)s)
Input: Behavior policy, critic, one-step policy: (arc+h)))) Qo(st, acs+h),, ). μs (8, z).
and Qu(se, (ss44))
'D + offline prior data.
+ offline prior data.
for every environment step f do if mod h then
for every environment step f do
ift mod h 0 then
acc+h ... dutth ~ f((()s))
aie+4 arg max, ii Qu(s, al.4k)
acs++ <- (s.,
Act with at receive 8t-1,
Act with de and receive st+1,
D <- D (( 8 r()
D + D U {( o,, ((,1,,,,
Update f via now-matching loss using D),
Update fe via flow- now-matching loss using D.
Update Qa via Eq (1I) using D.
Update Hy and Qo via Eq. (13, 15) using D.
Output: Qa.
Output: fc, Qo. (8)
C.1 QC-FQL
We build the implementation of our method on top of FQL [58], recently proposed offline RL/offline- to-online RL method that uses TD3+BC-style objective. It is implemented with one-step noise- conditioned policy (instead Gaussian policy that is commonly used RL) and it uses distillation loss from behavior flow-matching policy as the BC loss. To adapt this method to use action chunking, we simply apply FQL on the temporally extended action space behavior flow-matching policy generates sequence of actions, the one-step noise-conditioned policy predicts sequence of actions,
26

---

and the Q-network also takes in state and sequence of actions. More concretely, we train three networks:
1. Qo(8, ".......... .n): S x AA - R - the value function that takes in state and a sequence of actions (action chunk). In practice, we train an ensemble of Q networks. We denote the weight of the ensemble element as 8 = (01,
0x).
2. Hs (s, 2) SS x R Ah + IR the one-step caise--ondiiioeed policy that takes in a state and noise, and outputs sequence of actions conditioned on them.
3. fs, m, m, S x IR [0. RAA the flow-man fow-matching behavior policy parameterized by a velocity prediction network. The network predicts takes in a state, an intermediate state of the flow and time, and outputs the velocity direction that the intermediate action sequence should move in at the specified time. See Algorithm: 3 for more details on how this velocity prediction network is used to generate action from noise vector.
We denote our policy as (-(sss). It is implemented by first sampling a Gaussian noise 2 ~ N LAN) and run it through the one-step noise-conditiones policy [a1 three networks, sample high-level transition, (st, az 0141,'11 a1+h-1; Stth: rf) D where r Σk= 7f rete, to construct the following losses:
anl (p 2). To train these
(1) Critic loss:
K
2 Qā, aeth:"" .01t2h-1) ..4t2hh-))))))) (20)
(00 (sr, ae, ((4h--1) - r
1
L(Ok, w)
Σ
K
k'=1
where (ac+h
a0+24-1] ~ ( (1, K}.
(2) Actor loss:
L(v, (p + a ( 20)
of
(21)
where 2, NN(0, Ann) and (of with Algorithm from 24, and a e IR is a tunable parameter that controls the strength of the behavior regularization (higher leads to stronger behavior regularization).
a is the result of running the behavior policy fe (s, m, t)
(3) Flow-matching behavior policy loss:
L(g. w) = ( a jag
acth-1] + (1 - u)) u) (fat Oc4h-1] - 2,]1 (22)
where ((0, 1)), (O,,AA)).
w2. ... WMr and optimize the average loss Practically, sample batch of transitions {wi, (
for each network: L()) M N Dis W(),
1 EM
PMMEE DK L(Ok, ((v)) = + Di w(,, =
C.2 FQL
FQL [58] is a recently proposed offline RL/offline-to-nn RL method that uses TD3+BC-style objective. It is equivalent to our method with h = 1. For completeness, we write out the objectives for a transition sample w = (st, Ut, It, 8141):
K
I(Og, w) = Qo. Qo. (5)
- --)-)0
Σ
Q (Stt 2t (N(
(23)
K
K=1
I(b,, w) = Qo(st, Mo (8,, 2))) + a (p 21) - as 1?
(24)
of FlowODE Euler (st, Zt, fe, 1),22 -N(0, La),
(25)
I(S, w) = ( ua, + (1 u) u) (a, 2, 2 113, 2, ~ N(0, IA), U[(0 1)).
(26)
In practice, we sample a batch of transitions {1i, W2, wN) and optimize the average loss for each network: [()) $ EN, Σx L(Ox, w.), L(bb E/2 Σn L(v) w.), L(E) - E. L(E, w.).
27

---

Algorithm 3 FlowODE_ FloxODE_Bulerrss, Rt, Je, T); generate actions from the behavior flow policy [e(8, m, u) with Euler's method.
Input: State noise 2t and flow model fs(8, m, u), number of flow steps T.
m + 2t for Mric (I....,T) do m' f -ff(si, an (i-1)/
Output: m
C.3 FQL-n
To implement the n-step return baseline, take FQL and replace the 1-step TD update with the h-step TD update:
L(Ok: w) = (sr,
(st, at) ())())))((
h-1 ( T(4))
1
Σ
QB
(54thhHup Hp
(27)
K
P==0
P=1
where 2 ~NO0, IA) for all k' (1,2,... K... The actor loss and flow-matching loss remain the same as FQI..
C.4 QC
The flow-matching behavior policy trained with the same loss as used QC-FQL (Equation (22)). On top of the flow-matching behavior policy, we simply parameterize the policy implicitly by sampling multiple action chunks from the behavior policy and pick the one that maximizes the Q- value. Specifically, let (zt 2f ... N NO, 7 A and
[at
d++n-1]
FlowODE CearODDEEElle(stte fs:
The policy outputs the one action chunk out of N that maximizes the Q-value, [at where
dfth-1],
arg max ((s (a} [N[
a(+n-1))
Finally, we directly use this implicitly parameterize policy to generate actions for computing the TD target for our TD loss:
K
2 QB (Sethe "f "ia of )li2r-1)
L(Ok. w) = Qo (00 (8t, at, "
Σ
(28)
K
==1
where af afth""" 4442h-1 ~ ((((81(4)) (8+h))
The baselines BFN-n and BFN are implemented similarly to FQL-n and FQL by operating in the original action space.
C.5 RLPD, RLPD-AC, QC-RLPD
All the RLPD baseline results are obtained by running the official codebase (as linked in Ball et al. [7]) with additional modification to incorporate action chunking and behavior cloning. This baseline runs online RL from scratch using off-policy transitions where 50% of them come from the offline dataset and the other 50% come from the online replay buffer. essentially up-weights the online data more, allowing the online RL agent to learn more quickly. This is different from how QC-* BFN, BFN-n, FQL, FQL-n samples off-policy transitions (where we sample from the dataset that combines the offline dataset and online replay buffer data with no weighting). RLPD baselines all use Gaussian policy. This is also different from our method as our method uses noise-conditioned policy that can represent wider range of distributions. For RLPD-AC, we change all the actor and critic networks such that they work with an action chunk rather than single action. The baseline exactly the same as our method except that actor and the critic are updated the same as how RLPD
28

---

updates its actor and critic. For QC-RLPD, we add a behavior cloning loss in the actor loss as follows (highlighted in red below):
K Qo ((sr,a)) -a log (a)
((v) (x
(29)
K
C.6 SUPE-GT
For this baseline, adapt from a recently proposed skill-based offine-to-onllnee RL method [85]. The original method has additional modules such reward models and random distillation networks to deal with the problem setting where offline dataset is unlabeled (e.g., reward information is missing). We remove these two modules as we have ground truth reward information in our offline dataset. The method has two stages: (I) skill-pretraining and (2) online RL with skills. the first stage, the baseline uses trajectory VAE to learn conectt-ondiiioeed skill policy. In the second stage, the baseline uses RLPD to learn a high-level policy that outputs latent vector to command the latent-conditioned skill policy after every h steps. Such skill-based method in theory also allows temporally coherent actions for exploration and value backup speedup.
D Experiment Details
D.1 Computational resources
We use NVIDIA RTX-A5000 GPU to run our experiments. Each complete offline-to-online exper- iment run takes around 4-7 hours. To reproduce all our results in Table 6, estimate that it would take around
x
25
= 000 GPU hours. Each robomimic
hours per single run
#al methodds
of tasks
# of seeds
experiment run takes longer (around 8-12 hours). While each single run takes longer, we only run subset of strong methods/baselines on robomimic tasks. We estimate it would take around 10
3
X
5
1 350 GPU hours. In total, we estimate that it would
hours per single run
tasks
take around 10 350 GPU hours reproduce all the main results our paper.
D.2 Evaluation protocol
Unless specified otherwise, for all methods, we run 4 seeds on each OGBench task and seeds on each robomimic task. All plots use 95% confidence interval with stratified sampling (5000 samples). The success rate is computed by running the policy in the environment for 50 episodes and record the number of times that the policy succeeds at solving the task (and divide it by 50).
D.3 Hyperparameter tuning
QC, BFN, BFN-n. We tune the number of actions sampled, N, for the expected-maz operator. On OGBench domains, we sweep over {2, 4, 8, 16, 32, 64, 128) and select the best parameter for each domain and for each method on task2. We report the performance of each method with the best in Table and Figure (on all tasks). Table 5 summarizes the a value we use for each task.
QC-FQL, FQL., FQL-n. We tune the behavior regularization coefficient a. On OGBench domains, we take the default hyperparameter of FQL for each domain Odefault and tune all methods on task2 of cach domain with three choices of a: {C /3, Oddefault 30.effault (our Adefault comes from Table 6 in Park et al. [58]). On robomimic domain, we sweep over much large a values: {100, 1000, 10000}. We report the performance of each method with the best in in Table I and Figure (on all tasks). Table 4 summarizes the a value we use for each task.
RLPD, RLPD-AC, QC-RLPD. We sweep over (1) whether or not to use clipped double Q-learning (CDQ), and (2) whether or not to use entropy backup. We find that not using CDQ and not using entropy backup to perform the best for all of the RLPD baselines and use that across all domains.
Even though our method and the other FQL. baselines use = 2 critic ensemble size, we use = 10 critic ensemble size for RLPD to keep it the same the hyperparameter in the original paper [7].
29

---

Parameter
Value
Batch size (M)
256
Discount factor (7)
0.99
Optimizer
Adam
Learning rate
3 10
Target network update rate (r)
5 X 10-3
10 for RLPD, RLPD-AC, QC-RLPD, and SUPE-GT
Critic ensemble size (K)
2 for QC-FQL, FQL, FQL-n, QC-BEN, BEN, BEN-n
UTD Ratio
Number of flow steps (2)
10
Number of offline training steps
10 except RLPD-based approaches (0)
Number of online environment steps
1 10
Network width
512
Network depth
hidden layers
Table 3: Common hyperparameters.
Environments
FQL
FQL-n
QC-FQL
ReBRAC
scene-sparse+
300
100
300
0.1
puzzle- 1orzle--x3-Bee
-x3-sparse-*
100
100
300
0.1
cube-double-*
300
100
300
0.1
cube-triple-*
300
100
100
0.1
uube-quadruple-100k 100M-
300
100
100
0.1
lift
10000
10000
10000
can
10000
10000
10000
square
10000
10000
10000
Table 4: Behavior regularization coefficient (a)).
For QC-RLPD, we sweep over behavior regularization coefficient E 0,0001,0011,.11 and pick 0.01 since it works the best.
SUPE-GT. We tune the KL coefficient for the VAE {0.001, 0.003, 0.01, 0.03, 0.1) and pick 0.003 as it works the best.
skill skill peetrrrett from
ReBRAC. We tune the behavior regularization coefficient from (0.01,0.03, 0.1, 0.3, 1.0) and pick 0.1 as it works the best.
IFQL. We directly use the default hyperparameter from Park et al. [58] (N 32, 0.9) for this baseline. The first parameter, N = 32, means that for both online policy rollout and policy evaluation, sample 32 actions and pick the action that has the highest Q-value. The second parameter, 7 0.9 is the expectile coefficient.
IQL. We directly the default hyperparameter from Park al. [58] for this baseline. For cube-*, we use a = 0.3. For scene and puzzle- 3x33, we use a = 10.0. For the expectile coefficient, we 0.9.
E
Full Results
E.1
End-effector visualization
We provide more examples of the trajectory rollouts from QC and BFN over the course of online training on cube-triple-tasiee
In Figure we show first 9000 time steps (broken down into 9 subplots where each visualizes 1000 time steps). In Figure 9, we show another 9000 time steps but late in the training (from environment step 10*). The first example is the same as the one used in Figure 5.
30

---

Environments
BEN
BFN-n
QC-BFN
scene- spenees
4
32
puzzle 1erlle--33-app 3x3-aparae- cube-*
64
lift, can, square
16
Table 5: Number of actions sampled for the expected-max Q operator (N) for BFN methods.
BNN
B.
Figure 8: End-effector trajectory early in the training. Each subplot above shows the trajectory for consecutive of 1000 time steps. We include up Step 9000.
ac
87H 9C
8PN
6:N
DPM
Figure 9: End-effector trajectory visualization late in the training. Each subplot above shows the trajectory for consecutive of 1000 time steps (from Step 900000 to Step 909000).
E.2
OGBench results by individual task
Main results by task. Table 1 and Figure 10 shows the performance breakdown for all methods.
31

---

REP0
SIPE-0T
T0L.
TPQ)
PQI.
RPM
TPg
Fai.in
0D-41P0
0C-17a
CD-FQI
Q2
taaki
100
15
05
1160
160
1100
55
1300
08
1000
39
-110
1100
85-
1000
110
-100
-100
100
-100
100
15
11
1000
1100
023
1000
016
300
05
1300
1000
1300
72
100
1000
1000
300
1000
Carld
100
100
11
39
100
100
1000
1.00
055
300
05
1100
100
1000
56
62
100
100
160
50
300
300
1600
tarid
100
100
100
0-
01
20
1000
317
1000
1000
100
300
1100
100
1000
37
100
100
100
75
100
100
1000
Carkd
100
100
100
1.00
100
100
1000
01
100
07: -7 100
100
100
100
100
100
1000
org-
Looke?
100
100
1.00
100
1100
100
1000
tanks
100
100
13 1.00
1.00
031
1.00
119
3000
15:
558
52
300
300
160
515
1001
010
3001
3000
1600
100
100
100
0-+
977
1000
195
199
094
110
522
597
112
17:
115
100
011
1001
015
1000
516
1000
6F
1000
300
25
100
100
1000
S0450
05
50
972
90
02-
187
533
95
712
11
91
F1
R8
90
11
73
R1
92
195
85
800
16
70
55
87
86
83
88
84
8-1
task:
70
50
56
100
19
100
60
100
88
70
63
16
1000
100
70
300
83
10
73
38-
018
tarld
17
100
32
tazid
ang.
50
100
87
100
61
18->
1
T-
TI
13
17
11-
3-
70-
16
100
13 100
baakd
1
10 >
16 >
10
10
18
T.
65
TI
cobe-triple
11
10
11
13
75
11
41
11
0-+
1
10
10
16-
10-
0-
0-
2--
10-
task:
10
-
1
38
10
10
01
11
10
10
19
13--
87
11-0
0
11
11
11
T
11
1--
17
sabe-qadrsspee1000 1000
tasbd
13-0
144
1-
83
taskd
Mgg.
77
overall
(25 Luzka)
02
3-
31
08
23
37
30
50
30
50
38
80
32
86
Table 6: Complete OGBench oflinncto-online RL results. 4 seeds. 95% confidence interval.
QC FC
QC-FFL QCALDD
-
FQ
-Q- FOI IFOL
RABRAC IQL
SUFE-GT ALPSAC
-I- RIFD
FQL
BAV
puzzle bask
nuzte-5dd-qaame--arr spare-b222
pustle 2e3-spareeedddd
razzle 3x3 -prteeia--
L3
10
16-
poooosoe La
peoppoos 20-
coogapage
5.b
25
14
25
L3
10
16-
L3
20-
6.
10
15
20
60
10
15
55
Lo
25
3.0
3.-
Surcess Rate
20
65
19
15
28
60
85
16
15
2.
65
15
15
cube- pie
30-
2.0
65
19
15
60
85
25
10
15
2.
65
15
35
cobe-suddip
apequdd pepayy
35
65
05-
65
9000000080
Steps (K10f)
Figure 10: Complete OGBench offline-to-online RL results by task. seeds. 95% confidence interval.
32

---

QC
BFN
FQL
RLPD
IFQL
puzzle-3x3-sparg
scene-sparse
cube-double
1.00
200000000
1.00
0.50
0.30
0.50
0.25
0.25
0.25
BooO
Success Rate
0.00
0.00
0.00
0.0
0.5
2.0
0.5
1.0
2.0
0.0
1.0
1.5
cube-triple
qube-quadruple
1,00
1.00
1.00
ooo000o00
ooo000
00000000
pe0
0.25
0.25
0.00
0.00
-29.0999999
2990999999
0.00
2.0
1.0
1.0
1.5
Steps (x10*)
SN
FQL
RUO
puztle spart-aant
10
16
13
16
055
26
20
23
65
18
15
28
60
35
10
0.5
16
00
65
13
15
23
133
10
16.
10-
10-
8.
16
20
15
28
35
10
16
65
15
double
13
2.0
16-
10-
Succers faen
20
65
18
15
28
60
35
10
0.5
26
00
65
13
15
1.4
20
65
15
20
60
35
15
16
26
35
4.
2.0
65
18
35
55
15
20
03
25
13
35
Steps (x30F)
Figure 11: Full OGBench results by task (selected baselines). Top: summary plots by domain; Bottom: individual plots by task.
Selected baselines. Figure 11 shows the results for selected baselines (e.g., BFN, FQL, RLPD and RLPD-AC).
n-step return ablation results by task. Figure 12 shows the performance breakdown for Figure 4.
Q-chunking with Gaussian policies. The following plot shows the performance breakdown for Figure In addition, we include a new method for comparison, QC-RLPD, where we add a behavior cloning loss to RLPD-AC (RLPD with action chunking).
33

---

RC
BFN
punte behaamaaedd
35-
65
25-
05-
coete-ppa
scene trame-laad
Succees Datte
00
00
cubetrp wta.rrpppplyeeee..... uokd
cule cuce-bisle slay lost2
cabo www.pppaaa.....o...... lodk
cube Mistl play toki
cubuidiiee slay lasch
00
cabe-quede uusks
culo tust-
cabo cuod wadkd
cubl www.c-uudusspleee took
C.bs ind--ooainp
0.0
00
2.0
Figure 12: Full OGBench results by task (selected baselines).
RC-RLPD
cube pece-dduubeppppppe
cube double task
cube-double play-
oube poyylel
cube double playleeppyyttt tasks
1.0
00000000000000
1.0
2.0
10
10
0.5
0.5
05
0.0
0.0-6
5.0
0.
00-
0.3
02
04
05
38
1.5
0.0
0.4
86
08
1.0
0.
022
24
0.6
08
1.
00
04
56
08
140
5.0
02
34
056
08
1.
cube-tripeeppaayttte plast
aube pelo-brpplepply-taaae
cube-triple- cube-tripleplay-taast task
pibeetrrple-pla--takkk
cube-triple-pay-taatt
Success Rate
06
de
cube-quadripletaskk
tube
cube-quedruple-tadee
cube-quadniple-tats
3.0
0.0
02
5.
0.0
00
1.0
0.2
0.8
0.0
0.4
06
02
0.6
Online Steps (x10%)
Figure 13: Full RLPD results by task. QC-RLPD is RLPD-AC (RLPD on the temporally extended action space) where additionally add fixed behavior cloning coefficient of 0.01.
Action chunking size ablations. Figure 14 includes individual task results for the action chunking size ablation study. We also include results on the hardest cube Sabee-quue queeruuuaaueeeee task in Figure 15.
Rd.
CC-OL Fooi
OC-CC 11-02
COHO0 Pridl
0C404 Jodi
2040.17-58
Cale-
Supprxich
Figure 14: Action chunking size ablation on generally helps until 25. Surprisingly, h 25 the only chunk size where QC-FQL achieves non-trivial success on the hardest
Increasing the action chunking size
(none of other methods can).
34

---

QC-FQL (h=9)
QC-FQL (h=7) - QC-FQL (h=5)
QC-FQL (h=3) - FQL
FQL-n (n=9)
FQL-n (n=7)
FQL-n (n=5)
FQL-n (n=3)
cube-quadrrulle-00M-ttakk
1.0
Success Rate
0.8
0.6
0.4
0.2
0.0
0.0
0.5
1.0
1.5
2.0
Steps (x109)
Figure 15: Action chunking size ablation on cube--uaddrplle-pllay-aaass The n-step return baselines (i.e., FQL-n) can achieve good initial success during online learning but quickly collapse. In contrast, our method (i.e., QC-FQL) does not suffer from such collapse and solves the task consistently across reasonably large chunk sizes (e.g., (5, 7, 7.))). The results are over 5 seeds.
E.3 Robomimic ablation results
Figure 16 shows the performance of QC, QC-FQL, BFN-n, FQL-n, BEN, FQL our three robomimic tasks. This plot shows the performance breakdown for Figure (right).
QC
QC-FQL
BFN-n
FQL-n
BFN
FQL
lift
square
1.00
1.00
1.00
0000
Boonngo
Success Rate
0.50
0.25
0.25
0.25
0.00
0.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Steps (x10*)
Figure 16: Full robomimic ablation by task. For each method on each task, we use 5 seeds.
E.4 How resource efficient is Q-chunking?
In Figure 17, we report the runtime for our approach and our baselines on a representative task cube-tripp tew-trr-ppllakkkl In general, QC-FQL has comparable run-time as our basclines (e.g.. FQL and RLPD) for both offline and online. QC is slower for offline training as requires sampling 32 actions for each training example for the agent update (BFN is faster because it only needs sample 4 actions). For online training, we are doing one gradient update per environment step, and makes expensive than other methods.
QC only around 50%
Finally, we include the parameter count of each of these method on the representative domain cube- trii-e-ppe*** in Table 7 below (assuming h 5 for all Q-chunking methods).
35

---

Methods
Parameter Count (in millions)
QC BFN QC-FQL FQL RLPD
2 4.2
R 4.1
R 5.0
R 4.9
17.2
Table 7: Parameter count for each method. RLPD has much larger parameter count because it uses K critic networks whereas all the other methods and baselines only K = 2.
10
Offline Run-time (4)
Online Run-time (4)
QC
12.02
QC
15.07
QC-FQL
11.33
QC-FQL
4.24
BFN
11.59
BFN
3.33
FQL
10.06
FQL
2.97
RLPD
11.03
Figure 17: How long does each method take for one step milliseconds. Left: offline. Right: online (one agent training step and environment step). The runtime is measured using the default hyperparameters in our paper sube-triple-taske single RTX-A5000.
36


Reviews:
[{"review_id": "XUJPM2p7Nz", "reviewer": "Reviewer_nCbd", "summary": "This paper presents an actor-critic method for offline-to-online reinforcement learning. The proposed method, Q-learning with Action Chunking (Q-LAC), uses action chunks or macro-actions for learning (macro) action-values and acting in the environment. \n\nThe authors demonstrate that using action chunks with a relatively more straightforward actor-critic approach (RLPD) can lead to a deterioration in performance. Instead, they suggest building on top of Flow Q-learning (FQL), where action chunks can be constrained to be similar to the behavior observed in the offline data.\n\nExperiments in the offline-to-online setting show that the proposed approach outperforms prior methods (RLPD and FQL) on the more challenging domains from the OG-bench.", "questions": "Included in “Strengths and Weaknesses”", "limitations": "Limitations are touched in Discussions section, but could be elaborated upon.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The paper presents a relatively simple extension of FQL to incorporate temporally extended actions. \n- Further, this extension (Q-LAC) shows promising results on the OG-bench domains considered. In particular, it performs significantly better than relevant baselines (FQL, RLPD) on the more challenging tasks, where a long horizon and sparse reward pose the greatest challenge.\n\n### Weaknesses\n\n**W1.** A weakness of the present submission is in terms of novelty. Q-LAC feels like an incremental contribution on top of FQL. Perhaps more problematically, this connection is not really addressed before sections 4.2 and 5, even after which the exact relationship remains unclear. Could the authors clearly specify the differences between FQL and Q-LAC? Using the terminology of the paper, would it be correct to say Q-LAC = FQL-AC5?\n\n**W2.** Also related to novelty, the benefits of macro-actions in terms of more temporally coherent exploration and improved credit assignment are generally known (e.g. [1, 2, 3]), limiting the novelty of section 4.3. Lines 35 and 36 claim that chunking has not been widely used in RL; however, it has been employed through macro-actions (as noted in previous references) and for credit assignment [4]. \n\n**W3.** There are some aspects of the empirical analysis that would benefit from clarification.\n\nThe authors consider an FQL-n baseline that uses (biased) n-step returns. Due to the difference of $n$ for backup vs action chunking, it would also be helpful to see the performance of **FQL-n** for different choices of $n$, e.g., $n \\in \\{3, 5, 7, 9 \\}$. Could a different choice of $n$ match the performance of Q-LAC? FQL-5 also starts off quite well on cube-quadruple-100M, is there a reason for the sudden collapse?\n\nAnother question I had was regarding RLPD vs FQL. The FQL paper’s results show that FQL outperforms RLPD in the offline-to-online case, whereas the results in this paper (cube-double, cube-triple, robomimic from Figure 4, and aggregated results from Figure 1) show the opposite trend. Could the authors explain why this is the case?\n\n—-----------—-----------—-----------—-----------—-----------\n\nOn a minor note, it would be helpful to include some more details about FQL in the main paper’s background section or section 4.2. Perhaps sections 4.3, 5.1, and 5.2 could be compressed, and some of those details could be moved to the appendices.\n\nOverall, I appreciate the simplicity of the proposed method and the promising results (strengths), but I think the paper should adjust some claims related to novelty and further discuss related work (W1/2). I also have concerns related to the empirical analysis (W3).\n\nI remain open to increasing the score if the weaknesses and questions are adequately addressed and clarified.\n\nTypos: In the legend of Figure 7. D^comp instead of D^comb.\n\n#### References\n\n[1] McGovern, Amy, and Richard S. Sutton. \"Macro-actions in reinforcement learning: An empirical analysis.\" (1998)\n\n[2] Durugkar, Ishan P., et al. \"Deep reinforcement learning with macro-actions.\" arXiv preprint arXiv:1606.04615 (2016).\n\n[3] Tan, Aaron Hao, et al. \"Deep reinforcement learning for decentralized multi-robot exploration with macro actions.\" IEEE Robotics and Automation Letters 8.1 (2022)\n\n[4] Ramesh, Aditya, et al. \"Sequence compression speeds up credit assignment in reinforcement learning.\" ICML (2024).", "was_revised": true, "final_justification": "Please see my response to the authors:\n\nI would like to thank the authors for their detailed response. I greatly appreciate the time and effort spent to conduct the additional experiments. The response addresses my main concerns and I have increased my rating to \"accept\".\n\nIt's great to see that action chunking can be combined with other offline-to-online approaches. Similarly, Q-LAC performing better than FQL-n for other choices of n also supports the benefit of action chunking/Q-LAC.\n\nThe remaining criticism pertains to the novelty of the approach being low (action chunking + FQL). The authors point out that introducing action-chunking to work with these approaches is not straightforward as it doesn't work with RLPD and needs behavioral constraints and an expressive policy. In my opinion, the fact that action chunking didn't work with RLPD is enough for the current submission to be interesting.", "rebuttal": "Thanks for your detailed review and insightful comments. We addressed your main concern on **(W3) empirical analysis** by conducting a more comprehensive set of experiments **(11 baselines and 25 tasks in total)** and additional ablation studies requested by the reviewer on FQL-n, **(W1-W2) novelty** by clarifying the key principles behind our method Q-LAC that differentiate us from prior offline-to-online RL methods (e.g., FQL) and skill-based/macro-action-based methods. \n\n>## New results! (3 Q-LAC methods + 11 baselines on 25 OGBench tasks)\n\nThe newly experiments are summarized below (3 seeds for SUPE-GT and RLPD-AC-BC and 4 seeds for everything else). We report the success rate (%) after 1M offline training steps followed by 1M additional steps of online fine-tuning (offline $\\rightarrow$ online). Our methods, Q-LAC and Q-LAC-BFN, outperform other baselines by a large margin across the board. \n\nFor each row, we bold each number if its confidence interval overlaps with the best method’s confidence interval. Due to the character limit, we omit the confidence interval in the table and the performance on the individual task, but we are happy to provide them during the discussion period.\n\n| |Q-LAC (ours)|FQL|Q-LAC-BFN (ours)|BFN|Q-LAC-IFQL (ours)|IFQL|FQL-n|BFN-n|RLPD|RLPD-AC|RLPD-AC-BC|ReBRAC|IQL|SUPE-GT|\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n|puzzle-3x3-sparse (5 tasks)|$63$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}100}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}100}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}98}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}100}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}98}$→$97$|$\\mathbf{\\color{blue}98}$→$\\mathbf{\\color{purple}100}$|$58$→$91$|$0$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}100}$|$55$→$\\mathbf{\\color{purple}100}$|$0$→$20$|$0$→$\\mathbf{\\color{purple}100}$|\n|scene-sparse (5 tasks)|$\\mathbf{\\color{blue}84}$→$\\mathbf{\\color{purple}99}$|$58$→$95$|$\\mathbf{\\color{blue}84}$→$\\mathbf{\\color{purple}99}$|$\\mathbf{\\color{blue}85}$→$\\mathbf{\\color{purple}99}$|$\\mathbf{\\color{blue}83}$→$\\mathbf{\\color{purple}98}$|$68$→$73$|$18$→$70$|$57$→$\\mathbf{\\color{purple}97}$|$0$→$94$|$0$→$91$|$0$→$96$|$11$→$\\mathbf{\\color{purple}99}$|$0$→$39$|$0$→$\\mathbf{\\color{purple}92}$|\n|cube-double (5 tasks)|$39$→$\\mathbf{\\color{purple}100}$|$29$→$76$|$\\mathbf{\\color{blue}67}$→$\\mathbf{\\color{purple}98}$|$\\mathbf{\\color{blue}68}$→$79$|$12$→$79$|$11$→$59$|$11$→$77$|$11$→$65$|$0$→$\\mathbf{\\color{purple}98}$|$0$→$96$|$0$→$97$|$3$→$30$|$0$→$0$|$0$→$67$|\n|cube-triple (5 tasks)|$\\mathbf{\\color{blue}4}$→$\\mathbf{\\color{purple}54}$|$1$→$18$|$\\mathbf{\\color{blue}6}$→$\\mathbf{\\color{purple}64}$|$\\mathbf{\\color{blue}4}$→$23$|$0$→$0$|$0$→$3$|$0$→$1$|$0$→$0$|$0$→$41$|$0$→$11$|$0$→$11$|$0$→$0$|$0$→$0$|$0$→$0$|\n|cube-quadruple-100M (5 tasks)|$1$→$\\mathbf{\\color{purple}77}$|$0$→$3$|$\\mathbf{\\color{blue}4}$→$73$|$1$→$12$|$1$→$19$|$0$→$0$|$\\mathbf{\\color{blue}7}$→$37$|$0$→$0$|$0$→$0$|$0$→$7$|$0$→$11$|$1$→$20$|$0$→$0$|$0$→$0$|\n|aggregated (25 tasks)|$38$→$\\mathbf{\\color{purple}86}$|$37$→$58$|$\\mathbf{\\color{blue}52}$→$\\mathbf{\\color{purple}86}$|$\\mathbf{\\color{blue}51}$→$63$|$39$→$59$|$36$→$47$|$27$→$57$|$25$→$50$|$0$→$67$|$0$→$61$|$0$→$64$|$14$→$50$|$0$→$12$|$0$→$52$|\n\n*Note: for ReBRAC, IQL and IFQL, we use the same implementation and the same hyperparameter from the official FQL codebase. SUPE-GT is a baseline in our submission. We additionally ran it on all 25 OGBench tasks.*\n\n>## Novelty / What are the differences between FQL and Q-LAC?\n\nThe main difference between FQL and Q-LAC lies in the use of chunking in the actor and the critic. While this distinction might seem simple, it results in significant empirical improvements (see our new results above). For example, FQL fails to solve all but one task in the cube-triple and cube-quadruple domains. In contrast, Q-LAC achieves ~53% average success rate for cube-triple and ~77% average success rate for cube-quadruple.\n\nGetting action chunking to work in RL is not trivial. As also shown in the table, naïvely applying action chunking to standard online RL (RLPD) methods does not lead to good performance (RLPD-AC, RLPD-AC-BC). All RLPD approaches obtain low success rate (~10%) on the hardest domain (cube-quadruple). In our paper, we identify two principles that make action chunking work in RL: (1) learn both chunked policy and critic and perform TD-learning on the temporally extended action space (Section 4.1); (2) use expressive policy with behavior constraints (Section 4.2).\n\nFor this rebuttal, we show that these principles generalize beyond Q-LAC (based on FQL). We include two variants of Q-LAC, Q-LAC-BFN and Q-LAC-IFQL, a result of applying the same principles to two new offline-to-online RL baselines (BFN and IFQL, see below for more details on these two base algorithms). As shown in the table above, action chunking consistently improves the non-chunking counterpart (e.g., Q-LAC vs. FQL, Q-LAC-BFN vs. BFN, Q-LAC-IFQL vs. IFQL), supporting our claim that introducing action chunking into RL leads to improved performance. \n\nWe now describe the base algorithms of Q-LAC-BFN and Q-LAC-IFQL:\n1. **BFN** is built on top of an existing algorithm EMaQ [1] by replacing its behavior policy with a more expressive/modern flow BC policy. We call it best-of-n (BFN) because it works by sampling multiple actions from the BC policy and then picking the action that maximizes the critic value for online exploration and the TD target computation (see below for a detailed algorithm description). This type of implicit policy extraction is also used in IDQL [2] with the difference being that IDQL only uses it for evaluation and online exploration, and does not use it for TD target computation because it decouples value learning and policy learning. Despite it being extremely simple to implement, BFN is on par with FQL in our experiments.\n2. **IFQL** is an implementation of IDQL where the diffusion policy is replaced by a flow policy. It uses the IQL expectile loss to train Q-function and value-function. It also uses best-of-n sampling, but only for evaluation and online exploration. We use the implementation from the official FQL codebase.\n\n[1] “EMaQ: Expected-max q-learning operator for simple yet effective offline and online rl.\" \n\n[2] “IDQL: Implicit q-learning as an actor-critic method with diffusion policies.\"\n\n>## “Could a different choice of n [for FQL-n] match the performance of Q-LAC?”\n\nThis is a great question! The short answer: No. \n\nWe conducted new experiments with different choices of $n$ on cube-quadruple-task4 (one of the hardest tasks that we consider) where we report the performance at the end of the offline training, online at 200K steps, online at 500K steps, and online at 1M steps respectively (separated by the arrow. e.g., offline $\\rightarrow$ online@200K $\\rightarrow$ online@500K $\\rightarrow$ online@1M).\n\n|cube-quadruple-100M-task4|$h=9$|$h=7$|$h=5$ (our main results)|$h=3$|$h=1$|\n|---|---|---|---|---|---|\n|Q-LAC (with chunking)|$0$→$\\mathbf{\\color{brown}95}$→$\\mathbf{\\color{purple}98}$→$\\mathbf{\\color{magenta}97}$|$0$→$\\mathbf{\\color{brown}88}$→$\\mathbf{\\color{purple}94}$→$\\mathbf{\\color{magenta}93}$|$0$→$85$→$\\mathbf{\\color{purple}92}$→$\\mathbf{\\color{magenta}95}$|$0$→$0$→$0$→$0$|$0$→$0$→$0$→$0$|\n|FQL-n (without chunking)|$\\mathbf{\\color{blue}4}$→$8$→$2$→$3$|$\\mathbf{\\color{blue}4}$→$54$→$29$→$19$|$\\mathbf{\\color{blue}1}$→$46$→$24$→$13$|$0$→$1$→$0$→$0$|$0$→$0$→$0$→$0$|\n\nWhen $h=5$, we see a collapse in FQL-5 while Q-LAC continues to learn. This might be due to the biased value backup that FQL-5 uses, whereas Q-LAC is unbiased (see more discussions in Section 4.3, Benefit #1). When we increase the horizon ($h=7$, $h=9$), Q-LAC gets even more sample efficient online while the collapse is worse for FQL-7 and FQL-9. This demonstrates Q-LAC’s better scaling capability with respect to the horizon length. \n\n>## Additional references for macro-actions\n\nThanks for the additional references! Macro-actions are semantically equivalent to action chunks indeed and we will include the discussion in our paper and adjust our claims accordingly. However, our work leverages such temporally extended action differently from prior work [1,2]. These works leverage a discrete set of macro-actions (action sequences) that are either heuristically extracted from the prior experience or given in advance and a policy can choose between these macro-actions. In contrast, we use an expressive flow-matching based policy to directly parameterize a continuous space of action sequences, and are directly trained and fine-tuned online with RL. Conceptually, how prior work uses macro-actions is similar to those that use skills, options which we discuss in more detail in Section 2.\n\n[1] \"Deep reinforcement learning with macro-actions.\"\n\n[2] \"Deep reinforcement learning for decentralized multi-robot exploration with macro actions.\" \n\n>## In the FQL paper, RLPD underperforms FQL. In our submission, FQL underperforms RLPD. Why is that the case?\n\nThe FQL paper uses custom RLPD implementation which we have found to underperform the official implementation, so we used the official RLPD codebase instead. In addition, FQL uses 2 critics rather than 10 critics as used in the original RLPD codebase. To stay faithful to the original RLPD design, we maintained the use of 10 critics in our runs. \n\n>## Writing feedback\n\nWe have restructured Section 4 to include more details about FQL as well as better motivate the principles behind our method. We will also fix the typo you pointed out in our next revision.\n\nThank you again for your constructive feedback and detailed reviews. Your feedback has helped us improve our paper a lot. Thank you!! **Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?**"}, {"review_id": "SUiyYeAkxZ", "reviewer": "Reviewer_kQoc", "summary": "This paper proposes a method integrating action chunking within an Actor-Critic framework. ​The approach demonstrates improvements​ in value backup efficiency for accelerated learning while generating temporally coherent actions to enable effective online exploration. ​Experimental results provide evidence​ for the method's sample efficiency compared to baselines, notably in offline-to-online RL settings.", "questions": "1. Why does the action chunking formulation yield limited performance improvements in strongly Markovian or fully Markovian situation?\n\n2. Equation 8 claims that action chunking avoids bias in value estimation. What is the associated impact on estimation variance, and how does this affect the stability and convergence of the learning process?\n\n3. How does Q-LAC scale to environments with high-dimensional action spaces? Does effective Bellman updating with chunked high-dimensional inputs necessitate more complex Q-network architectures (e.g., deeper networks, attention mechanisms), and what computational trade-offs arise?", "limitations": "Yes, the author has included limitation discussion in appendix.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Pros**:\n1. The paper is well-structured and effectively presents the main idea.\n2. Integrating the action chunking into both critic learning and policy optimization process demonstrate its effectiveness on improving the sample efficiency especially in the online learning part.\n\n**Cons:**\n1. The core contribution appears largely incremental, primarily combining action chunking with FQL. Rigorous ablation studies are needed to isolate the individual efficacy of each design component. (e.g., Does action chunking generalize as a performance enhancer when integrated with ​​other​​ offline-to-online RL algorithms?)\n\n2. The computational demands appear significant, yet evaluation was restricted to limited tasks per OGbench environment. Comprehensive performance metrics (e.g., aggregated scores across all tasks) are absent, precluding a robust assessment of Q-LAC's overall effectiveness.\n\n3. It seems like in offline \"scene\" task results Q-LAC exhibits the highest variance among baselines, potentially undermined its online performance, yet this critical limitation lacks sufficient analysis regarding its cause and impact.\n\n3. The manuscript lacks substantive discussion or illustration explaining why action chunking is particularly well-suited to non-Markovian settings or adequately addresses potential side effects.\n\n4. Formal theoretical analysis is absent. Such analysis would significantly strengthen claims regarding the method's effectiveness and robustness.", "was_revised": true, "final_justification": "My final justification for the recommended score is 4. \n\nThe paper's primary contribution is introducing an effective action chunking mechanism to RL, thoroughly demonstrating its high efficacy in boosting sample efficiency and ease of deployment.  One limitation is that the idea of action chunking has been explored in prior work, which diminishes its novelty.", "rebuttal": "Thanks for the insightful feedback and detailed review. We discuss the novelty of our contribution below. Our main contribution is to **introduce effective action chunking into RL**, and while this modification might seem simple, our experiments show that it is **highly impactful**. Furthermore, during the rebuttal, we additionally show that **the same novel contribution** can be applied to **multiple different RL algorithms**, further supporting our claim that introducing action chunking into RL leads to improved performance. **Please see our response to nCbd for the summary table of our new results.**\n\n>## “The core contribution appears largely incremental”\n\nOur main contribution is to introduce effective action chunking into RL, and while this modification might seem simple, our experiments show that it is highly impactful. For example, FQL fails to solve all but one task in cube-triple and cube-quadruple domains. In contrast, Q-LAC solves 4 out of 5 tasks on both domains. In addition, getting action chunking to work in RL is not trivial. Furthermore, during the rebuttal, we additionally show that the same novel contribution can be applied to multiple different RL algorithms (Q-LAC [FQL], Q-LAC-BFN [BFN], Q-LAC-IFQL [IFQL]), further supporting our claim that introducing action chunking into RL leads to improved performance.\n\n>## “Does action chunking generalize as a performance enhancer when integrated with ​​other​​ offline-to-online RL algorithms?”\n\nYes it does! For this rebuttal, we include two variants of Q-LAC, Q-LAC-BFN, a result of applying the same principles to a new offline-to-online RL algorithm and Q-LAC-IFQL, a result of applying the same principles to a offline-to-online RL baseline IFQL (as considered in the FQL paper). Action chunking consistently improves the non-chunking counterpart (e.g., Q-LAC vs. FQL, Q-LAC-BFN vs. BFN, Q-LAC-IFQL vs. IFQL), supporting our claim that introducing action chunking into RL leads to improved performance. **Please see the summary of our new results in our response to Reviewer nCbd.**\n\n>## “Comprehensive performance metrics (e.g., aggregated scores across all tasks) are absent”\n\nOur summary table (see our response to Reviewer nCbd) now contains the aggregated scores across all tasks. \n\n>## “scene task results Q-LAC exhibits the highest variance among baselines, potentially undermined its online performance.”\n\nWe have fixed this in our newest experiments where Q-LAC can succeed 100% for all scene and puzzle-3x3 tasks online. In general, we have found that the original OGBench scene and puzzle-3x3 domains have high variances because the success rate is misaligned with the reward function (a shaped reward function that provides some reward for each sub-task finished). In some cases, succeeding in a task requires the agent to un-finish some of the sub-tasks. To eliminate any confounding factor from poor reward shaping, in all our new experiments, we use a sparse-reward function (-1/0) for both scene and puzzle-3x3 domains where a reward of -1 is given as long as the overall task has not been completed (regardless of the sub-task completion). With this modification, all the methods exhibit a much smaller variance. In addition, we now test on all five tasks rather than only two tasks for each domain. \n\n>## “The manuscript lacks substantive discussion or illustration explaining why action chunking is particularly well-suited to non-Markovian settings.”\n\nTo address your concern, we analyze the temporal coherency of the actions generated by Q-LAC and FQL. In particular, we look at the distance between the end-effector positions five time steps apart from our Q-LAC policy compared to FQL policy. A higher distance suggests a higher temporal coherency in the actions taken by Q-LAC and FQL policy. Please see our response to joyS (\"How does action chunking policy’s actions deviate from non-chunked policy?\") for the result table. As shown in the table, Q-LAC exhibits a higher temporal coherency, allowing it to explore in the environment better, and better online sample efficiency\n\n>## “Formal theoretical analysis is absent. Such analysis would significantly strengthen claims regarding the method's effectiveness and robustness.”\n\nFor theoretical rigor, we added a theorem that shows our method performs unbiased n-step return backup. **Please refer to our response to Reviewer joyS for more details.**\n\n>## “Why does the action chunking formulation yield limited performance improvements in strongly Markovian or fully Markovian situation?”\n\nWe are unsure how to best interpret this question as it is unclear to us which limited performed improvements that you are referring to. Here is our interpretation – “why does action chunking yields limited performance improvement on scene and puzzle-3x3”. In our new experiments (see our response to nCbd), we found that most baselines already perform quite well on both scene and puzzle-3x3 at around 100% success rate for most of the tasks. We believe that the limited performance improvements are due to the fact that these two domains are already saturated by existing algorithms. \nIf we interpret it wrong, could you please clarify the question?\n\n>## “Equation 8 claims that action chunking avoids bias in value estimation. What is the associated impact on estimation variance, and how does this affect the stability and convergence of the learning process?”\n\nTo address your question, we analyzed the convergence of the average Q-value of the state-action pair in the dataset over the course of offline training for Q-LAC, FQL-n and FQL on one of our hardest tasks, cube-quadruple-100M-task2 (averaged over 4 seeds). We compute the difference in Q-value from the previous time step and highlight the difference in red if it is greater than 10.\n| $\\mathbb{E}_{(s, a) \\sim \\mathcal{D}} \\left[Q(s, a)\\right]$ |Q-LAC | FQL-n | FQL |\n| --- |--- | --- | --- |\n| Step 5000 | $-321.2$ | $-377.2$ | $-270.1$ |\n| Step 55000 | $-270.4$ ($\\Delta = \\color{red}50.8$) | $-350.3$ ($\\Delta = \\color{red}26.9$) | $-382.8$ ($\\Delta = \\color{red}-112.7$) |\n| Step 105000 | $-262.7$ ($\\Delta = 7.7$) | $-322.4$ ($\\Delta = \\color{red}27.9$) | $-385.9$ ($\\Delta = -3.1$) |\n| Step 155000 | $-253.2$ ($\\Delta = \\color{red}9.5$) | $-317.0$ ($\\Delta = 5.3$) | $-377.0$ ($\\Delta = \\color{red}8.9$) |\n| Step 205000 | $-249.4$ ($\\Delta = 3.8$) | $-316.5$ ($\\Delta = 0.5$) | $-362.0$ ($\\Delta = \\color{red}15.0$) |\n| Step 255000 | $-247.0$ ($\\Delta = 2.4$) | $-314.8$ ($\\Delta = 1.7$) | $-345.7$ ($\\Delta = \\color{red}16.3$) |\n| Step 305000 | $-249.1$ ($\\Delta = -2.1$) | $-313.9$ ($\\Delta = 0.9$) | $-335.5$ ($\\Delta = \\color{red}10.1$) |\n| Step 355000 | $-245.4$ ($\\Delta = 3.7$) | $-311.9$ ($\\Delta = 2.0$) | $-323.7$ ($\\Delta = \\color{red}11.8$) |\n| Step 405000 | $-247.9$ ($\\Delta = -2.5$) | $-312.2$ ($\\Delta = -0.3$) | $-314.8$ ($\\Delta = \\color{red}9.0$) |\n| Step 455000 | $-246.8$ ($\\Delta = 1.1$) | $-310.4$ ($\\Delta = 1.8$) | $-310.4$ ($\\Delta = 4.4$) |\n| Step 505000 | $-249.9$ ($\\Delta = -3.1$) | $-310.4$ ($\\Delta = 0.1$) | $-308.0$ ($\\Delta = 2.5$) |\n| Step 555000 | $-246.0$ ($\\Delta = 3.8$) | $-308.4$ ($\\Delta = 1.9$) | $-305.0$ ($\\Delta = 3.0$) |\n| Step 605000 | $-249.3$ ($\\Delta = -3.3$) | $-308.7$ ($\\Delta = -0.3$) | $-305.2$ ($\\Delta = -0.2$) |\n| Step 655000 | $-245.2$ ($\\Delta = 4.1$) | $-307.2$ ($\\Delta = 1.5$) | $-302.3$ ($\\Delta = 2.9$) |\n| Step 705000 | $-247.4$ ($\\Delta = -2.1$) | $-307.7$ ($\\Delta = -0.5$) | $-296.7$ ($\\Delta = 5.6$) |\n| Step 755000 | $-245.2$ ($\\Delta = 2.2$) | $-306.4$ ($\\Delta = 1.4$) | $-294.2$ ($\\Delta = 2.5$) |\n| Step 805000 | $-249.0$ ($\\Delta = -3.9$) | $-306.8$ ($\\Delta = -0.4$) | $-299.0$ ($\\Delta = -4.8$) |\n\nBoth Q-LAC and FQL-n (FQL with n-step returns) have large Q-value differences in the first 100K to 150K steps and then have small Q-value differences for the rest of the offline training. In contrast, FQL has large Q-value differences for a longer time up to almost half way through the offline training (~400K steps). This suggests that both Q-LAC and FQL-n enjoy a faster convergence of the Q-value compared to FQL, providing evidence that n-step backup does accelerate value learning. It is worth highlighting that while this quantifies the value learning speed, it does not quantify how well the value is learned. In fact, in our experiments, even though FQL-n exhibits a faster value learning, it suffers from an online collapse problem (as shown here), possibly due to the bias in the n-step backup. In contrast, Q-LAC does not suffer from this issue, possibly due to its unbiased n-step backup (see more discussion in Section 4.3 Benefit #1).\n\n>## “How does Q-LAC scale to environments with high-dimensional action spaces? Does effective Bellman updating with chunked high-dimensional inputs necessitate more complex Q-network architectures, and what computational trade-offs arise?”\n\nWhile it is possible that the chunked action space requires a more complex Q-network, in our experiments, we find that using the same architecture (4 hidden layers with 512 units each) as the non-chunked FQL is sufficient to solve all our tasks. \nTo provide more insights for the scalability of our method, we analyzed the time and memory that action chunking brings and found that action chunking incurs almost no overhead on CPU & GPU memory and some but manageable overhead on the computational cost especially for online learning where environment steps are still the main bottleneck. This suggests that Q-LAC can be easily applied to existing RL algorithms without requiring large overhead on memory and time, making it scalable for more complex problems. **Please respond to Reviewer joyS for more details on the memory and time overhead of action chunking.**\n\nThank you again for your constructive feedback and detailed reviews. **Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?**"}, {"review_id": "68PV8sWhjD", "reviewer": "Reviewer_joyS", "summary": "This paper proposes Q-LAC, an offline to online RL algorithm that employs action chunking to accelerate temporal-difference value updates and promote temporally coherent exploration. Action chunking is a common algorithm in the field of embodied intelligence, and it is highly valuable to study its role in the domain of reinforcement learning. Empirical evaluations across a suite of challenging long-horizon, sparse-reward manipulation tasks demonstrate that Q-LAC outperforms prior approaches in both offline and online RL phases, particularly excelling in sample efficiency during fine-tuning.", "questions": "Please see weaknesses.", "limitations": "The topic of the article is highly valuable, yet many aspects could be explored more profoundly. Should the authors address my concerns, I would be more than willing to consider a higher rating.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- The motivation behind this article is excellent. It is highly meaningful to investigate the role of action chunking, which is commonly used in embodied intelligence algorithms, within the realm of reinforcement learning. \n\n- The proposed algorithm is concise and effective, with strong versatility.\n\n\n**Weaknesses** : The overall discussion and analysis in this article are not profound enough, and there are many points that could be expanded upon.\n\n- There is a lack of theoretical analysis regarding the impact of action chunking on online RL. Additionally, while the speedup and unbiased value backup claims are explained in Section 4.3, no formal analysis or bounds are provided.\n\n- When the action chunking size is 5, how much deviation is there between the five actions obtained by $a_{1,2,3,4,5} = \\pi_{ac}(s_1)$ and the five actions obtained by $a_{i} = \\pi(s_{i})$? How does this deviation affect the final performance and the two benefits?\n\n- The article mentions that the learning of value is accelerated, but there is no quantified indicator or experiment to measure the extent of this acceleration.\n\n- The ablation experiment on action chunking size is incomplete. Intuitively, as the size increases, the subsequent actions will become less accurate. Therefore, finding this critical point is crucial.\n\n- The article lacks analysis on whether action chunking will bring additional memory or time overhead.", "was_revised": true, "final_justification": "Considering that most of the concerns I raised have been addressed, I have accordingly increased my score from 3 to 4.", "rebuttal": "Thanks for your detailed review and insightful comments. We addressed your concern on the lack of analysis and discussion by adding **(1) theoretical results** that show Q-LAC performs unbiased n-step return backup, **(2) empirical analysis** that shows Q-LAC’s policy exhibit better **temporal coherency** compared to the non-chunked baseline, highlighting the effectiveness of action chunking for online exploration, **(3) empirical analysis** that quantifies the **value speedup effects of n-step returns**, (4) a more **complete ablation study for the effect of action chunk size** where we show when a too big of an action chunk size starts to hurt performance, and **(5) a detailed analysis of the CPU/GPU memory overhead and computational cost overhead** of our action chunking method Q-LAC.\n\n> ## “No formal analysis is provided about unbiased value backup”\n\nTo address your concern, we have included a theorem with proof below that shows the value backup for our chunked Q-function is unbiased. \n\n### [Theorem: Q-LAC performs unbiased n-step backup]\nLet $s_t, a_t, s_{t+1}, a_{t+1}, \\cdots, s_{t+n}$ be a trajectory segment generated by following a data collection policy $\\pi_\\beta(\\cdot | s)$ and $r_t, r_{t+1}, \\cdots r_{t+n-1}$ be the reward received at each corresponding time step. Let $V^\\pi(s_t)$ be the value for a policy $\\pi$ starting from state $s_t$. \nLet $Q^\\pi(s_t, a_t, a_{t+1}, \\cdots, a_{t+n-1})$ be the Q-value for a policy $\\pi$ starting from state $s_t$ and executing the action sequence $a_t, a_{t+1}, \\cdots, a_{t+n-1}$. \n\nUnder any policy $\\pi$ (possibly different from $\\pi_\\beta$), the $n$-step return estimate $\\hat{V}\\_{n} := \\sum_{t'=t}^{t+n-1}\\left[\\gamma^{t'-t}r_{t'}\\right] + \\hat{V}(s_{t+n})$ is unbiased for $Q^\\pi(s_t, a_{t}, \\cdots, a_{t+n-1})$ as long as $\\hat{V}(s_{t+n})$ is unbiased for $V^\\pi(s_{t+n})$. \n\n### [Proof]\nFrom the definition of $Q^\\pi(s_t, a_t, a_{t+1}, \\cdots, a_{t+n-1})$, we can write it as an expectation:\n$Q^\\\\pi(s_t, a_t, a_{t+1}, \\\\cdots, a_{t+n-1}) = \\\\mathbb{E}\\_{s\\_{t'} \\sim T(\\\\cdot | s\\_{t'}, a\\_{t'})} \\\\left[\\\\sum\\_{t'=t}^{t+n-1}\\\\left[\\\\gamma^{t'-t}r(s\\_{t'}, a\\_{t'})\\\\right] + V^\\pi(s\\_{t+n})\\\\right]$,\n\n$= \\\\mathbb{E}\\_{s\\_{t+n}, r\\_{t+1}, r\\_{t+2}, \\cdots, r\\_{t+n-1}}\\\\left[\\\\sum\\_{t'=t}^{t+n-1}\\\\left[\\\\gamma^{t'-t}r\\_{t'}\\\\right] + V^\\pi(s\\_{t+n})\\\\right]$\n\n$= \\\\mathbb{E}\\_{s\\_{t+n}, r\\_{t+1},  r\\_{t+2}, \\cdots, r\\_{t+n-1}}\\left[\\sum\\_{t'=t}^{t+n-1}\\\\left[\\\\gamma^{t'-t}r\\_{t'}\\\\right] + \\\\hat{V}(s\\_{t+n})\\\\right]$\n\n $= \\mathbb{E}\\left[\\hat{V}_{n}\\right]. \\square$\n\nThe second line uses the fact that the rewards $r_t, r_{t+1}, \\cdots, r_{t+h-1}$ are associated with the trajectory segment $s_t, a_t, \\cdots, s_{t+n}$. The third line uses the fact that $\\hat{V}(s_{t+n})$ is unbiased for $V^\\pi(s_{t+n})$. The fourth line uses the definition of $\\hat{V}_{\\mathrm{n\\text{-}step}}$.\n\nThe implication of this theorem is that if we use the n-step return backup for the chunked Q-function $Q(s\\_t, a\\_t, a\\_{t+1}, \\cdots, a\\_{t+n-1})$ (with a value estimate of $\\hat{V}\\_{s} \\\\leftarrow Q(s\\_t,  a\\_t, a\\_{t+1}, \\\\cdots, a\\_{t+n-1})$) to convergence it converges to the ground truth $Q$-value under the same policy $Q^\\pi(s\\_t, a\\_t, a\\_{t+1}, \\\\cdots, a\\_{t+n-1})$. In contrast, the n-step return backup on the original action space is known to not converge to the correct $Q$-value [1,2]. \n\nThis might be unsurprising because we are effectively treating the action chunk as the new action space and run RL with this transformed action space. Logically, if there is any RL algorithm that is guaranteed to converge to the optimal 1-step policy in the original action space, running the same algorithm on this transformed action space will also converge to the optimal action chunking policy. While it is true that the optimal action chunking policy cannot represent the optimal 1-step policy in some cases, in our experiments, we have found them to be sufficient **(see the new results in our response to Reviewer nCbd)**.\n\n[1] \"Revisiting Peng’s Q($λ$) for Modern Reinforcement Learning.\" \n\n[2] \"Revisiting fundamentals of experience replay.\"\n\n> ## How does action chunking policy’s actions deviate from non-chunked policy?\n\nTo address your question, we analyze the temporal coherency of the actions generated by the Q-LAC policy and the FQL policy in the beginning of online fine-tuning on task cube-triple-task2 (4 seeds). In particular, we look at the distance between the end-effector positions five time steps apart from our Q-LAC policy compared to FQL policy (e.g., $\\\\|\\mathbf{x}^{\\\\mathrm{eef}}\\_t - \\\\mathbf{x}^{\\\\mathrm{eef}}\\_{t+5}\\\\|\\_2$). A higher distance suggests a higher temporal coherency in the actions taken by Q-LAC and FQL policy. As shown in the table below, Q-LAC exhibits a higher temporal coherency, allowing it to explore in the environment better, and better online sample efficiency **(see our new results in our response to Reviewer nCbd)**.\n\n| Online Environment Steps |Q-LAC|FQL|\n| --- | --- | --- |\n|Step 5000|$\\\\mathbf{\\\\color{blue}0.070}$|$0.063$|\n|Step 10000|$\\\\mathbf{\\\\color{blue}0.070}$|$0.064$|\n|Step 15000|$\\\\mathbf{\\\\color{blue}0.069}$|$0.065$|\n|Step 20000|$\\\\mathbf{\\\\color{blue}0.069}$|$0.064$|\n|Step 25000|$\\\\mathbf{\\\\color{blue}0.068}$|$0.065$|\n|Step 30000|$\\\\mathbf{\\\\color{blue}0.068}$|$0.063$|\n|Step 35000|$\\\\mathbf{\\\\color{blue}0.067}$|$0.064$|\n|Step 40000|$\\\\mathbf{\\\\color{blue}0.067}$|$0.064$|\n|Step 45000|$\\\\mathbf{\\\\color{blue}0.066}$|$0.063$|\n\n> ## “The ablation experiment on action chunking size is incomplete.”\n\nWe ran a complete ablation study on more tasks with longer chunk sizes. In our original submission, we only ran on cube-triple-task4. Here we show an aggregation of all five cube-triple tasks to give a more complete picture. As shown in the figure, the final performance saturates at around $h=10$. When the action chunk size is too large ($h=50$), the learning fails. This is to be expected as the action prediction for longer chunks of actions is harder, and TD backup n-step return with a large $n$ reduces the stitching capability of Q-learning (e.g., the extreme case is Monte-Carlo regression where the algorithm can no longer do any stitching).\n\n| cube-triple (5 tasks) |$h=1$ | $h=5$ (our main results) | $h=10$ | $h=15$ | $h=25$ | $h=50$ |\n| --- |--- | --- | --- | --- | --- | --- |\n| Q-LAC | $\\mathbf{\\color{blue}1}$→$18$ | $\\mathbf{\\color{blue}4}$→$54$ | $\\mathbf{\\color{blue}2}$→$\\mathbf{\\color{purple}77}$ | $\\mathbf{\\color{blue}2}$→$\\mathbf{\\color{purple}71}$ | $1$→$\\mathbf{\\color{purple}76}$ | $0$→$0$ |\n\n> ## “The article lacks analysis on whether action chunking will bring additional memory or time overhead.”\n\nTo address your concern, we analyzed the time and memory that action chunking brings and found that action chunking incurs almost no overhead on CPU & GPU memory and some but manageable overhead on the computational cost especially for online learning where environment steps are still the main bottleneck.\n\n*Memory*\n\nAction chunking brings little memory overhead on both the CPU memory and GPU memory. For CPU memory, our replay buffer stores the transitions normally and we sample chunks of observations and actions for updating our agent. For GPU memory, action chunks make the input to the policy and the critic larger which make the first layer of our neural network (and the last layer of our neural network for the flow policy) larger. We include the parameter count below:\n- Q-LAC: ~5M parameters\n- FQL: ~4.9M parameters\n- RLPD: ~17.2M parameters\n\n*Time*\n\nIn terms of time, we report the time it requires for each step during the offline phase and the online phase below (in milliseconds). Compared to FQL, Q-LAC is ~40% more expensive offline, and only 10% more expensive in the online phase as the environment step takes most of the time during online training.\n\n|Methods|Q-LAC|FQL|RLPD|\n|---|---|---|---|\n|online time (ms/step)|$11.33 \\pm 1.08$|$10.06 \\pm 0.12$|$11.03 \\pm 1.53$|\n\n|Methods|Q-LAC|FQL|\n|---|---|---|\n|offline time (ms/step)|$4.24 \\pm 0.26$|$2.97 \\pm 0.04$|\n\n> ## “The article mentions that the learning of value is accelerated, but there is no quantified indicator or experiment to measure the extent of this acceleration.”\n\nTo address your question, we analyzed the convergence of the average Q-value of the state-action pair in the dataset over the course of offline training for Q-LAC, FQL-n and FQL on one of our hardest tasks, cube-quadruple-100M-task2 (averaged over 4 seeds). To gain insights on the convergence rate of the Q-value, we compute the difference in Q-value from the previous time step and highlight the difference if it is greater than 10. **Please refer to our response to kQoc for these results.**\n\nBoth Q-LAC and FQL-n (FQL with n-step returns) have large Q-value differences in the first 100K to 150K steps and then have small Q-value differences for the rest of the offline training. In contrast, FQL has large Q-value differences for a longer time up to almost half way through the offline training (~400K steps). This suggests that both Q-LAC and FQL-n enjoy a faster convergence of the Q-value compared to FQL, providing evidence that n-step backup does accelerate value learning. It is worth highlighting that while this quantifies the value learning speed, it does not quantify how well the value is learned. In fact, in our experiments, even though FQL-n exhibits a faster value learning, it suffers from an online collapse problem (as shown here), possibly due to the bias in the n-step backup. In contrast, Q-LAC does not suffer from this issue, possibly due to its unbiased n-step backup (see more discussion in Section 4.3 Benefit #1).\n\nThank you again for your constructive feedback and detailed reviews. **Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?**"}, {"review_id": "79FWNb0k1N", "reviewer": "Reviewer_6U2h", "summary": "This paper proposed Q-Learning with Action Chunking (Q-LAC), a reinforcement learning method that models action chunking by enabling the policy to output a sequence of $h$ actions from a given state. The critic is trained to evaluate the value of a state along with the $h$-step action sequence. The approach first trains in an offline dataset and then further trained through additional online learning, formalizing an offline-to-online setting.", "questions": "1. How were the baselines fine-tuned in the experiments to ensure they are performing properly? For example, in the original FQL paper (see Figure 12), FQL achieves near-optimal performance (close to 1.0) on tasks such as cube-double and scene. However, in this paper's Figure 4, FQL only reaches around 0.6 on these tasks. What accounts for this difference in performance?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is clearly written and well-organized.\n2. The idea of action chunking policy and a chunk-based Q-function is interesting and rarely explored in the RL context.\n3. The literature review is comprehensive.\n\n\n**Weaknesses**\n\n1. **The claimed advantages of action chunking over standard 1-step action policies and skill-based approaches are not convincing.** Action chunking breaks the Markov property and weakens the tight feedback loop between policy and environment. Let's denote action chunking policy as ACP and the normal 1-step action policy as N1P for convenience. Intuitively, ACP appears less favorable than N1P, it sacrifices reactivity and granularity, and may even be viewed as a degenerate case of N1Ps:\n\n   * N1P has finer granularity and higher real-time responsiveness. In ACP, no matter how the future states change, the policy blindly executes the pre-predicted chunk. There's no guarantee that the latter actions in the chunk are reasonable for the future states. In contrast, N1P reacts to every state, which is clearly more rational.\n   * If ACP is to be responsible for future $h$-step actions, under the same model scale, it requires the policy to have some foresight and predictive capability about the environment's transition model, i.e., some form of internal model-based reasoning. This increases modeling difficulty.\n   * Under the Markovian assumption: $\\pi^{Markovian}: s_t \\rightarrow a_t$, the policy outputs a single action based on the current state (given less information, do less behavior); Under non-Markovian assumption: $\\pi^{non-Markovian}: s_{t-n:t} \\rightarrow a_t$, the policy outputs a single action based on the history of states (given more information, do less behavior). In contrast, ACP outputs multiple actions based on one state: $\\pi^{ACP}: s_t \\rightarrow a_{t:t+h-1}$ (given less information, do more behavior). Clearly, ACP is the least favorable case. \n   * Furthermore, compared to skill-based methods (as mentioned at around Line 116), while skill-based methods are still step-wise policies internally, they can be interrupted or switched anytime, ACP, on the other hand, enforces hard commitment to an action sequence and cannot be interrupted, which is less flexible than skill-based methods.\n\n2. **Limited Contribution**. The proposed method can be considered as adding action chunking to the FQL baseline [1]. However, as noted in point 1, the theoretical justification and claimed benefits of action chunking are not well established. Simply integrating action chunking into FQL, without a solid theoretical grounding, offers limited innovation and contribution. Moreover, the reported performance comparisons with FQL deviate notably from those reported in the original FQL paper (see detailed Question 1 below), raising my concern about the claimed improvements empirically.\n\n3. In Figure 2, we can see that AC performs worse than the RLPD baseline in online RL, validating the above concerns. To address this, the paper introduces offline training and behavior regularization, leading to the results in Figure 3. However, these results don't convincingly demonstrate that Q-LAC's performance is due to action chunking or behavior regularization. It seems more likely that the offline training provides a better starting point for the policy (QLAC starts from around 0.3, while RLPD starts from 0), but RLPD quickly reaches the same performance. From this perspective, one main limitation of Q-LAC becomes that it relies heavily on offline data and behavior regularization. Therefore, I tend to view that the use of Q-LAC in offline-to-online scenarios seems to be more of a necessary compromise than a design choice.\n\n4. **Experimental comparisons raise several concerns:**\n\n   * As shown in Figure 4, in about half of the tasks, RLPD achieves performance equal to or better than Q-LAC. This again suggests that Q-LAC’s advantage likely comes from offline pretraining. For example, in the puzzle environment, Q-LAC already reaches the performance peak after offline training. I don’t think this convincingly demonstrates the advantage of Q-LAC, as it only shows improved sample efficiency in online learning, at the cost of more offline training.\n   * The core contribution of the paper is action chunking, but this point is not well supported by the experiments. If the impact of offline pretraining could be isolated, the results would be more convincing. Worryingly, Figure 2 shows that Q-LAC performs worse than the RLPD baseline in pure online settings; in Figure 4, in half of the environments (puzzle, robomimic, cube-quadruple), FQL-n performs better than FQL; and in all tasks, RLPD consistently outperforms RLPD-AC5. These results seem to contradict the claimed benefit of action chunking.\n   * The number of baselines (3) and evaluated tasks (6) is relatively small. For example, one of the baselines, FQL compared with 10 baselines on dozens of domains, which is a better experimental practice. Besides, experiments in Figure 5-7 only include one or two tasks. More comprehensive comparisons would strengthen the paper.\n\nIn summary, I think the action chunking policy proposed in this paper is an interesting idea, but it lacks sufficient theoretical support regarding MDPs, the Markov property, the Bellman equation, and policy convergence or optimality under this paradigm. The experimental results also fail to show clear advantages of action chunking itself. Therefore, I believe the contribution of this paper is limited.\n\n[1] Park, Seohong, Qiyang Li, and Sergey Levine. \"Flow Q-Learning.\" ICML, 2025.", "was_revised": true, "final_justification": "I acknowledge that the paper presents extensive experimental evaluations. However, I would like to raise my score from 2 to 3, but I still believe the work falls below the NeurIPS acceptance bar. My main concerns are as follows:\n\n1. The primary claimed contribution is introducing action chunking to the RL community and validating it through extensive experiments. In my view, this contribution is rather limited in novelty, since action chunking is not a new concept and is closely related to macro-actions, skills, and options in RL.\n2. The authors’ attempt to apply chunking to online RL was not successful, leading them to turn to the offline-to-online RL setting. I still think this seems more like a necessary compromise rather than a deliberate design choice.\n3. The authors emphasize additional designs, such as using expressive policies with behavior constraints. However, this is not novel either, as it has already been proposed in FQL, and thus cannot be counted as an original contribution. In essence, Q-LAC is equivalent to FQL + chunking.\n4. In the rebuttal, the authors noted that chunking can also improve other algorithms, such as BFN and IFQL. However, both are based on flow policies, suggesting that the observed improvement may still stem from the synergy with flow policy rather than chunking itself.\n\nIn summary, the work explores the application of action chunking to several RL algorithms, identifying cases where it improves performance and others where it does not, ultimately combining multiple existing strategies to achieve a cumulative enhancement. While the exploration is good, I don't see sufficiently novel or original contributions (e.g., action chunking, flow policy, and behavior constraint are all proposed by prior works), therefore, I give my score to be 3.", "rebuttal": "Thank you for your insightful comments and suggestions on our paper. We addressed your main concern about our experiments by **evaluating our method, all our baselines (11 baselines, with 3 new baselines) on 25 OGBench tasks (5 tasks for each of the 5 domains)**. In contrast, FQL evaluates 15 tasks for the offline-to-online setting (1 task for each domain). Our method outperforms all baselines across the board. In addition, we apply action chunking to two of our new baselines (BFN and IFQL) which yield two new variants of our method (Q-LAC-BFN, Q-LAC-IFQL) that also significantly improves the corresponding baselines. In particular, Q-LAC-BFN achieves strong performance across the board on par and sometimes better than Q-LAC. **Please see the summary of our new results in our response to Reviewer nCbd.**\n\nWe believe that there are also several **misunderstandings** of the reviewer on several **key results** of our paper that we would like to clarify below: \n\n>## “What accounts for this difference in performance for FQL compared to the original FQL paper?”\n \nOur results in Figure 4 were an aggregation of both task2 and task4. In contrast, for the offline-to-online RL experiments, the FQL paper only reported performance on the representative task for each domain (e.g., task2 for cube-double). The significantly harder task (e.g., task4) was not evaluated. For the rebuttal, we reran all the experiments on all 5 tasks for each domain, and our reported performance matches the FQL paper on the same task (e.g., task2 on cube-double).\n\n>## “In Figure 4, in half of the environments, FQL-n performs better than FQL … These results seem to contradict the claimed benefit of action chunking.” “As shown in Figure 4, in about half of the tasks, RLPD achieves performance equal to or better than Q-LAC.”\n\nThe results that show FQL-n performs better than FQL on these environments do not contradict the claimed benefit of action chunking. Instead, they show evidence that n-step return does help speed up value learning and Q-LAC uses chunking to enjoy a similar benefit of n-step return backup. In our new results, Q-LAC performs significantly better than FQL-n especially on the hardest domains (cube-triple and cube-quadruple). In terms of RLPD, while it is true that RLPD is a competitive baseline as it can match the online performance on the simpler tasks, it fails on the hardest cube-quadruple domain. In our newest experiments, both FQL and RLPD fail to solve all but one task in cube-quadruple domains. In contrast, Q-LAC solves 4 out of 5 tasks on cube-quadruple.\n\n>## “Q-LAC performs worse than the RLPD baseline in pure online settings” /  “Results don't convincingly demonstrate that Q-LAC's performance is due to action chunking or behavior regularization. It seems more likely that the offline training provides a better starting point for the policy”\n\nOn the hardest domain cube-quadruple, Q-LAC exhibits strong performance (solving 4/5 tasks) online with offline pre-trained performance being 0. On the other hand, RLPD methods do not succeed on any of these tasks. This provides strong evidence that the effectiveness of Q-LAC also comes from its ability to learn online effectively rather than just the advantage of being able to pre-train offline as suggested by the reviewer.\nWe would also like to clarify a couple of misunderstandings: (1) None of our methods are evaluated in the pure online setting. All the methods leverage an offline dataset. (2) Figure 2 shows Q-LAC outperforms RLPD, RLPD-AC and RLPD-ACBC after an initial pre-training phase which is opposite to what you seem to suggest in your review. (3) Our paper focuses on the offline-to-online setting which is common to use a pre-training phase [1-3].\n\n| |Q-LAC (ours)|Q-LAC-BFN (ours)|RLPD|RLPD-AC|RLPD-AC-BC|\n|---|---|---|---|---|---|\n|cube-double-task1|$66$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}84}$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}100}$|\n|cube-double-task2|$43$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}79}$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}98}$|$0$→$\\mathbf{\\color{purple}99}$|\n|cube-double-task3|$38$→$\\mathbf{\\color{purple}99}$|$\\mathbf{\\color{blue}68}$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}98}$|\n|cube-double-task4|$\\mathbf{\\color{blue}11}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}22}$→$92$|$0$→$\\mathbf{\\color{purple}98}$|$0$→$97$|$0$→$\\mathbf{\\color{purple}98}$|\n|cube-double-task5|$38$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}74}$→$\\mathbf{\\color{purple}99}$|$0$→$\\mathbf{\\color{purple}96}$|$0$→$87$|$0$→$88$|\n|cube-triple-task1|$\\mathbf{\\color{blue}19}$→$\\mathbf{\\color{purple}100}$|$\\mathbf{\\color{blue}13}$→$\\mathbf{\\color{purple}100}$|$0$→$\\mathbf{\\color{purple}100}$|$0$→$51$|$0$→$44$|\n|cube-triple-task2|$\\mathbf{\\color{blue}0}$→$\\mathbf{\\color{purple}89}$|$\\mathbf{\\color{blue}1}$→$\\mathbf{\\color{purple}89}$|$0$→$\\mathbf{\\color{purple}59}$|$0$→$2$|$0$→$8$|\n|cube-triple-task3|$0$→$\\mathbf{\\color{purple}52}$|$\\mathbf{\\color{blue}7}$→$\\mathbf{\\color{purple}73}$|$0$→$1$|$0$→$3$|$0$→$0$|\n|cube-triple-task4|$0$→$\\mathbf{\\color{purple}27}$|$0$→$\\mathbf{\\color{purple}54}$|$0$→$\\mathbf{\\color{purple}43}$|$0$→$0$|$0$→$0$|\n|cube-triple-task5|$0$→$0$|$0$→$0$|$0$→$0$|$0$→$0$|$0$→$0$|\n|cube-quadruple-100M-task1|$\\mathbf{\\color{blue}6}$→$\\mathbf{\\color{purple}98}$|$\\mathbf{\\color{blue}16}$→$\\mathbf{\\color{purple}100}$|$0$→$2$|$0$→$8$|$0$→$37$|\n|cube-quadruple-100M-task2|$0$→$\\mathbf{\\color{purple}97}$|$0$→$\\mathbf{\\color{purple}99}$|$0$→$0$|$0$→$26$|$0$→$17$|\n|cube-quadruple-100M-task3|$0$→$\\mathbf{\\color{purple}94}$|$\\mathbf{\\color{blue}3}$→$78$|$0$→$0$|$0$→$3$|$0$→$0$|\n|cube-quadruple-100M-task4|$0$→$\\mathbf{\\color{purple}95}$|$\\mathbf{\\color{blue}1}$→$\\mathbf{\\color{purple}88}$|$0$→$0$|$0$→$4$|$0$→$2$|\n|cube-quadruple-100M-task5|$0$→$0$|$0$→$0$|$0$→$0$|$0$→$0$|$0$→$0$|\n\n[1] \"Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning.\" \n\n[2] \"Flow Q-learning.\" \n\n[3] \"Adaptive policy learning for offline-to-online reinforcement learning.\"\n\n>## “RLPD consistently outperforms RLPD-AC5. These results seem to contradict the claimed benefit of action chunking.”\n\nRLPD-AC5 is not our method; the figure shows that naively using action chunking (RLPD-AC5) to an online RL algorithm (RLPD) does not work well.  Our key insight is that we need to use an expressive behavior constraint (e.g., behavior constraint on an expressive policy) to regularize the RL training (see Section 4.2). RLPD-AC5 (1) uses a Gaussian policy (much less expressive than a flow policy that is used in Q-LAC), and (2) does not use a behavior constraint whereas our method, Q-LAC, is based on FQL that deploys a behavior constraint.\n\n>## “[action chunking policy] sacrifices reactivity and granularity, and may even be viewed as a degenerate case of [1-step action policies]” / “[1-step action policy] has finer granularity and higher real-time responsiveness”\n\nOur action chunking policy is not a degenerate case of 1-step action policies. Our action chunking policy can not be fully represented by a 1-step action policy because action chunking policy can produce non-Markovian behavior that is not possible to achieve by a 1-step action policy. While 1-step action policy has higher real-time responsiveness, it also exhibits worse temporal coherency in our experiments **(see our response to Reviewer kQoc)**. Temporally correlated/non-Markovian noises have been known for improving online exploration [1, 2], and our method provides a convenient way to generate temporally coherent actions while also being informed by offline data as prior. While it is true that our action chunking policy can only represent a subset of all non-Markovian policies, in practice, we found that using an action chunking policy (Q-LAC) still consistently outperforms 1-step action policy (e.g., FQL).\n\n[1] \"Pink noise is all you need: Colored noise exploration in deep reinforcement learning.\" \n\n[2] \"Continuous control with deep reinforcement learning.\" \n\n>## Action chunking policies are limited as they cannot be switched and history-conditioned policies are more favorable. “Advantage of action chunking over 1-step action policies and skill-based approaches are not convincing”\n\nWhile there are limitations to action chunking policies, they do not diminish their strong empirical performances across the board **(see the table in our response to Reviewer nCbd)**. We show in our new experiments that Q-LAC performs better than all methods that use 1-step policies (FQL, RLPD, BFN) and the skill-based approach (SUPE-GT, see the description in our submission). We additionally include two variants of Q-LAC, Q-LAC-BFN and Q-LAC-IFQL, a result of applying action chunking to two newly added offline-to-online RL baselines, BFN and IFQL **(see more details in our response to Reviewer nCbd)**. As we have shown in the result table, action chunking consistently improves the non-chunking counterpart, supporting our claim that introducing action chunking into RL leads to improved performance.\n\n>## “action chunking policy proposed in this paper is an interesting idea, but it lacks sufficient theoretical support”\n\nWe are happy to hear that you think the action chunking policy proposed in this paper is an interesting idea. To provide more theoretical rigor and as recommended by other reviewers, we have included a theorem and proof below that shows the value backup for our chunked Q-function is unbiased. **Please refer to our response to Reviewer joyS for more details on the theorem and discussions.**\n\nThank you again for your constructive feedback and detailed reviews. **Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?**"}]

Author rebuttal:

Meta-review: This paper presents a simple yet effective actor-critic RL algorithm called Q-LAC. The key idea is to use temporally extended actions (i.e., action chunks) for learning (macro) action-values and acting in the environment. The proposed method is implemented upon FQL and tested on a range of long-horizon, sparse reward offline-to-online tasks, demonstrating strong performance. I think this is a timely study, especially since action chunking (ACT) has already become a default design choice for many embodied AI models. Developing an effective and simple offline-to-online finetuning RL method that is compatible with ACT can shed new light on the community. 

The paper initially received mixed reviews. The reviewers have raised concerns regarding its novelty, dependencies on FQL, and experiment evaluations. During the rebuttal, the authors have done a good job of convincing most reviewers by providing detailed discussion and comprehensive supplementary results. Most reviewers have raised their evaluations during the rebuttal phase and voted for acceptance. Although there is still one reviewer who holds negative opinions, I think the paper has reached the bar of NeurIPS.

I encourage the authors to further polish the paper in their final version based on the provided suggestions, especially,
- Include the new results and ablations during rebuttal.
- Discuss more about its dependency on FQL in the main text. The current version hides too many details in the Appendix.
- I hope the authors can add more discussion or conduct explorations on non-FQL implementations. As an FQL-based algorithm might be OK for running some small-scale tasks, but it can be hard to use on real-world, large-scale embodied AI models/tasks. Although the authors provided BFN and IFQL implementations during rebuttal, I think they are still somewhat restrictive (still heavily dependent on a flow BC policy). It will be more meaningful if Q-LAC can work on more flexible backbone algorithms.
Decision: Accept (poster)