Title: Reinforcement Learning with Action Chunking
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

offline-to-online RL

exploration
Abstract: We present Q-chunking, a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a *chunked* action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased $n$-step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.

Reinforcement Learning with Action Chunking

Abstract

We present Q-chunking , a simple yet effective recipe for improving reinforcement learning (RL) algorithms for long-horizon, sparse-reward tasks. Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning. Effective exploration and sample-efficient learning remain central challenges in this setting, as it is not obvious how the offline data should be utilized to acquire a good exploratory policy. Our key insight is that action chunking, a technique popularized in imitation learning where sequences of future actions are predicted rather than a single action at each timestep, can be applied to temporal difference (TD)-based RL methods to mitigate the exploration challenge. Q-chunking adopts action chunking by directly running RL in a 'chunked' action space, enabling the agent to (1) leverage temporally consistent behaviors from offline data for more effective online exploration and (2) use unbiased n -step backups for more stable and efficient TD learning. Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offlineto-online methods on a range of long-horizon, sparse-reward manipulation tasks.

State

Action

Environment

(2)

efficient value backup

by

'skipping' time steps

(1)

efficient exploration

with temporally coherent actions

Q(   ,             )

Q(   ,             )

π

(            |   )

π

(            |   )

Time

Steps

t

t+1

t+2

t+3

t+4

t+5

t+6

t+7

t+8

0.0

0.5

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Aggregated (5 OGBench Domains)

QC

BFN

FQL

RLPD

IFQL

.

1 Introduction

Reinforcement learning (RL) holds the promise of solving any given task based only on a reward function. However, this simple and direct formulation of the RL problem is often impractical: in complex environments, exploring entirely from scratch to learn an effective policy can be prohibitively expensive, as it requires the agent to successfully solve the task through random chance before learning a good policy. Indeed, even humans and animals rarely solve new tasks entirely from scratch, instead leveraging prior knowledge and skills from past experience. Inspired by this, a number of recent works have sought to incorporate prior offline data into online RL exploration [28, 39, 85]. But this poses a new set of challenges: the distribution of offline data might not match the policy that the agent should follow online, introducing distributional shift, and it is not obvious how the offline data should be leveraged to acquire a good online exploratory policy.

In the adjacent field of imitation learning (IL), a widely used approach in recent years has been to employ action chunking , where instead of training policies to predict a single action based on the state observation from prior data, the policy is instead trained to predict a short sequence of future actions (an 'action chunk') [90, 11]. While a complete explanation for the effectiveness of action chunking in IL remains an open question, its effectiveness can be at least partially ascribed to better handling of non-Markovian behavior in the offline data, essentially providing a more powerful tool for modeling the kinds of complex distributions that might occur in (for example) human-provided demonstrations or mixtures of different behaviors [90]. Action chunking has not been used widely in RL, perhaps because optimal policies in fully observed MDPs are Markovian [74], and therefore chunking may appear unnecessary.

We make the observation that, though we might desire a final optimal Markovian policy, the exploration problem can be better tackled with non-Markovian and temporally extended skills, and that action chunking offers a very simple and convenient recipe for obtaining this. Furthermore, action chunking provides a better way to leverage offline data (with a better handling of non-Markovian behavior in the data), and even improves the stability and efficiency of TD-based RL, by enabling unbiased n -step backups (where n matches the length of the chunk). Thus, in combination with pretraining on offline data, action chunking offers a compelling and very simple way to mitigate the exploration challenge in RL.

We present Q-learning with action chunking (or Q-chunking in short), a recipe for improving generic TD-based actor-critic RL algorithms in the offline-to-online RL setting (Figure 1). The key idea is to run RL at an action sequence level - (1) the policy predicts a sequence of actions for the next h steps and executes them one-by-one open loop, and (2) the critic takes in the current state and a sequence of actions and estimates the value of carrying out the whole sequence rather than a single action. The benefits of operating RL on this extended action space are two-fold: (1) the policy can be optimized to generate temporally coherent actions by regularizing it towards some prior behavior data that exhibit such coherency, (2) the critic trained with a standard TD-backup loss is effectively performing n -step backups, with no off-policy bias (that typically occurs in naïve n -step return methods), since the critic takes the full action sequence into account.

Our main contribution is QC , a practical offline-to-online RL algorithm that is instantiated from our Q-chunking recipe, essentially running RL with behavior regularization in the chunked action space. QC is simple to implement, requiring only training (1) an action chunking behavior policy using a standard flow-matching loss, and (2) a temporally extended critic with the standard TD-loss (Algorithm 1). QC achieves strong performance on a range of six challenging long-horizon, sparsereward domains, outperforming prior offline-to-online methods. Moreover, Q-chunking is a generic recipe that can be applied to existing offline-to-online algorithms with minimal modification. In this work, we demonstrate one such instantiation by applying it to FQL [58], resulting in QC-FQL (Algorithm 2), which shows significant improvements over the original method.

2 Related Work

Offline-to-online reinforcement learning methods focus on leveraging prior offline data to accelerate reinforcement learning online [88, 71, 37, 1, 89, 91, 7, 51, 92, 39]. The simplest way to tackle offlineto-online RL is to use an existing offline RL algorithm to first pretrain on the offline data and then use the same offline optimization objective to continue training online using a growing dataset that

combines the original offline data and the replay buffer data [50, 36, 33, 77, 58, 2, 42, 37]. While straightforward, this naïve approach often result in overly pessimistic that hinders exploration and consequently the online sample-efficiency. Several prior works have attempted to address this issue by adjusting the degree of pessimism online [92, 51, 42, 37, 82]. However, these approaches can be difficult to tune and sometimes stills fall short in online sample efficiency compared to a simple, wellregularized online RL algorithm learning from scratch on both offline data and online replay buffer data [7]. Our approach takes a step towards improving the sample efficiency of offline-to-online RL methods via value backup acceleration and temporally coherent exploration.

Action chunking is a technique popularized by roboticists for imitation learning (IL), where the policy predicts and executes a sequence of actions in an open-loop manner ('an action chunk') [90]. Action chunking has been shown to improve policy robustness [90, 23, 8], and handle non-Markovian behavior in offline data [90]. Existing RL methods that incorporate action chunking typically focus on fine-tuning a policy pre-trained with imitation learning [61]. Tian et al. [78] propose to learn a critic on action chunks by integrating n -step returns with a transformer. However, their method only applies chunking to the critic, while still optimizing a single-step actor. Li et al. [38] also observe that learning a critic over short action chunks removes the off-policy bias in n -step return backups, leading to more stable and effective value learning. There are key differences between their work and ours - Li et al. [38] operate in the online episodic RL setting [84, 31] and use Gaussian policies to predict parameters of motion primitives (MP) [63, 54], which are then used to generate full action sequences at the beginning of each episode. In contrast, we operate in the conventional offline-toonline RL setting and leverage more expressive flow-matching-based policies (as we find Gaussian policies to be ineffective as shown in Figure 2) to predict short action sequences directly in the raw action space. Seo and Abbeel [65] also train critics on action chunks and impose a behavior cloning loss that aligns with the principles behind Q-chunking. The key distinction from our work lies in their use of a multi-level, factorized critic architecture [66], which generates and refines action chunks from coarse to fine granularity via iterative discretization. At each level, the action space is discretized into bins, and the Q-function is modeled independently for each action dimension and timestep, conditioned on the whole action chunks predicted at the previous coarser level. While this factorized critic design enables tractable value-maximizing action sampling, it imposes strong structural assumptions on the action space, limiting policy expressiveness at each refinement level. In contrast, we make no such assumptions in our recipe which allows us to derive two general algorithms where both the critic and policy operate directly on action chunks without requiring factorization or iterative discretization/refinement.

Exploration with temporally coherent actions. Existing methods either rely on temporally correlated action noises [40] that are constructed through heuristics; hierarchically structured policies (see the next paragraph), which are often tricky to stabilize during online training; or pre-trained frozen skill policies [60, 85], which are not amendable for fine-grained online fine-tuning. Our method uses a single network to represent the policy to generate temporally extended action chunk and it is trained using a single objective function that is stable to optimize. There is also no frozen, pretrained components in our approach, ensuring its online fine-tuning flexibility.

Hierarchical reinforcement learning, options framework. Learning temporally extended actions have also been widely studied in the hierarchical reinforcement learning (HRL) literature [14, 16, 80, 13, 35, 81, 59, 62, 49, 3, 67, 60, 22, 87]. HRL methods typically train a space of low-level policies that can directly interact with the environment along with a high-level policy that selects among these low-level policies. These low-level policies can be hand-crafted [12], automatically discovered online [16, 35, 80, 81, 49], or pretrained using offline skill discovery methods [54, 47, 67, 3, 70, 60, 79, 52, 28, 20, 9, 57]. The options framework provides a slightly more sophisticated and more powerful formulation, where the low-level policy is additionally associated with learnable initiation condition and termination condition that makes utilization of the low-level policy more flexible [75, 46, 10, 44, 68, 69, 32, 13, 72, 53, 19, 4, 30, 5, 6, 15]. A long-lasting challenge in HRL is its bi-level optimization problem: when both low-level and high-level policies are updated during training, the high-level policies must optimize a moving objective function, which can lead to instability [49]. To mitigate this, some methods keep the low-level policies frozen after initial pretraining [3, 60, 85] to improve stability during online training. Our approach is a special case of HRL where the low-level skill executes a sequence of actions open-loop. This design choice allows us to collapse the bi-level optimization problem into a standard RL objective in a temporally extended action space, while retaining many of the exploration benefits associated with HRL methods. While

prior work in HRL has also explored such idea [45, 17], they often leverage a discrete set of action sequences that is either heuristically extracted from the prior experience or given in advance. In contrast, we use an expressive flow-matching based policy to directly parameterize a continuous space of action sequences, and are directly trained and fine-tuned online with RL.

Multi-step latent space planning and search is a technique commonly used in model-based RL methods where they use a learned model to optimize a short-horizon action sequence towards highreturn trajectories [53, 64]. These approaches work by training a dynamics model on an encoded latent space, where the model takes in a latent state and an action to predict the next latent state and the associated reward value. This latent dynamics model, along with a value network on the latent state, can then provide an estimate of the Q -value on-the-fly for any given action sequence starting from a given latent state by simply simulating the action sequence in the latent dynamics model. In contrast, we do not learn a latent dynamics model and instead train a Q -network to directly estimate the value of the action sequence. Lastly, these approaches operate in the purely online RL setting whereas we focus on the offline-to-online RL setting.

3 Background

Offline-to-online RL. In this paper, we consider an infinite-horizon, fully observable Markov decision process (MDP), ( S , A , ρ, T, r, γ ) , where S is the state space, A is the action space, T ( s ′ | s, a ) : S × A ↦→ ∆( S ) is the transition kernel, r ( s, a ) : S × A ↦→ R is the reward function, ρ : ∆( S ) is the initial state distribution and γ ∈ [0 , 1) is the discount factor. We also assume there is a prior offline dataset D that consists of transitions rollouts { ( s, a, s ′ , r ) } from M . The goal of offline-to-online RL is to find a policy π ( a | s ) : S ↦→ ∆( A ) that maximizes the expected discounted cumulative reward (or discounted return): η ( π ) := E s t +1 ∼ T ( s t ,a t ) ,a t ∼ π ( ·| s t ) [ ∑ ∞ t =0 γ t r ( s t , a t )] . Oftentimes, offline-toonline RL algorithms operate in two distinct phases: an offline phase where a policy is pretrained on the offline data D and an online phase where the policy is further fine-tuned online with environment interactions. Our approach follows the same regime.

Temporal difference and multi-step return. TD-based RL algorithms typically learn Q θ ( s, a ) to approximate the maximum expected discounted cumulative reward that a policy can receive starting from state s and action a by using a temporal difference (TD) loss [74]:

where ˆ V is an estimate of Q ( s t , a t ) that is commonly chosen as ˆ V 1 -step :

and s t , a t , s t +1 , r are sampled from some off-policy trajectories and ¯ θ is a delayed version of θ that does not allow the gradient to pass through for learning stability. When the TD error is minimized, the Q θ converges to the expected discounted value of the policy π ψ . As the effective horizon ˜ H = 1 / (1 -γ ) goes up, the learning slows down as the value only propagates 1 step backward (from s t +1 to s t ). To speed-up long-horizon value backup, a common strategy is to sample a lengthn trajectory segment, ( s t , a t , s t +1 , · · · , a t + n -1 , s t + n ) , and construct a n -step return from it [83, 74]:

where again r t = r ( s t , a t ) . This value estimate of Q ( s t , a t ) allows for a n times speed-up in terms of the number of time steps that the value can propagate back across. This estimator is sometimes referred to as the uncorrected n -step return estimator [18, 34] because it is biased when the data collection policy is different from the current policy π ψ . Nevertheless, due to the implementation simplicity of n -step return, it has been commonly adopted in large-scale RL systems [48, 26, 29, 86].

4 Q-Chunking

In this section, we first describe two main design principles of Q-chunking : (1) Q-learning on a temporally extended action space (the space of chunks of actions), and (2) behavior constraint in this extended action space, followed by practical implementations of Q-chunking ( QC , QC-FQL ) as effective TD-based offline-to-online RL algorithms.

4.1 Q-learning on a temporally extended action space

The first design principle of Q-chunking is to apply Q -learning on the temporally extended action space. Unlike normal 1-step TD-based actor-critic methods, which train a Q-function Q ( s t , a t ) and a policy π ( a t | s t ) , we instead train both the critic and the actor with a span of h consecutive actions: 1

In practice, this involves updating the critic and the actor on batches of transitions consisting of a random state s t , an action sequence a t : t + h followed by the state, and the state h steps into the future, s t + h . Specifically, we train Q θ with the following TD loss,

with a t + h : t +2 h ∼ π ψ ( ·| s t + h ) , and ¯ θ being the target network parameters that are often an exponential moving average of θ [25].

The TD loss above shares striking similarity to the n -step return in Equation 3 (with n matches h ) but with a crucial difference - the Q -function used in the n -step return backup takes in only one action (at time step t ) whereas our Q -function takes in the whole action sequence. The implication of this difference can be best explained after we write out the TD backup equations for standard 1-step TD, n -step return, and Q-chunking:

For the standard 1-step TD, each backup step propagates the value back by only 1 time step. n -step return propagates the value back h × faster, but can suffer from a biased value estimation issue when s t : t + h and a t : t + h are off-policy [18]. This is because the discounted sum of the n -step rewards r t : t + h from the dataset or replay buffer is no longer an unbiased estimate of the expected n -step rewards under the current policy π . Q-chunking value backup is similar to the n -step return where each step also propagates the value back by h time steps, but does not suffer from this biased estimation issue. Unlike n -step return where we are propagating the value to a 1-step Q -function, Q-chunking backup propagates the value back to a h -step Q -function that takes in the exact same actions that are taken to obtain the n -step rewards r t : t + h , eliminating the biased value estimation. We formalize this argument in Theorem A.1. As a result, Q-chunking value backup enjoys the value propagation speedup while maintaining an unbiased value estimate.

4.2 Behavior constraints for temporally coherent exploration

The second design principle of Q-chunking addresses the action incoherency issues by leveraging a behavior constraint in the objective for the π ψ :

where we denote π β ( a t : t + h | s t ) as the behavior distribution in the offline data D , and D as some distance metric that measures how different the learned policy π deviates from π β .

Intuitively, a behavior constraint on the temporally extended action sequence allows us to leverage temporally coherent action sequences in the offline dataset. This is particularly advantageous in

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple

Online Steps (×10 6 )

Success Rate

QC-RLPD

RLPD-AC

RLPD

the temporally extended action space compared to in the original action space because offline data often exhibit non-Markovian structure ( e.g. , from scripted policies [56], human tele-operators [43], or noisy expert policies for sub-tasks [56, 21]) that cannot be well captured by a Markovian behavior constraint. Temporally coherent actions are desirable for online exploration because they resemble temporally extended skills ( e.g. , moving in a certain direction for navigation, jumping motions for going over obstacles) that help traverse the environment in a structured way rather than using random actions that often result in data that is localized near the initial states. Imposing behavior constraint for an action chunking policy is a very simple way to approximately extract skills without the need of training policy with bi-level structure as often necessitated by skill-based methods (see more discussion in Section 2). We observe that Q-chunking, with such behavior constraints, can explore with temporally coherent actions (see Section 5.3), mitigating the exploration challenge.

4.3 Practical implementations

A key implementation challenge of Q-chunking is to enforce a good behavior constraint that captures the non-Markovian behavior at the action sequence level. One prerequisites of imposing a good behavior constraint is the ability of the policy to capture the complex behavior distribution ( e.g. , with a flow/diffusion policy). A Gaussian policy, a default choice in online RL algorithms, does not suffice. Indeed, if we naïvely take an off-the-shelf online algorithm, RLPD [7] for example, and apply Qchunking with a behavior cloning loss, we find that it often performs poorly (Figure 2).

To enforce a good behavior constraint, we start by using flow-matching objective [41] to train a behavior cloning flow policy to capture the behavior distribution. The flow policy is parameterized by a state-conditioned velocity field prediction model f ( s, z , u ) : S × R Ah × [0 , 1] ↦→ R Ah and we denote f ξ ( ·| s ) as the action distribution that the flow policy parameterizes, which serves as an approximation of the true behavior distribution in the offline data ( f ξ ≈ π β ). Now, we are ready to present our main method:

QC: Q-chunking with implicit KL behavior constraint. We consider a KL constraint on our policy through the learned behavior distribution:

While it is possible include the KL as part of the loss, estimating the KL divergence or log probability for flow models is practically challenging. Instead, we use best-ofN sampling [73] to maximize Q -value while imposing this KL constraint implicitly altogether. Practically, this involves first sampling N action chunks from the learned behavior policy f ξ ( ·| s ) ,

and then picking the action chunk sample that maximizes the temporally extended Q -function:

It has been shown in prior work that best-ofN sampling admits a closed-form upper-bound on the KL divergence from the original distribution [27]:

which approximately satisfies KL constraint implicitly (Equation 9). Tuning the value of N directly corresponds to the strength of the constraint.

Since we approximate the policy optimization (Equation 8) with the best-ofN sampling, we can completely avoid separately parameterizing a policy π ψ and only sample from the behavioral policy f ξ ( ·| s t ) . In particular, we use the best-ofN sampling to generate actions to both (1) interact with the environment, and (2) provide the action samples in the TD backup following Ghasemipour et al. [24]. As a result, our algorithm has only one additional loss function:

where again a ⋆ t + h := arg max a ∈{ a i t + h } Q ( s, a ) .

While QC is simple and easy to implement, it does come with some additional computational costs associated with best-of-N sampling. In our experiments, we experiment with two other variants ( QCFQL and QC-IFQL ) that leverage cheaper off-the-shelf offline/offline-to-online RL methods (FQL and IFQL respectively [58]). We present the FQL-version below as it performs better empirically.

QC-FQL: Q-chunking with 2-Wasserstein distance behavior constraint. For this variant of our method, we leverage the optimal transport framework to impose a Wasserstein distance ( W 2 ) constraint, again, through the learned behavior policy f ξ ( ·| s ) :

Following FQL [58], we parameterize the policy π ψ with a noise-conditioned action prediction model, µ ψ ( s, z ) : S × R Ah ↦→ R Ah , which directly outputs an action from Gaussian noise in one network forward pass. This noise-conditioned policy is trained to maximize the Q-chunking critic Q θ ( s t , a t : t + h ) while being regularized to be close to the behavioral cloning flow-matching policy via a BC loss that is shown to be an upper-bound on the squared 2-Wasserstein distance [58]:

where z 1 is the ODE solution from u = 0 to u = 1 following d z u = f ξ ( s t , z u , u )d u (the initial value z 0 is sampled from the unit Gaussian). The real-valued hyperparameter α directly controls the magnitude of the distillation loss. Finally, the TD loss remains the same as the previous section with the only difference in how we parameterize the policy:

where again z ∼ N (0 , I Ah ) .

Offline-to-online RL considerations. Since both variants of our methods use behavior constraint (implicit KL for QC, explicit W 2 for QC-FQL), we can also directly run them for offline RL pretraining. For both offline and online training, we use the same behavior constraint strength ( e.g. , N for QC and α for QC-FQL). We use the same algorithm for both offline and online training, and the only difference is whether there is environment interactions. See Section C, Algorithm 1 and Algorithm 2 for an overview of QC and QC-FQL during online training.

5 Experimental Results

We conduct extensive experiments to analyze the empirical effectiveness of our method on a range of long-horizon, sparse-reward domains. In particular, we are going to answer the following questions:

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

lift

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

can

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

square

Steps (×10 6 )

Success Rate

QC

BFN

FQL

RLPD

RLPD-AC

( Q1) How well do Q-chunking methods perform compared to prior offline-to-online RL methods?

( Q2) Why does action chunking helps online learning?

( Q3) How does chunk length, critic ensemble size, and update-to-data ratio affect performance?

5.1 Environments and Datasets

We consider six sparse reward robotic manipulation domains with tasks of varying difficulties. This includes 5 domains (5 tasks each) from OGBench [55], scene-sparse , puzzle-3x3-sparse , cube-double/triple/quadruple and 3 tasks from robomimic [43]. For OGBench, we use the default play-style datasets except for cube-quadruple where we use the larger 100M-size dataset. For robomimic, we use the multi-human datasets. See more details in Appendix B.

5.2 Comparisons

We compare with prior methods that speedup value backup as well as the previous best offline-toonline RL methods. We include a brief description of them below with more details in Section C.

BFN (best-ofN ) is a baseline that we propose to combine the expected-max Q operator [24] with an expressive behavior flow policy. BFN operates in the original action space and uses best-ofN

0.02

0.04

0.06

0.08

Online Steps (×10 6 )

0.04

0.06

Avg. of

x

eef

t

x

eef

t

+5 2

Temporal Coherency in EEF Positions

QC

BFN

QC

BFN

sampling to pick the action (out of N ) that maximizes the current Q -value. This baseline is an ablation to isolate the benefit of Q-chunking in QC .

FQL [58] is a recently proposed offline RL method that achieves strong offline and offline-to-online RL performance. This baseline is an ablation to isolate the benefit of Q-chunking in QC-FQL .

BFN-n/FQL-n. These baselines are the same as BFN/FQL but uses n -step backup with n > 1 (Equation 3) instead of the standard 1-step TD backup. This baseline enjoys the benefits of value backup speedup, but does not use chunked critic or actor, and potentially suffer from the bias issue.

RLPD [7], RLPD-AC . RLPD is a sample-efficient RL algorithm that treats offline data as additional off-policy data and learn from scratch online. RLPD-AC is the same as RLPD but operates on the temporally extended action space. Both of them do not use a behavior constraint.

We also compare with SUPE-GT [85], a recently proposed skill-based that is designed for offlineto-online RL. The original method is designed to deal with unlabeled dataset by learning a reward model with reward bonuses. We adapt it to our setting by directly using the ground truth rewards.

Finally, we compare with additional baselines: IQL [33], ReBRAC [76], IFQL (a baseline implemented in Park et al. [58] that combines IQL with rejection sampling policy extraction), and IFQL-n (IFQL with n -step return backup).

5.3 How well does our method compare to prior offline-to-online RL methods?

We report the performance of all Q-chunking methods and the baselines in Table 1 (OGBench) and a selective (strong) subset in Figure 3 (robomimic). QC achieves competitive performance offline, often matching or sometimes outperforming best prior methods. In the online phase, QC shows strong sample-efficiency, especially on the two hardest OGBench domains ( cube-triple/quadruple ), where it outperforms all prior methods (especially on cube-quadruple ) by a large margin. In addition, on OGBench domains, all Q-chunking methods (QC, QC-FQL, QC-IFQL) outperforms both their corresponding 1 -step TD counterpart (BFN, FQL, IFQL) and their correspdoning n -step return counterpart (BFN-n, FQL-n, IFQL-n). A similar trend continues on robomimic tasks as shown in Figure 4. Across OGBench and robomimic, n -step return baselines, which do not use chunked critics or policies, perform significantly worse than Q-chunking methods.

5.4 Why does action chunking help exploration?

We hypothesize in Section 4.2 that action chunking policy produce more temporally coherent actions and thus lead to better state coverage and exploration. In this section, we study to what degree that

0.0

0.5

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

robomimic (3 tasks)

QC

QC-FQL

BFN-n

FQL-n

BFN

FQL

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Action Chunk Size

1

5

10

15

25

50

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Critic Ensemble Size

QC (K=10)

QC (K=2)

BFN (K=10)

BFN (K=2)

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

Update-to-data Ratio

QC (UTD=1)

QC (UTD=5)

BFN (UTD=1)

BFN (UTD=5)

holds empirically. We first visualize the end-effector movements early in the training for QC and BFN (Figure 5, left). BFN 's trajectory contains many pauses (as indicated by a very big and dense cluster near the center of the visualization), especially when the end-effector is being lowered to pickup a cube. In contrast, QC has fewer pauses (fewer and shallower clusters) and a more diverse state coverage in the end-effector space. We include additional examples in Appendix E, Figure 8 and Figure 9. To get a quantitative measure of the temporal coherency in actions, we record the 3D end-effector position throughout training every 5 time steps: { x eef 0 , x eef 5 , · · · } and compute the average L 2 norm of the difference vector of two adjacent end-effector positions. This average norm would be small if there are any pauses or jittery motions, making a good proxy for measuring the temporal coherency in actions. As shown in Figure 5 (right), QC exhibits a higher action temporal coherency throughout training compared to BFN . This suggests that Q-chunking improves temporal coherency in actions, which explains the improved sample-efficiency that Q-chunking brings.

5.5 How does action chunk length, critic ensemble size, and UTD ratio affect performance?

In Figure 6 (left), we examine the performance of QC-FQL across different action chunk lengths ( h ∈ { 1 , 5 , 10 , 25 , 50 } ) on the cube-triple domain. Increasing the chunk length helps up to h = 10 , after which the asymptotic performance starts to drop. Although h = 25 shows faster early learning, it fails to achieve the same performance as h = 10 at the end of online fine-tuning. An even larger chunk length ( h = 50 ) fails to achieve any success. We suspect that overly large chunk sizes either hurt policy reactivity too much or make policy learning too difficult, as the network must predict a much longer action sequence at once. We use h = 5 in all our other experiments as h = 5 generally performs well and is cheaper to run. We also include the individual task breakdown in Figure 14 and additional ablation results on cube-quadruple in Figure 15. In Figure 6 (center), we study how the critic ensemble size affects the performance of our method. Using 10 critics improves both QC and BFN . We use K = 2 in our other experiments as it is cheap to run. Using K = 10 could potentially make Q-chunking perform much better on the benchmark tasks we consider. Finally, increasing the update-to-data ratio (UTD) does not improve the sample efficiency of QC (Figure 6, right).

6 Discussions

We demonstrate how action chunking can be integrated into an offline-to-online RL agent with a simple recipe. Our approach speeds up value backup and explores more effectively online with temporally coherent actions. As a result, it outperforms prior offline-to-online methods on a range of challenging long-horizon tasks. Our work serves as a step towards training non-Markovian policy for effective online exploration from prior offline data. Several challenges remain, opening promising directions for future research. First, our approach use a fixed action chunk, but it is unclear how to choose this size other than task-specific hyperparameter tuning. A natural next step would be to develop mechanisms that automatically determine chunk boundaries. Second, action chunking represents only a limited subclass of non-Markovian policies and may perform poorly in settings where a high-frequency control feedback loop is essential. Developing practical techniques for training more general non-Markovian policies for online exploration would further improve the online sample efficiency of offline-to-online RL algorithms.

Acknowledgments

This research was partially supported by RAI, and ONR N00014-22-1-2773. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at UC Berkeley. We would like to thank Seohong Park for providing the code infrastructure and the 100M dataset for cube-quadruple . We would also like to thank Ameesh Shah, Chuer Pan, Oleg Rybkin, Andrew Wagenmaker, Seohong Park, Yifei Zhou, Fangchen Liu, William Chen for discussions on the method and feedback on the early draft of the paper. Finally, we would like to thank NeurIPS review commitee for thoughtful comments and suggestions.

References

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 28955-28971. Curran Associates, Inc., 2022.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in neural information processing systems , 35:28955-28971, 2022.

Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. OPAL: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= V69LGwJ0lIN .

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In International Conference on Learning Representations , 2019.

Akhil Bagaria, Ben Abbatematteo, Omer Gottesman, Matt Corsaro, Sreehari Rammohan, and George Konidaris. Effectively learning initiation sets in hierarchical reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning , pages 1577-1594. PMLR, 2023.

Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 4788-4795. IEEE, 2024.

Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, and Abhishek Gupta. Selfsupervised reinforcement learning that transfers using random features. Advances in Neural Information Processing Systems , 36, 2024.

Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. Advances in neural information processing systems , 17, 2004.

Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , page 02783649241273668, 2023.

Murtaza Dalal, Deepak Pathak, and Russ R Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. Advances in Neural Information Processing Systems , 34:21847-21859, 2021.

Christian Daniel, Gerhard Neumann, Oliver Kroemer, and Jan Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research , 17(93):1-50, 2016.

Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information processing systems , 5, 1992.

Anita de Mello Koch, Akhil Bagaria, Bingnan Huo, Zhiyuan Zhou, Cameron Allen, and George Konidaris. Learning transferable sub-goals by hypothesizing generalizing features. 2025.

Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research , 13:227-303, 2000.

Ishan P Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan. Deep reinforcement learning with macro-actions. arXiv preprint arXiv:1606.04615 , 2016.

William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International conference on machine learning , pages 3061-3071. PMLR, 2020.

Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv:1703.08294 , 2017.

Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Unsupervised zero-shot reinforcement learning via functional reward encodings. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 13927-13942. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/frans24a.html .

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration. Advances in Neural Information Processing Systems , 34:11553-11564, 2021.

Abraham George and Amir Barati Farimani. One act play: Single demonstration behavior cloning with action chunking transformers. arXiv preprint arXiv:2309.10175 , 2023.

Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning , pages 3682-3691. PMLR, 2021.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Jacob Hilton. Kl divergence of max-of-n, 2023. URL https://www.jacobh.co.uk/bon_kl. pdf .

Hao Hu, Yiqin Yang, Jianing Ye, Ziqing Mai, and Chongjie Zhang. Unsupervised behavior extraction via random intent priors. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=4vGVQVz5KG .

Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations , 2018.

Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems , 32, 2019.

Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Advances in neural information processing systems , 21, 2008.

George Dimitri Konidaris. Autonomous robot skill acquisition . University of Massachusetts Amherst, 2011.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit Q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Tadashi Kozuno, Yunhao Tang, Mark Rowland, Rémi Munos, Steven Kapturowski, Will Dabney, Michal Valko, and David Abel. Revisiting peng's q ( λ ) for modern reinforcement learning. In International Conference on Machine Learning , pages 5794-5804. PMLR, 2021.

Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems , 29, 2016.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic Q-ensemble. In Conference on Robot Learning , pages 1702-1712. PMLR, 2022.

Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, and Gerhard Neumann. TOP-ERL: Transformer-based off-policy episodic reinforcement learning. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview. net/forum?id=N4NhVN30ph .

Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems , 36, 2024.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth. Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions. arXiv preprint arXiv:2303.17396 , 2023.

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In arXiv preprint arXiv:2108.03298 , 2021.

Shie Mannor, Ishai Menache, Amit Hoze, and Uri Klein. Dynamic abstraction in reinforcement learning via clustering. In Proceedings of the twenty-first international conference on Machine learning , page 71, 2004.

Amy McGovern and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. 1998.

Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut-dynamic discovery of sub-goals in reinforcement learning. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13 , pages 295-306. Springer, 2002.

Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv:1811.11711 , 2018.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PmLR, 2016.

Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems , 31, 2018.

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems , 36, 2024.

Soroush Nasiriany, Tian Gao, Ajay Mandlekar, and Yuke Zhu. Learning and retrieval from prior data for skill-based imitation learning. In Conference on Robot Learning , 2022.

Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. Advances in neural information processing systems , 30, 2017.

Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. Probabilistic movement primitives. Advances in neural information processing systems , 26, 2013.

Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. ArXiv , 2024.

Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv:2410.20092 , 2024.

Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In Forty-first International Conference on Machine Learning , 2024. URL https: //openreview.net/forum?id=LhNsSaAKub .

Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-learning. arXiv preprint arXiv:2502.02538 , 2025.

Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog) , 36(4):1-13, 2017.

Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning , pages 188-204. PMLR, 2021.

Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588 , 2024.

Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference on machine learning , pages 43444353. PMLR, 2018.

Stefan Schaal. Dynamic movement primitives-a framework for motor control in humans and humanoid robotics. In Adaptive motion of animals and machines , pages 261-280. Springer, 2006.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Younggyo Seo and Pieter Abbeel. Reinforcement learning with action sequence for dataefficient robot learning. arXiv preprint arXiv:2411.12155 , 2024.

Younggyo Seo, Jafar Uruç, and Stephen James. Continuous control with coarse-to-fine reinforcement learning. In 8th Annual Conference on Robot Learning , 2024. URL https: //openreview.net/forum?id=WjDR48cL3O .

Tanmay Shankar and Abhinav Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning , pages 8624-8633. PMLR, 2020.

Özgür ¸ Sim¸ sek and Andrew G Barto. Using relative novelty to identify useful temporal abstractions in reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning , page 95, 2004.

Özgür ¸ Sim¸ sek and Andrew G. Barto. Betweenness centrality as a basis for forming skills. Workingpaper, University of Massachusetts Amherst, April 2007.

Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=Ysuv-WOFeKR .

Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=yyBis80iUuU .

Aravind Srinivas, Ramnandan Krishnamurthy, Peeyush Kumar, and Balaraman Ravindran. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv:1605.05359 , 2016.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems , 33:3008-3021, 2020.

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998.

Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(1-2): 181-211, 1999.

Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems , 36:11592-11620, 2023.

Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-based soft actor-critic with n-step returns. arXiv preprint arXiv:2503.03660 , 2025.

Ahmed Touati, Jérémy Rapin, and Yann Ollivier. Does zero-shot reinforcement learning exist? In The Eleventh International Conference on Learning Representations , 2022.

Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. Advances in neural information processing systems , 29, 2016.

Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning , pages 3540-3549. PMLR, 2017.

Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. Advances in Neural Information Processing Systems , 36:47081-47104, 2023.

Christopher John Cornish Hellaby Watkins et al. Learning from delayed rewards. 1989.

Darrell Whitley, Stephen Dominic, Rajarshi Das, and Charles W Anderson. Genetic reinforcement learning for neurocontrol problems. Machine Learning , 13(2):259-284, 1993.

Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. arXiv preprint arXiv:2410.18076 , 2024.

Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602(7896): 223-228, 2022.

Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=jXe91kq3jAq .

Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetuning: Bridging sample-efficient offline and online reinforcement learning. Advances in neural information processing systems , 34:27395-27407, 2021.

Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=-Y34L45JR6z .

Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705 , 2023.

Han Zheng, Xufang Luo, Pengfei Wei, Xuan Song, Dongsheng Li, and Jing Jiang. Adaptive policy learning for offline-to-online reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11372-11380, 2023.

Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. arXiv preprint arXiv:2412.07762 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction are supported by the results in Section 4 and Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we discuss some limitations of our method in Section 6. Our assumptions and problem statement is included in Section 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: the proof of our theoretical result is available in Section A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we provide the algorithm in Algorithm 1 and Algorithm 2, and implementation details, including hyperparameter choices in Section C. We provide a public repo ( github.com/ColinQiyangLi/qc ) of our code that reproduces results in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, we have open-sourced our code at github.com/ColinQiyangLi/qc , which also includes instructions to download the data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we provide the details on the design choices in Section 5 and the hyperparameter values as well as the hyperparameter tuning process in Section C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All results in the paper have confidence intervals, denoting 95% confidence interval with four random seeds (five for robomimic results). We estimate the confidence intervals with bootstrapped sampling.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide additional information on computer resources in Section D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We adhere to the NeurIPS code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is on core reinforcement learning algorithm that has minimal societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: all code packages and datasets are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release code as an asset and provide detailed documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or nonstandard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We do not use LLM as an important component of this work.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Theoretical Justification

Proposition A.1 (Q-chunking performs unbiased n -step return backup) . Let s t , a t , · · · , s t + n be a trajectory segment generated by following a data collection policy π β ( a t , · · · , a t + n | s t ) ( i.e. , s t + k ∼ T ( · | s t + k -1 , a t + k -1 ) , ∀ k ∈ { 1 , · · · , n } , and r t , r t +1 , · · · r t + n -1 be the reward received at each corresponding time step ( i.e. , r t + k = r ( s t + k , a t + k ) ). Let V π ( s t ) be the value for an action chunking policy π ( a t , · · · , a t + n | s t ) starting from state s t . Let Q π ( s t , a t , a t +1 , · · · , a t + n -1 ) be the Q-value for π starting from state s t and executing the action sequence a t , a t +1 , · · · , a t + n -1 .

The n -step return estimate ˆ V n -step := ∑ t + n -1 t ′ = t [ γ t ′ -t r t ′ ] + ˆ V ( s t + n ) under the trajectory segment distribution described above is unbiased for Q π ( s t , a t , · · · , a t + n -1 ) as long as ˆ V ( s t + n ) is unbiased for V π ( s t + n ) .

Proof. From the definition of Q π ( s t , a t , a t +1 , · · · , a t + n -1 ) , we can write it as an expectation:

The second line uses the fact that the rewards r t , r t +1 , · · · , r t + h -1 are associated with the trajectory segment s t , a t , · · · , s t + n . The third line uses the fact that ˆ V ( s t + n ) is unbiased for V π ( s t + n ) . The fourth line uses the definition of ˆ V n -step .

The implication of this theorem is that if we use the n -step return backup for the chunked Qfunction Q ( s t , a t , a t +1 , · · · , a t + n -1 ) (with a value estimate of ˆ V s ← Q ( s t , a t , a t +1 , · · · , a t + n -1 ) ), it converges to the ground truth Q -value under the same policy Q π ( s t , a t , a t +1 , · · · , a t + n -1 ) . In contrast, using n -step return backup in the original action space does not converge to the correct Q -value ( i.e. , Q π ( s t , a t , a t +1 , · · · , a t + n -1 ) ).

B Domain Details

See an overview of the six domains we use in our experiments in Figure 7. We also include the dataset size, episode length and the action dimension in Table 2. In the following sections, we describe each domain in details.

B.1 OGBench environments.

We consider five manipulation domains from OGBench [55] and take the publicly available singletask versions of it in our experiments. For scene-sparse and puzzle-3x3-sparse , we sparsify the reward function such that the reward values are -1 when the task is incomplete and 0 when the task is completed. For cube-double/triple/quadruple , the RL agent needs to command an UR-5 arm to pick and place two/three/four cubes to target locations. In particular, cube-triple and cube-quadruple are extremely difficult to solve from offline data only, and often achieve zero success rate. The RL agent must explore efficiently online in these domains to solve the tasks. The cube-* domains provide a great test ground for sample-efficiency of offline-to-online RL algorithms which we primarily focus on. For cube-quadruple , we use the 100M-size dataset. The dataset is too big to fit our CPU memory, so we periodically (after every 1000 gradient steps) load in a 1 Msize chunk of the dataset for offline training. For online training of RLPD , QC-RLPD , we use the

same strategy where we load in a 1 M-size chunk of the dataset as the offline data and perform 50/50 sampling (e.g., 50% of the data comes from the 1 M-chunk of the offline data, 50% of the data comes from the online replay buffer). For online fine-tuning of QC-* , FQL , FQL-n , BFN , and BFN-n , we keep a fixed 1 M-size chunk of the offline dataset as the initialization of D and adds new data to D directly. The remaining 99 Mtransitions in the offline data are not being used online. We now describe each of the five domains in details:

scene-sparse : This domain involves a drawer, a window, a cube and two button locks that control whether the drawer and the window can be opened. These tasks typically involve a sequence of actions. For example, scene-task2 requires the robotic arm to unlock both locks, move the drawer and the window to the desired position, and then lock both locks. scene-task4 requires the robotic arm to unlock the drawer, open the drawer, put the cube into the drawer, close the drawer. The reward is binary: -1 if the desired configuration is not yet reached and 0 if the desired configuration is reached (and the episode terminates).

puzzle-3x3-sparse : This domain contains a 3 × 3 grid of buttons. Each button has two states represented by its color (blue or red). Pressing any button causes its color and the color of all its adjacent buttons to flip (red → blue and blue → red). The goal is to achieve a pre-specified configuration of colors. puzzle-3x3-task2 starts with all buttons to be blue, and the goal is to flip exactly one button (the top-left one) to be red. puzzle-3x3-task4 starts with four buttons (topcenter, bottom-center, left-center, right-center) to be blue, and the goal is to turn all the buttons to be blue. The reward is binary: -1 if the desired configuration is not yet reached and 0 if the desired configuration is reached (and the episode terminates).

cube-double/triple/quadruple : These three domains contain 2/3/4 cubes respectively. The tasks in the three domains all involve moving the cubes to their desired locations. The reward is -n wrong where n wrong is the number of the cubes that are at the wrong position. The episode terminates when all cubes are at the correct position (reward is 0).

B.2 Robomimic environments.

We use three challenging tasks from the robomimic domain [43]. We use the multi-human datasets that were collected by six human operators. Each dataset contains 300 successful trajectories. The three tasks are as described as follows.

lift : This task requires the robot arm to pick a small cube. This is the simplest task of the benchmark.

can : This task requires the robot arm to pick up a coke can and place in a smaller container bin.

square : This task requires the robot arm to pick a square nut and place it on a rod. The nut is slightly bigger than the rod and requires the arm to move precisely to complete the task successfully.

a)

scene

b)

puzzle-3x3

c)

cube-double

d)

cube-triple

e)

cube-quadruple

f)

lift

g)

can

h)

square

All of the three robomimic tasks use binary task completion rewards where the agent receives -1 reward when the task is not completed and 0 reward when the task is completed.

C Implementation Details

In this section, we provide more implementation details on both of our Q-chunking methods (QC, QC-FQL) and our baselines used in our experiments. For IFQL, ReBRAC, IQL, we directly use the implementation from Park et al. [58].

Algorithm 1

QC Algorithm 2 QC-FQL

C.1 QC-FQL

Webuild the implementation of our method on top of FQL [58], a recently proposed offline RL/offlineto-online RL method that uses TD3+BC-style objective. It is implemented with a one-step noiseconditioned policy (instead of a Gaussian policy that is commonly used in RL) and it uses a distillation loss from a behavior flow-matching policy as the BC loss. To adapt this method to use action chunking, we simply apply FQL on the temporally extended action space - the behavior flow-matching policy generates a sequence of actions, the one-step noise-conditioned policy predicts a sequence of actions,

and the Q -network also takes in a state and a sequence of actions. More concretely, we train three networks:

Q θ ( s, a 1 , · · · a h ) : S × A h ↦→ R -the value function that takes in a state and a sequence of actions (action chunk). In practice, we train an ensemble of Q networks. We denote the weight of the ensemble element as θ = ( θ 1 , · · · , θ K ) .

µ ψ ( s, z ) : S × R Ah ↦→ R Ah -the one-step noise-conditioned policy that takes in a state and a noise, and outputs a sequence of actions conditioned on them.

f ξ ( s, m, u ) : S × R Ah × [0 , 1] ↦→ R Ah -the flow-matching behavior policy parameterized by a velocity prediction network. The network predicts takes in a state, an intermediate state of the flow and a time, and outputs the velocity direction that the intermediate action sequence should move in at the specified time. See Algorithm 3 for more details on how this velocity prediction network is used to generate an action from a noise vector.

We denote our policy as π ψ ( ·| s ) . It is implemented by first sampling a Gaussian noise z ∼ N (0 , I Ah ) and run it through the one-step noise-conditioned policy [ a 1 · · · a h ] ← µ ψ ( s, z ) . To train these three networks, we sample a high-level transition, w = ( s t , a t , a t +1 , · · · , a t + h -1 , s t + h , r h t ) ∼ D where r h t = ∑ h -1 t ′ =0 γ t ′ r t + t ′ , to construct the following losses:

(1) Critic loss:

where [ a t + h · · · a t +2 h -1 ] ∼ π ψ ( ·| s t + h ) , k ∈ { 1 , 2 , · · · , K } .

(2) Actor loss:

where z t ∼ N (0 , I Ah ) and [ a ξ t · · · a ξ t + h -1 ] is the result of running the behavior policy f ξ ( s, m, t ) with Algorithm 3 from z t , and α ∈ R is a tunable parameter that controls the strength of the behavior regularization (higher α leads to stronger behavior regularization).

(3) Flow-matching behavior policy loss:

Practically, we sample a batch of transitions { w 1 , w 2 , · · · , w M } and optimize the average loss for each network: L ( θ ) = 1 M ∑ M i =1 ∑ K k =1 L ( θ k , w i ) , L ( ψ ) = 1 N ∑ M i =1 L ( ψ, w i ) , L ( ξ ) = 1 N ∑ M i =1 L ( ξ, w i ) .

C.2 FQL

FQL [58] is a recently proposed offline RL/offline-to-online RL method that uses TD3+BC-style objective. It is equivalent to our method with h = 1 . For completeness, we write out the objectives for a transition sample w = ( s t , a t , r t , s t +1 ) :

In practice, we sample a batch of transitions { w 1 , w 2 , · · · , w N } and optimize the average loss for each network: L ( θ ) = 1 N ∑ N i =1 ∑ K k =1 L ( θ k , w i ) , L ( ψ ) = 1 N ∑ N i =1 L ( ψ, w i ) , L ( ξ ) = 1 N ∑ N i =1 L ( ξ, w i ) .

Algorithm 3 FlowODE _ Euler ( s t , z t , f ξ , T ) : generate actions from the behavior flow policy f ξ ( s, m, u ) with Euler's method.

Input: State s t , noise z t and flow model f ξ ( s, m, u ) , number of flow steps T .

m 0 ← z t for i ∈ { 1 , · · · , T } do m i ← f ξ ( s t , m i -1 , ( i -1) /T ) Output: m T .

C.3 FQL-n

To implement the n -step return baseline, we take FQL and replace the 1-step TD update with the h -step TD update:

where z k ′ t ∼ N (0 , I A ) for all k ′ ∈ { 1 , 2 , · · · , K } . The actor loss and flow-matching loss remain the same as FQL.

C.4 QC

The flow-matching behavior policy is trained with the same loss as used in QC-FQL (Equation (22)). On top of the flow-matching behavior policy, we simply parameterize the policy π implicitly by sampling multiple action chunks from the behavior policy and pick the one that maximizes the Q -value. Specifically, let { z 1 t , z 2 t , · · · , z N t } ∼ N (0 , I Ah ) and

The policy outputs the one action chunk out of N that maximizes the Q -value, [ a i ⋆ t · · · a i ⋆ t + h -1 ] , where

Finally, we directly use this implicitly parameterize policy to generate actions for computing the TD target for our TD loss:

where a i ⋆ t + h , · · · , a i ⋆ t +2 h -1 ∼ π ( ·| s t + h ) .

The baselines BFN-n and BFN are implemented similarly to FQL-n and FQL by operating in the original action space.

C.5 RLPD, RLPD-AC, QC-RLPD

All the RLPD baseline results are obtained by running the official codebase (as linked in Ball et al. [7]) with additional modification to incorporate action chunking and behavior cloning. This baseline runs online RL from scratch using off-policy transitions where 50% of them come from the offline dataset and the other 50% come from the online replay buffer. It essentially up-weights the online data more, allowing the online RL agent to learn more quickly. This is different from how QC-* , BFN , BFN-n , FQL , FQL-n samples off-policy transitions (where we sample from the dataset that combines the offline dataset and online replay buffer data with no weighting). RLPD baselines all use Gaussian policy. This is also different from our method as our method uses noise-conditioned policy that can represent a wider range of distributions. For RLPD-AC , we change all the actor and critic networks such that they work with an action chunk rather than a single action. The baseline is exactly the same as our method except that actor and the critic are updated the same as how RLPD

updates its actor and critic. For QC-RLPD , we add a behavior cloning loss in the actor loss as follows (highlighted in red below):

C.6 SUPE-GT

For this baseline, we adapt from a recently proposed skill-based offline-to-online RL method [85]. The original method has additional modules such reward models and random distillation networks to deal with the problem setting where offline dataset is unlabeled ( e.g. , reward information is missing). We remove these two modules as we have ground truth reward information in our offline dataset. The method has two stages: (1) skill-pretraining and (2) online RL with skills. In the first stage, the baseline uses a trajectory V AE to learn latent-conditioned skill policy. In the second stage, the baseline uses RLPD to learn a high-level policy that outputs latent vector to command the latent-conditioned skill policy after every h steps. Such a skill-based method in theory also allows temporally coherent actions for exploration and value backup speedup.

D Experiment Details

D.1 Computational resources

We use NVIDIA RTX-A5000 GPU to run all our experiments. Each complete offline-to-online experiment run takes around 4-7 hours. To reproduce all our results in Table 6, we estimate that it would take around 6 ︸︷︷︸ hours per single run × 15 ︸︷︷︸ # of methods × 25 ︸︷︷︸ # of tasks × 4 ︸︷︷︸ # of seeds = 9 000 GPU hours. Each robomimic experiment run takes longer (around 8-12 hours). While each single run takes longer, we only run a subset of strong methods/baselines on robomimic tasks. We estimate it would take around 10 ︸︷︷︸ × 8 ︸︷︷︸ × 3 ︸︷︷︸ × 5 ︸︷︷︸ = 1 350 GPU hours. In total, we estimate that it would

take around 10 350 GPU hours to reproduce all the main results in our paper.

D.2 Evaluation protocol

Unless specified otherwise, for all methods, we run 4 seeds on each OGBench task and 5 seeds on each robomimic task. All plots use 95% confidence interval with stratified sampling (5000 samples). The success rate is computed by running the policy in the environment for 50 episodes and record the number of times that the policy succeeds at solving the task (and divide it by 50).

D.3 Hyperparameter tuning

QC , BFN , BFN-n . We tune the number of actions sampled, N , for the expected-max Q operator. On OGBench domains, we sweep over { 2 , 4 , 8 , 16 , 32 , 64 , 128 } and select the best parameter for each domain and for each method on task2 . We report the performance of each method with the best α in Table 1 and Figure 1 (on all tasks). Table 5 summarizes the α value we use for each task.

QC-FQL , FQL , FQL-n . We tune the behavior regularization coefficient α . On OGBench domains, we take the default hyperparameter of FQL for each domain α default and tune all methods on task2 of each domain with three choices of α : { α default / 3 , α default , 3 α default } (our α default comes from Table 6 in Park et al. [58]). On robomimic domain, we sweep over much large α values: { 100 , 1000 , 10000 } . We report the performance of each method with the best α in in Table 1 and Figure 1 (on all tasks). Table 4 summarizes the α value we use for each task.

RLPD , RLPD-AC , QC-RLPD . We sweep over (1) whether or not to use clipped double Q-learning (CDQ), and (2) whether or not to use entropy backup. We find that not using CDQ and not using entropy backup to perform the best for all of the RLPD baselines and use that across all domains. Even though our method and the other FQL baselines use K = 2 critic ensemble size, we use K = 10 critic ensemble size for RLPD to keep it the same as the hyperparameter in the original paper [7].

For QC-RLPD , we sweep over behavior regularization coefficient α ∈ { 0 . 001 , 0 . 01 , 0 . 1 } and pick 0 . 01 since it works the best.

SUPE-GT . We tune the KL coefficient for the VAE skill-pretraining from { 0 . 001 , 0 . 003 , 0 . 01 , 0 . 03 , 0 . 1 } and pick 0 . 003 as it works the best.

ReBRAC . We tune the behavior regularization coefficient from { 0 . 01 , 0 . 03 , 0 . 1 , 0 . 3 , 1 . 0 } and pick 0 . 1 as it works the best.

IFQL . We directly use the default hyperparameter from Park et al. [58] ( N = 32 , τ = 0 . 9 ) for this baseline. The first parameter, N = 32 , means that for both online policy rollout and policy evaluation, we sample 32 actions and pick the action that has the highest Q-value. The second parameter, τ = 0 . 9 is the expectile coefficient.

IQL . We directly use the default hyperparameter from Park et al. [58] for this baseline. For cube-* , we use α = 0 . 3 . For scene and puzzle-3x3 , we use α = 10 . 0 . For the expectile coefficient, we use τ = 0 . 9 .

E Full Results

E.1 End-effector visualization

We provide more examples of the trajectory rollouts from QC and BFN over the course of online training on cube-triple-task3 . In Figure 8, we show the first 9000 time steps (broken down into 9 subplots where each visualizes 1000 time steps). In Figure 9, we show another 9000 time steps but late in the training (from environment step 9 × 10 5 ). The first example is the same as the one used in Figure 5.

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

BFN

QC

E.2 OGBench results by individual task

Main results by task. Table 1 and Figure 10 shows the performance breakdown for all methods.

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task5

Steps (×10 6 )

Success Rate

QC

QC-FQL

QC-IFQL

QC-RLPD

BFN-n

FQL-n

IFQL-n

BFN

FQL

IFQL

ReBRAC

IQL

SUPE-GT

RLPD-AC

RLPD

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

puzzle-3x3-sparse

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

scene-sparse

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

cube-double

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

cube-triple

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

cube-quadruple

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

all

Steps (×10 6 )

Success Rate

QC

BFN

FQL

RLPD

IFQL

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-play-task5

Steps (×10 6 )

Success Rate

QC

BFN

FQL

RLPD

IFQL

Selected baselines. Figure 11 shows the results for selected baselines ( e.g. , BFN, FQL, RLPD and RLPD-AC).

n -step return ablation results by task. Figure 12 shows the performance breakdown for Figure 4. Q-chunking with Gaussian policies. The following plot shows the performance breakdown for Figure 2. In addition, we include a new method for comparison, QC-RLPD , where we add a behavior cloning loss to RLPD-AC ( RLPD with action chunking).

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

puzzle-3x3-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

scene-sparse-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-double-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-triple-play-task5

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-task1

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-task2

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-task3

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-task4

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

cube-quadruple-task5

Steps (×10 6 )

Success Rate

QC

QC-FQL

BFN-n

FQL-n

BFN

FQL

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double-play-task1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double-play-task2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double-play-task3

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double-play-task4

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-double-play-task5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple-play-task1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple-play-task2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple-play-task3

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple-play-task4

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-triple-play-task5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple-task1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple-task2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple-task3

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple-task4

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.5

1.0

cube-quadruple-task5

Online Steps (×10 6 )

Success Rate

QC-RLPD

RLPD-AC

RLPD

Action chunking size ablations. Figure 14 includes individual task results for the action chunking size ablation study. We also include results on the hardest cube-quadruple task in Figure 15.

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

cube-triple-play-singletask-task1-v0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

cube-triple-play-singletask-task2-v0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

cube-triple-play-singletask-task3-v0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

cube-triple-play-singletask-task4-v0

0.0

0.5

1.0

1.5

2.0

0.0

0.2

0.4

0.6

0.8

1.0

cube-triple-play-singletask-task5-v0

Steps (×10 6 )

Success Rate

FQL

QC-FQL (h=5)

QC-FQL (h=10)

QC-FQL (h=15)

QC-FQL (h=25)

QC-FQL (h=50)

0.0

0.5

1.0

1.5

2.0

Steps (×10 6 )

0.0

0.2

0.4

0.6

0.8

1.0

Success Rate

cube-quadruple-100M-task4

QC-FQL (h=9)

FQL-n (n=9)

QC-FQL (h=7)

FQL-n (n=7)

QC-FQL (h=5)

FQL-n (n=5)

QC-FQL (h=3)

FQL-n (n=3)

FQL

E.3 Robomimic ablation results

Figure 16 shows the performance of QC , QC-FQL , BFN-n , FQL-n , BFN , FQL our three robomimic tasks. This plot shows the performance breakdown for Figure 4 (right).

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

lift

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

can

0.0

0.5

1.0

1.5

2.0

0.00

0.25

0.50

0.75

1.00

square

Steps (×10 6 )

Success Rate

QC

QC-FQL

BFN-n

FQL-n

BFN

FQL

E.4 How resource efficient is Q-chunking?

In Figure 17, we report the runtime for our approach and our baselines on a representative task cube-triple-task1 . In general, QC-FQL has a comparable run-time as our baselines (e.g., FQL and RLPD ) for both offline and online. QC is slower for offline training as it requires sampling 32 actions for each training example for the agent update ( BFN is faster because it only needs to sample 4 actions). For online training, we are doing one gradient update per environment step, and it makes QC only around 50% more expensive than other methods.

Finally, we include the parameter count of each of these method on the representative domain cube-triple-* in Table 7 below (assuming h = 5 for all Q-chunking methods).

QC

QC-FQL

BFN

FQL

12.02

4.24

3.33

2.97

)

QC

QC-FQL

BFN

FQL

RLPD

15.07

11.33

11.59

10.06

11.03

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Q-Learning with Action Chunking (Q-LAC), a reinforcement learning method that models action chunking by enabling the policy to output a sequence of $h$ actions from a given state. The critic is trained to evaluate the value of a state along with the $h$-step action sequence. The approach first trains in an offline dataset and then further trained through additional online learning, formalizing an offline-to-online setting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is clearly written and well-organized.\n2. The idea of action chunking policy and a chunk-based Q-function is interesting and rarely explored in the RL context.\n3. The literature review is comprehensive.\n\n\n**Weaknesses**\n\n1. **The claimed advantages of action chunking over standard 1-step action policies and skill-based approaches are not convincing.** Action chunking breaks the Markov property and weakens the tight feedback loop between policy and environment. Let's denote action chunking policy as ACP and the normal 1-step action policy as N1P for convenience. Intuitively, ACP appears less favorable than N1P, it sacrifices reactivity and granularity, and may even be viewed as a degenerate case of N1Ps:\n\n   * N1P has finer granularity and higher real-time responsiveness. In ACP, no matter how the future states change, the policy blindly executes the pre-predicted chunk. There's no guarantee that the latter actions in the chunk are reasonable for the future states. In contrast, N1P reacts to every state, which is clearly more rational.\n   * If ACP is to be responsible for future $h$-step actions, under the same model scale, it requires the policy to have some foresight and predictive capability about the environment's transition model, i.e., some form of internal model-based reasoning. This increases modeling difficulty.\n   * Under the Markovian assumption: $\\pi^{Markovian}: s_t \\rightarrow a_t$, the policy outputs a single action based on the current state (given less information, do less behavior); Under non-Markovian assumption: $\\pi^{non-Markovian}: s_{t-n:t} \\rightarrow a_t$, the policy outputs a single action based on the history of states (given more information, do less behavior). In contrast, ACP outputs multiple actions based on one state: $\\pi^{ACP}: s_t \\rightarrow a_{t:t+h-1}$ (given less information, do more behavior). Clearly, ACP is the least favorable case. \n   * Furthermore, compared to skill-based methods (as mentioned at around Line 116), while skill-based methods are still step-wise policies internally, they can be interrupted or switched anytime, ACP, on the other hand, enforces hard commitment to an action sequence and cannot be interrupted, which is less flexible than skill-based methods.\n\n2. **Limited Contribution**. The proposed method can be considered as adding action chunking to the FQL baseline [1]. However, as noted in point 1, the theoretical justification and claimed benefits of action chunking are not well established. Simply integrating action chunking into FQL, without a solid theoretical grounding, offers limited innovation and contribution. Moreover, the reported performance comparisons with FQL deviate notably from those reported in the original FQL paper (see detailed Question 1 below), raising my concern about the claimed improvements empirically.\n\n3. In Figure 2, we can see that AC performs worse than the RLPD baseline in online RL, validating the above concerns. To address this, the paper introduces offline training and behavior regularization, leading to the results in Figure 3. However, these results don't convincingly demonstrate that Q-LAC's performance is due to action chunking or behavior regularization. It seems more likely that the offline training provides a better starting point for the policy (QLAC starts from around 0.3, while RLPD starts from 0), but RLPD quickly reaches the same performance. From this perspective, one main limitation of Q-LAC becomes that it relies heavily on offline data and behavior regularization. Therefore, I tend to view that the use of Q-LAC in offline-to-online scenarios seems to be more of a necessary compromise than a design choice.\n\n4. **Experimental comparisons raise several concerns:**\n\n   * As shown in Figure 4, in about half of the tasks, RLPD achieves performance equal to or better than Q-LAC. This again suggests that Q-LAC’s advantage likely comes from offline pretraining. For example, in the puzzle environment, Q-LAC already reaches the performance peak after offline training. I don’t think this convincingly demonstrates the advantage of Q-LAC, as it only shows improved sample efficiency in online learning, at the cost of more offline training.\n   * The core contribution of the paper is action chunking, but this point is not well supported by the experiments. If the impact of offline pretraining could be isolated, the results would be more convincing. Worryingly, Figure 2 shows that Q-LAC performs worse than the RLPD baseline in pure online settings; in Figure 4, in half of the environments (puzzle, robomimic, cube-quadruple), FQL-n performs better than FQL; and in all tasks, RLPD consistently outperforms RLPD-AC5. These results seem to contradict the claimed benefit of action chunking.\n   * The number of baselines (3) and evaluated tasks (6) is relatively small. For example, one of the baselines, FQL compared with 10 baselines on dozens of domains, which is a better experimental practice. Besides, experiments in Figure 5-7 only include one or two tasks. More comprehensive comparisons would strengthen the paper.\n\nIn summary, I think the action chunking policy proposed in this paper is an interesting idea, but it lacks sufficient theoretical support regarding MDPs, the Markov property, the Bellman equation, and policy convergence or optimality under this paradigm. The experimental results also fail to show clear advantages of action chunking itself. Therefore, I believe the contribution of this paper is limited.\n\n[1] Park, Seohong, Qiyang Li, and Sergey Levine. \"Flow Q-Learning.\" ICML, 2025.\n\nQUESTIONS:\n1. How were the baselines fine-tuned in the experiments to ensure they are performing properly? For example, in the original FQL paper (see Figure 12), FQL achieves near-optimal performance (close to 1.0) on tasks such as cube-double and scene. However, in this paper's Figure 4, FQL only reaches around 0.6 on these tasks. What accounts for this difference in performance?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Q-LAC, an offline to online RL algorithm that employs action chunking to accelerate temporal-difference value updates and promote temporally coherent exploration. Action chunking is a common algorithm in the field of embodied intelligence, and it is highly valuable to study its role in the domain of reinforcement learning. Empirical evaluations across a suite of challenging long-horizon, sparse-reward manipulation tasks demonstrate that Q-LAC outperforms prior approaches in both offline and online RL phases, particularly excelling in sample efficiency during fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n- The motivation behind this article is excellent. It is highly meaningful to investigate the role of action chunking, which is commonly used in embodied intelligence algorithms, within the realm of reinforcement learning. \n\n- The proposed algorithm is concise and effective, with strong versatility.\n\n\n**Weaknesses** : The overall discussion and analysis in this article are not profound enough, and there are many points that could be expanded upon.\n\n- There is a lack of theoretical analysis regarding the impact of action chunking on online RL. Additionally, while the speedup and unbiased value backup claims are explained in Section 4.3, no formal analysis or bounds are provided.\n\n- When the action chunking size is 5, how much deviation is there between the five actions obtained by $a_{1,2,3,4,5} = \\pi_{ac}(s_1)$ and the five actions obtained by $a_{i} = \\pi(s_{i})$? How does this deviation affect the final performance and the two benefits?\n\n- The article mentions that the learning of value is accelerated, but there is no quantified indicator or experiment to measure the extent of this acceleration.\n\n- The ablation experiment on action chunking size is incomplete. Intuitively, as the size increases, the subsequent actions will become less accurate. Therefore, finding this critical point is crucial.\n\n- The article lacks analysis on whether action chunking will bring additional memory or time overhead.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThe topic of the article is highly valuable, yet many aspects could be explored more profoundly. Should the authors address my concerns, I would be more than willing to consider a higher rating.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method integrating action chunking within an Actor-Critic framework. ​The approach demonstrates improvements​ in value backup efficiency for accelerated learning while generating temporally coherent actions to enable effective online exploration. ​Experimental results provide evidence​ for the method's sample efficiency compared to baselines, notably in offline-to-online RL settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros**:\n1. The paper is well-structured and effectively presents the main idea.\n2. Integrating the action chunking into both critic learning and policy optimization process demonstrate its effectiveness on improving the sample efficiency especially in the online learning part.\n\n**Cons:**\n1. The core contribution appears largely incremental, primarily combining action chunking with FQL. Rigorous ablation studies are needed to isolate the individual efficacy of each design component. (e.g., Does action chunking generalize as a performance enhancer when integrated with ​​other​​ offline-to-online RL algorithms?)\n\n2. The computational demands appear significant, yet evaluation was restricted to limited tasks per OGbench environment. Comprehensive performance metrics (e.g., aggregated scores across all tasks) are absent, precluding a robust assessment of Q-LAC's overall effectiveness.\n\n3. It seems like in offline \"scene\" task results Q-LAC exhibits the highest variance among baselines, potentially undermined its online performance, yet this critical limitation lacks sufficient analysis regarding its cause and impact.\n\n3. The manuscript lacks substantive discussion or illustration explaining why action chunking is particularly well-suited to non-Markovian settings or adequately addresses potential side effects.\n\n4. Formal theoretical analysis is absent. Such analysis would significantly strengthen claims regarding the method's effectiveness and robustness.\n\nQUESTIONS:\n1. Why does the action chunking formulation yield limited performance improvements in strongly Markovian or fully Markovian situation?\n\n2. Equation 8 claims that action chunking avoids bias in value estimation. What is the associated impact on estimation variance, and how does this affect the stability and convergence of the learning process?\n\n3. How does Q-LAC scale to environments with high-dimensional action spaces? Does effective Bellman updating with chunked high-dimensional inputs necessitate more complex Q-network architectures (e.g., deeper networks, attention mechanisms), and what computational trade-offs arise?\n\nLIMITATIONS:\nYes, the author has included limitation discussion in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an actor-critic method for offline-to-online reinforcement learning. The proposed method, Q-learning with Action Chunking (Q-LAC), uses action chunks or macro-actions for learning (macro) action-values and acting in the environment. \n\nThe authors demonstrate that using action chunks with a relatively more straightforward actor-critic approach (RLPD) can lead to a deterioration in performance. Instead, they suggest building on top of Flow Q-learning (FQL), where action chunks can be constrained to be similar to the behavior observed in the offline data.\n\nExperiments in the offline-to-online setting show that the proposed approach outperforms prior methods (RLPD and FQL) on the more challenging domains from the OG-bench.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The paper presents a relatively simple extension of FQL to incorporate temporally extended actions. \n- Further, this extension (Q-LAC) shows promising results on the OG-bench domains considered. In particular, it performs significantly better than relevant baselines (FQL, RLPD) on the more challenging tasks, where a long horizon and sparse reward pose the greatest challenge.\n\n### Weaknesses\n\n**W1.** A weakness of the present submission is in terms of novelty. Q-LAC feels like an incremental contribution on top of FQL. Perhaps more problematically, this connection is not really addressed before sections 4.2 and 5, even after which the exact relationship remains unclear. Could the authors clearly specify the differences between FQL and Q-LAC? Using the terminology of the paper, would it be correct to say Q-LAC = FQL-AC5?\n\n**W2.** Also related to novelty, the benefits of macro-actions in terms of more temporally coherent exploration and improved credit assignment are generally known (e.g. [1, 2, 3]), limiting the novelty of section 4.3. Lines 35 and 36 claim that chunking has not been widely used in RL; however, it has been employed through macro-actions (as noted in previous references) and for credit assignment [4]. \n\n**W3.** There are some aspects of the empirical analysis that would benefit from clarification.\n\nThe authors consider an FQL-n baseline that uses (biased) n-step returns. Due to the difference of $n$ for backup vs action chunking, it would also be helpful to see the performance of **FQL-n** for different choices of $n$, e.g., $n \\in \\{3, 5, 7, 9 \\}$. Could a different choice of $n$ match the performance of Q-LAC? FQL-5 also starts off quite well on cube-quadruple-100M, is there a reason for the sudden collapse?\n\nAnother question I had was regarding RLPD vs FQL. The FQL paper’s results show that FQL outperforms RLPD in the offline-to-online case, whereas the results in this paper (cube-double, cube-triple, robomimic from Figure 4, and aggregated results from Figure 1) show the opposite trend. Could the authors explain why this is the case?\n\n—-----------—-----------—-----------—-----------—-----------\n\nOn a minor note, it would be helpful to include some more details about FQL in the main paper’s background section or section 4.2. Perhaps sections 4.3, 5.1, and 5.2 could be compressed, and some of those details could be moved to the appendices.\n\nOverall, I appreciate the simplicity of the proposed method and the promising results (strengths), but I think the paper should adjust some claims related to novelty and further discuss related work (W1/2). I also have concerns related to the empirical analysis (W3).\n\nI remain open to increasing the score if the weaknesses and questions are adequately addressed and clarified.\n\nTypos: In the legend of Figure 7. D^comp instead of D^comb.\n\n#### References\n\n[1] McGovern, Amy, and Richard S. Sutton. \"Macro-actions in reinforcement learning: An empirical analysis.\" (1998)\n\n[2] Durugkar, Ishan P., et al. \"Deep reinforcement learning with macro-actions.\" arXiv preprint arXiv:1606.04615 (2016).\n\n[3] Tan, Aaron Hao, et al. \"Deep reinforcement learning for decentralized multi-robot exploration with macro actions.\" IEEE Robotics and Automation Letters 8.1 (2022)\n\n[4] Ramesh, Aditya, et al. \"Sequence compression speeds up credit assignment in reinforcement learning.\" ICML (2024).\n\nQUESTIONS:\nIncluded in “Strengths and Weaknesses”\n\nLIMITATIONS:\nLimitations are touched in Discussions section, but could be elaborated upon.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}