Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
Authors: Younggyo Seo
Pieter Abbeel
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Action Sequence
Humanoid
Primary area: reinforcement_learning
TL;DR: We introduce CQN-AS, a value-based RL algorithm that learns a critic network that outputs Q-values over a sequence of actions.

Abstract: Predicting a sequence of actions has been crucial in the success of recent behavior cloning algorithms in robotics. Can similar ideas improve reinforcement learning (RL)? We answer affirmatively by observing that incorporating action sequences when predicting ground-truth return-to-go leads to lower validation loss. Motivated by this, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based RL algorithm that learns a critic network that outputs Q-values over a sequence of actions, i.e., explicitly training the value function to learn the consequence of executing action sequences. Our experiments show that CQN-AS outperforms several baselines on a variety of sparse-reward humanoid control and tabletop manipulation tasks from BiGym and RLBench.

Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning
Younggyo Seo UC Berkeley mail mail@younggyy
Pieter Abbeel
UC Berkeley pabbeel@cs berkeley. edu
Abstract
Predicting sequence of actions has been crucial in the success of recent behavior cloning algorithms robotics. Can similar ideas improve reinforcement learning (RL)? We answer affirmatively by observing that incorporating action sequences when predicting ground-truth return-to-go leads to lower validation loss. Motivated by this, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based RL algorithm that learns a critic network that outputs Q-values over sequence of actions, i.e., explicitly training the value function learn the consequence of executing action sequences. Our experiments show that CQN-AS outperforms several baselines on a variety of sparse-rewan humanoid control and tabletop manipulation tasks from BiGym and RLBench. Code is available at: https: / hywu/gyouogoooooo/o me/cmnm-..........e.eeeeeee ccpp/a/g//////////a
BiGym
BiGym (25 Tasks)
RLBench (20 Tasks)
100
100
Success Rate (%)
Success Rate (%)
75
75
50
25
RL RLBench
2c4
4e4
6e4
Be4
1e5
1e4
2o4
3e4
Environment
Environment Steps
RL: CQN-AS (Ours)
CQN
DrQ-v2+
BC:
ACT
Figure 1: Summary results. Coarse-to- Q-Network with Action Sequence (CQN-AS) learns critic network with action sequence. CQN-AS outperforms various RL and BC baselines such as CQN (Seo et al., 2024), DrQ-v2+ (Yarats al., 2022), and ACT (Zhao al., 2023) on robotic tasks from BiGym (Chernyadev al., 2024) and RLBench (James et al., 2020).
1 Introduction
Predicting action sequences from expert trajectories is key idea in recent successful behavior cloning (BC; Pomerleau 1988) approaches in robotics. This has enabled policies effectively approximate the noisy, multi-modal distribution of expert demonstrations (Zhao et al., 2023; Chi et al., 2023). Can this idea similarly be useful for reinforcement learning (RI.)?
Our initial finding is affirmative: we make an intriguing observation that using action sequences can enhance value learning. Specifically, with humanoid demonstrations from BiGym (Chernyadev et al., 2024), we train regression models that predict the ground-truth return-to-ge i.c., the sum of discounted future rewards from the timestep f, given the current observation and action. In Figure 2a, we find that using an action sequence a {a,,... - as input results in lower validation
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Improvement L1 Loss (Seq. Length 1 50)
1e10
7.5e9
SAC
SAC-AS2
SAC
shle
18.1%
5ce
2. 2.5e9
TD3
CQN
drww Jep. nom1
14.0%
Targel Q-Values
2e5 4e5
665
8e5
1e6
No-op added
Return
Value
Return Value
caposants www.o...s..oo .II
Environment Sleps
19.1%
7.5e9
1e10
T03-A94
54
0
140.7
7.18
-17.45
16.25
219.5
236.6
20.3
18.8
wall suppscarr_.eoer
20.1%
2.5eg
294
144
0.56
0.27
4E8
1E8
202.9
185.4
16.4
18.7
2+5
4e5
665
865
1e6
Improvement (%)
Environment Steps
(a) Return-to-go prediction
(b) Value overestimation stand
(c) Effect of no-op actions
Figure 2: Analyses. (a) We measure the improvement in the validation LI loss of the return- to--o prediction model with different action sequence lengths. We find that using action sequence of length 50 results in the lower loss than using single-step action. (b) We find that SAC and TD3 with action sequences suffer from severe value overestimation in stand task from HumanoidBench, which leads to random near-zero performance. (c) Actor-critic algorithms like TD3 become vulnerable to value overestimation when redundant no-op actions are added to the action space. In contrast, a critic-only algorithm that uses discrete actions, CQN, is robust with high-dimensional action spaces.
losses than using single-step action a,. We hypothesize this is because action sequences, which can correspond to behavioral primitives such as going straight, make it easier for the model to learn the long-tee term outcomes compared evaluating the effect of individual single-step actions (see Appendix A for additional analysis based on a 2D Point-mass environment).
Building on this observation, train actor-critic algorithms (Haarnoja al., 2018: 8; Fujimoto et al., 2018) with action sequence on stand task from HumanoidlBnch (Sferrazza et al., 2024). Specifically, train the actor output action sequence and the critic to take action sequence inputs instead of single-step actions. However, we find these algorithms with action sequences suffer from severe value overestimation (see Figure 2b) and completely fail to solve the task. This is because a wider action space makes the critic more vulnerable to function approximation error (Fujimoto et al., 2018) and the actor excessively maximizes value functions by exploiting this estimation error. To further support this, in Figure 2c, we report additional toy experiments where we introduce redundant no-op actions for training TD3 agents on Cheetah Run task (Tassa al., 2020). Here, we find that actor-critic algorithms are indeed vulnerable to value overestimation with high- nigh-dimensional action spaces.
This result motivates algorithm, i.e., Coarse-to-fine Q-Network (CQN; Seo et al. 2024), which solves continuous control tasks with discrete actions. Because there is separate actor that may exploit value functions, i.e., CQN simply selects discrete actions with the highest Q-values, we find that training with action sequences stable and thus avoids value overestimation problem (see Figure 2c). In particular, we introduce Coarse-to-fine Q-Network with Action Sequence (CQN-AS), which learns a critic network that outputs Q-values over a sequence of actions (see Figure 3). By training the critic network to explicitly learn the consequence of taking a sequence of current and future actions, CQN-AS enables the RL agents effectively learn useful value functions on challenging robotic tasks.
design our RL algorithm with action sequence upon recent critic-only
Our experiments show that CQN-AS improves the performance of CQN on sparse-reward humanoid control tasks from BiGym benchmark (Chernyadev et al., 2024) that provides human-collected demonstrations and sparse-reward tabletop manipulation tasks from RL.Bench (James et al., 2020) that provide demonstrations generated via pootion-planning.. Considering that CQN-AS a critic-only algorithm that selects actions with the highest Q-value without a separate actor network, these results highlight the benefit of using action sequences in value learning.
Our contributions can be summarized as below:
We make observation that shows using action sequences can be useful for RL by enhancing value learning. We also show that standard actor-critic algorithms (Haarnoja et al., 2018; Fujimoto et al., 2018) suffer from value overestimation when trained with action sequences.
We introduce Coarse-to- foefne Q-Network with Action Sequence (CQN-AS) that trains critic network to output Q-values over action sequences. This critic-only algorithm successfully avoids value overestimation problem and enhances the base CQN algorithm.

---

Sequeres Step1
Sequenee Stop K
Zoam
Zeom in 16
Smaa&& & mrae
4
(an")
Selection
Selection
MLP
MLP
Lese indec
Decretostion
Decrrtontion
Dicreticet on
Bacretrzancn
Level
Level
Level2
Irputs
Actions Proviuss
Coarsc-to-fine inference procedure
(h) Architecture
Figure 3: Coarse-to-Fine Q-Network with Action Sequence. CQN-AS extends Coarse-to-Find Q- Network (CQN: Seo al. 2024), a critic-only RL algorithm for continuous control using discretized actions. (a) CQN progressively zooms into the action space by discretizing it into B bins and finding the bin with the highest Q-value further discretize at the next level. Last level's action sequence is used for controlling robots. CQN-AS generalizes this to action sequences by computing all K actions in parallel. (b) We train critic to predict Q-values over entire action sequences by extracting per-step features and aggregating them with recurrent network before projection to Q-values.
a demo-driven RI. setup that initializes training with expert demonstrations, we show that CQN-AS surpasses the performance of ACT (Zhao et al., 2023) BC algorithm that trains Transformer (Vaswani et al., 2017) to predict action sequences.
2 Preliminaries
We formulate a robotic control problem partially observable Markov decision process (Kaelbling et al., 1998; Sutton & Barto, 2018). At time step t, an RL agent encounters an observation o, executes an action de, receives reward re+1, and encounters new observation 0++1 from the environment. We aim to train a policy that maximizes the expected sum of rewards through RI. while using as few online samples as possible, optionally with access to modest amount of expert demonstrations.
Inputs and encoding Given visual observations of = {o," ..... O O.M from M cameras, encode each o using convolutional neural networks (CNN) into h, We then process them through a series of linear layers to fuse them into hς- If low-dimensional observations of available along with visual observations, we process them through a series of linear layers to obtain h We then use concatenated features h, [h; hooo] as inputs the critic network. In domains without vision low as h, without encoding the low-dimensional observations.
sensors, we simply use of
Coarse-to-fine Q-Network Coarse-to-inn Q-Network (CQN; Seo al. 2024) is critic-only RL algorithm that solves continuous control tasks with discrete actions. CQN trains an RI. agent to learn select coarse discrete actions in shallower levels with larger bin sizes, and then refine their choices by selecting finer-grained actions in deeper levels with smaller bin sizes. Specifically, CQN iterates the procedures of (i) discretizing the continuous action space into multiple bins and (ii) selecting the bin with the highest Q-value to further discretize. This reformulates the continuous control problem as a multi-level discrete control problem, allowing for the use of ideas from sample-efficient discrete RL algorithms (Mnih et al., 2015; Silver et al., 2017) for continuous control.
Formally, let a be an action level with of being the zero vector! We then define the coarse-to-fie to-fne critic to consist of multiple Q-networks which compute Q-values for actions at each level of. given the features h, and actions from the previous level af-1 as follows:
(b., (br,, of = [6b (br. of d a (
h-1 ER
For simplicity, we describe CQN and CQN-AS with a single- dineel-ommnsionall action in the main section. See Appendix C for full description with N dimensional actions.

---

where J.b denotes an action for each bin and B is the number of bins for each level. We note that CQN uses scalar values representing the center of each bin for previous level's action a enabling the network to locate itself without access to all previous levels' actions. We optimize each Q-network at level I with the following objective:
C' = (hs, a.,a. r++1 -y 2/ (hi+1, x x'-'(((t1)))
where & are delayed parameters for target network (Polyak & Juditsky, 1992) and n' is policy that outputs the action az at each level / via the inference steps with our critic, i.e., X r'(h)) de- Specifically, output actions at time step CQN first initializes constants a; oz and a; high with -1 and 1. Then the following steps are repeated for E T.):
Step 1 (Discretization): Discretize interval [at00 dz high into B uniform intervals, and each of these intervals become action space for Øz-
Step 2 (Bin selection): Find a bin with the highest Q-value and set of to the centroid of the bin.
Step 3 (Zoom-in): Set at and at
high to the minimum and maximum of the selected bin, which intuitively can be seen zooming-into each bin.
We then use the last level's action af as the action at time step I. For more details, including the inference procedure for computing Q-values, we refer readers to Appendix C.
3
Method
We present Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a value-based RL algorithm that learns critic network that outputs Q-values for sequence of actions 0t+++ {at, .... Oc+ K-1 for a given observation o,, Our main motivation comes from one of the key ideas in recent BC approaches: predicting action sequences, which helps resolve ambiguity when approximating noisy distributions of expert demonstrations (Zhao et al., 2023; Chi et al., 2023). Similarly, by explicitly learning Q- valuuuss of a sequence of actions from the given state, our approach mitigates the challenge of learning Q values with noisy trajectories. We provide the overview of CQN-AS in Figure 3.
3.1 Coarse-to-fine Critic with Action Sequence
Objective Let af {a!,..., K- be action sequence at level and of o be zero vector. We design our coarse-to to-fine critic network to consist of multiple Q-networks that compute Q-values for each action at sequence step le (1, K} and level I (1,...,
0/²h (h,, (erkk)k)) = [Qb Lk (he 1 o.t oettk) )
E R
b=1
1.b where d++k-1 denotes an action for each bin b step k. We optimize our critic network with the following objective:
(00* (hr, (h ah 1-1 1..tk)
Σ Te4: max 12 (h441, i d' F 1-1 (h++1))²
(2)
where is a hyperparameter for N-step return and T' is an action sequence policy that outputs the action sequence al.+ by following the similar inference procedure as in Section 2 (see Figure 3a). In practice, we compute Q-values for all sequence step (1...., K) in parallel, which is possible as Q-values for future actions depend only features h, but not previous actions.
Remarks on objective with N-step return We note that any N-step return can be used in Equa- tion 2 because the network can learn the long-term value of outputting action d4+; from bootstrapping. There is a trade-off: if one considers short N-step return, it can cause challenge as the setup becomes a delayed reward setup; but training with higher N N-step return may introduce variance (Sutton & Barto, 2018). In our considered setups, empirically find that using common values N {1,4) 43 works the best. We provide empirical analysis on the effect of N in Figure 8c.

---

Figure 4: Examples of robotic tasks. We study CQN-AS on 25 humanoid control tasks from BiGym (Chernyadev et al., 2024) and 20 tabletop manipulation tasks from RLBench (James et al., 2020).
Architecture Our critic network initially extracts features for each sequence step l and aggregates features from multiple steps with recurrent network (see Figure 3b). This architecture is helpful in cases where single-step action is already high nigh-dimensional so that concatenating them make inputs too high-dimensional. Specifically, let eg denote one-hot encoding for k. At each level 1, construct features for each sequence step k as h
h, d ct]- We then encode each 11,, with a shared MLP network and process them through GRU (Cho et al., 2014) to obtain s' [ ..... ..., f (h k.)) We find that this design empirically performs better than directly giving actions as inputs to GRU. We then use a shared projection layer to map each sy into s8 (s( (s),
I.b 1-1 Q-values at each sequence step k; Le., [40 (he, 1 CEtt k)
3.2 Action Execution and Training Details
Executing action with temporal ensemble With the policy that outputs an action sequence Ot.t+KK one question how to execute actions at time step E [t...., + K - -].. For this, we use temporal ensemble (Zhao et al., 2023) that computes dttt*K every time step, saves it to buffer, and executes a weighted average >. w,a;/ / Σ w, where aj denotes an action for step & computed at step t - i, w; exp(-m i) denotes weight that assigns higher value to more recent actions, with m hyperparameter that adjusts the weighting magnitude. We find this scheme outperforms the alternative of computing dt:++K every K steps and executing each action for subsequent K steps most tasks we considered, except on several tasks that need reactive control.
Storing training data When storing samples from the environment, we store a transition (o,,ū, Tr+1, oc+1) where â denotes an action executed time step For instance, if we use temporal ensemble for action execution, à; is weighted average of action outputs obtained from previous K time steps, i.e., å, E, w Σ Wt.
Sampling training data from a replay buffer When sampling training data from the replay buffer, we sample transition with action sequence, i.e., (Or, 0rz+K 1, Oz+1). If we sample time step t near the end of episode so that we do not have enough data to construct a full action sequence, we fill the action sequence with null actions. In particular, position control where we specify the position of joints or end effectors, we repeat the action from the last step so that the agent learns not to change the position. In torque control where specify the force apply, we set the action after the last. step to zero so that agent learns to not to apply force.
Experiment
We study CQN-AS on 25 humanoid control tasks from BiGym (Chernyadev et al., 2024) and 20 tabletop manipulation tasks from RLBench (James et al., 2020) (see Figure 4 for examples of robotic tasks). To focus on challenging robotic tasks that aim to induce policies generating realistic behaviors, we consider practical setup of demo-driven RL where we initialize training with a modest amount of expert demonstrations and then train with online data. In particular, our experiments are designed to investigate the following questions:
Can CQN-AS quickly match the performance of a recent BC algorithm (Zhao et al., 2023) and surpass it through online learning? How does CQN-AS compare to previous model-free RI. algorithms (Yarats et al., 2022; Seo et al., 2024)?
What is the effect of each component in CQN-AS?
Under which conditions is CQN-AS effective?
5

---

Move Patte
Move Two Flates
Sauceden To Hop
Sandwih Fli
Sandwith Rarrov
Succes Rato (S)
100
100
100
100
100
75
75
75-
75-
75-
mm
50
264 464 Gell
204 Aat Gmd Red
2e0 4ef 6nd
204 ded
Tel
Bell
204 And
Bieressberr Load Prates
Detterabber Lowd Cupa
Dattwaadhee Unioord Curlery
Toln Chipe
Phrt Capps
Suoods Rats (ha)
-00
75
16
0
2wd
And Awl
Swl
Ted
2w4
4nd
Bel
Twl
Twb
2wl Aef
Bie
Beil 1w5
Zed 4wd
Fart Bai 1ed
01 2 Aed Gad Fwd rad
Flp
Flip Cureery
Distweshher Cloco Trays
Ouphoarts Cose
Reach Torget Sinle
Success Rac
264
406
6o4 5o4 160
264
604
624
564
165
264
404
604 B64
165
254
404
6o4
854
106
204 404
664
Bod 10
Reach Targed Mudi Modell
Reach Targel Dual
Didrmealher Choas
Wall Cupboard Open
Drawera Open
Sucoess Raso (8)
15
50
25
25
2ed
464
Ball
Se4 1eb
204 Gac
Bal
Sa4
2ur Sea Aadi
Tec 2ur
Sel
4af bud
1:4. 204 3rl Auf
Sad
Will
Success Rate (N)
75
75-
75
82-
50-
50
00-
50-
18
26
25
25
26
1.504 Envromment Staps
1e6 1.504
2.504
Enwtromment Stape
Envronment Steps
Envrrrment Staps
Emvrrnment Stape
RL:
CON-AS (Ours)
CON
DrQ
BC:
ACT
Figure 5: BiGym results on 25 sparsely-rewarded mobile bi-manual manipulation tasks. All RL algorithms trained from scratch, with replay buffer initialized with 17 60 human-collected demonstrations, and with an auxiliary BC objective. We report the success rate over 25 episodes. The solid line and shaded regions represent the mean and confidence intervals, respectively, across runs.
Baselines We consider model-free RL baselines that learn deterministic policies, we find that stochastic policies struggle to solve fine-grained manipulation tasks. Specifically, we consider (i) Coarse-to-fine Q-Network (CQN; Seo et al. 2024), our backbone algorithm and (ii) DrQ-v2+, an optimized demo-driven variant of an actor-critic algorithm DrQ-v2 (Yarats et al., 2022) that uses a deterministic policy algorithm and data augmentation. We further consider (iii) Action Chunking Transformer (ACT; Zhao et al. 2023) that trains a transformer (Vaswani et al., 2017) policy to predict action sequences from expert demonstrations and utilizes temporal ensemble,
BC baseline.
Implementation details For training with expert demonstrations, we follow the setup of Seo et al. (2024). We keep separate replay buffer that stores demonstrations and sample half of training data from demonstrations. We also relabel successful online episodes as demonstrations and store them in the demonstration replay buffer. For CQN-AS, based on large margin loss (Hester et al., 2018). For actor-critic baselines, we use an auxiliary BC loss that minimizes 1.2 loss between the policy outputs and expert actions.
auxiliary BC loss from Seo et al. (2024)
4.1 BiGym Experiments
We study CQN-AS on mobile bi-manual manipulation tasks from BiGym (Chernyadev et al., 2024). BiGym's human-collected demonstrations are often noisy and multi-modal, posing challenges for RL algorithms. These algorithms must effectively leverage the information within demonstrations to learn strong initial behaviors, thereby mitigating exploration difficulties in sparsely rewarded tasks.
Setup We consider 25 BiGym tasks with 17 to 60 demonstrations We use RGB observations with 84x84 84 resolution from head, left_ ttiistttttt.t and right_wriie wrist cameras. We also use low- dimensional proprioceptive states. We use (i) absolute joint position control action mode and (ii)
2 BiGyy benchmark provides different number of successful demonstrations for each task. But we use the number of demonstrations for all algorithms. See Appendix B for more details.

---

S,cces Rat (50
100
Take Lid Off Baucepan
Open Drower
100
Stack Wine
100
Tollet Seat Up
100
Open Marowase
70
76-
70-
75-
75-
50
50
50
50-
50-
%
1u4
%
9,00ss Rat (10
Open Oven
1olato Plaze Off Cdlored Dish Rock
1.00
Turn Top
Put Money Safe
100
Phono On Basp
70
76
70-
75-
75
50
50
50
50-
50
%
1e4
Twe
Jel
%
Tell
364
2wl
Swc
1ed
2hd
Sicces Ras (83
100
Fut Books Booecheff
100
Swecep To Ductpon
100
Fick Up Cup
100
Open Ooor
100
Nezz On Grill
75
75
75-
71-
75
82
50
0
0
1n4
3e4
Twl
3ed
0
Sw4
Ted: Insert in Computer
Success Rate 150
100
Baskerball In Hoop
100
Lamp
100
Fress Switch
100
Put Rutbiih
100
73
75
75-
75
75
8
50
18
Twe
Tell
Sleps
RL:
CON-AS (Ours)
CON
DrQ+2+
BC:
ACT
Figure 6: RLBench results on 20 sparsely-rewarded tabletop manipulation tasks from RLBench (James et al., 2020). All RI algorithms are trained from scratch, with replay buffer initialized with 100 synthetic demonstrations generated via motion-planninn and with auxiliary BC objective. As expected, with synthetic demonstrations, CQN-AS achieves similar performance to CQN on most tasks. However, CQN-AS often significantly outperforms baselines several challenging, long-horizon tasks such as Open Oven. We report the success rate over 25 episodes. The solid line and shaded regions represent the mean and confidence intervals, respectively, across 4 runs.
floating base that replaces locomotion with classic controllers. We use the same set hyperparameters for all the tasks. Details BiGym experiments available in Appendix B.
Comparison to baselines Figure 5 shows that CQN-AS quickly matches the performance of ACT and outperforms it through online learning on most tasks, while other RL algorithms fail to do so especially on challenging Long-horizon tasks such as Move Plate and Saucepan To Hob. A notable result here that CQN-AS enables solving challenging BiGym tasks while other RL baselines completely fail as they achieve 0% success rate on many tasks.
Limitation However, CQN-AS struggles to achieve meaningful success rate on some of the long- horizon tasks that require interaction with delicate objects such cups cutlery. This leaves room for future work to incorporate advanced vision encoders (He et al., 2016) or critic architectures (Chebotar et al., 2023; Springenherg al., 2024).
4.2 RLBench Experiments
also study CQN-AS on manipulation tasks from RLBench (James et al., 2020). Unlike BiGym, RLBench provides synthetic demonstrations generated via motion planning, which cleaner and more consistent. This allows us to examine whether CQN-AS is also effective in settings with clean, unambiguous demonstrations where the effeet of each single-step action is casier to interpret.
Setup We use the official CQN implementation for collecting demonstrations and reproducing the baseline results on the same of tasks. We use RGB observations with 84x84 resolution from front, wrist, left_sshuutllle shoulder, and right_shoulder cameras. We also use low-dimensional proprioceptive states consisting of 7-dimensional joint positions and binary value for gripper open. We use 100 demonstrations and delta joint position control action mode. We use the same set of hyperparameters for all the tasks, particular, we use action sequence of length 4. More details on RIB RLBench experiments are available in Appendix B.
7

---

1000
Stand
1000
Walk
800
Run
Roach
Epeppe Rotur
7600
530
500
400
5000
MWW M
250
200
2600
0
206
BeB
8e6
107
266
4c6
666
Beb
167
266
4G6
6o5
Bo5
167
0
206
4e6
Bel
9c5
107
1.50
1000
400
1000
Simplo
Return
750
300
750
130
500
250
100
250
4c6
4a6
488
Environment
Environment Stops
Envirorment
Ervioommnn Steps
CON-AS (Ours)
CON
SAC
Figure 7: HumanoidBench results eight densely-rewarde humanoid control tasks (Sferrazza et al., 2024). All the experiments start from scratch and all the methods do not have an auxiliary BC objective. CQN-AS significantly improves the performance of underlying RL algorithm CQN, while outperforming model-free RL baseline, SAC. For CQN-AS and CQN, we report the results aggregated over runs. For SAC, report the results aggregated over 3 runs available from public website. The solid line and shaded regions represent the mean and confidence intervals.
CQN-AS also effective with clean demonstrations Because RLBench provides synthetic clean demonstrations, expected, Figure 6 shows that CQN-AS achieves similar performance CQN on most tasks, except 2/25 tasks where hurts the performance. But we still find that CQN-AS achieves quite superior performance CQN some challenging long-horizon tasks such Open Oven or Take Plate Off Colored Dish Rack. These results show that CQN-AS can be used in various benchmark with different characteristies.
4.3 HumanoidBench Experiments
To show that CQN-AS is generally applicable to tasks without demonstrations, we also study CQN-AS on densely-rewarder tasks from HumanoidBench (Sferrazza et al., 2024).
Setup We follow standard setup that trains RL agents from scratch. We use low-dimensional states consisting of proprioception and privileged task information as inputs. For tasks, we simply select the first 8 locomotion tasks in the benchmark. For baselines, (SAC) (Haarnoja et al., 2018). For SAC, we use the results available from HumanoidBench repository, which evaluated tasks with dexterous hands. For CQN-AS and CQN, also evaluate them tasks with hands. We use the same set hyperparameters for all the tasks (see Appendix B).
consider CQN and Soft Actor-Crite
Comparison baselines Figure shows that, by learning the critic network with action sequence, CQN-AS outperforms other model-free RL baselines, i.e., CQN and SAC, on most tasks. In particular, the difference between CQN-AS and baselines becomes larger as the task gets more difficult, e.g., baselines fail to achieve high episode return on Walk and Run tasks but CQN-AS achieves strong performance. This result shows that our idea of using action sequence can be applicable generic setup without demonstrations.
4.4 Ablation Studies, Analysis, Failure Cases
Effect of action sequence length Figure shows the performance of CQN-AS with different action sequence lengths. We find that training the critic network with longer action sequences tends to consistently improve performance, plateaus or decreases performance if the sequences get too long.
RL objective is crucial for strong performance Figure 8b shows the performance of CQN-AS without RL objective that trains the model only with BC objective successful demonstrations. We find this baseline significantly underperforms CQN-AS, which shows that RL objective enables the agent to leam from aall-an--eroor experiences.

---

107
100
Tousrmpan Hize
100
Move Plate
100
Saupapen To Hob
75
75
Rans
75
60-
50
25-
28-
25
464
God
105
2nd
46d
8:4
105
Jec
And
Sel
Fwel
Ted
264
Gaf
Sal
165
Enwirommen Steps
Envronment Bteps
Enveameett Siepe
Ern commend Siace
OON AB92
CONA811
AGN
OON-484
CQNA8E
CONAS
CONAS (No Ri)
(a) Effect of action sequence length
(b) Effect of RL objective
100
Saucepan
100
Sausspan Hob
Readi Target Single
Succers Rate (S)
50
20
50
25
25-
25
25-
464
Tes
0
Tes
404
165
214
Envirrmen Steps
Enhronment Stopo
Envrooment Buass
Env romment Stoos
nistopm1
stop-16
CON-AS
CON-AS (No Temporal Ensemble)
(c) Effect of N-step
(d) Efffect of temporal ensemble (m 0.01)
75
Esbocs Retum
60-
50-
500
500
25-
25-
250
250
0
Enaronmeet
Enséronment
Endrrmmnn
m=12
Effect of temporal ensemble magnitude
(I) Failure mode: Torque control
Figure 8: Ablation studies and analysis on the effect of (a) action sequence, (b) RL objective, (c) N-step return, and (d & e) temporal ensemble. (f) We also provide results on locomotion tasks from DeepMind Control Suite (Tassa et al., 2020), where CQN-AS fails to improve performance. The solid line and shaded regions represent the mean and confidence intervals, respectively, across runs.
Effect of N-step return Figure 8c shows experimental results with varying N N-step returns. We find that too high N-step return significantly degrades performance. We hypothesize this because the variance from N-step return makes it difficult to learn useful value functions.
Effect of temporal ensemble Figure 8d shows that performance degrades without temporal en- semble on Saucepan To Hob as temporal ensemble induces smooth motion and thus improves performance in fine-gan ann-grrined control tasks. But we also find that temporal ensemble is harmful on Reach Target Single. This is because temporal ensemble uses predictions from previous steps and thus makes it difficult to refine behaviors based on recent observations. Nonetheless, we use temporal ensemble for all the tasks as it helps on most tasks and we aim to use the same set of hyperparameters.
Effect of temporal ensemble magnitude We further provide results with different temporal ensemble magnitudes by adjusting hyperparameter m in Figure 8e. Here, higher m puts higher weights on recent actions and thus very high m corresponds using only first action. Similarly to previous paragraph, we find that higher leads to better performance on Reach Target Single that needs fast reaction, but degrades performance on Saucepan To Hob.
Failure case: Torque control Figure 8f shows that CQN-AS underperforms CQN on locomotion tasks with torque control, which are drawn from the DeepMind Control Suite (Tassa et al., 2020). We hypothesize that this performance degradation arises because sequences of joint positions tend to have clearer semantic structure in joint space, making them easier to learn from compared to sequences of raw torques. Addressing this failure case represents an interesting direction for future work.

---

5 Related Work
Behavior cloning with action sequence Recent behavior cloning approaches have shown that predicting a sequence of actions enables the policy to imitate noisy expert trajectories and helps in dealing with idle actions from human pauses during data collection (Zhao al., 2023: Chi et al., 2023). Notably, Zhao et al. (2023) train a transformer model (Vaswani et al., 2017) that predicts action sequence and Chi et al. (2023) train a denoising diffusion model (Ho et al., 2020) that approximates the action distributions. This idea has been extended to multi-task setup (Bharadhwaj et al., 2024), mobile manipulation (Fu et al., 2024b) and humanoid control (Fu et al., 2024a). Our work is inspired by this line of research and proposes to learn RL agents with action sequence.
Reinforcement learning with action sequence In the context of RL, Medini & Shrivastava (2019) propose to pre-compute frequent action sequences from expert demonstrations and augment the action space with these sequences. However, this idea introduces additional complexity and is not scalable to setups without demonstrations. One recent work relevant ours is Saanum et al. (2024) that encourage a sequence of actions from RL agents to be predictable and smooth. Our work differs in that directly incorporate action sequences into value learning. Recently, Ankile et al. (2024) point out that RL with action sequence is challenging and instead propose to use RL for learning a single-step policy that corrects action sequence predictions from BC. In contrast, show that RL with action sequence is feasible and improves performance of RL algorithms.
Multi-token prediction Recent large language models have incorporated a notably similar idea to predicting action sequences from demonstrations predicting multiple future tokens at once, or multi-token prediction (Gloeckle et al., 2024; Liu et al., 2024). For instance, Gloeckle et al. (2024) show that predicting multiple n future tokens in parallel with independent output heads improves the performance and can accelerate inference speed. DeepSeek-V3 V3 (Liu et al., 2024) also make similar observation but with a sequential multi-token prediction. It would be interesting to see whether our idea can be utilized for fine-tuning these models with multi-token prediction.
Hierarchical reinforcement learning Approaches that learn RL agents with temporally extended high-level actions, or options, have been well studied (Sutton, 1988). The key idea is to train high- level policies that output options by manually defining subgoals (Kulkarni et al., 2016; Dayan & Hinton, 1992) or learning options from data (Bacon et al., 2017; Vezhnevets et al., 2017; Nachum et al., 2018), and then train a low-level agent that learns to execute low-level actions conditioned on options. Our work is not directly comparable to these works as we do not abstract temporally extended actions but use raw action sequences for value learning.
Discussion
We have presented Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a critic-only RL algorithm that trains critic network to output Q-values over action sequences. Extensive experiments in benchmarks with various setups show that our idea not only improves the performance of the base algorithm but also allows for solving complex tasks where prior RL algorithms fail.
Limitations and future work One limitation of our work is the lack of real-world robot evaluation. Moreover, as discussed in Section 4.1 and Section 4.4, solving tasks involving small objects remains a limitation of our approach. One potential approach would be using strong pre-trained vision encoders, but we find that computational cost is often prohibitively large, which remains as an open problem. We are excited about future directions, including real-world RL with humanoid robots, incorporating advanced critic architectures (Kapturowski et al., 2023; Chebotar et al., 2023; Springenberg et al., 2024), bootstrapping RL agents from imitation learning (Hu et al., 2023; Xing et al., 2024) or offline RL (Nair et al., 2020; Lee et al., 2021), extending the idea to recent model-based RL approaches (Hafner al., 2023; Hansen al., 2024). extend parallel value learning scheme to autoregressive. multi-step Q-learning scheme (Kahn et al., 2018), fine-tuning vision- laaiion-lnguage-acae -cctinn models that use action sequence (Team et al., 2024: Doshi al., 2024) or language models that use multi-token prediction (Gloeckle et al., 2024; Liu et al., 2024) with our algorithm, to name but few.
10

---

Acknowledgements
We thank Stephen James and Richie Lo for the discussion on the initial idea of this project. This work was supported in part by Multidisciplinary University Research Initiative (MURT) award by the Army Research Office (ARO) grant No. W91 Pieter Abbeel holds concurrent appointments as a Professor at UC Berkeley and as an Amazon Scholar. This paper describes work performed at UC Berkeley and is not associated with Amazon.
References
Ankile, Lars, Simeonov, Anthony, Shenfeld, Idan, Tome, Marcel, and Agrawal, Pulkit. From imitation to cecthemmnnt-residee rl for precise visual assembly. arXiv preprint arXiv:2407.16677, 2024.
Ba, Jimmy Lei, Kiros, Jamie Ryan, and Hinton, Geoffrey E. Layer normalization. arXiv preprint 2016.
Bacon, Pierre-Luc, Harb, Jean, and Precup, Doina. The option-critic architecture. In Proceedings of the AAAI conference on artificial intelligence, 2017.
Bellemare, Marc G, Dabney, Will, and Munos, Rémi. A distributional perspective on reinforcement learning. International Conference on Machine Learning, 2017.
Bharadhwaj.
Homanga, Vakil, Jay. Sharma, Mohit, Gupta, Abhinav, Tulsiani, Shubham, and Kumar, Vikash. Roboagent: Generalization and efficiency robot manipulation via semantic augmenta- tions and action chunking. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024.
Chebotar, Yevgen, Vuong, Quan, Hausman, Karol, Xia, Fei, Lu, Yao, Irpan, Alex, Kumar, Aviral, Yu, Tianhe, Herzog, Alexander, Pertsch, Karl, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, 2023.
Chernyadev, Nikita, Backshall, Nicholas, Ma, Xiao, Lu, Yunfan, Seo, Younggyo, and James, Stephen. Bigym: A demo-driven mobile bi-manual manipulation benchmark. In Conference on Robot Learning, 2024.
Chi, Cheng, Feng, Siyuan, Du, Yilun, Xu, Zhenjia, Cousineau, Eric, Burchfiel, Benjamin, and Song, Shuran. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
Cho, Kyunghyun, Van Merriënboer, Bart, Gulcehre, Caglar, Bahdanau, Dzmitry, Bougares, Fethi, Schwenk, Holger, and Bengio, Yoshua. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
Dayan, Peter and Hinton, Geoffrey E. Feudal reinforcement learning. Advances in neural information processing systems, 1992.
Doshi, Ria, Walke, Homer, Mees, Oier, Dasari, Sudeep, and Levine, Sergey. Scaling cross-embodied learning: One policy for manipulation, navigation, locomotion and aviation. In Conference on Robot Learning, 2024.
Fu, Zipeng, Zhao, Qingqing, Wu, Qi, Wetzstein, Gordon, and Finn, Chelsea. Humanplus: Humanoid shadowing and imitation from humans. In Conference on Robot Learning, 2024a.
Fu, Zipeng, Zhao, Tony Z, and Finn, Chelsea. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning, 2024b.
Fujimoto, Scott, Hoof, Herke, and Meger, David. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, 2018.
Gloeckle, Fabian, Idrissi, Badr Youbi, Rozière, Baptiste, Lope7-Paz, David, and Synnaeve, Gabriel. Better faster large language models via multi-token prediction. arXiv preprint arXiv:2404.199 19737, 2024.
11

---

Haarnoja, Tuomas, Zhou, Aurick, Hartikainen, Kristian, Tucker, George, Ha, Sehoon, Tan, Jie, Kumar, Vikash, Zhu, Henry, Gupta, Abhishek, Abbeel, Pieter, al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2005905 2018.
Hafner, Danijar, Pasukonis, Jurgis, Ba, Jimmy, and Lillicrap, Timothy, Mastering diverse domains through world models. arXiv preprint arXiv:2301.0410 2023.
Hansen, Nicklas, Su, Hao, and Wang, Xiaolong. Td-mpe2: Scalable, robust world models for continuous control. In International Conference on Learning Representations. 2024.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Hendrycks, Dan and Gimpel, Kevin. 1006::16066 2016.
Gaussian error linear units (gelus). arXiv preprint
Hester, Todd, Vecerik, Matej, Pietquin, Olivier, Lanctot, Mare, Schaul, Tom, Piot, Bilal, Horgan, Dan, Quan, John, Sendonaris, Andrew, Osband, lan, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI conference on artificial intelligence, 2018.
Ho, Jonathan, Jain, Ajay, and Abbeel, Pieter. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Hu, Hengyuan, Mirchandani, Suvir, and Sadigh, Dorsa. Imitation bootstrapped reinforcement learning. arXiv preprint arXiv:2311.02/99. 2023.
James, Stephen, Freese, Mare, and Davison, Andrew J. Pyrep: Bringing v-rep deep robot learning. arXiv preprint arXiv: 2019.
James, Stephen, Ma, Zicong, Arrojo, David Rovick, and Davison, Andrew J. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Kaelbling, Leslie Pack, Littman, Michael I., and Cassandra, Anthony R. Planning and acting in partially observable stochastic domains. Artificial intelligence, 1998.
Kahn, Gregory, Villaflor, Adam, Ding, Bosen, Abbeel, Pieter, and Levine, Sergey. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE international conference on robotics and automation (ICRA), 2018.
Kapturowski, Steven, Campos, Victor, Jiang, Ray, Rakićević, Nemanja, van Hasselt, Hado, Blundell, Charles, and Badia, Adrià Puigdomenech. Human-level atari 200x faster. In International Conference on Learning Representations, 2023.
Kulkarni, Tejas D, Narasimhan, Karthik, Saeedi, Ardavan, and Tenenbaum, Josh. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Lee, Seunghyun, Seo. Younggyo, Lee, Kimin, Abbeel, Pieter, and Shin, Jinwoo. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning, 2021.
Liu, Aixin, Feng, Bei, Xue, Bing, Wang, Bingxuan, Wu, Bochao, Lu, Chengda, Zhao, Chenggang, Deng, Chengqi, Zhang, Chenyu, Ruan, Chong, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Loshchilov, Ilya and Hutter, Frank. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
Medini, Tharun and Shrivastava, Anshumali. Mimicking actions is good strategy for beginners: Fast reinforcement learning with expert action sequences, 2019. URL sttpss://pperrvvee. net bet/forrumidd=Hfzobbe
12

---

Mnih, Volodymyr. Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Mare G. Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K. Ostrovski, Georg, et al. Human- level control through deep reinforcement learning. Nature, 2015.
Nachum, Ofir, Gu, Shixiang Shane, Lee, Honglak, and Levine, Sergey, Data-efficient hierarchical reinforcement learning. Advances neural information processing systems, 2018.
Nair, Ashvin, Gupta, Abhishek, Dalal, Murtaza, and Levine, Sergey. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359) 2020.
Nakamoto, Mitsuhiko, Zhai, Simon, Singh, Anikait, Sobol Mark, Max, Ma, Yi, Finn, Chelsea, Kumar, Aviral, and Levine. Sergey. Cal-ql: Calibrated offline pre- tre-training for efficient online fine-tuning, Advances in Neural Information Processing Systems, 2023.
Polyak, Boris T and Juditsky, Anatoli B. Acceleration of stochastic approximation by averaging.
SIAM journal on control and optimization, 1992.
Pomerleau, Dean Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, 1988.
Rohmer, Eric, Singh, Surya PN, and Freese, Marc. V-rep: A versatile and scalable robot simulation framework. In JEEE/RSJ international conference on intelligent robots and systems, 2013.
Saanum, Tankred, Éltetō, Noémi, Dayan, Peter, Binz, Marcel, and Schulz, Eric. Reinforcement learning with simple sequence priors. Advances in Neural Information Processing Systems, 2024.
Seo, Younggyo, Uruç, Jafar, and James, Stephen. Continuous control with coarse-to-fine reinforce- ment learning. In Conference on Robot Learning. 2024.
Seyde, Tim, Werner, Peter, Schwarting, Wilko, Gilitschenski, Igor, Riedmiller, Martin, Rus, Daniela, and Wulfmeier, Markus. Solving continuous control via q-learning. In International Conference on Learning Representations, 2023.
Sferrazza, Carmelo, Huang, Dun-Ming, Lin, Xingyu, Lee, Youngwoon, and Abbeel, Picter. Hu- manoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. In Robotics: Science and Systems, 2024.
Silver, David, Schrittwieser, Julian, Simonyan, Karen, Antonoglou, Ioannis, Huang, Aja, Guez, Arthur, Hubert, Thomas, Baker, Lucas, Lai, Matthew, Bolton, Adrian, et al. Mastering the game of go without human knowledge. nature, 2017.
Springenberg, Jost Tobias, Abdolmaleki, Abbas, Zhang, Jingwei, Groth, Oliver, Bloesch, Michael, Lampe, Thomas, Brakel, Philemon, Bechtle, Sarah, Kapturowski, Steven, Hafner, Roland, et al. Offline actor-critic reinforcement learning scales to large models. In International Conference on Machine Learning, 2024.
Sutton, Richard S. Learning predict by the methods of temporal differences. Machine learning, 1988.
Sutton, Richard S and Barto, Andrew G. Reinforcement learning: An introduction. MIT press, 2018.
Tassa, Yuval, Tunyasuvunakool, Saran, Muldal, Alistair, Doron, Yotam, Liu, Siqi, Bohez, Steven, Merel, Josh, Erez, Tom, Lillicrap, Timothy, and Heess, Nicolas. dm_control: Software and tasks for continuous control. arXiv preprint arXiv:2006. 12983, 2020.
Team, Octo Model, Ghosh, Dibya, Walke, Homer, Pertsch, Karl, Black, Kevin, Mees, Oier, Dasari, Sudeep, Hejna, Joey, Kreiman, Tobias, Xu, Charles, et al. Octo: An open-source generalist robot policy. In Robotics: Science and Systems, 2024.
Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.
Vaswani, Ashish, Shazeer, Noam, Parmar, Niki, Uszkoreit, Jakob, Jones, Llion, Gomez, Aidan N, Kaiser, Lukasz, and Polosukhin, Illia. Attention all you need. In Advances in Neural Information Processing Systems, 2017.
13

---

Vezhnevets, Alexander Sasha, Osindero, Simon, Schaul, Tom, Heess, Nicolas, Jaderberg, Max, Silver, David, and Kavukcuoglu, Koray. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, 2017.
Wang. Ziyu, Schaul, Tom, Hessel, Matteo, Hasselt, Hado, Lanctot, Marc, and Freitas, Nando. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, 2016.
Xing, Jiaxu, Romero, Angel, Bauersfeld, Leonard, and Scaramuzza, Davide. Bootstrapping rein- forcement learning with imitation for vision-based agile flight. In Conference on Robot Learning, 2024.
Yarats, Denis, Fergus, Rob, Lazaric, Alessandro, and Pinto, Lerrel. Mastering visual continuous con- trol: Improved lata-augmented reinforcement learning. In International Conference on Learning Representations, 2022.
Zhao, Tony Z, Kumar, Vikash, Levine, Sergey, and Finn, Chelsea. Learning fine-grained bimanual manipulation with low-cost hardware, In Robotics: Science and Systems, 2023.
14

---

A Motivating Experiments
Additional details For Figure 2a, use the same demonstrations used in our main experiments (see Appendix B for more details). For SAC and TD3 experiments with action sequences in Figure 2b, we implemented our code based on the official HumanoidBench repository. We use the hyperparam- eters in the repository for training SAC agents. For TD3, we use the standard deviation of 0.2 for exploration. We report the average target Q-values recorded throughout experiments. For Figure 2c, we train SAC and CQN agents with 6 original actions and 294 no-op actions with [-1, 1] action bounds and use an environment wrapper that slices out no-op actions.
Experiment with 2D Point-mass environment To further motivate the use of action sequence for value learning, we train DQN agents (Mnih et al., 2015) on 2D Point-mass environment with discrete action spaces. In particular, we train a Raw agent that trains with the raw discrete action space consisting of single-step accelerations parameterized by discrete headings (cardinal and discrete directions) and magnitude level, resulting in 8 total actions. We compare this against Sequence agent that trains with the discrete action space that consists of smooth 5-step acceleration sequences parameterized by cubic Bezier curves instead of single-step accelerations. Both agents operate 2D double-integrator environment where the goal is to reach a target position. In Figure 9, as expected, find that training DQN agent with pre-defined action sequences lead faster convergence. Our main experimental results Section further show that, CQN-AS can achieve similar benefit of using action sequences without pre-defined set of action sequences various challenging continuous control benchmarks.
-2
End
Distance to Goal
-4
M
Goal
-6
-8
DQN w/ Action Sequence
Start
DQN w/ Raw Actions
-10
25000 Environment Sleps
50000
(a) 2D Point-mat mass task
(b) Experimental Results
Figure 9: 2D Point-mass experiments (a) We consider simple toy environment where the goal is reach the region around a randomly spawned goal point. (b) We show that, as expected, training with pre-defined action sequences lead faster convergence. The solid line and shaded regions represent the mean and confidence intervals, respectively, across 10 runs.
15

---

B Experimental Details
BiGym BiGym² (Chernyadev et al., 2024) is built upon MuJoCo (Todorov et al., 2012). We use Unitree HI with two parallel grippers. We find that demonstrations available in the recent version of BiGym are not all successful. Therefore we adopt the strategy of replaying all the demonstrations and only use the successful ones as demonstrations, instead of discarding the failed demonstrations, store them in a replay buffer as failure experiences. To avoid training with too few demonstrations, we exclude the tasks where the ratio of successful demonstrations is below 50%. Table I shows the rewarded mobile bi-manual manipulation tasks used in our experiments.
list of 25
Table 1: BiGym tasks with their maximum episode length and number successful demonstrations.
Task
Length
Demos
Task
Length
Demos
Move Plate
300
51
Cupboards Close All
620
53
Move Two Plates
550
30
Reach Target Single
100
30
Saucepan To Hob
440
28
Reach Target Multi Modal
100
60
Sandwich Flip
620
34
Reach Target Dual
100
50
Sandwich Remove
540
Dishwasher Close
375
44
Dishwasher Load Plates
560
17
Wall Cuphoard Open
300
44
Dishwasher Load Cups
750
58
Drawers Open All
480
45
Dishwasher Unload Cutlery
620
29
Wall Cupboard Close
300
60
Take Cups
420
32
Dishwasher Open Trays
380
57
Put Cups
425
43
Drawers Close All
200
59
Flip Cup
550
45
Drawer Top Open
200
40
Flip Cutlery
500
43
Drawer Top Close
120
51
Dishwasher Close Trays
320
62
HumanoidBench HumanoidBench' (Sferrazza et al., 2024) is built upon MuJoCo (Todorov et al., 2012). We use Unitree HI with two dexterous hands. consider the first 8 locomotion tasks the benchmark: Stand, Walk, Run, Reach, Hurdle, Crawl, Maze, Sit Simple. We use proprioceptive states and privileged task information instead visual observations. Unlike BiGym and RLBench experiments, we do not utilize dueling network (Wang et al., 2016) and distributional critic (Bellemare et al., 2017) in HumanoidBench for faster experimentation.
RLBench RLBencc (James et al., 2020) is built upon CoppeliaSim (Rohmer et al., 2013) and PyRep (James et al., 2019). We use a 7-DoF Franka Panda robot arm and parallel gripper. Following the setup of Seo et al. (2024), we increase the velocity and acceleration of the arm by 2 times. For all experiments, use 100 demonstrations generated via motion-planning, Table 2 shows the list of 20 sparsely-rewarded visual manipulation tasks used in our experiments.
Table 2: RLBench tasks with their maximum episode length used in experiments.
Task
Length
Task
Length
Take Lid Off Saucepan
100
Put Books On Bookshelf
175
Open Drawer
100
Sweep To Dustpan
100
Stack Wine
150
Pick Up Cup
100
Toilet Seat Up
150
Open Door
125
Open Microwave
125
Meat On Grill
1.50
Open Oven
225
Basketball In Hoop
125
Take Plate Off
150
Lamp On
100
Colored Dish Rack
Turn Tap
125
Press Switch
100
Put Money In Safe
150
Put Rubbish In Bin
150
Phone on Base
175
Insert Usb In Computer
100
bhttpe://github.co
bench
'https://githu
16

---

Hyperparameters We use the same set of hyperparameters across the tasks in each domain. For hyperparameters shared across CQN and CQN-AS, we use the same hyperparameters for both algorithms for fair comparison. We provide detailed hyperparameters for BiGimm and RI RLBench experiments in Table 3 and HumanoidBench experiments Table 4
Table 3: Hyperparameters for demo-driven vision-based experiments in BiGym and RLBench
Hyperparameter
Value
Image resolution
84 x 84 x 3
Image augmentation
RandomShift (Yarats et al., 2022)
Frame stack
4 (BiGym)/ 8 (RIBeenc)
CNN Architecture
Conv (c=[32, 64, 128, 256]. s=2, p=l)
Linear (c=[512, 512, 64, 512. 512]. bias=False) (BiGym)
MLP Architecture
Linear (c=[64, 512, 512], bias=False) (RLBench)
CNN & MLP- Activation
SiLU (Hendrycks & Gimpel, 2016) and LayerNorm (Ba et al., 2016)
GRU - Architecture
GRU (c=[512), bidirectional=False)
Dueling network
True
C51 Atoms
51
C51 Vmin: Vmax
-2, 2
Action sequence
16 (BiGym) 4 (RLBench)
Temporal ensemble weight m
0.01
Levels
3
Bins
5
BC loss (Cec) scale
1.0
RI. loss (Cxl) scale
0.1
Relaheling demonstrations
True
Data-driven action scaling
True
Action mode
Absolute Joint (BiGym), Delta Joint (RLBench)
Exploration noise
< ~~ (0000.01)
Target critic update ratio (r)
0.02
N-step return
Batch size
256
Demo batch size
256
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Learning
Se-5
Weight decay
0.1
Computing hardware For BiGym and Humanoid experiments, use NVIDIA A5000 GPU with 24GB VRAM. With A5000, each BiGym experiment with 100K environment steps take 16 hours, and each HumanoidBench experiment with 10M environment steps take 40 hours. For RLBench experiments, we use NVIDIA RTX 2080Ti GPU, with which each experiment with 30K environment steps take 6.5 hours. We find that CQN-AS takes 13% more memory compared to CQN and is 40% slower than CQN. Overall, CQN-AS is around 33% slower than running CQN because larger architecture slows down both training and inference.
Baseline implementation For CQN (Seo et al., 2024) and DrQ-v2+ (Yarats et al., 2022), we use the implementation available from the official CQN implementation', For ACT (Zhao et al., 2023), we use the implementation from RoboBase repository". For SAC (Haarnoja et al., 2018), D3eamerV3 (Hafner et al., 2023), and TD-MPC2 (Hansen et al., 2024), we use results provided in HumanoidBench repository (Sferrazza et al., 2024).
fttpp:///gthab..om/youugggo/// hatppp///hhha..cc////ooooeoooobs httppp///ggaabbooo////reeeffeaaa bench
17

---

Table Hyperparameters for state-based experiments in HumanoidBench
Hyperparameter
Value
MLP Architecture
Linear (c=[512, 512], bias=False)
CNN & MLP - Activation
SiLU (Hendrycks & Gimpel, 2016) and LayerNorm (Ba et al., 2016) GRU (c=[512], bidirectionalFalse)
GRU Architecture
Dueling network
True
Action sequence
4
Temporal ensemble weight Levels
0.01
Bins
5
RL loss (Cxt) scale
1.0
Action mode
Absolute Joint
Exploration noise
~~ 0.000.01
Target critic update ratio (7)
1.0
Target critic update interval (r)
Update-to-data ratio (UTD)
0.5
N-step return
3
Batch size
128
Optimizer
AdamW (Loshchilov Hutter, 2019)
learning rate
5e-5
Weight decay
0.1
C
Full description of CQN and CQN-AS
This section provides the formulation of CQN and CQN-AS with n-dimensional actions.
C.1 Coarse-to-fine Q-Network
Let a be action level and dimension and at {a!'" ..... J.:N be actions at level with af being zero vector. We then define coarse-to-fin critic to consist of multiple Q-networks:
Qa" (he, af a a (b,, a."" a.i.D. o of-1")] b=1
for E {l...., L} and E {1,..., N}
(3)
Where B denotes the number of bins. We optimize the critic network with the following objective:
Σ,E, (Q "0e""(he of a a Tt+1 max Q (h4+1, d', t'' (h(+1))
(4)
where are delayed parameters for a target network (Polyak & Juditsky, 1992) and nt is policy that outputs the action a at each level / via the inference steps with critie, i.e., <((h.) a
Action inference To output actions at time step & with the critic, CQN first initializes constants n.high with -1 and for each n. Then the following steps are repeated for / E (1,..., L):
a, nolow and o;
Step (Discretization): Discretize an interval [a, n.high into B uniform intervals, and
each of these intervals become action space for Q/
Step 2 (Bin selection): Find the bin with the highest Q-value, set a to the centroid of the selected bin, and aggregate actions from all dimensions to a!.
Step 3 (Zoom-in): Set a; n_low and a, n.bigh to the minimum and maximum of the selected bin, which intuitively can be seen as zooming-into each bin.
We then use the last level's action a/- the action at time step
18

---

Computing Q-values To compute Q-values for given actions a,, CQN first initializes constants hihgg with -1 and 1 for each n. We then repeat the following steps for 1 E (1...., L.J: n, and dz
dz
n.high into B uniform intervals, and Step (Discretization): Discretize interval la,- aa each of these intervals become an action space for Q/"".
Step 2 (Bin selection): Find the bin that contains input action a,, compute ot" for the selected interval, and compute Q-values Q0" "h,(,,,, di I.n
, 1)
Step 3 (Zoom-in): Set af and aj nh to the minimum and maximum of the selected bin, which intuitively can be seen zooming-into each bin.
then use set of Q-values (Q)" (hr, of )3= for given actions a..
C.2
Coarse-to-fine Critic with Action Sequence
Let 8 {a/: - be an action sequence at level and a K be zero vector. Our critic network consists of multiple Q-networks for each level I, dimension n, and sequence step k:
in 1-1 (he, d 1: a(++k) Qunak
lmb [0 ( (be, aitk-1 aitk-1 a b=1
B
(5)
for E (1, L}, [I.....N) and l E fi...., K}
We optimize the critic network with the following objective:
ΣΣΣ (oe (he, ag a r++1 - max Q ml
f (ha+,, a', x (le+1))
(6)
where The is action sequence policy that outputs the action sequence a In practice, compute Q Q-values for all sequence step & E [1.... } and all action dimension E (1.....V} in parallel. This can be seen as extending the idea of Seyde et (2023), which learns decentralized Q-networks for action dimensions, into action sequence dimension. As mentioned in Section 3.1, we find this simple scheme works well on challenging tasks with high-dimensional action spaces.
Architecture Let eg denote an one-hot encoding for k. For each level (, we construct features for each sequence step ls as h [br, 1-1 - ex] We encode each h with shared MLP network and process them through GRU (Cho et al., 2014) obtain s fgnu (J (b/,hb))))))))) ( M (h/,,)), We use a shared projection layer to map each st into Q-values at each sequence step k, i.e., 18 18 (s/_a) = (12%* (h, (1 {1, N} at the same time with a big linear layer, which follows the design Seo et al. (2024).
We compute Q-values all dimensions n
D Additional Preliminary Experiments
Offiine RL with CQN-AS To further investigate whether CQN-AS formulation is compatible with offline RL, we conduct preliminary experiments on BiGym's Sandwich Remove task. Specifically, we combine CQN-AS with Cal-QL (Nakamoto et al., 2023) and train on the dataset that consists of 26 successful demonstrations and 10 failed trajectories. We find that CQN-AS Cal- QL achieves 33 (+: 6.8) % while CQN + Cal-QL achieves (# 14) %, which shows CQN-AS can be indeed effective for offline setup. We leave further exhaustive investigation as an interesting future work.
Experiments with ResNet-18 To investigate if using larger and stronger pre-trained vision encoders such as ResNet (He et al., 2016) can improve performance, we tried running CQN-AS with ResNet- 18 encoder on BiGym tasks, However, we find that it requires GPU with least 48GB memory and is extremely slow to train. We will leave this direction of incorporating larger vision encoders in an efficient manner as future direction.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have claimed that incorporating action sequences into reinforcement learning be beneficial in the abstract and introduction, which clearly reflects the main contribution of this paper - CQN-AS that uses action sequences value learning. We have provided experimental results and analysis in the paper to support the claims in Figure Figure 5, Figure 6, and Figure 8.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed our limitations in Section 4.1, Section 4.4, and Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the detailed experimental details along with the source code to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
21

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We have provided the source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
oniooicyy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ wtc/public// /etiiii/g////////eeeeeieiiiissee Sadeee////dSmmmiiic StstSusummisiiiii. wwwsssoolllloooeoooesoso onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided sufficient details used in our experiments along with the source code that contains hyperparameters required for reproducing the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We have provided error bars aggregated over runs for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided detailed information about compute costs in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttp://eeerpp/eeeeeeeeeee cc httpppulic/tiiicuuideeeeeeeeeeee
Answer: [Yes]
Justification:
We are committed to conforming with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed broader impacts of this work ??.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work do not require safeguards as do not foresee the imminent risks from using our algorithms.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited references for benchmarks used in the paper. Each benchmark provides licenses for assets used in their simulations in their papers or websites.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We have not introduced new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We have not conducted relevant experiments in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We have conducted relevant experiments in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
25

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Developing our algorithm have not involved LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http///eurrippppee ctcpiipps.cc/coffeeeeeeee httrrrpps.cc/coofeeeneeee/////// for what should or should not be described.
26


Reviews:
[{"review_id": "Y8KQ1KsyRM", "reviewer": "Reviewer_H7zC", "summary": "This paper presents Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel value-based multi-step RL algorithm. CQN-AS trains a critic network to output Q-values over discretized action sequences in a coarse-to-fine manner. By generating multiple actions in parallel, it is less affected by compounding errors and makes value prediction easier. By completely removing the actor network, it is more robust to function approximation error. Experiments on multiple challenging benchmarks demonstrate that CQN-AS outperforms baseline algorithms in most tasks, particularly in sparse reward scenarios.", "questions": "1. Can CQN-AS be extended to the offline RL setting?\n\n2. Would replacing the GRU with a transformer network improve performance on long-horizon tasks?\n\n3. The paper mentions that removing the actor network prevents the actor from overexploiting the critic, making the algorithm more robust to function approximation errors in the critic. However, when performing Q-learning with discretized actions, the algorithm still selects the action that maximizes the next Q-value, which could also correspond to regions with large approximation errors. Why does this not lead to overestimation in such cases?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The authors begin by analyzing two empirical observations: (1) the benefit of multistep actions for return-to-go prediction, and (2) the value overestimation in standard actor-critic algorithms when combined with multistep actions. These analyses effectively motivate their proposed method, resulting in a presentation that is logically coherent and clearly motivated.\n\n2. The writing is clear and the structure is well-organized throughout. Figure 2 effectively illustrates the core idea of the algorithm.\n\n3. The authors conduct extensive experiments to validate the effectiveness of the algorithm, along with comprehensive ablation studies that highlight the contribution of each component.\n\n### Weaknesses\nI agree with the authors' acknowledgment of the limitations and weaknesses of their method: (1) inferior performance on long-horizon tasks, (2) lack of real-world robot experiments, and (3) suboptimal results under torque control. However, the algorithm is well-motivated and innovative, and the experimental results strongly support the proposed motivation. I believe this work has the potential to inspire many future studies. In my opinion, it is worthy of acceptance and presentation at NeurIPS.", "was_revised": true, "final_justification": "I am keeping my recommendation for acceptance.", "rebuttal": "Dear Reviewer H7zc,\n\nWe really appreciate your time and effort for reviewing our paper. We provide our responses below.\n\n---\n\n**Q1: CQN-AS for offline RL?**\n\n**A1.** We find positive results in our preliminary experiments. We combined CQN-AS with Cal-QL [1] and trained it on the dataset that consists of 26 successful demonstrations and 10 failed episodes. We find that CQN-AS + Cal-QL achieves 33 ($\\pm$ 6.8) \\% while CQN + Cal-QL achieves 7 ($\\pm$ 14) \\%, which shows CQN-AS can be indeed effective for offline setup.\n\n[1] Nakamoto, Mitsuhiko, et al. \"Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning.\" Advances in Neural Information Processing Systems 36 (2023): 62244-62269.\n\n---\n\n**Q2: Would GRU -> Transformer improve performance on long-horizon tasks?**\n\n**A2.** This is an interesting question. While using Transformer for RL has a potential to achieve better performance on long-horizon tasks (as shown in offline RL papers like [2,3]), we find that GRU is faster to train and leads to stronger performance in our considered setups. We hypothesize this is because it is non-trivial to train Transformer models from scratch using unstable training signals from temporal difference learning, which leaves a room for interesting future work.\n\n[2] Chebotar, Yevgen, et al. \"Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions.\" Conference on Robot Learning. PMLR, 2023.\n\n[3] Springenberg, Jost Tobias, et al. \"Offline actor-critic reinforcement learning scales to large models.\" arXiv preprint arXiv:2402.05546 (2024).\n\n---\n\n**Q3: How does Q-learning with discrete actions avoid severe value overestimation?**\n\n**A3.** As you mentioned, Q-learning with discrete actions may still suffer from overestimation due to max operation during training. However, in this paper, we find that a combination of (i) a critic network that takes high-dimensional action sequences as inputs and (ii) the actor that is trained to output actions using gradients from the critic leads to severe value overestimation problems. We hypothesize this is because the critic network tends to have more function approximation errors from high-dimensional actions and the actor can easily exploit this error. On the other hand, critic-only algorithms can avoid this problem, leading to stable training with action sequences.\n\nTo further support our point, we conducted additional toy experiments where we introduced redundant no-op actions for training TD3 and CQN agents on DMC Cheetah Run tasks. Specifically, we trained agents with 6 original actions and 294 no-op actions with [-1, 1] action bounds and used an environment wrapper that slices out no-op actions. In this setup, we found that CQN (critic-only algorithm with discrete actions) is indeed robust to the overestimation problem. In particular, we find that TD3 suffers from severe value overestimation with more no-op actions, while CQN with such no-op actions achieves similar performance with the original CQN agent. This clearly shows that a critic-only algorithm is more robust to training with high-dimensional action spaces.\n\n| Additional No-op actions | Episode Return (CQN) | Target Q-Values (CQN) | Episode Return (TD3) | Target Q-Values (TD3) |\n| :--- | :--- | :--- | :--- | :--- |\n| 0 | 236.61 | 18.84 | 140.67 | 16.25 |\n| 54 | 219.50 | 20.27 | 7.18 | -17.45 |\n| 144 | 185.38 | 16.42 | 0.56 | 1.00E+08 |\n| 294 | 202.94 | 18.66 | 0.27 | 4.00E+08 |\n\nBased on this observation, one potential alternative is to use expectile regression for Q-learning as in IQL [4], which can avoid value overestimation problems by avoiding the explicit maximization step in TD-learning. We will leave this as a future work and include relevant discussion in the final version.\n\n[4] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" International Conference on Learning Representations 2022"}, {"review_id": "ux0dK7PlQZ", "reviewer": "Reviewer_5gur", "summary": "This paper introduces the Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a novel reinforcement learning (RL) algorithm. The core idea is to enhance value learning in robotics by training a critic network to predict Q-values for sequences of actions rather than individual actions. This approach is motivated by observations that action sequences lead to lower validation loss in return-to-go prediction and that standard actor-critic methods struggle with value overestimation when using action sequences. Experiments show that CQN-AS successfully avoids this overestimation and demonstrates improved performance on various robotic simulation benchmark environments.", "questions": "* The author mentioned that CQN-AS may struggle to “achieve meaningful success rates on long-horizon tasks” (4.1). I think the paper would be made stronger if the authors include some potential experiments showing how some of the potential extensions can make CQN-AS perform even better.\n* Can you elaborate on on the baselines details of SAC and TD3 you used in obtaining figure 2b). I also don't entirely agree with your claim that action sequencing causes severe value over estimation from by looking at this plot - it looks from the plot that in fact TD3 and SAC may have even more value overestimation compared to the ones using action sequences.\n* In appendix table 3, there are mentions of hyperaparameters for C51. Do you use a distributional critic for both CQN and CQN-AS? If so, it would be nice to know how much a distributional critic helps with addressing value estimation. Given a distributional critic has bounded value, this means that you won't be able to see severe value overestimation as those reported in fig 2b.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* The design of the CQN-AS is well-motivated by the observation that using action sequences lead to lower critic loss.\nExtensive evaluation on the performance of CQN-AS.\n* The evaluation is thorough.\n* Clear ablation of the individual algorithmic choices and their merits in (4.4).\n\nWeaknesses\n* Extension to use action sequences based on CQN looks like an incremental extension so I wouldn’t rate this work highly in terms of of novelty.", "was_revised": true, "final_justification": "Thank the authors for the response. The rebuttals address my points. Please consider incoporating some the suggestions into the final version.", "rebuttal": "Dear Reviewer 5gur,\n\nWe really appreciate your time and effort for reviewing our paper. We provide our responses below.\n\n---\n\n**Q1: Limited Novelty of CQN-AS.**\n\n**A1.** While the architectural novelty of CQN-AS may be limited, we would like to emphasize our paper has significant contributions: we provide interesting observations from training popular actor-critic algorithms with action sequences, build a new algorithm based on such observations, and provide experimental results and analysis that could be useful for future research.\n\n---\n\n**Q2: The paper could be stronger with some potential experiments showing how some of the potential extensions can make CQN-AS perform even better.**\n\n**A2.** As we noted in Section 4.1, CQN-AS struggles to achieve meaningful success rate on some of the long-horizon *tasks that require interaction with delicate objects such as cups or cutlery*). We believe this can be resolved by incorporating a better perception stack, e.g., incorporating a stronger vision encoder and experimenting with higher-resolution visual observations. While we tried to run experiments per your suggestion, i.e., increasing the image resolution and incorporating ResNet-18, we found that it requires a GPU with at least 48GB memory and is extremely slow to train. We will leave this direction of incorporating larger vision encoders in an efficient manner as a future direction.\n\n---\n\n**Q3: Are TD3-AS and SAC-AS really suffering from value overestimation in Figure 2(b)? Can you provide more details of SAC and TD3 baselines in Figure 2(b)?**\n\n**A3.** We would like to clarify that TD3 and SAC with action sequences indeed suffer from severe value overestimation, as shown in the follow table:\n\n| Method | Episode return | Target Q values |\n|---|---|---|\n| TD3-AS1 | 113.97807 | 9.19 |\n| TD3-AS2 | 4.68642 | 6e9 |\n| TD3-AS4 | 27.52748 | 5e9 |\n| SAC-AS1 | 198.16332 | 16.13 |\n| SAC-AS2 | 16.27714 | 3e9 |\n| SAC-AS4 | 22.638 | 1e9 |\n\nNonetheless, we agree that the figure can be confusing to readers. We will make the figure clearer to understand. For details on SAC and TD3, we used the standard implementation and hyperparameters for training SAC and TD3 for Figure 2(b), for instance, we used the hyperparameters from HumanoidBench official repository for training SAC. For TD3, we used standard deviation of 0.2 for exploration. We will include these details in the final version as well.\n\n---\n\n**Q4: Did you use a distributional critic for both CQN and CQN-AS? What’s the effect of the distributional critic on RL with action sequence?**\n\n**A4:** We used C51 for both CQN and CQN-AS. To answer your question, we run additional experiments that don't use C51 for CQN-AS on BiGym’s Saucepan To Hob task, and find that both CQN-AS with distributional critic (86 $\\pm$ 19\\%) and CQN-AS without distributional critic (82.6 $\\pm$ 16\\%) achieve similar performance. This result shows that using a critic-only algorithm with discrete actions is indeed more crucial in enabling RL with action sequences compared to the use of a distributional critic."}, {"review_id": "7BFyYrnnfv", "reviewer": "Reviewer_YKFp", "summary": "The paper introduces Coarse-to-fine Q-Network with Action Sequence (CQN-AS), a value-based RL method that explicitly predicts Q-values over sequences of actions rather than individual actions. The main claim is that modeling the return-to-go over action sequences leads to more accurate value estimation and better performance, especially in sparse-reward settings. This is supported by empirical results showing that CQN-AS often outperforms baseline methods on tasks from BiGym and RLBench.\n\nA key contribution is the formulation of the value function over sequences, which encourages the agent to reason more effectively about temporally extended behaviors. The authors support this claim through comparative results on several benchmarks and provide ablation studies showing that longer action sequences correlate with improved performance. However, while average performance improves, in some tasks gains are modest or inconsistent, which somewhat weakens the universality of the claim. A deeper analysis of when and why CQN-AS works best would help solidify its impact.", "questions": "Some tasks show only marginal gains over baselines. Can the authors provide more detailed per-task results or analysis to clarify where CQN-AS is most effective?\n\nThe paper mentions using “17 to 60 demonstrations” per task. How is this number chosen? Performance can vary a lot wrt the number of demos used. Can you clarify the choice?\n\nThe ablation on sequence lengths shows significant performance improvements with longer action sequences. Is there an upper bound or threshold beyond which increasing the sequence length causes performance to degrade?\n\nHave the authors considered comparisons to sequence models?", "limitations": "The authors are transparent about the method’s limitations on long-horizon tasks, where predicting the return of long action sequences becomes increasingly difficult. This is a meaningful constraint, especially for complex tasks requiring extended planning or multi-stage behaviors. While CQN-AS shows promise on mid-range horizons, its performance may degrade as the sequence length increases due to compounding uncertainty and training instability. This limits its scalability in more complex environments. It would be interesting to see future extensions that incorporate hierarchical sequence modeling or hybrid approaches that decouple high-level planning from low-level execution. Such strategies could help overcome the bottlenecks associated with long sequence modeling and extend the applicability of the method to broader domains.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper presents a well-motivated problem and a clearly defined method. Experiments cover a diverse set of challenging sparse-reward tasks from BiGym and RLBench, supported by a reasonable set of ablations.\n\nThe paper is clearly written and well-structured. The presentation of ideas, contributions, and results is easy to follow. Ablation studies and visualizations help convey the impact of different design decisions effectively.\n\nThe idea of leveraging action sequences for value-based RL is timely and relevant, bridging insights from behavior cloning to reinforcement learning. However, in some cases/tasks, it is not clear that the proposed method significantly outperforms the baselines.\n\nThe method builds on existing ideas in sequence modeling and action prediction, applying them in a novel way to Q-value estimation. While the high-level motivation is not entirely new, the specific formulation and empirical exploration are original and interesting.\nThe method draws inspiration from recent trends in behavior cloning and sequence modeling (e.g., decision transformers), but applies these ideas in a novel way within a value-based RL framework—distinct from purely model-free or policy-gradient approaches. To the best of my knowledge, there is no prior value-based method that directly models Q-values over explicit action sequences as done here. This design is original and well-motivated, representing a meaningful contribution to the space of temporal abstraction and sequential reasoning in RL.\n\nSome experimental design choices are not well justified. For example, the statement that \"17 to 60 demonstrations\" were used leaves ambiguity: using 17 versus 60 demos can have a large impact on performance, particularly in sparse-reward settings. Clarifying this and providing more control over such variables would strengthen the experimental section.", "was_revised": true, "final_justification": "I believe my questions have been addressed, I confirm my score, and I have completed the mandatory acknowledgement.", "rebuttal": "Dear Reviewer YKFp,\n\nWe really appreciate your time and effort for reviewing our paper. We provide our responses below.\n\n---\n\n**Q1: Some tasks show only marginal gains over baselines. Can the authors provide more detailed per-task results or analysis to clarify where CQN-AS is most effective?**\n\n**A1.** We find that CQN-AS tends to be more effective on relatively long-horizon tasks compared to baselines. For instance, results on RLBench’s long-horizon tasks like Open Oven or Take Plate Off Colored Dish Rack support this. On the other hand, we find that CQN-AS can be weaker than baselines on very dynamic tasks such as reach target due to the use of temporal ensemble. Finally, we find that CQN-AS does not exhibit stronger performance on tasks like on BiGym’s flip cutlery task, where we hypothesize all methods fail due to challenges in perception. We will include more detailed discussion in the final version. Thank you for your suggestion!\n\n---\n\n**Q2: Clarification on “17 to 60 demonstrations” per task**\n\n**A2.** As we’ve written in Appendix C, we used different number of demonstrations for different tasks, but we used the same number of demonstrations for all algorithms within each task. This is because not all human demonstrations in BiGym are not successfully replayed in simulation, and therefore we used demonstrations that we were able to replay in simulations, where the available number of demonstrations was different for each task. Specific number of demonstrations used for each task is available in Table 1, Appendix C. We will clarify this in the caption.\n\n---\n\n**Q3: Is there an upper bound or threshold beyond which increasing the sequence length causes performance to degrade?**\n\n**A3.** We find that using longer action sequences ($K  > 16$) does not improve performance and sometimes degrade performance. We hypothesize this is because training with too long action sequences can be difficult, offsetting the benefit of learning with action sequences. We also note that the best action sequence length can be different for each dataset that has different action frequencies. We will include the relevant results and discussion in the final version.\n\n---\n\n**Q4: Comparison to sequence models**\n\n**A4.** While using sequence models like Transformer for RL has a potential to achieve better performance on long-horizon tasks (as shown in offline RL papers like [1,2], we find that GRU is faster to train and leads to stronger performance in our considered setups. We hypothesize this is because it is non-trivial to train Transformer models from scratch using unstable training signals from td learning, which leaves a room for interesting future work.\n\n[1] Chebotar, Yevgen, et al. \"Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions.\" Conference on Robot Learning. PMLR, 2023.\n\n[2] Springenberg, Jost Tobias, et al. \"Offline actor-critic reinforcement learning scales to large models.\" arXiv preprint arXiv:2402.05546 (2024)."}, {"review_id": "ZRLdwyCarS", "reviewer": "Reviewer_yZtr", "summary": "Following a recent trend in behavioral cloning for robot learning, the authors augment an existing reinforcement learning (RL) algorithm, Coarse-to-Fine Q-Network (CQN) with the capability to deal with action sequences. They evaluate their algorithm, CQN-AS, on 3 sets of benchmarks (BiGym, RLBench and HumanoidBench) and show that CQN-AS generally outperforms several RL baselines which do not play action sequences and behavioral cloning (BC) baselines that do. They also provide an ablation study.", "questions": "*Major issues:*\n\n- In the introduction, the authors outline that actor-critic algorithms dealing with action sequences suffer from over-estimation bias. Then they advocate for critic-only algorithms with the argument that \"there is no separate actor that may exploit value functions\" and they pretend that as their contribution, CQN-AS, is based on a critic-only approach, it will avoid the value overestimation problem. First, the argument is wrong: DQN is a critic-only algorithm (the most basic one) and it suffers from over-estimation bias, as the mitigation with Double DQN clearly shows. Second, the authors do not measure over-estimation bias in CQN-AS in the empirical study, so they do not show that it mitigates over-estimation bias. A study of the role of over-estimation bias in RL algorithms dealing with action sequences could be a useful contribution of this paper...\n\n- About the N-step return, lines 112-117 the authors suggest that using from 1 to 3 steps works the best, but their empirical study only provides results with 1 (i.e. not N-step return) and 4 steps, which performs worse, hence suggesting that one should avoid using N-step return. This inconsistency must be solved, either by providing results with 2 and 3 steps, or by removing the N-step return idea from the main paper (showing that it fails in some Appendix may help readers avoid to try this again).\n\n- In Figures 5 and 6, the performance of all algorithms including CQN-AS always starts at 0, though the policy is pre-trained with a few (17 to 60) expert demonstrations. Do the figures start before training with the demonstrations? Otherwise, could the authors explain why?\n\n- Did all the baselines benefit from the same 17 to 60 expert demonstrations pre-training process? I assume it is the case for CQN, but for DrQ-v2+, it is less obvious. Otherwise I'm afraid the comparison is not scientifically valid.\n\n- I don't find the message from the comparison between *human-collected* and *synthetic (clean)* demonstrations (BiGym vs RLBench experiments) convincing enough. The authors do not provide compelling evidence that this feature is playing a key role, it could be other factors in the different benchmarks that explain the difference. A controlled experiment where using synthetic versus human-collected data is the only changing factor (e.g. same benchmark, just changing the dataset) would be necessary to evaluate the role of this feature. Besides, as it only plays a role in pre-training and then balanced sampling, it is unclear why it should make such a difference.\n\n- What are the differences between CQN-AS with the RL part ablated and BC-ACT? If they are close to be the same, I don't find Fig. 7b useful, as Fig. 5 already shows more or less the same results.\n\n*More local points:*\n\n- The authors could replace Figures 5 and 6 with performance profiles (see the Rliable library and \"Reinforcement learning at the edge of the statistical precipice\", Agrawal et al. NeurIPS 2021) and reject the full set of curves to an Appendix. The saved space could be used to include HumanoidBench results into the main paper\n- The authors seem to be using balanced sampling of expert demonstrations, they should give a ref to this approach (Ross&Bagnell 2012 or the more recent RLPD paper)\n- Did the authors try to ablate expert pre-training?\n- On the effect of action length, why didn't the authors try longer sequences (32, 64, etc.) as they found that the longer, the better? Using 50 is a common practice in Diffusion policy based BC...\n- The studies of the effect of temporal ensemble and of its magnitude (Figs. 7d and 7e) could be merged into a single study and the value of m is not given in Fig 7d.\n\n- In the conclusion: \"One limitation of our work is the lack of real-world robot evaluation,as RL with fragile humanoid hardware remains a significant challenge\" -> Well, this is not the right argument, as some robots are not fragile humanoids :)", "limitations": "The authors do not discuss the computational aspects of CQN-AS wrt CQN: is it faster or slower to train? Faster or slower at inference time? Does it require a larger memory? All these aspects will be important when switching to real robots", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper fills a gap: it seems that the benefits of using action sequences in RL has not been shown before\n- The paper is rather clear\n- The empirical study uses a large set of benchmarks\n- Results are generally good\n\nWeaknesses:\n- several points need further clarification (see below)\n- we could expect a larger effort on understanding WHY action sequences help. Despite ablations, the paper does not provide empirical clues on that. Maybe using much simpler environments coudl help.\n- the idea that the effect has to do with over-estimation bias is not supported (see below)\n- an empirical comparison to (Saanum et al. 2024) would improve the paper\n- the title is inadequate as there is no robot in this work", "was_revised": true, "final_justification": "The authors did a good job in clarifying their points during the rebuttal, I'm now more convinced of the value of the paper, so I switched from a brderline reject to a borderline accpet.", "rebuttal": "Dear Reviewer yZtr,\n\nWe really appreciate your time and effort for reviewing our paper. We provide our responses below.\n\n---\n\n**Q1.  Empirical evidence that shows why using action sequences is helpful.**\n\n**A1.** In Introduction, we hypothesized that using action sequences can be helpful for training RL agents by making it easier to learn value functions. Our initial observation showed that using action sequences is helpful for predicting return-to-go from robotic demonstrations. We also showed that CQN-AS, which uses action sequences, achieves strong performance, directly supporting our hypothesis. Here, the key insight is that CQN-AS is a *critic-only algorithm* where actions are selected greedily based on the learned Q-values. The strong performance of CQN-AS is therefore a direct consequence of the critic learning a more accurate value function. We will further clarify this in the final version.\n\n---\n\n**Q2. DQN also suffers from over-estimation bias. Further study on the effect of action sequences on over-estimation in RL algorithms could be a useful contribution**\n\n**A2. ** As you mentioned, RL algorithms with discrete actions also suffer from over-estimation bias as well. However, we would like to clarify that our focus in this work is on the over-estimation problem that happens when using high-dimensional actions as inputs to the critic network.\n\nIn particular, what we find in this paper is that a combination of (i) a critic network that takes high-dimensional action sequences as inputs and (ii) the actor that is trained to output actions using gradients from the critic leads to severe value overestimation problems. We hypothesize this is because the critic network tends to have more function approximation errors from high-dimensional actions and the actor can easily exploit this error. On the other hand, critic-only algorithms can avoid this problem, leading to stable training with high-dimensional action sequences.\n\nTo further support our point, we conducted additional toy experiments where we introduced redundant no-op actions for training TD3 and CQN agents on DMC Cheetah Run tasks. Specifically, we trained agents with 6 original actions and 294 no-op actions with [-1, 1] action bounds and used an environment wrapper that slices out no-op actions. In this setup, we found that CQN (critic-only algorithm with discrete actions) is indeed robust to the overestimation problem. In particular, we find that TD3 suffers from severe value overestimation with more no-op actions, while CQN with such no-op actions achieves similar performance with the original CQN agent. This clearly shows that a critic-only algorithm is more robust to training with high-dimensional action spaces.\n\n| Additional No-op actions | Episode Return (CQN) | Target Q-Values (CQN) | Episode Return (TD3) | Target Q-Values (TD3) |\n| :--- | :--- | :--- | :--- | :--- |\n| 0 | 236.61 | 18.84 | 140.67 | 16.25 |\n| 54 | 219.50 | 20.27 | 7.18 | -17.45 |\n| 144 | 185.38 | 16.42 | 0.56 | 1.00E+08 |\n| 294 | 202.94 | 18.66 | 0.27 | 4.00E+08 |\n\nBased on this observation, one potential alternative is to use expectile regression for Q-learning as in IQL [1], which can avoid value overestimation problems by avoiding the explicit maximization step in TD-learning. We will leave this as a future work and include relevant discussion in the final version.\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" International Conference on Learning Representations 2022\n\n---\n\n**Q3. It is unclear why the properties of demonstrations should make such a difference.**\n\n**A3.** This is because our setup is a very challenging sparse-reward task setup, which makes it extremely difficult to solve the tasks unless we use demonstrations for guiding the training. This necessitates RL algorithms to effectively learn useful value functions from demonstrations and then use information from online trial-and-error experiences to further improve the performance. Therefore the properties of demonstrations become quite important -- if the characteristics of demonstrations (such as noisyness of trajectories) pose challenges in learning value functions from them, RL algorithms would suffer to learn initial meaningful behaviors and thus learn from online data as well in our setup.\n\n---\n\n**Q4. Comparison between human-collected and synthetic demonstrations needs more controlled experiments.**\n\n**A4.** Thank you for your suggestion. To support our claim that value learning is easier with smoother and clean trajectories, we conducted preliminary experiments that train a return-to-go prediction model with smoothed trajectories from BiGym’s Move Plate task. In particular, we apply Gaussian filtering to demonstrations with different smoothing magnitudes, and train the model to predict return-to-go conditional on observation-action pairs. While these smoothed trajectories are not actually usable for policy learning as they have lost their precision, we conducted this experiment as a preliminary experiment for understanding the effect of demonstration characteristics on value learning. As shown in the below table, we find that the validation loss decreases with more smoothing, which shows that demonstration qualities can affect value learning, which thus affects RL training as we mentioned in **A3.**\n\n| Smoothing | Val Loss | Improvement |\n| :--- | :--- | :--- |\n| Baseline (no smoothing) | 0.0211 | |\n| Gaussian (sigma=0.5) | 0.0203 | 3.84\\% |\n| Gaussian (sigma=1.0) | 0.0188 | 10.78\\% |\n| Gaussian (sigma=1.5) | 0.0185 | 12.37\\% |\n| Gaussian (sigma=2.0) | 0.0185 | 12.22\\% |\n| Gaussian (sigma=2.5) | 0.0183 | 13.34\\% |\n\n---\n\n**Q5. The title is inadequate as there is no robot in this work.**\n\n**A5**. We respectfully disagree with the assessment of the title. Our use of simulation is in line with the standard and widely accepted practice in robot learning research, where algorithms are developed and validated for robotic agents.\n\n---\n\n**Q6. Why do performances start from 0 despite pre-training with expert demonstrations?**\n\n**A6.**  We would like to clarify that there is no pre-training. We just initialize the training by initializing the replay buffer with expert demonstrations (and maintaining a separate demonstration replay buffer as well). Therefore all the methods’ performance starts from 0. We agree that this is confusing and we missed the explanation on this. We will include sentences to clarify our setup.\n\n---\n\n**Q7. Did all baselines use the same 17 to 60 expert demonstrations?**\n\n**A7.** Yes. All methods use the same number of demonstrations. As we’ve written in Appendix C, we used different number of demonstrations for *different tasks*, but we used the same number of demonstrations for all algorithms within each task. We will clarify this in the final version.\n\n---\n\n**Q8. Differences between CQN-AS without RL and ACT?**\n\n**A8.** As we’ve written in line 163-165, ACT is an Action Chunking Transformer [3] that trains a transformer model to output action sequences. On the other hand, CQN-AS without RL shares the same architecture with CQN-AS and also the discrete action space as well, which allows us to ablate the effect of RL objective. \n\n[2] Zhao, Tony Z., et al. \"Learning fine-grained bimanual manipulation with low-cost hardware.\" Robotics: Science and Systems 2023\n\n---\n\n**Q9. Inconsistency with regard to N-step return**\n\n**A9.** This is a good point. We wanted to emphasize that there is no need to set $N$ to be equal to the length of the action sequence ($K$), as that will train RL algorithms with very large $N$-step returns leading to unstable training. Our experiments find that the $N$-step of 1 or 4 (3 was a typo, we will fix this) works better than very large $N$ such as 16. But it looks like our intent was not clear. We will further clarify this and will move it to Appendix per your suggestion.\n\n---\n\n**Q10. An empirical comparison to (Saanum et al. 2024) would improve the paper.**\n\n**A10.** Thank you for the suggestion. While we agree that such comparison would improve our paper, but it is quite non-trivial to compare our work to [Saanum et al., 2024] considering that (i) it has never been tested in our main setup that uses sparse-reward tasks and (ii) its implementation is not open-sourced. \n\n---\n\n**Minor - Q1. Using Rliable library, adding references to balanced sampling, combining Figure 7(d) and (e)**\n\n**Minor - A1.** Thank you for your suggestions. We will incorporate them.\n\n---\n\n**Minor - Q2. Performance with longer action sequences?**\n\n**Minor - A2.** We find that using longer action sequences ($K  > 16$) does not improve performance and sometimes degrade performance. We hypothesize this is because training with too long action sequences can be difficult, offsetting the benefit of learning with action sequences. We also note that the best action sequence length can be different for each dataset that has different action frequencies. We will include the relevant results and discussion in the final version.\n\n---\n\n**Minor - Q3. Remarks on fragile humanoid hardware**\n\n**Minor - A3.** We will incorporate your point that not all robots are fragile.\n\n---\n\n**Minor - Q4. Computational cost of CQN-AS compared to CQN is missing**\n\n**Minor - A4.** As we’ve written in Appendix C, training of CQN-AS is around 33% slower than CQN. At inference time, CQN-AS takes 13% more memory compared to CQN and is 40% slower than CQN. We believe these costs are definitely worthy considering the effectiveness of CQN-AS over CQN as shown in our BiGym experiments. Moreover, to further reduce the cost at inference time, we can consider adjusting the frequency of the temporal ensemble instead of computing action sequences at every time step. We will include the relevant discussion in the final version."}]

Author rebuttal:

Meta-review: The comments from reviewers do note overall issues with the paper.

1. Extension to use action sequences based on CQN looks like an incremental extension, so I wouldn’t rate this work highly in terms of novelty.
2. However, the method's performance is apparent, yet the authors do not use a reliable method to evaluate the performance.
3. Given the lack of real hardware experiments, the authors should be careful about calling this robot learning, and not just reinforcement learning.

Overall, the paper appears to provide a clean step forward in research; however, it would also benefit from additional references to related work on learning multi-step q-functions.

Kahn, Gregory, et al. "Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation." 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.
Decision: Accept (poster)