Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Authors: Jijia Liu
Feng Gao
Qingmin Liao
Chao Yu
Yu Wang
Venue: ICML 2025 poster (2025)
Keywords: Value-based Reinforcement Learning
Continuous Control
Auto-Regressive
Suboptimal Demonstration
Primary area: reinforcement_learning->deep_rl
TL;DR: We propose Auto-Regressive Soft Q-learning (ARSQ), a value-based RL algorithm that models Q-values in a coarse-to-fine, auto-regressive manner, improving policy learning in continuous control tasks especially with suboptimal data.

Abstract: Reinforcement learning (RL) for continuous control often requires large amounts of online interaction data. 
Value-based RL methods can mitigate this burden by offering relatively high sample efficiency. 
Some studies further enhance sample efficiency by incorporating offline demonstration data to “kick-start” training, achieving promising results in continuous control. 
However, they typically compute the Q-function independently for each action dimension, neglecting interdependencies and making it harder to identify optimal actions when learning from suboptimal data, such as non-expert demonstration and online-collected data during the training process.
To address these issues, we propose Auto-Regressive Soft Q-learning (ARSQ), a value-based RL algorithm that models Q-values in a coarse-to-fine, auto-regressive manner.
First, ARSQ decomposes the continuous action space into discrete spaces in a coarse-to-fine hierarchy, enhancing sample efficiency for fine-grained continuous control tasks. 
Next, it auto-regressively predicts dimensional action advantages within each decision step, enabling more effective decision-making in continuous control tasks. 
We evaluate ARSQ on two continuous control benchmarks, RLBench and D4RL, integrating demonstration data into online training. On D4RL, which includes non-expert demonstrations, ARSQ achieves an average 1.62× performance improvement over SOTA value-based baseline. On RLBench, which incorporates expert demonstrations, ARSQ surpasses various baselines, demonstrating its effectiveness in learning from suboptimal online-collected data.

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Jijia Liu Feng Gao | Qingmin Liao |
Chao Yu 12 Yu Wang' !
Abstract
1. Introduction
Reinforcement learning (RL) for continuous con- trol often requires large amounts of online interac- tion data. Value-based RL methods can mitigate this burden by offering relatively high sample ef- ficiency. Some studies further enhance sample efficiency by incorporating offline demonstration data to "kick-start" training, achieving promis- ing results continuous control. However, they typically compute the Q-function independently for each action dimension, neglecting interdepen- dencies and making it harder identify optimal actions when learning from suboptimal data, such as non-expert demonstration and online-collected data during the training process. To address these issues, we propose Auto- Regresseee Soft Q- learning (ARSQ), a value-based RL algorithm that models Q-values in coarse-to-fine, auto- regressive manner. First, ARSQ decomposes the continuous action space into discrete spaces in a coarse-to-fine finne hierarchy, enhancing sample effi- ciency for fine-grained continuous control tasks. Next, it rugo-regresyvvely predicts dimensional action advantages within each decision step, en- abling more effective decision-making in continu- ous control tasks. We evaluate ARSQ on two con- tinuous control benchmarks, RLBench and D4RL, integrating demonstration data into online train- ing. On D4RL, which includes non-expert demon- strations, ARSQ achieves an average 1.62x per- formance improvement over SOTA value-based baseline. On RLBench, which incorporates expert demonstrations, ARSQ surpasses various base- lines, demonstrating its effectiveness in learning from suboptimal online-collected data.
Deep reinforcement learning (RL) has demonstrated remark- able performance across various continuous control domains (Haarnoja et al., 2018; Schulman et al., 2017). However, these breakthroughs often come at the cost of extensive online interactions, which are required for effective con- vergence (Berner et al., 2019; Mníh et al., 2015). This reliance on large-scale exploration poses major challenge in real-world applications, where data collection can be ex- pensive, time-consuming, or even risky. To alleviate this burden, value-based RL methods, which directly approx- imate the Q-function rather than parameterizing policy, have gained popularity due to their improved sample elfi- ciency (Seyde et al., 2024; Tavakoli et al., 2021; Seyde et al., 2023) and have shown advances continuous control tasks by discretizing each of the dimensions of continuous action spaces (Seo et al., 2024). Moreover, some studies integrate offline demonstration data into training to further accelerate early learning, reducing the dependence on purely online exploration (Ball et al., 2023). In this paper, we adopt this training paradigm to address continuous control using value- based RL, incorporating offline data into the online training process.
For value-based RI, the discretization scheme results in an exponentially large discrete action space, making RL train- ing and exploration challenging. To mitigate this, existing value-based methods often estimate the Q-value for each action dimension independently (Metz et al., 2017; Seyde et al., 2023). However, this simplification comes with limi- tation it neglects interdependencies between action dimen- sions, potentially leading suboptimal decision-making. When training data exhibits multiple modes, such as a mix of optimal and suboptimal demonstrations, independently estimating Q-values can bias action selection toward the most frequent behaviors rather than the truly optimal ones. This limitation is particularly pronounced in the early stages of learning, when the agent relies heavily on imperfect of- fline data and lacks sufficient online refinement.
'Tsinghua University, Beijing, China Beiijnng Zhong- guancun Academy, Beijing, China. Correspondence to: Chao Yu eateyuchho@gmail.com... Yu Wang <yu- wang@ @tsinnguaa.duu.nn
Consider a simple one-step decision-making task with two-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Project page is https: / /ttttssssssssss////////ee/eeeess http://ssssssssss...ooooo google. conolee.co//eee view/
soottt-.

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressiv Soft Q-Network
1G
10
10
Q () 0.42
Q a) 1.00
2
Optimal Mode
r= 1.0
Action Dim 2
Action Dim 2
Action
-1.0
10'0- 10
00
Suboptimal Mode
-25
0.1
(s.
0.11
((..a) = -0.52
0.20
0
-0.85
-1.
-25
0.
05
10
-1.
-0.
0.
45
1.0
-0.5
0.0
45
Action Dim
Action Dim
1
Action Dim 1
(a) An example dataset for one-step decision-making environment.
(b) Q function given by independent ac- tion decomposition.
(c) Q function given by auto-regressive
action decomposition (Ours).
Figure I. A motivating example of how Q decomposition influences policy training, as detailed in Appendix C.I.
dimensional actions (41, a2) E === [-1,1] C R2, shown in Fig. 1, where an agent selects an action (a1, given state s and receives a reward r before the episode terminates. Suppose the training dataset consists three distinct modes: one optimal mode with r = 1, two suboptimal modes with -.1 0.1 and r 1, with the latter occurring more fre- quently. If the suboptimal modes are more prevalent in the dataset, conventional Q-learning approaches that esti- mate action dimensions independently, i.e., (s,, a:), could undervalue the optimal mode. This bias can hinder the cor- rect identification and reinforcement of the optimal action mode, leading to slow convergence and degraded policy performance.
learning and imitation learning baselines. Results indicate that ARSQ consistently surpasses these baselines, achiev- ing up to 1.62 performance over existing value-basee RL when trained with suboptimal demonstrations on D4RL. Ab- lation studies further highlight the significance of ARSQ's key components, confirming its effectiveness in continuous control tasks.
Our contributions include:
We extend Soft Q-learning framework to value-based reinforcement learning with dimensional advantage estimation.
We propose the ARSQ algorithm to capture dependen- cies action dimensions and enhance learning from suboptimal data.
To address this issue, we propose Auto nuto-RRerressvee Soft Q- learning (ARSQ), novel approach that captures cross- dimensional dependencies in discretized high-dimensional action spaces. Instead of treating each dimension inde- pendently, ARSQ adopts an auto-regressive structure, se- quentially estimating advantages for each action dimension conditioned the previously selected dimensions. This allows the method to better model interdependencies, ensur- ing that correlated action dimensions are jointly optimized rather than selected in isolation. Additionally, ARSQ adopts a coarse-to-fine hierarchical discretization strategy inspired by CQN (Seo et al., 2024), further enhancing sample effi- ciency for fine-grained continuous control. We theoretically show that the original Q function can be expanded into an regressive formulation with dimensional advantage estimation under the framework of soft Q-learning. Our ap- proach integrates these insights into an auto-regressive soft Q-network, which is specifically designed for continuous control tasks.
Through extensive experiments, we demonstrate that ARSQ can learn better policies when data suboptimal- ity arises from either offline datasets or data collected online.
2. Related Works
Value-based RL for Continuous Control. Despite their inherently straightforward critic-only framework, value- based reinforcement learning (RL) algorithms have achieved notable success (Mnih et al., 2015; Silver et al., 2017: Schrit- twieser et al., 2020; Seyde et al., 2024; Seo et al., 2024). Although these algorithms are primarily designed for dis- crete action spaces, recent efforts have sought adapt them to continuous control by discretizing the continuous action space (Tavakoli et al., 2018: 8; Seyde et al., 2023). However, the curse of dimensionality remains a significant challenge, as the number of discretization bins increases exponentially with the action dimension (Lillicrap, 2015). To address this issue, some studies have modified the Markoy Decision
To evaluate ARSQ, we conduct extensive experiments on the D4RL and RLBench continuous control benchmarks, challenging it against variety of widely used reinforcement
2

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Process (MDP) of the environment, transforming it into sequential decision-making problem along the action dimen- sion (Metz et al., 2017; Chebotar et al., 2023). Other ap- proaches treat each action dimension independently, gener- ating the Q function separately for each dimension (Tavakoli et al., 2018; 2021; Seyde et al., 2023; 2024), akin to treating each action dimension as a multi-agent RL problem (Foer- ster et al., 2018; Yu et al., 2022). Recent research (Seo et al., 2024) has employed coarse-to-fine discretization approach to improve sample efficiency. However, treating each action dimension independently may disrupt the correlation be- tween different action dimensions, potentially diminishing performance in policy optimization. Some studies (Seo & Abbeel, 2024) have attempted to solve this issue through action sequence prediction. Our approach generates actions in an auto-regressive manner, considering the correlations between dimensions and improving policy learning, which is orthogonal to (Seo & Abbeel, 2024).
by the tuple 7,,,, do). Here, S is the continuous state space, A is the continuous action space, y (0, is the discount factor, p(s' | a) is the transition dynamics, r(s, a) is the reward function, and do(s) is the distribution of the initial state. In addition to interacting with the envi- ronment online, we assume access to a pre-collected dataset D = {(ss 0, Ti, 6() J. which can substantially reduce sam- ple complexity and provide broader state-action coverage.
3.2. Soft Q Learning
To improve policy exploration, maximum entropy RL en- hances the reward by adding an entropy term (Ziebart et al., 2008; Haarnoja al., 2017; 2018), so the optimal policy seeks to maximize entropy at every state it visits. The ob- jective is defined as
T
J(x) Σ
E (s, (r(s, + a+((T((ss)) (I)
Online RL with Offline Demonstration. Deep reinforce- ment learning often requires a large amount of online inter- actions to achieve convergence (Berner et al., 2019; Mnih et al., 2015). To address this challenge, many methods have been proposed that leverage offline demonstrations to guide online exploration and accelerate policy training (Ra- jeswaran et al., 2018; Ball et al., 2023). Some approaches involve performing offline RL pretraining before initiating online RL training (Lee et al., 2022; Nakamoto et al., 2023; LEI et al., 2024; Hu al., 2024). However, these approaches often depend on expensive offline pretraining. To mitigate this, some works explore incorporating offline demonstra- tion data directly into the training process. One strategy initializes the replay buffer with offline data (Hester et al., 2018; Ball et al., 2023), while another balances sampling between online and offline data to improve training stabil- ity (Zhang et al., 2023; Hansen et al., 2023). Additionally, certain methods explicitly introduce a behavior cloning loss to leverage high-quality demonstrations for better guidance (Rudner al., 2021; Rajeswaran al., 2018; Nair et al., 2018). In this work, adopt the paradigm of integrating offline demonstrations into training to enhance sample effi- ciency in continuous control tasks. Specifically, we improve value-based RL by introducing an auto-regressive structure that sequentially estimates advantage for each action dimen- sion. This design enables better handling of suboptimal data, whether from offline demonstrations or trajectories collected during training.
where H. is entropy, T is the episode length and Pr is the trajectory distribution induced by policy æ. The temperature parameter a dictates how much importance is placed on the entropy term in comparison to the reward. Let the soft Q-function and soft value function defined as:
Qion an) = ret
E( Jupe
Σ
80
re (re-1 + aH (T" (1
(2)
(=n) log exp ( (' da'
(3)
Then the optimal policy for Eq. (1) is given by
x (arls)) = exp (t
( (Qion (Se, at) - Vion (si))
(4)
Similar to the standard Q-function and value function, the Q-function can be connected to the value function at a future state using a soft Bellman equation.
Qion (St, a.) = r + 7
[V (51+1)
(5)
The proof can be found in (Ziebart et al., 2008; Haarnoja et al., 2017).
4. Method
3. Preliminaries
In this section, we begin by discussing the process of dis- cretizing multi-dimennional actions in a coarse-to-fine man- ner. Building on this, we extend the soft Q-learning theory with a focus on the dimensional soft advantage. Subse- quently, we introduce our Auto- Regreseegees Soft Q-learning (ARSQ) algorithm, which is overviewed in Fig. 2.
3.1. Problem Formulation
In this paper, we consider the standard RL setting with the addition of a pre-collected dataset D for continuous control. The problem can be represented as MDP, defined
3

---

Learning from Suboptimal Data in Continuous Control via Auto-Regresse Soft Q-Network
Auto-regressive Network
Conditioning
Action Dim
Action Dim
Action Dim
Advantage
Mask
Level
coarse-to-fine Discretization
Sunde
Semple
Saaat
Policy
staf) xxp
((rl))
Dim
Dim
actiion 15.1,9)
Dimensional conditioning
Level
Coarse-to porrse-to-iine Conditioning
Level 2
Figure 2. The ARSQ algorithm. The action space is discretized using coarse-to-fine approach. By predicting dimensional soft advantages, ARSQ generates actions in an auto-regressive manner within single decision- makinn step.
4.1. Coarse-to-fine Action Discretization
expressed as
To apply Q-learning (Mnih et al., 2015) in continuous domain, straightforward approach is to discretize the ac- tion space (Tang & Agrawal, 2020; Seo et al., 2024). For a continuous action of d dimensions a, = (a ag ........ . a²) E RDD, the discretized action a = (u a², a...,a") can represented by
( (2
Venn = a log
Σ exp a'EA
(8)
And we omit the subscript for 8, and a, further stream- line the expression of the policy, we define the soft advan- tage.
Definition 4.1 (Soft Advantage). The soft advantage of at is given by
a² arg max la bil
where b = .b.,.... bn) are the centers of B discretization intervals, or bins, which typically provide a uniform separa- tion of the given action space. However, obtaining finer separation of the action space necessitates a greater number of bins, thereby increasing the computational load when assessing the Q function for each discrete action bin.
(s, = (s. a) V (s)
Similar to the advantage policy gradient-basee RL algo- rithms, the soft advantage assesses how much taking action a at state s beneficial. Thus, the optimal policy in Eq. (4) can be expressed
To address this issue, discretization approach (Seo et al., 2024), similar to the method used in (Yan et al., 2015) for computer vision, as illustrated in Fig. 2. With L levels and uniform separation bins each level, the discrete action for dimension at level 2 is expressed as:
apply coarse-to foneine action
T (als) = exp ('
=exp
(10)
Considering the multi-dimensional action space, it still re- mains necessary use neural network to output B values in the final layer, as per the DQN (Mnih al., 2015).
-L-igds BL-1
However, outputting such large number of Q values im- poses significant computational burden on the neural net- work. Inspired by auto-regression (Brown et al., 2020), we address this problem by make the policy æ generate action a (a), a² .......)) P) auto-regressively along the action dimensions.
Here, [:] represents the floor function.
During inference, the policy generates discrete actions progressively through each level (a (1.2 ... a (1.1.. These are then combined to produce the final discrete action.
For clarity, treats discrete action discussed in Sec. 4.1 in one level. The multi-level coarse-to-fin fine discrete action can be considered as additional action dimensions, without compromising generalization. We first define the dimen- sional soft advantage to represent the auto-regressive policy at dimension d.
4.2. Dimensional Soft Advantage for Policy Representation
Building on action discretization, we initially extend soft Q-learning to discrete spaces. The soft value function is

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Definition 4.2 (Dimensional Soft Advantage). The dimen- sional soft advantage of the action a at state consid- ering the previous generated dimensional actions a a-d
Algorithm Auto-Regressive Soft Q Algorithm (ARSQ)
Initialize 0,,,, ø1,2 for A and V
Assign target parameters 0,, Ø; + 8,, 6,. Offine dataset 'D, replay buffer R t- D. for each epoch do for each environment step do select a, with Ag, and As, (10, 16)
) is expressed by (a',"""a"" 1)
L (s,
T(a"ss x erp
(11)
S p( |8c; ax)
However, the dimensional soft advantage is not normalized for each action dimension d. We propose the following theorem to establish connection between the dimensional soft advantage and the soft advantage.
R < RU {st ac. Tt, St-1)
end for
for each gradient step do
Sample mini-batch bp. by from D, R
Theorem 4.3. If the dimensional soft advantage (s, a a²)) satisfies
Calculate Ln - LRI. + BL with bn (15, 18)
Calculate Ln = LRL. with bn (18)
Update me, according Va, (CD LR)
Σ exp (_,4(s,a-4,00) ad
Update Va according to V (LD + Li)
(12)
Update target networks 0. + p0; + (1 p)0; and
Ø, f --p; + p)o: end for end for
for all dimension d, then the soft advantage can then be expressed as the summation of the dimensional soft advan- tages
a ,a,)) a(s,a)
(13)
actions present in the offline dataset to be preferred over other actions. Specifically, we define the loss as
Proof. See Appendix A.
C Σ max 4.00 A4.0 a -d ,aa) ad
(15)
Additionally, Eq. (12) shows that the exponential of the dimensional soft advantage represent valid probability distribution. Using Eq. (11) together with Theorem 4.3, the dimensional soft advantage serves as bridge between policy representation and Q prediction.
A (s, a a Cm)
where a denotes the expert action observed in the offline dataset, and Gm is hyper-paramette controlling the margin. This objective encourages the soft advantages of expert actions to be at least C'n higher than those of other actions.
Since we not introduce additional elements policy optimization. the Q-iteration follows the same update rule as soft Q-learning. Based on Eq. (5), have
Policy Representation. As discussed in Sec. 4.2, ARSQ predicts dimensional soft advantages, which function as both components of the Q function and policy representa- tion. The network architecture is illustrated in Fig. 3. In practical design, the soft value Vun and the dimensional soft advantage A are predicted using two separate neu- ral networks. The advantage prediction network estimates the dimensional soft advantage for each action dimension, based on the partially generated action from previous di- mensions, creating an auto-regressive sequence. practical design, we use a globally-shared MLP in the advantage net- work, with separate heads predict the dimensional soft advantages.
a+) + Ti- YEs, r~p(s) (st41) (14)
The maximum entropy policy described in Eq. (4) can be obtained by repeatedly applying Eq. (14) until it converges.
4.3. Auto-Rette RettoRRereesive Soft Q-learning
Building the theory outlined in See. 4.2, we introduce the Auto-Regressin Soft Q-learning (ARSQ) algorithm. The pseudo code for the ARSQ algorithm is presented in Al- gorithm I. We will discuss the various design choices of ARSQ.
Behavior Cloning Objective. To leverage offline demon- stration data during online training, we introduce an ad- ditional behavior cloning loss term. Following previous works (Hester al., 2018; Seo al., 2024), we encourage
Another challenge is applying the constraint of the dimen- sional soft advantage as per Eq. (12). Here, we enforce a hard constraint by normalizing each output head through log-sum-exp subtraction, ensuring consistency across out-
5

---

Learning from Suboptimal Data in Continuous Control via Auto-Regresive Soft Q-Network
observation
Ø
a
Infer
Infer
Infor
Value Network
Advantage Shared Backbone
Infer
V (5))
Head
Head 2
Head
Head
3
4
u (s,, 0.-
u² (Sp. a 2,)
u a=)
(s
1 xd - log Σ (ep) ad o8
Advantage Constraint
(sp, 0.)
A A a 2,)
A a ,)
A ( a (,)
Figure 3. Network architecture of ARSQ. The soft value Van and the dimensional soft advantage A are predicted by two separate networks. The advantage network utilizee shared backbone, advantage constraints are applied to its output.
puts.
for each dimension in an auto-regressive sequence, while coarse-to- to-fee conditioning involves generating actions for each dimension from coarse fine. In practice, we imple- ment coarse-to-fine conditioning prior dimensional condi- tioning. Specifically, dimensional conditioning serves the inner conditioning, while coarse-to-fine fine conditioning acts as the outer conditioning across levels. We explore swapping the order of conditioning in Sec. 5.4, and the results indicate that the current design better captures interdependencies between action dimensions.
A dd ",a")=
(s a,)))
alog Σon exp () ur a
(16)
where ud is the output of the d-th output head.
Furthermore, stabilize training and address the over- estimation problem (Fujimoto et al., 2018; van Hasselt et al., 2016), we implement double Q-learning approach with two separate value networks and their corresponding tar- get networks. Specifically, we maintain two online value networks & and V along with their respective target soft and V Each target network is updated as an exponential moving average (EMA) of its respective online network parameters. The value target is then com- puted by taking the minimum of the two target networks' predictions
5. Experiment
We design our experiments to investigate the following ques- tions: (i) What is ARSQ's performance when the offline dataset is suboptimal? (ii) What is ARSQ's performance when online collected data is suboptimal? (iii) How do various design factors of ARSQ affect the performance?
networks V
seft
Benchmarks. We evaluate our approach two continuous control benchmarks: D4RL (Fu et al., 2020) and RLBench (James et al., 2020). Both domains provide access to online interaction data and a limited number of demonstrations, enabling us to assess the performance of ARSQ in diverse settings. We present representative results here due lim- ited space and leave full results in Appendix D.
Yz (pps) min (Vioh (5:41), Vigh (s++1)) (17)
Thus the resulting optimization objective becomes
(vrh (s:) + (st, y1) (18)
LRL
Baselines. We use CQN (Seo al., 2024), state-of-the- art value-based RL method for continuous control, as our baseline. CQN employs a coarse-to-fine action selection strategy and independently predicts Q-values for each action dimension. Additionally, CQN trains using a combination of online training and offline demonstrations. Besides, we also include DrQ-v2 (Yarats et al., 2022), renowned actor- critic algorithm designed for vision-based RL, along with its enhanced version, DrQ-v2+, as benchmarks. We also
where A is the soft advantage function parameterized by 0,
Auto-regressive Conditioning. In Sec. 4.2, we explained the process of handling discrete action in one coarse-to- fine level. With multi-level coarse-to- fine action discretiza- tion, the auto regressiee conditioning encompasses two as- pects. Dimensional conditioning refers to generating actions
6

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
CON
100
ARSQ
91.5
85.8
Normalized Return
62
75.0
76.1
35.8
51.6
52.8
50
41.3
37.1
20.0
12.0
2.5
balfchwetah-mtr
balicheeiahem
halfchettah-mee
happes-mr
hogper-m
happer-me
walker2d-me
#uker2d-m
walker2d-me
Total
Figure 4. D4RL main results. mr, m, and me represent mediumm-replay. medium, and medium-expeer respectively.
feature ACT (Zhao et al., 2023) and CQN-style behavior cloning (BC) policy among our baselines. Details about the baselines can be found in Appendix C.3.
enhances policy performance.
100
CON
5.1. Performance on D4RL
ARSQ (Ours)
Normalized Return
Main Results. To evaluate ARSQ's performance when the offline dataset is suboptimal, we consider three distinct locomotion tasks from the D4RL benchmark, each with three datasets of varying quality. The medium dataset is gathered using medium-level policy, whereas the medium- expert dataset comprises combination of medium-level and expert demonstrations. The medium-replay dataset includes data ranging from completely random to medium-level. The input to the model consists of state representations, while the output corresponds to torques applied at each hinge joint. dense reward is provided encourage completing the task, staying alive, and discourage vigorous actions that consume excessive energy.
69.8
62.-
53.1
46.7
43.2
middle Dalaset Quality (%)
top
Figure 5. D4RL results different demonstration quality aver- aged over tasks, with each containing datasets respectively. We report the normalized return provided by D4RI.
We evaluate ARSQ, CQN (Seo et al., 2024), and BC in this setting. At the beginning of online training, the replay buffer for both ARSQ and CQN initialized with an offline dataset, and online data added as the training progresses. Additionally, both ARSQ and CQN incorporate the BC objective (Eq. (15)) towards offline dataset. The BC baseline trained solely offline using the offline dataset with the BC objective. We report the converged performance of ARSQ, CQN and BC, averaged over three random seeds.
Analysis on Demonstration Quality. To better investi- gate the influence dataset quality, we rank trajectories by episode return for each dataset, and labeling the top 30%, middle 30%, and bottom 30% of the data offline demon- strations. The behavior cloning objective is applied only to these offline demonstrations. We report the converged performance ARSQ, CQN and BC over three random seeds. As illustrated in Fig. 5, ARSQ consistently outperforms both CQN and BC across all three levels of demonstration quality. Notably, when using the bottom 30% of data as of- fline demonstrations, ARSQ achieves approximately 2.0x the final performance of CQN. In contrast, with the lowest demonstration quality, CQN performs slightly worse than BC, revealing CQN's sensitivity to demonstration quality, which negatively affects its online training. These results further validate the effectiveness our method when using suboptimal offline datasets.
As shown in Fig. 4, ARSQ exhibits outstanding performance across all nine datasets, demonstrating its ability to effec- tively identify suboptimal actions and learn efficiently from the available offline data. ARSQ surpasses CQN, par- ticularly in the medium-replay and medium-exper eeppert datasets, where optimal data is not predominant, highlighting that ARSQ less biased toward frequently observed suboptimal actions. Notably, both ARSQ and CQN outperform BC, indicating that conducting reinforcement learning online

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
1.00 open_oven
take_plate
pick_up_co cup
open_door
meat_on
press, press_switch
Success Rate
50-
Steps
ARSQ (Ours)
CQN
v2Q
DrQ
ACT
Figure 6. RLBench results on different tasks. Each experiment begins with 100 expert demonstrations, and all RL methods include a hehavior cloning objective.
5.2. Performance on RLBench
jectories based on total reward, and we re-evaluate DWBC under the same conditions on additional datasets.
To further evaluate ARSQ's performance, we focus on six tasks from RLBench (James al., 2020). The agent re- ceives input RGB images and proprioceptive states and outputs the change in joint angles to control the robot arm. Unlike D4RL,, the reward is sparse, offering binary value (0 or 1) only at the final timestamp. Although each task is provided with 100 expert demonstrations, the agent might gather unsuccessful trajectories during its interaction with the environment. This setup allows us examine the per- formance when the data collected online suboptimal.
The results are presented in Tab. 1. All data is sourced from the respective papers, with the reevaluated DWBC results marked by "*" Both ARSQ and re-evaluated DWBC are assessed using 10 trajectories over three random seeds. ARSQ demonstrates superior overall performance compared to the other baselines, indicating its capability to effectively manage suboptimal data under fully offline setting.
5.4. Ablation Studies
In this domain, evaluate the performance of ARSQ, CQN, DrQ-v2+, DrQ-v2, ACT and BC. All reinforcement learning methods incorporate the behavior cloning objective (Eq. (15)) towards expert demonstrations and successful trajectories collected online. Results are averaged over three random seeds.
In this section, we evaluate the impact of key design factors in ARSQ: auto-regressive conditioning (Fig.2) and advan- tage prediction network (Fig.3).
Ablation on Auto-regressive Conditioning. We consider several variants of ARSQ on auto-regressive conditioning.
As shown in Fig. 6, ARSQ demonstrates superior perfor- mance compared to all other algorithms, highlighting its effectiveness in online learning with suboptimal collected data. Additionally, ARSQ exceeds ACT, highlighting the importance of reinforcement learning in online training.
Swap: We reverse the conditioning order, applying dimensional conditioning first, followed by coarse-to- fine conditioning.
w/o CF Cond.: We remove the coarse-to-Bee to-fine condi- tioning and output actions at multiple levels simultane- ously.
5.3. Performance under Fully Offline Setting
To further examine the performance of ARSQ, we conduct experiments a fully offline setting, where the algorithm learns solely from a predetermined dataset. We utilize nine locomotion datasets from D4RL, as outlined in Sec. 5.1. For offline RL methods, we employ CQL (Kumar al., 2020), IQL (Kostrikov et al., 2022), TD3+BC (Fujimoto & Gu, 2021), Onestep RL (Brandfonbrener et al., 2021), and RvS- R (Emmons et al., 2022) as baselines. For offline imitation learning methods capable of handling suboptimal data, we use filtered BC (Chen et al., 2021; Emmons et al., 2022), Decision Transformer (Chen al., 2021), and DWBC (Xu al., 2022) as baselines. For DWBC, We adopt its best performance under "Setting 2" from its original paper (Xu et al., 2022), which mark top 5% trajectories as expert tra-
w/o Dim Cond.: We remove the dimensional condition- ing and instead output all action dimensions simultane- ously at each level.
w/o CF: We replace the coarse-to-fin fine structure entirely by discretizing each action dimension into B² bins and then applying dimensional conditioning.
Plain: We remove both the coarse-to-fine structure and dimensional conditioning.
We report results on hopper-mediam-esper and hopper- mediam-replay from D4RL, as well as Open Oven from RLBench, all evaluated across three random seeds. As depicted in Fig. 7, Swap demonstrates slight decline in performance, underscoring the effectiveness of the current
8

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Table 1. Performance under fully offline setting.
Dataset
CQL
IQL
TD3+BC
Onestep RL
RvS-R
Filt. BC
DT
DWBC
ARSQ (Ours)
halfcheetah-m
44.0
47.4
48.3
48.4
41.6
42.5
42.6
$41.4
43.7
hopper-m
58.5
66.3
59.3
59.6
60.2
56.9
67.6
*56.0
99.2 0.5
walker2d-m
72.5
78.3
83.7
81.8
71.7
75.0
74.0
*72.3
81.2
halfcheetah-mr
45.5
44.2
44.6
38.1
38.0
40.6
36.6
38.9
0.1
hopper-mr
95.0
94.7
60.9
97.5
73.5
75.9
82.7
73.0
90.7
walker2d-m
77.2
73.9
81.8
49.5
60.6
62.5
66.6
59.8
74.0 + 2.6
hefe
91.6
86.7
90.7
93.4
92.2
92.9
86.8
*93.1
92.4 1.2
hopper-me
105.4
91.5
98.0
103.3
101.7
110.9
107.6
*110.4
110.9 + 1.0
walker2d-me
108.8
109.6
110.1
113.0
106.0
109.0
108.1
*108.3
107.9 + 0.3
Total
698.5
692.4
677.4
684.6
645.5
666.2
672.6
653.2
741.1
100
100
Nomal Regum
Bate
100
37.5
922
50
Succiss
44.5
835
455
25
Normalous
ARSO
Septraate
Lmnal Shaned
30k
20k
Environment
Seei Cad..
win of
Cond.
ARS0
wis wib Dim Cond.
Separase
Seap
M/ Cond.
Plain
Figure 8, Ablation on shared backbone in D4RL (left) and RL- Bench (right).
Figure 7. Ablation on auto-regressive conditioning D4RL (left) and RLBench (right).
conditioning order design. Additionally, removing any of the components degrades performance to varying degrees. When all components are removed, as Plain, the per- formance is at lowest, emphasizing the significance of dimensional and coarse-to-fine action generation.
6. Conclusion
In this paper, we introduced Auto-Regressive Soft Q- learning (ARSQ). a novel value-based RL approach tailored for continuous control tasks with suboptimal data. ARSQ addresses the limitations of existing value-basee bbste methods by adopting an auto-regressive structure that sequentially estimates soft advantage for each action dimension, thereby capturing cross-dimensional dependencies. Through empiri- cal evaluations, we show that ARSQ significantly surpasses existing methods, highlighting its effectiveness in learning from suboptimal data.
Ablation on Shared Backbone. The advantage network of ARSQ utilizes shared backbone to reduce the number of parameters and speed up the learning process. To assess the impact of this choice, we introduce two variants. The network architecture these two variants can be found in Appendix C.3.
Separate: We employ separate networks for each action dimension.
For future directions, adaptive coarse-to-fine discretiza- tion can be used to balance control granularity with the overhead additional bins. Another approach to explore grouping unrelated dimensions to shorten the conditioning chain length, thereby speeding up computation.
Level Shared:
We employ shared networks for each coarse-to-fine level.
We report results hopper-medium from D4RL and Open Oven from RLBench over three random seeds. As shown in Fig. 8, the standard ARSQ consistently performs well in both environments. In contrast, using either a level-shared or separate backbone results in diminished performance. This demonstrates the effectiveness of the shared backbone design.
A Acknowledgements
This work was supported by National Natural Science Foundation of China NNo.6246115, 62325405), Postdoc- toral Fellowship Program CPSF under Grant Number (GZC20240830, 2024M761676), China Postdoctoral Sci- ence Special Foundation 2024T170496.

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Impact Statement
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 15084-15097. Curran Associates, Inc., 2021.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline RL via supervised learn- ing? In International Conference on Learning Represen- tations, 2022. URL https: htttts://openrere// /ppeneeieeeeeeeeeeeeeeeeee net/ forum?id=S3 -d==8874XAIPPR-
Ba, J. L. Layer normalization. arXiv: 1607.06400 2016.
arXiv preprint
Ball, P. J., Smith, L.., Kostrikov, I., and Levine, S. Efficient online reinforcement learning with offline data. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 1577-1594. PMLR, 23-29 Jul 2023.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelli- gence, 32(1), Apr. 2018, doi: 16.1609/aaii v32i1. 1794.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint urXiv:2004.07219, 2020.
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint 1912.0660,, 2019.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 20132-20145 Curran Associates, Inc., 2021.
Brandfonbrener, D., Whitney, W., Ranganath, R., and Bruna, Offline rl without off off-policy evaluation. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 4933-4946. Curran Associates, Inc., 2021.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th In- ternational Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 1587- 1596. PMLR, 10-15 Jul 2018.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan. A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G.,
Haarnoja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-based policies. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, vol- ume 70 of Proceedings of Machine Learning Research, pp. 1352-1361. PMLR, 06-11 Aug 2017.
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J.,
Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray. S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018
pp. 1877
Chebotar, Y., Vuong, Q.. Hausman, K., Xia, F., Lu, Y., Irpan, A., Kumar, A., Yu, T., Herzog, A., Pertsch, K., Gopalakrishnan, K., Ibarz, J., Nachum, O., Sontakke, S. A., Salazar, G., Tran, H. T., Peralta, J., Tan, C., Man- junath, D., Singh, J., Zitkovich, B., Jackson, T., Rao, K., Finn, C., and Levine, S. Q-transformer: Scalable offline reinforcement learning via autoregressive qufunctions. In Tan, J.. Toussaint, M., and Darvish, K. (eds.), Proceed- ings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 3909-3928. PMLR, 06 09 Nov 2023.
Hansen, N., Lin, Y., Su, H., Wang, X., Kumar, V., and Rajeswaran, A. Modem: Accelerating visual model- based reinforcement learning with demonstrations. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: opwnreview net/ forum? Corum?i==ddTnneggiie
10

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Haykin, S. Neural networks: a comprehensive foundation. Prentice Hall PTR, 1998.
and Learning Systems, 33((2):699--7019 2022. doi: 10. 20009/NNNSS200213300080
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: 2016.
Lillicrap, T. Continuous control with deep reinforcement
learning. arXiv preprint arXiv:1509.02977 2015.
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, Leibo, J., and Gruslys, A. Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaai w0t.///aa.......11111.... v3211.11777.
Metz, I.., Ibarz, J., Jaitly, N., and Davidson, J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv: /t0ivv/700.0000 05035, 2017.
I., Dulac-Aroodd, G., Agapiou, Leibo
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Hu, H.. Yang. Y., Ye, J., Wu, C., Mai, Z., Hu, Y., Lv, T., Fan, C., Zhao, Q. and Zhang. C. Bayesian design principles for offline-to-online reinforcement learning. In Salakhut- dinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the s1st International Conference on Machine Learning, vol- 235 of Proceedings of Machine Learning Research, pp. 19491-19515, PMLR, 21-27 Jul 2024.
Nair, A., McGrew, B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 6292-6299, 2018. doi: 10.1109/CRR.22018.44660 8446316......
James, S., Ma, Z., Arrojo, D. R., and Davison, A. J. RI- bench: The robot learning benchmark learning environ- ment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020. doi: 10.1109/LRA.200022747000
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C.,, Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Informa- tion Processing Systems, volume 36, pp. 62244 62269. Curran Associates, Inc., 2023.
Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. In International Confer- ence on Learning Representations, 2022. URL https: //openreview net forum? ? 6dd
Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schul- man, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.). Advances in Neural Information Process- ing Systems, volume 33, pp. 1179-1191. Curran Asso- ciates, Inc., 2020.
Rudner, G. Lu, C.. Osbome, M. A., Gal, Y., and Teh, Y.
On pathologies in kl-regularized reinforcement learning from expert demonstrations. In Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Sys- tems, volume 34, pp. 28376 28389. Curran Associates, Inc., 2021.
Lec, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Faust, A., Hsu, D., and Neu- mann, G. (eds.), Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 1702-1712. PMLR, 08-11 Nov 2022.
Schrittwieser, J., Antonoglou. I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D.,, Graepel. T., et al. Mastering atari, go. chess and shogi by planning with learned model. Nature, 588(7839): 604-609, 2020.
LEI, K., He. Z., Lu, C., Hu, K., Gao. Y., and Xu, H. Uni-o4: Unifying online and offline deep reinforcement learning with multi-step on-policy optimization. In The Twelfth International Conference on Learning Representations, 2024. URL https:////........... http://wwnweeeeeeeeeeeeee n forum? id=tbFBh3IMKi.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06447 2017.
Li, Z., Liu, F., Yang, W., Peng, S., and Zhou, J. survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks
Seo, Y. and Abbeel, P. Reinforcement learning with action sequence for data-efficient robot learning. arXiv preprint A.iv:2411.12/55 2024.

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressve Soft Q-Network
Seo, Y., Uruç, J., and James, S. Continuous control with coarse-to-fine reinforcement learning. In 8th Annual Conference Robot Learning, 2024, URL https: net. w/opennevvew.nee////eeeeeeeeeeee
In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Yarats, D., Fergus, R., Lazaric, A., and Pinto, L. Master- ing visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https:// openreview net worumrirddiioo Lardd__S-_yy___
Seyde, T., Werner, P., Schwarting, W., Gilitschenski, I., Riedmiller, M., Rus, D., and Wulfmeier, M. Solv- ing continuous control via q-learning. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///pperrreeeeeeeeeee wttpi//wwwwe//oorrrt//e id-U5X0GxAgeeS.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and WU, Y. The surprising effectiveness of ppo in cooperative multi-agent games. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 2461 1661-44424. Curran Associates, Inc., 2022.
Seyde, T., Werner, P., Schwarting. W., Wulfmeier, M., and Rus, D. Growing Q-networks: Solving continuous con- trol tasks with adaptive control resolution. In Abate, A., Cannon, M., Margellos, K., and Papachristodoulou, A. (eds.), Proceedings of the 6th Annual Learning for Dy- namics Control Conference, volume 242 of Proceedings of Machine Learning Research, pp. 1666-1661. PMLR, 17-17 Jul 2024.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging offline-to- online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ //tp//ppppeeeeeeeeeeeeeeeeeoeeee net n id= Y34445JR62.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge, nature, 550(76766:354-55,, 2017.
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware, 2023.
Tang, Y. and Agrawal, Discretizing continuous action space for on-policy optimization. Proceedings of the AAAI Conference on Artificial Intellience, 34(04):59811 5988, Apr. 2020. doi: 10.1609/aami.v344000 v34104.6099.
Ziebart, B. D., Maas, A.. Bagnell, J. A., and Dey, A. K.
Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd National Conference on Artificial
Intelligence Volume 3, AA pp. 1433-1438. AAAL Press, 2008 ISBN 9781577353683.
Tavakoli, A., Pardo, F., and Kormushev, P. Action branching architectures for deep reinforcement learning. Proceed- ings of the AAAI Conference on Artificial Intelligence, 32 (1), Apr. 2018. doi: 10. 1111/669aai.vv33 wit....322..11111 11798.
Tavakoli, A., Fatemi, M., and Kormushev, P. Learning to rep- resent action values as a hypergraph on the action vertices. In International Conference on Learning Representations, 2021. URL https: / http:::://ppeeeeeeeeeeee www.reviw...w..................p (d=xv_s64Fix)
wwt.iiewweeeeeeeeeeeeeeeeeeeeeee net/ forum?
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double q-learning Proceedings of the AAAI Conference on Artificial Intelligence, 30)1), Mar. 2016. doi: 10. 10.1609/a 330.9/aai.v00.1.... v30i1.10295.
Xu, H. Zhan, X., Yin, H., and Qin, H. Discriminator- weighted offline imitation learning from suboptimal demonstrations. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learn- ing Reseurch, pp. 24742. PMLR, 17-23 Jul 2022.
Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., and Yu, Y. Hd-enn: Hierarchical deep convo- lutional neural networks for large scale visual recognition.
12

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
A. Proof of Theorem 4.3
First, we express the policy using conditional probability, and then replace it with Eq. (11).
(als)
II (a ( aaa
dm1
D
cap a
Z(s a
d=1
(19)
I ezp
1 A a aaa)))
II 2 (s, 4-4)
1 D A A a a-, aa4)
erp
a 2dd=
11
(ds a
We can then apply Eq. (12), resulting in
D
x <als) = erp
1
Σ
Ass a²))
14 (s,
(20)
d=1
Recall that the policy raals) can be represented using the soft advantage as shown Eq. (10). Therefore, we have
Σ (s, a ",at) a)
(21)
B. Implementation Details
B.1. Action Selection
As illustrated in Algorithm 1, the action selection process receives inputs from Ag, and Aø, and produces a Eq. (10) and Eq. (16) describe the action selection process utilizing single soft advantage network. To leverage the benefits of double network, we employ two advantage networks to generate more precise actions. This process is detailed in Algorithm 2.
Algorithm 2 ARSQ Action Selection with Double Q Network
Input: parameter 01,2 for A state 8
Output: action ar Initialize output action a, 0 for each action dimension d do Compute A Ad, (s,, a,, a ) for each ad (16) Compute A (a = min; Ad,0; (st. ar, ad)
Compute 7 (a)) = exp (a)) (10)
Normalize æd by x (ad) = En *
(a
Sample discrete action at dimension d with (a")
Append action a, a, U {a
end for
B.2. Variant of Behavior Cloning Objective
As discussed in See. 4.3, we incorporate behavior cloning objective to effectively utilize offline demonstration data during online training, as defined in Eq. (15).
Following prior works (Kumar et al., 2020), we also employ variant of this objective, expressed as:
C BC- = max
log Σ exp (Ad.0: (s, a aaa)) ( a a a.)) Om
(22)
13

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
where of is the expert action and C. is predefined margin constant.
We observe that this variant objective achieves better performance in scenarios where action modes are concentrated, such as in the medium and medium-exper series of datasets in D4RL. Consequently, we adopt this variant objective when working with such datasets.
B.3. Network Architecture
In RLBench tasks, observations consist of a combination of RGB images and low-dimensional states. To compute the dimensional soft advantage for given dimension, we first input the RGB images and now-dimensional states into Convolutional Neural Network (CNN) (Li et al., 2022) encoder and a Multi-Layer Perceptron (MIP) (Haykin, 1998) encoder, respectively, to extract feature representations. These representations are then used to predict the soft value. Concurrently, the feature representations are combined with actions from previous dimensions and coarse-to-fing levels to create auto-regressive conditioning. An ML.P-based shared backbone and output head are then utilized determine the dimensional soft advantage for the given dimension.
In D4RL tasks, observations consist solely of fow-dimensional states, and feature representations are derived directly from these states.
B.4. Hyper-parameters
Table 2. Typical hyper-parameters of ARSQ in D4RI, and RLBench.
Syyer-parameee
D4RL
RLBench
Image resolution
84 84 x RandomShift
Image augmentation
Frame stack
1
8
CNN Encoder
/
Conv (c=[32, 64, 128, 256], s=2, p=1)
Backbone
Linear (512, 512)
Linear (512, 512, 512, bias=False)
Output Head Layers
1
Activation
Tanh
SiLU & LayerNorm
Coarse-to-fine Levels
2
3
Coarse-to-fine Bins
7
5
Batch Size
512
512
Optimizer Learning Rate
Adam
AdamW (weight decay = 0.1)
3e-4
5e-5
Temperature Coefficient a
0.01
0.001
Target Critic Update Ratio (r)
0.005
0.02
BC Margin Cm Action Roll-out Network
-0.01
Current
Target
The hyperparameters of ARSQ are presented in Table 2. We provide the typical hyperparameters for ARSQ in D4RL (hopper-medium) and RLBench (Open Oven). In RLBench, ARSQ employs RandomShift (Yarats et al., 2022) for image augmentation. Additionally, ARSQ utilizes SiLU (Hendrycks & Gimpel, 2016) and LayerNorm (Ba, 2016) activation functions RLBench.
C. Experiment Setup
C.1. Motivating Example Setup
As introduced in Sec. and illustrated in Fig. 1, we consider a motivating example to demonstrate the impact of Q decomposition on policy training. The dataset is depicted in Fig. la, with each point to be a data point in the dataset. The color of the data points indicates the reward of the data point. To illustrate the Q function of value-based RL algorithms, we first discretize the action space with 2 bins in each action dimension.
14

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Q function given by independent action decomposition is an example of DecQN (Seyde et al., 2023), well in CQN (Seo al., 2024), which features just single coarse-to-fine level. In this setting, we employ separate tabular Q functions, (1) and Qss, a2), for action dimension and action dimension 2. The Q function is learned by gradient descent.
For the function obtained through auto-regressive action decomposition, we employ both tabular soft advantage functions, (s, ai) and (3 01, (2) for action dimension and action dimension 2, and a tabular soft value function (s). The Q value reported in Fig. ic is sum of the soft value and the dimensional soft advantage of the corresponding dimensions, i.e., Q(s, 01, 02) = (s) + A (s, + (s, The soft advantage functions and the soft value function are simultaneously learned through gradient descent.
C.2. Environment and Dataset
D4RL Gym Environment. D4RL (Fu et al., 2020) provides datasets for various tasks evaluate the performance of reinforcement learning. In this context, use 3 Gym Locomotion tasks and datasets from D4RL to assess the performance of ARSQ and other baselines. These tasks are illustrated in Fig. 9. The agent's observations include its states, such as the angle and velocity of each rotor. The agent's ' actions consist of torques applied between the robot's links, constrained within the range (-1,1). The reward is dense, offering incentives for task completion and survival, while penalizing excessive energy- mnrgy-consuming actions.
Figure 9. D4RL Gym tasks used in experiment.
D4RL Dataset. In D4RL, we use the medium-replay, medium, and medium-expert datasets for tasks involving half-cheetah, hopper, and walker2d. In Section 5.1, to examine the impact of dataset quality, we rank trajectories based on episode returns within these nine datasets. Specifically, we compute the total reward for each data chunk within each dataset. We then rank these data chunks and select the top, middle, and bottom 30% accordingly. This is akin to rank trajectories but is easier to handle.
To better demonstrate the suboptimal nature of the datasets, we plot a histogram of the data chunk rewards, as shown in Fig. 10.
RLBench Environment. RLBench (James al., 2020) serves as a benchmark and learning environment for robot control. We have selected 20 tasks from RLBench and present results for 6 of them in Sec. 5. An illustration of the environment be seen in Fig. 11. The input consists of RGB images with resolution of 84 x 84, captured from four camera angles: front, wrist, left-shoulder, and right-shoulder, along with a history of the past seven observations. The output specifies the change in joint angles at each time step, utilizing the delta JointPosition mode provided by RLBench. In our experiments, we use a binary sparse reward system (0 or 1), which is awarded only at the final timestamp of an episode to indicate task success.
C.3. Baselines and Evaluation Details
Main Results Baselines. As mentioned in Sec. 5.1, within D4RL, we utilize the implementation from (Seo et al., 2024) and modify its CNN-based encoder to an MLP-based encoder as the CQN baseline. The BC baseline originates from CQN but operates with the RI. learning objective turned off and without any online environment interaction.
15

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressee Soft Q-Network
halfcheetth-medium-rrppaa
halfcheetah-medium
halfthhetah-medlum-experrr
140
250
8-
200
Count
100
40
2
20
0.0
1.0
hopper-medium-replay
hopper-medium
hopper-medimm-eppet
140
350
200
100-
250-
Count
550-
150
100-
50-
20
50
1.0
0.2
0.8
1.0
0.2
0.8
1.0
walke 22dmmmeium
walke wewerr2--ddiu-exppeee
175
150-
125
400
Count
100-
300
73
40-
200
50-
20
100
0.0
0.7
0.4
0.8
0.8
1.0
1.2
0.0
0.2
0.4
0.8
0.8
3.0
1.2
0.0
0.2
0.4
0.6
1.0
1.2
Narmalized Reward
Normalized Reward
Normalized Reward
Figure 10. Histogram of reward D4RL datascts.
In RLBench, we adopt DrQ-v2+, an optimized variant of DrQ-v2 proposed by (Seo et al., 2024), as our baseline. DrQ-v2+ incorporates several optimization strategies introduced the CQN algorithm. Specifically, compared DrQ-v2, DrQ-v2+ employs a distributional critic instead of standard critic network. utilizes an exploration strategy with small Gaussian noise, and features optimized network architectures and hyperparameters tailored for RLBench tasks. These enhancements strengthen DrQ-v2+'s performance, making it more robust baseline than DrQ-v2. Additionally, DrQ-v2+ has been open-sourced by (Seo et al., 2024).
Ablation Study Baselines. As mentioned in Sec. 5.4, we utilize the Separate and Level Shared backbone baselines for an ablation study to explore the effectiveness of the shared backbone in the advantage network. The network architectures of these baselines are illustrated in Fig. 12 and Fig. 13.
D. Additional Results
Sensitivity of Temperature Coefficient a. Our methods are derived from Soft Q-learning, which aims to achieve a maximum-entropy policy. The temperature coefficient a in Eq. (1) affects the balance hetween maximizing policy entropy and the reward from the environment. We conducted experiments to examine how varying a impacts policy learning.
As shown in Fig. 14, a very high a results in reduced performance and unstable training, whereas a very low a also hampers policy improvement by restricting exploration.
Training Curves of D4RL Main Results. In Section 5.1, we discuss the converged performance of ARSQ, CQN, and BC. The training curves for each task are shown in Fig. 15. ARSQ converges after approximately 25,000 to 50,000 environment steps and generally outperforms the CQN and BC baselines across most tasks. This further demonstrates ARSQ's strength in managing suboptimal data.
16

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
Figure 11. Example of RLBench tasks used experiment.
observation
a
arr PHE
Infer
Infer
Infer
Value Network
Backbone 1
Backbone 2
Backbone
Backbone
Infer (5a)e(5)
Head
Head 3
Head
xt (5, 0-)
ur a
x (X.. ?,)
(8,,
At m uf log Σ Ceppet ktt
Advantage Constraint
(sp d.)
A2(sa 2,)
A (5,a %,)
A A a
Figure 12. Network architecture of Separate backbone baseline in ablation study.
D4RL Results per Task for Different Demonstration Quality. In Sec. 5.1, we present the D4RL results, averaged over all 9 datasets, based on varying demonstration quality. The results for each task are illustrated in Fig. 16. ARSQ consistently outperforms the CQN and BC baselines in nearly every task, demonstrating its ability maintain stable performance across datasets of varying quality.
RLBench Results in All 20 Tasks. In Sec. 5.2, present results for six selected tasks from RLBench. The complete results for all 20 tasks are displayed in Fig. 17. These results indicate that ARSQ performs comparably or better across these tasks, showcasing its ability to learn effectively even when the data collected online is not optimal.
E. Computational Cost Analysis
As discussed in Sec. 4.3, ARSQ generates actions in each dimension in an auto-regressive manner. To analyze the overhead, conducted experiments both D4RL (hopper-medium) and RLBench (Open Oven) tasks. The training and inference times for ARSQ and CQN were evaluated 1,000 times and averaged. These experiments were conducted on single Nvidia RTX 3090 graphics card.
The results are shown in Fig. 3. ARSQ exhibits similar training times to CQN, due to the parallel optimization implemented and the batch training nature of the auto-regressive model. However, ARSQ experiences higher inference latency compared CQN. We aim address this issue by grouping the action dimensions and outputting the grouped dimensional actions ruto-regressivell. solution we plan explore in future work.
F. Performance under Fully Online Setting
In addition to the main experimental results presented in Sec. 5.1, we assess the performance of ARSQ in fully online setting. We compare the fully online reinforcement learning performance of ARSQ and CQN on the hopper task, using PPO(Schulman al., 2017) as baseline for comparison. The results are depicted in Fig. 18, with all experiments conducted
17

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
observttio
at
ot
Infer
Inter
Infer
Infer
Value Network
Backbone
Backbone
Inter
(
Head
Head
N² (S, 0.)
At ut luy Σ (o
Advantage Constraint
Ca A-)
(' 3,)
a' w
Level 1
Level 2
Figure 13. Network architecture of Level Shared backbone baseline ablation study.
100
Moraltze Retum
92.2
762
50
10k
2ók
30k
Environment Steps
1=
e.
dede
sedeed
0.1
0.01
0.001
Figure 14. Sensitivity of temperature coefficient a, evaluated on hopper-medium from D4RL and Open Oven from RLBench over three random seeds.
using three random seeds. We also illustrate the performance of both vanilla and offline ARSQ using the one of the hopper dataset, specifically hopper-mediumt replayy.
Online ARSQ achieves a similar converged performance to vanilla ARSQ, albeit requiring more environment steps. This highlights the importance of using offline datasets enhance sample efficiency. Furthermore, ARSQ with online interaction achieves higher final performance than in the offline setting, suggesting the necessity of online interaction enhance policy performance. Additionally, online ARSQ demonstrates greater sample efficiency than CQN and PPO, underscoring its potential as versatile reinforcement learning algorithm.
G. Error Analysis of Q Prediction and Action Discretization
To further examine the error introduced by action discretization, as discussed in Sec. 4, we designe more complex case study similar to Fig. 1. We create a one-step environment featuring two-dimensional action space (01;02) E A = 1-1, 1]2 R The agent performs action at the initial timestamp, receives reward, and the episode concludes. The Q function's 's ground truth landscape, akin to the reward landscape, is illustrated in Fig. 19a. There is one optimal action mode, two sub-optimal action modes, and two negative action modes.
We uniformly sample 2,000 data points from the environment to form a dataset. This dataset then used to train agents with independent Q decomposition for each action dimension, ARSQ without hierarchical coarse-to-fine action discretization, and the standard ARSQ. The resulting landscapes are displayed in Fig. 19, and Q prediction errors are displayed in Fig. 20. When independently decomposing for each action dimension, the agent learns blurred Q landscape, complicating the identification of optimal actions. ARSQ without coarse-to-fine fine action discretization produces Q landscape similar to the vanilla ARSQ but with more "glitches,"" likely because too many action bins make it difficult for the dataset to cover them comprehensively. This underscores the importance of coarse-to-fine action discretization.
Furthermore, we sample 1,000 data points in the proposed environment and calculate the Q prediction error against the ground truth for all three methods discussed, over three random seeds. The results are presented Tab. 4. Independent Q
18

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressiv Soft Q-Network
halfchestab-mr
halfcheetah-m
halfchoetah-me
Normalized Retur
25k
50k
75k
100k
25k
S0k
75k
100k
25k
50K
75k
100k
hopper-mr
hopper-m
hopper-me
Normalized Retur
25k
50k
75k
100k
25k
S0k
75k
100k
25k
50k
75k
100k
walker2d-mr
walker2d-m
walker2d-me
Normalized Return
25k
50k
75k
200k
100k
25k
50k
75k
100k
Env. Steps
Env. Steps
Env. Steps
ARSQ (Ours)
CQN
BC
Figure 15. Training curves of D4RL main results, evaluated over three random seeds.
decomposition results in significant error increase compared to ARSQ and continuous learning. Additionally, ARSQ without coarse-to-fine fine action discretization also results in higher Q error, further highlighting the necessity of our action discretization strategy.
19

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
halfchecth--mdium-replay
halfcheetah-medde
halfcheetah--eeiuu-kkae
100
100
100
Noralizdd
424
30
50
-
top
hogper-mediumreelayy
hopper-medium
hopper-medium-expert
1231
200 128
1306
1.00
910
100
100
Nomalized
Sia
13
114
13:011
bottom 30%
Middle 30%
top 30%
bottom 30%
middle 30%
top 30%
bottom 30%
middle 30%
top 30%
walker2d-mediumeee ummprrll
walker2d-medium
walker22d-mdduu-expper
10L4
Normaized
50-
50
50
13.3
bottomn 30%
middle Quality
top 30%
buttom 30%
middle 30%
bottom 30%
middle 30% Quality
30%
Quality
BC
CON
ARSO [Ours]
Figure 16. D4RL results per task different demonstration quality, evaluated three random seeds.
Table 3. Computational time in D4RL and RLBench (ms).
D4RL
RLBench
ARSQ Inference
4.1
32.1
ARSQ Training
12.2
290.5
CQN Inference
2.6
6.9
CQN Training
11.6
260.5
Table 4. Q prediction enror with different action discretization strategy.
Discretization Method
Error
Independent Decomposition
17.57 + 0.67
ARSQ w/o Coarse-to-fine
0.50 0.21
ARSQ
0.16 0.02
20

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
open, drawer
stack wine
toilet
open microwave
146
5400000 R40
114
23c
33%
24
136
284
Jbc
ogen oven
turn. tap
put money
phone
29c
36f
2
1kc
20
30c
13k
30c
10
3kc
sweep.
open,
75-
5000000
38c
30x
basketball
switch
rubbish
insert usb
Succes Ro1e
26
Stgpa
Enn. Stopa
Supa
Erv.
Stuse
AA50 (Dural
CON
DrQ
3/2+2
ACT
Figure 17. RLBench results in all 20 tasks.
100
Normalizes lized Retur
50
100k
200k
300k
400k
Env. Steps
ARSQ
CQN (online)
ARSQ (online)
PPO
ARSQ (offline)
Figure 18. Performance under fully online settings, hopper task ropper-mediam-replayy dataset.
21

---

Learning from Suboptimal Data in Continuous Control via Auto-Regressiv Soft Q-Network
00
Actan
(a) Ground truth.
(b) Independent action decom- position.
(c) ARSQ w/o coarse-to-fine discretization.
(d) ARSQ.
Figure Visualization Q prediction with different action discretization stralegy.
-12
A
Dex
(a) Independent action decom- position. Figure 20. Q prediction emrrr with different action discretization strategy.
(b) ARSQ w/o coarse-to to-fine discretization.
(c) ARSQ.
22


Reviews:
[{"review_id": "7P47XxhNjd", "reviewer": "Reviewer_5ENF", "summary": "This paper studies the problem of reinforcement learning for continuous control with action discretization, specifically focusing on offline RL (D4RL benchmark) and RL from demonstrations (RLBench) as problem settings. A key limitation of prior work that leverages action discretization is the explosion in dimensionality that occurs if the continuous action space is jointly binned in a grid-like manner across all dimensions. Another option that prior work has explored is the use of separate action binning across each dimension, which avoids the aforementioned explosion but in turn makes an assumption about action dimension independence. This work proposes ARSQ, a method for coarse-to-fine hierarchical action discretization in an auto-regressive framework that overcomes both of these two issues by auto-regressively conditioning action dimension/coarse-to-fine sampling on the previously sampled ones (nicely illustrated in Figure 2). Experiments are conducted on D4RL and RLBench, and the proposed method outperforms CQN, an existing method for coarse-to-fine discretization, as well as recent methods for imitation learning (BC, ACT) and online RL (DrQ-v2).\n\n## Post-rebuttal assessment\n\nI appreciate the detailed response to my comments. The additional comparisons and clarifications address my concerns. I am raising my score from Weak Accept -> Accept under the assumption that these changes (along with those requested by my fellow reviewers) will be included in the camera-ready revision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed and constructive feedback. We appreciate the positive remarks about\nthe motivation, clarity, originality, and empirical results of our work.\n\nBelow, we respond to the reviewer’s concerns point-by-point:\n\n## 1. Additional Baseline Results on D4RL\n\nWe agree with the reviewer that additional baseline comparisons on the well-established D4RL benchmark would strengthen\nour evaluation.\nFollowing the reviewer’s suggestion, we have included results from several offline RL methods (CQL[1], IQL[2],\nTD3+BC[3], Onestep RL[4], RvS-R[5]) and offline imitation methods (Filtered BC[5], Decision Transformer(DT)[6],\nDWBC[7]).\n\nWe have included the results in Table 1\nof [Supplementary Material](https://anonymous.4open.science/r/icml25-Submission9509_2/fig_5ENF.pdf).\nAll data are sourced from the respective papers, and we re-evaluate DWBC with extending datasets (marked by \"*\").\nThese results are summarized below for convenience:\n\n| Dataset        | CQL    | IQL   | TD3+BC | Onestep RL | RvS-R | Filt. BC | DT    | DWBC    | **ARSQ (Ours)** |\n|----------------|--------|-------|--------|------------|-------|----------|-------|---------|-----------------|\n| halfcheetah-m  | 44.0   | 47.4  | 48.3   | *48.4*     | 41.6  | 42.5     | 42.6  | \\*41.4  | 43.7 ± 0.6      |\n| hopper-m       | 58.5   | 66.3  | 59.3   | 59.6       | 60.2  | 56.9     | 67.6  | \\*56.0  | *99.2 ± 0.5*    |\n| walker2d-m     | 72.5   | 78.3  | *83.7* | 81.8       | 71.7  | 75.0     | 74.0  | \\*72.3  | 81.2 ± 0.9      |\n| halfcheetah-mr | *45.5* | 44.2  | 44.6   | 38.1       | 38.0  | 40.6     | 36.6  | 38.9    | 41.1 ± 0.1      |\n| hopper-mr      | 95.0   | 94.7  | 60.9   | *97.5*     | 73.5  | 75.9     | 82.7  | 73.0    | 90.7 ± 4.4      |\n| walker2d-mr    | 77.2   | 73.9  | *81.8* | 49.5       | 60.6  | 62.5     | 66.6  | 59.8    | 74.0 ± 2.6      |\n| halfcheetah-me | 91.6   | 86.7  | 90.7   | *93.4*     | 92.2  | 92.9     | 86.8  | \\*93.1  | 92.4 ± 1.2      |\n| hopper-me      | 105.4  | 91.5  | 98.0   | 103.3      | 101.7 | *110.9*  | 107.6 | \\*110.4 | *110.9 ± 1.0*   |\n| walker2d-me    | 108.8  | 109.6 | 110.1  | *113.0*    | 106.0 | 109.0    | 108.1 | \\*108.3 | 107.9 ± 0.3     |\n| Total          | 698.5  | 692.4 | 677.4  | 684.6      | 645.5 | 666.2    | 672.6 | 653.2   | **741.1**       |\n\nThe above results demonstrate that ARSQ achieves competitive performance with existing offline RL/IL\nalgorithms, further demonstrating its potential as a versatile RL algorithm.\n\n[1] Kumar et al. (2020). Conservative q-learning for offline reinforcement learning. NeurIPS 33.  \n[2] Kostrikov et al. (2021). Offline reinforcement learning with implicit q-learning. arXiv:2110.06169.  \n[3] Fujimoto et al. (2021). A minimalist approach to offline reinforcement learning. NeurIPS 34.  \n[4] Brandfonbrener et al. (2021). Offline rl without off-policy evaluation. NeurIPS 34.  \n[5] Emmons et al. (2021). Rvs: What is essential for offline rl via supervised learning? arXiv:2112.10751.  \n[6] Chen et al. (2021). Decision transformer: Reinforcement learning via sequence modeling. NeurIPS 34.  \n[7] Xu et al. (2022). Discriminator-weighted offline imitation learning from suboptimal demonstrations. ICML.\n\n## 2. Clarification on Number of Seeds\n\nWe apologize for the previous lack of clarity regarding the number of random seeds used.\nTo clarify, all experiments were conducted using three different random seeds.\n\nWe will ensure this information is clearly stated in the revised version of the paper.\n\n## 3. Details on the DrQ-v2+ Baseline\n\nWe thank the reviewer for pointing out this omission.\nDrQ-v2+ is an enhanced variant of DrQ-v2, proposed and open-sourced by Seo et al[8].\nIt incorporates several key improvements, including a distributional critic, an exploration strategy using small\nGaussian noise, and optimized hyperparameters.\nThese enhancements make DrQ-v2+ a more competitive baseline compared to the original DrQ-v2.\n\nWe will include a detailed description of the DrQ-v2+ baseline in the revised paper.\n\n[8] Seo et al. Continuous Control with Coarse-to-fine Reinforcement Learning. CoRL.\n\n## 4. Clarification on Target Value Computation\n\nSpecifically, we train two separate value networks and maintain two corresponding target networks, which are updated\nusing exponential moving averages of the online value networks. The value targets are computed as the minimum of the two\ntarget network outputs.\n\nWe will clarify this in the revised paper.\n\n## 5. Improved Figure Readability\n\nWe thank the reviewer for highlighting the readability issues in some figures. We will improve this in the revised\npaper.\n\nWe greatly appreciate the reviewer's insightful feedback, which has helped us significantly improve the manuscript. We\nwould be happy to incorporate any additional suggestions from the reviewer.", "claims_and_evidence": "The main claims of the paper (that auto-regressive conditioning is beneficial, and that ARSQ outperforms existing methods on competitive benchmarks) are supported by empirical evidence. I believe that the proposed method is well motivated based on observed limitations in prior work, and the illustrative example in Figure 1 is helpful for understanding the problem. Ablations support the claim that the specific formulation of auto-regressive conditioning used in the proposed method is beneficial and favorable over a number of alternatives (e.g. conditioning only on dimensions or coarse-to-fine levels).", "methods_and_evaluation_criteria": "The experimental setup is appropriate for evaluation of the method in question. D4RL and RLBench are competitive, commonly used environments in related literature, and the two benchmarks span both low-dimensional state representations as well as RGB images as inputs, and cover both pure offline RL and RLfD. Results are mostly convincing.\n\nI presently have three concerns regarding the evaluation:\n\n* There are few baseline results included for D4RL. I find this somewhat surprising given that it is a well established benchmark with lots of benchmark results readily available in prior literature. I would recommend that the authors include a few more baseline comparisons, e.g. IQL [1] and CQL [2]. I understand that this submission focuses on action discretization and that the aforementioned methods do not, but it would be helpful to have a set of established results on this benchmark for comparison. I believe that numbers can be extracted directly from the respective papers.\n\n* Number of seeds is not specified, except for L350 referencing D4RL. It would be greatly appreciated if the authors could clearly detail the number of seeds used for each experiment/figure.\n\n* Given that DrQ-v2+ is not a well established method but rather proposed as a baseline in prior work CQN, it would be helpful to include a description of this method in the submission (to make it self-contained) as well as a (potentially brief) comment on how the baseline was obtained/implemented (presumably on top of the public implementation of DrQ-v2).\n\n[1] Kostrikov et al., Offline Reinforcement Learning with Implicit Q-Learning, https://arxiv.org/abs/2110.06169 (2021)\n\n[2] Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning, https://arxiv.org/abs/2006.04779 (2020)", "theoretical_claims": "I have not checked the theoretical claims in detail, but did not encounter any glaring issues while reading through the paper.", "experimental_designs_or_analyses": "See my previous comments in the \"methods and evaluation criteria\" field.", "relation_to_broader_scientific_literature": "The work is generally well positioned wrt prior literature on the topic of action discretization.", "essential_references_not_discussed": "I am not aware of any works that are currently missing from the list of references, but it is possible that I missed some.", "other_strengths_and_weaknesses": "In summary\n\n**Strengths:** The paper is generally well written and easy to follow. The illustrations are helpful for understanding the problem setting and proposed method. The experimental setup appears solid (aside from the few concerns already mentioned), and empirical performance gains are substantial. I have no concerns regarding the originality of the work. I believe that the contributions will be of interest to the ICML community.\n\n**Weaknesses:** I have a few concerns regarding the experimental evaluation, namely lack of (1) baseline results for D4RL, (2) clarity regarding number of seeds, and (3) details regarding the DrQ-v2+ baseline. I believe that these issues, while important, can easily be addressed during the rebuttal period.", "other_comments_or_suggestions": "It is clear from Eq. 17-18 how the value target is computed, but the couple of lines preceding that are a bit ambiguous. Maybe the authors can consider rephrasing it to make it explicit that they train two value networks and use two target networks that (presumably) are exponential moving averages of the two online value networks + that the value target then is computed as the minimum of the two targets. This is standard practice in the field but may not be obvious to uninitiated readers.\n\nAlso minor, but some of the figures are rather small and/or have small fonts which makes them difficult to read. I suggest that the authors revisit the formatting of the paper to mitigate that.", "questions_for_authors": "I would like the authors to address my previous comments regarding the experimental evaluation / weaknesses. Details about baselines such as DrQ-v2 and DrQ-v2+ could potentially be added to the appendices.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZjxdOjM3W", "reviewer": "Reviewer_L88h", "summary": "The paper proposes ARSQ, a value-based reinforcement learning method to improve learning from suboptimal data in continuous control tasks.  Previous methods estimate Q-values independently for each action dimension, neglecting their interdependencies, leading to biased action selection with mixed-quality data. ARSQ addresses this by modeling Q-values in an auto-regressive manner, A coarse-to-fine hierarchical discretization is porposed in improving efficiency in high-dimensional action spaces. Experiments on D4RL and RLBench benchmarks show that ARSQ achieves sota performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback.\nSupplementary Material for our response is at\n[THIS LINK](https://anonymous.4open.science/r/icml25-Submission9509_2/fig_L88h.pdf).\n\nBelow, we address each point raised:\n\n## 1. Analysis of the Nature of Suboptimality\n\nWe agree that explicitly analyzing the nature of suboptimality is important, and provide two additional analyses to\nbetter illustrate suboptimality of datasets:\n\n- Trajectory Reward Analysis (Fig. 1 in Supplementary Material): We visualize histograms of trajectory rewards in D4RL\n  datasets, intuitively showing varying data quality.\n\n- Case Study (Fig. 2 in Supplementary Material): We introduce a simplified environment and compare ARSQ against action dimension independent Q decomposition methods.\n  The learned Q landscape demonstrates ARSQ's ability to learn accurate Q-values despite suboptimal data.\n\n## 2. Comparison to Alternative Discretization Strategies\n\nWe agree that ablations of alternative discretization strategies are important. In fact, we have provided some analyses\nalong these lines in Section 5.3, where we evaluate several alternative discretization variants, including\nvariants without hierarchical coarse-to-fine discretization (w/o CF Cond., w/o CF), variants that generate actions\nindependently for each dimension (w/o Dim Cond., Plain), and a variant\nthat swaps the conditioning order (Swap).\n\nThe results show that ARSQ consistently outperforms all other variants, highlighting the effectiveness of our\ndiscretization strategy.\n\n## 3. Comparison to Offline RL and Offline IL Methods\n\nWe appreciate this valuable suggestion.\nIn response, we have included additional comparisons between ARSQ and several popular offline RL and IL methods, in\nTable 1 of Supplementary Material.\nRegarding mentioned baselines:\n\n- **SafeDICE** primarily targets scenarios with labeled non-preferred trajectories. Thus, we adopt its preliminary\n  version, **DWBC**, which is more applicable to our setting.\n- **EDAC** is orthogonal and can be integrated to ARSQ in principle. However, due to time constraints, we leave this\n  integration for future work.\n- **IQL** and other offline RL/IL baselines are included in our revised paper.\n\nThese comparisons further support the effectiveness of ARSQ in learning from suboptimal data.\n\n## 4. Analysis of Theorem 4.3\n\nThe normalization condition in Theorem 4.3 (Eq. 12) is indeed essential for the conclusion that the soft advantage can\nbe decomposed into the summation of dimensional soft advantages (Eq. 13). Without this assumption, dimensional\ndecomposition does not hold. To ensure this normalization constraint in practice, we\napply a hard constraint during training via log-sum-exp subtraction (Eq. 16), ensuring consistency and theoretical\nvalidity.\n\nWe also identified a typo in Eq. 16, where the temperature parameter $\\alpha$ was inadvertently omitted. The corrected\nequation is:\n\n$$ A^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^d) = u^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^d) - \\alpha log \\sum _{a^{d'}} \\exp ( \\frac{1}{\\alpha} u^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^{d'}) )$$\n\nWe will correct this typo in the revised manuscript.\n\n## 5. Convergence to Optimal Q-function under Approximation Errors\n\nOur algorithm updates value and advantage functions via soft Bellman iteration (Eq. 14). Haarnoja et al. [1][2] have\ndemonstrated that, in the absence of approximation errors, the optimal soft\nBellman operator possesses a unique fixed point and is a contraction mapping in the infinity norm with contraction\nfactor $\\gamma < 1$. Thus, we expect our value iteration to converge close to the optimal solution if approximation\nerrors remain bounded.\n\n[1] Haarnoja et al. (2017). Reinforcement learning with deep energy-based policies. ICML.  \n[2] Haarnoja et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic\nactor. ICML.\n\n## 6. Hierarchical Discretization Bias and Error Analysis with Suboptimal Data\n\nWe conducted an error analysis in a simplified 2D environment, in Fig. 2 & 3 and Tab. 2 in Supplementary Material.\n\n| Discretization | Q Prediction Error |\n|----------------|--------------------|\n| Independent    | 17.57 ± 0.67       |\n| ARSQ w/o CF    | 0.50 ± 0.21        |\n| ARSQ           | 0.16 ± 0.02        |\n\nThe results demonstrate that both dimensional conditioning and hierarchical coarse-to-fine (CF) discretization\nsignificantly reduce Q-value prediction bias, further highlighting the necessity of our action discretization strategy.\n\n## 7. Generalization to Unseen Tasks or Out-of-Distribution Environments\n\nWe agree that analyzing ARSQ's generalization to unseen tasks or out-of-distribution environments is highly relevant.\nHowever, such analysis is beyond our current scope. We acknowledge this as an important future research direction.\n\nWe thank the reviewer again for their valuable feedback, which significantly improves the clarity and rigor of our\npaper.", "claims_and_evidence": "The paper offers good foundation for many of its claims, particularly through evaluations on standard benchmarks such as D4RL and RLBench. Still, tackling some remaining issues would further enhance the submission:\n\n1. While ARSQ demonstrates improved performance when trained on suboptimal datasets, the nature of the suboptimality is not deeply analyzed.\n\n2.The discretization method is discussed, but there is no direct comparison with alternative discretization strategies to prove its superiority.", "methods_and_evaluation_criteria": "The proposed ARSQ and evaluation criteria are largely appropriate for the problem of RL from suboptimal data in continuous control.", "theoretical_claims": "The theoretical claims in the paper are generally well-motivated.", "experimental_designs_or_analyses": "The experimental design in the paper is generally well-structured, leveraging D4RL and RLBench as benchmark datasets, comparing against strong baselines, and including ablation studies to isolate key design choices. \n\nHowever, there are several areas where the soundness and validity of the experiments could be improved: There lacks comparison to offline RL methods (such as IQL, CQL, etc.), and RL/IL methods that handle suboptimal data (SafeDICE, EDAC). The reviewer believe a further comparison could enhance the soundness.", "relation_to_broader_scientific_literature": "The key contributions of the paper build upon several existing themes in reinforcement learning (RL), particularly in value-based RL for continuous control, handling suboptimal data, and hierarchical action representations.", "essential_references_not_discussed": "[1] Jang, Y., Kim, G. H., Lee, J., Sohn, S., Kim, B., Lee, H., & Lee, M. (2023). SafeDICE: offline safe imitation learning with non-preferred demonstrations. Advances in Neural Information Processing Systems, 36, 74921-74951.\n\n[2] An, G., Moon, S., Kim, J. H., & Song, H. O. (2021). Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34, 7436-7447.\n\n[3] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1 Theorem 4.3 assumes a normalization condition on the dimensional soft advantage function, Is there empirical validation or an approximation analysis to show its impact when the condition is violated?\n\n2. The hierarchical discretization approach breaks down continuous action selection, but does it introduce bias in Q-learning updates? Would the method still converge to the optimal Q-function under function approximation errors?\n\n3. Can you provide an error analysis of ARSQ’s predictions when trained on suboptimal data? How well does ARSQ generalize to unseen tasks or out-of-distribution environments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ngxMNDVhHZ", "reviewer": "Reviewer_9ufZ", "summary": "This work introduces an algorithm for continuous control with discretized actions. Building upon coarse-to-fine Q learning, this algorithm further models advantages and policies in an autoregressive fashion, breaking the limiting assumption of independence between action dimensions. Update rules are derived from a soft variant of Q-learning, and are combined with a behavior cloning component. This method is evaluated on offline-to-online settings on D4RL and RLBench, displaying strong performance compares to purely offline baselines, or methods modeling action dimensions independently (Seo et al., 2024).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and valuable suggestions. We appreciate your careful reading and constructive\nfeedback.\nSupplementary Material for our response is at\n[THIS LINK](https://anonymous.4open.science/r/icml25-Submission9509_2/fig_9ufZ.pdf).\n\nBelow we address your specific questions and concerns one by one and provide clarifications and additional\nresults as requested.\n\n## 1. Clarifications on Theoretical Claims (Eq. 3 and Eq. 10)\n\nWe are greatly thankful for your careful reading and for pointing out the ambiguity in the theoretical claims.\n\nIndeed, the exponential of advantages is inherently normalized due to its definition.\nIn Soft Q Learning[1], the soft value function is defined using the soft Q function (Eq. 4)\n\n$$V^*_{\\text{soft}}(s_t) = \\alpha \\log \\int_{A} \\exp ( \\frac{1}{\\alpha} Q^*_{\\text{soft}}(s_t, a') ) da'$$\n\nNote that the soft value is a *softmax* of Q, and is *different* from the common concept of value\nfunction in RL. By construction, this directly implies:\n\n$$\\int_{A} \\exp ( \\frac{1}{\\alpha} ( Q^*_{\\text{soft}}(s_t, a_t) - V^*_{\\text{soft}}(s_t) )) da_t = 1$$\n\nwhich means that the exponentiated soft advantages are already normalized.\nThis equality, crucial to our theoretical claims (Eq. 3 and Eq. 10 in our manuscript), has been rigorously\nproven in Theorem 1 of the original Soft Q-Learning paper [1].\n\nTo address the reviewer’s valuable comment, We will explicitly clarify this point in the revised paper to strengthen the\ntheoretical justification.\n\n[1] Haarnoja et al. (2017). Reinforcement learning with deep energy-based policies. ICML.\n\n## 2. Clarification on Theorem 4.3\n\nThank you for highlighting the complexity of Theorem 4.3.\n\nTheorem 4.3, which exactly means the exponential of the dimensional soft advantage represents a valid\nprobability distribution, allows us to express the overall soft advantage function as the sum of the dimensional soft\nadvantages.\nIn this formulation, the dimensional soft advantage serves as a critical link between auto-regressive policy\nrepresentation and Q-value prediction.\n\nWe agree with the reviewer that the conditions of Theorem 4.3 are non-trivial and may not be satisfied naturally. To\naddress this, we enforce a hard constraint by normalizing the output of the dimensional advantage prediction network.\nSpecifically, we apply a log-sum-exp subtraction, as defined in Eq. (16). This normalization ensures that Theorem 4.3\nholds, thereby validating the correctness of all subsequent derivations.\n\nAdditionally, we identified a typo in Eq. (16), missing temperature parameter $\\alpha$. The corrected equation is:\n\n$$ A^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^d) = u^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^d) - \\alpha log \\sum _{a^{d'}} \\exp ( \\frac{1}{\\alpha} u^d(\\mathbf{s}_t, \\mathbf{a}^{-d}, a^{d'}) )$$\n\nWe will rearrange the theoretical claims and correct this typo accordingly in the revised manuscript.\n\n## 3. Evaluation on Fully Offline and Fully Online Settings\n\nWe agree with the reviewer that evaluating ARSQ in both fully offline and fully online settings is essential for\ncompleteness.\n\n**In the fully offline setting**, we have experiments in Tab. 1 in Supplementary Material, comparing ARSQ against\nrepresentative offline RL and imitation learning methods. ARSQ demonstrates superior overall\nperformance across tasks, highlighting its effectiveness on suboptimal offline data.\n\n**In the fully online setting**, we compare ARSQ against online CQN and PPO[2] in Fig. 1 in Supplementary Material.\nOnline ARSQ achieves better sample efficiency than CQN and PPO, underscoring its potential as\na general-purpose reinforcement learning algorithm.\nHowever, we observe that although online ARSQ eventually matches the converged performance of the standard ARSQ, it\nrequires approximately $4\\times$ more environment interactions\nto reach comparable performance, highlighting the importance of using offline datasets to enhance sample efficiency.\n\n[2] Schulman et al. (2017). Proximal policy optimization algorithms. arXiv:1707.06347.\n\n## 4. Training Curves for D4RL Main Results (Fig. 4)\n\nThank you for pointing out the absence of training curves. We have added detailed training curves for each task in Fig.\n2 in Supplementary Material, clearly showing the number of online environment steps (approximately 25k–50k steps until\nconvergence).\n\nWe will also include these training curves in the revised manuscript to enhance clarity.\n\n## 5. Tasks Evaluated in Figure 7\n\nThank you for pointing out the lack of clarity regarding the tasks evaluated.\nFigure 7 reports results on two D4RL tasks (*hopper-medium-expert* and *hopper-medium-replay*), and one RLBench task\n(*Open Oven*).\n\nWe will clarify this in the revised manuscript.\n\nWe sincerely appreciate your constructive feedback, which has significantly improved the quality and clarity of our\npaper. We hope these clarifications and additional results address your concerns, and we are happy to further discuss\nany remaining questions.", "claims_and_evidence": "The main claim put forward by this work is in the empirical effectiveness of the proposed method. This claim is indeed supported by convincing evidence.", "methods_and_evaluation_criteria": "The evaluation criteria are consistent with the scope of this paper. However, the offline-to-online setup is rather restrictive. Can this ARSQ be applied, e.g., to fully online or fully offline settings?", "theoretical_claims": "Yes, the theoretical results are overall correct, although imprecise.\n\n- I am skeptical about Equation 3 and 10: why can we assume that the exponentiated advantages are already normalized? Should the $=$ be replaced by a $\\propto$? The rest of the derivations would still hold, as far as I can tell.\n\n- Moreover, I find the theorem to be unnecessarily complicated. If I understand correctly, Eq. 12 is equivalent to assuming that exponentiated advantages represent a valid probability distribution. In this case, it's important to state this more clearly, and remark that this assumption is almost always violated.", "experimental_designs_or_analyses": "The analyses are overall sounds, with minor issues with clarity as described in my questions.", "relation_to_broader_scientific_literature": "The proposed method proposes a solution to the important tradeoff between expressivity and tractability in continuous control with discretized actions. The autoregressive model proposed appears to solve the issues that arise when treating action dimensions independently, which is the standard approach in the literature, as far as I know.", "essential_references_not_discussed": "I am not aware of missing important references. My knowledge of related literature is however limited.", "other_strengths_and_weaknesses": "Strengths:\n- The presentation is overall good, except for the aforementioned imprecisions in Section 4\n- Empirical improvements appear to be significant.\n\nWeaknesses:\n- The theory is not precise, as described above (Eq. 3 and 10, Theorem 4.3).\n- The experimental evaluation does not mention important information (see question below).", "other_comments_or_suggestions": "None. The paper is well written, and I could not find any language issues nor typos.", "questions_for_authors": "1. Can you comment on my questions on Theoretical claims? Clarifications in this regard would help me confirm my score.\n2. Why was the offline-to-online setup chosen? What prevents an evaluation in fully online or offline settings?\n3. How many online steps are performed in Figure 4? Why are bar plots reported instead of training curves (as done in Figure 6)?\n4. What tasks are evaluated in Figure 7? Is it an aggregate score over tasks?\n\n## Update after rebuttal\nConsidering all of my comments were addressed, I have updated my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of reinforcement learning for continuous control with action discretization, specifically the explosion in dimensionality that occurs if the continuous action space is jointly binned in a grid-like manner across all dimensions. One solution that prior work has explored is the use of separate action binning across each dimension, which avoids the aforementioned explosion but in turn makes an assumption about action dimension independence. This work proposes a coarse-to-fine auto-regressive framework that overcomes both of these two issues. 

Reviewers agree that the claims in the paper are supported by empirical evidence  and the proposed method is well-motivated. There were initial concerns about a lack of baselines and number of seeds used, but the authors were able to produce additional results during the rebuttal period that assuaged concerns. There were also concerns about assumptions for the theoretical contribution which were later clarified. If these changes are incorporated into the camera ready, this paper is a strong contribution to ICML.
Decision: Accept (poster)