Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning
Authors: Jeonghye Kim
Suyoung Lee
Woojun Kim
Youngchul Sung
Venue: NeurIPS 2024 poster (2024)
Keywords: offline reinforcement learning
return conditioned supervised learning
Q function
dynamic programming
Primary area: reinforcement_learning

Abstract: Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce $Q$-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of $Q$-functions. By analyzing $Q$-function over-generalization, which impairs stable stitching, QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value-based methods, consistently achieving or exceeding the highest trajectory returns across diverse offline RL benchmarks. QCS represents a breakthrough in offline RL, pushing the limits of what can be achieved and fostering further innovations.

Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learning
Jeonghye Kim', Suyoung Lee', Woojun Kim², Youngchul Sung'
KAIST *Carnegie Mellon University
Abstract
Offline reinforcement learning (RL) has progressed with ceturn-ondiitoned su- pervised learning (RCSL), but its lack stitching ability remains a limitation. We introduce Q-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of Q-functions. By analyzing Q-function over-geneee nee-genneraizztion, which impairs stable stitching, QCS adaptively integrates Q-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value- based methods, consistently achieving or exceeding the maximum trajectory re- turns across diverse offline RL benchmarks. The project page is available
Introduction
Offline reinforcement learning (RL) is vital framework for acquiring decision-making skills from fixed datasets, particularly when online in- teractions are impractical. This is especially rel- evant in fields such as robotics, autonomous driv- ing, and healthcare, where the costs and risks of real-time experimentation significant.
QCS Policy Training
RCSL
QUAM
Optimnal
Loss Function
Q-Aid
Offline Dataset
Sub Sub-optimal
Loss Function
promising approach offline RL return- conditioned supervised learning (RCSL) [T0] [2] 20 By framing offline RL as sequence model- ing tasks, RCSL allows agent to leverage past experiences and condition on the target outcome, facilitating the generation of future actions that are likely to achieve desired outcomes. This method builds recent advancements in super- vised learning (SL) [4 II] 30), and thus benefits from the inherent stability and scalability of SL. However, RCSL is significantly limited by its lack of 'stitching ability', the ability to combine
Figure 1: Conceptual idea of QCS: Follow RCSL when learning from optimal trajectories where it predicts actions confidently but the Q-function may stitch incorrectly. Conversely, refer the Q- function when learning from sub-optimal trajecto- ries where RCSL is less certain but the Q-function is likely accurate.
suboptimal trajectory segments to form better overall trajectories [13 27
133
16
7 58
As a result,
its effectiveness is restricted to the best trajectories within the dataset.
Conversely, the Q-function possesses the ability to stitch together multiple sub-optimal trajecto- ries, dissecting and reassembling them into an optimal trajectory through dynamic programming, Therefore, to address the weakness of RCSL, prior works have attempted to enhance stitching ability condition-
through the Q-function [53] 16]. However, these prior works employ the Q-function ing factor for RCSL and do not fully leverage the Q-function's 's stitching ability, resulting in either negligible performance improvements or even reduced performance. The primary challenge lies
"Corresoondence to Youngchul Sung.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RCSL Methods (DREG))
Value-Sa Methods (o3--B/QQxxaaaa
Combined RCSL-Vake Methods SDTTSOTKGOTAACCPO
QCS (Ours)
B.: Max Trajectory Return
methods each gr
MujoCo Medium
MujoCa Modium-Rappe Rapiay
MujoCo Kedium-Expert
AniMaaee Large
Ntrmalized Return
110
70
Figure 2: Mean normalized return in MuJoCo medium, medium ueplllay,, medium expert, and AntMaze large. The scores of RCSL, the value-based methods, and the combined methods represent the maximum mean performances within their respective groups. The full scores are in Section 6.2
in the fact that utilizzng the Q-function through conditioning, without managing the conditions for stable Q-guided stitching, can result in a sub-optimal algorithm.
In this work, we aim to fully synergize the stable and scalable learning framework of RCSL with the stitching ability of the Q-function. To effectively utilize the Q-function, is crucial to identify when it can benefit RCSL and to integrate its assistance, termed Q-aid. Our contributions to achieving the effective utilization of Q-aid RCSL are as follows: (1) We discovered that in-sample Q-learning on an expert dataset, which predominantly consists of optimal actions with similar Q-values within a constrained action range, causes the Q-function to receive improper learning signals and become over-generalized. (2) To prevent errors from this Q-generalization and to incorporate stitching ability within RCSL's stable framework, we propose Q Q-Aided Conditional Supervised Learning (QCS), which adaptively integrates Q-aid into the RCSL's loss function based on trajectory returns.
Despite its simplicity, the effectiveness of QCS empirically demonstrated across offline RL bench- marks, showing significant advancements over existing state-of-the-an (SOTA) methods, including both RCSL and value-based methods. Especially, QCS surpasses the maximal dataset trajectory return across diverse MuJoCo datasets, under varying degrees of sub- catb-optiaalty,, as shown in Fig. 2 Furthermore, QCS significantly outperforms the baseline methods in the challenging AntMaze Large environment, This notable achievement underscores the practical effectiveness of QCS in offline RL.
Preliminaries
We consider Markov Decision Process (MDP) 66, described as a tuple M ASP Po. is the state space, and A the action space. P Sx + A(S) is the transition dynamics, Po A(S) the initial state distribution, r: S x A IR the reward function, and ạ [0, 1) is discount factor. The objective of offline RL is to learn policy x (-1ss) that maximizes the expected cumulative discounted reward, ( -P..e) [Eeo 7"r(,,, (5),a)))) a.)]. using static dataset D (7) J
comprising set of trajectories ((), Each trajectory (() consists of transitions over time horizon T. collected from unknown behavior policy B.
2.1 Value-Based Offfine Reinforcement Learning
Offline RL effectively employs off- policy RL techniques, allowing a divergence between the behavior policy 3 used for data acquisition and the target policy π being optimized [25] 14 23 Off-policy methods primarily utilize the Q-function, which is learned through temporal-differeme (TD) boot- strapping. In actor-critic off-policy approaches, both the Q-function Qo and the policy â updated iteratively, This process can cause a shift in the action distribution, leading â to select actions that significantly deviate from those in the training dataset. These deviations can inadvertently result in overestimation errors, especially for incorrect Q-values through interactions with the environment.
(OOD) actions, as offline RL cannot correct
Unlike actor-critic methods, in-sample learning methods use only in-sample actions learn optimal Q-function, thereby preventing the querying of OOD action Q-values during training [34 32 23 [11. Implicit Q-Learning (IQI.) 23 a representative in-sample learning method. It utilizes expectile regression, defined as L
1(u < 0)(u² where 17 E [0.5, 1), to formulate the

---

asymmetrical loss function for the value network V.. Through this loss, V& can approximate the implicit maximum of the TD target, max, a). Formally, for a parameterized critic Qo(s.a) a) with target critic (5, a), the value loss function is given by
Lv (v) (s,a)-D
2 (Q
Intuitively, this loss function suggests placing more emphasis when Qg is greater than (s). Subse- quently, the critic network Qo updated by treating the learned (s)) as (ex)) Qg(s',a') where Ds'') denotes the in-sample actions for the given state s', i... (s', e') DD:
(o)
E "...,''--.. (r(s,.a) + (V') Qo(s,
We use IQL. pretrain the Q-function used to aid RCSL, found that this method, without conservatism during Q-function training, can provide good stitching ability when well integrated. A comparison with different Q-learning method, CQL [25]. is provided in Appendix H..
2.2 Return-Condittoned Supervised Learning (RCSL)
RCSL is an emerging approach to addressing challenges in offline RL. It focuses on learning the action distribution conditioned on relurn-to-go (RTG), defined as the cumulative sum of future rewards R, Σ.- I' through supervised learning (SL) 10
12 20 Due to the stability of SL,
RCSL is capable of learning decision-makinn by extracting and mimicking useful information from the dataset. In particular, Decision Transformer (DT) [10] applies the Transformer architecture 41 to reframe the RL as a sequence modeling problem. constructs input sequences to the Transformer by using sub-trajectories, each spanning K timesteps and comprising RTGs, states, and
R., st). The model is then trained to predict the action a based on Tr Recently, Kim et al. (20) proposed Decision ConvForme (DC) to simplify the attention module of DT and better model the local dependency in the dataset, yielding performance gains over DT with reduced complexity. These methods have shown effective planning capabilities, but they lack stitching ability, which causes difficulties with datasets that contain many sub-optimal trajectories. This will be discussed in more detail in Section 3.1
actions:
Tr-K+1++ (R,-K+1, St-k Qt--- K-: st-1, 1 04-1
2.3 Neural Tangent Kernel of Q-Function
The Neural Tangent Kernel (NTK) [19 provides insightful analysis of function approximation errors of Q-function, Qo. especially those related to generalization. The NTK, denoted as ko a, 8, a), is defined as the inner product of two gradient vectors, VoQu(5, ā) and VoQa as, i.e., ko a, 8, := VoQa(3, a)' VoQa(s, a). The NTK offers a valuable perspective on the impact of parameter updates in function approximation, particularly in gradient descent scenarios. It essentially measures the degree of influence a parameter update for one state- action pair (s, a) exerts on another pair (5, a). A high value of ke( 5, a.s, a) implies that single update in the Qe for the pair (s,a) could lead to substantial changes for the pair (3, a). We guide the readers to Appendix Difor a deeper understanding of the NTK.
3 When Is Q-Aid Beneficial for RCSL?
When is it beneficial for RCSL to receive assistance from the Q-function, denoted as Qp. and how should this assistance be provided? To explore this, we trained two policies, RCSL policy and max-Q policy that selects the best action according to Qs. on two different quality datascts from D4RL [13 MuJoCo, comparing their performances in Table [ Note that the performance is not directly linked to the policy's 's accuracy across all states; even if the agent accurately predicts actions in several states, errors in some states can lead to path deviations and accumulated errors, resulting in a test-time distribution shift and a lower trajectory return. However, these results can provide insight into when Q-aid might be helpful.
For the RCSL algorithm, we used the Decision Transformer (DT) 10]. To train the max-Q policy, we first trained the Qo using the in-sample Q-learning method outlined in Eqs. [1] and (2). Then, we extracted the max-Q policy to select the action that directly maximizes Qo for each state s by using a 3 3-layer MLP and the loss function Lmax-Q (Ø) E.-D [-Qo (s, max-QQ(s)).
3

---

Table I: Performance comparison of DT and max-Q on expert and medium-rep.l quality datasets in MuloCo.
balfchoetab-o
halfcheetal-m-11
hopper-e
hopper-m-r
walker2d-e
walker2d-m-1
DT max-Q
91.4 1.7
36.6 = 0.8
110.1 = 0.9
82.7 + 7.0
109.2 = 1.5
66.6 + 3.0
-4.1 1.1
52.8 = 0.4
1.8 1.0
92.1 + 2.6
-0.2 a 0.6
91.2 a 1.9
we see that the dataset quality favoring RCSL contrasts with that benefiting the max- Q policy. RCSL tends to perform well by mimicking actions in high-return trajectory datasets 31. However, this method is less effective with datasets predominantly containing suboptimal trajectories, even though RTG conditioning helps predict actions that yield higher returns. On the other hand, the max-Q policy excels with suboptimal datasets but shows notably poor results with optimal datasets. From these observations, motivating question arises: "Why does the simple max-Q) Q policy outperforms RCSL on suboptimal datasets yet fails on optimal datasets? If so, how can we effectively combine the two methods to achieve optimal performance?"
Observing Table
29
3.1 How Can Max-Q Policy Surpass RCSL in Suboptimal Datasets?
We present a toy example demonstrating the limitation of RCSL, as illustrated in Fig. 3 Sup- pose the dataset is composed of two sub-optimal trajectories. At the initial state s², the agent has two options: the 1 action connected to trajectory 2 (the orange trajectory) with an RTG of 5, and the - action connected to trajectory 1 (the pur- ple trajectory) with an RTG of 6. RCSL makes the agent choose the - action with a high RTG and follow the path of trajectory which is not optimal. This example demonstrates that RCSL alone is insufficient for the agent to learn to as- semble the parts of beneficial sub-trajectories.
Goal
Trijectory I (Sccead)
Ingactory 2 ('alund)
Opti Policy
Start
Fall
Figure 3: An example demonstrating the limit of RCSL: The dataset consists of two trajectories, with time limit of T = 3 and a discount factor y = 1. The black dashed arrow represents the optimal policy yielding a maximum return of 7.
In contrast, Q-function can develop stitching ability. Consider the example in Fig. 3 again. We can compute the Q-values for the actions 1 and + at state 81 with dynamic programming: Q(s².t) = 3 + max (o(s",)),Q0
)))= 7, Q(s², (0(ss,-)) -) = 1+ +mx
=6. =
With the Q-values, the agent will select the t action at s and then the + action at s². Consequently, using the Q-function, the agent can select the optimal action that yields the maximum return of 7. Therefore, integrating RCSL with Q-function in situations with abundant sub-optimal trajectories can be beneficial for developing the stitching ability required for optimal decision-making
3.2 Why Does Max-Q Policy Struggle with Optimal Datasets?
Despite the potential advantages of using Q-values, incorporating values from a learned Q- Q-function, Qs. to aid RCSL can introduce errors due inaccuracies in learning. These inaccuracies are particularly significant when Qa is learned from optimal trajectories. Suppose we have an optimal policy n². Optimal trajectories are visit logs containing actions performed by n², yielding the best Q-value for a given state s. Due to the stochasticity of π², multiple similar actions can be sampled from σ², namely aj,az, a?........"".. (s()) x² x²(sss for given state s. In this case, we have (sssa;)) & as(s)a)) and aj a: Vi.j E (1,2....,nn(.. actions. When learning Qu from such limited information, where the values at the narrow action points are almost identical for each given state, it is observed that the learned Qa tends to be over-generalized to the OOD action region. This means that the nearly identical value at the in-sample actions aj aa......"....................... (x(s) is extrapolated to OOD actions, yielding a nearly flat Q-value over the entire action space for each given state, i.e., Qo GOOD) & (s, a²) with Qo a) being a function of s only. This over over- generalization makes Qo noise- sensiisee,, potentially assigning high values incorrect actions and causing state distribution shifts in the test phase, shown in Fig. 7
due to the optimality of these
We present a simple experiment to verify that learning Qo indeed induces over-generalization when trained on optimal trajectories. The experiment consists of an MDP with micddineniional discrete

---

Yecarr delinet
dooset dley essien
Gt au Nedium ditaset
Endrwerent
True
Expert datseer
Quda Expert detaret Regrestonn
Wv Expettanntt t0L
Soare
-00
Adtons
(a)
(b)
(C)
(d)
Figure 4: (a) the view of the environment and true Q calculated through value iteration, (b) training datasets with color representing the true Q for each sample, (c) Qo learned through regression with medium dataset (upper) and an expert dataset (bottom), (d) Qe learned through IQL. with a medium dataset (upper) and an expert dataset (bottom).
states and actions, each divided into 500 bins. This environment simulates a car, where the state indicates the agent's position, ranging from -5 to 5, as illustrated in Fig. (a). The action range is between -1 and I, allowing the agent to move according to the direction and twice the magnitude of the action. The objective is to reach position O, which grants a reward of 100, while larger actions incur penalties given as -30-a². a² Due to its discrete nature, we can compute the true optimal Q-values through value iteration 337. which is shown in the bottom row of Fig. (a).
With this environment, we generated two datasets, medium and expert. The medium dataset consisted of actions varying within the range of +0.5 perturbed from the optimal action determined by the true optimal Q-values, while the expert dataset consisted of actions varying within the range of #0.05 perturbed from the optimal action. (Refer to Fig. 4 (b)) We then adopted a 3-layer MLP as the structure of Qa and performed regression to follow the true Q-value at each sample point (8, a) in the trajectories. Note that in-sample Q-learning can essentially be regarded as regression with the target value obtained from bootstrapping.
The learned Qe with the medium um and expert datasets are shown in Fig. [c).. Indeed, the learned Qs with the expert dataset, containing nearly-optimal actions, shows that the value is flat over the entire action space for each state. This means that the nearly identical value of in-sample expert actions with small spread is projected to the entire action space for each state. In contrast, the learned Qo with the medium um dataset well estimates the true Q Q-function. This is because the medium dataset has diverse actions with diverse values for each state that facilitate the regression process. We additionally present the results from Qo obtained through IQL in Fig. (d), which shows a similar trend to the results from regression.
In-Sample Actions
Qals-a)
In In-Sample Actions
Qa(aa))
Qo(a.0)
In-Sumple Actions Actions
In-Sample Actions
000 Autions
(000
420
800
-13
0.5
0.5
00
85
1.
1.0
1.0
Artton the acten
Aetten Dee andes d
3.0
1.
Inverted Double
Inverted Double
(a) Pendulum Medium
(b) Pendulum Expert (c) Hopper Medium-Replay
(d) Hopper Expert
Figure 5: We present the estimated @o(s,a) for à e A and the normalized NTK ka(s, a, 8, aee Qo(s, 2 across four datasets with a 1D action space for Inverted Dou- ble Pendulum and a 3D action space for Hopper. In these figures, we fix the state and the fixed reference action aref at zero (marked *), and sweep over all actions axes for action dimensions and color to represent Q-values in 3D plots. Additionally, in the NTK plot, we only include the high-NTK regions for values over 0.9. Refer Appendix E for details.
A. For Hopper, we use
5

---

The over over-genera serr-eneralliztion tendency in Qe with optimal trajectories is not limited to the simple experi- ment above but also applies to complex RL tasks. We analyze how Qs(s.-) varies over the action
space in the Gym Inverted Double Pendulum ( and MuJoCo Hopper environments [39 g trained on expert and uuuuulliityyy datasets with IQI.. The details of the analysis are in Appendix F As depicted in the upper row of Fig. 5 (a) and (b), and on the left side of the upper row of (c) and (d),
the expert dataset shows concentrated action distribution, while the medium dataset has a broader spread, as expected. The concentration and similarity of true Q-values of the actions for given state in the expert dataset cause over-generalization in Qa(s,)) :), yielding constant flat values across the entire action space. This is further supported by the results in Appendix E.2 which visualize the weights of the learned Q-function.
For a deeper understanding of the over generalization in Qo, we analyze the gradient similarity, captured as the Neural Tangent Kernel (NTK), between an arbitrary action and the reference action Orr for a given state s. A higher NTK value signifies that the gradient update of Qo(s, (ze) has substantial impact on Qo a). This indicates that even when dre and á are dissimilar or unrelated actions, high NTK value suggests that the Q2 network is misjudging the relationship between these actions and over-generalizing. In Fig. 5 Qa trained with the expert dataset shows uniformly high normalized NTK values across actions, indicating that the gradient at one action equally affects all others. In contrast, Qo trained with the medium dataset shows NTK values that are higher near the reference action and decrease with action distance, reflecting more precise generalization. This analysis reveals that datasets consisting of optimal trajectories exhibit more aggressive generalization, which can negatively impact the accuracy of the learned Q-function in offline RL.
4 Q-Aided Conditional Supervised Learning
According to Section 3 RCSL faces challenges with suboptimal datasets, whereas Qo can effectively serve critic for stitching ability, favoring the use of Q-aid. In contrast, in an optimal dataset, Qe tends to over- over-generalize, leading to inaccuracies of learned Qs. while RCSL excels by mimicking the optimal behavior without requiring external assistance. Recognizing this dynamic, it is crucial to integrate Q-aid into RCSL adaptively. The following subsections explain how to effectively adjust the level of Q-aid and facilitate the integration of the two methods, leading the proposal of Q-Aided Conditional Supervised Learning (QCS).
4.1 Controlling Q-Aid Based on Trajectory Returns
Given the complementary relationship, how can we adjust the degree of Q-aid? Since RCSL's preference for mimicking datasets and the Q-function's over-generalization issue tied to trajectory optimality, we can apply varying degrees of Q-aid based on the trajectory return for each sub- trajectory in RCSL. Therefore, we set the degree of Q-aid, denoted as the QCS weight w( (R(r)) for trajectory T, continuous, monotone-decreasing function of the return of r, R(T), such that
VTI, T2, R(TI) R(r2)= w(R(T))) 2 w(R(T2)),
where continuity is imposed for gradual impact change. Among various choices, we find that simple options such as linear decay are sufficient to produce good results, i.e., w ((((T)) = A- (R" R(T)) with some À > O, where R" represents the optimal return of the task. Practically, R. can be obtained from an expert dataset or from the maximum value in the dataset. For details on how to calculate R', please refer to Appendix F Note that R(T) differs from RTG R. which is the sum of future rewards after timestep é and decreases as timestep goes, thereby failing to represent the trajectory's optimality accurately.
4.2 Integrating Q-Aid into the RCSL Loss Function
Instead of using the Q-unction as the conditioning factor for RCSL. as in previous works, we propose a more explicit approach by integrating Q-assistance into the loss function and dynamically adjusting the degree of assistance based on Section 4.1 As a result, the overall policy loss is given as follows:
K-1
CCCS (6) ET-D
| T (Tea+ 11 X-R R(7)) 20 (8t+z, To (T(Tee+j))
(3)
K
RCSL
QCS weight
Aid

---

where (.-)) denotes the fixed Q-function pretrained with IQL. R(F) is the return of the entire trajectory т containing the sub-trajectory Th:t K-1+ The overall input to the policy time is the sub- trajectory of context length K starting from time t, T==+K-1 (R,, St, az..... 1e- c
Our new loss function enables adaptive learning strategies depending on the trajectory's quality to which the subtrajectory belongs. For optimal trajectories, action selection follows RCSL. On the other hand, for suboptimal trajectories T with R(T) R", the Q-aid term kicks in and its impact increases as R(T) decreases. We describe the details of the QCS weight w(R(r)) and the policy update with the loss function in Appendix 0.2 and our full algorithm's 's pseudocode in Appendix A
4.3 Implementation
For implementing Tg. general RCSL policy can be used. When K = 1, meaning only the current time step considered to estimate the action, we use an MLP network. When K 2 2,
Base Architecture.
we use a history-based policy network, such as DT [10] or DC 20)
Conditioning. We consider two conditioning approaches as proposed by RvS 12 one for tasks maximizing returns and the other for tasks aiming at reaching specific goals. For return-maxinizing tasks, we employ RTG conditioning, and our algorithm is named QCS-R. For goal-reaching tasks, we additionally use subgoal conditioning, and our algorithm is named QCS-G. For subgoal selection, we randomly select state that the agent will visit in the future. The ablations on conditioning are in Appendix H.2
5 Related Work
Prompting RCSL with Dynamic Programming. Recent studies have recognized the limitations of RCSL in stitching abilities (27 7 (58. Our work contributes to the ongoing efforts to imbue RCSL with this capability. Notably, Q-learning Decision Transformer (QDT) [53] and Advantage Conditioned Transformer (ACT) [16] have proposed integrating dynamic programming into RCSL by modifying the RTG prompt to Q-value advantage prompt. Our approach, QCS, parallels these efforts by leveraging dynamic programming for action guidance and trajectory stitching. However, unlike these methods, which implicitly incorporate dynamic programming through conditioning, QCS explicitly augments its loss function with the learned Q-function.
Incorporating RCSL with Stitching Ability. In a distinct vein, recently proposed Critic-Guided Decision Transformer (CGDT) 43 identifies the gap between target RTG and expected returns of actions as key to RCSL's limited stitching. To mitigate this, it adjusts DT's output with the critic network's 's Monte-Carla return predictions and target RTG, In contrast, QCS uses Q-values Icarned through dynamic programming to guide actions, enhancing stitching ability explicitly. Another approach, the Elastic Decision Transformer (EDT) (49), recommends variable context lengths during inference, using longer contexts for optimal trajectories and shorter ones for sub-optimal trajectories to identify optimal paths better. QCS similarly adapts based on trajectory optimality but differentiates itself by modifying its learning approach during training, leveraging the complementary strengths of the Q-function and RCSL.
Furthermore, POR [50] integrates imitation learning techniques with stitching ability by generating high-value states using additional networks and value functions. These states are then used as conditions for predicting actions. Unlike QCS, which focuses action stitching, POR emphasizes state stitching, allowing agents to choose actions that lead to high-value states, albeit with the need for additional networks. By concentrating on action stitching, QCS can avoid the computational demands associated with high-dimensional state prediction.
State-Adaptive Balance Coefficient Regarding the sub- crajecttry-adappiie weight used in QCS, FamO20 42 employs state-adaptive weight coefficients to balance policy improvement and con- straints in the offline-to-onlnee RL framework. Although Fam020 is an offline-to-on method that incorporates additional online samples, we provide performance comparison with this work in Appendix G.2 to further demonstrate the effectiveness of QCS.
7

---

Experiments
In the experiment section, we conduct various experiments across different RL benchmarks answer the following questions:
How well does QCS perform decision- makiin compared to prior SOTA methods across datasets of varying quality and tasks with diverse characteristics, especially those requiring stitching ability?
To what extent does the dynamic nature of QCS weights, informed by trajectory return, contribute to effective decision- making, and how robust are these dynamic weights to hyperparameters?
Can QCS effectively acquire stitching ability while preventing test-time distribution shift?
6.1 Experimental Setup
To address a range of questions, we conduct comprehensive benchmarking against 12 representative baselines that are state-of-the-art in each category. For the value-based
Baseline Methods.
category, we assess 4 methods: TD3+BC 14, IQL [23], and CQL [25], SQL [52]. For RCSL,
assess 3 methods: DT [10]. DC 200 RvS 12] Additionally, we evaluate 5 advanced RCSL methods
proposed integrate stitching capabilities: QDT [53), EDT [49]. CGDT 43), ACT [16], and POR
[50]. For more details on the setup and the baselines, refer to Appendix B
Benchmarks. We evaluated QCS against various baselines using datasets with diverse characteristics,
including tasks focused on return maximization or goal-reaching, and those with dense or sparse rewards and varying sub-optimality levels.
AntMaze Umaze
AntMaze
AntMaze
Adroit Pen
(a) Halfcheetah
(b) Hopper
(c) Walker2d
(d)
(c) Medium
Large
Figure 6: Views of tasks used in our experiments.
Our primary focus was on the D4RL 13 MuJoCo, AntMaze, and Adroit domains. The MuloCo domain [39. 8 features several continuous locomotion tasks with dense rewards. We conducted experiments in three environments: Halfcheetah, Hopper, and Walker2d, utilizing three distinct v2 datasets-medium medi repllayy, and medium- pediumeepeet-eeace representing different levels of data quality. AntMaze is a domain featuring goal-reaching environments with sparse rewards, encompassing variously sized and shaped maps. is ideal testing bed for evaluating agent's capability to stitch trajectories and perform long-range planning. We conduct experiments using six datasets: umaze, mmaze-diverse, large-divers where umaze, medium, um, and large indicate map sizes, and play and diverse refer to data collection strategies. The Adroit domain evaluate the effectiveness of algorithms in high- dighenimensional robotic manipulation tasks. In our experiments, we utilize the human and cloned datasets for the pen task.
samummll.yyaammmmammmmmmmamamm medium-diverss
large-playy and
355 comprises various tasks designed to
Performance results for the MuJoCo and AntMaze domains are presented in Table 2 and Table while results for the Adroit domain included in Appendix G
Hyperparameters and Backbone Architecture. We adopted two sets of hyperparameters per domain to determine the gradient of the monotonic decreasing function w w(R(T)). The detailed hyperparameters we used are provided in Appendix and the impact of is detailed Appendix 6.3 Additionally, implemented QCS based on DT. DC, and simple MLP, and compared the performance of each. Detailed results for each architectural choice are provided Appendix H.2 We observed that the DC-based approach performs best, although the performance gap is minor.
Evaluation Metric. In all evaluations of QCS, we assess the expert-normalized returns [33 of 10 episodes at each evaluation checkpoint (every 10² gradient steps). Subsequently, we compute the running average these normalized returns over ten consecutive checkpoints. We report the mean and standard deviations of the final scores across five random seeds.

---

6.2 Overall Performance
Table 2: Performance of QCS and baselines the MuJoCo domain. The dataset names are abbreviated as follows: medium to 'm', medium- medium-reples replaay to 'm-r', mediim-eeepeeeeeeeeeeee expert to 'm-e'. The boldface numbers denote the maximum score or comparable one among the algorithms.
Value-Based Method
Combined Mothod
Ours
Dataset
TD3+RC
IQI.
CQI.
SQI.
RvS-R
QDT
CGDT
ACT
QCS-R
halfeheetah-m
48.3
47.4
44.0
48.3
42.6
43.0
41.6
42.3
42.5
43.0
49.1
48.8
59.0 0.4
hopper-m
59.3
66.3
58.5
75.5
67.6
92.5
60.2
66.5
63.5
96,9
67.8
78.6
96.4 3.7
walkker2d m
83.7
78.3
72.5
84.2
74.0
79.2
71.7
67.1
72.8
79.1
80.9
81.1
88.2 1.1
hulfcheetah-me m-r
44.6
44.2
45.5
44.8
36.6
41.3
38.0
35.6
37.8
40.4
43.0
43.5
54.1 0.8
hopper-
60.9
94.7
95.0
99.7
82.7
94.2
73.5
52.1
89.0
93.4
98.4
98.9
100.4 1.1
walker2d-m-r
77.2
81.2
66.6
76.6
58.2
78.1
76.6
halfcheetal----e
90.7
86.7
91.6
94.0
86.8
93.0
92.2
93.6
96.1
94.7
93.3 1.8
hopper-m-e
98.0
91.5
105.4
111.8
107.6
110.4
101.7
107.6
111.5
90.0
110.2
walker2d-m-a
110.1
109.6
108.8
110.0
108.1
109.6
106.0
109.3
113.3
109.1
116.6 :: 2,4
average
75.3
77.0
77.6
83.1
74.7
82.2
71.7
82.4
79.6
80.1
90.3
Table 3: Performance of QCS and baselines in the AntMaze domain. The dataset names are abbrevi- ated as follows: umaze to 'u', -maze-diverse to 'u-d', medium-play to 'm-p', medium-diveen to 'm-d', Jarge-play to 'l-p', and large-divers to 'I-d'. The boldface numbers denote the maximum score or comparable one among the algorithms.
Value-Based Method
RCSL
Combined POR
Ours
Dataset
TD3+BC
IQL
CQL
SQL
DI
DC
RvS-R
RvS-G
QCS-R
QCS-G
antmaze-u
78.6
87.5
74.0
92.2
65.6
85.0
64.4
65.4
90.6
92,7 + 3.9
92,5 4.6
untmaze-u-d
71.4
62.2
84.0
74.0
51.2
78.5
70.1
60.9
71.3
72.3 12.4
82.5 8.2
antmaze-m-p
10.6
71.2
61.2
80.2
4.3
33.2
4.5
58.1
84.6
81.6 6.9
84.8 11.5
untmaze-m-d
3,0
70.0
53.7
79.1
1.2
27.5
7.7
67.3
79.2
79.5 5.8
75.2 11,9
untmaze 1p
0.2
39.6
15.8
53.2
0.0
4.8
3.5
32.4
58.0
68.7 7.8
70.0 9.6
antmazo-I-d
0.5
73.4
7
77.3 11.2
averago
27.3
63.0
50.6
71.8
20.5
40.2
25.6
53.5
76.2
77.6
80.4
As shown in Table and Table 3, QCS significantly outperforms prior value-based methods, RCSL, and combined methods across the datasets. Specifically, QCS outperforms both IQL and DC, upon which it is based, across all datasets, unlike the contradictory results between RCSL and the max-Q policy shown in Table [ This empirically confirms that QCS successfully combines the strengths of both RCSL and the QQffnnctin.. A particularly remarkable achievement of QCS is its ability to substantially improve efficiency goal-reaching tasks, AntMaze, especially in Large environments, where prior RCSL methods exhibited notably low performance. This enhancement is largely attributed the stitching ability introduced by the Q-aid of QCS. These results underscore QCS's robustness and superiority in wide array of offline RL contexts. The training curves for Tables and 3 are shown in Appendix!! demonstrating stable learning curves across all datasets.
6.3 Ablation Studies
To further analyze how each design element influences performance, we conducted additional experiments. More ablation studies detailed in Appendix H including the use of Q-function trained by CQL and the impact of base architecture and conditioning.
The Importance of Weights Relative to Trajectory Return.
To assess the impact dynamically setting the QCS weight w( ((((T)) based on trajectory return, we compare our ap- proach with constant QCS weight, w ((()) c. We test five constant weights e é 21,2.5, 5, 7.5, 10} and report the maximum score among these values in Table 4 The QCS method with the dynamic weight based on trajectory return outperforms the highest scores obtained with various con- stant weight settings across datasets, as shown in Table
Table 4: Comparison of constant QCS weight and the dynamic weight.
Dalasel
Constant Weight
Dynamic
Weight
mujoco-m
74.7
81.2
mujoco-m-r
75.4
82.9
mujoco-m-e
104.2
106.7

---

This demonstrates that our dynamic weight control, grounded in trajectory return, is more effective and robust in integrating Q-aids.
Impact of the QCS weight x. We examined the effect À by varying it from 0.2 to 1.5. As shown in Table except for the walker2d-medium, with changing À either matched or surpassed the performance of existing value-based methods and RCSL's representative methods, including IQL, CQL, DT, DC, and RvS. This demonstrates QCS's relative robustness regarding hyperparameters. For walker2d-medium, we found that performance begins decrease when À exceeds the initial setting of 0.5. While increasing the gradient steps from 500K to 1M improves performance at À further increasing À to 1.5 leads to greater instability.
found that the smallest values achieved
Table Performance of QCS in the Mujoco domain with varying values. The boldface numbers denote the maximum score comparable one.
A - 0.2
A 0.5
A
A 1.5
hulfchestah medium
53.7 0.4
57.7
59.0 0.4
59.0 0.2
hopper-medium walker2d medium
75.5+7.11 (500K) 87.6=3.9
halfchectah-medde mediuu--pada
0.8
52.8 a
=
54.2 #:
hopper-mediun-rephe walker2d medium replay
98.5 a
100.4 1.1
99.4 =
100.5 0.7
83.3 5.7
93.2 + 2.5
94.1 = 2.0
92.3 + 3.7
Test Time State Distribution Shift.
To validate whether QCS effectively acquires stitching ability while preventing shift in the test-time state distribution, as discussed in Sec-
tion 3.2 we present Fig. 7 This figure compares the state distributions explored by RCSL, max-
Q, and QCS policies during evaluation. RCSL and max-Q, representing QCS's extremes, were trained using specific loss configurations: RCSI. loss as QCS loss in Eq. with À = 0 and max-Q loss as QCS loss without the RCSL term, i.c.. selecting actions as argmax (s. Fig. illlustrates RCSL's adherence to dataset states, contrasting with the notable state distribution shift of the max-Q policy. QCS inherits RCSL's stability but surpasses its performance, indicating an effective blend of transition recombination without straying excessively from the state distribution.
(a) RCSL
(b) max- Q
(c) QCS (Ours)
Figure 7: t-SNE [40 analysis of states vis- ited by policies trained with RCSL, max-Q (argmax (s. and QCS losses dur- ing evaluation, alongside dataset's states in Malker2d-medine
7 Conclusion
In conclusion, QCS effectively combines the stability of RCSL with the stitching ability of the Q-function. Anchored by thorough observation of Q-function generalization error, QCS adeptly modulates the extent Q-assistance. This strategic fusion enables QCS to exceed the performance of existing SOTA methods in both efficacy and stability, particularly in complex offline RL benchmarks encompassing a wide range of optimality.
In addressing our initial motivating question on integrating RCSL and Q-function, QCS opens up promising future research directions. While we have established a correlation between trajectory return and the mixing weight, have considered simple linear weights to control the level of Q-aid. is also plausible that the mixing weight might be influenced by other dataset characteristics, such the dimensions of the state and actions. We believe QCS will stand motivating work, inspiring new advancements in the field.
10

---

Acknowledgments
This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.202--0-00469, Develop- ment of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones, 50%) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2021R1 A2C200000 Information Theory-Based Reinforcement Learning for Generalized Environments, 50%).
References
[1] J. Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint
2019.
[2] A. F. Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375
2018.
[3] G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances Neural Information Processing Systems, 34:7436- 7447, 2021.
[4] Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arXiv: 1607.66400 2016.
[5] J. Ball, L. Smith, Kostrikov, and Levine, Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
[6] R. Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pages 679-684 1957.
[7] D. Brandfonbrener. A. Bietti, J. Buckman. R. Laroche, and J. Bruna. When does return- conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542-1553. 2022.
[8] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 010.6001566 0444000000.1.....000 2016.
[9] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
[10] L. Chen, K. Lu, A. Rajeswaran, Lee, Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances Neural Information Processing Systems, 34:1 15097, 2021.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[12] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What essential for offline RL. via supervised learning? In International Conference on Learning Representations, 2022. URL http://pprrrreeeeeeeeeeeerroraa/
[13] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning, arXiv preprint arXiv:2004.072 079 2020.
[14] S. Fujimoto and S. S. Gu. A minimalist approuch to offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-20144 2021.
[15] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference Machine Learning, pages 2052-2062. PMLR, 2019.

---

[16] C. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, and Y. Yu. Act: Empowering decision transformer with dynamic programming via advantage conditioning. arXiv preprint wrX1::230.055/55 2023.
[17] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning, pages 1037-1037. PMLR, 2020.
[18] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR. 2018.
[19] A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
[20] J. Kim, S. Lee, W. Kim, and Y. Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. In International Conference on Learning Representations, 2024.
[21] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[22] I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
[23] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning, In International Conference on Learning Representations, 2021.
[24] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019,
[25] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
[26] A. Kumar, R. Agarwal, T. Ma, A. Courville, G. Tucker, and S. Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. International Conference on Learning Representations, 2021.
[27] A. Kumar. J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05611 2022.
[28] J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems, 32, 2019.
[29] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXivv2005.01663, 2020.
[30] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[31] I. Mediratta, Q You, M. Jiang, and R. Raileanu. A study of generalization in offline reinforce- ment learning. NeurlPS 2023 Workshop on Generalization in Planning, 2023.
[32] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint 2xXiv:000600935/ 2020.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, hegh-peeformance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[34] X. B. Peng, A. Kumar, G. Zhang and S. Levine. weiigte regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv:1910.00117 2019.
12

---

[35] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems XIV, 2018.
[36] H. Singh, N. Misra, V. Hnizdo, A. Fedorowiez, and Demchuk. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 2003.
[37] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[38] R. S. Sutton, A. G. Barto, et al. Introduction to reinforcement learning, volume 135. MIT press Cambridge, 1998.
[39] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[40] I.. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(11), 2008.
[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[42] S. Wang, Q. Yang. J. Gao. M. Lin, H. Chen, L. Wu, N. Jia, S. Song, and Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[43] Y. Wang. C. Yang. Y. Wen, Y. Liu, and Y. Qiao. Critic-guided decision transformer for offline reinforcement learning. arXiv preprint arXiv:2312.137/6 2023.
[44] Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[45] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[46] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
[47] Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:191..11366 2019.
[48] Y. Wu, S. Zhai, N. Srivastava, J. Susskind, J. Zhang, R. Salakhutdinov, and H. Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.081400 2021.
[49] Y.H. Wu, X. Wang, and M. Hamaya. Elastic decision transformer. In Advances in Neural Information Processing Systems, 2023.
[50] H. Xu, L. Jiang, L. Jianxiong, and X. Zhan. A policy-guided imitation approach for offline reinforeement learning. Advances in Neural Information Processing Systems, 35:4085 2022.
[51] H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2022.
[52] H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization, arXiv preprint 2r3vv:2303. 15810, 2023.
13

---

[53] T. Yamagata, A. Khalil, and R. Gantos-Rodriguer Qleaaniig decision transformer. Leveraging dynamic programming for conditional sequence modelling offline rl. In International Conference on Machine Learning, pages 38989-39007. PMLR, 2023.
[54] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14129-14142, 2020.
[55] T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems, 34:28954-28867, 2021.
[56] Y. Yue, R. Lu, Kang, Song, and G. Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. In Advances in Neural Information Processing Systems, 2023.
[57] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In International Conference on Machine Learning, pages 27042-27059, PMLR, 2022.
[58] Z. Zhou, C. Zhu, R. Zhou, Q Cui, A. Gupta, and S. Du. Free from bellman completeness: Trajectory stitching via model-based Conference on Learning Representations, 2024.
supervised learning. In International
14

---

Appendices
A
Pseudocode
The QCS algorithm first learns the Q-function using a dynamic programming and then trains the policy based on the aid of the learned Q-function. Detailed pseudocode can be found in Algorithm In this work, we utilized IQL [23] as the Q training algorithm, but other Q training algorithms can be employed. A comparison with using Q-function trained by CQL 25 be found in Appendix
Algorithm 1 IQL-aided QCS
Hyperparameters: Total critic gradient steps M, critie learning rate Olerities target update rate X total policy gradient steps N, policy learning rate (ppllcyy context length K Initialize parameters: 6, sb and Φ
IQL Pretraining for m = to M do
v
O'eiitic V (wb using Eq.
0 - 0 - Ctaritic V L (f) using Eq.
0 x0 (1 x)0
end for
// QCS Policy Training for = N do
Sample trajectory ~~ D Sample sub-trajectory Tht+K-1 r with random initial timestep
② + Opolicy V OCS ( (Ø) using Eq.
end
B Baseline Details
We evaluated the performance of QCS against twelve different baseline methods. This group consists of four value-based methods: TD3+BC [14), IQL [23. CQL (25) and SQL [52]: three RCSL algorithms: DT (10), DC [20]. and RvS [12;; and five combined methods that signify progress in RCSL by integrating stitching abilities: QDT 53 EDT [49], and CGDT 43 ACT [16], and POR [50] The performance for these baselines was sourced from their respective original publications, with two exceptions. For CQL [25]. the performance data in the original paper was based on the MuJoCo v0 environment, which differs from the v2 version used in our study. Therefore, for CQL, we referenced the performance score reported in 23 to ensure a consistent and fair comparison across all methods.
In addition, for antmaze-mat -mediu and antmaze-large, since there were no reported DT [10 and DC [20] scores, conducted evaluations using the official codebase. When training on antmaze-mediue and large, we used 512 as the embedding dimension in the default hyperparame- ter setting. We found that removing the positional embedding slightly improved performance, as also discussed in Zheng et al. (57). so we trained without it. For the target RTG, we used values of 1 and 100 and reported the higher score obtained from the two values.
C Dataset Return Distributions
To gain deeper understanding of the scenarios in which offline RL is applied and the necessity of learning good policies, we plotted the trajectory return distributions for three different datasets in each of the three MuJoCo environments in Fig. 8 For these return distribution histograms, we set the number of bins 50. The "Count" label denotes the number of trajectories corresponding to each normalized return.
15

---

QCS
Trajectory
Halfcheetah Medium
Hopper Medium
Walker2d Medium
250
150
200
Count
Normalzzed
Halfcheetah Medium--Replay
Hopper Medium-Replan Repply
Walker2d Medium-Replay
Count
Halfcheetah Medium-Expert
Hopper Sedilum-Exper
Walker2d Medium-Expen
500
Count
603
200
Normalized
Return
Normalized
Figure 8: Distribution of trajectory returns in the MuJoCo datasets, including the dataset's maximum trajectory return and the QCS score.
As shown in Fig. S the medium-rephe datasets encompass wide varieties of returns. Additionally, the medium-exper's dataset, a combination of the medium and expert datasets, exhibits two peaks, indicating division in the range of returns. This observation reveals that each dataset exhibits distinct distribution pattern of returns. In this graph, alongside the return distribution, the dataset's maximum trajectory return and the score of our QCS method are also presented. QCS is observed to achieve results that are close to or surpass the maximum return. This is particularly notable in datasets with diverse return distribution characteristics, such the medi um-replay ay dataset, where the distribution of low return trajectories is prevalent.
D Brief Derivation of the Neural Tangent Kernel
To understand the influence of parameter updates in function approximation across different state-
action pairs, the Neural Tangent Kernel (NTK) emerges as a crucial tool [19 28 26 56). For comprehensive insights, we direct readers to Achiam et al. [1. while here we distill the essential concepts. The NTK framework becomes particularly relevant in the context of deep Q-learning
where the parameterized Q Q-function, denoted as Q9, is updated as follows (37:-
8'
E [6g a) (e).
s,a-p
where a is the learning rate, p is the distribution transitions in the dataset, and 50 = r(s, a) + y max, Qa(s', d') (s,s, a) is the temporal difference (TD) error. On the other hand, the first-order Taylor expansion around 0 at an out-of-sample pair (8, a) yields
(5, Q0 a) Vo (0'-
(5)
Substituting
into
we have
(s. @0(5.a) + +a E [ a, 0, (s, 8,0-p
(6)
where ko,5,ā, referred to the NTK, is defined the inner product between two gradient vectors VoQo a) and voQo(s,a). i.e. a,( =
16

---

(6 together with 7 explains how the parameter update with function approximation for sample pair a) affects the Q-value change for another sample pair (5, a). When the NTK (n s, a) is large, the TD-error 50( s, a) has more pronounced impact on the update difference Qi (5, Qo 8, a).
Thus, the single update based on the TD-error at sample pair (8, a) can induce a significant change in the Q-function for another pair @s.@).
E Details and Extended Analysis of Q-Function and NTK
In Section 3.2
we conduct an NTK analysis of the Q-function trained with IQL in the Inverted Double Pendulum and Hopper environments, which have state dimensions of 2 and 11, and action dimensions of I and 3, respectively. This section details the analysis methods, provides extended results, and offers further clarity by presenting the action distributions for the datasets of each environment.
E.1 Analysis Methods
Inverted Double Pendulum. We chose the Inverted Double Pendulum for analysis of Q- values and NTK due its one-dimensional action space. For training the Q-function, as no prior open-source offline dataset existed for this environment, we first created one. The dataset was generated by training an online Soft Actor-Critic (I8), using an implementation in RLkit, available at https://gi thub.
We created two datasets: expert and medium. The expert dataset consists of 10 samples generated by an optimal policy, while the medium dataset includes 109 samples from a medium policy, whose performance is approximately one-third of the optimal policy. Given the continuous nature of the state and action spaces in the Inverted Double Pendulum, which complicates analysis, we initially quantized both spaces. For state quantization, we set the range from minimum of -5 to a maximum of 10 (the minimum and maximum values across all dimensions in both datasets) and divided each dimension into 80 equal segments. For action quantization, the range was set from -1 1, divided into 500 equal segments.
When plotting the Q-values, calculated the Q-value for each quantized state across all quantized actions. Fig. shows the results for the state chosen in each dataset, based on the highest count of in-sample actions. In the NTK analysis, computed the following equation for the reference action
and the remaining quantized actions with index E .....,,000, where 01 -1, 4500 1, and Oset = 0.
MuJoCo Hopper. For Hopper environment, we use open-source D4RL [13] copper-exper's and
datasets. In the case of Hopper, similar to what was done in the Inverted Double Pendulum environment,
quantized the continuous state and action space for analysis. More specifically, for the state space, we divided the values of each dimension into 100 equal segments, ranging from -10 to 10. As for the action space, dimension into 50 equal segments, ranging from -1 to 1. In the case of Hopper, with its 3D action dimension, visualizing it similarly the 1D action dimension in the Inverted Double Pendulum posed a challenge. Consequently, in the 3D plots, we assigned each axis to one of the action dimensions and utilized color indicate the Q-value, as shown in Fig. 5 Additionally, in NTK analysis, representing the relationships between the reference action and all quantized actions within a single graph is challenging. We marked high-NTK regions in gray, where the normalized NTK values are greater than 0.9.
divided the values of each
E.2 Q-Network Weights Visualization
Figure g illustrating the neural network's learned weights for actions in two distinct datasets. provides a compelling visual representation of the verrgeneraaliztton results presented in Figure
S Specifically, the figure displays the first layer's weight matrix W1 from a two-layer MLP Q- network trained on the Inverted Double Pendulum using Implicit L-Leannig (IQL). This network defined as Q(s, a) = W2ReLU ( (s,,s, a) + b1) + by, where W, is a key focus due its direct interaction with the concatenated state and action inputs. The dimensions of the weight matrix W, are 32 x (dim(S) + dim(AA)) where dSm(S)= = 11 and dim(A) = represent the dimensions of the state and action spaces, respectively. The figure contrasts the learned weights in datasets with different action spreads and the diversity of the Q-values: a narrow spread (expert dataset) and wider spread (medium dataset).
17

---

.5
Wal
W;
Wih--
H159en
State Dim Action Dim
Action Dim
State Dim Action Dim
Action Dim
Double Pendulum Expert
Inverted Double Pendulum Medium
(a)
(b)
Figure 9: The first layer's weight matrix W1 of a two-layer MLP Q-network trained on the Inverted Double Pendulum using IQL. The matrix dimensions 32 (dim(S) + dim(.A).. For the expert dataset, the action-related elements of W are learned as zero, indicating the network's training to not differentiate actions across all states.
For the expert dataset (Fig. aa)), the action-related elements of W (the last column elements) Icarned as zero. This intriguing result indicates that the network, during its training, learns not to differentiate between actions across all states, leading to uniformly flat Q-values for all actions. Such behavior is characteristic of datasets with narrower action spread, where the actions are more clustered and coherent. The network's 's tendency to not distinguish between different actions such dataset is direct consequence of the limited diversity, requiring less differentiation in the action representation.
In contrast, for the medium dataset (Fig. Ib),, which represents wider action spread, the action- related elements of W₁ show variation. This variation signifies that the network has learned to differentiate between actions to a greater extent, necessity in dataser where actions are more diverse and dispersed. The network's 's capacity to distinguish between various actions and assign different levels of importance each reflects the need for more nuanced understanding of the action space datasets with a wider spread.
This visual evidence from the learned weights substantiates our understanding of how neural networks adapt their learning based on diversity in the action space the dataset. In datasets with narrower action spreads with similar Q-values, the network leamn a more uniform approach towards different actions, while in those with wider spreads with diverse Q-values, adopts a more differentiated and discerning strategy. This adaptive learning aligns with the principles of regression demonstrating the network's 's response to the diversity and distribution of actions in the training data.
E.3 Extended NTK Visualization
s.
-1.0
-1.0
6.0
0.0
1.0
0.0
1.0
-.5
D5
-25
-3.0
055
-10
60
1.0
0.3
10
05
0.0
63
200
03
0.0
03
1.0
05
03 10
0.5
03
0.0
055
1.0
00 100
Pendulum Medium
Double Pendulum Expert
Figure 10: Normalized NTKs ka a, s, V for three different fixed states from each dataset and for all reference action a A and contrastive action E A. The figures below illustrate the cross sections of figures above d = 0.0.
18

---

we visualize the NTK for fixed state and reference action aeef zero the Inverted Double Pendulum environment. Here, we present the extended results with three different fixed states and a varying reference action across the action space Ore [-1.0, 1.0]. In Fig. T0. the diagonal symmetry of the normalized NTK as a function of action distance observed. Generally, in the medium dataset, the NTK is high between two actions that are close to each other and low between actions that are far apart. However, in the expert dataset, this distinction becomes blurred, regardless of the proximity of the actions.
In Fig. (a) and (b) in Section 3.2
E.4 Action Distributions
Inverted Double Pendulum Export
Inverted Double Pendulum Modium Ico
COUIT
%
Hopper Expart
Happor Nedium
Hoaper Modium-Rcpizy
LU
1.
Lu
0.5
0.c.
Distance
Figure 11: The average L2 distance between different actions within each quantized state in the Inverted Double Pendulum and MuJoCo Hopper environments. All histograms are plotted with 50 bins.
In this subsection, we examine the action distributions of datasets within the Inverted Double Pendulum and MuJoCo Hopper environments. The continuous nature and multi-dimensionality of these action spaces pose significant challenges for directly visualizing the exact action distributions. To address this, we define the average action distribution spread of dataset D, quantified as the expectation over the states of the dataset as:
H(D) == E.ED (s)))D(s)
[l]a a|2]]
(8)
We then visualize the distribution of the L2 distance among all actions within each quantized state across the dataset using Eq. (8). This approach is based on the characteristies of the action dimensions in both the Inverted Double Pendulum and the Hopper, which are bounded between -1 and 1. For action distribution visualization, we maintain the same state and action quantization as outlined in Appendix E.1 Fig. presents the results. As depicted in this figure, in both the Inverted Double Pendulum and Hopper environments, the expert datasets exhibit a small average L.2 distance between actions coexisting in the same quantized state. This indicates denser clustering of actions within these datasets, which is linked to high-return datasets typically exhibiting a more concentrated action distribution on average since they primarily perform exploitation actions instead of exploration actions.
F Calculating R.
In Section 4.1 define the QCS weight w(R(T) as X XX- R(r)), where R² is the optimal return for the task. For calculating R", we consider the two methods described below.
(i) Set with the optimal return for the environment. our experiments, we set R for environments with optimal returns as follows: Hopper (R" 3500), Walker2d (R* 5000), Halfcheetah (R* 11000), and AntMaze (R* = 1). Note that prior RCSI. algorithms such as Decision Transformer [10] and RvS [2] used predefined Rº for target RTG conditioning
19

---

during inference. Therefore, using R based on the optimal return introduces no additional assumptions compared to previous RCSL methods. As noted in Appendix p.2] QCS does not use R' for target RTG conditioning but instead relies on the maximum trajectory return, requiring only one R per algorithm.
ii) Set R. to the maximum trajectory return within the dataset. An alternative approach for setting R' is to use the maximum trajectory return from the dataset. When obtaining the true optimal return from the environment challenging. the maximum trajectory return can serve as approximation. Table presents additional results using this method for R
Table 6: QCS performance with R' as the optimal environment and maximum dataset returns.
Dataset
(optimal return)
(max return)
hallcheetah- medium
59.0 = 0.4
55.2 0.5
hopper-medium
96.4 = 3.7
97. a 3.0
walker2d-mcdiom
88.2 1.1
87.4 a: 2.1
halfcheetah medium replay
54.1 0.8
52.1 + 0.7
hopper meetee--ddiiuillla
100.4 + 1.1
99.8 1.2
walleerd-mmdiuu--rplaa
94.1 = 2.0
90.6 3,2
As shown in Table 6 setting R' with the optimal enviromment return slightly better than setting with the maximum dataset return, but setting with the maximum dataset return still outperforms the baselines. Therefore, we propose using the optimal environment return for R" however, when is hard to determine, using the maximum dataset return be good alternative.
G More Experiments Results
G.1 Additional Performance Comparison
In addition to the performance comparison the MuJoCo and AntMaze domains, as discussed in Section 6.2 we also compare the performance of QCS in the Adroit domain using extensive baselines, similar to those mentioned in Section 6..2 Since there are no reported results for TD3+BC [14). SQL
[52. RvS [2], QDT [53] EDT 49 CGDT 43,. ACT
[6] POR 50 in the Adroit domain, CQL 25 and RCSL baselines (DT [10],
only compare with the value-based baselines (IQL 23
DC
20
RvS 12
For DT and DC, we evaluate the score using their official codebase.
Table 7 displays the performance QCS alongside the baseline performances. As indicated by the results, QCS-R outperforms other baselines in the Adroit Pen task. This outcome reiterates that QCS a robust framework, excelling in range of tasks with varying features. It also underscores findings from our earlier experiments, which demonstrate that strategically blending RCSL with Q-function can significantly enhance performance.
Table 7: The performance QCS and baselines in the Adroit domain. The boldface numbers denote the maximum score.
Method
Dataset
IQL
COL
DT
DC
QCS-R
pen-human
71.5
37.5
62.9
74,2
83.9 = 10.2
pen cloned average
37.3
39.2
28.7
50.0
66.5 9.5
54.4
38.4
45.8
62.1
75.2
G.2 Comparison with FamO20
FamO20 42 is an offline-to-onlinn RL method that facilitates adtativ balancing between policy improvement and constraints. During offline pre-traininn traiirann it develops a set of policies with various balance coefficients. In the subsequent online tinettunigg phase, Fam020 determines the most suitable policy for each state by selecting the corresponding balance coefficient from this set. The major difference between QCS and FamO20 is that FamO20 additionally uses 10" online samples to find suitable balance coefficient, while QCS only utilizes the offline dataset and adjusts the balance coefficient (QCS weight) based the trajectory return. Moreover, unlike FamO2O,
20

---

which utilizes state-adaptive balance coefficient, QCS is based on historical architecture and uses uub--rajectory-adaptie balance coefficient. Although it is not fair comparison between FamO2O, an offline-to-online algorithm, and QCS, purely offline algorithm, we present the performance comparison demonstrate the effectiveness of QCS even when compared with an offlin--toonnline algorithm.
Table Performance Comparison between FamO20 and QCS in the MuJoCo domain.
Table Performance Comparison between FamO20 and QCS in the AntMaze domain.
Dataset
FamO20 (offline Lo online)
QCS (offline)
halfchectah-m
59.2
59.0 + 0.4
Dataset
FamO20 (offline to online)
QCS (offline)
hopper-ng
90.7
96.4 3.7
walker2d.m
85.5
88.2 1.1
antmaze-u
96.7
92.3 + 4.6
antmaze-H u-d
70.8
82.5 8.2
antmaze-ma m-p
93.0
84.8 11.5
hopper-m-r
97.6
100.4 = 1.1
antmaze-m-
93,0
75.2 11.9
walker2d-m-rr
92.9
94.1 =
animace-1-
60.7
70.0 9.6
hallcheetalh-nn m-e
93.1
93.3 1.8
antmaze-I-d
64.2
77.3 11.2
hopper mi
87.3
110.2 = 2.4
walker2d-m-e
112.7
116.6 2.4
average
79.7
80.4
average
85.8
90.3
H
More Ablation Studies
H.1
Comparing Assistance from Actor-Critic Learned Q- V-Vuuees
To compare the performance of QCS using Q-function learned through actor-critic algorithms, we use representative actor-critic algorithms such as CQL [25] for benchmarking. As shown in Table 10 the performance of CQL-aided QCS generally improved compared to the original CQL, but does not match the performance of IQL-aided QCS for the MuJoCo domain. The reason can be attributed to two factors: (1) the Q-function may be bounded by the actor's representation ability, and (2) CQL might impose excessive conservatism on the Q-function. Moreover, in the case of - diverse, IQL-aided QCS underperforms CQL, but CQL-aided QCS outperforms CQL. Since QCS is general framework that proposes a new combination of RCSL and Q-function on trajectory return, there is a wide range of potential integrations of RCSL and offline Q-learning methods. The most impactful aspect will differ depending on the characteristies of each RCSL and Q-learning method when combined. Exploring this would be an interesting research area, which we leave future work.
Table 10: The performance of CQL, CQL-aided QCS, and IQL-aided QCS. The dataset names are abbreviated as follows: medium as 'm', medium-replay as 'm-r', and imaze-dive divrse as 'u-d'.
Dataset
CQL.
CQl QCS
IQL-aided QCS
mejoco-medium
58.3 1.2
68.1 1.5
81.2 1.8
mujoco-medium-reply
72.6 4.1
75.7 5.8
82.9
antmaze -iave-divvreee
84.0
85.2
82.5
H.2 Impact of Base Architecture and Conditioning
In Section we discussed QCS variants with different base architectures and conditioning. To assess the impact of these on performance, we conducted additional comparisons between QCS implementations with and without conditioning across three base architectures: DT, DC, and MLP. Table reveells that the choice of base architecture does not significantly impact performance, except for the Adroit Pen. However, conditioning proves to be particularly beneficial for complex tasks and datasets with diverse trajectory optimality. Generally, we found that the DC base architecture advantageous.
21

---

Table 11: Comparison of the base architecture of QCS and the ablations on conditioning. For the MuloCo and Adroit domains, we utilize QCS-R, and for the AntMaze domain, we utilize QCS-G for evaluation. The dataset names are abbreviated as follows: medium to 'm', medium-replay 'm-r', medium- exxperrt to 'm-e', umaze 'u', umaze riazree 'u-d', medium-play to p", medium-diverse to 'm-d', large-play to 'l-p', and large- -iveree to 'l-d', The boldface number represents the higher value when comparing the base architecture.
halfcheetah
Dutaset
DT D'T'uueed
DC based
MLP-based
Dataset
Condition
DXC-based
DC-based
hopper-m
halicheelah hopper-m
walker2d-m
85.4
88.2
halfeheetah--m-t
53.7
54.1
53.2
walker2d m
83.9
88.2
hopper-mtr
99.1
100.4
102.4
balfehoctah-m--
53.6
54.1
walker2d-mir
90.9
94.1
93.3
hopper-m-r
76.6
100.4
hulfcchettlh mn
94.4
93.3
84.0
wa walker2d-m--r
90.8
94.1
hopper-m-e
110.2
110.2
110.4
antmazc-m-p
80.3
84.8
walker2d
antmaze-m--
71.2
75.2
antmaze-I-p
41.2
70.0
antmaaze-u
antmaze-I-d
33.0
77.3
antmaze-m-p
75.2
82.1
pen-buman pen- pen-cloned
antmare m-d
72.1
75.2
80.1
antmare-l-p
66.2
70.0
67.7
antmaze-I-d
75.3
77.3
68.9
pen-human
76.8
83.9
59.4
pen clonel
40.2
66.5
44.0
22

---

I Training Curves
OCS-R
QCS-G
hallcheetah-medlim
hopper-medium
walker2d Balker/d-mmellum
100
Normalized Ruturn
80
Normalized Return
80
40
60
30
Norma
20
20
10
2
Gradient Steps (10')
Gradient Steps (1.07)
Gradient Steps (107)
maficheetah-mediune replay
hoaper-medlumrrplaay
waller22-medduu-rrppay
100
50
Normalizen Returm
Normelized Return
80
40
Normalized
30
20
20
2
Gradient Steps (1.0))
Gradient Steps (1.F))
Credient Steps (10F)
hal/cheetah medium expert
hepper-medlum expert
Walker2d medi expert
100
125
Normalized Reetum
100
Normalized Return
100
75
Normaltred
40
50
20
25
0
0.
0
3
0
1
3
4
2
Steps
(30')
umnaze
umaze-diverse
medium-ppay
100
100
100
Normalized Retum
Normalized Return
Nomalized
60
60
60
40
40
40
3
6
9
10
2
5
8
10
2
5
10
Gradient
(207)
Gradient (107)
Gradient
(107)
medium-diverse
large-ploy
large-diiverse
100
100
100
80
Hormalized Returm
80
Normalized Return
Normalized
40
40
40
Gradient Steps (10P)
Gradient Steps (10F)
Gradient Steps (10F)
Figure 12: Training curves of QCS-R and QCS-G in the MuloCo and AntMaze domains.
23

---

Implementation and Hyperparameters
J.1 Training the Q-Function
QCS utilizes Q-aid, where the Q-function is learned through IQL [23], using the open-source imple- mentation of IQL (nttps://github: com.//aacchrry/gg-prrooooo
and following the common
nyperparameters recommended by the authors, as listed in Table 12 For AntMaze, we set the expectile 0.8, whereas for other domains, we it to 0.7. Moreover, inspired by RLPD [5] and SPOT 46 we employed Layer Normalization [4] and larger discount 0.995 for the Q and V networks in AntMaze. For fair comparison, we retrained IQL using these modified hyperparameters. and the results are shown in Table 13 Since this modified setting had a negative effect on IQL, we used the IQL performances from the IQL paper
23 for Table
Table 12: Common hyperparameters for QCS Q-function training.
Hyperparameter
Value
Optimizer
Adam
21
Learning rate
3e-4
Batch size
256
Target update rate
5c-3
Hidden dim
256
Nonlinearity function
ReLU
Table 13: Comparison of the IQL. performances reported in the IQL paper modified hyperparameters.
23
with our results using
(reported
IQL hyper-params)
antmaze-u
87.5
87.9
untmazz-ud
67.2
38.7
antmazo-m-p
71.2
50.7
antmazo-m-d
70.0
45.3
antmaze- 1p
39.6
16.3
antmaze-1dd
47.5
9.3
J.2 Policy Training
Detailed description of the loss function.
A. AR' - -R(r(0))
CQCS EBBD
(1) (eattet)
11
(1()
(2)
K
Q
(esttt))
RCSL
Aid
where each component of the loss function is as follows:
The batch sampled over the entire dataset D (c.g., hopper medium):
B (1) Phttt *atta K-
[B]
i-th sub-trajectory the batch for i= 1,..., B:
(
(()) (1) (
50)
(1)
R St ai
Dataset-level Q-normalizer:
1101 0
1
Σ
Q
101
(s,a),
D
(s.a)ED
i.c., the dataset-level Q-normalizer QIOL is the mean of the Q-value for all samples in the dataset.

---

Implementations. After training the Q-function, we train our policy with three different base architectures: DT TO. DC (20), and MLP. For DT-based QCS, we utilize the official DT codebase (https://githe thub com http:////d/eiioooneeeea transformerr) for our implementation. Sim- ilarly, for DC-based QCS, we use the official DC codebase (https://gtt thtgg.thu..... wwt.ccoo//abieeeee beanie0n/ Docision-ConvForee for our implementation.
Hyperparameters. For the AntMaze domain, we used 10th gradient steps, and for the other domains, we used x 10 gradient steps for training the policy. For all domains and base architectures, QCS uses a dropout rate of 0.1, ReLU as the nonlinearity function, weight decay of le-4, and LambdaLR scheduler 33 with linear warmup of 10 gradient steps. In addition, we use context length K of 20 for DT-baaeed QCS, 8 for DC-based QCS, and for MLP-based QCS. We found that the impact of action information and positional embedding on performance was negligible, so we excluded them from training. QCS-R, we set our target RTG the highest trajectory return in the dataset. For the MuloCo and Adroit domains, we evaluate the target RTG at double this value. In the AntMaze domain, we test it 100 times the value. This method aims to leverage the RTG generalization effect observed by Kim et al. [20] We then report the best score achieved across the two target RTGs, From Table 14
15
we provide detailed hyperparameter settings for actor training.
Table 14: Per-domain hyperparameters of DT- based QCS and DC-based QCS.
Table 15: Per-domain hyperparameters of MLP. based QCS.
Elyperurameeer
Hlpperpuramettr
Hiddeem dim
256
512
128
Hiddemm din
1024
1024
256
à layers
# layers
Batch size
64
256
64
Batch size
64
256
64
Learning rate
le-4
3e-4
30-4
Learning rate
1c-4
3e-4
3e-4
Our analysis suggests setting the QCS weight wi R(T)) as a continuous, monotone-decreasing function the trajectory return R(T). We explored various functional forms, including linear, quadratic, and exponential decay, but found that simple linear decay ((((T)) * - [(T)) suffices. addition, we found that for some datasets, clipping w (((T) to minimum of 10 is beneficial, particularly for
QCS Weight Relative to Trajectory Return.
and QCS-R AntMaze, except umaze-divers The choice of for each dataset is presented Table
16 to 18
Table 16: À on MuJoCo.
Table 17: on AntMaze.
Table 18: À on Adroit.
Dataset
Dataset
À
Dataset
aifcheetah--ediine
antmaze-umaze
0.2
pen-human 0.01
halfcheetah- wathhheetah-mmiiu----e medium-replay
antmazz-umaze-divense
0.05
pen-cloned 0.01
halfcheetah medium -epertt-expert
0.5
antmaze-medium play
0.2
hopper-medium medium
0.5
untmaze-nmediun-diivrree
0.2
0.5
antmaze-large-play
0.2
hopper-medee medium- expert
0.5
antmaze-large-divee
0.2
walker2d-meding
0.5
malker2d-medium-reeee
walker22d-medium-expent
K Training Time
We compare QCS training time complexity with IQL 233. CQL 25 and QDT 53
QCS requires a pre-trained Q learned using the IQL method, while QDT requires pre-trained Q learned using the CQL method. Therefore, the time was measured by incorporating the Q pretraining time for both algorithms.
The training times for IQI, CQL., QDT, and QCS are as follows: IQL. 80 min, CQI. min, QDT 400 min, and QCS 215 min.
25

---

The results show that QCS takes longer than IQL but has total time similar to CQL. Notably, compared QDT, which requires CQL pretraining, QCS can be trained in nearly half the time but demonstrates superior performance to QDT as shown in our main results in Table 2
L Limitations
In this paper, we leveraged the complementary relationship between RCSL and Q-function over generalization to determine the QCS weight linear function of the trajectory return, which is readily obtainable from the dataset. This approach was tested on MuJoCo, AntMaze, and Adroit, where it showed promising results. However, depending the task, more advanced method that can efficiently evaluate Q-functions's over-generalization and provide appropriate Q-aid might be necessary. Additionally, this method entails the extra burden of pre-training the Q-function.
M Borader Impacts
This research is centered on enhancing the strengths of two promising approaches in the field of offline reinforcement learning: RCSL and value-basee methods. By overcoming each of their limitations and creating better trajectories than the maximum quality trajectories of existing datasets, it contributes to the advancement of offline reinforcement learning. As foundational research in machine learning. this study does not lead to negative societal outcomes.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction reflect all contributions in the paper well.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
The paper does not deal with theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
The project page is available at [bttger//baaaeeo..cc////bllalaat
qca
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//guuu guwde//ddee/ee/ Coc//ooodeeesiseeeeeeo hwtpbbbmsssiooooooooiooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section 6.1 (.1aa and Appendix J
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In our main experimental results, Table and Table 3 in Section and Table 7 in Appendix[Gi GJ we provide the mean and standard deviations for the five random seeds.
Moreover, we provide the training curves in Fig. 12 Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
K
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp::/neuurj:: http:pp.....ppplcccttccccseeeee.
Answer: [Yes]
Justification:
The authors have read the https://heuripsscc/ppbbb
Ethi cthiisGuueeliies and ensured that this paper conforms
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix
M
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix E. and
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See our anonymized zip file.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "yLb5wasdAB", "reviewer": "Reviewer_WnjH", "summary": "This submission proposes an algorithm that combines the stability of return-conditioned supervised learning (RCSL) with the stitching capability of Q-functions. The submission tests their algorithm in the MuJoCo domain with medium, medium-replay, and medium-expert datasets. The performance of the submission’s proposed method is higher on average in the MuJoCo domain.", "questions": "Why did the authors bold their algorithm while SQL is the highest performing algorithm for hopper-m-e in Table 2? (111.8>110.2)", "limitations": "Please see weaknesses.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper tests their proposal in the main benchmarks.", "weaknesses": "The HQIL algorithm [1] outperforms the proposed method of the submission. However, this is not mentioned, and further there is no comparison to HQIL.\n\nTable 1 might need some standard deviations.\n\nTo be able to interpret Table 2 correctly, the table needs to have standard deviations of the previous methods as well.\n\nPOR method can also perform above 70 in the antmaze-large-diverse dataset. \n\nOther studies also report results for antmaze-medium-replay dataset. Why did the authors omit this baseline when reporting results?\n\nTable 8 again needs standard deviations.\n\nThe submission only compares their algorithm to old baselines. However, there are currently algorithms that outperforms the proposal of the submission.\n\nSection 3.2. argues about something that is already expected/known to anyone that knows basic reinforcement learning, i.e. how the $Q$-learning update works. Should this section really occupy two pages in the main body of the paper?\n\n*“QCS represents a breakthrough in offline RL,pushing the limits of what can be achieved and fostering further innovations.”* \n\nThis statement might be too strong to describe the submission's contributions\n\n[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions, NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's efforts in providing constructive feedback to improve our work. **Our detailed response to the reviewer's comments is posted below and also in the global response G1-G4.**\n\n&nbsp;\n\n### **W1. Comparison with HIQL.**\n\nThank you for introducing the highly effective algorithm HIQL [Ref.2] for the goal conditioning task. RL tasks can be divided into two categories: return-maximizing tasks, which aim to earn the maximum return, like MuJoCo, and goal-reaching tasks, which aim to reach the goal with a higher success ratio, like Antmaze. HIQL specializes in goal-reaching tasks as it uses a hierarchical approach that generates high-valued subgoals with a higher-level policy and generates actions with a lower-level policy conditioned on those generated subgoals. HIQL achieves superior performance on Antmaze tasks, but due to the nature of the algorithm, which involves generating and conditioning on subgoals, it has limitations when applied to tasks like MuJoCo.\n\nIn Table R2, we compare QCS-R and QCS-G with HIQL in the Antmaze medium and large datasets. QCS previously used a hidden dimension of 256 for training the Q-function, and we observed that HIQL uses a hidden dimension of 512. Therefore, we re-trained the Q-function with an increased hidden dimension of 512 and confirmed that this setting is beneficial for QCS. In Table R2, we mark the original QCS-R/QCS-G score as QCS-R(256)/QCS-G(256) and the new score as QCS-R(512)/QCS-G(512).\n\nAs shown in Table R2, our new QCS score is comparable to HIQL, which is specialized for goal conditioning tasks, and particularly, QCS-R performs even better than HIQL. This new result reaffirms that our algorithm, despite being general, is effective even when compared to specialized algorithms.\n\n&nbsp;\n\n**[Table R2. Performance comparison between QCS and HIQL.**]\n\n| Dataset     | HIQL          | QCS-R(256)     | QCS-R(512)     | QCS-G(256)     | QCS-G(512)     |\n|-------------|---------------|----------------|----------------|----------------|----------------|\n| antmaze-m-p | 84.1±10.8     | 81.6±6.9       | **93.1±7.0**   | 84.8±11.5      | 88.5±6.3       |\n| antmaze-m-d | 86.8±4.6      | 79.5±5.8       | **88.7±5.4**   | 75.2±11.9      | 84.9±8.4       |\n| antmaze-l-p | 88.2±5.3      | 68.7±7.8       | **89.2±6.3**   | 70.0±9.6       | 84.2±10.5      |\n| antmaze-l-d | **86.1±7.5**  | 70.6±5.6       | 84.2±10.6      | 77.3±11.2      | 77.1±7.0       |\n| **average** | **86.3**      | 75.1           | **88.8**       | 76.8           | 83.7           |\n\n&nbsp;\n\n[Ref.2] Park, Seohong, et al. \"Hiql: Offline goal-conditioned rl with latent states as actions.\" NeurIPS 2023.\n\n&nbsp;\n\n### **W2. Recentness of Baselines.**\n\nWe want to note that we compared baselines including very recent works such as ACT (AAAI 2024), CGDT (AAAI 2024), DC (ICLR 2024), FamO2O (NeurIPS 2023), EDT (NeurIPS 2023), QDT (ICML 2023), SQL (ICLR 2023). Especially since the NeurIPS 2024 submission deadline was May 22, considering this date, the works for AAAI 2024 and ICLR 2024 can be regarded as very recent, having been published less than three months ago. \n\nWe continuously track the latest works, and after the NeurIPS submission deadline, we found a new work, Q-value Regularized Transformer(QT) [Ref.6] that proposes a new way of combining RCSL and Q-learning, published at ICML 2024. Note that according to NeurIPS regulations, QT is considered `Contemporaneous Work' because it appeared online after the submission. Since the work appeared after our submission, we are not obligated to compare it, but to help the reviewer better appreciate the quality of our work, we additionally compare QCS with QT in Table R3 in response W3 using the same evaluation metrics as QCS. \n\n[Ref.6] Hu, Shengchao, et al. \"Q-value Regularized Transformer for Offline Reinforcement Learning.\" ICML 2024.\n\n&nbsp;\n\n### **W3. Comparison with QT and POR with the same evaluation metric of QCS.**\nAs mentioned in our manuscript Section 6.1, we report QCS as the **last running average score**. However, QT selects the best score throughout the whole training process (as confirmed through email with the QT author), which is not suitable for offline RL settings. Offline RL assumes a situation with limited online interaction, but finding the best score relies heavily on a significant amount of online interaction. Therefore, we re-ran the QT through the official QT code with author-recommended hyperparameters and verified the last running average score. \n\nRegarding POR, mentioned by the reviewer for the antmaze large score, we previously reported POR scores from the POR paper. We were unable to confirm the exact evaluation metric, but the authors shared the training curve of POR on the POR github. From this, we verified the last running average score of POR.\n\nTo summarize, in Table R3 and Table R9 in the global response PDF, we compare QCS with QT and POR using the last running average score. As can be seen in Tables, QCS outperforms both POR and QT in MuJoCo and Antmaze domains, with POR's Antmaze large scores being below 70, demonstrating that QCS is a robust and high-performing algorithm. \n\n&nbsp;\n\n**[Table R3. Performance comparison between QCS (ours), POR, and QT. We evaluate each algorithm based on the last running average score.]**\n\n| Dataset        | POR (last)      | QT (last)      | QCS (ours, last)      |\n|----------------|-----------------|----------------|-----------------|\n| antmaze-u      | 87.9±4.0        | 51.3±15.7      | **92.5**±4.6        |\n| antmaze-u-d    | 65.5±6.0        | 57.9±9.6       | **82.5**±8.2        |\n| antmaze-m-p    | **84.6**±5.4        | 32.7±11.3      | **84.8**±11.5       |\n| antmaze-m-d    | **75.6**±4.8        | 22.7±23.6      | **75.2**±11.9       |\n| antmaze-l-p    | 56.5±5.2        | 0±0.0          | **70.0**±9.6        |\n| antmaze-l-d    | 63.0±4.0        | 0±0.0          | **77.3**±11.2       |\n| **average**    | 72.2        | 27.4       | **80.4**        |\n\n&nbsp;", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qw2TQy4UJE", "reviewer": "Reviewer_K7ih", "summary": "Offline reinforcement learning (RL) has advanced with return-conditioned supervised learning (RCSL) but still lacks stitching ability. Q-Aided Conditional Supervised Learning (QCS) combines RCSL's stability with the stitching capability of Q-functions, addressing Q-function over-generalization. QCS adapts Q-aid integration into RCSL's loss function based on trajectory returns, significantly outperforming RCSL and value-based methods in offline RL benchmarks. This innovation pushes the limits of offline RL and fosters further advancements.", "questions": "- how the subgoals are chosen? Is not it complicated to choose them in a more complex environment?", "limitations": "The conditioning part seems to be controled by the algorithm which makes it hard to use in any environment.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- it is well-written and fluent. \n\n- They brought clear definitions which make it easy to follow the context. \n\n- They compared with different baselines.", "weaknesses": "The innovation of the works is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing comments and positive feedback on our work. We hope that our response below will clarify our algorithm and innovations.\n\n&nbsp;\n\n### **W1. The innovation of QCS.**\n\nWe summarize the innovations of QCS as follows:\n* We first analyze the strengths and weaknesses of RCSL and Q-learning in various dataset qualities and discover that RCSL and Q-learning can be complementary depending on the quality of the dataset.\n* Especially, we identify the Q-function over-generalization problem when performing Q-learning with an optimal quality dataset.\n* Based on the analysis of Q-function over-generalization, we propose a novel algorithm that combines RCSL and Q-learning, called QCS. QCS demonstrates superior performance compared to various recent and SOTA baselines.\n\nWe want to emphasize that our analysis of the complementary conditions of RCSL and Q-learning, as well as the Q-function over-generalization problem, is novel and has not been addressed in previous works. The superiority of the QCS algorithm, which is based on these analyses, further confirms that our analyses are necessary and can lead to a deeper understanding within the offline RL community.\n\n&nbsp;\n\n### **Q1. Regarding the subgoal selection.**\n\nFor QCS-G, we follow the subgoal selection as proposed by RvS [Ref.7]. As per RvS, during the training phase, we randomly select a subgoal from among the states between the next state and the final state. For the evaluation phase, we prefix the subgoal as the goal state. We select subgoals based on the states present in the dataset, and there is no need for a special algorithm; we simply choose randomly among the next states, incurring no additional computational cost or difficulty.\n\nMoreover, as can be seen in Table 3 of our manuscript, the QCS-R score, which only utilizes reward information, shows performance comparable to QCS-G. This demonstrates that the QCS algorithm can achieve good performance even without subgoal selection.\n\n&nbsp;\n\n[Ref.7] Emmons, Scott, et al. \"RvS: What is Essential for Offline RL via Supervised Learning?.\" ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c14SzEPbyx", "reviewer": "Reviewer_fTMz", "summary": "Authors of this paper introduce a new algorithm called Q-aided Conditional Supervised Learning (QCS) for offline DRL scenario, which combines the stability of return-conditioned supervised learning (RCSL) and the stitching ability of Q-functions. The primary contributions are 1). Identify the strength and weakness of RCSL and Q-learning in different experimental settings and 2) propose a new approach to combine the them to achieve better performance.", "questions": "1. For calculating R*, for each update using equation (3), do you need to calculate R* for each trajectory? Could you provide more detailed information on how it is calculated?\n\n2. I think the hyper-parameter K has very limited discussion on its impact on the performance. Do you perform any ablation studies on how K will affect the model performance?\n\n3. How sensitive is the QCS algorithm to the scaling factor $\\lambda$ ? I noticed that, for one environment with different offline settings, (for example, halfcheetah-medium-replay and halfcheetah-medium-expert), OCS uses different $\\lambda$ values. Does one fixed $\\lambda$ work well within the same environments? I think this is important for understanding the adaptability and robustness of the method across different settings.", "limitations": "The paper briefly mentions the potential limitation of using a simple linear function to combine RCSL and Q-function.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written. The paper has a logical structure on the problem definition. It first demonstrates the strength and weakness of RCSL and Q-learning in different offline settings through experimental results and then explain the reasons through simple toy examples to help understanding. Finally, it propose the novel approach QCS based on the reasoning above. \n\n2. The paper has rigorous experimental set-up and SOTA performance. The proposed algorithm is tested on various offline D4RL benchmarking datasets and demonstrate SOTA performance across the datasets.\n\n3. Very interesting study on the strength and weakness of RCSL and Q-learning in different experimental settings, especially the discovery of Q-learning over generalization problem.", "weaknesses": "1. If I understand correctly, QCS requires a pre-trained Q-function using IQL, while the experimental results on D4RL benchmarking seems very impressive, there appears to be a lack of detailed discussion concerning the computational workload.\n\n2. The weight term (which is important and essential) in the algorithm is dynamic and depends on R*, the optimal return of the task. Calculating R* across environments and dataset settings can be a bit tricky and discussion on how calculation is performed is a also a bit limited and unclear. \n\n3. Different benchmark experiments seem to require different hyper-parameter $\\lambda$ values to achieve SOTA performance. Not sure how sensitive QCS is to $\\lambda$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments that help clarify our algorithm and for the positive feedback on our work. As suggested, we have included further explanations and experimental results to enhance the significance of our study. Due to the rebuttal word limit, **we have addressed the response regarding context length in the global response G5.**\n\n&nbsp;\n\n### **W1. Regarding complexity.**\n\nWe compare QCS training time complexity with IQL, CQL, and QDT [Ref.8]. QCS requires a pre-trained Q learned using the IQL method, while QDT requires a pre-trained Q learned using the CQL method. Therefore, the time was measured by incorporating the Q pretraining time for both algorithms.\n\nThe training times for IQL, CQL, QDT, and QCS are as follows:\nIQL - 80 min, CQL - 220 min, QDT - 400 min, and QCS - 215 min.\n\nThe results show that QCS takes longer than IQL but has a total time similar to CQL. Notably, compared to QDT, which requires CQL pretraining, QCS can be trained in nearly half the time but demonstrates superior performance to QDT as shown in our main results in Table 2.\n\n&nbsp;\n\n[Ref.8] Yamagata, Taku, et al. \"Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modeling in offline rl.\" ICML 2023.\n\n&nbsp;\n\n### **W2/Q1. Regarding calculating R***\n\nThe detailed explanation of how to obtain R$^*$ is as follows. We propose two ways of calculating R$^*$, and we will add this explanation to our manuscript.\n\n* (1) Set R$^*$ with the optimal return for the environment.\n\nFor our main experiments, we set R$^*$ for the environments with the optimal return for the environment as follows: Hopper - 3500, Walker2d - 5000, Halfcheetah - 11000, and Antmaze - 1. Note that prior RCSL algorithms, such as Decision Transformer [Ref.8] or RvS [Ref.7], used predefined R$^*$ for target RTG conditioning at the inference stage. Therefore, setting R$^*$ with optimal return requires no further assumptions compared to prior RCSL works. As mentioned in Appendix J.2 of our manuscript, for target RTG conditioning, QCS does not use R$^*$ but instead uses the maximum trajectory return, only requiring one R$^*$ per algorithm.\n\n* Set R$^*$ to the maximum trajectory return within the dataset.\n\nAnother way of setting R$^*$ is to use the maximum trajectory return within the dataset. In situations where obtaining the optimal environment return is difficult, we can infer the optimal return using the maximum trajectory return. Therefore, in Table R4 we provide additional results using the maximum trajectory return as R$^*$.\n\n&nbsp;\n\n**[Table R4. QCS performance when R\\* is the maximum dataset return and the optimal environment return.]**\n\n| Dataset                   | QCS (optimal env return) | QCS (max dataset return) |\n|---------------------------|--------------------------|--------------------------|\n| halfcheetah-medium        | 59.0±0.4                 | 55.2±0.5                 |\n| hopper-medium             | 96.4±3.7                 | 97.1±3.0                 |\n| walker2d-medium           | 88.2±1.1                 | 87.4±2.1                 |\n| halfcheetah-medium-replay | 54.1±0.8                 | 52.1±0.7                 |\n| hopper-medium-replay      | 100.4±1.1                | 99.8±1.2                 |\n| walker2d-medium-replay    | 94.1±2.0                 | 90.6±3.2                 |\n\n\nAs shown in Table R4, setting R$^*$ with the optimal environment return is slightly better than setting it with the maximum dataset return, but setting it with the maximum dataset return still outperforms the baselines. Therefore, we propose using the optimal environment return for R$^*$; however, when it is hard to determine, using the maximum dataset return can be a good alternative.\n\n&nbsp;\n\n[Ref.7] Emmons, Scott, et al. \"RvS: What is Essential for Offline RL via Supervised Learning?.\" ICLR 2022.\n\n[Ref.8] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" NeurIPS 2022.\n\n&nbsp;\n\n### **W3/Q3. Impact of the QCS weight $\\lambda$.**\n\nWe wish to direct the reviewer's attention to Table 9 in Appendix H.2 of our original manuscript, which investigates the effect of $\\lambda$ by varying it from 0.2 to 1.5. We additionally test the QCS with $\\lambda$ 0.5 and 1 with MuJoCo medium-expert datasets and the result can be seen in Table R5. \n\nAs shown in Table 9 and R5, except for the walker2d-medium and halfcheetah-medium-expert, we found that even the smallest values achieved with changing $\\lambda$ either matched or surpassed the performance of existing value-based methods and RCSL's representative methods, including IQL, CQL, DT, DC, and RvS. This demonstrates QCS's relative robustness regarding hyperparameters. For walker2d-medium and halfcheetah-medium-expert, we found that when $\\lambda$ exceeds the initial setting of 0.5, performance begins to decrease. We anticipate that further research will address these issues.\n\nOverall, setting $\\lambda$ to 0.5 consistently produces good performance, even though it results in performance degradation of around 2 points in some environments. **Compared to many offline RL algorithms that require tuning more than 10 sets of hyperparameters depending on the environment, achieving stable performance with only one or two hyperparameter adjustments is a major strength of QCS.**\n\n&nbsp;\n\n**[Table R5. Impact of the QCS weight $\\lambda$ in MuJoCo medium-expert datasets.]**\n\n| Dataset                   | $\\lambda$=0.5           | $\\lambda$=1             |\n|---------------------------|-------------------------|-------------------------|\n| halfcheetah-medium-expert | 93.3±1.8                | 84.6±4.2                |\n| hopper-medium-expert      | 110.2±2.4               | 110.8±2.8               |\n| walker2d-medium-expert    | 117.4±2.0               | 116.6±2.4               |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Romz14gvE2", "reviewer": "Reviewer_Ndxq", "summary": "This paper proposes an offline reinforcement learning framework by combining return-conditioned supervised learning with in-sample Q-learning. By demonstrating their relative merits on offline datasets with different behavior policies, an overall loss function is designed to integrate Q assistance into RCSL. Extensive experiments empirically validate its superior performance with compared baselines on several domains and datasets.", "questions": "All my questions are listed in the weakness part.", "limitations": "No limitation issues.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe studied problem is important. Empowering RCSL with stitching ability is crucial to facilitate its capability for pursuing optimal policies. \n2.\tThe analysis and experiments of RCSL and Q-learning on datasets with optimal and suboptimal policies provide valuable insights. Based on that, the proposed QCS method can be naturally optimized on the proposed loss function by considering conditional supervising learning and maximizing Q-function simultaneously.\n3.\tEmpirical experimental results are promising with a bunch of baselines on several datasets and domains.", "weaknesses": "1.\tThe definition of degree of Q-aid as w(R($\\tau$)) is not completely reasonable. As described, it should be defined by the degree of optimality of the behavior policy used to generate this sequence. However, a sequence with a higher return does not necessarily indicate it was generated by a superior policy, even from a statistical perspective. This definition only makes sense if the initial states of the compared trajectories are the same.\n2.\tThe analysis of the Q-Greedy Policy appears somewhat redundant. The focus of this paper is on integrating Q-learning into GCSL, rather than addressing challenges in Q-learning.\n3.\tConversely, the experimental section seems overly brief. I suggest moving more experimental details from the appendix to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments on the strengths of our work. Your valuable suggestions will help us further improve and clarify our work.\n\n&nbsp;\n\n### **W1. The definition of degree of Q-aid.**\n\nThank you for pointing out a good point. We have dealt with environments where the deviations in the initial state are marginal to affect the scale of the returns significantly, but in practical scenarios, differences may arise. In such cases, a higher trajectory return may not always indicate optimality. Considering how to set the weights in a more practical setting with significant differences in the initial state could be a good follow-up work.\n\n&nbsp;\n\n### **W2. Regarding writing.**\n\nAs the reviewer mentioned, the focus of our work is combining Q-learning with RCSL. To develop an algorithm for this purpose, we first (1) experimentally demonstrated that the Q-greedy policy and RCSL can be complementary depending on the quality of the dataset, and (2) analyzed the reasons for this through an examination of Q generalization. Based on this analysis, we determined the QCS weight and developed the QCS algorithm. Therefore, we believe that discussing the Q-greedy policy is necessary to explain the need for and development process of our algorithm. However, we will improve the writing to make this point brief and clear. However, we will improve the writing to make this point brief and clear. Additionally, we will refine our manuscript to include more experimental details in the main section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We express our deepest gratitude to all the reviewers for their time and effort in evaluating our work and providing valuable advice. Our responses to the reviewers' comments have been left as replies to each review. Moreover, due to the rebuttal word limit, we have posted the responses that could not be addressed to individual reviewers here and specified which reviewer each response is for. Additionally, the content regarding the standard deviation pointed out by the reviewer WnjH can be found in the PDF attachment.

&nbsp;

### **G1. Regarding Antmaze-medium-replay dataset. - reply to reviewer WnjH** 

According to [Ref.1], there are six datasets (antmaze-umaze, antmaze-umaze-diverse, antmaze-medium-play, antmaze-medium-diverse, antmaze-large-play, antmaze-large-diverse) for the Antmaze task, and we could not find an antmaze-medium-replay dataset. Neither the baselines we compared nor the HIQL [Ref.2] mentioned by the reviewer referred to the antmaze-medium-replay dataset. However, we recognize that new benchmarks are continuously being updated and our knowledge may be incomplete. If the reviewer can provide information on where we can find the antmaze-medium-replay dataset or any prior work that evaluates using that dataset, we will conduct additional tests.

&nbsp;

[Ref.1] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv 2020.

[Ref.2] Park, Seohong, et al. "Hiql: Offline goal-conditioned rl with latent states as actions." NeurIPS 2023.

&nbsp;

### **G2. Regarding Section 3.2.  - reply to reviewer WnjH**

We'd like to kindly ask the reviewer for specific parts in section 3.2 that seem to be unnecessary. We would be happy to revise and improve our paper accordingly.

In Section 3.2, what we are trying to convey is not 'how the Q-learning update works,' but rather **analyzing why a Q function trained with an optimal dataset tends to over-generalize, and why this is not the case when trained with a medium-quality dataset.** This analysis is novel and has not been addressed in previous works. Since we are comparing and analyzing the Q function based on dataset quality, the analysis naturally begins with how the Q function is updated. However, this is merely the starting point of the analysis; the crucial point is how in-sample Q-learning progresses according to dataset quality. We believe that properly analyzing offline Q-learning in various settings helps establish a logical structure, provides insight into the problem definition, and naturally leads to the development of the QCS algorithm. We will update the manuscript to clearly convey our key points.

&nbsp;

### **G3. Regarding Standard deviation - reply to reviewer WnjH**

Thank you for pointing this out and allowing us to make our score reporting more complete. The standard deviations for Tables 1, 2, and 8 in our manuscript have been added to Tables R6, R7, and R8 in the PDF attachment within the global response. Since some baselines do not report the standard deviation for their own algorithms [Ref.3, Ref.4, etc.], we add standard deviation except for those works.

&nbsp;

[Ref.3] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline Reinforcement Learning with Implicit Q-Learning." ICLR 2022.

[Ref.4] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.

&nbsp;

### **G4. Regarding wrong bolding. - reply to reviewer WnjH**
Thank you for pointing out. We made a mistake with the bold in the hopper-medium-expert dataset and will correct it. We also noticed that for the hopper-medium-replay dataset, the correct SQL score is 99.7, but we mistakenly reported it as 101.7 higher than 99.7. Since 101.7 was the maximum score we previously marked in bold, we will correct the score and remove the boldface.

&nbsp;

### **G5. Impact of the context length K. - reply to reviewer fTMz**

We conducted additional experiments by varying the context length of QCS. As seen in the results, the QCS is highly robust to changes in context length. The QCS is based on the DC [Ref.5], which emphasizes the local context in reinforcement learning, allowing it to achieve good results even with a short context length.

&nbsp;

**[Table R1. QCS with varying context length $K$.]**

| Dataset | K=2 | K=8 | K=20 |
|---------|-----|-----|------|
| halfcheetah-medium | 60.2±0.6 | 59.0±0.4 | 59.2±0.3 |
| hopper-medium | 97.7±2.5 | 96.4±3.7 | 95.5±5.1 |
| walker2d-medium | 88.4±0.9 | 88.2±1.1 | 87.8±2.0 |
| halfcheetah-medium-replay | 54.0±0.4 | 54.1±0.8 | 52.6±0.8 |
| hopper-medium-replay | 100.1±1.5 | 100.4±1.1 | 99.0±3.4 |
| walker2d-medium-replay | 86.4±5.1 | 94.1±2.0 | 88.6±4.1 |

&nbsp;

[Ref.5] Kim, Jeonghye, et al. "Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making." ICLR 2024.

Meta-review: The submission presents an analysis of the failure modes of RCSL and value-based approaches in Offline RL. While this analysis may not be particularly novel or surprising, it effectively establishes a solid foundation for developing a new solution that mitigates these failures. The proposed algorithm, though simple, delivers compelling empirical results that validate the soundness of the approach. Based on these strengths, I recommend accepting this submission as a poster.
Decision: Accept (poster)