Title: Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: offline reinforcement learning

return conditioned supervised learning

Q function

dynamic programming
Abstract: Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce $Q$-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of $Q$-functions. By analyzing $Q$-function over-generalization, which impairs stable stitching, QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value-based methods, consistently achieving or exceeding the highest trajectory returns across diverse offline RL benchmarks. QCS represents a breakthrough in offline RL, pushing the limits of what can be achieved and fostering further innovations.

Adaptive Q -Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Abstract

Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce Q -Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of Q -functions. By analyzing Q -function over-generalization, which impairs stable stitching, QCS adaptively integrates Q -aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and valuebased methods, consistently achieving or exceeding the maximum trajectory returns across diverse offline RL benchmarks. The project page is available at https://beanie00.com/publications/qcs .

1 Introduction

Offline reinforcement learning (RL) is a vital framework for acquiring decision-making skills from fixed datasets, particularly when online interactions are impractical. This is especially relevant in fields such as robotics, autonomous driving, and healthcare, where the costs and risks of real-time experimentation are significant.

A promising approach in offline RL is returnconditioned supervised learning (RCSL) [10, 12, 20]. By framing offline RL as sequence modeling tasks, RCSL allows an agent to leverage past experiences and condition on the target outcome, facilitating the generation of future actions that are likely to achieve desired outcomes. This method builds on recent advancements in super-

Optimal

Sub-optimal

RCSL

QCS Policy Training

Q-Aid

RCSL

Q-Aid

vised learning (SL) [41, 9, 11, 30], and thus benefits from the inherent stability and scalability of SL. However, RCSL is significantly limited by its lack of 'stitching ability', the ability to combine suboptimal trajectory segments to form better overall trajectories [13, 27, 53, 16, 7, 58]. As a result, its effectiveness is restricted to the best trajectories within the dataset.

Conversely, the Q -function possesses the ability to stitch together multiple sub-optimal trajectories, dissecting and reassembling them into an optimal trajectory through dynamic programming. Therefore, to address the weakness of RCSL, prior works have attempted to enhance stitching ability through the Q -function [53, 16]. However, these prior works employ the Q -function as a conditioning factor for RCSL and do not fully leverage the Q -function's stitching ability, resulting in either negligible performance improvements or even reduced performance. The primary challenge lies

Value-Based Methods

RCSL Methods

QCS (Ours)

Combined RCSL-Value Methods

(DT/DC/RvS)

(TD3+BC/IQL/CQL/SQL)

(QDT/EDT/CGDT/ACT/POR)

*Best methods in each group

MujoCo Medium

MujoCo Medium-Replay

MujoCo Medium-Expert

Normalized Return

90

90

110

70

Average

50

60

60

80

30

in the fact that utilizing the Q -function through conditioning, without managing the conditions for stable Q -guided stitching, can result in a sub-optimal algorithm.

In this work, we aim to fully synergize the stable and scalable learning framework of RCSL with the stitching ability of the Q -function. To effectively utilize the Q -function, it is crucial to identify when it can benefit RCSL and to integrate its assistance, termed Q -aid. Our contributions to achieving the effective utilization of Q -aid in RCSL are as follows: (1) We discovered that in-sample Q -learning on an expert dataset, which predominantly consists of optimal actions with similar Q -values within a constrained action range, causes the Q -function to receive improper learning signals and become over-generalized. (2) To prevent errors from this Q -generalization and to incorporate stitching ability within RCSL's stable framework, we propose Q -Aided Conditional Supervised Learning (QCS), which adaptively integrates Q -aid into the RCSL's loss function based on trajectory returns.

Despite its simplicity, the effectiveness of QCS is empirically demonstrated across offline RL benchmarks, showing significant advancements over existing state-of-the-art (SOTA) methods, including both RCSL and value-based methods. Especially, QCS surpasses the maximal dataset trajectory return across diverse MuJoCo datasets, under varying degrees of sub-optimality, as shown in Fig. 2. Furthermore, QCS significantly outperforms the baseline methods in the challenging AntMaze Large environment. This notable achievement underscores the practical effectiveness of QCS in offline RL.

2 Preliminaries

We consider a Markov Decision Process (MDP) [6], described as a tuple M=(S , A , P , ρ 0 , r, γ ) . S is the state space, and A is the action space. P ∶ S ×A ↦ ∆ (S) is the transition dynamics, ρ 0 ∈ ∆ (S) is the initial state distribution , r ∶ S × A ↦ R is the reward function, and γ ∈ [ 0 , 1 ) is a discount factor. The objective of offline RL is to learn a policy π (⋅∣ s ) that maximizes the expected cumulative discounted reward, E a t ∼ π (⋅∣ s t ) ,s t + 1 ∼P(⋅∣ s t ,a t ) [∑ ∞ t = 0 γ t r ( s t , a t )] , using a static dataset D = { τ ( i ) } D i = 1 comprising a set of trajectories τ ( i ) . Each trajectory τ ( i ) consists of transitions over a time horizon T , collected from an unknown behavior policy β .

2.1 Value-Based Offline Reinforcement Learning

Offline RL effectively employs off-policy RL techniques, allowing a divergence between the behavior policy β used for data acquisition and the target policy π being optimized [25, 14, 23]. Off-policy methods primarily utilize the Q -function, which is learned through temporal-difference (TD) bootstrapping. In actor-critic off-policy approaches, both the Q -function ˆ Q θ and the policy ˆ π are updated iteratively. This process can cause a shift in the action distribution, leading ˆ π to select actions that significantly deviate from those in the training dataset. These deviations can inadvertently result in overestimation errors, especially for out-of-distribution (OOD) actions, as offline RL cannot correct incorrect Q -values through interactions with the environment.

Unlike actor-critic methods, in-sample learning methods use only in-sample actions to learn the optimal Q -function, thereby preventing the querying of OOD action Q-values during training [34, 32, 23, 51]. Implicit Q -Learning (IQL) [23] is a representative in-sample learning method. It utilizes expectile regression, defined as L 2 η ( u ) = ∣ η -✶ ( u < 0 )∣ u 2 where η ∈ [ 0 . 5 , 1 ) , to formulate the

asymmetrical loss function for the value network V ψ . Through this loss, V ψ can approximate the implicit maximum of the TD target, max a Q ˆ θ ( s, a ) . Formally, for a parameterized critic Q θ ( s, a ) with a target critic Q ˆ θ ( s, a ) , the value loss function is given by

Intuitively, this loss function suggests placing more emphasis when Q ˆ θ is greater than V ψ ( s ) . Subsequently, the critic network Q θ is updated by treating the learned V ψ ( s ′ ) as max a ′ ∈D( s ′ ) Q ˆ θ ( s ′ , a ′ ) , where D( s ′ ) denotes the in-sample actions for the given state s ′ , i.e., ( s ′ , a ′ ) ∈ D :

We use IQL to pretrain the Q -function used to aid RCSL, as we found that this method, without conservatism during Q -function training, can provide good stitching ability when well integrated. A comparison with a different Q -learning method, CQL [25], is provided in Appendix H.1.

2.2 Return-Conditioned Supervised Learning (RCSL)

RCSL is an emerging approach to addressing challenges in offline RL. It focuses on learning the action distribution conditioned on return-to-go (RTG), defined as the cumulative sum of future rewards ˆ R t = ∑ T t ′ = t r t ′ through supervised learning (SL) [10, 12, 20]. Due to the stability of SL, RCSL is capable of learning decision-making by extracting and mimicking useful information from the dataset. In particular, Decision Transformer (DT) [10] applies the Transformer architecture [41] to reframe the RL as a sequence modeling problem. It constructs input sequences to the Transformer by using sub-trajectories, each spanning K timesteps and comprising RTGs, states, and actions: τ t -K + 1 ∶ t = ( ˆ R t -K + 1 , s t -K + 1 , a t -K + 1 , ..., ˆ R t -1 , s t -1 , a t -1 , ˆ R t , s t ) . The model is then trained to predict the action a t based on τ t -K + 1 ∶ t . Recently, Kim et al. [20] proposed Decision ConvFormer (DC) to simplify the attention module of DT and better model the local dependency in the dataset, yielding performance gains over DT with reduced complexity. These methods have shown effective planning capabilities, but they lack stitching ability, which causes difficulties with datasets that contain many sub-optimal trajectories. This will be discussed in more detail in Section 3.1.

2.3 Neural Tangent Kernel of Q -Function

The Neural Tangent Kernel (NTK) [19] provides insightful analysis of function approximation errors of Q -function, Q θ , especially those related to generalization. The NTK, denoted as k θ ( ¯ s, ¯ a, s, a ) , is defined as the inner product of two gradient vectors, ∇ θ Q θ ( ¯ s, ¯ a ) and ∇ θ Q θ ( s, a ) , i.e., k θ ( ¯ s, ¯ a, s, a ) ∶= ∇ θ Q θ ( ¯ s, ¯ a ) ⊺ ∇ θ Q θ ( s, a ) . The NTK offers a valuable perspective on the impact of parameter updates in function approximation, particularly in gradient descent scenarios. It essentially measures the degree of influence a parameter update for one state-action pair ( s, a ) exerts on another pair ( ¯ s, ¯ a ) . A high value of k θ ( ¯ s, ¯ a, s, a ) implies that a single update in the Q θ for the pair ( s, a ) could lead to substantial changes for the pair ( ¯ s, ¯ a ) . Weguide the readers to Appendix D for a deeper understanding of the NTK.

3 When Is Q -Aid Beneficial for RCSL?

When is it beneficial for RCSL to receive assistance from the Q -function, denoted as Q θ , and how should this assistance be provided? To explore this, we trained two policies, RCSL policy and a maxQ policy that selects the best action according to Q θ , on two different quality datasets from D4RL [13] MuJoCo, comparing their performances in Table 1. Note that the performance is not directly linked to the policy's accuracy across all states; even if the agent accurately predicts actions in several states, errors in some states can lead to path deviations and accumulated errors, resulting in a test-time distribution shift and a lower trajectory return. However, these results can provide insight into when Q -aid might be helpful.

For the RCSL algorithm, we used the Decision Transformer (DT) [10]. To train the maxQ policy, we first trained the Q θ using the in-sample Q -learning method outlined in Eqs. (1) and (2). Then, we extracted the maxQ policy to select the action that directly maximizes Q θ ( s, ⋅) for each state s by using a 3-layer MLP and the loss function L maxQ ( ϕ ) = E s ∼D [-Q θ ( s, maxQ ϕ ( s ))] .

±

±

±

±

±

±

Observing Table 1, we see that the dataset quality favoring RCSL contrasts with that benefiting the maxQ policy. RCSL tends to perform well by mimicking actions in high-return trajectory datasets [29, 31]. However, this method is less effective with datasets predominantly containing suboptimal trajectories, even though RTG conditioning helps predict actions that yield higher returns. On the other hand, the maxQ policy excels with suboptimal datasets but shows notably poor results with optimal datasets. From these observations, a motivating question arises: 'Why does the simple maxQ policy outperforms RCSL on suboptimal datasets yet fails on optimal datasets? If so, how can we effectively combine the two methods to achieve optimal performance?'

3.1 How Can MaxQ Policy Surpass RCSL in Suboptimal Datasets?

We present a toy example demonstrating the limitation of RCSL, as illustrated in Fig. 3. Suppose the dataset is composed of two sub-optimal trajectories. At the initial state s 1 , the agent has two options: the ↑ action connected to trajectory 2 (the orange trajectory) with an RTG of 5, and the → action connected to trajectory 1 (the purple trajectory) with an RTG of 6. RCSL makes the agent choose the → action with a high RTG and follow the path of trajectory 1, which is not optimal. This example demonstrates that RCSL alone is insufficient for the agent to learn to assemble the parts of beneficial sub-trajectories.

S 4

S 6

3

1

1

Optimal Policy

S 2

S 1

Goal

Fail

Start

Trajectory 1 (Succeed)

Trajectory 2 (Failure)

S 3

1

1

4

In contrast, a Q -function can develop stitching ability. Consider the example in Fig. 3 again. We can compute the Q -values for the actions ↑ and → at state s 1 with dynamic programming: Q ( s 1 , ↑) = 3 + max ( Q ( s 2 , →) , Q ( s 2 , ↘)) = 7 , Q ( s 1 , →) = 1 + max ( Q ( s 3 , →) , Q ( s 3 , ↖)) = 6 .

With the Q -values, the agent will select the ↑ action at s 1 and then the → action at s 2 . Consequently, using the Q -function, the agent can select the optimal action that yields the maximum return of 7. Therefore, integrating RCSL with Q -function in situations with abundant sub-optimal trajectories can be beneficial for developing the stitching ability required for optimal decision-making.

3.2 Why Does MaxQ Policy Struggle with Optimal Datasets?

Despite the potential advantages of using Q -values, incorporating values from a learned Q -function, Q θ , to aid RCSL can introduce errors due to inaccuracies in learning. These inaccuracies are particularly significant when Q θ is learned from optimal trajectories. Suppose we have an optimal policy π ∗ . Optimal trajectories are visit logs containing actions performed by π ∗ , yielding the best Q -value for a given state s . Due to the stochasticity of π ∗ , multiple similar actions can be sampled from π ∗ , namely a ∗ 1 , a ∗ 2 , . . . , a ∗ n ( s ) ∼ π ∗ (⋅∣ s ) for a given state s . In this case, we have Q ∗ ( s, a ∗ i ) ≈ Q ∗ ( s, a ∗ j ) and a ∗ i ≈ a ∗ j ∀ i, j ∈ { 1 , 2 , . . . , n ( s )} due to the optimality of these actions. When learning Q θ from such limited information, where the values at the narrow action points are almost identical for each given state, it is observed that the learned Q θ ( s, a ) tends to be over-generalized to the OOD action region. This means that the nearly identical value at the in-sample actions a ∗ 1 , a ∗ 2 , . . . , a ∗ n ( s ) is extrapolated to OOD actions, yielding a nearly flat Q -value over the entire action space for each given state, i.e., Q θ ( s, a OOD ) ≈ Q θ ( s, a ∗ 1 ) with Q θ ( s, a ) being a function of s only. This over-generalization makes Q θ noise-sensitive, potentially assigning high values to incorrect actions and causing state distribution shifts in the test phase, as shown in Fig. 7.

We present a simple experiment to verify that learning Q θ indeed induces over-generalization when trained on optimal trajectories. The experiment consists of an MDP with one-dimensional discrete

(a)

(b)

(c)

(d)

Medium dataset

Qe via Medium dataset (Regression)

Q via Medium dataset (IQL)

100

Environment

5

5

5

s

States

90

-5

0

5

80

-5

-5

True Q

Expertdataset

Q via Expert dataset (Regression)

Qevia Expertdataset (IQL)

5

5

5

5

-70

States

States

States

States

60

-5

5

Actions

Actions

Actions

1

Actions

1

50

states and actions, each divided into 500 bins. This environment simulates a car, where the state indicates the agent's position, ranging from -5 to 5, as illustrated in Fig. 4 (a). The action range is between -1 and 1, allowing the agent to move according to the direction and twice the magnitude of the action. The objective is to reach position 0, which grants a reward of 100, while larger actions incur penalties given as -30 ⋅ a 2 . Due to its discrete nature, we can compute the true optimal Q -values through value iteration [37], which is shown in the bottom row of Fig. 4 (a).

With this environment, we generated two datasets, medium and expert . The medium dataset consisted of actions varying within the range of ±0.5 perturbed from the optimal action determined by the true optimal Q -values, while the expert dataset consisted of actions varying within the range of ±0.05 perturbed from the optimal action. (Refer to Fig. 4 (b)) We then adopted a 3-layer MLP as the structure of Q θ and performed regression to follow the true Q -value at each sample point ( s, a ) in the trajectories. Note that in-sample Q -learning can essentially be regarded as regression with the target value obtained from bootstrapping.

The learned Q θ with the medium and expert datasets are shown in Fig. 4 (c). Indeed, the learned Q θ with the expert dataset, containing nearly-optimal actions, shows that the value is flat over the entire action space for each state. This means that the nearly identical value of in-sample expert actions with a small spread is projected to the entire action space for each state. In contrast, the learned Q θ with the medium dataset well estimates the true Q -function. This is because the medium dataset has diverse actions with diverse values for each state that facilitate the regression process. We additionally present the results from Q θ obtained through IQL in Fig. 4 (d), which shows a similar trend to the results from regression.

O O D   A cti o n s

I n - S a m

pl e  A cti o n s

-. 

-. 

. 

. 

-. 

-. 

. 

. 

. 





O O D   A cti o n s

I n - S a m

pl e  A cti o n s

-. 

-. 

. 

. 

-. 

-. 

. 

. 

. 





200

50

400

250

(a)

Inverted Double

Pendulum Medium

(b)

Inverted Double

Pendulum Expert

(c) Hopper Medium-Replay

(d) Hopper Expert

In-Sample Actions

Qe(s,a)

In-Sample Actions

Qe(s,a)

Qo(s,a)

Qo(s,a)

Action dim

Action dim

Action dim

Action dim

Action dim 2

Action dim 2

Action dim 2

Action dim 2

e wp uo

Action dim 3

Action dim 3

Action dim 3

kg(8,a,s,aret)

ko（s,a,s,aret)

ko（s,a,s,aref)

VQe(s,aret)12

>0.9

>0.9

VQe(s,aref)2

VQe(s, aret)12

aref

aref

★

Action dim

★

Action dim

Action dim 2

Action dim 2

Action dim 3

Action dim 3

The over-generalization tendency in Q θ with optimal trajectories is not limited to the simple experiment above but also applies to complex RL tasks. We analyze how Q θ ( s, ⋅) varies over the action space in the Gym Inverted Double Pendulum [8] and MuJoCo Hopper environments [39, 8] trained on expert and medium -quality datasets with IQL. The details of the analysis are in Appendix E. As depicted in the upper row of Fig. 5 (a) and (b), and on the left side of the upper row of (c) and (d), the expert dataset shows concentrated action distribution, while the medium dataset has a broader spread, as expected. The concentration and similarity of true Q -values of the actions for a given state in the expert dataset cause over-generalization in Q θ ( s, ⋅) , yielding constant flat values across the entire action space. This is further supported by the results in Appendix E.2, which visualize the weights of the learned Q -function.

For a deeper understanding of the over-generalization in Q θ , we analyze the gradient similarity, captured as the Neural Tangent Kernel (NTK), between an arbitrary action ¯ a and the reference action a ref for a given state s . A higher NTK value signifies that the gradient update of Q θ ( s, a ref ) has a substantial impact on Q θ ( s, ¯ a ) . This indicates that even when a ref and ¯ a are dissimilar or unrelated actions, a high NTK value suggests that the Q θ network is misjudging the relationship between these actions and over-generalizing. In Fig. 5, Q θ trained with the expert dataset shows uniformly high normalized NTK values across actions, indicating that the gradient at one action equally affects all others. In contrast, Q θ trained with the medium dataset shows NTK values that are higher near the reference action and decrease with action distance, reflecting more precise generalization. This analysis reveals that datasets consisting of optimal trajectories exhibit more aggressive generalization, which can negatively impact the accuracy of the learned Q -function in offline RL.

4 Q -Aided Conditional Supervised Learning

According to Section 3, RCSL faces challenges with suboptimal datasets, whereas Q θ can effectively serve as a critic for stitching ability, favoring the use of Q -aid. In contrast, in an optimal dataset, Q θ tends to over-generalize, leading to inaccuracies of learned Q θ , while RCSL excels by mimicking the optimal behavior without requiring external assistance. Recognizing this dynamic, it is crucial to integrate Q -aid into RCSL adaptively. The following subsections explain how to effectively adjust the level of Q -aid and facilitate the integration of the two methods, leading to the proposal of Q -Aided Conditional Supervised Learning (QCS).

4.1 Controlling Q -Aid Based on Trajectory Returns

Given the complementary relationship, how can we adjust the degree of Q -aid? Since RCSL's preference for mimicking datasets and the Q -function's over-generalization issue is tied to trajectory optimality, we can apply varying degrees of Q -aid based on the trajectory return for each subtrajectory in RCSL. Therefore, we set the degree of Q -aid, denoted as the QCS weight w ( R ( τ )) for a trajectory τ , as a continuous, monotone-decreasing function of the return of τ , R ( τ ) , such that

where continuity is imposed for gradual impact change. Among various choices, we find that simple options such as linear decay are sufficient to produce good results, i.e., w ( R ( τ )) = λ ⋅ ( R ∗ -R ( τ )) with some λ > 0 , where R ∗ represents the optimal return of the task. Practically, R ∗ can be obtained from an expert dataset or from the maximum value in the dataset. For details on how to calculate R ∗ , please refer to Appendix F. Note that R ( τ ) differs from RTG ˆ R t which is the sum of future rewards after timestep t and decreases as timestep t goes, thereby failing to represent the trajectory's optimality accurately.

4.2 Integrating Q -Aid into the RCSL Loss Function

Instead of using the Q -function as the conditioning factor for RCSL as in previous works, we propose a more explicit approach by integrating Q -assistance into the loss function and dynamically adjusting the degree of assistance based on Section 4.1. As a result, the overall policy loss is given as follows:

where Q IQL θ (⋅ , ⋅) denotes the fixed Q -function pretrained with IQL. R ( τ ) is the return of the entire trajectory τ containing the sub-trajectory τ t ∶ t + K -1 . The overall input to the policy at time t is the subtrajectory of context length K starting from time t , τ t ∶ t + K -1 = ( ˆ R t , s t , a t , . . . , ˆ R t + K -1 , s t + K -1 ) ⊂ τ.

Our new loss function enables adaptive learning strategies depending on the trajectory's quality to which the subtrajectory belongs. For optimal trajectories, action selection follows RCSL. On the other hand, for suboptimal trajectories τ with R ( τ ) < R ∗ , the Q -aid term kicks in and its impact increases as R ( τ ) decreases. We describe the details of the QCS weight w ( R ( τ )) and the policy update with the loss function in Appendix J.2 and our full algorithm's pseudocode in Appendix A.

4.3 Implementation

Base Architecture. For implementing π ϕ , a general RCSL policy can be used. When K = 1 , meaning only the current time step is considered to estimate the action, we use an MLP network. When K ≥ 2 , we use a history-based policy network, such as DT [10] or DC [20].

Conditioning. We consider two conditioning approaches as proposed by RvS [12]: one for tasks maximizing returns and the other for tasks aiming at reaching specific goals. For return-maximizing tasks, we employ RTG conditioning, and our algorithm is named QCS-R. For goal-reaching tasks, we additionally use subgoal conditioning, and our algorithm is named QCS-G. For subgoal selection, we randomly select a state that the agent will visit in the future. The ablations on conditioning are in Appendix H.2.

5 Related Work

Prompting RCSL with Dynamic Programming. Recent studies have recognized the limitations of RCSL in stitching abilities [27, 7, 58]. Our work contributes to the ongoing efforts to imbue RCSL with this capability. Notably, Q -learning Decision Transformer (QDT) [53] and Advantage Conditioned Transformer (ACT) [16] have proposed integrating dynamic programming into RCSL by modifying the RTG prompt to Q -value or advantage prompt. Our approach, QCS, parallels these efforts by leveraging dynamic programming for action guidance and trajectory stitching. However, unlike these methods, which implicitly incorporate dynamic programming through conditioning, QCS explicitly augments its loss function with the learned Q -function.

Incorporating RCSL with Stitching Ability. In a distinct vein, recently proposed Critic-Guided Decision Transformer (CGDT) [43] identifies the gap between target RTG and expected returns of actions as key to RCSL's limited stitching. To mitigate this, it adjusts DT's output with the critic network's Monte-Carlo return predictions and target RTG. In contrast, QCS uses Q -values learned through dynamic programming to guide actions, enhancing stitching ability explicitly. Another approach, the Elastic Decision Transformer (EDT) [49], recommends variable context lengths during inference, using longer contexts for optimal trajectories and shorter ones for sub-optimal trajectories to identify optimal paths better. QCS similarly adapts based on trajectory optimality but differentiates itself by modifying its learning approach during training, leveraging the complementary strengths of the Q -function and RCSL.

Furthermore, POR [50] integrates imitation learning techniques with stitching ability by generating high-value states using additional networks and value functions. These states are then used as conditions for predicting actions. Unlike QCS, which focuses on action stitching, POR emphasizes state stitching, allowing agents to choose actions that lead to high-value states, albeit with the need for additional networks. By concentrating on action stitching, QCS can avoid the computational demands associated with high-dimensional state prediction.

State-Adaptive Balance Coefficient Regarding the sub-trajectory-adaptive weight used in QCS, FamO2O [42] employs state-adaptive weight coefficients to balance policy improvement and constraints in the offline-to-online RL framework. Although FamO2O is an offline-to-online method that incorporates additional online samples, we provide a performance comparison with this work in Appendix G.2 to further demonstrate the effectiveness of QCS.

6 Experiments

In the experiment section, we conduct various experiments across different RL benchmarks to answer the following questions:

How well does QCS perform in decision-making compared to prior SOTA methods across datasets of varying quality and tasks with diverse characteristics, especially those requiring stitching ability?

To what extent does the dynamic nature of QCS weights, informed by trajectory return, contribute to effective decision-making, and how robust are these dynamic weights to hyperparameters?

Can QCS effectively acquire stitching ability while preventing test-time distribution shift?

6.1 Experimental Setup

Baseline Methods. To address a range of questions, we conduct a comprehensive benchmarking against 12 representative baselines that are state-of-the-art in each category. For the value-based category, we assess 4 methods: TD3+BC [14], IQL [23], and CQL [25], SQL [52]. For RCSL, we assess 3 methods: DT [10], DC [20], RvS [12]. Additionally, we evaluate 5 advanced RCSL methods proposed to integrate stitching capabilities: QDT [53], EDT [49], CGDT [43], ACT [16], and POR [50]. For more details on the setup and the baselines, refer to Appendix B.

Benchmarks. Weevaluated QCS against various baselines using datasets with diverse characteristics, including tasks focused on return maximization or goal-reaching, and those with dense or sparse rewards and varying sub-optimality levels.

(a) Halfcheetah

(b) Hopper

(c) Walker2d

(d)

AntMaze

Umaze

(e)

AntMaze

Medium

(f)

AntMaze

Large

(g)

Adroit

Pen

Our primary focus was on the D4RL [13] MuJoCo, AntMaze, and Adroit domains. The MuJoCo domain [39, 8] features several continuous locomotion tasks with dense rewards. We conducted experiments in three environments: Halfcheetah, Hopper, and Walker2d, utilizing three distinct v2 datasetsmedium , medium-replay , and medium-expert -each representing different levels of data quality. AntMaze is a domain featuring goal-reaching environments with sparse rewards, encompassing variously sized and shaped maps. It is an ideal testing bed for evaluating an agent's capability to stitch trajectories and perform long-range planning. We conduct experiments using six v2 datasets: umaze , umaze-diverse , medium-play , medium-diverse , large-play , and large-diverse , where umaze , medium , and large indicate map sizes, and play and diverse refer to data collection strategies. The Adroit domain [35] comprises various tasks designed to evaluate the effectiveness of algorithms in high-dimensional robotic manipulation tasks. In our experiments, we utilize the human and cloned datasets for the pen task.

Performance results for the MuJoCo and AntMaze domains are presented in Table 2 and Table 3, while results for the Adroit domain are included in Appendix G.

Hyperparameters and Backbone Architecture. We adopted two sets of hyperparameters per domain to determine the gradient of the monotonic decreasing function w ( R ( τ )) . The detailed hyperparameters we used are provided in Appendix J, and the impact of λ is detailed in Appendix 6.3. Additionally, we implemented QCS based on DT, DC, and a simple MLP, and compared the performance of each. Detailed results for each architectural choice are provided in Appendix H.2. We observed that the DC-based approach performs best, although the performance gap is minor.

Evaluation Metric. In all evaluations of QCS, we assess the expert-normalized returns [13] of 10 episodes at each evaluation checkpoint (every 10 3 gradient steps). Subsequently, we compute the running average of these normalized returns over ten consecutive checkpoints. We report the mean and standard deviations of the final scores across five random seeds.

6.2 Overall Performance

As shown in Table 2 and Table 3, QCS significantly outperforms prior value-based methods, RCSL, and combined methods across the datasets. Specifically, QCS outperforms both IQL and DC, upon which it is based, across all datasets, unlike the contradictory results between RCSL and the maxQ policy shown in Table 1. This empirically confirms that QCS successfully combines the strengths of both RCSL and the Q -function. A particularly remarkable achievement of QCS is its ability to substantially improve efficiency in goal-reaching tasks, AntMaze, especially in Large environments, where prior RCSL methods exhibited notably low performance. This enhancement is largely attributed to the stitching ability introduced by the Q -aid of QCS. These results underscore QCS's robustness and superiority in a wide array of offline RL contexts. The training curves for Tables 2 and 3 are shown in Appendix I, demonstrating stable learning curves across all datasets.

6.3 Ablation Studies

To further analyze how each design element influences performance, we conducted additional experiments. More ablation studies are detailed in Appendix H, including the use of Q -function trained by CQL and the impact of base architecture and conditioning.

The Importance of Weights Relative to Trajectory Return.

To assess the impact of dynamically setting the QCS weight w ( R ( τ )) based on trajectory return, we compare our approach with a constant QCS weight, w ( R ( τ )) = c . We test five constant weights c ∈ { 1 , 2 . 5 , 5 , 7 . 5 , 10 } and report the maximum score among these values in Table 4. The QCS method with the dynamic weight based on trajectory return outperforms the highest scores obtained with various constant weight settings across datasets, as shown in Table 4.

This demonstrates that our dynamic weight control, grounded in trajectory return, is more effective and robust in integrating Q -aids.

Impact of the QCS weight λ . We examined the effect of λ by varying it from 0.2 to 1.5. As shown in Table 5, except for the walker2d-medium, we found that even the smallest values achieved with changing λ either matched or surpassed the performance of existing value-based methods and RCSL's representative methods, including IQL, CQL, DT, DC, and RvS. This demonstrates QCS's relative robustness regarding hyperparameters. For walker2d-medium, we found that performance begins to decrease when λ exceeds the initial setting of 0.5. While increasing the gradient steps from 500K to 1M improves performance at λ = 1 , further increasing λ to 1.5 leads to greater instability.

±

Test Time State Distribution Shift.

To validate whether QCS effectively acquires stitching ability while preventing a shift in the test-time state distribution, as discussed in Section 3.2, we present Fig. 7. This figure compares the state distributions explored by RCSL, maxQ , and QCS policies during evaluation. RCSL and maxQ , representing QCS's extremes, were trained using specific loss configurations: RCSL loss as QCS loss in Eq. 3 with λ = 0 and maxQ loss as QCS loss without the RCSL term, i.e., selecting actions as argmax a ∈A Q IQL θ ( s, a ) . Fig. 7 illustrates RCSL's adherence to dataset states, contrasting with the notable state distribution shift of the maxQ policy. QCS inherits RCSL's

±

±

±

Dataset State

Test Time Visit State

(a) RCSL

(b) max-

Q

(c) QCS (Ours)

stability but surpasses its performance, indicating an effective blend of transition recombination without straying excessively from the state distribution.

7 Conclusion

In conclusion, QCS effectively combines the stability of RCSL with the stitching ability of the Q -function. Anchored by thorough observation of Q -function generalization error, QCS adeptly modulates the extent of Q -assistance. This strategic fusion enables QCS to exceed the performance of existing SOTA methods in both efficacy and stability, particularly in complex offline RL benchmarks encompassing a wide range of optimality.

In addressing our initial motivating question on integrating RCSL and Q -function, QCS opens up promising future research directions. While we have established a correlation between trajectory return and the mixing weight, we have considered simple linear weights to control the level of Q -aid. It is also plausible that the mixing weight might be influenced by other dataset characteristics, such as the dimensions of the state and actions. We believe QCS will stand as a motivating work, inspiring new advancements in the field.

Acknowledgments

This work was supported in part by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00469, Development of Core Technologies for Task-oriented Reinforcement Learning for Commercialization of Autonomous Drones, 50%) and in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2021R1A2C2009143 Information Theory-Based Reinforcement Learning for Generalized Environments, 50%).

References

J. Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894 , 2019.

A. F. Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

G. An, S. Moon, J.-H. Kim, and H. O. Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in Neural Information Processing Systems , 34:74367447, 2021.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning , pages 1577-1594. PMLR, 2023.

R. Bellman. A markovian decision process. Journal of Mathematics and Mechanics , pages 679-684, 1957.

D. Brandfonbrener, A. Bietti, J. Buckman, R. Laroche, and J. Bruna. When does returnconditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems , 35:1542-1553, 2022.

G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems , 33:1877-1901, 2020.

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in Neural Information Processing Systems , 34:15084-15097, 2021.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=S874XAIpkR-.

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems , 34:20132-20145, 2021.

S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052-2062. PMLR, 2019.

C. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, and Y. Yu. Act: Empowering decision transformer with dynamic programming via advantage conditioning. arXiv preprint arXiv:2309.05915 , 2023.

A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning , pages 1025-1037. PMLR, 2020.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems , 31, 2018.

J. Kim, S. Lee, W. Kim, and Y. Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. In International Conference on Learning Representations , 2024.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

I. Kostrikov, R. Fergus, J. Tompson, and O. Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2021.

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

A. Kumar, R. Agarwal, T. Ma, A. Courville, G. Tucker, and S. Levine. Dr3: Value-based deep reinforcement learning requires explicit regularization. In International Conference on Learning Representations , 2021.

A. Kumar, J. Hong, A. Singh, and S. Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05618 , 2022.

J. Lee, L. Xiao, S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in Neural Information Processing Systems , 32, 2019.

S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10012-10022, 2021.

I. Mediratta, Q. You, M. Jiang, and R. Raileanu. A study of generalization in offline reinforcement learning. In NeurIPS 2023 Workshop on Generalization in Planning , 2023.

A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 32, 2019.

X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems XIV , 2018.

H. Singh, N. Misra, V. Hnizdo, A. Fedorowicz, and E. Demchuk. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences , 2003.

R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

R. S. Sutton, A. G. Barto, et al. Introduction to reinforcement learning , volume 135. MIT press Cambridge, 1998.

E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012.

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017.

S. Wang, Q. Yang, J. Gao, M. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Y. Wang, C. Yang, Y. Wen, Y. Liu, and Y. Qiao. Critic-guided decision transformer for offline reinforcement learning. arXiv preprint arXiv:2312.13716 , 2023.

Z. Wang, A. Novikov, K. Zolna, J. S. Merel, J. T. Springenberg, S. E. Reed, B. Shahriari, N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems , 33:7768-7778, 2020.

Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2022.

J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:31278-31291, 2022.

Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Y. Wu, S. Zhai, N. Srivastava, J. Susskind, J. Zhang, R. Salakhutdinov, and H. Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Y.-H. Wu, X. Wang, and M. Hamaya. Elastic decision transformer. In Advances in Neural Information Processing Systems , 2023.

H. Xu, L. Jiang, L. Jianxiong, and X. Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:4085-4098, 2022.

H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations , 2022.

H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810 , 2023.

T. Yamagata, A. Khalil, and R. Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning , pages 38989-39007. PMLR, 2023.

T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Modelbased offline policy optimization. Advances in Neural Information Processing Systems , 33: 14129-14142, 2020.

T. Yu, A. Kumar, R. Rafailov, A. Rajeswaran, S. Levine, and C. Finn. Combo: Conservative offline model-based policy optimization. Advances in Neural Information Processing Systems , 34:28954-28967, 2021.

Y. Yue, R. Lu, B. Kang, S. Song, and G. Huang. Understanding, predicting and better resolving q-value divergence in offline-rl. In Advances in Neural Information Processing Systems , 2023.

Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In International Conference on Machine Learning , pages 27042-27059. PMLR, 2022.

Z. Zhou, C. Zhu, R. Zhou, Q. Cui, A. Gupta, and S. S. Du. Free from bellman completeness: Trajectory stitching via model-based return-conditioned supervised learning. In International Conference on Learning Representations , 2024.

Appendices

A Pseudocode

The QCS algorithm first learns the Q -function using a dynamic programming and then trains the policy based on the aid of the learned Q -function. Detailed pseudocode can be found in Algorithm 1. In this work, we utilized IQL [23] as the Q training algorithm, but other Q training algorithms can be employed. A comparison with using a Q -function trained by CQL [25] can be found in Appendix H.1.

Algorithm 1 IQL-aided QCS

B Baseline Details

We evaluated the performance of QCS against twelve different baseline methods. This group consists of four value-based methods: TD3+BC [14], IQL [23], CQL [25] and SQL [52]; three RCSL algorithms: DT [10], DC [20], and RvS [12]; and five combined methods that signify progress in RCSL by integrating stitching abilities: QDT [53], EDT [49], and CGDT [43], ACT [16], and POR [50]. The performance for these baselines was sourced from their respective original publications, with two exceptions. For CQL [25], the performance data in the original paper was based on the MuJoCo v0 environment, which differs from the v2 version used in our study. Therefore, for CQL, we referenced the performance score reported in [23] to ensure a consistent and fair comparison across all methods.

In addition, for antmaze-medium and antmaze-large , since there were no reported DT [10] and DC [20] scores, we conducted evaluations using the official codebase. When training on antmaze-medium and large , we used 512 as the embedding dimension in the default hyperparameter setting. We found that removing the positional embedding slightly improved performance, as also discussed in Zheng et al. [57], so we trained without it. For the target RTG, we used values of 1 and 100 and reported the higher score obtained from the two values.

C Dataset Return Distributions

To gain a deeper understanding of the scenarios in which offline RL is applied and the necessity of learning good policies, we plotted the trajectory return distributions for three different datasets in each of the three MuJoCo environments in Fig. 8. For these return distribution histograms, we set the number of bins to 50. The 'Count' label denotes the number of trajectories corresponding to each normalized return.

QCS

HalfcheetahMedium

250

200

Count

150

100

50

0

0

20

40

60

80

100

120

Normalized Return

HalfcheetahMedium-Replay

15

1

Count

10

5

0

20

40

60

80

100

120

Normalized Return

HalfcheetahMedium-Expert

自

400

Count

300

200

100

0

0

20

40

60

80

100

120

Normalized Return

HopperMedium

250

:1

200

Count

150

100

50

:!

0

0

20

40

60

80

100

120

Normalized Return

HopperMedium-Replay

OTTNTOTTn

300

Count

200

100

0

0

20

40

60

80

100

120

Normalized Return

HopperMedium-Expert

600

500

Count

400

300

200

100

0

0

20

40

60

80

100

120

Normalized Return

Walker2dMedium

150

Count

100

1

一

50

0

0

20

40

60

80

100

120

Normalized Return

Walker2dMedium-Replay

150

!:

125

Count

100

75

50

1:

25

0

0

20

40

60

80

100

120

Normalized Return

Walker2d Medium-Expert

600

Count

400

200

Q

0

20

40

60

80

100

120

Normalized Return

As shown in Fig. 8, the medium-replay datasets encompass wide varieties of returns. Additionally, the medium-expert dataset, a combination of the medium and expert datasets, exhibits two peaks, indicating a division in the range of returns. This observation reveals that each dataset exhibits a distinct distribution pattern of returns. In this graph, alongside the return distribution, the dataset's maximum trajectory return and the score of our QCS method are also presented. QCS is observed to achieve results that are close to or surpass the maximum return. This is particularly notable in datasets with diverse return distribution characteristics, such as the medium-replay dataset, where the distribution of low return trajectories is prevalent.

D Brief Derivation of the Neural Tangent Kernel

To understand the influence of parameter updates in function approximation across different stateaction pairs, the Neural Tangent Kernel (NTK) emerges as a crucial tool [19, 28, 1, 26, 56]. For comprehensive insights, we direct readers to Achiam et al. [1], while here we distill the essential concepts. The NTK framework becomes particularly relevant in the context of deep Q -learning, where the parameterized Q -function, denoted as Q θ , is updated as follows [37]:

where α is the learning rate, ρ is the distribution of transitions in the dataset, and δ θ ( s, a ) = r ( s, a ) + γ max a ′ Q θ ( s ′ , a ′ )-Q θ ( s, a ) is the temporal difference (TD) error. On the other hand, the first-order Taylor expansion around θ at an out-of-sample pair ( ¯ s, ¯ a ) yields

Substituting (4) into (5), we have

where k θ ( ¯ s, ¯ a, s, a ) , referred to as the NTK, is defined as the inner product between two gradient vectors ∇ θ Q θ ( ¯ s, ¯ a ) and ∇ θ Q θ ( s, a ) , i.e.,

(6) together with (7) explains how the parameter update with function approximation for a sample pair ( s, a ) affects the Q -value change for another sample pair ( ¯ s, ¯ a ) . When the NTK k θ ( ¯ s, ¯ a, s, a ) is large, the TD-error δ θ ( s, a ) has a more pronounced impact on the update difference Q θ ′ ( ¯ s, ¯ a ) -Q θ ( ¯ s, ¯ a ) . Thus, the single update based on the TD-error at a sample pair ( s, a ) can induce a significant change in the Q -function for another pair ( ¯ s, ¯ a ) .

E Details and Extended Analysis of Q -Function and NTK

In Section 3.2, we conduct an NTK analysis of the Q -function trained with IQL in the Inverted Double Pendulum and Hopper environments, which have state dimensions of 2 and 11, and action dimensions of 1 and 3, respectively. This section details the analysis methods, provides extended results, and offers further clarity by presenting the action distributions for the datasets of each environment.

E.1 Analysis Methods

Inverted Double Pendulum. We chose the Inverted Double Pendulum for analysis of Q -values and NTK due to its one-dimensional action space. For training the Q -function, as no prior open-source offline dataset existed for this environment, we first created one. The dataset was generated by training an online Soft Actor-Critic [18], using an implementation in RLkit, available at https://github.com/rail-berkeley/rlkit.git .

We created two datasets: expert and medium . The expert dataset consists of 10 5 samples generated by an optimal policy, while the medium dataset includes 10 5 samples from a medium policy, whose performance is approximately one-third of the optimal policy. Given the continuous nature of the state and action spaces in the Inverted Double Pendulum, which complicates analysis, we initially quantized both spaces. For state quantization, we set the range from a minimum of -5 to a maximum of 10 (the minimum and maximum values across all dimensions in both datasets) and divided each dimension into 80 equal segments. For action quantization, the range was set from -1 to 1, divided into 500 equal segments.

When plotting the Q -values, we calculated the Q -value for each quantized state across all quantized actions. Fig. 5 shows the results for the state chosen in each dataset, based on the highest count of in-sample actions. In the NTK analysis, we computed the following equation for the reference action and the remaining quantized actions with index i ∈ 1 , . . . , 500 , where a 1 = -1 , a 500 = 1 , and a ref = 0 .

MuJoCo Hopper. For Hopper environment, we use open-source D4RL [13] hopper-expert and hopper-medium-replay datasets. In the case of Hopper, similar to what was done in the Inverted Double Pendulum environment, we quantized the continuous state and action space for analysis. More specifically, for the state space, we divided the values of each dimension into 100 equal segments, ranging from -10 to 10. As for the action space, we divided the values of each dimension into 50 equal segments, ranging from -1 to 1. In the case of Hopper, with its 3D action dimension, visualizing it similarly to the 1D action dimension in the Inverted Double Pendulum posed a challenge. Consequently, in the 3D plots, we assigned each axis to one of the action dimensions and utilized color to indicate the Q -value, as shown in Fig. 5. Additionally, in NTK analysis, representing the relationships between the reference action and all quantized actions within a single graph is challenging. We marked high-NTK regions in gray, where the normalized NTK values are greater than 0.9.

E.2 Q -Network Weights Visualization

Figure 9, illustrating the neural network's learned weights for actions in two distinct datasets, provides a compelling visual representation of the over-generalization results presented in Figure 5. Specifically, the figure displays the first layer's weight matrix W 1 from a two-layer MLP Q -network trained on the Inverted Double Pendulum using Implicit Q -Learning (IQL). This network is defined as Q ( s, a ) = W 2 ReLU ( W 1 ( s, a ) + b 1 ) + b 2 , where W 1 is a key focus due to its direct interaction with the concatenated state and action inputs. The dimensions of the weight matrix W 1 are 32 ×( dim (S) + dim (A)) , where dim (S) = 11 and dim (A) = 1 represent the dimensions of the state and action spaces, respectively. The figure contrasts the learned weights in datasets with different action spreads and the diversity of the Q -values: a narrow spread (expert dataset) and a wider spread (medium dataset).

(a)

-50

20

W1

Wi[,-1]

StateDimActionDim

Action Dim

InvertedDoublePendulumExpert

(b)

-50

20

W1

W[:,-1]

StateDimActionDim

Action Dim

InvertedDoublePendulumMedium

For the expert dataset (Fig. 9(a)), the action-related elements of W 1 (the last column elements) are learned as zero. This intriguing result indicates that the network, during its training, learns not to differentiate between actions across all states, leading to uniformly flat Q -values for all actions. Such behavior is characteristic of datasets with a narrower action spread, where the actions are more clustered and coherent. The network's tendency to not distinguish between different actions in such a dataset is a direct consequence of the limited diversity, requiring less differentiation in the action representation.

In contrast, for the medium dataset (Fig. 9(b)), which represents a wider action spread, the actionrelated elements of W 1 show variation. This variation signifies that the network has learned to differentiate between actions to a greater extent, a necessity in a dataset where actions are more diverse and dispersed. The network's capacity to distinguish between various actions and assign different levels of importance to each reflects the need for a more nuanced understanding of the action space in datasets with a wider spread.

This visual evidence from the learned weights substantiates our understanding of how neural networks adapt their learning based on the diversity in the action space of the dataset. In datasets with narrower action spreads with similar Q -values, the network learns a more uniform approach towards different actions, while in those with wider spreads with diverse Q -values, it adopts a more differentiated and discerning strategy. This adaptive learning aligns with the principles of regression demonstrating the network's response to the diversity and distribution of actions in the training data.

E.3 Extended NTK Visualization

1

0.3

1

0.3

IIVQ(s,a)I12

IIVQ(s,a)I12

-1.0

-1.0

10

-0.5

10

-0.5

0

a0.0

10

a0.0

10

0.5

10

0.5

1.0

10

1.0

-1.0-0.50.0

0.51.0-1.0-0.50.0

0.51.0-1.0-0.50.0

0.51.0

0S000S0-0-0TS000S0-0-0S000S0-0

a

a

a

a=aref

a=aref

a=aref

a=aref

a=aref

a=aref

1.0

1.0

0.5

1.0-0.5

0.0

0.5

1.0-1.0-0.50.0

0.5

1.0-1.0-0.50

0.0

0.5

1.0

0.5

-1.0-0.5

0.0

0.51.0-1.0-0.50.0

0.51.0-1.0-0.5

0.0

0.51.0

a

a

In v e r t ed D

oubl e  Pe n d ul um M edi um

In v e r t ed D

oubl e  Pe n d ul um E x pe r t

In Fig. 5 (a) and (b) in Section 3.2, we visualize the NTK for a fixed state and a reference action a ref at zero in the Inverted Double Pendulum environment. Here, we present the extended results with three different fixed states and a varying reference action across the action space a ref ∈ [-1 . 0 , 1 . 0 ] . In Fig. 10, the diagonal symmetry of the normalized NTK as a function of action distance is observed. Generally, in the medium dataset, the NTK is high between two actions that are close to each other and low between actions that are far apart. However, in the expert dataset, this distinction becomes blurred, regardless of the proximity of the actions.

E.4 Action Distributions

InvertedDoublePendulumExpert

InvertedDoublePendulumMedium

1e6

1e6

m

3

Count

Count

2

1

1

0

0

Distance

Distance

HopperExpert

Hopper Medium

Hopper Medium-Replay

1.0le7

1.01e7

1.0le7

Count

Count

Count

0.5

0.5

0.5

0.0

0

2

0.0

2

0.0

2

Distance

Distance

Distance

In this subsection, we examine the action distributions of datasets within the Inverted Double Pendulum and MuJoCo Hopper environments. The continuous nature and multi-dimensionality of these action spaces pose significant challenges for directly visualizing the exact action distributions. To address this, we define the average action distribution spread of a dataset D , quantified as the expectation over the states of the dataset as:

We then visualize the distribution of the L2 distance among all actions within each quantized state across the dataset using Eq. (8). This approach is based on the characteristics of the action dimensions in both the Inverted Double Pendulum and the Hopper, which are bounded between -1 and 1. For action distribution visualization, we maintain the same state and action quantization as outlined in Appendix E.1.

Fig. 11 presents the results. As depicted in this figure, in both the Inverted Double Pendulum and Hopper environments, the expert datasets exhibit a small average L2 distance between actions coexisting in the same quantized state. This indicates a denser clustering of actions within these datasets, which is linked to high-return datasets typically exhibiting a more concentrated action distribution on average since they primarily perform exploitation actions instead of exploration actions.

F Calculating R ∗

In Section 4.1, we define the QCS weight w ( R ( τ )) as λ ⋅ ( R ∗ -R ( τ )) , where R ∗ is the optimal return for the task. For calculating R ∗ , we consider the two methods described below.

(i) Set R ∗ with the optimal return for the environment. In our experiments, we set R ∗ for the environments with optimal returns as follows: Hopper ( R ∗ = 3500 ), Walker2d ( R ∗ = 5000 ), Halfcheetah ( R ∗ = 11000 ), and AntMaze ( R ∗ = 1 ). Note that prior RCSL algorithms such as Decision Transformer [10] and RvS [12] used predefined R ∗ for target RTG conditioning

during inference. Therefore, using R ∗ based on the optimal return introduces no additional assumptions compared to previous RCSL methods. As noted in Appendix J.2, QCS does not use R ∗ for target RTG conditioning but instead relies on the maximum trajectory return, requiring only one R ∗ per algorithm.

(ii) Set R ∗ to the maximum trajectory return within the dataset. An alternative approach for setting R ∗ is to use the maximum trajectory return from the dataset. When obtaining the true optimal return from the environment is challenging, the maximum trajectory return can serve as an approximation. Table 6 presents additional results using this method for R ∗ .

±

±

As shown in Table 6, setting R ∗ with the optimal environment return is slightly better than setting it with the maximum dataset return, but setting it with the maximum dataset return still outperforms the baselines. Therefore, we propose using the optimal environment return for R ∗ ; however, when it is hard to determine, using the maximum dataset return can be a good alternative.

G More Experiments Results

G.1 Additional Performance Comparison

In addition to the performance comparison in the MuJoCo and AntMaze domains, as discussed in Section 6.2, we also compare the performance of QCS in the Adroit domain using extensive baselines, similar to those mentioned in Section 6.2. Since there are no reported results for TD3+BC [14], SQL [52], RvS [12], QDT [53], EDT [49], CGDT [43], ACT [16], POR [50] in the Adroit domain, we only compare with the value-based baselines (IQL [23], CQL [25]) and RCSL baselines (DT [10], DC [20], RvS [12]). For DT and DC, we evaluate the score using their official codebase.

Table 7 displays the performance of QCS alongside the baseline performances. As indicated by the results, QCS-R outperforms other baselines in the Adroit Pen task. This outcome reiterates that QCS is a robust framework, excelling in a range of tasks with varying features. It also underscores findings from our earlier experiments, which demonstrate that strategically blending RCSL with Q -function can significantly enhance performance.

G.2 Comparison with FamO2O

FamO2O [42] is an offline-to-online RL method that facilitates state-adaptive balancing between policy improvement and constraints. During offline pre-training, it develops a set of policies with various balance coefficients. In the subsequent online fine-tuning phase, FamO2O determines the most suitable policy for each state by selecting the corresponding balance coefficient from this set. The major difference between QCS and FamO2O is that FamO2O additionally uses 10 6 online samples to find a suitable balance coefficient, while QCS only utilizes the offline dataset and adjusts the balance coefficient (QCS weight) based on the trajectory return. Moreover, unlike FamO2O,

which utilizes a state-adaptive balance coefficient, QCS is based on historical architecture and uses a sub-trajectory-adaptive balance coefficient. Although it is not a fair comparison between FamO2O, an offline-to-online algorithm, and QCS, a purely offline algorithm, we present the performance comparison to demonstrate the effectiveness of QCS even when compared with an offline-to-online algorithm.

H More Ablation Studies

H.1 Comparing Assistance from Actor-Critic Learned Q -Values

To compare the performance of QCS using Q -function learned through actor-critic algorithms, we use representative actor-critic algorithms such as CQL [25] for benchmarking. As shown in Table 10, the performance of CQL-aided QCS generally improved compared to the original CQL, but it does not match the performance of IQL-aided QCS for the MuJoCo domain. The reason can be attributed to two factors: (1) the Q -function may be bounded by the actor's representation ability, and (2) CQL might impose excessive conservatism on the Q -function. Moreover, in the case of antmaze-umaze-diverse , IQL-aided QCS underperforms CQL, but CQL-aided QCS outperforms CQL. Since QCS is a general framework that proposes a new combination of RCSL and Q -function on trajectory return, there is a wide range of potential integrations of RCSL and offline Q -learning methods. The most impactful aspect will differ depending on the characteristics of each RCSL and Q -learning method when combined. Exploring this would be an interesting research area, which we leave as future work.

H.2 Impact of Base Architecture and Conditioning

In Section 4.3, we discussed QCS variants with different base architectures and conditioning. To assess the impact of these on performance, we conducted additional comparisons between QCS implementations with and without conditioning across three base architectures: DT, DC, and MLP. Table 11 reveals that the choice of base architecture does not significantly impact performance, except for the Adroit Pen. However, conditioning proves to be particularly beneficial for complex tasks and datasets with diverse trajectory optimality. Generally, we found that the DC base architecture is advantageous.

I Training Curves

QCS-R

QCS-G

60

Return

50

40

Normalized

30

20

10

0

1

2

3

4

5

Gradient Steps (105)

Return

50

40

Normalized

30

20

10

0

1

2

3

4

5

Gradient Steps (105)

100

Return

80

60

Normalized

40

20

0

0

1

2

3

4

5

umaze

100

Return

80

R

60

ize

Normal

40

20

O

0

12345678910

GradientSteps(105)

medium-diverse

100

Return

80

60

zed

N

Normal

40

20

O

0

12345678910

GradientSteps (105)

hopper-medium

100

eturn

80

R

60

N

40

rm

ION

20

0

0

1

2

3

4

5

GradientSteps(105)

hopper-medium-replay

100

80

R

60

N

40

ION

20

0

1

2

3

4

5

hopper-medium-expert

Return

100

75

zed

N

50

25

0

0

1

2

3

4

5

GradientSteps(105)

umaze-diverse

100

eturn

80

R

60

Z

40

orm

ION

20

0

0

12345678910

GradientSteps(105)

large-play

100

eturn

80

R

60

7

40

ION

20

0

0

12345678910

GradientSteps(105)

walker2d-medium

100

Return

80

R

60

alized

Normal

40

20

0

0

1

2

3

4

5

GradientSteps(105)

walker2d-medium-replay

100

eturn

80

R

60

orm

40

ION

20

0

0

1

2

3

4

5

GradientSteps(105)

walker2d-medium-expert

125

Return

100

75

50

ION

25

0

0

1

2

3

4

5

GradientSteps(105)

medium-play

100

eturn

80

Ret

60

zed

Normal

N

40

20

0

1

2345678910

Gradient Steps (105)

large-diverse

100

Return

80

60

ized

Normal

40

20

0

0

12345678910

Gradient Steps (105)

J Implementation and Hyperparameters

J.1 Training the Q -Function

QCS utilizes Q -aid, where the Q -function is learned through IQL [23], using the open-source implementation of IQL ( https://github.com/Manchery/iql-pytorch ) and following the common hyperparameters recommended by the authors, as listed in Table 12. For AntMaze, we set the expectile to 0.8, whereas for other domains, we set it to 0.7. Moreover, inspired by RLPD [5] and SPOT [46], we employed Layer Normalization [4] and a larger discount 0.995 for the Q and V networks in AntMaze. For a fair comparison, we retrained IQL using these modified hyperparameters, and the results are shown in Table 13. Since this modified setting had a negative effect on IQL, we used the IQL performances from the IQL paper [23] for Table 3.

J.2 Policy Training

Detailed description of the loss function.

where each component of the loss function is as follows:

The batch sampled over the entire dataset D (e.g., hopper-medium ):

i -th sub-trajectory in the batch for i = 1 , . . . , B :

Dataset-level Q -normalizer:

i.e., the dataset-level Q -normalizer ¯ Q IQL is the mean of the Q -value for all samples in the dataset.

Implementations. After training the Q -function, we train our policy with three different base architectures: DT [10], DC [20], and MLP. For DT-based QCS, we utilize the official DT codebase ( https://github.com/kzl/decision-transformer ) for our implementation. Similarly, for DC-based QCS, we use the official DC codebase ( https://github.com/beanie00/ Decision-ConvFormer ) for our implementation.

Hyperparameters. For the AntMaze domain, we used 10 6 gradient steps, and for the other domains, we used 5 × 10 5 gradient steps for training the policy. For all domains and base architectures, QCS uses a dropout rate of 0.1, ReLU as the nonlinearity function, a weight decay of 1e-4, and a LambdaLR scheduler [33] with a linear warmup of 10 4 gradient steps. In addition, we use a context length K of 20 for DT-based QCS, 8 for DC-based QCS, and 1 for MLP-based QCS. We found that the impact of action information and positional embedding on performance was negligible, so we excluded them from training. In QCS-R, we set our target RTG to the highest trajectory return in the dataset. For the MuJoCo and Adroit domains, we evaluate the target RTG at double this value. In the AntMaze domain, we test it at 100 times the value. This method aims to leverage the RTG generalization effect observed by Kim et al. [20]. We then report the best score achieved across the two target RTGs. From Table 14 to 15, we provide detailed hyperparameter settings for actor training.

Table 15: Per-domain hyperparameters of MLPbased QCS.

QCS Weight Relative to Trajectory Return. Our analysis suggests setting the QCS weight w ( R ( τ )) as a continuous, monotone-decreasing function of the trajectory return R ( τ ) . We explored various functional forms, including linear, quadratic, and exponential decay, but found that a simple linear decay w ( R ( τ )) = λ ( R ∗ -R ( τ )) suffices. In addition, we found that for some datasets, clipping w ( R ( τ )) to a minimum of 10 is beneficial, particularly for walker2d-medium-expert and QCS-R AntMaze, except umaze-diverse . The choice of λ for each dataset is presented in Table 16 to 18.

K Training Time

We compare QCS training time complexity with IQL [23], CQL [25], and QDT [53]. QCS requires a pre-trained Q learned using the IQL method, while QDT requires a pre-trained Q learned using the CQL method. Therefore, the time was measured by incorporating the Q pretraining time for both algorithms.

The training times for IQL, CQL, QDT, and QCS are as follows: IQL - 80 min, CQL - 220 min, QDT - 400 min, and QCS - 215 min.

Dataset antmaze-umaze

antmaze-umaze-diverse antmaze-medium-play

antmaze-large-diverse

λ

0.2

0.05

0.2

0.2

0.2

0.2

Dataset

λ

pen-human pen-cloned

0.01

0.01

The results show that QCS takes longer than IQL but has a total time similar to CQL. Notably, compared to QDT, which requires CQL pretraining, QCS can be trained in nearly half the time but demonstrates superior performance to QDT as shown in our main results in Table 2.

L Limitations

In this paper, we leveraged the complementary relationship between RCSL and Q -function overgeneralization to determine the QCS weight as a linear function of the trajectory return, which is readily obtainable from the dataset. This approach was tested on MuJoCo, AntMaze, and Adroit, where it showed promising results. However, depending on the task, a more advanced method that can efficiently evaluate Q -functions's over-generalization and provide appropriate Q -aid might be necessary. Additionally, this method entails the extra burden of pre-training the Q -function.

M Borader Impacts

This research is centered on enhancing the strengths of two promising approaches in the field of offline reinforcement learning: RCSL and value-based methods. By overcoming each of their limitations and creating better trajectories than the maximum quality trajectories of existing datasets, it contributes to the advancement of offline reinforcement learning. As foundational research in machine learning, this study does not lead to negative societal outcomes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction reflect all contributions in the paper well.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not deal with theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The project page is available at https://beanie00.com/publications/ qcs .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 6.1 and Appendix J.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In our main experimental results, Table 2 and Table 3 in Section 6, and Table 7 in Appendix G, we provide the mean and standard deviations for the five random seeds. Moreover, we provide the training curves in Fig. 12 in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have read the https://neurips.cc/public/ EthicsGuidelines and ensured that this paper conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See our anonymized zip file.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an offline reinforcement learning framework by combining return-conditioned supervised learning with in-sample Q-learning. By demonstrating their relative merits on offline datasets with different behavior policies, an overall loss function is designed to integrate Q assistance into RCSL. Extensive experiments empirically validate its superior performance with compared baselines on several domains and datasets.\n\nSTRENGTHS:\n1.\tThe studied problem is important. Empowering RCSL with stitching ability is crucial to facilitate its capability for pursuing optimal policies. \n2.\tThe analysis and experiments of RCSL and Q-learning on datasets with optimal and suboptimal policies provide valuable insights. Based on that, the proposed QCS method can be naturally optimized on the proposed loss function by considering conditional supervising learning and maximizing Q-function simultaneously.\n3.\tEmpirical experimental results are promising with a bunch of baselines on several datasets and domains.\n\nWEAKNESSES:\n1.\tThe definition of degree of Q-aid as w(R($\\tau$)) is not completely reasonable. As described, it should be defined by the degree of optimality of the behavior policy used to generate this sequence. However, a sequence with a higher return does not necessarily indicate it was generated by a superior policy, even from a statistical perspective. This definition only makes sense if the initial states of the compared trajectories are the same.\n2.\tThe analysis of the Q-Greedy Policy appears somewhat redundant. The focus of this paper is on integrating Q-learning into GCSL, rather than addressing challenges in Q-learning.\n3.\tConversely, the experimental section seems overly brief. I suggest moving more experimental details from the appendix to the main paper.\n\nQUESTIONS:\nAll my questions are listed in the weakness part.\n\nLIMITATIONS:\nNo limitation issues.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors of this paper introduce a new algorithm called Q-aided Conditional Supervised Learning (QCS) for offline DRL scenario, which combines the stability of return-conditioned supervised learning (RCSL) and the stitching ability of Q-functions. The primary contributions are 1). Identify the strength and weakness of RCSL and Q-learning in different experimental settings and 2) propose a new approach to combine the them to achieve better performance.\n\nSTRENGTHS:\n1. The paper is well written. The paper has a logical structure on the problem definition. It first demonstrates the strength and weakness of RCSL and Q-learning in different offline settings through experimental results and then explain the reasons through simple toy examples to help understanding. Finally, it propose the novel approach QCS based on the reasoning above. \n\n2. The paper has rigorous experimental set-up and SOTA performance. The proposed algorithm is tested on various offline D4RL benchmarking datasets and demonstrate SOTA performance across the datasets.\n\n3. Very interesting study on the strength and weakness of RCSL and Q-learning in different experimental settings, especially the discovery of Q-learning over generalization problem.\n\nWEAKNESSES:\n1. If I understand correctly, QCS requires a pre-trained Q-function using IQL, while the experimental results on D4RL benchmarking seems very impressive, there appears to be a lack of detailed discussion concerning the computational workload.\n\n2. The weight term (which is important and essential) in the algorithm is dynamic and depends on R*, the optimal return of the task. Calculating R* across environments and dataset settings can be a bit tricky and discussion on how calculation is performed is a also a bit limited and unclear. \n\n3. Different benchmark experiments seem to require different hyper-parameter $\\lambda$ values to achieve SOTA performance. Not sure how sensitive QCS is to $\\lambda$.\n\nQUESTIONS:\n1. For calculating R*, for each update using equation (3), do you need to calculate R* for each trajectory? Could you provide more detailed information on how it is calculated?\n\n2. I think the hyper-parameter K has very limited discussion on its impact on the performance. Do you perform any ablation studies on how K will affect the model performance?\n\n3. How sensitive is the QCS algorithm to the scaling factor $\\lambda$ ? I noticed that, for one environment with different offline settings, (for example, halfcheetah-medium-replay and halfcheetah-medium-expert), OCS uses different $\\lambda$ values. Does one fixed $\\lambda$ work well within the same environments? I think this is important for understanding the adaptability and robustness of the method across different settings.\n\nLIMITATIONS:\nThe paper briefly mentions the potential limitation of using a simple linear function to combine RCSL and Q-function.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
Offline reinforcement learning (RL) has advanced with return-conditioned supervised learning (RCSL) but still lacks stitching ability. Q-Aided Conditional Supervised Learning (QCS) combines RCSL's stability with the stitching capability of Q-functions, addressing Q-function over-generalization. QCS adapts Q-aid integration into RCSL's loss function based on trajectory returns, significantly outperforming RCSL and value-based methods in offline RL benchmarks. This innovation pushes the limits of offline RL and fosters further advancements.

STRENGTHS:
- it is well-written and fluent. 

- They brought clear definitions which make it easy to follow the context. 

- They compared with different baselines.

WEAKNESSES:
The innovation of the works is limited.

QUESTIONS:
- how the subgoals are chosen? Is not it complicated to choose them in a more complex environment?

LIMITATIONS:
The conditioning part seems to be controled by the algorithm which makes it hard to use in any environment.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes an algorithm that combines the stability of return-conditioned supervised learning (RCSL) with the stitching capability of Q-functions. The submission tests their algorithm in the MuJoCo domain with medium, medium-replay, and medium-expert datasets. The performance of the submission’s proposed method is higher on average in the MuJoCo domain.\n\nSTRENGTHS:\nThe paper tests their proposal in the main benchmarks.\n\nWEAKNESSES:\nThe HQIL algorithm [1] outperforms the proposed method of the submission. However, this is not mentioned, and further there is no comparison to HQIL.\n\nTable 1 might need some standard deviations.\n\nTo be able to interpret Table 2 correctly, the table needs to have standard deviations of the previous methods as well.\n\nPOR method can also perform above 70 in the antmaze-large-diverse dataset. \n\nOther studies also report results for antmaze-medium-replay dataset. Why did the authors omit this baseline when reporting results?\n\nTable 8 again needs standard deviations.\n\nThe submission only compares their algorithm to old baselines. However, there are currently algorithms that outperforms the proposal of the submission.\n\nSection 3.2. argues about something that is already expected/known to anyone that knows basic reinforcement learning, i.e. how the $Q$-learning update works. Should this section really occupy two pages in the main body of the paper?\n\n*“QCS represents a breakthrough in offline RL,pushing the limits of what can be achieved and fostering further innovations.”* \n\nThis statement might be too strong to describe the submission's contributions\n\n[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions, NeurIPS 2023.\n\nQUESTIONS:\nWhy did the authors bold their algorithm while SQL is the highest performing algorithm for hopper-m-e in Table 2? (111.8>110.2)\n\nLIMITATIONS:\nPlease see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}