Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Authors: jeonghye  kim
suyoung  lee
woojun  kim
youngchul  sung
  kaist
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:43:26

Abstract: Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce $Q$-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of $Q$-functions. By analyzing $Q$-function over-generalization, which impairs stable stitching, QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and value-based methods, consistently achieving or exceeding the highest trajectory returns across diverse offline RL benchmarks. QCS represents a breakthrough in offline RL, pushing the limits of what can be achieved and fostering further innovations.

Main paper: 
 preliminaries: 
 We consider a Markov Decision Process (MDP) [6] , described as a tuple M = (S, A, P, ρ 0 , r, γ). S is the state space, and A is the action space. P ∶ S × A ↦ ∆(S) is the transition dynamics, ρ 0 ∈ ∆(S) is the initial state distribution, r ∶ S × A ↦ R is the reward function, and γ ∈ [0, 1) is a discount factor. The objective of offline RL is to learn a policy π(⋅|s) that maximizes the expected cumulative discounted reward, E at∼π(⋅|st),st+1∼P(⋅|st,at) [∑ ∞ t=0 γ t r(s t , a t )], using a static dataset D = {τ (i) } D i=1 comprising a set of trajectories τ (i) . Each trajectory τ (i) consists of transitions over a time horizon T , collected from an unknown behavior policy β.Offline RL effectively employs off-policy RL techniques, allowing a divergence between the behavior policy β used for data acquisition and the target policy π being optimized [25, 14, 23] . Off-policy methods primarily utilize the Q-function, which is learned through temporal-difference (TD) bootstrapping. In actor-critic off-policy approaches, both the Q-function Qθ and the policy π are updated iteratively. This process can cause a shift in the action distribution, leading π to select actions that significantly deviate from those in the training dataset. These deviations can inadvertently result in overestimation errors, especially for out-of-distribution (OOD) actions, as offline RL cannot correct incorrect Q-values through interactions with the environment.Unlike actor-critic methods, in-sample learning methods use only in-sample actions to learn the optimal Q-function, thereby preventing the querying of OOD action Q-values during training [34, 32, 23, 51] . Implicit Q-Learning (IQL) [23] is a representative in-sample learning method. It utilizes expectile regression, defined as L 2 asymmetrical loss function for the value network V ψ . Through this loss, V ψ can approximate the implicit maximum of the TD target, max a Q θ (s, a). Formally, for a parameterized critic Q θ (s, a) with a target critic Q θ (s, a), the value loss function is given byEQUATIONIntuitively, this loss function suggests placing more emphasis when Q θ is greater than V ψ (s). Subsequently, the critic network Q θ is updated by treating the learned V ψ (s ′ ) as max a ′ ∈D(s ′ ) Q θ (s ′ , a ′ ), where D(s ′ ) denotes the in-sample actions for the given state s ′ , i.e., (s ′ , a ′ ) ∈ D:EQUATIONWe use IQL to pretrain the Q-function used to aid RCSL, as we found that this method, without conservatism during Q-function training, can provide good stitching ability when well integrated. A comparison with a different Q-learning method, CQL [25] , is provided in Appendix H.1.RCSL is an emerging approach to addressing challenges in offline RL. It focuses on learning the action distribution conditioned on return-to-go (RTG), defined as the cumulative sum of future rewards Rt = ∑ T t ′ =t r t ′ through supervised learning (SL) [10, 12, 20] . Due to the stability of SL, RCSL is capable of learning decision-making by extracting and mimicking useful information from the dataset. In particular, Decision Transformer (DT) [10] applies the Transformer architecture [41] to reframe the RL as a sequence modeling problem. It constructs input sequences to the Transformer by using sub-trajectories, each spanning K timesteps and comprising RTGs, states, and actions:τ t-K+1∶t = ( Rt-K+1 , s t-K+1 , a t-K+1 , ..., Rt-1 , s t-1 , a t-1 , Rt , s t ).The model is then trained to predict the action a t based on τ t-K+1∶t . Recently, Kim et al. [20] proposed Decision ConvFormer (DC) to simplify the attention module of DT and better model the local dependency in the dataset, yielding performance gains over DT with reduced complexity. These methods have shown effective planning capabilities, but they lack stitching ability, which causes difficulties with datasets that contain many sub-optimal trajectories. This will be discussed in more detail in Section 3.1.The Neural Tangent Kernel (NTK) [19] provides insightful analysis of function approximation errors of Q-function, Q θ , especially those related to generalization. The NTK, denoted as k θ (s, ā, s, a), is defined as the inner product of two gradient vectors,∇ θ Q θ (s, ā) and ∇ θ Q θ (s, a), i.e., k θ (s, ā, s, a) ∶= ∇ θ Q θ (s, ā) ⊺ ∇ θ Q θ (s, a).The NTK offers a valuable perspective on the impact of parameter updates in function approximation, particularly in gradient descent scenarios. It essentially measures the degree of influence a parameter update for one state-action pair (s, a) exerts on another pair (s, ā). A high value of k θ (s, ā, s, a) implies that a single update in the Q θ for the pair (s, a) could lead to substantial changes for the pair (s, ā). We guide the readers to Appendix D for a deeper understanding of the NTK.3 When Is Q-Aid Beneficial for RCSL?When is it beneficial for RCSL to receive assistance from the Q-function, denoted as Q θ , and how should this assistance be provided? To explore this, we trained two policies, RCSL policy and a max-Q policy that selects the best action according to Q θ , on two different quality datasets from D4RL [13] MuJoCo, comparing their performances in Table 1 . Note that the performance is not directly linked to the policy's accuracy across all states; even if the agent accurately predicts actions in several states, errors in some states can lead to path deviations and accumulated errors, resulting in a test-time distribution shift and a lower trajectory return. However, these results can provide insight into when Q-aid might be helpful.For the RCSL algorithm, we used the Decision Transformer (DT) [10] . To train the max-Q policy, we first trained the Q θ using the in-sample Q-learning method outlined in Eqs. ( 1) and (2) . Then, we extracted the max-Q policy to select the action that directly maximizes Q θ (s, ⋅) for each state s by using a 3-layer MLP and the loss function Observing Table 1 , we see that the dataset quality favoring RCSL contrasts with that benefiting the max-Q policy. RCSL tends to perform well by mimicking actions in high-return trajectory datasets [29, 31] . However, this method is less effective with datasets predominantly containing suboptimal trajectories, even though RTG conditioning helps predict actions that yield higher returns. On the other hand, the max-Q policy excels with suboptimal datasets but shows notably poor results with optimal datasets. From these observations, a motivating question arises: "Why does the simple max-Q policy outperforms RCSL on suboptimal datasets yet fails on optimal datasets? If so, how can we effectively combine the two methods to achieve optimal performance?"L max-Q (ϕ) = E s∼D [-Q θ (s, max-Q ϕ (s))].3.1 How Can Max-Q Policy Surpass RCSL in Suboptimal Datasets? The dataset consists of two trajectories, with a time limit of T = 3 and a discount factor γ = 1. The black dashed arrow represents the optimal policy yielding a maximum return of 7. We present a toy example demonstrating the limitation of RCSL, as illustrated in Fig. 3 . Suppose the dataset is composed of two sub-optimal trajectories. At the initial state s 1 , the agent has two options: the ↑ action connected to trajectory 2 (the orange trajectory) with an RTG of 5, and the → action connected to trajectory 1 (the purple trajectory) with an RTG of 6. RCSL makes the agent choose the → action with a high RTG and follow the path of trajectory 1, which is not optimal. This example demonstrates that RCSL alone is insufficient for the agent to learn to assemble the parts of beneficial sub-trajectories.In contrast, a Q-function can develop stitching ability. Consider the example in Fig. 3 again. We can compute the Q-values for the actions ↑ and → at state s 1 with dynamic programming:Q(s 1 , ↑) = 3 + max (Q(s 2 , →), Q(s 2 , ↘)) = 7, Q(s 1 , →) = 1 + max (Q(s 3 , →), Q(s 3 , ↖)) = 6.With the Q-values, the agent will select the ↑ action at s 1 and then the → action at s 2 . Consequently, using the Q-function, the agent can select the optimal action that yields the maximum return of 7. Therefore, integrating RCSL with Q-function in situations with abundant sub-optimal trajectories can be beneficial for developing the stitching ability required for optimal decision-making. 
why does max-q policy struggle with optimal datasets?: 
 Despite the potential advantages of using Q-values, incorporating values from a learned Q-function, Q θ , to aid RCSL can introduce errors due to inaccuracies in learning. These inaccuracies are particularly significant when Q θ is learned from optimal trajectories. Suppose we have an optimal policy π * . Optimal trajectories are visit logs containing actions performed by π * , yielding the best Q-value for a given state s. Due to the stochasticity of π * , multiple similar actions can be sampled from π * , namely a * 1 , a * 2 , . . . , a * n(s) ∼ π * (⋅|s) for a given state s. In this case, we haveEQUATION, . . . , n(s)} due to the optimality of these actions. When learning Q θ from such limited information, where the values at the narrow action points are almost identical for each given state, it is observed that the learned Q θ (s, a) tends to be over-generalized to the OOD action region. This means that the nearly identical value at the in-sample actions a * 1 , a * 2 , . . . , a * n(s) is extrapolated to OOD actions, yielding a nearly flat Q-value over the entire action space for each given state, i.e., Q θ (s, a OOD ) ≈ Q θ (s, a * 1 ) with Q θ (s, a) being a function of s only. This over-generalization makes Q θ noise-sensitive, potentially assigning high values to incorrect actions and causing state distribution shifts in the test phase, as shown in Fig. 7 .We present a simple experiment to verify that learning Q θ indeed induces over-generalization when trained on optimal trajectories. The experiment consists of an MDP with one-dimensional discrete states and actions, each divided into 500 bins. This environment simulates a car, where the state indicates the agent's position, ranging from -5 to 5, as illustrated in Fig. 4 (a). The action range is between -1 and 1, allowing the agent to move according to the direction and twice the magnitude of the action. The objective is to reach position 0, which grants a reward of 100, while larger actions incur penalties given as -30 ⋅ a 2 . Due to its discrete nature, we can compute the true optimal Q-values through value iteration [37] , which is shown in the bottom row of Fig. 4 (a) .With this environment, we generated two datasets, medium and expert. The medium dataset consisted of actions varying within the range of ±0.5 perturbed from the optimal action determined by the true optimal Q-values, while the expert dataset consisted of actions varying within the range of ±0.05 perturbed from the optimal action. (Refer to Fig. 4 (b)) We then adopted a 3-layer MLP as the structure of Q θ and performed regression to follow the true Q-value at each sample point (s, a) in the trajectories. Note that in-sample Q-learning can essentially be regarded as regression with the target value obtained from bootstrapping.The learned Q θ with the medium and expert datasets are shown in Fig. 4 (c ). Indeed, the learned Q θ with the expert dataset, containing nearly-optimal actions, shows that the value is flat over the entire action space for each state. This means that the nearly identical value of in-sample expert actions with a small spread is projected to the entire action space for each state. In contrast, the learned Q θ with the medium dataset well estimates the true Q-function. This is because the medium dataset has diverse actions with diverse values for each state that facilitate the regression process. We additionally present the results from Q θ obtained through IQL in Fig. 4 (d) , which shows a similar trend to the results from regression. k θ (s, ā, s, a ref )/∥∇ θ Q θ (s, a ref )∥ 2 2across four datasets with a 1D action space for Inverted Double Pendulum and a 3D action space for Hopper. In these figures, we fix the state s and the fixed reference action a ref at zero (marked as ★), and sweep over all actions ā ∈ A. For Hopper, we use axes for action dimensions and color to represent Q-values in 3D plots. Additionally, in the NTK plot, we only include the high-NTK regions for values over 0.9. Refer to Appendix E for details.The over-generalization tendency in Q θ with optimal trajectories is not limited to the simple experiment above but also applies to complex RL tasks. We analyze how Q θ (s, ⋅) varies over the action space in the Gym Inverted Double Pendulum [8] and MuJoCo Hopper environments [39, 8] trained on expert and medium-quality datasets with IQL. The details of the analysis are in Appendix E. As depicted in the upper row of Fig. 5 (a) and (b), and on the left side of the upper row of (c) and (d), the expert dataset shows concentrated action distribution, while the medium dataset has a broader spread, as expected. The concentration and similarity of true Q-values of the actions for a given state in the expert dataset cause over-generalization in Q θ (s, ⋅), yielding constant flat values across the entire action space. This is further supported by the results in Appendix E.2, which visualize the weights of the learned Q-function.For a deeper understanding of the over-generalization in Q θ , we analyze the gradient similarity, captured as the Neural Tangent Kernel (NTK), between an arbitrary action ā and the reference action a ref for a given state s. A higher NTK value signifies that the gradient update of Q θ (s, a ref ) has a substantial impact on Q θ (s, ā). This indicates that even when a ref and ā are dissimilar or unrelated actions, a high NTK value suggests that the Q θ network is misjudging the relationship between these actions and over-generalizing. In Fig. 5 , Q θ trained with the expert dataset shows uniformly high normalized NTK values across actions, indicating that the gradient at one action equally affects all others. In contrast, Q θ trained with the medium dataset shows NTK values that are higher near the reference action and decrease with action distance, reflecting more precise generalization. This analysis reveals that datasets consisting of optimal trajectories exhibit more aggressive generalization, which can negatively impact the accuracy of the learned Q-function in offline RL. 
q-aided conditional supervised learning: 
 According to Section 3, RCSL faces challenges with suboptimal datasets, whereas Q θ can effectively serve as a critic for stitching ability, favoring the use of Q-aid. In contrast, in an optimal dataset, Q θ tends to over-generalize, leading to inaccuracies of learned Q θ , while RCSL excels by mimicking the optimal behavior without requiring external assistance. Recognizing this dynamic, it is crucial to integrate Q-aid into RCSL adaptively. The following subsections explain how to effectively adjust the level of Q-aid and facilitate the integration of the two methods, leading to the proposal of Q-Aided Conditional Supervised Learning (QCS).Given the complementary relationship, how can we adjust the degree of Q-aid? Since RCSL's preference for mimicking datasets and the Q-function's over-generalization issue is tied to trajectory optimality, we can apply varying degrees of Q-aid based on the trajectory return for each subtrajectory in RCSL. Therefore, we set the degree of Q-aid, denoted as the QCS weight w(R(τ )) for a trajectory τ , as a continuous, monotone-decreasing function of the return of τ , R(τ ), such that∀τ 1 , τ 2 , R(τ 1 ) < R(τ 2 ) ⇒ w(R(τ 1 )) ≥ w(R(τ 2 )),where continuity is imposed for gradual impact change. Among various choices, we find that simple options such as linear decay are sufficient to produce good results, i.e., w(R(τ)) = λ ⋅ (R * -R(τ ))with some λ > 0, where R * represents the optimal return of the task. Practically, R * can be obtained from an expert dataset or from the maximum value in the dataset. For details on how to calculate R * , please refer to Appendix F. Note that R(τ ) differs from RTG Rt which is the sum of future rewards after timestep t and decreases as timestep t goes, thereby failing to represent the trajectory's optimality accurately.Instead of using the Q-function as the conditioning factor for RCSL as in previous works, we propose a more explicit approach by integrating Q-assistance into the loss function and dynamically adjusting the degree of assistance based on Section 4.1. As a result, the overall policy loss is given as follows:EQUATIONwhere Q IQL θ (⋅, ⋅) denotes the fixed Q-function pretrained with IQL. R(τ ) is the return of the entire trajectory τ containing the sub-trajectory τ t∶t+K-1 . The overall input to the policy at time t is the subtrajectory of context length K starting from time t, τ t∶t+K-1 = ( Rt , s t , a t , . . . , Rt+K-1 , s t+K-1 ) ⊂ τ.Our new loss function enables adaptive learning strategies depending on the trajectory's quality to which the subtrajectory belongs. For optimal trajectories, action selection follows RCSL. On the other hand, for suboptimal trajectories τ with R(τ ) < R * , the Q-aid term kicks in and its impact increases as R(τ ) decreases. We describe the details of the QCS weight w(R(τ )) and the policy update with the loss function in Appendix J.2 and our full algorithm's pseudocode in Appendix A.Base Architecture. For implementing π ϕ , a general RCSL policy can be used. When K = 1, meaning only the current time step is considered to estimate the action, we use an MLP network. When K ≥ 2, we use a history-based policy network, such as DT [10] or DC [20] .Conditioning. We consider two conditioning approaches as proposed by RvS [12] : one for tasks maximizing returns and the other for tasks aiming at reaching specific goals. For return-maximizing tasks, we employ RTG conditioning, and our algorithm is named QCS-R. For goal-reaching tasks, we additionally use subgoal conditioning, and our algorithm is named QCS-G. For subgoal selection, we randomly select a state that the agent will visit in the future. The ablations on conditioning are in Appendix H.2. 
related work: 
 Prompting RCSL with Dynamic Programming. Recent studies have recognized the limitations of RCSL in stitching abilities [27, 7, 58] . Our work contributes to the ongoing efforts to imbue RCSL with this capability. Notably, Q-learning Decision Transformer (QDT) [53] and Advantage Conditioned Transformer (ACT) [16] have proposed integrating dynamic programming into RCSL by modifying the RTG prompt to Q-value or advantage prompt. Our approach, QCS, parallels these efforts by leveraging dynamic programming for action guidance and trajectory stitching. However, unlike these methods, which implicitly incorporate dynamic programming through conditioning, QCS explicitly augments its loss function with the learned Q-function.Incorporating RCSL with Stitching Ability. In a distinct vein, recently proposed Critic-Guided Decision Transformer (CGDT) [43] identifies the gap between target RTG and expected returns of actions as key to RCSL's limited stitching. To mitigate this, it adjusts DT's output with the critic network's Monte-Carlo return predictions and target RTG. In contrast, QCS uses Q-values learned through dynamic programming to guide actions, enhancing stitching ability explicitly. Another approach, the Elastic Decision Transformer (EDT) [49] , recommends variable context lengths during inference, using longer contexts for optimal trajectories and shorter ones for sub-optimal trajectories to identify optimal paths better. QCS similarly adapts based on trajectory optimality but differentiates itself by modifying its learning approach during training, leveraging the complementary strengths of the Q-function and RCSL. Furthermore, POR [50] integrates imitation learning techniques with stitching ability by generating high-value states using additional networks and value functions. These states are then used as conditions for predicting actions. Unlike QCS, which focuses on action stitching, POR emphasizes state stitching, allowing agents to choose actions that lead to high-value states, albeit with the need for additional networks. By concentrating on action stitching, QCS can avoid the computational demands associated with high-dimensional state prediction.Regarding the sub-trajectory-adaptive weight used in QCS, FamO2O [42] employs state-adaptive weight coefficients to balance policy improvement and constraints in the offline-to-online RL framework. Although FamO2O is an offline-to-online method that incorporates additional online samples, we provide a performance comparison with this work in Appendix G.2 to further demonstrate the effectiveness of QCS. 
experiments: 
 In the experiment section, we conduct various experiments across different RL benchmarks to answer the following questions:• How well does QCS perform in decision-making compared to prior SOTA methods across datasets of varying quality and tasks with diverse characteristics, especially those requiring stitching ability?• To what extent does the dynamic nature of QCS weights, informed by trajectory return, contribute to effective decision-making, and how robust are these dynamic weights to hyperparameters?• Can QCS effectively acquire stitching ability while preventing test-time distribution shift?Baseline Methods. To address a range of questions, we conduct a comprehensive benchmarking against 12 representative baselines that are state-of-the-art in each category. For the value-based category, we assess 4 methods: TD3+BC [14] , IQL [23] , and CQL [25] , SQL [52] . For RCSL, we assess 3 methods: DT [10] , DC [20] , RvS [12] . Additionally, we evaluate 5 advanced RCSL methods proposed to integrate stitching capabilities: QDT [53] , EDT [49] , CGDT [43] , ACT [16] , and POR [50] . For more details on the setup and the baselines, refer to Appendix B.Benchmarks. We evaluated QCS against various baselines using datasets with diverse characteristics, including tasks focused on return maximization or goal-reaching, and those with dense or sparse rewards and varying sub-optimality levels. Our primary focus was on the D4RL [13] MuJoCo, AntMaze, and Adroit domains. The MuJoCo domain [39, 8] features several continuous locomotion tasks with dense rewards. We conducted experiments in three environments: Halfcheetah, Hopper, and Walker2d, utilizing three distinct v2 datasets-medium, medium-replay, and medium-expert-each representing different levels of data quality. AntMaze is a domain featuring goal-reaching environments with sparse rewards, encompassing variously sized and shaped maps. It is an ideal testing bed for evaluating an agent's capability to stitch trajectories and perform long-range planning. We conduct experiments using six v2 datasets: umaze, umaze-diverse, medium-play, medium-diverse, large-play, and large-diverse, where umaze, medium, and large indicate map sizes, and play and diverse refer to data collection strategies. The Adroit domain [35] comprises various tasks designed to evaluate the effectiveness of algorithms in high-dimensional robotic manipulation tasks. In our experiments, we utilize the human and cloned datasets for the pen task.Performance results for the MuJoCo and AntMaze domains are presented in Table 2 and Table 3 , while results for the Adroit domain are included in Appendix G.Hyperparameters and Backbone Architecture. We adopted two sets of hyperparameters per domain to determine the gradient of the monotonic decreasing function w(R(τ )). The detailed hyperparameters we used are provided in Appendix J, and the impact of λ is detailed in Appendix 6.3. Additionally, we implemented QCS based on DT, DC, and a simple MLP, and compared the performance of each. Detailed results for each architectural choice are provided in Appendix H.2. We observed that the DC-based approach performs best, although the performance gap is minor.Evaluation Metric. In all evaluations of QCS, we assess the expert-normalized returns [13] of 10 episodes at each evaluation checkpoint (every 10 3 gradient steps). Subsequently, we compute the running average of these normalized returns over ten consecutive checkpoints. We report the mean and standard deviations of the final scores across five random seeds. As shown in Table 2 and Table 3 , QCS significantly outperforms prior value-based methods, RCSL, and combined methods across the datasets. Specifically, QCS outperforms both IQL and DC, upon which it is based, across all datasets, unlike the contradictory results between RCSL and the max-Q policy shown in Table 1 . This empirically confirms that QCS successfully combines the strengths of both RCSL and the Q-function. A particularly remarkable achievement of QCS is its ability to substantially improve efficiency in goal-reaching tasks, AntMaze, especially in Large environments, where prior RCSL methods exhibited notably low performance. This enhancement is largely attributed to the stitching ability introduced by the Q-aid of QCS. These results underscore QCS's robustness and superiority in a wide array of offline RL contexts. The training curves for Tables 2 and 3 are shown in Appendix I, demonstrating stable learning curves across all datasets.To further analyze how each design element influences performance, we conducted additional experiments. More ablation studies are detailed in Appendix H, including the use of Q-function trained by CQL and the impact of base architecture and conditioning.The Importance of Weights Relative to Trajectory Return. To assess the impact of dynamically setting the QCS weight w(R(τ )) based on trajectory return, we compare our approach with a constant QCS weight, w(R(τ )) = c. We test five constant weights c ∈ {1, 2.5, 5, 7.5, 10} and report the maximum score among these values in Table 4 . The QCS method with the dynamic weight based on trajectory return outperforms the highest scores obtained with various constant weight settings across datasets, as shown in Table 4 .This demonstrates that our dynamic weight control, grounded in trajectory return, is more effective and robust in integrating Q-aids.Impact of the QCS weight λ. We examined the effect of λ by varying it from 0.2 to 1.5. As shown in Table 5 , except for the walker2d-medium, we found that even the smallest values achieved with changing λ either matched or surpassed the performance of existing value-based methods and RCSL's representative methods, including IQL, CQL, DT, DC, and RvS. This demonstrates QCS's relative robustness regarding hyperparameters. For walker2d-medium, we found that performance begins to decrease when λ exceeds the initial setting of 0.5. While increasing the gradient steps from 500K to 1M improves performance at λ = 1, further increasing λ to 1.5 leads to greater instability. To validate whether QCS effectively acquires stitching ability while preventing a shift in the test-time state distribution, as discussed in Section 3.2, we present Fig. 7 . This figure compares the state distributions explored by RCSL, max-Q, and QCS policies during evaluation. RCSL and max-Q, representing QCS's extremes, were trained using specific loss configurations: RCSL loss as QCS loss in Eq. 3 with λ = 0 and max-Q loss as QCS loss without the RCSL term, i.e., selecting actions as argmax a∈A Q IQL θ (s, a). Fig. 7 illustrates RCSL's adherence to dataset states, contrasting with the notable state distribution shift of the max-Q policy. QCS inherits RCSL's stability but surpasses its performance, indicating an effective blend of transition recombination without straying excessively from the state distribution.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: See Section 6.1 and Appendix J. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 In conclusion, QCS effectively combines the stability of RCSL with the stitching ability of the Q-function. Anchored by thorough observation of Q-function generalization error, QCS adeptly modulates the extent of Q-assistance. This strategic fusion enables QCS to exceed the performance of existing SOTA methods in both efficacy and stability, particularly in complex offline RL benchmarks encompassing a wide range of optimality.In addressing our initial motivating question on integrating RCSL and Q-function, QCS opens up promising future research directions. While we have established a correlation between trajectory return and the mixing weight, we have considered simple linear weights to control the level of Q-aid. It is also plausible that the mixing weight might be influenced by other dataset characteristics, such as the dimensions of the state and actions. We believe QCS will stand as a motivating work, inspiring new advancements in the field.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: In our main experimental results, Table 2 and Table 3 in Section 6, and Table 7 in Appendix G, we provide the mean and standard deviations for the five random seeds. Moreover, we provide the training curves in Fig. 12 in Appendix I. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Appendix E.1 and J.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 Figure 1 : Conceptual idea of QCS: Follow RCSL when learning from optimal trajectories where it predicts actions confidently but the Q-function may stitch incorrectly. Conversely, refer to the Qfunction when learning from sub-optimal trajectories where RCSL is less certain but the Q-function is likely accurate.Offline reinforcement learning (RL) is a vital framework for acquiring decision-making skills from fixed datasets, particularly when online interactions are impractical. This is especially relevant in fields such as robotics, autonomous driving, and healthcare, where the costs and risks of real-time experimentation are significant.A promising approach in offline RL is returnconditioned supervised learning (RCSL) [10, 12, 20] . By framing offline RL as sequence modeling tasks, RCSL allows an agent to leverage past experiences and condition on the target outcome, facilitating the generation of future actions that are likely to achieve desired outcomes. This method builds on recent advancements in supervised learning (SL) [41, 9, 11, 30] , and thus benefits from the inherent stability and scalability of SL. However, RCSL is significantly limited by its lack of 'stitching ability', the ability to combine suboptimal trajectory segments to form better overall trajectories [13, 27, 53, 16, 7, 58] . As a result, its effectiveness is restricted to the best trajectories within the dataset.Conversely, the Q-function possesses the ability to stitch together multiple sub-optimal trajectories, dissecting and reassembling them into an optimal trajectory through dynamic programming. Therefore, to address the weakness of RCSL, prior works have attempted to enhance stitching ability through the Q-function [53, 16] . However, these prior works employ the Q-function as a conditioning factor for RCSL and do not fully leverage the Q-function's stitching ability, resulting in either negligible performance improvements or even reduced performance. The primary challenge lies in the fact that utilizing the Q-function through conditioning, without managing the conditions for stable Q-guided stitching, can result in a sub-optimal algorithm.In this work, we aim to fully synergize the stable and scalable learning framework of RCSL with the stitching ability of the Q-function. To effectively utilize the Q-function, it is crucial to identify when it can benefit RCSL and to integrate its assistance, termed Q-aid. Our contributions to achieving the effective utilization of Q-aid in RCSL are as follows: (1) We discovered that in-sample Q-learning on an expert dataset, which predominantly consists of optimal actions with similar Q-values within a constrained action range, causes the Q-function to receive improper learning signals and become over-generalized. (2) To prevent errors from this Q-generalization and to incorporate stitching ability within RCSL's stable framework, we propose Q-Aided Conditional Supervised Learning (QCS), which adaptively integrates Q-aid into the RCSL's loss function based on trajectory returns.Despite its simplicity, the effectiveness of QCS is empirically demonstrated across offline RL benchmarks, showing significant advancements over existing state-of-the-art (SOTA) methods, including both RCSL and value-based methods. Especially, QCS surpasses the maximal dataset trajectory return across diverse MuJoCo datasets, under varying degrees of sub-optimality, as shown in Fig. 2 . Furthermore, QCS significantly outperforms the baseline methods in the challenging AntMaze Large environment. This notable achievement underscores the practical effectiveness of QCS in offline RL. 
 
 Appendix: The QCS algorithm first learns the Q-function using a dynamic programming and then trains the policy based on the aid of the learned Q-function. Detailed pseudocode can be found in Algorithm 1. In this work, we utilized IQL [23] as the Q training algorithm, but other Q training algorithms can be employed. A comparison with using a Q-function trained by CQL [25] can be found in Appendix H.1.Hyperparameters: Total critic gradient steps M , critic learning rate α critic , target update rate χ, total policy gradient steps N , policy learning rate α policy , context length K Initialize parameters: θ, θ, ψ, and ϕ// IQL Pretraining for m = 1 to M do ψ ← ψ -α critic ∇L V (ψ) using Eq. 1 θ ← θ -α critic ∇L Q (θ) using Eq. 2 θ ← χθ + (1 -χ) θ end for // QCS Policy Training for n = 1 to N doSample trajectory τ ∼ D Sample sub-trajectory τ t∶t+K-1 ∼ τ with random initial timestep t ϕ ← ϕα policy ∇L QCS π (ϕ) using Eq. 3 end forWe evaluated the performance of QCS against twelve different baseline methods. This group consists of four value-based methods: TD3+BC [14] , IQL [23] , CQL [25] and SQL [52] ; three RCSL algorithms: DT [10] , DC [20] , and RvS [12] ; and five combined methods that signify progress in RCSL by integrating stitching abilities: QDT [53] , EDT [49] , and CGDT [43] , ACT [16] , and POR [50] . The performance for these baselines was sourced from their respective original publications, with two exceptions. For CQL [25] , the performance data in the original paper was based on the MuJoCo v0 environment, which differs from the v2 version used in our study. Therefore, for CQL, we referenced the performance score reported in [23] to ensure a consistent and fair comparison across all methods.In addition, for antmaze-medium and antmaze-large, since there were no reported DT [10] and DC [20] scores, we conducted evaluations using the official codebase. When training on antmaze-medium and large, we used 512 as the embedding dimension in the default hyperparameter setting. We found that removing the positional embedding slightly improved performance, as also discussed in Zheng et al. [57] , so we trained without it. For the target RTG, we used values of 1 and 100 and reported the higher score obtained from the two values.To gain a deeper understanding of the scenarios in which offline RL is applied and the necessity of learning good policies, we plotted the trajectory return distributions for three different datasets in each of the three MuJoCo environments in Fig. 8 . For these return distribution histograms, we set the number of bins to 50. The 'Count' label denotes the number of trajectories corresponding to each normalized return. As shown in Fig. 8 , the medium-replay datasets encompass wide varieties of returns. Additionally, the medium-expert dataset, a combination of the medium and expert datasets, exhibits two peaks, indicating a division in the range of returns. This observation reveals that each dataset exhibits a distinct distribution pattern of returns. In this graph, alongside the return distribution, the dataset's maximum trajectory return and the score of our QCS method are also presented. QCS is observed to achieve results that are close to or surpass the maximum return. This is particularly notable in datasets with diverse return distribution characteristics, such as the medium-replay dataset, where the distribution of low return trajectories is prevalent.To understand the influence of parameter updates in function approximation across different stateaction pairs, the Neural Tangent Kernel (NTK) emerges as a crucial tool [19, 28, 1, 26, 56] . For comprehensive insights, we direct readers to Achiam et al. [1] , while here we distill the essential concepts. The NTK framework becomes particularly relevant in the context of deep Q-learning, where the parameterized Q-function, denoted as Q θ , is updated as follows [37] :EQUATIONwhere α is the learning rate, ρ is the distribution of transitions in the dataset, andδ θ (s, a) = r(s, a) + γ max a ′ Q θ (s ′ , a ′ ) -Q θ (s, a)is the temporal difference (TD) error. On the other hand, the first-order Taylor expansion around θ at an out-of-sample pair (s, ā) yieldsQ θ ′ (s, ā) = Q θ (s, ā) + ∇ θ Q θ (s, ā) ⊺ (θ ′ -θ).(5) Substituting ( 4) into (5), we haveEQUATIONwhere k θ (s, ā, s, a), referred to as the NTK, is defined as the inner product between two gradient vectorsEQUATION(6) together with (7) explains how the parameter update with function approximation for a sample pair (s, a) affects the Q-value change for another sample pair (s, ā). When the NTK k θ (s, ā, s, a) is large, the TD-error δ θ (s, a) has a more pronounced impact on the update differenceQ θ ′ (s, ā) -Q θ (s, ā).Thus, the single update based on the TD-error at a sample pair (s, a) can induce a significant change in the Q-function for another pair (s, ā).In Section 3.2, we conduct an NTK analysis of the Q-function trained with IQL in the Inverted Double Pendulum and Hopper environments, which have state dimensions of 2 and 11, and action dimensions of 1 and 3, respectively. This section details the analysis methods, provides extended results, and offers further clarity by presenting the action distributions for the datasets of each environment.Inverted Double Pendulum.We chose the Inverted Double Pendulum for analysis of Qvalues and NTK due to its one-dimensional action space. For training the Q-function, as no prior open-source offline dataset existed for this environment, we first created one. The dataset was generated by training an online Soft Actor-Critic [18] , using an implementation in RLkit, available at https://github.com/rail-berkeley/rlkit.git.We created two datasets: expert and medium. The expert dataset consists of 10 5 samples generated by an optimal policy, while the medium dataset includes 10 5 samples from a medium policy, whose performance is approximately one-third of the optimal policy. Given the continuous nature of the state and action spaces in the Inverted Double Pendulum, which complicates analysis, we initially quantized both spaces. For state quantization, we set the range from a minimum of -5 to a maximum of 10 (the minimum and maximum values across all dimensions in both datasets) and divided each dimension into 80 equal segments. For action quantization, the range was set from -1 to 1, divided into 500 equal segments.When plotting the Q-values, we calculated the Q-value for each quantized state across all quantized actions. Fig. 5 shows the results for the state chosen in each dataset, based on the highest count of in-sample actions. In the NTK analysis, we computed the following equation for the reference action and the remaining quantized actions with index i ∈ 1, . . . , 500, where a 1 = -1, a 500 = 1, and a ref = 0.MuJoCo Hopper. For Hopper environment, we use open-source D4RL [13] hopper-expert and hopper-medium-replay datasets. In the case of Hopper, similar to what was done in the Inverted Double Pendulum environment, we quantized the continuous state and action space for analysis. More specifically, for the state space, we divided the values of each dimension into 100 equal segments, ranging from -10 to 10. As for the action space, we divided the values of each dimension into 50 equal segments, ranging from -1 to 1. In the case of Hopper, with its 3D action dimension, visualizing it similarly to the 1D action dimension in the Inverted Double Pendulum posed a challenge. Consequently, in the 3D plots, we assigned each axis to one of the action dimensions and utilized color to indicate the Q-value, as shown in Fig. 5 . Additionally, in NTK analysis, representing the relationships between the reference action and all quantized actions within a single graph is challenging. We marked high-NTK regions in gray, where the normalized NTK values are greater than 0.9. For the expert dataset (Fig. 9(a) ), the action-related elements of W 1 (the last column elements) are learned as zero. This intriguing result indicates that the network, during its training, learns not to differentiate between actions across all states, leading to uniformly flat Q-values for all actions. Such behavior is characteristic of datasets with a narrower action spread, where the actions are more clustered and coherent. The network's tendency to not distinguish between different actions in such a dataset is a direct consequence of the limited diversity, requiring less differentiation in the action representation.In contrast, for the medium dataset (Fig. 9(b) ), which represents a wider action spread, the actionrelated elements of W 1 show variation. This variation signifies that the network has learned to differentiate between actions to a greater extent, a necessity in a dataset where actions are more diverse and dispersed. The network's capacity to distinguish between various actions and assign different levels of importance to each reflects the need for a more nuanced understanding of the action space in datasets with a wider spread. This visual evidence from the learned weights substantiates our understanding of how neural networks adapt their learning based on the diversity in the action space of the dataset. In datasets with narrower action spreads with similar Q-values, the network learns a more uniform approach towards different actions, while in those with wider spreads with diverse Q-values, it adopts a more differentiated and discerning strategy. This adaptive learning aligns with the principles of regression demonstrating the network's response to the diversity and distribution of actions in the training data. Generally, in the medium dataset, the NTK is high between two actions that are close to each other and low between actions that are far apart. However, in the expert dataset, this distinction becomes blurred, regardless of the proximity of the actions. In this subsection, we examine the action distributions of datasets within the Inverted Double Pendulum and MuJoCo Hopper environments. The continuous nature and multi-dimensionality of these action spaces pose significant challenges for directly visualizing the exact action distributions. To address this, we define the average action distribution spread of a dataset D, quantified as the expectation over the states of the dataset as:EQUATIONWe then visualize the distribution of the L2 distance among all actions within each quantized state across the dataset using Eq. ( 8). This approach is based on the characteristics of the action dimensions in both the Inverted Double Pendulum and the Hopper, which are bounded between -1 and 1. For action distribution visualization, we maintain the same state and action quantization as outlined in Appendix E.1.Fig. 11 presents the results. As depicted in this figure, in both the Inverted Double Pendulum and Hopper environments, the expert datasets exhibit a small average L2 distance between actions coexisting in the same quantized state. This indicates a denser clustering of actions within these datasets, which is linked to high-return datasets typically exhibiting a more concentrated action distribution on average since they primarily perform exploitation actions instead of exploration actions.In Section 4.1, we define the QCS weight w(R(τ )) as λ ⋅ (R * -R(τ )), where R * is the optimal return for the task. For calculating R * , we consider the two methods described below.(i) Set R * with the optimal return for the environment. In our experiments, we set R * for the environments with optimal returns as follows: Hopper (R * = 3500), Walker2d (R * = 5000), Halfcheetah (R * = 11000), and AntMaze (R * = 1). Note that prior RCSL algorithms such as Decision Transformer [10] and RvS [12] used predefined R * for target RTG conditioning during inference. Therefore, using R * based on the optimal return introduces no additional assumptions compared to previous RCSL methods. As noted in Appendix J.2, QCS does not use R * for target RTG conditioning but instead relies on the maximum trajectory return, requiring only one R * per algorithm.(ii) Set R * to the maximum trajectory return within the dataset. An alternative approach for setting R * is to use the maximum trajectory return from the dataset. When obtaining the true optimal return from the environment is challenging, the maximum trajectory return can serve as an approximation. Table 6 presents additional results using this method for R * . As shown in Table 6 , setting R * with the optimal environment return is slightly better than setting it with the maximum dataset return, but setting it with the maximum dataset return still outperforms the baselines. Therefore, we propose using the optimal environment return for R * ; however, when it is hard to determine, using the maximum dataset return can be a good alternative.In addition to the performance comparison in the MuJoCo and AntMaze domains, as discussed in Section 6.2, we also compare the performance of QCS in the Adroit domain using extensive baselines, similar to those mentioned in Section 6.2. Since there are no reported results for TD3+BC [14] , SQL [52] , RvS [12] , QDT [53] , EDT [49] , CGDT [43] , ACT [16] , POR [50] in the Adroit domain, we only compare with the value-based baselines (IQL [23] , CQL [25] ) and RCSL baselines (DT [10] , DC [20] , RvS [12] ). For DT and DC, we evaluate the score using their official codebase.Table 7 displays the performance of QCS alongside the baseline performances. As indicated by the results, QCS-R outperforms other baselines in the Adroit Pen task. This outcome reiterates that QCS is a robust framework, excelling in a range of tasks with varying features. It also underscores findings from our earlier experiments, which demonstrate that strategically blending RCSL with Q-function can significantly enhance performance. FamO2O [42] is an offline-to-online RL method that facilitates state-adaptive balancing between policy improvement and constraints. During offline pre-training, it develops a set of policies with various balance coefficients. In the subsequent online fine-tuning phase, FamO2O determines the most suitable policy for each state by selecting the corresponding balance coefficient from this set. The major difference between QCS and FamO2O is that FamO2O additionally uses 10 6 online samples to find a suitable balance coefficient, while QCS only utilizes the offline dataset and adjusts the balance coefficient (QCS weight) based on the trajectory return. Moreover, unlike FamO2O, which utilizes a state-adaptive balance coefficient, QCS is based on historical architecture and uses a sub-trajectory-adaptive balance coefficient. Although it is not a fair comparison between FamO2O, an offline-to-online algorithm, and QCS, a purely offline algorithm, we present the performance comparison to demonstrate the effectiveness of QCS even when compared with an offline-to-online algorithm. To compare the performance of QCS using Q-function learned through actor-critic algorithms, we use representative actor-critic algorithms such as CQL [25] for benchmarking. As shown in Table 10 , the performance of CQL-aided QCS generally improved compared to the original CQL, but it does not match the performance of IQL-aided QCS for the MuJoCo domain. The reason can be attributed to two factors: (1) the Q-function may be bounded by the actor's representation ability, and (2) CQL might impose excessive conservatism on the Q-function. Moreover, in the case of antmaze-umaze-diverse, IQL-aided QCS underperforms CQL, but CQL-aided QCS outperforms CQL. Since QCS is a general framework that proposes a new combination of RCSL and Q-function on trajectory return, there is a wide range of potential integrations of RCSL and offline Q-learning methods. The most impactful aspect will differ depending on the characteristics of each RCSL and Q-learning method when combined. Exploring this would be an interesting research area, which we leave as future work. In Section 4.3, we discussed QCS variants with different base architectures and conditioning. To assess the impact of these on performance, we conducted additional comparisons between QCS implementations with and without conditioning across three base architectures: DT, DC, and MLP. Table 11 reveals that the choice of base architecture does not significantly impact performance, except for the Adroit Pen. However, conditioning proves to be particularly beneficial for complex tasks and datasets with diverse trajectory optimality. Generally, we found that the DC base architecture is advantageous. QCS utilizes Q-aid, where the Q-function is learned through IQL [23] , using the open-source implementation of IQL (https://github.com/Manchery/iql-pytorch) and following the common hyperparameters recommended by the authors, as listed in Table 12 . For AntMaze, we the expectile to 0.8, whereas for other domains, we set it to 0.7. Moreover, inspired by RLPD [5] and SPOT [46] , we employed Layer Normalization [4] and a larger discount 0.995 for the Q and V networks in AntMaze. For a fair comparison, we retrained IQL using these modified hyperparameters, and the results are shown in Table 13 . Since this modified setting had a negative effect on IQL, we used the IQL performances from the IQL paper [23] for Table 3 . Detailed description of the loss function.EQUATIONwhere each component of the loss function is as follows:• The batch sampled over the entire dataset D (e.g., hopper-medium):B = {τ (1) t1∶t1+K-1 , . . . , τ (B) t B ∶t B +K-1 } , B = |B|.• i-th sub-trajectory in the batch for i = 1, . . . , B:τ (i) ti∶ti+K-1 = ( R(i) ti , s (i) ti , a (i) ti , . . . , R(i) ti+K-1 , s (i) ti+K-1 ) . • Dataset-level Q-normalizer: QIQL θ = 1 |D| ∑ (s,a)∈D Q IQL θ (s, a) ,i.e., the dataset-level Q-normalizer QIQL is the mean of the Q-value for all samples in the dataset.QCS Weight Relative to Trajectory Return. Our analysis suggests setting the QCS weight w(R(τ )) as a continuous, monotone-decreasing function of the trajectory return R(τ ). We explored various functional forms, including linear, quadratic, and exponential decay, but found that a simple linear decay w(R(τ )) = λ (R * -R(τ )) suffices. In addition, we found that for some datasets, clipping w(R(τ )) to a minimum of 10 is beneficial, particularly for walker2d-medium-expert and QCS-R AntMaze, except umaze-diverse. The choice of λ for each dataset is presented in Table 16 to 18. We compare QCS training time complexity with IQL [23] , CQL [25] , and QDT [53] . QCS requires a pre-trained Q learned using the IQL method, while QDT requires a pre-trained Q learned using the CQL method. Therefore, the time was measured by incorporating the Q pretraining time for both algorithms.The training times for IQL, CQL, QDT, and QCS are as follows: IQL -80 min, CQL -220 min, QDT -400 min, and QCS -215 min.The results show that QCS takes longer than IQL but has a total time similar to CQL. Notably, compared to QDT, which requires CQL pretraining, QCS can be trained in nearly half the time but demonstrates superior performance to QDT as shown our main results in Table 2 .In this paper, we leveraged the complementary relationship between RCSL and Q-function overgeneralization to determine the QCS weight as a linear function of the trajectory return, which is readily obtainable from the dataset. This approach was tested on MuJoCo, AntMaze, and Adroit, where it showed promising results. However, depending on the task, a more advanced method that can efficiently evaluate Q-functions's over-generalization and provide appropriate Q-aid might be necessary. Additionally, this method entails the extra burden of pre-training the Q-function.This research is centered on enhancing the strengths of two promising approaches in the field of offline reinforcement learning: RCSL and value-based methods. By overcoming each of their limitations and creating better trajectories than the maximum quality trajectories of existing datasets, it contributes to the advancement of offline reinforcement learning. As foundational research in machine learning, this study does not lead to negative societal outcomes.Answer: [Yes] Justification: The project page is available at https://beanie00.com/publications/ qcs. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.η (u) = |η -1(u < 0)|u2 where η ∈ [0.5, 1), to formulate the 


Decision: The submission presents an analysis of the failure modes of RCSL and value-based approaches in Offline RL. While this analysis may not be particularly novel or surprising, it effectively establishes a solid foundation for developing a new solution that mitigates these failures. The proposed algorithm, though simple, delivers compelling empirical results that validate the soundness of the approach. Based on these strengths, I recommend accepting this submission as a poster.

Reviews: review: paper_summary: paper_summary: This submission proposes an algorithm that combines the stability of return-conditioned supervised learning (RCSL) with the stitching capability of Q-functions. The submission tests their algorithm in the MuJoCo domain with medium, medium-replay, and medium-expert datasets. The performance of the submission’s proposed method is higher on average in the MuJoCo domain.
strength_weakness: strength_weakness: The HQIL algorithm [1] outperforms the proposed method of the submission. However, this is not mentioned, and further there is no comparison to HQIL.

Table 1 might need some standard deviations.

To be able to interpret Table 2 correctly, the table needs to have standard deviations of the previous methods as well.

POR method can also perform above 70 in the antmaze-large-diverse dataset. 

Other studies also report results for antmaze-medium-replay dataset. Why did the authors omit this baseline when reporting results?

Table 8 again needs standard deviations.

The submission only compares their algorithm to old baselines. However, there are currently algorithms that outperforms the proposal of the submission.

Section 3.2. argues about something that is already expected/known to anyone that knows basic reinforcement learning, i.e. how the $Q$-learning update works. Should this section really occupy two pages in the main body of the paper?

*“QCS represents a breakthrough in offline RL,pushing the limits of what can be achieved and fostering further innovations.”* 

This statement might be too strong to describe the submission's contributions

[1] HIQL: Offline Goal-Conditioned RL with Latent States as Actions, NeurIPS 2023.
questions: questions: Why did the authors bold their algorithm while SQL is the highest performing algorithm for hopper-m-e in Table 2? (111.8>110.2)
limitations: limitations: Please see weaknesses.
paper_summary: paper_summary: Offline reinforcement learning (RL) has advanced with return-conditioned supervised learning (RCSL) but still lacks stitching ability. Q-Aided Conditional Supervised Learning (QCS) combines RCSL's stability with the stitching capability of Q-functions, addressing Q-function over-generalization. QCS adapts Q-aid integration into RCSL's loss function based on trajectory returns, significantly outperforming RCSL and value-based methods in offline RL benchmarks. This innovation pushes the limits of offline RL and fosters further advancements.
strength_weakness: strength_weakness: The innovation of the works is limited.
questions: questions: - how the subgoals are chosen? Is not it complicated to choose them in a more complex environment?
limitations: limitations: The conditioning part seems to be controled by the algorithm which makes it hard to use in any environment.
paper_summary: paper_summary: Authors of this paper introduce a new algorithm called Q-aided Conditional Supervised Learning (QCS) for offline DRL scenario, which combines the stability of return-conditioned supervised learning (RCSL) and the stitching ability of Q-functions. The primary contributions are 1). Identify the strength and weakness of RCSL and Q-learning in different experimental settings and 2) propose a new approach to combine the them to achieve better performance.
strength_weakness: strength_weakness: 1. If I understand correctly, QCS requires a pre-trained Q-function using IQL, while the experimental results on D4RL benchmarking seems very impressive, there appears to be a lack of detailed discussion concerning the computational workload.

2. The weight term (which is important and essential) in the algorithm is dynamic and depends on R*, the optimal return of the task. Calculating R* across environments and dataset settings can be a bit tricky and discussion on how calculation is performed is a also a bit limited and unclear. 

3. Different benchmark experiments seem to require different hyper-parameter $\lambda$ values to achieve SOTA performance. Not sure how sensitive QCS is to $\lambda$.
questions: questions: 1. For calculating R*, for each update using equation (3), do you need to calculate R* for each trajectory? Could you provide more detailed information on how it is calculated?

2. I think the hyper-parameter K has very limited discussion on its impact on the performance. Do you perform any ablation studies on how K will affect the model performance?

3. How sensitive is the QCS algorithm to the scaling factor $\lambda$ ? I noticed that, for one environment with different offline settings, (for example, halfcheetah-medium-replay and halfcheetah-medium-expert), OCS uses different $\lambda$ values. Does one fixed $\lambda$ work well within the same environments? I think this is important for understanding the adaptability and robustness of the method across different settings.
limitations: limitations: The paper briefly mentions the potential limitation of using a simple linear function to combine RCSL and Q-function.
paper_summary: paper_summary: This paper proposes an offline reinforcement learning framework by combining return-conditioned supervised learning with in-sample Q-learning. By demonstrating their relative merits on offline datasets with different behavior policies, an overall loss function is designed to integrate Q assistance into RCSL. Extensive experiments empirically validate its superior performance with compared baselines on several domains and datasets.
strength_weakness: strength_weakness: 1.	The definition of degree of Q-aid as w(R($\tau$)) is not completely reasonable. As described, it should be defined by the degree of optimality of the behavior policy used to generate this sequence. However, a sequence with a higher return does not necessarily indicate it was generated by a superior policy, even from a statistical perspective. This definition only makes sense if the initial states of the compared trajectories are the same.
2.	The analysis of the Q-Greedy Policy appears somewhat redundant. The focus of this paper is on integrating Q-learning into GCSL, rather than addressing challenges in Q-learning.
3.	Conversely, the experimental section seems overly brief. I suggest moving more experimental details from the appendix to the main paper.
questions: questions: All my questions are listed in the weakness part.
limitations: limitations: No limitation issues.

Comments: title: Further clarification on previous response
comment: ### **Observation of Q Overgeneralization is one of our contributions.**

Starting from the 164th sentence in our manuscript (slightly below the sentence mentioned by the reviewer)—"We present a simple experiment to verify that learning $Q_\theta$ indeed induces over-generalization when trained on optimal trajectories."—this phenomenon is illustrated throughout Fig. 4 and Fig. 5 in Section 3.2.

&nbsp;

### **The topic 'Why Does the Q-Greedy Policy Struggle with Optimal Datasets?' is addressed in Section 3.2 by demonstrating the phenomenon of Q generalization and validating it through various methods across different domains.**
Section 3.2 explains that the Q function tends to over-generalize when trained on an optimal dataset, which is why the Q-Greedy Policy struggles with optimal datasets. Throughout the section, we demonstrate that this overgeneralization phenomenon can particularly occur when training with an optimal dataset, using various domains including toy environment, Gym Inverted Double Pendulum, and MuJoCo Hopper, as well as various methods such as Q-value analysis and NTK analysis  (please see Figures 4 and 5 in our manuscript). The appendix provides an in-depth analysis that couldn’t be fully covered in the main text due to space constraints, but it is already addressed in the main body.

&nbsp;

### **The results reported in Appendix E (the MuJoCo Hopper results) are additional analyses that connect to the Hopper results shown in Figure 5 of the main paper.**

In Section 3.2, Figure 4, we present the results of a toy experiment, and immediately afterward, in Figure 5, we extend this experiment to demonstrate that a similar phenomenon occurs in both the Gym Inverted Double Pendulum and MuJoCo Hopper environments. We would like to note that the main paper includes not only the results of the toy experiment but also the results for Hopper. Therefore, the results in Appendix E are not a sudden comparison involving a change of domain and an expansion of the dimension from the toy example, but rather a more detailed analysis of the results presented for Hopper in Figure 5.
We appreciate the reviewer's additional comments aimed at further clarifying our work.

&nbsp;

### **POR changed their official score after their camera-ready version.**

In the POR GitHub, around Nov 2022, questions were raised regarding the reproduction of POR results. Since the POR authors could not access their previous code due to the copyright issue, they re-implemented the work and presented the results anew. While most of the results matched the previous ones, they did not match the hopper-medium-v2 dataset. As a result, they replaced the score with the new results and updated the arxiv version accordingly. Through a comment on GitHub, the author recommended using the new results instead of the conference version, which the reviewer had checked. The exact author's comment is **"We have updated our paper in the arxiv to reflect that and the mismatch results in the hopper-medium-v2 dataset (lower it from 98.2 to 78.6). If you want to compare POR with your work, you can refer to this paper version."**. Due to NeurIPS rebuttal policies, we cannot include external links, but we provide the address of this comment: github.com/ryanxhr/POR/issues/2. Therefore, our POR score reporting is accurate and aligns with the author's intent.

&nbsp;

### **Comparison with QT in Table R9.**

Surpassing QT in 13 out of the 15 datasets we compared, with the remaining 2 scores being similar to QT within the range of standard deviation, cannot be a reason for rejection but rather demonstrates the superiority of QCS. (QCS mean 86.3 > QT mean 60.9)

We want to emphasize that the offline RL benchmark features a variety of state and action dimensions, as well as different goals, making it a diverse and challenging benchmark where it is difficult for a single algorithm to perform well across all datasets compared to all baselines. For example, both POR and QT have instances where their performance is similar to or even lower than that of previous studies when examined closely at each dataset. For example, POR's performance of 76.6 on walker2d-medium-replay is lower than TD3+BC's 81.8, and QT's reported performance of 59.3 on antmaze-medium-diverse is lower than IQL's 70.0. However, POR and QT are recognized for their sufficiently good results due to their generally strong performance. **Slightly lower or similar scores to baselines on a few datasets cannot be grounds for rejection. We believe that what truly matters is how robustly the performance has improved compared to previous works.**

&nbsp;

### **Comparison of QCS to the Best Score**

As mentioned in the previous response, we were unable to confirm the exact evaluation metric for POR. Since we have already compared the scores from the POR paper provided by the POR author in Tables 2 and 3 of our original manuscript, we conduct an additional comparison for QT with the best score. Table R10 shows the results of the comparison using the best score. As can be seen from the table, the QCS score has generally increased compared to the running average score that we previously reported. Compared to QT, it also shows superior results on average in MuJoCo (six datasets are better, two are similar, and one is worse), and especially demonstrates better performance with a large gap in Antmaze. While we knew that reporting the best score would naturally result in better performance, we want to emphasize again, as mentioned in our previous response, that we believe the last running average score is a more appropriate metric for offline settings, which is why we reported using this metric. We have demonstrated better performance than QT, but we would also like to clarify once again that QT should not be a reason for rejection, as it qualifies as concurrent work under NeurIPS policies.

&nbsp;

**Table R10. Performance comparison between QCS (ours) and QT. We evaluate each algorithm based on the best score.**

| Dataset            | QT (best)           | QCS (best)         |
|--------------------|---------------------|--------------------|
| halfcheetah-m      | 51.4±0.4            | **60.6**±0.3       |
| hopper-m         | 96.9±3.1          | **99.7**±0.3       |
| walker2d-m       | 88.8±0.5           | **92.3**±0.4       |
| halfcheetah-m-r    | 48.9±0.3        | **55.5**±0.5       |
| hopper-m-r       | 102.0±0.2           | **103.4**±0.5      |
| walker2d-m-r     | **98.5**±1.1        | **98.6**±0.3       |
| halfcheetah-m-e  | **96.1**±0.2        | 95.6±0.2           |
| hopper-m-e     | **113.4**±0.4       | **113.1**±0.6      |
| walker2d-m-e     | 112.6±0.6           | **118.3**±1.2      |
| **average**      | **89.8**            | **93.0**           |
| antmaze-u        | 96.7±4.7            | **100.0**±0.0      |
| antmaze-u-d        | 96.7±4.7            | **98.0**±4.0       |
| antmaze-m-d        | 59.3±0.9            | **100.0**±0.0      |
| antmaze-l-d        | 53.3±4.7            | **92.0**±7.5       |
| **average**        | 76.5                | **97.5**           |
I thank the authors for their response and providing the standard deviation results. 

The results reported in the original paper [1] for POR are different from what the submission reports. For instance, for the hopper-m dataset for POR the original paper reports 98.2 ±1.6, however the submission reports POR results as 78.6 ± 7.2 for the hopper-m dataset. Given that the performance of the submission’s proposed algorithm QCS-R is 96.4 ± 3.72, this means the submission does not outperform a prior algorithm POR. The way the results are reported in the submission is quite misleading and incorrect. 

[1] A Policy-Guided Imitation Approach for Offline Reinforcement Learning, NeurIPS 2022.

Looking at the results reported in Table 9 in the attached pdf, in the hopper-m-e dataset the performance of QT is 108.2± 3.6, and the performance of QCS is 110.2± 2.41. For the halfcheetah-m-e dataset the performance of QT is 94.0±0.2 the performance of QCS 93.3±1.78. These results are within one standard deviation. 

*“When learning $Q_\theta$ from such limited information, where the values at the narrow action points are almost identical for each given state, it is observed that the learned $Q_\theta$ tends to be over-generalized to the OOD action region.”*

Where was this observed? It would be good to have a reference here if this observation comes from prior work.

How would the results look like for POR and QT if you were reporting results as the prior work did, i.e. selecting the best score throughout the whole training process as you confirmed with the QT author through email, instead of the last running average score? Before changing the metrics we report the results, perhaps we can provide results in both metrics to be able to provide robust comparisons with prior work. 

While the question of “Why Does Q-Greedy Policy Struggle with Optimal Datasets?” is one of the main questions that the submission focuses on that takes up more than two pages in the main body of the paper, but in reality this question has been only investigated in one game in the MuJoCo environment presented in the Appendix. I do not think this is sufficient enough evidence that the experiments of the submission support the claim that is being made here.

Looking at the results reported in Appendix E, MuJoCo Hopper results, it seems that as the dimension of the action space increases the results reported in the main paper about the toy example becomes less relevant. 

I thank the authors for their rebuttal.  The submission needs substantial re-writing. I will keep my original score.
We are pleased to hear that our response has resolved your concerns. Thank you once again for leaving a thoughtful comment and for dedicating your time and effort to reviewing our work.
Thanks for the response addressing my concern. I increase my score to 6.
Adding on to our previous response, random selection of subgoals occurs during training to learn a generalized function for various optimal and non-optimal subgoals.  However, during evaluation, instead of using random subgoals, the agent always uses the fixed ultimate goal of the task, which is known for each task. We hope this helps to further clarify your concerns regarding our subgoal conditioning method.
Thank you for bringing up a good point. As we mentioned in our previous response, subgoal conditioning follows the prior work [Ref.7], and our explanation of the effect of the subgoal is summarized below.

Let's first reconsider return conditioned supervised learning. In the case of return conditioning, the agent basically learns $a = f_\theta(s, \hat{R})$, where $\hat{R}$ is the return-to-go, i.e., sum of rewards received from the current timestep until the end of each episode. During training, the joint function $f_\theta$ of $(s, \hat{R})$ is learned with   various $\hat{R}$ values from good or bad trajectories in the dataset.  Basically we learn the  function $f_\theta$ on the $(s, \hat{R})$ plane hoping for generalization over a wide region of $(s, \hat{R})$ with various $(s,\hat{R})$ data points.  Later during the test phase, we set a high target return-to-go as $\hat{R}$ so that  the model will output an action 
$a$ for each $s$ with  high return-to-go. That is, we use $a=f_\theta(s, \hat{R}\_{target})$, a function  from $s$ to $a$, where $\hat{R}\_{target}$ is a fixed high target value. 

Similarly, subgoal conditioning case can be considered to learn a function $a = f_\psi(s, \hat{s}\_{goal})$, where the subgoal $ \hat{s}\_{goal}$ is a randomly-picked state  between the next step and  the episode end  in each episode.  
If $\hat{s}\_{goal}$ is from a bad episode, the produced action will be bad.  If $\hat{s}\_{goal}$ is from a good episode, the produced action will be good. 
But, again, our goal is to learn an entire function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide  joint region of $(s, \hat{s}\_{goal})$, where $\hat{s}\_{goal}$ ranges from good to bad (hoping for interpolation and extrapolation) so that in the later test phase, we use this function $a = f_\psi(s, \hat{s}\_{goal})$ generalized over a wide  joint region of $(s, \hat{s}\_{goal})$ by fixing $\hat{s}\_{goal}$ to be the ultimate goal of the task, which is known for each task. 

We hope our additional response has addressed all your concerns. If your concerns have been resolved or if you have any further questions, please feel free to let us know.

&nbsp;

[Ref.7] Emmons, Scott, et al. "RvS: What is Essential for Offline RL via Supervised Learning?." ICLR 2022.
Thanks for the response. 

How randomly selected subgoals can always achieve to a good performance?