A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
Authors: Yunpeng Qing
Shunyu Liu
Jingyuan Cong
Kaixuan Chen
Yihe Zhou
Mingli Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Learning
Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO.

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
Yunpeng Qing', Shunyu Liu?; Jingyuan Cong', Kaixuan Ch Yihe Zhou', Mingli S.ng2.3
College of Computer Science and Technology. Zhejiang University
State Key Laboratory of Blockchain and Data Security, Zhejiang University
Hangzhou High-Teech Zone (Binjiang) Institute of Blockchain and Data Security fqingyunpeng, liushunyu, kcj51, chenkx, zhouyihe, brooksong) (ew.enu..eeeeuu..eee..o. edu. cn
Abstract
Offline reinforcement learning endeavors leverage offline datasets craft effee- tive agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the -ifdiitriitttinn problem. However, existing works often suffer from the constraint conflict issue when offline datasets collected from multiple behavior policies, Le., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weigh weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, introduce novel Advantage- Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data conditional variables. Then the agent can follow such dis- policy entangled action distribution constraints to optimize the advantage-awan towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at http://littubb.ooo/lllansooo////
Introduction
Offline Reinforcement Learning (RL) aims to learn effective control policies from pre- collected datasets without online exploration, and has witnessed its unprecedented success in various
real-world applications, including robot control [57 45 24 25), power grid control 2 49 26 36),
etc. A formidable challenge of offline RL lies in the DistribDisioottion (OOD) problem [1]] involving the distribution shift between data induced by the learned policy and data collected by the behavior policy. Consequently, the direct application of conventional online RL methods inevitably exhibits extrapolation error [34], where the unseen state-action pairs are erroneously estimated. To tackle this OOD problem, offline RL methods attempt to impose proper conservatism on the learning agent within the distribution of the dataset, such as restricting the learned policy with a regularization
term 19 9
penalizing the value overestimation of OOD actions 20 18
Despite the promising results achieved, offline RL often encounters the constraint conflict issue when dealing with the mixed-quality dataset [40 [2 D (28). Specifically, when training data are collected from multiple behavior policies with distinct returns, existing works still treat each sample constraint
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

[-10,
-
LAPO
400
LAPO VAE
(-9,0), 8
(11,
A2PO
A2PO VAEiE=
Optimal Retur
A2PO VAEIE- -1)
300
Returns
Dataset
10
200
Optimal Action
Action
45%
100
-5
46%
Sub- Optiaaa
Action
Action
0.0
Training Iterati ons (x10*)
0.6
1.0
Action Range
One-step Jump Task and Offfine Dataset
(b) Learning Curves of
(c) VAE Behavior Distributions
A2PO and LAPO
Figure 1: A didactic experiment. (a) The visualization of the toy one-step jump task and the composition of the mixed-quality dataset. The agent starts at position 0 and can make a one-step jump [-10, 10] reach new position and receive a reward r. (b) Learning curves of A2PO and LAPO. (c) VAE-generated action distributions of A2PO and LAPO at the initial state. LAPO VAE conditions only on the state, while A2PO VAE conditions on both the state and the advantage
equally with no regard for the differences in data quality and diversity. This oversight results in
improper constrain conflict actions [3] T6 50 15]. ultimately leading to further suboptimal
58 28 employ weighted sampling to prioritize training transitions with high advantage values from the offline dataset. However, we argue that these AW methods implicitly reduce the diverse behavior policies associated with the offline dataset into narrow one from the viewpoint of the dataset redistribution. As result, this redistribution operation of AW may exclude a significant number of critical transitions during training, imposing erroneous constraints on agent learning. To exemplify the above issue of AW, we conduct a didactic experiment on the recent advanced AW method, LAPO [5], as shown in Figure [] The toy one-step jump task requires the agent to jump over obstacles and reach two designated goal positions with different rewards. The offline dataset mainly contains failed attempts, with only few successful transitions, making it very challenging for the agent to learn an effective policy. The results in Figure demonstttee that LAPO performs poorly in this task. Furthermore, Figure reveals that the AW redistribution does not effectively prioritize either optimal or suboptimal actions modeling the behavior policy. Instead, the AW redistribution can lead to an incorrect on bad actions, which results in unreliable policy optimization.
outcomes. To resolve this concern, the Advantage- -eigggee (AW) methods 5 43
In this paper, we propose Advantage-Aware Policy Optimization, abbreviated as A2PO, to explicitly learn the advantage-aware policy with disentangled behavior policies from the mixed-quality offline dataset. Unlike previous AW methods devoted to dataset redistribution while reducing the data diversity, the proposed A2PO directly conditions the agent policy on the advantage values of all training data without any prior preference. Technically, A2PO comprises two alternating stages, behavior policy disentangling and agent policy optimization. The former stage introduces Condi- tional Variational Eitto-- (CVAE) [4] to disentangle different behavior policies into separate action distributions by modeling the advantage values of collected state- actt-nction pairs as conditioned variables. The latter stage further imposes an explicit advantage-aware policy constraint on the training agent within the support of disentangled action distributions. The advantage-conditioned CVAE can models the behavior policy distribution (Figure which is further utilized to construct advantage-awan constraint for agent optimization toward high advantage values, resulting in an lb). Ib).
effective docision-making policy (Figure
To sum up, our main contribution is the first dedicated attempt towards advantage-aware policy optimization to alleviate the constraint conflict issue under the mixed-quality offline dataset. The proposed A2PO can achieve advantage-aweer aware policy constraint derived from different behavior policies, where customized CVAE is employed to infer diverse action distributions associated with the behavior policies by modeling advantage values as conditional variables. Extensive experiments conducted on the D4RL benchmark [8. including both single-quality and mixed-quality datasets, demonstrate that the proposed APPO method yields significantly superior performance to other advanced offline RL basclines, as well as the advantage-weight competitors.

---

2 Related Works
Offline RL can be broadly classified into four categories: policy constraint [1] value regular-
ization 13 14| model-based method [53] [2,, and return-condiin condiinond supervised learning 7 23]. Policy constraint methods impose constraints on the learned policy to be close to the behavior policy 19 Previous studies directly introduce the explicit constraint on policy learning. such as behavior cloning [2], maximum mean discrepancy [19], or maximum likelihood estimation [48] In contrast, recent efforts [29 46 mainly focus on realizing the policy constraints implicitly by approximating the formal optimal policy derived from KI.-divergence constraint. On the other hand, value regularization methods make constraints on the value function to alleviate the overestimation of OOD action. Researchers try to approximate the lower bound of the value function with the O-regularization term for conservative action selection 20 27 Model-based methods construct the environment dynamies to estimate state-action uncertainty for OOD penalty [7] 6). Several works also converts offline RL into a return-conditione' supervised learning task. Decision Trans- former (DT) [3] builds transformer policy conditioned on both the current state and the additional sum return signal with supervised learning. Yamagata et al. [51] improve the stitching ability of DT policy on sub-optimal samples by relabeling the return signal with Q-learning results. However, in the context of offline RL with a mixed-quality dataset and no access to the trajectory return signals, all these methods treat each sample equally without considering data quality, thereby resulting in improper regularization and further suboptimal learning outcomes.
Advantage-weighted Offline RL Method employs weighted sampling to prioritize training transi- tions with high advantage values from the offline dataset. To enhance sample efficiency. Peng et al. 31 introduce advantage- weighted maximum likelihood loss by directly caleulating advantage values via trajectory return. [29] further use the critic network to estimate advantage values for advantage-weighted policy training. This technique has been incorporated
subroutine in other works [18 50) for agent policy extraction. Recently, AW methods have also been well studied in addressing the constraint conflict issue that arises from the mixed-quality dataset (5
58 40). Several
studies present Selvanttge-weightee behavior cloning as a direct objective function [58] or an explicit
policy constraint 332 (5) propose the Latent Serantage-Weigheee
Policy Optimization (LAPO)
framework, which employs an advantage-eiighted loss to train CVAE for generating high-advantage actions based on the state condition. Besides AW methods, Hong al. [15 enhance the classical offline RL training objective with the weight of subsequent return. On the other hand, Hong et al. [16] directly learning the optimal policy density as the weight function to enable sampling from high-performing policies. However, this AW mechanism inevitably diminishes the data diversity in the dataset. In contrast, our A2PO directly conditions the agent policy on both the state and the estimated advantage value, enabling effective utilization of all samples with varying quality.
3 Preliminaries
We formalize the RL task as a Markov Decision Process (MDP) [35 defined by a tuple M = (S, A, P.r.Y. Y.P.., Po), where S represents the state space, AA represents the action space, P: S SxAXS - [0, 1] denotes the environment dynamics, r : S x AA- + R denotes the reward function, 7 E (0. 1] is the discount factor, and Po the initial state distribution. At each time step f, the agent observes the state 8; E S and selects an action a; E A according to its policy æ, This action leads to a transition to the next state S based on the dynamics distribution P. Additionally, the agent receives a reward signal rz. The goal of RL is to learn an optimal policy x that maximizes the expected return:
o" = arg max, E, 1> > >r In offline RL, the agent can only learn from an offline dataset without online interaction with the environment. In the single-quality settings, the offline dataset D = {(s,, at Ti, st-1) f==1 1..... .N) with N transitions is collected by only one behavior policy Tg. In the mixed-qualig settings, the offline dataset D = U, ((s,,, 0i,t, 81,+11 t t= = .N]
is collected by multiple behavior policies {
We evaluate the learned policy x by the action value function Q* a)
E,, DEee 7'r(s), az) So = s, a). The state value function is defined as (* Ex- QQ (s.a), a)), while the advantage function is defined as (s, = Q* a) - (s). For
continuous control, our A2PO implementation uses the TD3 algorithm [I0 based on the actor-critic framework as a basic backbone for its robust performance. The actor network Tw, known as the
learned policy, is parameterized by w, while the critic networks consist of the Q-network Qa
3

---

Offline
Behavior Policy Disentangling
CVAE
M
(s,a)
Qa V, Critic
l
(o(t e)
Disentangled Action
Mrnes-Quvality Offiiine Doroect
pol-la,0)
Encoder
Decoder
Aral Tayl a)
Inserpple Actions
c e|l
xu(-le)
po(a|a,
Advaniage-awar Constmint
Louigaity
E'
Actor
Po(
Higha Quaalie
(19) A
c" li
mu(-le")
PalalF,e7
Optimal Policy Improvement
Highoqullit Policy
Agent Policy Optimization
Figure 2: An illustrative diagram of the Advantage Aware Policy Optimization (A2PO) method.
parameterized by and the V-network Vi parameterized by The actor-critic framework involves two steps: policy evaluation and policy improvement. During policy evaluation phase, the Q-network Qs is optimized by the temporal-differen
(TD) loss 42]:
Lo( =
(ma)))))) [Q((s, (
(r(s,a) + (Qg(d',
where and @ are the parameters of the target networks that are regularly updated by online parameters and w to maintain learning stability. The V-network Vi can also be optimized by the similar TD loss. For policy improvement in continuous control, the actor network Th be optimized by the deterministic policy gradient loss [39 38]:
L (w) = E.DD E.- Qo (s))]
(2)
Note that offline RL will impose conservative constraints on the optimization losses to tackle the OOD problem. Moreover, the performance of the final learned policy Tw highly depends on the quality of the offline dataset D associated with the behavior policies {
4 Methodology
In this section, we provide details of our proposed A2PO approach, consisting of two key components: behavior policy disentangling and agent policy optimization. In the behavior policy disentangling phase, we disentangle behavior policies with CVAE specifically modeling the action distribution conditioned on the advantage values of collected state-action pairs. By taking different advantage inputs. the newly formed CVAE allows the agent to infer distinct action distributions that are as- sociated with various behavior policies. Then in the agent policy optimization phase, the action distributions derived from the advantage condition serve as disentangled behavior policies, estab- lishing an advantage-aware policy constraint to guide agent training. An overview of our A2PO is illustrated in Figure 2
4.1 Behavior Policy Disentangling
To realize behavior policy disentangling, adopt CVAE to relate the behavior distribution of different specific behavior policies T3. to the advantage condition variables, which is quite different from previous methods [II] (56) utilizing CVAE only for approximating the overall mixed-quality behavior policy set { conditioned only on specific state s. Concretely, we have made adjustments to the architecture of the CVAE to be advantage-aware. The encoder q (z)a, c) is fed with condition and action a to project them into a latent representation z. Given specific condition and the encoder output the decoder (a/2,, e) captures the correlation between condition e and latent representation 2 to reconstruct the original action a. Unlike previous meth- ods [ 5 48 predicting action solely based on the state s, consider both state and advantage value & for CVAE condition. The atate-ad condition is formulated as:
E.
(3)

---

Therefore, given the current state and the advantage value & as joint condition, the CVAE model is able to generate corresponding action a with varying quality positively correlated with the advantage value ç. For state-action pair (s, a), the advantage value & can be computed as follows:
tanh min Qe, (s. Vo(s)),
(4)
where two Q-networks with the min(-) operation are adopted to ensure conservatism in offline RL settings IT. Moreover, we employ the tanh(.) function to normalize the advantage condition within the range of (-1,1). This operation prevents excessive outliers from impacting the performance of CVAE, improving the controllability of generation. The optimization Q-networks and V-network will be described in the following section.
The CVAE model is trained using the state-advantage condition e and the corresponding action a. The training objective involves maximizing the Empirical Lower Bound (ELBO) 41 on the log-likelihood of the sampled minibatch:
(C, )) EE (alo,)) [lg(ps (a|2, c))] + KL
KL (9p(2)a, p(=)]]
(5)
where is the coefficient for trading off the KL-divergence loss term, and p(z) denotes the prior distribution of setting to be (O,, 1). The first log- log-liielhhood term encourages the generated action to match the real action as much as possible, while the second KL divergence term aligns the latent variable distribution with the prior distribution p(z).
At each round of CVAE training, minibatch of state-action pairs (s, a) is sampled from the offline dataset. These pairs are fed to the critic network Qo and V to get corresponding advantage condition C by Eq. 4 Then the advantage-aware CVAE is subsequently optimized by Eq. (5). By incorporating the ad advantage condition ç. the CVAE captures the relation between & and the action distribution of the behavior policies, as shown the upper part of Figure 2 This further enables the CVAE to generate actions based on the state-advantage condition in a manner where the action quality positively correlated with ç. Furthermore, the advantage-awan aware CVAE is utilized to establish an advantage-awea aware policy constraint for agent policy optimization in the next stage.
4.2 Agent Policy Optimization
The agent is constructed using the actor-critic framework [42]. The critic comprises two Q-networks Q and one V-network V₂ to approximate the value of the agent policy. The actor, advantage- aware policy (-lc). with input e = s sI| e. generates a latent representation & based on the state s and the designated advantage condition &. This latent representation ž, along with c, is then fed into the decoder pq: for an recognizable action ag:
0g ~ ( (2,),, where ~ (: lc).
(6)
With this form, the advantage-aware policy Fw is expected produce action with different qualities that are positively correlated to the designated advantage input which is normalized within (-1, 1) in Eq. 4). Therefore, the output optimal action a" is obtained by C* s 5* input with e = 1. It should be noted that the critic networks are to approximate the expected value of the optimal policy T..(-lc") The agent optimization, following the actor-critic framework, encompasses policy evaluation and policy improvement steps. During the policy evaluation step, the critic is updated through the minimization of the temporal difference loss with the optimal policy (-(c") Specifically, for the V-network vg. we employ the one-step Bellman operator to approximate the state value under the current agent-aware policy, conditioned on the optimal advantage input 5* = 1, as follows:
LTD = E (s.a. [ y min Qo (d',at) Vo())
3" (-ee)
dp-po(13".c)
where Qg is the target network updated softly. As for the Q-networks, both of the two Q-network entities Qo, are optimized with agent policy (-(c") following Equation I
For the policy improvement, the actor loss is defined as:
Exe(w) = - EAND. EA (o"") [Qo, (s, oz)] + E (p a.)
(8)
a- (e))
a (18.0)
5

---

where o² in the first term is the optimal action generated with the fixed maximum advantage condition E* = 1 input and ag in the second term is obtained with the advantage condition & derived from the critic based on Eq. 4 applied to the sampled batch. Meanwhile, following TD3+BC [9], we add a normalization coelficient À a/ Lis..a.) [Q(s,, d;)l) to the first term to keep the scale balance between Q value objective and regularization, where a is a hyperparameter to control the scale of the normalized Q value. The first term encourages the optimal policy condition on c" to select actions that yield the highest expected returns represented by the Q-value. This aligns with the policy improvement step conventional RI. approaches [22]. The second behavior cloning term explicitly imposes constraints on the advantage-aware policy, ensuring the policy selects in in-sample actions that adhere to the advantage condition & determined by the critic. Therefore, the suboptimal samples with low advantage condition will not disrupt the optimization of optimal policy Tw (-lc)). And they enforce valid constraints on the corresponding policy (-cc), as shown in the lower part of Figure It should be noted that the decoder Pú is fixed during both policy evaluation and improvement.
Our A2PO implementation selects TD3+BC as the base backbone for its robustness. The general framework derived above is thoroughly described in Appendix A
5 Experiments
To illustrate the effectiveness of the proposed A2PO method, we conduct experiments on the D4RL
benchmark [8]. We aim to answer the following questions: (1) Can A2PO outperform the advanced
offline RL methods in both the single-quality datasets and mixed-quality datasets? (Section 5.2
(2) How do different components of A2PO contribute to the overall performance? (Section 5.3 and
Appendix D G (3) How does A2PO perform under mixed -uxlity datasets with varying single- quality samples? (Section 5.5 and Appendix H (4) Can the A2PO agent effectively estimate the
quality of different transitions? (Section 5.4 and Appendix I (5) How does the time overhead of
A2PO compare to other baselines? (Section 5.6
5.1 Experiment Settings
Tasks and Datasets. We consider four different domains of tasks in D4RL benchmark [8] Gym, Maze, Adroit, and Kitchen. Each domain contains several tasks and corresponding distinct datasets. We conduct experiments for each Gym task using single-quality and mixed-quality datasets. The single-quality datasets are generated with the medium behavior policy. The mixed-quality datasets are the combinations of random, medium, and expert single-quality datasets, including medium-experr, medium-expert, and
medium-reee replay, only includes the first two mixed-quality datasets. Thus, following Hong et al. [6 [15,, we manually construct the last three mixed-quality datasets by combining the corresponding single-quality datasets in D4RL with equal proportions. For the other domains of tasks, the corresponding D4RL datasets exhibit a significant level of diversity to evaluate the effectiveness of our A2PO algorithm.
The D4RL benchmark
Comparison Methods and Hyperparameters. We compare the proposed A2PO to several advanced offline RI. methods: BCQ [1I] TD3+BC [9. CQL 20). EQL. (50) especially the advantage-weightedd
offline RI. methods: AWAC 29], IQL. 18, CQL+AW (.S. LAPO [5). Besides, we also select the vanilla BC method [33] the model-based offline RL method MOPO [53], and the emerging diffusion- based method Diffusson-QU QL [47), for comparison. We report the performance of baselines using the best results reported from their own paper. More comparison results can be found in Appendix C
The detailed hyperparameters of A2PO are given in Appendix B.2
5.2 Comparison on D4RL Benchmarks
Results for Gym Tasks. The experimental results of all compared methods in the D4RL Gym tasks are presented in Table For the single-quality medium dataset and mixed-qualit medium-expi expert and medium--eplan datasets from D4RL, our A2PO achieves state-of-the art results with low variance. Meanwhile, both conventional offline RL approaches like EQL and advantage- addintage-weighhed weighted approaches like LAPO still learn acceptable policy, indicating that the conffict issue hardly occurs in these datasets with low diversity. However, the newly constructed mixed-quality datasets, namely random-medium undom-expert and random-men mediom--dddm--eeeee highlight the issue of substantial gaps between behavior policies. The results these datasets reveal significant drop in performance for all other baselines.

---

Table 1: Test returns of our proposed A2PO and baselines on the Gym tasks. + corresponds one standard deviation of the performance on 5 random seeds. The performance is measured by the normalized scores at the last training iteration. Bold indicates the best performance in each task.
Source
Task
BC
BCQ
TD3+BC CQL MOPO
EQL Diffusion-O0 QL AWAC
IQL
CQL+AW LAPO| A2PO (Ours)
hatfcheetah
42.6
47.0
48.3
44,0
42.3
47.2
51.1
43.5
47.4
49.0
46.0
40.
medium
bopper
52.9
56.7
59.3
58.5
28.0
74.6
90.5
57.0
66.3
71,0
51.6
80,3:40
walker2d
75.3
72.6
83.7
72.5
17.8
83.2
87.0
72.4
78.3
83.0
80.8
84,9202
halfcheetah
36.6
40.4
44.6
45.5
53.1
44.5
47.8
44.2
47.0
41,9
44,8
medium replay
hopper
18.1
53.3
60.9
95.0
67.5
98.1
101.3
37.2
94.7
99.0
50.1
101.6:13
walker2d
26.0 52.1
81.8
81.6
39.0
76.6
95.5
27.0
73.9
87.0
60.6
82.8
mediumm expert
halfcheetah
55.2 89.1
90.7
91.6
63.3
90.6
96.8
42.8
86.7
84.0
95.6.00.
hopper
52,5
81.8
98.0
105.4
23.7
105.5
111.1
55,8
91.5
91.0
111.0
113..4.0
walker2d
107.5
109.0
110.1
108.8
44.6
110.2
110.1
74.5
109.6
109.0
110.9
112.
halfcheetah)
2.3
12.7
47.7
31,9
52.7
42.3
48.4
46.5
42.2
46.5
48.5
random
medium
hopper
23.2
9.2
7.4
3.3
19.9
1.7
6.9
19.5
6.2
22.6
4.2
62.1-28
walker2d
19.2
0.2
10.7
0.2
40.2
31.4
0.0
54.6
82.0
23.6
82,3=04
random
halfcheetah)
13.7
2.1
43.1
15.0
18.5
47.4
86.1
28.6
80.7
90.3
hopper
10.1
8.5
78.8
7.8
17.2
68.6
102.0
84.7
58.5
109.6
82.3
112.5
expert
walker2d
14.7
0.6
7.0
0.3
4.6
9.1
56.3
11.7
90.9
108.6
0.4
109.1
random
halfcheetah)
2.3
15.9
62.3
13.5
26.7
42.8
81.2
61.6
76.8
90,621.
medium
hopper
27.4
4.0
60.5
9.4
13.3
72.4
70.1
8.6
57.9
71.8
66.6
107.8.044
expert
walker2d
24.6
2.4
15.7
0.1
56.4
61.0
56.6
90.8
58.3
97.7-66
Gym Total
[604.2 657.6
1010.6
784.4
628.8
1107.2
1302.1
710.9 1183.9
1376.9
1026.8]
1563.3
Table 2: Test returns of our proposed A2PO and baselines on the Maze, Kitchen, and Adroit tasks.
Task
BC
BCQ
TD3-BC CQL
MOPO
EQL Diffusion-O1 QL
AWAC
IQL
CQL+AW
LAPO A2PO (Ours)
mazedd uunaaee
0.5
24.8
24.2
5.7
-15.4
56.5
66.7
94.5
56.2
19.6
78.0
133,3:98
mazeld medium
0.7
22.5
31.5
5.0
19.0
36.3
100.6
31.4
25.7
22.6
43.2
114.9.179
mazedd -aagge
1.1
43.0
128.5
12.5
-0.5
$7.0
116.3
43,9
45.7
10.3
69.7
156.4158
antmaze umare diverse
45.6
55.0
71.4
84.0
0.0
50.8
66.2
49.3
62.2
$4.0
0.0
72.6:103 102
antmace medium diverse
0.0
0.0
3.0
53.7
0.0
62.2
78.6
0.7
70.0
24.0
30.2
80,2:40
antmaze-large large- diverse
0.0
2.2
0.0
14.9
0.0
38.0
56.6
1.0
47.5
40.0
22.3
52.1 179
Maze Total
47.9
147.5
260.6
175.8
3.1
300.8
485.0
220.8
307.3
170.5
243.4
609.5
kitchen-complete
33,8
8.1
0.8
43.8
40.1
70.3
84,0
3.8
62.5
30,2
53.2
69 29..2:44
kitchen-partial
33.8
18.9
0.0
49.8
6.7
74.5
60.5
0.3
46.3
36.0
53.7
75.8:24
kitchon-mixed
47,5
10.6
0.8
51.0
17.3
55.6
62.6
0.0
51.0
50,5
62.4
64.2s3.1
Kitchen Total
115.1
37.6
1.6
144.6
64.1
200.4
207.1
4.1
159.8
116.7
169.3
209.2
pen-human
24.4
12.3
-3.7
37.5
54.6
44.3
72.8
4.3
71.5
-3.0
68.1
68. 68.9a59
pen cloned
56.9
28.0
1.7
39.2
10.7
46.9
$7.3
..8
37.3
-255
55.8
85.0.2.3
Adroit Total
91.3 40.3
2.0
76.7
65.3
91.2
130.1
3.5
108.8
-5.5
123.9
153.9
Instead, our A2PO continues to achieve the best performance on these datasets. When considering the total scores across all datasets, A2PO outperforms the next best-performing AW method, CQL+AW, by over 21 %. The results reveal the exceptional ability of A2PO to capture and utilize high-quality interactions within the dataset in order to enforce a reasonable advantage-aware policy constraint and further obtain an optimal agent policy.
Results for Maze, Kitchen, and Adroit Tasks. Table 2 presents the experimental results of all the compared methods on the D4RL Maze, Kitchen, and Adroit tasks. The D4RL datasets for these tasks exhibit varying patterns in behavior policy samples. For instance, the Antmaze datasets highly sub-optimal, while the Adroit datasets have a narrow state-action distribution. Among the offline RL baselines and AW methods, A2PO delivers remarkable performance in these challenging tasks and showcases the robust representation capabilities of the advantage-aware policy.
5.3 Ablation Analysis
Different Advantage condition during training. The performance comparison of different advan- tage condition computing methods for agent training is given in Figure 3 Eq.
4 obbains continuous advantage condition E in the range of (-1, 1). To evaluate the effectiveness of the continuous com- puting method, we design discrete form of advantage condition: Eass = sgn(E) (f) l where sgn(.) the symbolic function, and lggs> is the indicator function returning 1 if the absolute value of is greater than the hyperparameter of threshold e, otherwise 0. Thus, the advantage condition Eass is constrained to discrete value of (- 1). Another special form of advantage condition is Eix = for all state-action pairs, in which the advantage-awer ability is lost. Figure 3a shows that setting
7

---

Return Diference
Retur Diference
Retur Diffrencee
Return Difference
-50
-50
-50
-50
-75
-75
-75
-75
Etx
Eai with 0.0
(c) Eas with 0.1 (d) Edi with 0.5
Figure 3: Test return difference of A2PO with different discrete advantage conditions during training compared with original A2PO with continuous advantage condition during training. Task abbrevia-
tions are listed in Appendix B.l Test returns are reported in Appendix D
nt: 1s.5=10)
nt. (s.5=00)
nl- =.E==10)
Normalized Retums
1000
Normaltzee marmaarreed Return
1000
Normalized lozd Returns
100
Norma Returs
200
50
29
25
8.0 0.2 0.4 0.6 0.6 1.0
8.0
0.2
0.4 0.6 0.8 1.0
8.0 0.2 0.6 0.8 1.0
8.0 0.2 0.4 0.6 0.8 100
Training Iterations (x10%)
Training lterations (x10*)
Training Iterations (x10*)
Training iterations (x10f)
(a) halfcheetah (medium-expert)
(b) halfcheetah
(c) hopper (medium-expertt)
(d) hopper (random- mrinloo-mmediu--xpprte
(randon--medium-exppr))
Figure Learning curves A2PO under different fixed advantage inputs during the test while using the original continuous advantage condition for training. Test returns are reported in Appendix E
Eix without explicitly advantage-awand mechanism leads to significant performance decreasing, especially in the new mixed-quality dataset. Meanwhile, Eai with different values of threshold achieve slightly inferior results than the continuous &. This outcome strongly supports the efficiency of continuous Ç Although the Edis signals are more stable, Eais hidden the concrete advantage value, causing mismatch between the advantage value and the sampled transition.
Different Advantage Condition for Test. The performance comparison of different discrete advan- tage conditions input for the test is given in Figure 4 To ensure clear differentiation, we select from (- }). The different designated advantage conditions & are fixed input for the actor, leading to different policies Tw (-|s. 5). The final outcomes demonstrate the partition of returns corresponding to the policies with different Ç. Furthermore, the magnitude of the gap increases as the offline dataset includes samples from more diverse behavior policies. These observations provide strong evidence for the success of A2PO disentangling the behavior policies under the multi-quality dataset.
1.0
100
1.0
100
0.5
80
0.5
80
0.0
0.0
40
-0.5
-0.5
20
20
1.0
1.0
(a) walker2d
(b) walker2d
(c) hopper
(d) hopper
(Advantage condition)
(Actual Return)
(Advantage condition)
(Actual Return)
Figure 5: Visualization of A2PO latent representation after applying PCA with different advantage conditions and actual returns in the walker2d- meduurrr--m and hopper-medium--eene -replay tasks. Each data point indicates a latent representation Σ based on the initial state and different advantage conditions sampled uniformly from [-1, 1). The actual return is measured under the corresponding sampled advantage condition. The value magnitude is indicated with varying shades of color.

---

5.4 Visualization
Figure D presents the visualization of A2PO latent representation. The uniformly sampled advantage condition & combined with the initial state s, are fed into the actor-network to get the latent representa- tion generated by the final layer of the actor. The result demonstrates that the representations converge according to the advantage and the actual return. Moreover, upon comparing Figure 5(a,b,, as well as Figure 5(c,d) separately, we observe that the latent action representation follows the same alteration pattern based on the actual real return. These observations demonstrate that our advantage-aware policy effectively capture policies with different returns by the designated advantage input &. This provides compelling evidence for the efficacy of the A2PO policy construction. More experiments of advantage estimation conducted on different tasks and datasets are presented in Appendix !
5.5 Robustness
Figure 6 presents the experimental results of A2PO across mixed-quality datasets with varying proportions of single-quality samples.
== 0%
d= 20%
== 40%
a= 100%
10%
30%
50%%
Following the methodology of [15] [6], we evaluate the effectiveness of A2PO on three mixed-quality datasets: medium-expent expert, random-
Normalized Returns
100
medium, and random-expert. These datasets consist of total number of x 10 transitions. We vary the proportions a of higher quality sam- ples and (1-o) of lower quality samples. The re- sults demonstrate that our A2PO effectively cap- tures and infers high-quality potential behavior policies for proper policy regularization, even with small proportion of high-quality samples. Additionally, as the a becomes larger, the vari- ance decreases. Thus, A2PO demonstrates its robustness in handling variations in the propor- tions of different single-quality samples, guar- anteeing consistently high performance.
Halfcheetah
Hopper
Walker2d
Figure 6: Compare the returns of A2PO under random-experi dataset with different high-quality data proportions in the Gym tasks. Detail returns are reported in Appendix
5.6 Time Overhead
We measure the training times of A2PO as well as other 300
baselines, which are presented in Table 7 The experi- ments are performed on cluster of A40 GPUs under sercheetah-mediun-eepeee
oveneed
scenarios for x 10 steps.
100
Although the A2PO runtime is longer than the lightweight algorithms like IQL due to CVAE training, our A2PO is more efficient compared to other AW methods such LAPO and CQL+AW.
BC
BC9
TD3+ at
+C
Eat.
CoL coL
AW
AMAC
sat
LAPO Azpo
Figure 7: Compare the time overhead of A2PO and other baselines.
6 Conclusion
In this paper, we propose novel approach, termed as A2PO, to tackle the constraint conflict issue on mixed-quality offline datasets with advantage-aware policy constraints. Specifically, A2PO utilizes CVAE effectively disentangle the action distributions associated with various behavior policies. This is achieved by modeling the advantage values of all training data as conditional variables. Consequently, advantage-awe agent policy optimization can be focused on maximizing high advantage values while conforming to the disentangled distribution constraint imposed by the mixed-quality dataset. Experimental results show that A2PO successfully decouples the underlying behavior policies and significantly outperforms advanced offline RI. competitors. For our future work, we will extend A2PO to multi-task offline RI. scenarios characterized by a greater diversity of behavior policies and more prominent constraint conflict issue.

---

Limitations. The limitation of A2PO is that it incorporates CVAE during training, which may lead to quite large time overhead. However, the results presented in Section 5.6]show that the time overhead of A2PO remains reasonably acceptable when compared to other baseline methods.
7 Acknowledgement
This work was supported part by the Hangzhou Joint Funds of the Zhejiang Provincial Natural Science Foundation of China under Grant No. LHZSD24F020001, in part by the Fundamental Research Funds for the Central Universities under Grant No. 226-2024-00058. and in part by the Zhejiang Province High-Leveel Talents Special Support Program "Leading Talent of Technological Innovation of Te'TTe-Theeee Ten-Thousands Talents Program" under Grant No. 2022R52046.
References
[1] Y. Chebotar, Q. Vuong, A. Irpan. K. Hausman, F. Xia, Y. Lu, A. Kumar, T. Yu, A. Herzog, K. Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. arXiv preprint arXiv:2309. 10100000000100 2023.
[2] K. Chen, W. Luo, S. Liu, Y. Wei, Y. Zhou, Y. Qing. Q. Zhang. J. Song, and M. Song. Powerformer: A section-adaptive transformer for power flow adjustment. arXiv preprint arXiv:2401.02771, 2024.
[3] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Annual Conference Neural Information Processing Systems, 2021.
[4] X. Chen, Z. Zhou, Z. Wang, C. Wang, Y. Wu, and K. Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2020.
[5] X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y. Gao, W. L. Bin, C. Finn, and C. Zhang, Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2022.
[6] C. Diehl, T. S. Sievernich, M. Krüger, F. Hoffmann, and T. Bertram. Uncertaint--wware model- based offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters, 8(2):1167-1174, 2023.
[7] S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rys: What essential for offline rl via supervised learning? arXiv preprint 12X 2021.
2021.
[8] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[9] Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. In Annual Conference Neural Information Processing Systems, 2021.
[10] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning. 2018.
[11] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In International Conference on Machine Learning. 2019.
[12] C. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, and Y. Yu. Act: Empowering decision transformer with dynamic programming via advantage conditioning. arXiv preprint 055555 2023.
[13] K. Ghasemipour, S. Gu, and O. Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. In Annual Conference Neural Information Processing Systems, 2022.
[14] Hong, A. Kumar, and Levine. Confidence-conditionee value functions for offline reinforce- ment learning, arXiv preprint arXiv:2212.046017 2022.
10

---

[15] Z.-W. Hong, P. Agrawal, R. T. d. Combes, and R. Laroche. Harnessing mixed offline reinforce- ment learning datasets via trajectory weighting, arXiv preprint arXiv:2306.1308. 2023.
[16] Z. Hong, A. Kumar, S. Karnik, A. Bhandwaldar, A. Srivastava, J. Pajarinen, R. Laroche, A. Gupta, and P. Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. In Anmual Conference on Neural Information Processing Systems, 2023.
[17] M. O. Karabag and U. Topcu. On the sample complexity of vanilla model-based offline reinforcement learning with dependent samples. arXiv preprint arXiv:2303. 042668, 2023.
[18] Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2 2021.
[19] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via hootstrapping error reduction. In Annual Conference on Neural Information Processing Systems, 2019.
[20] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2020.
[21] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
[22] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (tt.:0299999711 2015,
[23] H. Liu and P. Abbeel. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv:2305. 16554, 2023.
[24] S. Liu, Y. Qing, S. Xu, H. Wu, J. Zhang, J. Cong, T. Chen, Y. Liu, and M. Song. Curricular subgoals for inverse reinforcement learning. arXiv preprint arXiv:2366.08232 2023.
[25] S. Liu, Y. Zhou, J. Song. T. Zheng, K. Chen, T. Zhu, Z. Feng, and M. Song. Contrastive identity- aware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11595-11603, 2023.
[26] S. Liu, W. Luo, Y. Zhou, K. Chen, Q. Zhang, H. Xu, Q. Guo, and M. Song. Transmission interface power flow adjustment: A deep reinforcement learning approach based on multi-task attribution map. JEEE Transactions on Power Systems, 39(2):3324-3335, 2024.
[27] J. Lyu, X. Ma, X. Li, and Z. Lu. Mildly conservative q-learning for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2022.
[28] Y. Mao, C. Wu, X. Chen, H. Hu, J. Jiang, T. Zhou, T. Lv, C. Fan, Z. Hu, Y. Wu, et al. Stylized offline reinforcement learning: Extracting diverse high-quality behaviors from heterogeneous datasets. In International Conference on Learning Representations, 2023.
[29] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint S.Ki:::00060933550 2020.
[30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Z. Yang. Z. Veti,, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, perforeefore deep learning library. In Annual Conference on Neural Information Processing Systems, 2019.
[31] X. B. Peng, A. Kumar, G. Zhang, and S. Levine. advantage-weighhen regressioo Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[32] Z. Peng, C. Han, Y. Liu, and Z. Zhou. Weighted policy constraints for offline reinforcement learning. In AAAI Conference on Artificial Intelligence, 2023.
[33] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3(1):88-9) -788889-111777 1991.

---

[34] R. F. Prudencio, M. R. Maximo, and E. L. Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[35] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[36] Y. Qing, S. Liu, J. Song, H. Wang, and M. Song. A survey on explainable reinforcement learning: Concepts, algorithms, challenges. arXiv preprini (ttp::22211.666555 2022.
[37] Y. Ran, Y-C. Li, F. Zhang, Z. Zhang, and Y. Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717. PMLR, 2023.
[38] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprini arXiv: (efi::77770634 2017.
[39] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 2014.
[40] A. Singh, A. Kumar, Q. Vuong, Y. Chebotar, and Levine. Offline rl with realistic datasets: Heteroskedastictyy and support constraints. arXiv preprint arXiv:2211.01052, 2022.
[41] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Annual Conference on Neural Information Processing Systems, 2015.
[42] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] Q. Tian, K. Kuang, Liu, and Wang. Learning from good trajectories in offline multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence, 2023.
[44] Q. Vuong, A. Kumar, S. Levine, and Y. Chebotar. Dasco: Dual-generator adversarial support constrained offline reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2022.
[45] A. Wagenmaker and A. Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference Machine Learning, 2023.
[46] S. Wang. Q. Yang. Gao, M. G. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang. Train once, get family: State-adaptive balances for offline-to-online reinforcement learning. arXiv preprint arXiv:2310.17990 17966, 2023.
[47] 7.. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.00 063,,,1996 2022.
[48] J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems, 2022.
[49] Xu, S. Liu, Y. Qing, Y. Zhou, Y. Wang, and M. Song. Temporal prototype-aware learning for active voltage control on power distribution networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024.
[50] H. Xu, L.. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv: /5810, 2023.
[51] T. Yamagata, A. Khalil, and R. Cantos-Rodrighe Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference Machine Learning. 2023.
[52] Yang, X. Chen, S. Wang, and Zhang. Model-based offline policy optimization with adversarial network. arXiv preprint arXiv:230.002157, 2023.
12

---

[53] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Annual Conference on Neural Information Processing Systems, 2020.
[54] Y. Yue, B. Kang, X. Ma, Z. Xu, G. Huang, and S. Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:2210.09241, 2022.
[55] Q. Zheng, M. Henaff, B. Amos, and A. Grover. Semi-supervised offline reinforcement learning with action-free traeectories In International Conference Machine Learning, 2023.
[56] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2021.
[57] Y. Zhou, S. Liu, Y. Qing, K. Chen, T. Zheng, Y. Huang, J. Song, and M. Song. Is centralized training with decentralized execution framework centralized enough for marl? arXiv preprint arXiv:2305. /7552, 2023.
[58] Z. Zhuang. K. Lei, J. Liu, D. Wang, and Y. Guo. Behavior proximal policy optimization. arXiv preprint 23Xi1:2302.11312, 2023.
13

---

A Pseudocode
To make the proposed A2PO method clearer for readers, the pseudocode is provided in Algorithm)
Algorithm 1 Advantage-Aware Policy Optimization (A2PO)
Input: offline dataset D, CVAE training step K, total training step T, soft update rate т. Initialize: CVAE encoder QG and decoder Pvs actor network Rws critic networks Qo and Va-
for 1 to T' do
Sample random minibatch of transitions B = {(,s((s,,, a,s,,,,,,)) D. Calculate & = tanh(min;=11,2 Qo, (8, a) (=())), = == =
# Behavior Policy Disentangling
if K then
Optimize CVAE encoder 4o and decoder Pv: according to Eq. as
(evAE(Fs, v)) -ED (=1a,))) [lg((p KL (a. [(=))]]
end if
Agent Policy Optimization
Optimize critic networks Qo and Vo according to Eq. and Eq. 7 as
Lw,(, Φ) = E (~))
Σ | Ir+y min (s)., min (s', -V))
(tle") of ppmp ,*,
Optimize actor network The according to Eq. Blas
Lac(w) - A
Qm (8, (8. + (s,)) (-
[(a az)²].
(1
Update the target networks with 0 - 1-11 (1-T)0 + r0, 6 (1 (1- TOb
end for
B Experiment Details
B.1 Task Abbreviations and Task Versions
In order to improve the readability and conciseness, we adopt abbreviations for the Gym tasks throughout the main text. The corresponding abbreviations for each task are provided in Table 3 For the specific task versions, we use the "-v2' version for the Gym tasks, the '-vl' version for the Maze2d tasks, the "-v0' version for the Antmaze tasks, the "-v0' version for the Kitchen tasks, the '-y1' version for the Adroit tasks.
Table 3: The abbreviation of the corresponding Gym task and dataset.
Dataset
halfcheetah
hopper walker2d
random medium
he-r
h-r
w-r
he-m h-m w-m
expert
he-e h-e w-e
medium-replay
he-m-r h-m-r w-m-r
medium-expert
he-m-e
h-m-e w-m-e
random-medium
he-r-m
h-r-m w-r-m
random- runeot-expert
he-m-e
h-m-e w-m-c
random-medius
medium-expert
he-r-m-e
h-r-m-e
w-r-m-e
14

---

B.2 Implementation Details
In this section, we provide the implementation details of our experiments. We conducted our experiments using PyTorch 3.8 [30] on cluster of 4 A40 GhUs.The source code will be made publicly available upon the publication of this paper.
The critic, actor, and CVAE networks are all constructed using a 2-layer MLP with 256 hidden units. They are updated using the Adam optimizer with a learning rate of 3 10 Additionally, we employ a target critic network with a soft-update rate of 5 x 10 Following the TD+3BC approach [9], we incorporate normalization, policy noise, and policy clipping during the training process. or the hyperparameter value in Q normalization, we adopted the strategy proposed by Wang et al. [47], selecting values of 5.0 and 6.0 for the antmaze- waaaiemm/dadaedddiiiiiii diverse tasks respectively robust and stable Q Learning, In contrast, we chose a = 0.005 for the kitchen- kitchen-partial task to facilitate policy regularization. For other tasks, setting a = 1.0 yields state-of-the-an results. As for the CVAE step K, 0.2M steps demonstrate sufficient robustness for most tasks, while the optimal settings for the -uw//2ddmmmar/mmmiiu tasks are 0.4M and 0.1M steps, respectively. Among the tasks, the continuous form of the advantage signal Ç. described in Eq. + consistently produces outstanding results across most scenarios. However, the advantage distribution may vary significantly across all the tasks, necessitating the fine-tuning of advantage computation methods based on online evaluations. The maze2d-umaze task requires a discrete form with E = 0.1. In the case of the maze2d-mediun task, we utilize
promote
value of 0.2. For both the maze2d- and antmaze- umaze tasks, we increase this to e = 0.8. In contrast, for the kitchen-partiall partial/mixxd tasks, we adopt a value of c = 0.0. Further details on this sensitivity can be found in Section 5.3 and Appendix D
Since all of the original papers of other baselines do not provide the complete results of all the tasks, we make implementation of the baselines. The CQL,, IQL and MOPO baselines are implemented using the implementations provided at respectively. The remaining baselines, including BCQ,
ggwhomas/igl-pytonn
and
github.com/yihaosuu 124/OlflincRLL---
CQL. TD3+BC, EQL, Diffusion-O QL, AWAC. CQL+AW, and LAPO, are implemented using the original implementations provided by the authors of the respective papers. These implementa- tions can be found at: BCQ gitub..oo//stjim/BCC) TD3+BC ggthub.com//tujjmm BC Diffusion-QU
EQL
github.com/ZZhendngg Wang//iffuunii
folicies-for-ffflini
AWAC ggthub.ccomrall-beeeeee.ec
COL+AW
and LAPO juthub.com/ppchen///l.PPP
of
C Additional Comparisons with More Baselines
In this section, we have added the data rebalance baseline ReD 54 novel policy-constrain baseline PRDC (37) and return-onniitione baselines Decision Transformer and %BC 31 the comparison baselines to further evaluate the superiority of our A2PO. The results are illustrated in Table Our A2PO method demonstrates comparable performance to various offline RL methods. While the performance of A2PO is slightly inferior to that of PRDC in the medium and medium-replay dataset, surpasses all additional baselines in the majority of scenarios.
D Analysis of Advantage Condition Input for Training
In this section, we provide full comparison of different advantage condition computing methods for training on Gym tasks in Ta These computing methods are thoroughly described in Section 5.3 The A2PO algorithm degenerates into TD3+BC under Êix which treats each sample constraint equally with no regard for the differences in the data quality. In this case, it achieves significantly worse performance compared using varied & values, particularly on the mendom-expert datasets which highlight the substantial gap between the behavior policies. This observation highlights the potential risks of constraint conflict issues in policy regularization methods. On the other hand, the discrete Eass yields slightly inferior performances and larger variances compared to the continuous Ç, when different threshold values e are used. While the Ças signals demonstrate more stability, the discretization process fails to accurately capture the advantage value associated with each sampled transition. Consequently, there a mismatch between the advantage value and the actual sampled transition, which negatively impacts the overall algorithm performance. However, by utilizing the
15

---

Table 4: Test returns of proposed A2PO and additional baselines on the locomotion tasks. The italic results the baselines are obtained from the corresponding original paper.
Source
Task
%BC
DT
IQL+ReD
PRDC
A2PO (Ours)
halfcheetah
42.5
42.6
47.6
63.5
47.1
medium
hopper
56.9
67.6
66.0
100.3
80.3=4.0
walker2d
75.0
74.0
78.6
85.2
84.9=022
halfcheetah
40.6
36.6
44.3
55.0
44.8=022
medium replay
hopper
75.9
82.7
101.0
100.1
101.6=13
walker2d
62.5
66.6
79.5
92.0
82.8
halfcheetah
92.9
86.8
92.6
94.5
95.6=005
medium expert
hopper
110.9
82.7
101.0
109.2
113.4
walker2d
109.0
108.1
110.5
111.2
112.1.02
halfcheetah
40.1
42,0
42.0
56.5
48.5
random
hopper
21.0
3.1
6.7
5.5
62.1=28
medium
walker2d
32.0
66.9
60.0
5.5
82.3=004
halfcheetah
7.7
10.3
42.4
1.3
90.3=1.6
random
hopper
2.4
90.2
16.7
24.8
112.5 1.3
expert
walker2d
53.4
103.4
93.2
1.1
109.1
random
halfcheetah
29.1
42.6
39.1
10.5
90.6
medium hopper
62.0
46.1
31.3
88.5
107.8 10.4
expert walker2d
10.6
78.8
52.0
4.9
97.7
Table 5: Test returns of our proposed A2PO with different advantage conditions during training. = corresponds one standard deviation of the average evaluation of the performance random seeds. The performance is measured by the normalized scores at the last training iteration. Bold indicates the best performance each task.
Source
Task
Eix Edis; 0.0
Edis; 0.1 Édis; é 0.5 Continuous
halfcheetah
39.610.6
40.7=08
41.0au.5
41.2=05
44.8.0.
medium replay
hopper
74.8+112
96.9-1.7
85.2+6.0
93.3
101.6+1.
walker2d
61.9+1.6
63.6+5.6
73.4
69.1 +67
82.8
halfcheetah
93.1
94.1 :0.4
94.5
94.8 0.0
95.6
medium expert
hopper
62.5
110.5.07
108.6.1.5
107.6
113.4
walker2d
10..310.1
110.7103
110.710.1
110.716.7
112.1 10.2
halfcheetah
5,811.1
26.0
21.916.2
22.91377
90.311.6
random
hopper
51.61318
107.8137
110.411.0
95.8
112.511.3
expert
walker2d
63.1+33.1
73.6=52.0
109.6+0.0
109.9=0.1
109.1+14
random halfcheetah
63.2+1.2
73.7+5.4
71.6+2.7
71.2+37
90.6+1.6
medium
hopper
65.7+38.2
108.0+1.
96.1+155 111..
108.0-o.1
107.8+0.
expert
walker2d
89.0+13.5
96.9+13.8
54.5+54.
108.7+3.3
97.746.7
continuous advantage value, our A2PO is able to better adapt to the varying environment dynamics and complexities found in the mixed-quality dataset, thereby further getting state-of-the-an performance with low variance. This enables more precise capture of the relationship between the advantage condition and the disentangled behavior policies.
E Analysis of Advantage Condition Input for Testing
In this section, we provide full comparison of different fixed advantage condition inputs for testing on Gym tasks in Table 6 as supplement for Figure 4 It should be noted that in contrast to the ablation study shown in Appendix D which investigates different methods for computing ad advantages
16

---

Table 6: Test returns of A2PO with different discrete advantage conditions for test while using the original continuous advantage condition during training.
Source
Task
nT |=(,)s,5=-11 To (-18,(
(-1s,,
halfcheetah
87.2+0.3
93.2+0.4
96.3=0033
expert
hopper
84.94222
106.646.7
111.7=10.4
walker2d
7.9+3.1
48,9+12.7
112.4#0.2
halfcheetah
4.7+5.6
21.5+8.6
44,8+0.2
medium replay
hopper
11.4+6.
25.3
101.6#11.
walker2d
2.0134
22.543.7
82.8.1.7
halfcheetah
40.4+0.6
64.5+4.6
95.6+0.5
medium
hopper
37.1+162
76.9=24.4
113.420.5
expert
walker2d
5.840.9
73.0=5.8
112.1=0.2
halfcheetahl
2.8+4.2
79.92119
90.3=11.6
random expert
hopper
0,8+0.0
11.5#9.1
112.5=1.3
walker2d
-0.1+0.0
3,14
109.1#14
random
halfcheetah
2.9+199
54,3+7.0
90.6=1.6
medium
hopper
7.947.
31.0#19.1
107.8=04
expert walker2d
1.51166
9.2+5.9
97.7=167
during CVAE and agent learning, this section utilizes the original continuous advantage computation method. However, the designated is varied during the testing process. On hand, the CVAE decoder Ps: can generate CVAE-approximated behavior policies by varying the input Ç. On the other hand, for policy improvement regulates the advantage-aware policy closely resemble samples from the same advantage condition. This indicates that the quality of the generated action should be positively correlated with the specified action input. The experimental outcomes demonstrate the relationship between varying values of & and the corresponding A2PO policy returns: the agent performance consistently improves as the fixed advantage condition input increases. Moreover, the gap between the returns increases as the offline dataset includes more diverse behavior policies. These observations serve as strong evidence for the effectiveness of A2PO in effectively disentangling the behavior policies within multi-quality dataset.
F Analysis of the CVAE Policy
The CVAE policy corresponds to the CVAE decoder (v 20, c²), where is sampled from (( 1). e* s =l As described in Section 4.1 the advantage-awarc CVAE utilizes the advantage condition computed by the agent critic to construct the ELBO loss, which is formulated as sraantage-guider supervised learning 12]. The CVAE decoder Pe: (a] c²) generates action outputs of varying qualities based on the target advantage input signal Ç. By conditioning
the maximum nor- malized advantage value ç* to get the optimal action of CVAE and agent policy, we present thorough comparison results of the CVAE policy and agent policy in Table 7 The performance of the CVAE policy indicates that it only demonstrates superior performance in a limited set of tasks and datasets, environments. The A2PO agent consistently
specifically the hopper-random and walker2d-randie outperforms the CVAE agent in the majority of cases. This performance gap is particularly significant in the newly constructed mix-quality datasets including random-expaa expprtt and random-mediune medium-expeee expert highlighting the conflict issue. These findings suggest that A2PO achieves well-disentangled behavior policies and an optimal agent policy, surpassing the capabilities of CVAE-reconstructed behavior policies.
G Analysis of A2PO Policy Optimization
In this section, we evaluate the effectiveness of the policy regularization term within the policy improvement step as shown in Equation 8 Previous approaches [56 5 that construct agent policies within the CVAE latent space do not incorporate the BC regularization term, the latent action
17

---

Table 7: Test returns of A2PO with CVAE policy agent policy.
CVAE policy
Agent Policy <11|2",
Source
Task
(- c*)
halfcheetah
45.7=0.3
47.1=022
medium
hopper
57.1
80.3+4.0
walker2d
81.9=0.7
84,9+0.2
halfcheetah
39,2+
44.8+0.2
medium replay
hopper
91.5+1114
101.6+1..
walker2d
63,429.5
82.8=1.7
halfcheetah
93,4+099
95.6+0.5
medium expert
hopper
112.2=0.6
113,420.5
walker2d
110.5
112.1=0.2
halfcheetah
41.1=0.9
48.5
random medium
hopper
15.5=11.7
62.1+28
walker2d
41.9=6.0
82.3++04
halfcheetah
36.9+15.9
90.3+116
random
hopper
81.4
112.5+113
expert
walker2d
-0.1
109.1
random
halfcheetah
66.2+660
90.6+1.4
medium
hopper
56.7
107.8=0.4
expert walker2d
22.7=6.1
97.7=6.7
space inherently imposes constraints on the action outputs. In contrast, our advantage-aware policy integrates the BC term into the actor loss to explicitly ensure appropriate pessimism. Meanwhile, to guarantee the policy action outputs are consistent with the advantage condition &, we develop variant of A2PO, designated A2PO (vne) which constrains the agent output to the optimal CVAE policy (+120 ~ (0, I), c²). Unlike this variant, the original A2PO utilizes the MSE loss to align the agent action output (-11²", e²) with the sampled action To assess the impact of our A2PO architecture and the BC regularization, we conduct an ablation study by comparing the results of different A2PO variants and LAPO, demonstrated in Table 8 The results indicate that even without the BC regularization term, A2PO consistently outperforms LAPO in most tasks. Furthermore, the original formulation of the BC term in the main text can enhance performance in most cases. This comparison highlights the superior performance achieved by A2PO, showcasing its effective disentanglement of action distributions from different behavior policies to enforce reasonable advantage-aware policy constraint and obtain an optimal agent policy.
H Analysis of A2PO Under Mixed-quality Dataset with Different Proportion
In this section, we conducted additional experiments assess the robustness of A2PO under various mixed-quality datasets. The mixed datasets consist of a fixed number of x 10t offline samples. These datasets generated by combining of either expert or medium dataset (high-return) and 1- a of random or medium dataset (low-return). The quantitative results are presented in Table 9 The results demonstrate that our A2PO algorithm is capable of achieving expert-level performance even with a limited number of high-quality samples. Moreover, as the proportion a increases, the A2PO policy becomes more stable. These results indicate the robustness and effectiveness of our A2PO algorithm in deriving optimal policies across diverse structures of offline datasets.
I Advantage Visualization
In this section, we illustrate the distribution initial state-action pairs offline datasets across various tasks, along with the corresponding actual return and estimated advantage values. Figure & and present a comparative analysis of the actual return, LAPO, and our A2PO advantage approximation. The findings indicate that LAPO exhibits limited discrimination in assessing transition advantages,
18

---

Table 8: Test returns of LAPO, A2PO and its variants in relation to policy regularization.
Source
Task
LAPO
A2PO w/o BC A2PO(a
A2PO (Ours)
halfcheetah
46.0+n.1
46.8+
46.6+0.1
47.1+0.2
medium
hopper
51.6=26
70.1
71.9=9.4
80.3+4.0
walker2d
80.8+1.3
82.0=1.1
82.0=0.7
84.9+0.2
halfcheetah
41.9+0.5
42,0=0.3
40,3=1.2
44.8+00,
medium replay
hopper
50. 10112112
96,5=15
96. +220
101.6#1.3
walker2d
60.6+10.5
71,1=8.0
79.9=1.6
82.8+1177
halfcheetah
94,2+0.5
94.3+000
94,8=003
95.6+0.5
medium expert
hopper 111.0+0.4
107.3+20
87.7+19.0
113.4+0.5
walker2d
110.920.2
111.6=0.1
111.5=0.3
112.1:0.2
halfcheetah
52.6+17.3
61.4+6.3
93.1 +6.2
90.3+1.6
random
hopper
82.3+19.0
113.2=1.2
81.8=0.2
112.5#1.3
expert
walker2d
0,4
66,8=1100
96,8=3.3
109.1#14
halfcheetah
18.5+1.0
43.2+005
41,0=1.6
48.5+0.3
random
hopper
4.2+3.1
25.7=9.2
40,9 40.9
62.1+28
medium
walker2d
23.6+34.0
72.3+4.4
57,8#3.4
82.3+0.4
random halfcheetah|
71.1+04
70.8+422
89.0 39.0+4.8
90.6+1.6
medium
hopper
66.6+19.3
86.5+7.3
12.8+4.0
107.8+0.
expert walker2d
60.4243.2
110.4
63.0
97.726.7
Total
1026.8
1342.0
1287.0
1563.3
while A2PO effectively distinguishes between transitions of varying data quality. These results underscore the limitations of the AW method and highlight the superiority of our A2PO approach.
J Boarder Impact
This paper presents an offline advantage-aware learning approach that leverages the estimated advantage condition to deal with mixed-quality datasets. The advantage-aware concept brings new perspective to the solution of real-world RL tasks, facilitating a more practical and effective utilization of the offline datasets. It has the potential to enhance the robustness of the agent towards the varying offline datasets from real-world RL scenarios, where the pre-collected offline datasets are noisy and often not well-organized as the D4RL standardized datasets.
Table 9: Test returns of A2PO under random-expert dataset with different component proportions in the Gym tasks. The datasets consist of 10th samples, where a of the samples originate from the expert dataset and the remaining
of the samples come from the random dataset.
Source
Task
a=0%
1=10%
a=20%
a=30%
4=40%
5=50%
o= 100%
halfchectah
47.1 87.3
90.4206
91.4
94,8
95.6 05.6203
96.2-103
medium expert
hopper
89.3140
72.8
98.7
90.3122
92.245.3
113.4:05
111.7:94
walker2d
84.9+02
79.4+39
95.2493
94.745.3
111.5+23
1
112.440.2
halfcheetah
25.62110
48.510.
48.3
48.4am)
48.1
48.5.0.3
47. 11.2
random medium
hopper
18.4204
27.81-1.
68
68.1122
56.5133
60.012.0
62.1428
89.31:10
walker2d
3.6117
36.0a119
77.7412
72,3112
74.1111
82,3404
84,9
halfcheetah
25.6+10 81.0+1.
86.2+0.1
90.6403
90.7+0.
90.3+1.6
96.2+t0.
random
hopper
18.4+04 40.3+69
77.0+73
108.7+07
111.1+07
112,5+1.3
111.7+04
expert
walker2d
3.611.7 67.2
96.1474
108.0ao
109.2104
109. 111.4
112.410.2
19

---

100
1.0
1.0
80
0.5
0.5
-05
0.5
20
-1.
1.0
Actual Return
LAPO Advantage Estimation
A2PO Advantage Estimation
(a) walker2d-randome expert
100
1.0
1.0
80
0.5
60
0.0
0.0
40
-0.5
00.5
20
Actual Return
LAPO Advantage Estimation
A2PO Advantage Estimation
(b) hopper-random-exper
Figure 8: Comparison of proposed A2PO method and the recent advanced AW method (LAPO) in advantage estimation for mixed-quality offline datasets (random-expal expeerr) in Gym tasks. Each data point represents initial state-action pair in the offline dataset after applying PCA while varying shades of color indicate the magnitude of the actual return or advantage value.
20

---

100
1.0
0.5
0.5
60
0.0
0.0
40
20
0
1.0
1.0
Actual Return
LAPO Advantage Estimation
A2PO Advantage Estimation
(a) halfcheetah-randdee -ranmhhh---rraddddmmmmd-u--e-eer
100
1.0
1.0
80
0.5
60
0.0
0.0
-.5
0.5
20
-1.0
1.0
Actual Return
LAPO Advantage Estimation
A2PO Advantage Estimation
(b)
80
0.5
0.5
60
0.0
0.0
40
-05
20
1.0
Actual Return
LAPO Advantage Estimation
A2PO Advantage Estimation
(c) hopper-random- med.adm---mmmiiuu-eeeeeet
Figure 9: Comparison of our proposed A2PO method and the recent advanced AW method (LAPO) in advantage estimation for mixedd-uulity offline datasets tasks. Each data point represents an initial state-actin pair in the offline dataset after applying PCA while varying shades of color indicate the magnitude of the actual return or advantage value.
eereddim--xpper in locomotion
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 0
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
Justification:
See supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiiicgguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:::////uis..eeeeeeeee http://ccccppppccccccccccactaaae
Answer:
[Yes]
Justification:
See supplemental material.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix !
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
26

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "zLlf1UqBOS", "reviewer": "Reviewer_3MzD", "summary": "The paper introduces A2PO, a novel approach to offline reinforcement learning that addresses the constraint conflict issue in mixed-quality datasets by using a Conditional Variational Auto-Encoder to disentangle action distributions and optimize policies towards high advantage values. A2PO demonstrates superior performance over existing methods on the D4RL benchmark, showcasing its effectiveness in leveraging diverse offline datasets for more robust policy learning.", "questions": "This paper focuses on the constraint conflict issue in offline reinforcement learning when dealing with mixed-quality datasets collected from multiple behavior policies. By using a Conditional Variational Auto-Encoder (CVAE), A2PO disentangles the action distributions of different behavior policies present in the mixed-quality dataset. The CVAE models the advantage values as conditional variables, allowing it to generate action distributions that reflect the behavior policies' distinct characteristics.\nIn general, this paper is interesting and well-written. One question is that since we can directly select the good trajectories according to the rewards, is there any ablation study to show that CVAE can do better than this simple method?", "limitations": "see questions part", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "see questions part", "weaknesses": "see questions part", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your supportive review of our method, experiments, and writing skills. We will address your doubts here.\n \n**[Q1. since we can directly select the good trajectories according to the rewards, is there any ablation study to show that CVAE can do better than this simple method?]**\n\n  Thanks for your constructive suggestions! We have additionally conducted ablation study on our CVAE in the A2PO method. To illustrate the effectiveness of our CVAE approach compared to merely selecting high-quality trajectories for training, we eliminate the CVAE module and implement the trajectory selection method proposed in [1,2], which involves selecting only the top 10% of trajectories based on their returns for agent training.\n\n  The results shown in Table R1 clearly indicate that A2PO without CVAE [top-10%] consistently lags behind the original A2PO in most cases, especially on the mixed-qualities datasets like 'walker2d-random-medium' and 'walker2d-random-expert'.   This is mainly due to the challenge of determining the hyperparameter $k$ for selecting the top-$k$ trajectories for training across different tasks and datasets. A smaller $k$ filters out more data, limiting the agent's awareness of the action distribution and task information. On the other hand, a larger $k$ introduces constraint conflicts that hinder agent policy improvement.   Instead, our A2PO utilizes CVAE conditioned on advantage value to obtain disentangled behavior constraints.  By optimizing the advantage-aware policy based on these disentangled action distribution constraints, we are able to achieve higher returns by directing the agent toward high advantage values.\n\n  \n\n  Table R1. Test returns of A2PO and A2PO w/o CVAE [selecting top-10% trajectory for training]. **Bold** indicates the best performance among the two algorithms.\n\n  | Env                              | A2PO w/o CVAE [top-10%] | A2PO              |\n  | -------------------------------- | ----------------------- | ----------------- |\n  | halfcheetah-medium               | 42.1$\\pm$0.1            | **47.1**$\\pm$0.2  |\n  | hopper-medium                    | 61.2$\\pm$9.0            | **80.3**$\\pm$4.0  |\n  | walker2d-medium                  | 17.9$\\pm$1.2            | **84.9**$\\pm$0.2  |\n  | halfcheetah-medium-replay        | 35.6$\\pm$2.9            | **44.8**$\\pm$0.2  |\n  | hopper-medium-replay             | 89.8$\\pm$1.1            | **101.6**$\\pm$1.3 |\n  | walker2d-medium-replay           | 73.9$\\pm$3.4            | **82.8**$\\pm$1.7  |\n  | Halfcheetah-medium-expert        | 87.2$\\pm$0.2            | **95.6**$\\pm$0.5  |\n  | hopper-medium-expert             | 103.9$\\pm$0.9           | **113.4**$\\pm$0.5 |\n  | walker2d-medium-expert           | 111.1$\\pm$0.1           | **112.1**$\\pm$0.2 |\n  | halfcheetah-random-medium        | 45.9.$\\pm$0.2           | **48.5**$\\pm$0.3  |\n  | hopper-random-medium             | 60.1$\\pm$1.5            | **62.1**$\\pm$2.8  |\n  | walker2d-random-medium           | 3.2$\\pm$1.8             | **82.3**$\\pm$0.4  |\n  | halfcheetah-random-expert        | 54.1$\\pm$2.1            | **90.3**$\\pm$1.6  |\n  | hopper-random-expert             | 105.8$\\pm$1.0           | **112.5**$\\pm$1.3 |\n  | walker2d-random-expert           | 10.5$\\pm$1.2            | **109.1**$\\pm$1.4 |\n  | halfcheetah-random-medium-expert | 50.3.$\\pm$1.0           | **90.6**$\\pm$1.6  |\n  | hopper-random-medium-expert      | 101.3$\\pm$3.9           | **107.8**$\\pm$0.4 |\n  | walker2d-random-medium-expert    | 15.2$\\pm$0.6            | **97.7**$\\pm$6.7  |\n\n  [1] Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS 2021.\n\n  [2] Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaeETd2kR7", "reviewer": "Reviewer_e3Zq", "summary": "This paper presents A2PO. A2PO is an offline RL method for learning with datasets that was collected by a diverse set of poliices. The aim of the method is to disentangle the data being collected by each policy by using advantage calculation and then using this information to better learn policies from the dataset. Specifically, they use a VAE and condition it on advantage and state and then train in latent states. They perform a thorough set of experiments on benchmarks to compare to baseline methods. They also perform an ablation study to find the important qualities in their method.", "questions": "None.", "limitations": "The limitation they address is that the method is slower than others, but no slower than some baselines they provide. I believe this is addressed well enough and improving upon the computation speed is clearly out of scope.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality. Average strength. The originalty of this work is what I would expect. They have with a novel idea of disentangling the policies using advantage estimation which builds upon the previous idea of advantage estimation. I think this work is of solid novelty.\n\nQuality. Major strength. The emperical analysis in this work is done very well. They compare to many baselines and show their method is clearly superior in many of the domains. They also perform a very good ablation study that shows the quality of their choises. I think this is a high point of the paper.\n\nClarity. Major strength. The paper is very well written, provides excellent figures and is easy to follow. Great job here.\n\nSignificance. Average strength. The work here seems like it takes a good step forward in the domain of offline RL with mixed quality datasets. The significance here is solid due to their impressive results but no more than I would expect from a good paper.", "weaknesses": "I don't have any major comments here. I believe this paper is complete, written well, has impressive results and is easy to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive support of our work. In the future, we will make further improvements to the A2PO to contribute to the offline rl community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9dEb7gathu", "reviewer": "Reviewer_xTq4", "summary": "The paper proposes using CVAE to train an agent and using a decoder, which conditions on state and advantage value, to generate actions. The Q function and V function are trained simultaneously, and the policy is trained in a TD3-BC style. Experiments show that the method can outperform baselines in most tasks.", "questions": "1. In line 169, what prior distribution of $p(z)$ did you choose?\n\n2. When training the Q and V functions (Eq(7)), the authors claimed that training simultaneously can stabilize training. Do the authors have any intuition, theory, or literature to back up such a design? I ask because, in my memory, most methods that need to estimate the V function train Q and V separately, such as IQL[1] and SAC[2]. Simultaneous training may cause instability, similar to training GANs.\n\n3. In Table 1, I found some baseline results inconsistent with the original paper. For example, hopper-medium-v2 is 90.5 for Diffusion-QL in the original paper, but 70.3 in this paper; antmaze-umaze-diverse is 66.2 for Diffusion-QL in the original paper, but 24 in this paper. Most of the results for Diffusion-QL are not consistent with the original paper, while in line 234, the authors claim their results are from the original paper. Can the authors explain the mismatch?\n\n4. In line 463, the authors state they use the v2 version for antmaze tasks, while most other baselines, including IQL and Diffusion-QL, use the v0 version. In my experience, the v2 version dataset can always provide a higher score than the v0 version when using the same algorithm. It is unfair to compare A2PO's v2 results with the v0 results of other baselines.\n\n5. I found that the training curve of hopper random-medium-expert in Figure 4 is unstable. Can the authors provide more training curves for other environments?\n\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[2]Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).", "limitations": "My main concern is about the empirical experiments. It seems the algorithm doesn't have competitive performance compared to diffusion-based policy algorithms.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to follow, and the motivation and intuition for the algorithm design are reasonable. The experiments are comprehensive and justify the advantages of the method claimed by the authors. Conditioning on the advantage value, instead of using it as a coefficient, is interesting, allowing for sampling to make the advantage value extreme ($\\xi=1$) to get the optimal action.", "weaknesses": "I found some results in the experimental section problematic. More details are in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for affirming the idea's novelty and the experiment's comprehensiveness. In this section, we aim to address your concerns.\n\n**[Q1: In line 169, what prior distribution of $p(z)$ did you choose?]**\n\nSorry for the confusion. As stated in line 167 of the original paper, the prior distribution of $p(z)$ is chosen to be $\\mathcal N(0,I)$.\n\n**[Q2: When training the Q and V functions (Eq(7)), the authors claimed that training simultaneously can stabilize training. Do the authors have any intuition, theory, or literature to back up such a design?]** \n\nThank you for your insightful question. We apologize for the confusion caused by the misdescription in the manuscript. In fact, we did not train the Q and V functions simultaneously; instead, we employed an alternating training approach for policy evaluation following the LAPO method [1].  Specifically,  the Q-function is still optimized with the Bellman equation: $L_Q(\\theta)=\\mathbb E_{(s,a,r,s')\\sim \\mathcal D}\\sum_i [r+\\gamma V_{\\hat \\phi}(s')-Q_{\\theta_i}(s,a)]^2$, while the V-function is optimized with the the expectation on Q-value: $L_V(\\phi)=\\mathbb E_{s\\sim \\mathcal D}[V_\\phi(s)-\\mathbb E_{ \\tilde z^*\\sim \\pi_\\omega (\\cdot|c^*), a^*_\\xi\\sim p_\\psi(\\cdot|\\tilde z^*, c^*) }Q_\\hat\\theta(s,a_\\xi^*)]^2$. Corresponding to the implementation presented in the supplementary material, in line 230-238 of`latent_a.py`, the V-function update codes are:\n\n```python \nadv_sample = torch.ones((next_s.size()[0], 1)).to(self.device)        #get \\xi^*\nlatent_a = self.actor_target(torch.cat((s, adv_sample), dim=1))\t      #get \\tilde z^*\nhat_a = self.vae.decode(torch.cat((s, adv_sample), dim=1), latent_a)  #get a_\\xi^*\ntemp_q1, temp_q2 = self.critic_target(s,hat_a)                        #get Q value\ntarget_v = torch.min(temp_q1, temp_q1) * self.doubleq_min + torch.max(temp_q1, temp_q1) * (1 - self.doubleq_min) \ncurrent_v = self.critic.v(s)\nloss = ... + F.mse_loss(current_v, target_v)\n```\n\n\n\n**[Q3: In Table 1, I found some baseline results inconsistent with the original paper.]**\n\nSorry for the confusion. We have systematically reviewed the score tables for the 'medium', 'medium-replay', and 'medium-expert' datasets, rectifying any recording errors identified. The mismatch scores reported in the original manuscript were obtained by re-running the corresponding source code, which was accidently mixed with the officially reported scores.\nWe give a new comparison table between these baselines and A2PO, as illustrated in Table R1, and we will update the paper accordingly.\n\n**[Q4: It is unfair to compare A2PO's v2 results with the v0 results of other baselines.]**\n\nThanks for your constructive comments. We have additionally tested A2PO using the v0 version for the AntMaze tasks. For BC, BCQ, TD3+BC, CQL, AWAC, IQL, and Diffusion-QL, we have directly referenced the scores from the Diffusion-QL paper [1]. For CQL+AW, we have utilized the scores from the CQL+AW paper [2]. For EQL and LAPO—neither of which provide scores for the v0 version—we have re-evaluated their performance, presenting both the scores and standard deviations. The results are summarized in Table R2, indicating that our A2PO continues to achieve performance comparable to that of advanced baselines such as Diffusion-QL.\n\n**[Q5: I found that the training curve of hopper random-medium-expert in Figure 4 is unstable. Can the authors provide more training curves for other environments. ]**\n\nThanks for the insightful comment. We have provided various A2PO training curves in the supplementary pdf, in which the A2PO is trained with 1M timesteps while performing online evaluation at each 2K timesteps. The results show that the A2PO maintains overall stability during training on most of the tasks and datasets.\n\n**[Q6: My main concern is about the empirical experiments. It seems the algorithm doesn't have competitive performance compared to diffusion-based policy algorithms.]**\n\nWe appreciate your valuable comment; however, we respectfully disagree with the assertion that A2PO does not exhibit competitive performance compared to diffusion policy for the following reasons:\n\n1. Even taking into account the comparative data from the Diffusion-QL paper, our A2PO method still outperforms diffusion-QL in 11 out of 18 gym tasks, 4 out of 6 maze tasks, 2 out of 3 kitchen tasks, and 1 out of 2 adroit tasks. This demonstrates that our algorithm exhibits a comparative advantage in most task scenarios over the diffusion policy.\n2. The primary focus of our research is not on benchmarking against diffusion-based methods but on addressing the conflict constraint issue, which current AW methods fail to solve effectively. Our extensive experiments indicate that A2PO significantly outperforms existing AW methods, such as LAPO.\n3. Our approach is orthogonal to diffusion-based methods. Specifically, Diffusion-QL utilizes diffusion as the policy to implicitly capture the multimodal action distribution, while it overlooks the intrinsic quality of different actions. In contrast, our A2PO explicitly leverages the advantage value to disentangle the action distributions of interrelated behavior policies, ensuring that the quality of different actions is positively correlated with the advantage value. Therefore, our advantage-aware mechanism can be effectively integrated with diffusion-based methods, enhancing their capacity to identify multimodal data of varying quality.\n\n In conclusion, our core contribution lies not in the model itself but in the introduction of an advantage-aware policy.\n\n[1] Latent-Variable Advantage-Weighted Policy Optimization for Offline RL. NeurIPS 2022\n\n[2] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. ICLR 2023\n\n[3] Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting. ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWfTroNAfb", "reviewer": "Reviewer_M64c", "summary": "The authors propose an advantage aware offline RL algorithm for datasets consisting of data from multiple behavior policies. The algorithm consists of two steps, (1) behavior policy disentangling: Wherein a CVAE is trained to output actions conditioned on normalized advantage and state. (2) Policy optimization, where the policy is optimized based on the actions generated by the CVAE. The authors demonstrate the performance of the algorithms on standard benchmarks", "questions": "See weaknesses", "limitations": "The authors address limitations", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow\n\n2. The experimental section is extensive", "weaknesses": "1. **The Agent policy optimization is unclear**\n\n    1. Why is a behavior regularization term (2nd term in (8)) needed when you have a generative model trained on the dataset? \n\n    2. Why not condition the 2nd in eq 8 term on c*? \n\n2. **Missing some return conditioned baselines**\n\n    1. Comparison with return conditioned approaches such as %BC and Decision Transformers are needed to understand the advantage of the proposed approach\n\n    2. Comparisons with data rebalancing techniques [1] can help understand methods with mixed datasets\n\n[1] Yue, Yang, et al. \"Boosting offline reinforcement learning via data rebalancing.\" arXiv preprint arXiv:2210.09241 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments on the experiment's completeness and the writing's quality. We will address your concerns in this section.\n\n**[Q1.1: The Agent policy optimization is unclear: Why is a behavior regularization term (2nd term in (8)) needed when you have a generative model trained on the dataset?]**\n\nSorry for the confusion.  First of all,  as an offline RL method with policy constraints, A2PO requires the agent policy to closely align with the behavior policy. In order to achieve this, we incorporate a behavior regularization term that constrains the action distribution of the advantage-aware agent based on the computed advantage value.  \n\nMeanwhile, although latent action space inherently imposes a constraint on the action output [1], the regularization term ensures that the actions chosen by the A2PO policy align with the advantage condition $\\xi$ determined by the critic, thereby providing a more precise policy constraint.   To confirm our perspective, we conducted an ablation study by eliminating the regularization term in Table 7 and Appendix F of the original paper. The results of this study demonstrate that the regularization term in A2PO enhances its performance in most cases, providing strong support for the necessity and effectiveness of the regularization term.\n\n**[Q1.2: The Agent policy optimization is unclear: Why not condition the 2nd in eg 8 term on *c*?]**\n\nThanks for your insightful comment. By directly constraining the agent's output on $c^*$, the agent will become fully constrained on the 'optimal' action generated by the CVAE. However, this approach may introduce additional computing errors due to the training of the CVAE. Instead, we propose utilizing the critic-computed $c$ and constraining the agent on the precisely sampled action. This not only avoids errors but also allows the agent to be aware of the multiple action distributions of behavior policies within the datasets.\n\nTo further verify our statement, we conducted an ablation study on the 'random-medium', 'random-expert' and 'random-medium' dataset by changing the regularization term from the original advantage-aware form $\\mathbb E_{\\substack{(s,a)\\sim \\mathcal D, \\tilde{z}\\sim \\pi_\\omega(\\cdot|c), \\\\ a_\\xi\\sim p_\\psi(\\cdot|\\tilde{z}, c)}}\\big[(a-a_\\xi)^2\\big]$ to $\\mathbb E_{\\substack{(s,a)\\sim \\mathcal D, \\tilde{z}^*\\sim \\pi_\\omega(\\cdot|c^*), \\\\ a_\\xi^*\\sim p_\\psi(\\cdot|\\tilde{z}^*, c^*),\\\\ a_\\text{cvae}^*\\sim p_\\psi(\\cdot|z, c^*), z\\sim \\mathcal N(0,I)}}\\big[(a_\\xi^*-a_\\text{cvae}^*)^2\\big]$.  Therefore, the output of the A2PO optimal policy $a_\\xi^*$ is constrained only by the CVAE-inferred best action $a_\\text{cvae}^*$.. The results are presented in Table R1， which show that our original form outperforms the variation in most cases, providing support for our regularization term.\n\nTable R1. Test returns of A2PO and A2PO constrined on $a_\\text{cvae}^*\\sim p_\\psi(\\cdot|z=\\mathcal N(0,I), c^*)$. **Bold** indicates the best performance among the two algorithms.\n\n| Env                              | A2PO constrined on  $a_\\text{cvae}^*$ | A2PO              |\n| -------------------------------- | ------------------------------------- | ----------------- |\n| halfcheetah-random-medium        | 41.0$\\pm$1.6                          | 48.5$\\pm$0.3      |\n| hopper-random-medium             | 40.9$\\pm$2.0                          | **62.1**$\\pm$2.8  |\n| walker2d-random-medium           | 57.8$\\pm$3.4                          | **82.3**$\\pm$0.4  |\n| halfcheetah-random-expert        | **93.1**$\\pm$6.2                      | 90.3$\\pm$1.6      |\n| hopper-random-expert             | 81.8$\\pm$0.2                          | **112.5**$\\pm$1.3 |\n| walker2d-random-expert           | 96.8$\\pm$3.3                          | **109.1**$\\pm$1.4 |\n| halfcheetah-random-medium-expert | 89.0$\\pm$4.8                          | **90.6**$\\pm$1.6  |\n| hopper-random-medium-expert      | 12.8$\\pm$4.0                          | **107.8**$\\pm$0.4 |\n| walker2d-random-medium-expert    | 63.0$\\pm$4.5                          | **97.7**$\\pm$6.7  |\n\n**[Q2: Missing some return conditioned baselines：1. Comparison with return conditioned approaches such as %BC and Decision Transformers are needed to understand the advantage of the proposed approach;  2.Comparisons with data rebalancing techniques can help understand methods with mixed datasets]**\n\nThanks for your constructive suggestions! We have additionally compared our A2PO with the return condition baselines 10%BC and Decision Transformer [2], and Offline RL with data rebalancing technique ReD and its variation DeReD[3]. The results are presented in Table R2. We directly take the score from the original paper for 'medium', 'medium-replay' and 'medium-expert' datasets.  For the new 'random-medium',  'random-expert' and 'random-medium-expert' datasets, due to the time limit, we only evaluate IQL with ReD. The results show that our A2PO outperforms both the return condition baselines and data rebalancing techniques in most cases, which further demonstrates the superiority of our advantage-aware methods to tackle the constrain conflict issue. \n\n\n[1] PLAS: Latent Action Space for Offline Reinforcement Learning. CoRL 2021\n\n[2] Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS 2021\n\n[3] Boosting Offline Reinforcement Learning via Data Rebalancing. arXiv 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HgRNNsQM0n", "reviewer": "Reviewer_W3XG", "summary": "This paper proposes an offline reinforcement learning method called A2PO, which aims to solve the problem of constraint conflicts in mixed-quality datasets collected from multiple behavior policies. A2PO optimizes offline learning by explicitly constructing advantage-aware policy constraints, especially in the presence of data of variable quality. Specifically, A2PO employs a CVAE to disentangle the distribution of actions under different behavior policies, modeling the advantage values of all training data as conditional variables. The agent can then optimize the advantage-aware policies based on these disentangled action distribution constraints to achieve high advantage values. Through extensive experiments on single- and mixed-quality datasets from the D4RL benchmark, A2PO demonstrates its superior performance over other offline RL baselines and advantage-weighted competitors.", "questions": "1.\tAlthough this paper presents a new approach, the comparison with existing techniques may not be in depth enough and lacks an analysis of why the algorithm works. In addition, could the authors further discuss the conditions under which A2PO may fail?\n2.\tThis paper focuses on the analysis and comparison of advantage-weighted offline RL methods, ignoring recent work on policy constraints in the same relative as A2PO, such as PRDC: Policy Regularization with Dataset Constraint for Offline. It would be more convincing if the authors could add research and experimental comparisons of related work to demonstrate the advantages of A2PO among its relative methods (even SOTA methods).\n3.\tRegarding implementation details, how can we ensure a sufficient number of high-quality samples in the dataset to support training when selecting ξ = 1? Additionally, how can we guarantee the accuracy of the introduced advantage information?\n\nIn general, this work demonstrates commendable quality and I would be willing to raise my score if my concerns could be addressed.", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors introduce CVAE to deal with the problem of disentangling behavior policies in mixed-quality datasets, which is novel to me.\n2.\tThe paper is well-structured and clearly written.\n3.\tSeveral analytical experiments are also provided to help understand the method", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your support on the idea, writing, and experiment. And we will solve your doubts with text description and experimental support.\n\n  **[Q1: Although this paper presents a new approach, the comparison with existing techniques may not be in depth enough and lacks an analysis of why the algorithm works.]**\n\n  Sorry for the confusion.   In our original manuscript, we have compared our A2PO with other advanced offline RL methods. These techniques include policy constraint methods such as TD3+BC, value regularization methods like CQL, model-based method MOPO, diffusion-based method Diffusion-QL, and advantage-weighted methods like LAPO. Moreover, we have additionally conducted experiments to evaluate the return conditioned methods DT and %BC [1], as well as the data rebalancing techniques ReD [2] and its variation DeReD, as presented in Table R1. To validate the efficacy of the A2PO components, we have also conducted various ablation studies in the original paper. And we have the following key observations:\n\n  1. In Figure 1 of the paper, the toy demo showcases the erroneous constraint issue of the AW method and highlights the precise behavior distribution modeling capability of our A2PO, while the following comparing experiments further confirmed this point.\n  2. In comparison to conventional offline rl methods like TD3+BC/EQL and return-conditioned methods such as DT/%BC, the superiority of our A2PO becomes increasingly apparent as the substantial gaps between behavior policies become larger from medium, medium-expert to random-expert, random-expert-medium.\n  3. Although diffusion-based method Diffusion-QL precisely models the action distribution and gets comparable performance in several scenarios, our A2PO outperforms Diffusion-QL in most cases, particularly on mixed-quality datasets while being less time-consuming, as shown in Figure 7 of the paper.\n  4. The ablation study on the CVAE and the advantage-aware policy constraint (Appendices E, F) demonstrates the effectiveness of advantage-aware policy optimization.\n\n  These key observations highlight the effectiveness of A2PO to capture high-quality interactions within the dataset for a reasonable advantage-aware policy constraint and policy optimization.  Unlike previous AW methods focusing on redistributing the dataset but inadvertently reduce data diversity, our A2PO precisely disentangles and models the mixed behavior policies with advantage-condition technique on CVAE, then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values, therefore consistently improving the performance.\n\n  **[Q2: In addition, could the authors further discuss the conditions under which A2PO may fail?]**\n\n  Thank you for the constructive comments. The results of the original paper indicate that our A2PO does not perform well when the offline dataset distribution is narrow, such as halfcheetah-medium. When the dataset distribution is severely restricted, A2PO struggles to efficiently train both the CVAE and the agent model, which can result in task failure.\n\n  **[Q3: This paper focuses on the analysis and comparison of advantage-weighted offline RL methods, ignoring recent work on policy constraints in the same relative as A2PO, such as PRDC: Policy Regularization with Dataset Constraint for Offline Reinforcement Learning.]** \n\n  Thanks for the insight suggestion! We have additionally conducted experiments to compare the recently proposed policy-constrained method, PRDC [3] with our A2PO The results are presented in Table R1, which show that our A2PO consistently achieves superior performance on the majority of the gym, maze, antmaze, and adroit tasks. This comparison clearly underscores the superior performance and effectiveness of advantage-aware policy constraints employed in A2PO for offline policy optimization. \n\n\n**[Q4: Regarding implementation details, how can we ensure a sufficient number of high-quality samples in the dataset to support training when selecting ξ = 1? ]**\n\n  It is challenging to ensure that the dataset comprises an adequate number of high-quality samples. We address this limitation by utilizing the generalization capabilities of the generative model CVAE. By selecting $\\xi=1$, the advantage-aware CVAE can generate and infer the optimal actions. As indicated in Table 6 of Appendix E, by directly inputting $\\xi=1$, the CVAE-generated action $a\\sim p_\\psi(\\cdot|z\\sim\\mathcal N(0,I),c)$ can also achieve high performance in various scenarios. Furthermore, we have also investigated the scenario where the dataset contains only a small quantity of high-quality samples, as shown in Appendix H. The results demonstrate that our A2PO algorithm is capable of achieving expert-level performance, even with a limited number of high-quality samples. These findings underscore the generalization ability of the advantage-aware CVAE and the robustness of our A2PO algorithm in deriving optimal policies across diverse structures of offline datasets.\n\n**[ Q5: How can we guarantee the accuracy of the introduced advantage information? ]**\n\n  Thanks for the insightful comments. The advantage value is determined by both the critic and the advantage estimation method. It is important to clarify that we do not propose a novel advantage computation method; rather, we adopt the widely accepted baseline in offline reinforcement learning, TD3+BC [4], as our framework. We utilize the advantage definition $ A(s, a) = Q(s, a) - V(s) $ for advantage estimation and subsequent normalization, ensuring that the agent maintains an effective estimation.\n\n[1] Decision Transformer: Reinforcement Learning via Sequence Modeling. NeurIPS 2021\n\n[2] Boosting Offline Reinforcement Learning via Data Rebalancing. arXiv 2022\n\n[3] Policy Regularization with Dataset Constraint for Offline Reinforcement Learning. ICML 2023\n\n[4] A Minimalist Approach to Offline Reinforcement Learning. NeurIPS 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Please refer to the attachment for the figures of all results during rebuttal.

Meta-review: Strong reviews, and all concerns addressed during the discussion period. Clear accept.
Decision: Accept (poster)