Title: A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Deep Learning

Reinforcement Learning
Abstract: Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO.

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

Abstract

Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel AdvantageAware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO.

1 Introduction

Offline Reinforcement Learning (RL) [11, 4] aims to learn effective control policies from precollected datasets without online exploration, and has witnessed its unprecedented success in various real-world applications, including robot control [57, 45, 24, 25], power grid control [2, 49, 26, 36], etc . A formidable challenge of offline RL lies in the Out-Of-Distribution (OOD) problem [21], involving the distribution shift between data induced by the learned policy and data collected by the behavior policy. Consequently, the direct application of conventional online RL methods inevitably exhibits extrapolation error [34], where the unseen state-action pairs are erroneously estimated. To tackle this OOD problem, offline RL methods attempt to impose proper conservatism on the learning agent within the distribution of the dataset, such as restricting the learned policy with a regularization term [19, 9] or penalizing the value overestimation of OOD actions [20, 18].

Despite the promising results achieved, offline RL often encounters the constraint conflict issue when dealing with the mixed-quality dataset [40, 12, 1, 28]. Specifically, when training data are collected from multiple behavior policies with distinct returns, existing works still treat each sample constraint

，

，

Sub-Optimal

Action

(a) One-step Jump Task

and Offline Dataset

0.0

0.2

0.4

0.6

0.8

1.0

Training Iterations (×10 6 )

5

0

5

10

Returns

Optimal Return

LAPO

A2PO

(b) Learning Curves of

A2PO and LAPO

10

5

0

5

10

Action Range

0

100

200

300

400

LAPO VAE

A2PO VAE(

=1)

A2PO VAE( =

1)

Dataset

(c) VAE Behavior

Distributions

equally with no regard for the differences in data quality and diversity. This oversight results in improper constrain on conflict actions [5, 16, 50, 15], ultimately leading to further suboptimal outcomes. To resolve this concern, the Advantage-Weighted (AW) methods [5, 43, 58, 28] employ weighted sampling to prioritize training transitions with high advantage values from the offline dataset. However, we argue that these AW methods implicitly reduce the diverse behavior policies associated with the offline dataset into a narrow one from the viewpoint of the dataset redistribution. As a result, this redistribution operation of AW may exclude a significant number of critical transitions during training, imposing erroneous constraints on agent learning. To exemplify the above issue of AW, we conduct a didactic experiment on the recent advanced AW method, LAPO [5], as shown in Figure 1. The toy one-step jump task requires the agent to jump over obstacles and reach two designated goal positions with different rewards. The offline dataset mainly contains failed attempts, with only a few successful transitions, making it very challenging for the agent to learn an effective policy. The results in Figure 1b demonstrate that LAPO performs poorly in this task. Furthermore, Figure 1c reveals that the AW redistribution does not effectively prioritize either optimal or suboptimal actions in modeling the behavior policy. Instead, the AW redistribution can lead to an incorrect on bad actions, which results in unreliable policy optimization.

In this paper, we propose Advantage-Aware Policy Optimization , abbreviated as A2PO, to explicitly learn the advantage-aware policy with disentangled behavior policies from the mixed-quality offline dataset. Unlike previous AW methods devoted to dataset redistribution while reducing the data diversity, the proposed A2PO directly conditions the agent policy on the advantage values of all training data without any prior preference. Technically, A2PO comprises two alternating stages, behavior policy disentangling and agent policy optimization . The former stage introduces a Conditional Variational Auto-Encoder (CVAE) [41] to disentangle different behavior policies into separate action distributions by modeling the advantage values of collected state-action pairs as conditioned variables. The latter stage further imposes an explicit advantage-aware policy constraint on the training agent within the support of disentangled action distributions. The advantage-conditioned CVAE can models the behavior policy distribution (Figure 1c), which is further utilized to construct advantage-aware constraint for agent optimization toward high advantage values, resulting in an effective decision-making policy (Figure 1b).

To sum up, our main contribution is the first dedicated attempt towards advantage-aware policy optimization to alleviate the constraint conflict issue under the mixed-quality offline dataset. The proposed A2PO can achieve advantage-aware policy constraint derived from different behavior policies, where a customized CVAE is employed to infer diverse action distributions associated with the behavior policies by modeling advantage values as conditional variables. Extensive experiments conducted on the D4RL benchmark [8], including both single-quality and mixed-quality datasets, demonstrate that the proposed A2PO method yields significantly superior performance to other advanced offline RL baselines, as well as the advantage-weighted competitors.

2 Related Works

Offline RL can be broadly classified into four categories: policy constraint [11, 44], value regularization [13, 14], model-based method [55, 52], and return-conditioned supervised learning [7, 23]. Policy constraint methods impose constraints on the learned policy to be close to the behavior policy [19]. Previous studies directly introduce the explicit constraint on policy learning, such as behavior cloning [9], maximum mean discrepancy [19], or maximum likelihood estimation [48]. In contrast, recent efforts [29, 46] mainly focus on realizing the policy constraints implicitly by approximating the formal optimal policy derived from KL-divergence constraint. On the other hand, value regularization methods make constraints on the value function to alleviate the overestimation of OOD action. Researchers try to approximate the lower bound of the value function with the Q-regularization term for conservative action selection [20, 27]. Model-based methods construct the environment dynamics to estimate state-action uncertainty for OOD penalty [17, 6]. Several works also converts offline RL into a return-conditioned supervised learning task. Decision Transformer (DT) [3] builds a transformer policy conditioned on both the current state and the additional sum return signal with supervised learning. Yamagata et al. [51] improve the stitching ability of DT policy on sub-optimal samples by relabeling the return signal with Q-learning results. However, in the context of offline RL with a mixed-quality dataset and no access to the trajectory return signals, all these methods treat each sample equally without considering data quality, thereby resulting in improper regularization and further suboptimal learning outcomes.

Advantage-weighted Offline RL Method employs weighted sampling to prioritize training transitions with high advantage values from the offline dataset. To enhance sample efficiency, Peng et al. [31] introduce an advantage-weighted maximum likelihood loss by directly calculating advantage values via trajectory return. [29] further use the critic network to estimate advantage values for advantage-weighted policy training. This technique has been incorporated as a subroutine in other works [18, 50] for agent policy extraction. Recently, AW methods have also been well studied in addressing the constraint conflict issue that arises from the mixed-quality dataset [5, 58, 40]. Several studies present advantage-weighted behavior cloning as a direct objective function [58] or an explicit policy constraint [32]. [5] propose the Latent Advantage-Weighted Policy Optimization (LAPO) framework, which employs an advantage-weighted loss to train CVAE for generating high-advantage actions based on the state condition. Besides AW methods, Hong et al. [15] enhance the classical offline RL training objective with the weight of subsequent return. On the other hand, Hong et al. [16] directly learning the optimal policy density as the weight function to enable sampling from high-performing policies. However, this AW mechanism inevitably diminishes the data diversity in the dataset. In contrast, our A2PO directly conditions the agent policy on both the state and the estimated advantage value, enabling effective utilization of all samples with varying quality.

3 Preliminaries

We formalize the RL task as a Markov Decision Process (MDP) [35] defined by a tuple M = ⟨S , A , P, r, γ, ρ 0 ⟩ , where S represents the state space, A represents the action space, P : S×A×S → [0 , 1] denotes the environment dynamics, r : S × A → R denotes the reward function, γ ∈ (0 , 1] is the discount factor, and ρ 0 is the initial state distribution. At each time step t , the agent observes the state s t ∈ S and selects an action a t ∈ A according to its policy π . This action leads to a transition to the next state s t +1 based on the dynamics distribution P . Additionally, the agent receives a reward signal r t . The goal of RL is to learn an optimal policy π ∗ that maximizes the expected return: π ∗ = arg max π E π [∑ ∞ k =0 γ k r t + k ] . In offline RL, the agent can only learn from an offline dataset without online interaction with the environment. In the single-quality settings, the offline dataset D = { ( s t , a t , r t , s t +1 ) | t = 1 , · · · , N } with N transitions is collected by only one behavior policy π β . In the mixed-quality settings, the offline dataset D = ⋃ i { ( s i,t , a i,t , r i,t , s i,t +1 ) | t = 1 , · · · , N } is collected by multiple behavior policies { π β i } i M =1 .

We evaluate the learned policy π by the action value function Q π ( s, a ) = E π [ ∑ ∞ t =0 γ t r ( s t , a t ) | s 0 = s, a 0 = a ] . The state value function is defined as V π ( s ) = E a ∼ π [ Q π ( s, a )] , while the advantage function is defined as A π ( s, a ) = Q π ( s, a ) -V π ( s ) . For continuous control, our A2PO implementation uses the TD3 algorithm [10] based on the actor-critic framework as a basic backbone for its robust performance. The actor network π ω , known as the learned policy, is parameterized by ω , while the critic networks consist of the Q-network Q θ

Behavior Policy Disentangling

Agent Policy Optimization

Offline Dataset

Mixed-Quality

Offline Dataset

CVAE

Disentangled

Action

Distributions

Advantage-aware

Constraint

Optimal Policy

Improvement

High Q-value

In-sample Actions

Actor

Critic

Encoder

Decoder

+

Low-quality Policy

High-quality Policy

+

parameterized by θ and the V-network V ϕ parameterized by ϕ . The actor-critic framework involves two steps: policy evaluation and policy improvement. During policy evaluation phase, the Q-network Q θ is optimized by the temporal-difference (TD) loss [42]:

where ˆ θ and ˆ ω are the parameters of the target networks that are regularly updated by online parameters θ and ω to maintain learning stability. The V-network V ϕ can also be optimized by the similar TD loss. For policy improvement in continuous control, the actor network π ω can be optimized by the deterministic policy gradient loss [39, 38]:

Note that offline RL will impose conservative constraints on the optimization losses to tackle the OOD problem. Moreover, the performance of the final learned policy π ω highly depends on the quality of the offline dataset D associated with the behavior policies { π β i } .

4 Methodology

In this section, we provide details of our proposed A2PO approach, consisting of two key components: behavior policy disentangling and agent policy optimization . In the behavior policy disentangling phase, we disentangle behavior policies with a CVAE specifically modeling the action distribution conditioned on the advantage values of collected state-action pairs. By taking different advantage inputs, the newly formed CVAE allows the agent to infer distinct action distributions that are associated with various behavior policies. Then in the agent policy optimization phase, the action distributions derived from the advantage condition serve as disentangled behavior policies, establishing an advantage-aware policy constraint to guide agent training. An overview of our A2PO is illustrated in Figure 2.

4.1 Behavior Policy Disentangling

To realize behavior policy disentangling, we adopt a CVAE to relate the behavior distribution of different specific behavior policies π β i to the advantage condition variables , which is quite different from previous methods [11, 5, 56] utilizing CVAE only for approximating the overall mixed-quality behavior policy set { π β i } M i =1 conditioned only on specific state s . Concretely, we have made adjustments to the architecture of the CVAE to be advantage-aware. The encoder q φ ( z | a, c ) is fed with condition c and action a to project them into a latent representation z . Given specific condition c and the encoder output z , the decoder p ψ ( a | z, c ) captures the correlation between condition c and latent representation z to reconstruct the original action a . Unlike previous methods [11, 5, 48] predicting action solely based on the state s , we consider both state s and advantage value ξ for CVAE condition. The state-advantage condition c is formulated as:

Therefore, given the current state s and the advantage value ξ as a joint condition, the CV AE model is able to generate corresponding action a with varying quality positively correlated with the advantage value ξ . For a state-action pair ( s, a ) , the advantage value ξ can be computed as follows:

where two Q-networks with the min( · ) operation are adopted to ensure conservatism in offline RL settings [11]. Moreover, we employ the tanh( · ) function to normalize the advantage condition within the range of ( -1 , 1) . This operation prevents excessive outliers from impacting the performance of CVAE, improving the controllability of generation. The optimization of Q-networks and V-network will be described in the following section.

The CVAE model is trained using the state-advantage condition c and the corresponding action a . The training objective involves maximizing the Empirical Lower Bound (ELBO) [41] on the log-likelihood of the sampled minibatch:

where α is the coefficient for trading off the KL-divergence loss term, and p ( z ) denotes the prior distribution of z setting to be N (0 , 1) . The first log-likelihood term encourages the generated action to match the real action as much as possible, while the second KL divergence term aligns the latent variable distribution with the prior distribution p ( z ) .

At each round of CVAE training, a minibatch of state-action pairs ( s, a ) is sampled from the offline dataset. These pairs are fed to the critic network Q θ and V ϕ to get corresponding advantage condition ξ by Eq. (4). Then the advantage-aware CVAE is subsequently optimized by Eq. (5). By incorporating the advantage condition ξ , the CVAE captures the relation between ξ and the action distribution of the behavior policies, as shown in the upper part of Figure 2. This further enables the CVAE to generate actions a based on the state-advantage condition c in a manner where the action quality is positively correlated with ξ . Furthermore, the advantage-aware CVAE is utilized to establish an advantage-aware policy constraint for agent policy optimization in the next stage.

4.2 Agent Policy Optimization

The agent is constructed using the actor-critic framework [42]. The critic comprises two Q-networks Q θ i =1 , 2 and one V-network V ϕ to approximate the value of the agent policy. The actor, advantageaware policy π ω ( ·| c ) , with input c = s || ξ , generates a latent representation ˜ z based on the state s and the designated advantage condition ξ . This latent representation ˜ z , along with c , is then fed into the decoder p ψ for an recognizable action a ξ :

With this form, the advantage-aware policy π ω is expected to produce action with different qualities that are positively correlated to the designated advantage input ξ which is normalized within ( -1 , 1) in Eq. 4). Therefore, the output optimal action a ∗ is obtained by c ∗ = s || ξ ∗ input with ξ ∗ = 1 . It should be noted that the critic networks are to approximate the expected value of the optimal policy π ω ( ·| c ∗ ) . The agent optimization, following the actor-critic framework, encompasses policy evaluation and policy improvement steps. During the policy evaluation step, the critic is updated through the minimization of the temporal difference loss with the optimal policy π ω ( ·| c ∗ ) . Specifically, for the V-network v ϕ , we employ the one-step Bellman operator to approximate the state value under the current agent-aware policy, conditioned on the optimal advantage input ξ ∗ = 1 , as follows:

where Q ˆ θ is the target network updated softly. As for the Q-networks, both of the two Q-network entities Q θ i are optimized with agent policy π ω ( ·| c ∗ ) following Equation 1.

For the policy improvement, the actor loss is defined as:

where a ∗ ξ in the first term is the optimal action generated with the fixed maximum advantage condition ξ ∗ = 1 input and a ξ in the second term is obtained with the advantage condition ξ derived from the critic based on Eq. 4 applied to the sampled batch. Meanwhile, following TD3+BC [9], we add a normalization coefficient λ = α/ ( 1 N ∑ ( s i ,a i ) | Q ( s i , a i ) | ) to the first term to keep the scale balance between Q value objective and regularization, where α is a hyperparameter to control the scale of the normalized Q value. The first term encourages the optimal policy condition on c ∗ to select actions that yield the highest expected returns represented by the Q-value. This aligns with the policy improvement step in conventional RL approaches [22]. The second behavior cloning term explicitly imposes constraints on the advantage-aware policy, ensuring the policy selects in-sample actions that adhere to the advantage condition ξ determined by the critic. Therefore, the suboptimal samples with low advantage condition ξ will not disrupt the optimization of optimal policy π ω ( ·| c ∗ ) . And they enforce valid constraints on the corresponding policy π ω ( ·| c ) , as shown in the lower part of Figure 2. It should be noted that the decoder p ψ is fixed during both policy evaluation and improvement.

Our A2PO implementation selects TD3+BC [9] as the base backbone for its robustness. The general framework derived above is thoroughly described in Appendix A.

5 Experiments

To illustrate the effectiveness of the proposed A2PO method, we conduct experiments on the D4RL benchmark [8]. We aim to answer the following questions: (1) Can A2PO outperform the advanced offline RL methods in both the single-quality datasets and mixed-quality datasets? (Section 5.2) (2) How do different components of A2PO contribute to the overall performance? (Section 5.3 and Appendix D-G) (3) How does A2PO perform under mixed-quality datasets with varying singlequality samples? (Section 5.5 and Appendix H) (4) Can the A2PO agent effectively estimate the quality of different transitions? (Section 5.4 and Appendix I) (5) How does the time overhead of A2PO compare to other baselines? (Section 5.6)

5.1 Experiment Settings

Tasks and Datasets. We consider four different domains of tasks in D4RL benchmark [8]: Gym, Maze, Adroit, and Kitchen. Each domain contains several tasks and corresponding distinct datasets. We conduct experiments for each Gym task using single-quality and mixed-quality datasets. The single-quality datasets are generated with the medium behavior policy. The mixed-quality datasets are the combinations of random , medium , and expert single-quality datasets, including medium-expert , medium-replay , random-medium , medium-expert , and random-medium-expert . The D4RL benchmark only includes the first two mixed-quality datasets. Thus, following Hong et al. [16, 15], we manually construct the last three mixed-quality datasets by combining the corresponding single-quality datasets in D4RL with equal proportions. For the other domains of tasks, the corresponding D4RL datasets exhibit a significant level of diversity to evaluate the effectiveness of our A2PO algorithm.

Comparison Methods and Hyperparameters. We compare the proposed A2PO to several advanced offline RL methods: BCQ [11], TD3+BC [9], CQL [20], EQL [50], especially the advantage-weighted offline RL methods: AWAC [29], IQL [18], CQL+AW [15], LAPO [5]. Besides, we also select the vanilla BC method [33], the model-based offline RL method MOPO [53], and the emerging diffusion-based method Diffusion-QL [47], for comparison. We report the performance of baselines using the best results reported from their own paper. More comparison results can be found in Appendix C. The detailed hyperparameters of A2PO are given in Appendix B.2.

5.2 Comparison on D4RL Benchmarks

Results for Gym Tasks. The experimental results of all compared methods in the D4RL Gym tasks are presented in Table 1. For the single-quality medium dataset and mixed-quality medium-expert and medium-replay datasets from D4RL, our A2PO achieves state-of-the-art results with low variance. Meanwhile, both conventional offline RL approaches like EQL and advantage-weighted approaches like LAPO still learn acceptable policy, indicating that the conflict issue hardly occurs in these datasets with low diversity. However, the newly constructed mixed-quality datasets, namely random-medium , random-expert , and random-medium-expert , highlight the issue of substantial gaps between behavior policies. The results on these datasets reveal a significant drop in performance for all other baselines.

Instead, our A2PO continues to achieve the best performance on these datasets. When considering the total scores across all datasets, A2PO outperforms the next best-performing AW method, CQL+AW, by over 21%. The results reveal the exceptional ability of A2PO to capture and utilize high-quality interactions within the dataset in order to enforce a reasonable advantage-aware policy constraint and further obtain an optimal agent policy.

Results for Maze, Kitchen, and Adroit Tasks. Table 2 presents the experimental results of all the compared methods on the D4RL Maze, Kitchen, and Adroit tasks. The D4RL datasets for these tasks exhibit varying patterns in behavior policy samples. For instance, the Antmaze datasets are highly sub-optimal, while the Adroit datasets have a narrow state-action distribution. Among the offline RL baselines and AW methods, A2PO delivers remarkable performance in these challenging tasks and showcases the robust representation capabilities of the advantage-aware policy.

5.3 Ablation Analysis

Different Advantage condition during training. The performance comparison of different advantage condition computing methods for agent training is given in Figure 3. Eq. 4 obtains continuous advantage condition ξ in the range of ( -1 , 1) . To evaluate the effectiveness of the continuous computing method, we design a discrete form of advantage condition: ξ dis = sgn ( ξ ) · 1 | ξ | >ϵ , where sgn ( · ) is the symbolic function, and 1 | ξ | >ϵ is the indicator function returning 1 if the absolute value of ξ is greater than the hyperparameter of threshold ϵ , otherwise 0 . Thus, the advantage condition ξ dis is constrained to discrete value of {-1 , 0 , 1 } . Another special form of advantage condition is ξ fix = 1 for all state-action pairs, in which the advantage-aware ability is lost. Figure 3a shows that setting

hc-m-r

hc-m-e

hc-r-e

hc-r-m-e

h-m-r

h-m-e

h-r-e

h-r-m-e

w-m-r

w-m-e

w-r-e

w-r-m-e

75

50

25

0

25

Return Difference

(a)

ξ

fix

= 1

hc-m-r

hc-m-e

hc-r-e

hc-r-m-e

h-m-r

h-m-e

h-r-e

h-r-m-e

w-m-r

w-m-e

w-r-e

w-r-m-e

75

50

25

0

25

Return Difference

(b)

ξ

dis

with

ϵ

= 0

.

0

hc-m-r

hc-m-e

hc-r-e

hc-r-m-e

h-m-r

h-m-e

h-r-e

h-r-m-e

w-m-r

w-m-e

w-r-e

w-r-m-e

75

50

25

0

25

Return Difference

(c)

ξ

dis

with

ϵ

= 0

.

1

hc-m-r

hc-m-e

hc-r-e

hc-r-m-e

h-m-r

h-m-e

h-r-e

h-r-m-e

w-m-r

w-m-e

w-r-e

w-r-m-e

75

50

25

0

25

Return Difference

(d)

ξ

dis

with

ϵ

= 0

.

5

(

s

,

=1.0)

(

s

,

=0.0)

(

s

, =

1.0)

0.0

0.2

0.4

0.6

0.8

1.0

Training Iterations (×10 6 )

0

25

50

75

100

Normalized Returns

(a) halfcheetah

(medium-expert)

0.0

0.2

0.4

0.6

0.8

1.0

Training Iterations (×10 6 )

0

25

50

75

100

Normalized Returns

(b) halfcheetah

(random-medium-expert)

0.0

0.2

0.4

0.6

0.8

1.0

Training Iterations (×10 6 )

0

25

50

75

100

Normalized Returns

(c) hopper

(medium-expert)

0.0

0.2

0.4

0.6

0.8

1.0

Training Iterations (×10 6 )

0

25

50

75

100

Normalized Returns

(d) hopper

(random-medium-expert)

ξ fix = 1 without explicitly advantage-aware mechanism leads to a significant performance decreasing, especially in the new mixed-quality dataset. Meanwhile, ξ dis with different values of threshold ϵ achieve slightly inferior results than the continuous ξ . This outcome strongly supports the efficiency of continuous ξ . Although the ξ dis signals are more stable, ξ dis hidden the concrete advantage value, causing a mismatch between the advantage value and the sampled transition.

Different Advantage Condition for Test. The performance comparison of different discrete advantage conditions input for the test is given in Figure 4. To ensure clear differentiation, we select ξ from {-1 , 0 , 1 } . The different designated advantage conditions ξ are fixed input for the actor, leading to different policies π ω ( ·| s, ξ ) . The final outcomes demonstrate the partition of returns corresponding to the policies with different ξ . Furthermore, the magnitude of the gap increases as the offline dataset includes samples from more diverse behavior policies. These observations provide strong evidence for the success of A2PO disentangling the behavior policies under the multi-quality dataset.

1.0

0.5

0.0

0.5

1.0

(a) walker2d

(Advantage condition)

0

20

40

60

80

100

(b) walker2d

(Actual Return)

1.0

0.5

0.0

0.5

1.0

(c) hopper

(Advantage condition)

0

20

40

60

80

100

(d) hopper

(Actual Return)

5.4 Visualization

Halfcheetah

Hopper

Walker2d

100

80

60

40

20

0

Figure 5 presents the visualization of A2PO latent representation. The uniformly sampled advantage condition ξ combined with the initial state s , are fed into the actor-network to get the latent representation generated by the final layer of the actor. The result demonstrates that the representations converge according to the advantage and the actual return. Moreover, upon comparing Figure 5(a,b), as well as Figure 5(c,d) separately, we observe that the latent action representation follows the same alteration pattern based on the actual real return. These observations demonstrate that our advantage-aware policy effectively capture policies with different returns by the designated advantage input ξ . This provides compelling evidence for the efficacy of the A2PO policy construction. More experiments of advantage estimation conducted on different tasks and datasets are presented in Appendix I.

5.5 Robustness

Figure 6 presents the experimental results of A2PO across mixed-quality datasets with varying proportions of single-quality samples. Following the methodology of [15, 16], we evaluate the effectiveness of A2PO on three mixed-quality datasets: medium-expert , randommedium , and random-expert . These datasets consist of a total number of 1 × 10 6 transitions. We vary the proportions σ of higher quality samples and (1 -σ ) of lower quality samples. The results demonstrate that our A2PO effectively captures and infers high-quality potential behavior policies for proper policy regularization, even with a small proportion of high-quality samples. Additionally, as the σ becomes larger, the variance decreases. Thus, A2PO demonstrates its robustness in handling variations in the proportions of different single-quality samples, guaranteeing consistently high performance.

5.6 Time Overhead

We measure the training times of A2PO as well as other baselines, which are presented in Table 7. The experiments are performed on a cluster of 4 A40 GPUs under halfcheetah-medium-expert-v2 scenarios for 1 × 10 6 steps. Although the A2PO runtime is longer than the lightweight algorithms like IQL due to CVAE training, our A2PO is more efficient compared to other AW methods such as LAPO and CQL+AW.

6 Conclusion

In this paper, we propose a novel approach, termed as A2PO, to tackle the constraint conflict issue on mixed-quality offline datasets with advantage-aware policy constraints. Specifically, A2PO utilizes a CV AE to effectively disentangle the action distributions associated with various behavior policies. This is achieved by modeling the advantage values of all training data as conditional variables. Consequently, advantage-aware agent policy optimization can be focused on maximizing high advantage values while conforming to the disentangled distribution constraint imposed by the mixed-quality dataset. Experimental results show that A2PO successfully decouples the underlying behavior policies and significantly outperforms advanced offline RL competitors. For our future work, we will extend A2PO to multi-task offline RL scenarios characterized by a greater diversity of behavior policies and a more prominent constraint conflict issue.

=0%

=10%

=20%

=30%

=40%

=50%

=100%

Halfcheetah

Hopper

Walker2d

0

20

40

60

80

100

Normalized Returns

BC

BCQ

TD3+BC

Diff-QL

EQL

CQL

CQL+AW

AWAC

IQL

LAPO

A2PO

0

100

200

300

time overhead

Limitations. The limitation of A2PO is that it incorporates CV AE during training, which may lead to quite a large time overhead. However, the results presented in Section 5.6 show that the time overhead of A2PO remains reasonably acceptable when compared to other baseline methods.

References

Y. Chebotar, Q. Vuong, A. Irpan, K. Hausman, F. Xia, Y. Lu, A. Kumar, T. Yu, A. Herzog, K. Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. arXiv preprint arXiv:2309.10150 , 2023.

K. Chen, W. Luo, S. Liu, Y. Wei, Y. Zhou, Y. Qing, Q. Zhang, J. Song, and M. Song. Powerformer: A section-adaptive transformer for power flow adjustment. arXiv preprint arXiv:2401.02771 , 2024.

L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Annual Conference on Neural Information Processing Systems , 2021.

X. Chen, Z. Zhou, Z. Wang, C. Wang, Y. Wu, and K. Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2020.

X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y. Gao, W. Li, L. Bin, C. Finn, and C. Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2022.

C. Diehl, T. S. Sievernich, M. Krüger, F. Hoffmann, and T. Bertram. Uncertainty-aware modelbased offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters , 8(2):1167-1174, 2023.

S. Emmons, B. Eysenbach, I. Kostrikov, and S. Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751 , 2021.

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2021.

S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning , 2018.

S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , 2019.

C. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, and Y. Yu. Act: Empowering decision transformer with dynamic programming via advantage conditioning. arXiv preprint arXiv:2309.05915 , 2023.

K. Ghasemipour, S. S. Gu, and O. Nachum. Why so pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters. In Annual Conference on Neural Information Processing Systems , 2022.

J. Hong, A. Kumar, and S. Levine. Confidence-conditioned value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607 , 2022.

Z.-W. Hong, P. Agrawal, R. T. d. Combes, and R. Laroche. Harnessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.13085 , 2023.

Z.-W. Hong, A. Kumar, S. Karnik, A. Bhandwaldar, A. Srivastava, J. Pajarinen, R. Laroche, A. Gupta, and P. Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. In Annual Conference on Neural Information Processing Systems , 2023.

M. O. Karabag and U. Topcu. On the sample complexity of vanilla model-based offline reinforcement learning with dependent samples. arXiv preprint arXiv:2303.04268 , 2023.

I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Annual Conference on Neural Information Processing Systems , 2019.

A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2020.

S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

H. Liu and P. Abbeel. Emergent agentic transformer from chain of hindsight experience. arXiv preprint arXiv:2305.16554 , 2023.

S. Liu, Y. Qing, S. Xu, H. Wu, J. Zhang, J. Cong, T. Chen, Y. Liu, and M. Song. Curricular subgoals for inverse reinforcement learning. arXiv preprint arXiv:2306.08232 , 2023.

S. Liu, Y. Zhou, J. Song, T. Zheng, K. Chen, T. Zhu, Z. Feng, and M. Song. Contrastive identityaware learning for multi-agent value decomposition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 11595-11603, 2023.

S. Liu, W. Luo, Y. Zhou, K. Chen, Q. Zhang, H. Xu, Q. Guo, and M. Song. Transmission interface power flow adjustment: A deep reinforcement learning approach based on multi-task attribution map. IEEE Transactions on Power Systems , 39(2):3324-3335, 2024.

J. Lyu, X. Ma, X. Li, and Z. Lu. Mildly conservative q-learning for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2022.

Y. Mao, C. Wu, X. Chen, H. Hu, J. Jiang, T. Zhou, T. Lv, C. Fan, Z. Hu, Y. Wu, et al. Stylized offline reinforcement learning: Extracting diverse high-quality behaviors from heterogeneous datasets. In International Conference on Learning Representations , 2023.

A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Annual Conference on Neural Information Processing Systems , 2019.

X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

Z. Peng, C. Han, Y. Liu, and Z. Zhou. Weighted policy constraints for offline reinforcement learning. In AAAI Conference on Artificial Intelligence , 2023.

D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation , 3(1):88-97, 1991.

R. F. Prudencio, M. R. Maximo, and E. L. Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems , 2023.

M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Y. Qing, S. Liu, J. Song, H. Wang, and M. Song. A survey on explainable reinforcement learning: Concepts, algorithms, challenges. arXiv preprint arXiv:2211.06665 , 2022.

Y. Ran, Y.-C. Li, F. Zhang, Z. Zhang, and Y. Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning , pages 28701-28717. PMLR, 2023.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning , 2014.

A. Singh, A. Kumar, Q. Vuong, Y. Chebotar, and S. Levine. Offline rl with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052 , 2022.

K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Annual Conference on Neural Information Processing Systems , 2015.

R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction . MIT press, 2018.

Q. Tian, K. Kuang, F. Liu, and B. Wang. Learning from good trajectories in offline multi-agent reinforcement learning. In AAAI Conference on Artificial Intelligence , 2023.

Q. Vuong, A. Kumar, S. Levine, and Y. Chebotar. Dasco: Dual-generator adversarial support constrained offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2022.

A. Wagenmaker and A. Pacchiano. Leveraging offline data in online reinforcement learning. In International Conference on Machine Learning , 2023.

S. Wang, Q. Yang, J. Gao, M. G. Lin, H. Chen, L. Wu, N. Jia, S. Song, and G. Huang. Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning. arXiv preprint arXiv:2310.17966 , 2023.

Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193 , 2022.

J. Wu, H. Wu, Z. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning. In Annual Conference on Neural Information Processing Systems , 2022.

F. Xu, S. Liu, Y. Qing, Y. Zhou, Y. Wang, and M. Song. Temporal prototype-aware learning for active voltage control on power distribution networks. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2024.

H. Xu, L. Jiang, J. Li, Z. Yang, Z. Wang, V. W. K. Chan, and X. Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.15810 , 2023.

T. Yamagata, A. Khalil, and R. Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning , 2023.

J. Yang, X. Chen, S. Wang, and B. Zhang. Model-based offline policy optimization with adversarial network. arXiv preprint arXiv:2309.02157 , 2023.

T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Modelbased offline policy optimization. In Annual Conference on Neural Information Processing Systems , 2020.

Y. Yue, B. Kang, X. Ma, Z. Xu, G. Huang, and S. Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:2210.09241 , 2022.

Q. Zheng, M. Henaff, B. Amos, and A. Grover. Semi-supervised offline reinforcement learning with action-free trajectories. In International Conference on Machine Learning , 2023.

W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , 2021.

Y. Zhou, S. Liu, Y. Qing, K. Chen, T. Zheng, Y. Huang, J. Song, and M. Song. Is centralized training with decentralized execution framework centralized enough for marl? arXiv preprint arXiv:2305.17352 , 2023.

Z. Zhuang, K. Lei, J. Liu, D. Wang, and Y. Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312 , 2023.

A Pseudocode

To make the proposed A2PO method clearer for readers, the pseudocode is provided in Algorithm 1.

Algorithm 1 Advantage-Aware Policy Optimization (A2PO)

Input: offline dataset D , CVAE training step K , total training step T , soft update rate τ . Initialize: CVAE encoder q φ and decoder p ψ , actor network π ω , critic networks Q θ and V ϕ .

Sample random minibatch of transitions B = { ( s, a, r, s ′ ) } ∼ D . Calculate ξ = tanh(min i =1 , 2 Q θ i ( s, a ) -V ϕ ( s )) , ξ ∗ = 1 , c = s || ξ, c ∗ = s || ξ ∗ .

# Behavior Policy Disentangling if i ≤ K then

Optimize CVAE encoder q φ and decoder p ψ according to Eq. 5 as

end if

# Agent Policy Optimization

Optimize critic networks Q θ and V ϕ according to Eq. 1 and Eq. 7 as

Optimize actor network π ω according to Eq. 8 as

Update the target networks with ˆ θ ← (1 -τ ) ˆ θ + τθ, ˆ ϕ ← (1 -τ ) ˆ ϕ + τϕ . end for

B Experiment Details

B.1 Task Abbreviations and Task Versions

In order to improve the readability and conciseness, we adopt abbreviations for the Gym tasks throughout the main text. The corresponding abbreviations for each task are provided in Table 3. For the specific task versions, we use the '-v2' version for the Gym tasks, the '-v1' version for the Maze2d tasks, the '-v0' version for the Antmaze tasks, the '-v0' version for the Kitchen tasks, the '-v1' version for the Adroit tasks.

B.2 Implementation Details

In this section, we provide the implementation details of our experiments. We conducted our experiments using PyTorch 3.8 [30] on a cluster of 4 A40 GPUs.The source code will be made publicly available upon the publication of this paper.

The critic, actor, and CV AE networks are all constructed using a 2-layer MLP with 256 hidden units. They are updated using the Adam optimizer with a learning rate of 3 × 10 -4 . Additionally, we employ a target critic network with a soft-update rate of 5 × 10 -3 . Following the TD+3BC approach [9], we incorporate Q normalization, policy noise, and policy clipping during the training process. or the hyperparameter value α in Q normalization, we adopted the strategy proposed by Wang et al. [47], selecting values of 5 . 0 and 6 . 0 for the antmaze-medium/large-diverse tasks respectively to promote robust and stable Q Learning. In contrast, we chose α = 0 . 005 for the kitchen-partial task to facilitate policy regularization. For other tasks, setting α = 1 . 0 yields state-of-the-art results. As for the CVAE step K , 0.2M steps demonstrate sufficient robustness for most tasks, while the optimal settings for the maze2d-umaze/medium tasks are 0.4M and 0.1M steps, respectively. Among the tasks, the continuous form of the advantage signal ξ , as described in Eq. 4, consistently produces outstanding results across most scenarios. However, the advantage distribution may vary significantly across all the tasks, necessitating the fine-tuning of advantage computation methods based on online evaluations. The maze2d-umaze task requires a discrete form with ϵ = 0 . 1 . In the case of the maze2d-medium task, we utilize an ϵ value of 0 . 2 . For both the maze2d-large and antmaze-umaze tasks, we increase this to ϵ = 0 . 8 . In contrast, for the kitchen-partial/mixed tasks, we adopt a value of ϵ = 0 . 0 . Further details on this sensitivity can be found in Section 5.3 and Appendix D.

Since all of the original papers of other baselines do not provide the complete results of all the tasks, we make implementation of the baselines. The CQL, IQL and MOPO baselines are implemented using the implementations provided at github.com/young-geng/cql, gwthomas/iql-pytorch, and github.com/yihaosun1124/OfflineRL-Kit, respectively. The remaining baselines, including BCQ, CQL, TD3+BC, EQL, Diffusion-QL, AWAC, CQL+AW, and LAPO, are implemented using the original implementations provided by the authors of the respective papers. These implementations can be found at: BCQ github.com/sfujim/BCQ, TD3+BC github.com/sfujim/TD3_BC, EQL github.com/ryanxhr/IVR, Diffusion-QL github.com/Zhendong-Wang/DiffusionPolicies-for-Offline-RL, AWAC github.com/rail-berkeley/rlkit/tree/master/examples/awac, CQL+AW github.com/Improbable-AI/harness-offline-rl, and LAPO github.com/pcchenxi/LAPOofflienRL.

C Additional Comparisons with More Baselines

In this section, we have added the data rebalance baseline ReD [54]; novel policy-constrain baseline PRDC [37] and return-conditioned baselines Decision Transformer and %BC [3] as the comparison baselines to further evaluate the superiority of our A2PO. The results are illustrated in Table 4. Our A2PO method demonstrates comparable performance to various offline RL methods. While the performance of A2PO is slightly inferior to that of PRDC in the medium and medium-replay dataset, it surpasses all additional baselines in the majority of scenarios.

D Analysis of Advantage Condition Input for Training

In this section, we provide a full comparison of different advantage condition computing methods for training on Gym tasks in Table 5. These computing methods are thoroughly described in Section 5.3. The A2PO algorithm degenerates into TD3+BC under ξ fix = 1 , which treats each sample constraint equally with no regard for the differences in the data quality. In this case, it achieves significantly worse performance compared to using varied ξ values, particularly on the random-expert datasets which highlight the substantial gap between the behavior policies. This observation highlights the potential risks of constraint conflict issues in policy regularization methods. On the other hand, the discrete ξ dis yields slightly inferior performances and larger variances compared to the continuous ξ , when different threshold values ϵ are used. While the ξ dis signals demonstrate more stability, the discretization process fails to accurately capture the advantage value associated with each sampled transition. Consequently, there is a mismatch between the advantage value and the actual sampled transition, which negatively impacts the overall algorithm performance. However, by utilizing the

continuous advantage value, our A2PO is able to better adapt to the varying environment dynamics and complexities found in the mixed-quality dataset, thereby further getting state-of-the-art performance with low variance. This enables a more precise capture of the relationship between the advantage condition and the disentangled behavior policies.

E Analysis of Advantage Condition Input for Testing

In this section, we provide a full comparison of different fixed advantage condition inputs for testing on Gym tasks in Table 6 as a supplement for Figure 4. It should be noted that in contrast to the ablation study shown in Appendix D which investigates different methods for computing advantages

during CVAE and agent learning, this section utilizes the original continuous advantage computation method. However, the designated ξ is varied during the testing process. On one hand, the CVAE decoder p ψ can generate CVAE-approximated behavior policies by varying the input ξ . On the other hand, Eq. 8 for policy improvement regulates the advantage-aware policy to closely resemble samples from the same advantage condition. This indicates that the quality of the generated action should be positively correlated with the specified action input. The experimental outcomes demonstrate the relationship between varying values of ξ and the corresponding A2PO policy returns: the agent performance consistently improves as the fixed advantage condition input ξ increases. Moreover, the gap between the returns increases as the offline dataset includes more diverse behavior policies. These observations serve as strong evidence for the effectiveness of A2PO in effectively disentangling the behavior policies within a multi-quality dataset.

F Analysis of the CVAE Policy

The CVAE policy corresponds to the CVAE decoder p ψ ( a | z 0 , c ∗ ) , where z 0 is sampled from N (0 , 1) ,and c ∗ = s || ξ ∗ . As described in Section 4.1, the advantage-aware CVAE utilizes the advantage condition computed by the agent critic to construct the ELBO loss, which is formulated as advantage-guided supervised learning [12]. The CVAE decoder p ψ ( a | z 0 , c ∗ ) generates action outputs of varying qualities based on the target advantage input signal ξ . By conditioning on the maximum normalized advantage value ξ ∗ to get the optimal action of CV AE and agent policy, we present thorough comparison results of the CVAE policy and agent policy in Table 7. The performance of the CVAE policy indicates that it only demonstrates superior performance in a limited set of tasks and datasets, specifically the hopper-random and walker2d-random environments. The A2PO agent consistently outperforms the CVAE agent in the majority of cases. This performance gap is particularly significant in the newly constructed mix-quality datasets including random-expert and random-medium-expert highlighting the conflict issue. These findings suggest that A2PO achieves well-disentangled behavior policies and an optimal agent policy, surpassing the capabilities of CV AE-reconstructed behavior policies.

G Analysis of A2PO Policy Optimization

In this section, we evaluate the effectiveness of the policy regularization term within the policy improvement step as shown in Equation 8. Previous approaches [56, 5] that construct agent policies within the CVAE latent space do not incorporate the BC regularization term, as the latent action

space inherently imposes constraints on the action outputs. In contrast, our advantage-aware policy integrates the BC term into the actor loss to explicitly ensure appropriate pessimism. Meanwhile, to guarantee the policy action outputs are consistent with the advantage condition ξ , we develop a variant of A2PO, designated A2PO [ a ∗ cvae ] , which constrains the agent output to the optimal CVAE policy p ψ ( ·| z 0 ∼ N (0 , I ) , c ∗ ) . Unlike this variant, the original A2PO utilizes the MSE loss to align the agent action output π ( ·| ˜ z ∗ , c ∗ ) with the sampled action a . To assess the impact of our A2PO architecture and the BC regularization, we conduct an ablation study by comparing the results of different A2PO variants and LAPO, as demonstrated in Table 8. The results indicate that even without the BC regularization term, A2PO consistently outperforms LAPO in most tasks. Furthermore, the original formulation of the BC term in the main text can enhance performance in most cases. This comparison highlights the superior performance achieved by A2PO, showcasing its effective disentanglement of action distributions from different behavior policies to enforce a reasonable advantage-aware policy constraint and obtain an optimal agent policy.

H Analysis of A2PO Under Mixed-quality Dataset with Different Proportion

In this section, we conducted additional experiments to assess the robustness of A2PO under various mixed-quality datasets. The mixed datasets consist of a fixed number of 1 × 10 6 offline samples. These datasets are generated by combining σ of either an expert or medium dataset (high-return) and 1 -σ of a random or medium dataset (low-return). The quantitative results are presented in Table 9. The results demonstrate that our A2PO algorithm is capable of achieving expert-level performance even with a limited number of high-quality samples. Moreover, as the proportion σ increases, the A2PO policy becomes more stable. These results indicate the robustness and effectiveness of our A2PO algorithm in deriving optimal policies across diverse structures of offline datasets.

I Advantage Visualization

In this section, we illustrate the distribution of initial state-action pairs in offline datasets across various tasks, along with the corresponding actual return and estimated advantage values. Figure 8 and 9 present a comparative analysis of the actual return, LAPO, and our A2PO advantage approximation. The findings indicate that LAPO exhibits limited discrimination in assessing transition advantages,

while A2PO effectively distinguishes between transitions of varying data quality. These results underscore the limitations of the AW method and highlight the superiority of our A2PO approach.

J Boarder Impact

This paper presents an offline advantage-aware learning approach that leverages the estimated advantage condition to deal with mixed-quality datasets. The advantage-aware concept brings a new perspective to the solution of real-world RL tasks, facilitating a more practical and effective utilization of the offline datasets. It has the potential to enhance the robustness of the agent towards the varying offline datasets from real-world RL scenarios, where the pre-collected offline datasets are noisy and often not as well-organized as the D4RL standardized datasets.

Actual Return

LAPO Advantage Estimation

A2PO Advantage Estimation

0

20

40

60

80

100

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

(a) walker2d-random-expert

Actual Return

LAPO Advantage Estimation

A2PO Advantage Estimation

0

20

40

60

80

100

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

(b) hopper-random-expert

Actual Return

LAPO Advantage Estimation

A2PO Advantage Estimation

0

20

40

60

80

100

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

(a) halfcheetah-random-medium-expert

Actual Return

LAPO Advantage Estimation

A2PO Advantage Estimation

0

20

40

60

80

100

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

(b) walker2d-random-medium-expert

Actual Return

LAPO Advantage Estimation

A2PO Advantage Estimation

0

20

40

60

80

100

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

(c) hopper-random-medium-expert

一

一

一

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix B

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix B

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: See supplemental material.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an offline reinforcement learning method called A2PO, which aims to solve the problem of constraint conflicts in mixed-quality datasets collected from multiple behavior policies. A2PO optimizes offline learning by explicitly constructing advantage-aware policy constraints, especially in the presence of data of variable quality. Specifically, A2PO employs a CVAE to disentangle the distribution of actions under different behavior policies, modeling the advantage values of all training data as conditional variables. The agent can then optimize the advantage-aware policies based on these disentangled action distribution constraints to achieve high advantage values. Through extensive experiments on single- and mixed-quality datasets from the D4RL benchmark, A2PO demonstrates its superior performance over other offline RL baselines and advantage-weighted competitors.\n\nSTRENGTHS:\n1.\tThe authors introduce CVAE to deal with the problem of disentangling behavior policies in mixed-quality datasets, which is novel to me.\n2.\tThe paper is well-structured and clearly written.\n3.\tSeveral analytical experiments are also provided to help understand the method\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1.\tAlthough this paper presents a new approach, the comparison with existing techniques may not be in depth enough and lacks an analysis of why the algorithm works. In addition, could the authors further discuss the conditions under which A2PO may fail?\n2.\tThis paper focuses on the analysis and comparison of advantage-weighted offline RL methods, ignoring recent work on policy constraints in the same relative as A2PO, such as PRDC: Policy Regularization with Dataset Constraint for Offline. It would be more convincing if the authors could add research and experimental comparisons of related work to demonstrate the advantages of A2PO among its relative methods (even SOTA methods).\n3.\tRegarding implementation details, how can we ensure a sufficient number of high-quality samples in the dataset to support training when selecting ξ = 1? Additionally, how can we guarantee the accuracy of the introduced advantage information?\n\nIn general, this work demonstrates commendable quality and I would be willing to raise my score if my concerns could be addressed.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
The authors propose an advantage aware offline RL algorithm for datasets consisting of data from multiple behavior policies. The algorithm consists of two steps, (1) behavior policy disentangling: Wherein a CVAE is trained to output actions conditioned on normalized advantage and state. (2) Policy optimization, where the policy is optimized based on the actions generated by the CVAE. The authors demonstrate the performance of the algorithms on standard benchmarks

STRENGTHS:
1. The paper is well written and easy to follow

2. The experimental section is extensive

WEAKNESSES:
1. **The Agent policy optimization is unclear**

    1. Why is a behavior regularization term (2nd term in (8)) needed when you have a generative model trained on the dataset? 

    2. Why not condition the 2nd in eq 8 term on c*? 

2. **Missing some return conditioned baselines**

    1. Comparison with return conditioned approaches such as %BC and Decision Transformers are needed to understand the advantage of the proposed approach

    2. Comparisons with data rebalancing techniques [1] can help understand methods with mixed datasets

[1] Yue, Yang, et al. "Boosting offline reinforcement learning via data rebalancing." arXiv preprint arXiv:2210.09241 (2022).

QUESTIONS:
See weaknesses

LIMITATIONS:
The authors address limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes using CVAE to train an agent and using a decoder, which conditions on state and advantage value, to generate actions. The Q function and V function are trained simultaneously, and the policy is trained in a TD3-BC style. Experiments show that the method can outperform baselines in most tasks.\n\nSTRENGTHS:\nThe paper is easy to follow, and the motivation and intuition for the algorithm design are reasonable. The experiments are comprehensive and justify the advantages of the method claimed by the authors. Conditioning on the advantage value, instead of using it as a coefficient, is interesting, allowing for sampling to make the advantage value extreme ($\\xi=1$) to get the optimal action.\n\nWEAKNESSES:\nI found some results in the experimental section problematic. More details are in the Questions section.\n\nQUESTIONS:\n1. In line 169, what prior distribution of $p(z)$ did you choose?\n\n2. When training the Q and V functions (Eq(7)), the authors claimed that training simultaneously can stabilize training. Do the authors have any intuition, theory, or literature to back up such a design? I ask because, in my memory, most methods that need to estimate the V function train Q and V separately, such as IQL[1] and SAC[2]. Simultaneous training may cause instability, similar to training GANs.\n\n3. In Table 1, I found some baseline results inconsistent with the original paper. For example, hopper-medium-v2 is 90.5 for Diffusion-QL in the original paper, but 70.3 in this paper; antmaze-umaze-diverse is 66.2 for Diffusion-QL in the original paper, but 24 in this paper. Most of the results for Diffusion-QL are not consistent with the original paper, while in line 234, the authors claim their results are from the original paper. Can the authors explain the mismatch?\n\n4. In line 463, the authors state they use the v2 version for antmaze tasks, while most other baselines, including IQL and Diffusion-QL, use the v0 version. In my experience, the v2 version dataset can always provide a higher score than the v0 version when using the same algorithm. It is unfair to compare A2PO's v2 results with the v0 results of other baselines.\n\n5. I found that the training curve of hopper random-medium-expert in Figure 4 is unstable. Can the authors provide more training curves for other environments?\n\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline reinforcement learning with implicit q-learning.\" arXiv preprint arXiv:2110.06169 (2021).\n\n[2]Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).\n\nLIMITATIONS:\nMy main concern is about the empirical experiments. It seems the algorithm doesn't have competitive performance compared to diffusion-based policy algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents A2PO. A2PO is an offline RL method for learning with datasets that was collected by a diverse set of poliices. The aim of the method is to disentangle the data being collected by each policy by using advantage calculation and then using this information to better learn policies from the dataset. Specifically, they use a VAE and condition it on advantage and state and then train in latent states. They perform a thorough set of experiments on benchmarks to compare to baseline methods. They also perform an ablation study to find the important qualities in their method.\n\nSTRENGTHS:\nOriginality. Average strength. The originalty of this work is what I would expect. They have with a novel idea of disentangling the policies using advantage estimation which builds upon the previous idea of advantage estimation. I think this work is of solid novelty.\n\nQuality. Major strength. The emperical analysis in this work is done very well. They compare to many baselines and show their method is clearly superior in many of the domains. They also perform a very good ablation study that shows the quality of their choises. I think this is a high point of the paper.\n\nClarity. Major strength. The paper is very well written, provides excellent figures and is easy to follow. Great job here.\n\nSignificance. Average strength. The work here seems like it takes a good step forward in the domain of offline RL with mixed quality datasets. The significance here is solid due to their impressive results but no more than I would expect from a good paper.\n\nWEAKNESSES:\nI don't have any major comments here. I believe this paper is complete, written well, has impressive results and is easy to follow.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThe limitation they address is that the method is slower than others, but no slower than some baselines they provide. I believe this is addressed well enough and improving upon the computation speed is clearly out of scope.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper introduces A2PO, a novel approach to offline reinforcement learning that addresses the constraint conflict issue in mixed-quality datasets by using a Conditional Variational Auto-Encoder to disentangle action distributions and optimize policies towards high advantage values. A2PO demonstrates superior performance over existing methods on the D4RL benchmark, showcasing its effectiveness in leveraging diverse offline datasets for more robust policy learning.

STRENGTHS:
see questions part

WEAKNESSES:
see questions part

QUESTIONS:
This paper focuses on the constraint conflict issue in offline reinforcement learning when dealing with mixed-quality datasets collected from multiple behavior policies. By using a Conditional Variational Auto-Encoder (CVAE), A2PO disentangles the action distributions of different behavior policies present in the mixed-quality dataset. The CVAE models the advantage values as conditional variables, allowing it to generate action distributions that reflect the behavior policies' distinct characteristics.
In general, this paper is interesting and well-written. One question is that since we can directly select the good trajectories according to the rewards, is there any ablation study to show that CVAE can do better than this simple method?

LIMITATIONS:
see questions part
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14