DASCO: Dual-Generator Adversarial Support Constrained Offline Reinforcement Learning
Authors: Quan Vuong
Aviral Kumar
Sergey Levine
Yevgen Chebotar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Offline Reinforcement Learning
Generative Adversarial Networks

Abstract: In offline RL, constraining the learned policy to remain close to the data is essential to prevent the policy from outputting out-of-distribution (OOD) actions with erroneously overestimated values. In principle, generative adversarial networks (GAN) can provide an elegant solution to do so, with the discriminator directly providing a probability that quantifies distributional shift. However, in practice, GAN-based offline RL methods have not outperformed alternative approaches, perhaps because the generator is trained to both fool the discriminator and maximize return - two objectives that are often at odds with each other. In this paper, we show that the issue of conflicting objectives can be resolved by training two generators: one that maximizes return, with the other capturing the "remainder" of the data distribution in the offline dataset, such that the mixture of the two is close to the behavior policy. We show that not only does having two generators enable an effective GAN-based offline RL method, but also approximates a support constraint, where the policy does not need to match the entire data distribution, but only the slice of the data that leads to high long term performance. We name our method DASCO, for Dual-Generator Adversarial Support Constrained Offline RL. On benchmark tasks that require learning from sub-optimal data, DASCO significantly outperforms prior methods that enforce distribution constraint.


Dual Generator Offline Reinforcement Learning
Quan Vuong', Aviral Kumar Sergey Levine Yevgen Chebotar'
'Google Research, UC Berkeley
Abstract
In offline RI., constraining the learned policy to remain close to the data is essential to prevent the policy from outputting out-of-distributtin (OOD) actions with erro- neously overestimated values. In principle, generative adversarial networks (GAN) can provide elegant solution to do so, with the discriminator directly providing a probability that quantifies distributional shift. However, practice, GAN-based offline RL methods have not performed as well alternative approaches, perhaps because the generator is trained to both fool the discriminator and maximize return two objectives that can be odds with each other. In this paper. show that the issue conflicting objectives can be resolved by training two generators: one that maximizes return, with the other capturing the "remainder" of the data distribution in the offline dataset, such that the mixture of the two close to the behavior policy. We show that not only does having generators enable effective GAN-based offline RL method, but also approximates a support constraint, where the policy does not need match the entire data distribution, but only the slice of the data that leads to high long term performance. We name our method DASCO, for Dual-Generator Adversarial Support Constrained Offline RL. On benchmark tasks that require learning from sub-optimal data, DASCO significantly outperforms prior methods that enforce distribution constraint.
Introduction
Offline reinforcement learning (RL) algorithms aim to extract policies from datasets of previously logged experience. The promise of offline RL to extract decision making engines from existing data [32]. Such promise is especially appealing in domains where data collection is expensive or dangerous, but large amounts of data may already exists (e.g., robotics, autonomous driving, task- oriented dialog systems). Real-world datasets often consist of both expert and sub-optimal behaviors for the task of interest and also include potentially unrelated behavior corresponding to other tasks. While not all behaviors in the dataset relevant for solving the task of interest, even sub-optimal trajectories can provide an RL algorithm with some useful information. In principle, if offline RL algorithms can combine segments of useful behavior spread across multiple sub-optimal trajectories together, the combined segments can then perform better than any behavior observed in the dataset.
Effective offline RL requires estimating the value of actions other than those that were taken in the dataset, so as to pick actions that are better than the actions selected by the behavior policy. However, this requirement introduces a fundamental tension: the offline RL method must generalize to new actions, but should not attempt to use actions in the Bellman backup for which the value simply cannot be estimated using the provided data. These are often referred to in the literature out-of- distribution (OOD) actions [28]. While a wide variety of methods have been proposed to constrain offline RL to avoid OOD actions [26, 14, 1]. the formulation and enforcement of such constraints can be challenging, and might introduce considcrable complexity, such as the need to explicitly estimate the behavior policy [49] or evaluate digeensnnnii integrals [30]. Generative adversarial networks (GANs) in principle offer an appealing and simple solution: use the discriminator to estimate whether action is in-distribution, and train the policy as the "generator" in the GAN formulation to fool
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

this discriminator. Although some prior works have proposed variants on this approach [49], it has been proven difficult in practice as GANs can already suffer from instability when the discriminator is too powerful. Forcing the generator (i.e., the policy) to simultaneously both maximize reward and fool the discriminator only exacerbates the issue of an overpowered discriminator.
We propose a novel solution that enables the effective use of GANs in offline RL, in the process not only mitigating the above challenge but also providing more appealing form of support constraint that leads to improved performance. Our key observation is that the generative distribution in GANs can be split into two separate distributions, one that represents the "good parts" of the data distribution and becomes the final learned policy, and an auxiliary generator that becomes the policy's s complement, such that their mixture is equal to the data distribution. This formulation removes the tension between maximizing rewards and matching the data distribution perfectly; as long as the learned policy is within the support of the data distribution, the complement will pick up the slack and model the "remainder" of the distribution, allowing the two generators together to perfectly fool the discriminator. however the policy ventures outside of the data support, the second generator cannot compensate for this mistake, and the discriminator will push the policy back inside the support. We name our method DASCO, for Dual-Generator Adversarial Support Constrained Offline RL.
Experimentally, we demonstrate the benefits of our approach, DASCO, on standard benchmark tasks. For offline datasets that consist of combination of expert. sub-optimal and noisy data, our method outperforms distribution instrbutiio-cconstraneed offline RL methods by a large margin.
2 Related Work
Combining sub-optimal trajectories to obtain nigh-performing policies is central promise of offline RL. During offline training, querying the value function on unseen actions often leads to value over-estimation and collapse in learning progress. To avoid querying the value functions on out- of-distribution actions, existing methods encourage the learned policies to match the distribution of the behavior policies. This principle has been realized with variety of practical algorithms [22, 49, 41, 42, 49. 29, 26, 25, 47, 12, 9, 15, 21, 37, 10. 34j. For example, by optimizing the policies with respect to a conservative lower bound of the value function estimate [30], only optimizing the policies on actions contained in the dataset [26]. or jointly optimizing the policy on the long-term return and behavior cloning objective [12]. While explicitly enforcing distribution constraint by adding the behavior cloning objective allows for good performance on near-optimal data, this approach fails to produce good trajectories on sub-optimal datasets [26]. Methods that implicitly enforce distribution constraints, such as CQL and IQL, have seen more successes on such datasets. However, they still struggle to produce near-optimal trajectories when the actions of the dataset generation policies are corrupted with noise or systematic biases (a result we demonstrate in Section 5).
However, enforcing distribution constraints to avoid value over-estimation may not be necessary. It is sufficient to ensure the learned policies do not produce actions that are too unlikely under the dataset generation policy. That is, it not necessary for the learned policy to fully cover the data distribution, only to remain in-support [29, 27, 32, 49, 51, 8]. Unfortunately, previous methods that attempt to instantiate this principle into algorithms have not seen as much empirical success as algorithms that penalize the policies for not matching the action distribution of the behavior policies. In this paper, we propose a new GAN-based offline RL algorithm whose use of dual generators naturally induce support constraint and has competitive performance with recent offline RL methods. In number of prior works, GANs have been used in the context of imitation learning to learn from expert data [20, 33, 19, 35]. In this work, we show that dual-generator GANs can be used to learn from sub-optimal data in the context of offline RI.
To enforce the support constraint, we use the discriminator score to measure how likely an action is under the behavior policy. To measure uncertainty, one could also use an ensemble of value functions, as is done in [2. 3]. We include [2] as baseline in Table and demonstrate the our method has significantly higher performance. Nevertheless, it would be interesting as future work to combine the discriminator score and an ensemble of value functions to devise a higher performing algorithm.
3 Background
Let M = (S,A, P. P,R,R 7) define a Markov decision process (MDP), where S and A are state and action spaces, P:S A S + E__ is a state-transition probability function, R: S x A + Ris
2

---

reward function and y is discount factor. Reinforcement learning methods aim at finding a policy T (als) that maximizes the expected discounted reward R(r) -i= 7' R(s;, az) over trajectories T (Sp, dpy..., Sy, ar) with time horizon T induced by the policy π.
In this work, we concentrate on the offline or off- off--poliy RL setting, i.e. finding an optimal policy given a dataset of previously collected experience popular family of methods for offline learning are based on training a Q-function through dynamic programming using temporal-diiffrence (TD) learning [48, 44J. Such methods train Q-function to satisfy the Bellman equation:
D by a behavior policy TT. particularly
Q(s..a)) R(st, ax) + y (8)++,
In Q-learning, the policy is replaced with maximization, such that (a]s)) arg max, Qa(8, a), while actor-critic methods optimize a separate parametric policy (a(a))) that maximizes the Q- function. In this work, we extend the Soft Actor-Critic (SAC) method [18] for learning from diverse offline datasets.
Generative Adversarial Networks (GANs) [17] enable modeling a data distribution pp through an adversarial game between generator G and discriminator D:
min max Exupp log D(x)) + Ez=p(s) log(1 D(G(2)))
D
For this two player zero-sum game, [17] shows that for a fixed generator G, the optimal discriminator is (z)x)=
(a() and the optimal generator matches the data distribution (z PD-
(D( + pc(
GAN has been extended to the offline RL setting by interpreting the discriminator function as a measure of how likely an action is under the behavior policy, and jointly optimizing the policy to maximize an estimate of the long-term return and the discriminator function [49]:
min max Bs...po.............o.. [log( (log(D(s,ssoooo a))] + Be--p,,o~7(a))loggl - D(s, a))] - -E--pp (Q),
(2)
where Q(s, a) is trained via the Bellman operator to approximate the value function of the policy T (a]s)) This leads to iterative policy evaluation and policy improvement rules for the actor and the policy [49]. During the htth update step, given the most recent values for the policy nh the value function Qt, and the discriminator D&, we perform the following updates to obtain the next values for the value function and the policy:
Q++1 + --rg min Mi E. ((R(s, + "Bsr 12 (s'),'))) - Qrarge((s,a)))
(3)
A+1 arg max EawD) [Q (s, a) log D (s)
where the log (alss) term in the policy objective aims at regularizing the learnt policy to prevent from outputting OOD actions. In practice, training the policy to maximize both the value function and discriminator might lead to conflicting objectives for the policy and thus poor performance on either objective. This can happen when the data contains a mixture of good and bad actions. Maximizing the value function would mean avoiding low-reward behaviors. On the other hand, maximizing the discriminator would require outputting all in-distribution actions, including sub-optimal ones. Our approach alleviates this conflict and enables in support maximization of the value function when learning from mixed-quality datasets.
4 Dual-Generator Adversarial Support Constraint Offline RL
We now present our algorithm, which uses a novel dual-generator GAN in combination with weighting method to enable GAN-based offline RL that constrains the learned policy to remain within the support of the data distribution. We call our method Dual- gene-ageeratrr Adversarial Support Constraint Offline RL (DASCO). We will first introduce the dual-generator training method generically. for arbitrary generators that must optimize a user-specified function f f(xr) within the support of the data distribution in Section 1. We will then show this method can be incorporated into a complete offline RI. algorithm Section 4.2 in combination with our proposed weighting scheme, and then summarize the full resulting actor-critic method in Section 4.3.
3

---

4.1 Dual generator in-support optimization
volx) (G)
pola)
F(a)
In this section, we will develop an approach for performing a joint optimization of adversarial and secondary objectives of the generator GAN framework, which we will then apply to offline RI. This is necessary component for performing the joint optimization in Eq. 2 with- out introducing a conflict of these objectives. All proofs for theorems presented in this section are Appendix
Pulir)
Py(a)
Paua
Figure I: Visualizations to illustrate the benefit of dual generators over single generator when max- imizing a secondary objective f(x) in the GAN framework. In both figures, PD (x: is the data dis- tribution. The x-axis is one-dimensional sample space. Left: In this figure, since there is only sin- gle generator, the generator G is trained to jointly maximize the objective fa)) and matches the data distribution pp (4)). The distribution pa induced by the generator is thus not very good either maximizing the objective f(r) or matching the data distribution. Right: In this figure, have two generators, inducing two distributions PG and Paux- By introducing the auxiliary generator Gauz into the GAN framework, the primary generator can better maximize the objective f(x) while stay- ing within the support of the data distribution pp. The mixed distribution also perfectly matches the
Let's consider general objective that requires training a generator G to fool the diseriminator D while also optimizing the expected value of some other function f:
min max G D
Egppp [lo (x()))
+ p()) - D(G(2)))
+ E (x) [[(())
(4)
where the first two terms are the same the objective of the GAN formulation. We have also added an additional term [f where f is a mapping from the generator output to scalar value. The third term represents secondary objective that the generator should optimize.
Theorem 4.1 The optimal genera- for of Eq. 4 induces a distribution -(z)~
data distribution, i.c. Py (z(x) + Pauz PD(x).
2 Note that in these two figures, the primary gen- erator aims to maximize f(z) (instead of mini- mize) to allow for more intuitive interpretation. In Appendix B, we provide numerical example to further illustrate the benefit of the dual generator technique on 1D example.
(z)x) = po(x) 2- e- where v > is the Lagrange multiplier that ensures that (z()) normalized to
We can see that by adding a secondary objective function for the generator, in general, the opti- mal generator does not attempt to match the data distribution (r(x) anymore, but instead tries to (xx)-v
match the data distribution weighted by 2 e- f(x))=" We expect that such case, the discriminator
clearly has an advantage in the two player zero-sum game and will be able to distinguish between real samples and sample generated by the generator.
To allow the generator to specialize in optimizing the secondary objective function, we propose to introduce second auxiliary generator that matches the portion of the data distribution that is not
Pg + Paur
well captured by the primary generator. Let Pmix
consider the min- min-max problem:
2
min max Ez-pp (log(( (x))) + Ex-ppix [log(1 - D(x)) + Ez-p, [f()).
(5)
G.G.sa D
where we mix samples from the primary generator G and the auxiliary generator Gauz to generate samples that can fool the discriminator. The mixing is indicated by the distribution Pmiz in the second term of Eq. 5. The first and third term of Eq. 5 are the same as the objective in Eq. 4.
We next theoretically demonstrate the benefit of adding the auxiliary generator to the GAN formulation with the following Theorem.
Theorem 4.2 (Informal) The primary generator PG performs in-support optimization of f(x)-
We first note that the optimal solution of the mixed distribution from Eq. 5 is the real data distribution:
Pauz (x) + D2 (x)
(x(x)
(6)
2

---

Accordingly, the optimal auxiliary generator distribution can be expressed as
Pawe (x) 2pp (z)
Let z0 to be the element inside the support of the data distribution pD that minimizes f. That is:
arg min f(x)
=ESupp(p))
When optimizing the secondary objective f(x), the primary generator will maximize the probability mass of in-support samples that maximize fx)). However, Eq. 7 introduces constraint that enforces 2pp (2: - (x) for Paua (x) remain a valid distribution. This leads us to conclude that the optimal primary generator Pg assigns the following probability to To:
(x0)) (2)) if 2pp(2o) <
(8)
otherwise
Interestingly, if the global optimum zo is not taking the full probability mass, the rest of the probability mass is redistributed to the next best in-support optimum, which we can define recursively:
2pp(x;) ri-1
if E <1
For Vi E
arg min [(x), pf((;)) = (eSupp(oo)\((
if Σ ppgp)((a)>>>
(9)
() =1
We provide more explanation for the solution in Eq. 9. In the first case, pf(2,) (pp( if 20= (1rz) < 1. That is, if the optimal solution for the primary generator Py assign the probability (x;( to the jth in support minima of fxx) without the total sum of probability assigned Pg (1z) going over 1, then the primary generator P2 will assign the probability 2pp(2;; (x;) to dz-
In the second case, (xa) probability 2pp(at)) to the ith in support minima of (xx) the total sum of probability assigned Po=o (2rr) goes over 1, then the primary generator P will assign the remaining probability 1- 1-E (z²) to 24. In the third case, the generator assigns probability 0 to because all the probability has already been assigned.
r(,)) if P
1. That is, if by assigning the
2j=0
To summarize the benefit of dual generator, we note that by introducing auxiliary generator and mixing with the primary generator, not only does the optimal solution for the mixed distribution match the real data distribution, but also the primary generator can better optimize the secondary objective f on the part of the domain of f that within the support of the data distribution pD. To better illustrate the benefit, provide visual explanation of the benefit in Figure 1.
4.2 Update rules for offline reinforcement learning
We will now incorporate the dual- qual-generator method to train policies for offline RL, based on optimiz- ing the joint objective from Eq. 5. The updates for the actor and the critic are generally similar to Eq. 3. However, simply combining Eq. and Eq. still allow the policy to exploit errors in the value function during the policy improvement step. We therefore augment the policy improvement step with an adaptive weight on the Q-value.
More concretely, as the policy improvement step samples actions from the current policy iterate n optimize the policy objective, as we observe our experiments. To alleviate this issue, use the probability assigned to the sampled actions by the discriminator to weight the value function estimates in the policy objective, leading to the following updates:
Qu+1 + arg min E, E.a.N'
[(R(s. a) + "a (Q' ,a)I) Qtarget(s, a)) a))
(10)
Q
(*(ssa) (p(
π f arg max D.a..
Q @ + log D (s,
(1I)
where ap the action from the offline dataset. The term (s, as) down-weights the contribution of the gradient of the value function to the policy update if the discriminator deems the sampled
5

---

action too unlikely. We further calibrate the probability (*, a) by dividing it with the probability (* ap (s)) that the discriminator assigns to the dataset action @p(s). It should be noted that during optimization the gradients are not propagated into these weights.
Next, we define the update rules for the auxiliary generator and the discriminator. We mix the samples from the tt iterate of the policy nh and the distribution Paur induced by the hthh iterate of the
π + Paur At every iteration k, we update the tth
auxiliary generator Gaaz: that is, let Pmix
iterate of the auxiliary generator G aur and discriminator D using the objectives:
G t arg min Ra- [og(1 - (8, ou0
(12)
Gaua
D arg max Ex-po log(D* (s.a))] Ex-pmiz [log(1 (s,
(13)
4.3 Algorithm summary
Algorithm provides step-by-step description of our algorithm. At every training step, we sample a batch transitions from the offline dataset and proceed to update the parameters of the value function, the policy, the auxiliary generator and the discriminator in that order.
Algorithm DASCO algorithm summary
Initialize Q-function Qo, policy Tg auxiliary generator Gauz,gh, discriminator D
2: for training step k in [I....N] _N} do
3: (s, a, r, 87) + D: Sample batch of transitions from the dataset
4: gk+1 + Update Q-function Qe using the Bellman update in Eq. 10
5: @++1 + Update policy गे using the augmented objective in Eq. 11
gi + Update auxiliary generator Gaur,y using the objective in Eq. 12
6:
7:
w k+1
+ Update discriminator D using mixed samples from To and Gauz, as in Eq. 13
5
Experiments
Our experiments aim answering the following questions:
1. When learning from offline datasets that require combining actions from sub-optimal trajec- tories, does DASCO outperform existing methods that based distribution constraints?
2. On standard benchmarks such as D4RL [1]], how does DASCO compare against recent methods?
3. Are both the dual generator and the probability ratio weight important for the performance of DASCO?
5.1 Comparisons on standard benchmarks and new datasets
For our first set of experiments, might encounter when using offline RL algorithms on real world data. These datasets introduce additional learning challenges and require the algorithm to combine actions in different trajectories to obtain good performance. We use the existing AntMaze environments from the D4RL suite [11]: antmaze-medhie and antmaze-large. In these two environments, the algorithm controls 8-DoF "Ant" quadruped robot to navigate 2D maze to reach desired goal locations. The D4RL benchmark generates the offline datasets for these two environments using two policies: 1. a low-level goal reaching policy that outputs torque commands to move the Ant to nearby goal location and a high-level waypoint generator provide sub-goal, that guide the low-level goal-reaching policy to the desired location. We use the same two policies to generate two new classes of datasets.
introduce four new datasets to simulate the challenges one
For the noisy dataset, we add Gaussian noise to the action computed by the low-level goal-reaching policy. The noise variance depends on the 2D location of the Ant in the maze larger some 2D) regions than others. We intend this dataset to be representative of situations where the data generation

---

Table 1: Performance comparison to baselines when learning from the noisy and biased AntMaze datasets. Our method outperforms the baselines significantly. The value in parenthesis indicates the standard deviation of mean episode return, computed over 3 different runs.
Dataset
BEAR
EDAC
CQL
IQL
DASCO (Ours)
antmaze-large-bias
61.7 (3.5)
41.0 (7.9)
63.9 (6.0)
lntmaze-lagee-nissy
50.3 (2.3)
39.0 (6.4)
54.3 (2.0)
antmaze-medium-bias
0.0 (0.0)
0.0 (0.0)
66.7 (2.9)
48.0 (5.9)
90.2 (2.4)
antmaze-medium-nniiy
0.0 (0.0)
(0.0)
55.7 (4.7)
44.3 (1.7)
86.3 (4.5)
noisy and biased antmaze-v2 total
234.4
172.3
294.7
Table Performance comparison distribution-coostraane baselines on AntMaze tasks in D4RL.
Our algorithm outperforms the baselines when learning from the diverse and play datasets.
Dataset
CQL
IQL
DASCO (Ours)
antmaze-umaze
97.0 (0.8)
90.3 (1.9)
99.2 (0.0)
antmaze-umaze-dNie diverse
58.7 (12.2)
70.3 (4.6)
89.0 (1.7)
autmaze-nediumn-ply
77.0 (1.6)
82.7 (0.5)
92.3 (1.5)
antmaze-mecdun--divens
80.0 (0.0)
82.3 (1.9)
87.1 (0.4)
antmaze-large-play
53.3 (4.6)
(3.1)
64.4 (1.7)
aatmaze-large-divrsse
48.0 (2.9)
50.0 (3.6)
74.1 (2.8)
antmaze total
414.0
431.3
506.1
policies more deterministic in some states than others [31] robot picking up object has many good options to approach the object, but when the robot grasps the object, its behavior should be more deterministic ensure successful grasp without damaging or dropping the object [38].
For biased dataset, in addition to adding Gaussian noise to the actions as it is done in the noisy dataset, we also add bias to the actions computed by the low-level policy. The values of the bias also depend on the current 2D location of the Ant in the maze. This setting is meant to simulate learning from relabelled data, where the dataset was generated when the data generation policies were per- forming a different task than the tasks we are using the dataset to learn to perform. Relabelling offline data is popular method for improving the performance of offline RL algorithms [46, 43], especially when we have much more data for some tasks than others [23]. In the AntMaze environment, offline RL algorithms must combine data from sub-optimal trajectories to learn behaviors with high returns. In addition, noisy and biased datasets present a more challenging learning scenarios due to the added noise and systematic bias which vary non-uniformly based on the 2D location of the Ant.
Table illustrates the performance comparison our method and representative methods that enforce distribution constraints, either through optimizing conservative lower bound of the value estimates (CQL) or only optimizing the policy on actions in the dataset using Advantage Weighted Regression [41] (IQL). Our method outperforms both CQL and IQL. In these tasks, to ensure fair comparison between different methods, we perform oracle offline policy selection to obtain the performance estimates for CQL, IQL, and method. We describe how tune syper-paramette baselines in Appendix E. We also include results of recent methods, such as EDAC [2] and BEAR [28]. We also compare the performance standard AntMaze tasks when learning from the datasets in the D4RL benchmark without modifications in Table 2. In these tasks, our method outperforms IQL by large margin two diverse datasets.
for the
By comparing the results in Table (learning from noisy and biased datasets) and Table (learning from existing offline datasets in D4RL), also note that our proposed algorithm outperforms distribution-consrain offline RL algorithms (CQL, IQL) more consistently when tested on the noisy and biased datasets. For the results in these two tables, the definition of the antmaze-medium and antmaze-large environments are the same. The only axis variation the learning setup is the noise and systematic bias added to the actions of the dataset generation policies. We therefore conclude that our algorithm is more robust to the noise and systematic bias added to the actions than distribution-conntrainedd offline RL algorithms.
Next, we evaluate our approach on Gym locomotion tasks from the standard D4RL, suite. The performance results on these tasks are illustrated in Table 3. Our method is competitive with BC,
7

---

Table 3: Performance comparison with recent offline RL algorithms on the Gym locomotion tasks
Dataset
BC
10%BC
DT [7]
AWAC [39]
Onestop RL [6]
TD3+BC [13]
COMBO [50]
CQL
IQL
DASCO (Ours)
36.6
40.6
36.6
40.5
38.1
44,6
55.1
45.5
44.2
44.7
18.1
75.9
82.7
37.2
97.5
60.9
89.5
95.0
94.7
101.7
walker2d-mee medium- replay
26.0
62.5
66.6
27.0
49.5
81.8
56.0
77.2
73.9
74.5
halfcheerah-ne saafeheerah-meiium---- exppett
55.2
92.9
86.8
42.8
93.4
90.7
90.0
91.6
86.7
93.8
supper-mediunne expert
52.5
110.9
107.6
55.8
103.3
98.0
111.1
105.4
91.5
110.9
walker2d medium expert
107.5
109.0
108.1
74.5
113.0
110.1
103.3
108.8
109.6
108.3
500.6
Table 4: Ablation for training without and with auxiliary generator. The dual generator technique, which trains the auxiliary generator in addition to the policy, is crucial to obtain good performance.
Dataset
Without
With
malfcheetta--mediu-eeee
79.8 (3.4)
93.8 (0.1)
hopper-mediin-eepprr
95.1 (1.6)
110.9 (0.8)
antmaze-large-hne
55.0 (2.3)
63.9 (6.0)
antmaze-large. harge-noisy
45.1 (L.8)
54.3 (2.0)
one-step offline RL methods [5], and multi-step distribution-constraiit RI. methods 30). This is not surprising because in these tasks, the offline dataset contains large number trajectories with high returns. In Table 3, 10%BC refers to performing BC using the top 10% trajectories with the highest episode return.
5.2 Ablations
We conduct three different sets of experiments to gain more insights into our algorithm. The first experiment measures the importance having an auxiliary generator. We recall that there are two benefits to having the auxiliary generator. Firstly, without the auxiliary generator, the generator does not in general match the data distribution (Theorem 4.1). As such, the discriminator has an unfair advantage in learning how to distinguish between real and generated examples. Secondly, the auxiliary generator plays the role of a support player and learns to output actions that are assigned non-zero probability by the data distribution, but have low Q values. The support player allows the policy to concentrate on in-support maximization of the Q-function (Theorem 4.2). Table demonstrates that having an auxiliary generator clearly leads to performance improvement across different task families, from Gym locomotion tasks to AntMaze navigation tasks.
The second experiment compares the performance of the policy and the auxiliary generator on a subset of the Gym locomotion and AntMaze tasks (Table 5). The difference in the performance of the policy and auxiliary generator illustrates their specialization of responsibility: the policy learns to output actions that lead to good performance, while the auxiliary generator learns to model the "remainder" of the data distribution. If this "remainder" also contains good action, then the auxiliary generator will have non-trivial performance. Otherwise, the auxiliary generator will have poor performance.
In the Gym locomotion tasks, the auxiliary generator has non-trivial performance, but it is still worse than the policy. This demonstrates that: 1. By optimizing the policy to maximize the long-term return and the discriminator function, the policy can outperform the auxiliary generator, which only maximizes the discriminator function, 2. The dataset contains large fraction medium performance level actions contained in continuous trajectories, which the auxiliary generator has learnt to output. In contrast, in the bias and noisy AntMaze tasks, the auxiliary generator fails to obtain non-zen zero performance while the policy has strong performance. This reflects the necessity of carefully picking a subset of the in- suppot actions to obtain good performance.
The third set of experiments illustrates the importance of weighing the value function in the policy objective by the probability computed by the discriminator, as described in Eq. 11. Doing so provides a second layer of protection against exploitation of errors in the value function by the policy. Table 6 illustrates that this very important for the AntMaze tasks, which require combining optimal and sub-optimal trajectories to obtain good performance. Perhaps this is because learning from such trajectories necessitates many rounds offline policy evaluation and improvement steps, with each
round creating opportunity for the policy exploit the emrors in the value estimates. On the other
8

---

Table 5: Policy vs Auxiliary Generator. The auxiliary generator has reasonable performance on the easier locomotion tasks and is significantly worse than the policy on the harder AntMaze tasks.
Dataset
Auxiliary Generator
Policy
halfcheetah ceeidduuu-epppree
48.5 (2.1)
93.8 (0.1)
Supper-medium-expeee
70.4 (0.9)
110.9 (0.8)
antmaze-large-bia
0.0 (0.0)
63.9 (6.0)
antmaze-large-noisy
0.0 (0.0)
54.3 (2.0)
Table Ablation for dynamic weighting of value function estimates in the policy objective. When learning from datasets that require combining actions across trajectories, such as the AntMaze tasks, using the dynamic weighting vital obtaining good performance.
Dataset
Without
With
halfchhetah-meedin--eexye medium--exprrr
91.1 (1.1)
93.8 (0.1)
hopper-medium-expee
106.7 (2.9)
110.9 (0.8)
antmaze- antmaze-large- large-play
0.0 (0.0)
64.4 (1.7)
antmaze-largee lang--z--aree-diverse
0.0 (0.0)
74.1 (2.8)
hand, the dynamic weight is less important in the Gym locomotion tasks, presumably because significant fraction the corresponding offline datasets has high returns and therefore incorporating sub-optimal data is less criticial to obtain high performance.
6 Conclusions
In this paper, we introduced DASCO, a GAN-based offline RL method that addresses the challenges of training policies as generators with discriminator to minimize deviation from the behavior policy by means of two modifications: an auxiliary generator to turn the GAN loss into support constraint, and value function weight in the policy objective. The auxiliary generator makes it possible for the policy to focus on maximizing the value function without needing to match the entirety of the data distribution, only that part of that has high value, effectively trrnng the standard distributional constraint that would be enforced by conventional GAN into kind of support constraint. This technique may in fact be of interest other settings where there a need to maximize some objective in addition to fooling discriminator, and applications of this approach outside of reinforcement learning are an exciting direction for future work. Further, since our method enables GAN- strategies attain good results range of offline RL benchmark tasks, it would also be interesting in future work to consider other types of GAN losses that induce different divergence measures. We also plan to explore robust methods for offline policy and
selection in the future.
Potential negative societal impact Because our focus on developing a generic offline RI. algorithm and not its application in any particular domain, our method inherits the potential negative societal impacts that any offline RL algorithm might have, such data bias and lack of robustness to distributional shift. We also note that how perform effective offline policy selection and evaluation the general case remain challenging. Therefore, the offline learned RL policies might not be easily deployed to solve real world problems where online policy evaluation is expensive.
References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. CoRR, abs/2110.01548, 2021. URL https://arxiv. org/ abs/2110 01548.
[3] Chenjia Bai, Lingxiao Wang. Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning, 2022. URL https: / //wxivv.vv htggabb//220222 11566.

---

[4] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[5] David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off- off-po evaluation. arXiv preprint arXiv:2206.0888 2021.
[6] David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline RL without ooo-po evaluation. CoRR, abs/2106.08909, 2021. URL https:///axxiii org/ /bbs/ 2106. 08909.
[7] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.013: 2021.
[8] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage- policy optimization for offline rl. arXiv preprint arXiv:2203.0888 089499 2022.
[9] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning, 2019. URL https:
/ //arxiv. org/ org/abs//990 aas.//910. 12179.
[10] Paul Daoudi, Merwan Barlier, Ludovic Dos Santos, and Aladin Virmaux. Density estimation for conservative q-learning, 2022. URL httptts///peeereeiiee wwwprrewwww...eeffeeeeeeeeeeee meet/forrmmid-aae
[11] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv, 2020. URL https: //axiivaaivvvai/iai org/pdf 22004. 07219.
[12] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. arXiv preprini arXiv:2 106.068600 2021.
[13] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. CoRR, (ins/200600866) 2021. URL ptttts::/arrte http:///aawwwwwwaabaaaee borg/abs/20e /206 06860.
[14] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration, 2018. URL https: //tpxii////vvvv////a org/ wwwpg/g////aareeer/ooo/eeeeeeee/ 1aa.. 02900.
[15] Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for of- fline hindsight information matching. In International Conference on Learning Representations, 2022. URL https: /ttp::::///////ie/eeeeeeee net/
[16] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Neural Information Processing Systems (NIPS), 2014.
[17] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL
https: /ttpx://arxiv... org/ abs/1406..666
[18] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In arXiv, 2018. URL https: / //arxiv.org wttpf///11100....... 01290. pdf.
[19] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav S. Sukhatme, and Joseph Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, NeurIPS, pages 1235-1245, 2017.
[20] J. Ho and S. Ermon. Generative adversarial imitation learning. In Neural Information Processing Systems (NIPS), 2016.
[21] Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference Learning Representations, 2022. URL attpps://opereeviee
[22] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint 0rXiv:1907.00453 2019.
[23] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic
10

---

reinforcement learning at scale. CoRR, abs/2104.08212, 2021. URL abs/2104. 08212.
[24] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL https://arxiv org/ abs/ 0bbs/442.6990.
[25] Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q-learnin. arXiv preprint 0rXiv:2110.061 2021.
[26] Nya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. arXiv preprint 103.0805, 2021.
[27] Aviral Kumar. Data-driven deep reinforcement learning. https://bair: http://baarrbebeeee..e.e. edu/
blog/2019/ 1t/11//00//////// 2019. BAIR Blog.
[28] Aviral Kumar. Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. In Neural Information Processing Systems (NeurlPS), 2019.
[29] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-pooliy q-learning via bootstrapping error reduction. In Advances Neural Information Processing Systems, pages 11761-11771 2019.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
[31] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning?, 2022.
[32] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01664 2020.
[33] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, NeurlPS, pages 2017.
[34] Liu Liu, Ziyang Tang, Lanqing Li, and Dijun Luo. Robust imitation learning from corrupted demonstrations, 2022. URL https: http:///ppeeeeeeeeeeeeeeeeeeeeet mtt//ooruuridd-UECcrrrrooo
[35] Zhihan Liu, Yufeng Zhang. Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Provably efficient generative adversarial imitation learning for online and offline setting with linear function approximation. CoRR, abs/2108.0877 2021. URL
/208 08765.
[36] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks, 2016. URL. taattps:/aarxeee www.aaiwwvv.aaaaaaaaaaaae org/ hatg///b///// abs/1611. 1611. 04076.
[37] Linghui Meng, Muning Wen, Yaodong Yang. chenyang le, Xi yun Li, Weinan Zhang, Jun Wang, and Bo XU. Offline pre-trained multi-agent decision transformer, woee//ffrummdddNOIoooo MQ...rr..
Li, Haifeng Zhang, Ying Wen,
2022. URL https: /pperr/eeeeeeeeeeeeeeeee
[38] Richard M. Murray, S. Shankar Sastry, and Li Zexiang. A Mathematical Introduction to Robotic Manipulation. CRC Press, Inc., USA, 1st edition, 1994. ISBN 0849379814.
[39] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online re- inforcement learning with offline datasets. CoRR, abs/2006.09359, 2020. URL https:
//axxiv.org abs/2006 09359.
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perforrmncc deep learning library, 2019. URL http:/ //wxx//xvvv.//////////////////a/ htt//bbs/1912/ habs/1912.01703. cite 90mmmmmmmmnnmnm 12 pages, 3 figures, NeurIPS 2019.
[41] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable oof-po reinforcement learning. arXiv preprint arXiv:1910.0011 2019.
[42] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, and Martin Riedmiller. Keep doing what worked:
11

---

Behavioral modelling priors for offline reinforcement learning. arXiv preprint crX1v::0022.88:99 2020.
[43] Charlie Snell, Mengjiao Yang, Justin Fu, Yi Su, and Sergey Levine. Context-awaree language
modeling for goal-oriented dialogue systems, 2022. URL 10198.
22204
[44] Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
[45] Casper Kaae Sønderby, Jose Caballero, Lucas Theis, Wenzhe Shi, and Ferenc Huszár. Amor- tised map inference for image super-resolution, 2016. URL 04490.
[46] Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Hao Su, and Henrik Iskov Christensen. Multi-task batch reinforcement learning with metric learning. CoRR, abs/1909. 1373, 2019.
URL http: /tatt:://xxiiv org/ abs/ 1909..999000 1909.11373.
[47] Ziyu Wang, Alexander Novikov, Konrad Zoina, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. arXiv preprini arXiv:2006. 15134, 2020.
[48] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):27--292,
1992.
[49] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rX1v:1911.1161,, 2019.
[50] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline mosel-bbasd policy optimization. arXiv preprint arXiv:2102.08363 2021.
[51] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint htt::201100721333 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see the experimental results in Section 5 and avenues for future work in the Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discussed the potential negative impacts in the Conclusion of the paper
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We will include the proofs for all theorems in the Appendix
(b) Did you include complete proofs of all theoretical results? [Yes] We will include the proof's for all theorems in the Appendix
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material a URL)? [Yes] We will include these results in the Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We will include this information in the Appendix
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We will include this information in the Appendix
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix D
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the suppiemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "vVVTzlmxhA9", "reviewer": "Reviewer_kSDN", "summary": "This paper proposes a modified GAN-based policy learning objective that modifies existing work. The use of a second (auxiliary) generator to capture additional objective is quite interesting. The algorithm outperforms existing offline RL methods in noisy and biased dataset that resemble robotics control and data relabeling. There are some problems with writing on the experiment section of the paper and additional experiments need to be run. If these problems can be fixed, it will be a good paper meeting the acceptance criteria.\n\nI'm happy to update my score if my questions are addressed and/or if I missed several key factors/findings of the paper.", "questions": "- Can Table 1 include more baselines, beyond CQL and AWR (or IQL)? Or is DASCO performing worse or on-par if you include other baselines?\n- Can the authors explain how to sample from $p_{mix}$?\n- Can authors explain a bit more on why does the biased dataset resemble data relabeling? In your case, you are adding bias/noise to state and action. However, normal transition relabeling is to relabel reward instead.", "limitations": "The paper lacks experimental evidence that convinces me this algorithm can solve a unique challenge that other algorithms can't (even though the introduction/conclusion makes me believe that there should be such use case).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The first half of the paper is very well written.\n- I quickly read through the theorem and the proof. The approach generally seems principled, but I'm unable to comment on how \"innovative\" this modification is and whether it is an incremental improvement from the previous GAN-based offline policy RL objective.\n- The biased and noisy datasets are designed with specific scenarios in mind. DASCO does outperform two other baselines with a large margin.\n\nWeakness\n- The second half of the paper is a bit harder to interpret. \n  - Starting Section 5 (Experiments), some details of the experimental setup gets lost. For example, baseline 10% BC is not explained neither in main text nor appendix (but please correct me if I missed it). \n  - Line 226-227 says \"Advantage Weighted Regression [31] (IQL)\", first of all, it should be AWR, not IQL (a different paper). The citation suggests AWR as well. So is the experiment done with IQL or AWR? \n- Lack of better use case\n  - Proposing new algorithms will always have its value -- we shouldn't force every new algorithm to be \"SOTA\" in order to get published. The idea discussed in this paper seems interesting enough that I don't think I'm too hang up on the fact that Table 2 and Table 3 shows close performance with other baselines. However, I am a believer that there should be some unique use cases that this algorithm can and will be used by others. Table 1 seems to be getting this point across, but 2 tasks / settings are very limited and it's only compared against 2 baselines, therefore not very trustworthy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uRw_Ls3w7i", "reviewer": "Reviewer_PvFC", "summary": "This paper proposes a GAN-based method to learn an offline policy. \n\n- The policy is trained as the generator of GAN to make the policy close to the supported distribution. Meanwhile, the policy is trained to maximize rewards as an ordinary RL algorithm. \n- To handle the potential conflicts in these two objectives, the authors use two separate distributions to learn the policy and the auxiliary generator, respectively. \n- The experiments on mixed datasets with sub-optimal data obtain good performance compared to several previous methods.", "questions": "1. Can you explain more about \"the policy does not need to match the entire data distribution, but only the slice of the data that leads to high long term performance\" in the abstract?\n\n2. What is the relationship between Theorem 4.1 and the proposed method? The data distribution weight seems hard to implement. What information can we obtained from the form of the data distribution weight?\n\n3. In Eq.8, why the optimal primary generator  $p^*_g=2P_D$ if $2P_D<1$? What is the \"next\" best in-support maxima in Eq. 9? Does the probabilities of $p^*_g(x_j)$ are sorted when you sum $j=0$ to $i$? Can you give an example to explain what distribution of $p^*_g(x_j)$ should be in simple cases?\n\n4. The ablation study shows dynamic weighting in Eq. 11 is important. It seems that the policy has be constrained to be close to the bebavior policy through adversarial training, why the dynamic weighting is still important especially in antmaze tasks?\n\n5. The relationship between this paper and uncertainty-based methods should be discussed, like EDAC (NeurIPS 21) and PBRL (ICLR 22).\n", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The idea that uses GAN to model the state-action distribution of offline dataset is novel. Compared to directly learn a behavior policy, the GAN-based method is more flexible and enables the network to learn complex distribution from a mixed dataset. \n- The policy constraints is applied implicitly by using the policy network as the generator of GAN. \n\nWeaknesses\n- Some conclusions and theoretical analysis need more explanation (see questions)\n- Several related works should be discussed. \n- Since the method applies adversarial training that is usually considered unstable, it is better to release the public code to ensure reproducibility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "X6hs0wcKEGd", "reviewer": "Reviewer_uYzh", "summary": "This paper proposed to resolve the conflict issue when incorporating a generative adversarial network (GAN) in an offline RL setting. The authors argue that optimizing the action-value function and the discriminator during policy improvement causes poor performance on either objective. The authors showed that using an auxiliary generator alleviates the conflict issue, and RL policy attains in-support optimization of the Q-value function. Additional importance weight on the Q-value further reduced the possibility of extrapolation errors. Numerical results show that the proposed method produced competitive performance in the standard benchmark tasks and the best performance in the noisy/biased datasets.\n\n", "questions": "1. GAN has a stability issue. Training of both generator and discriminator requires careful design (architecture, update period, and so on). I suspect that inserting an additional generator still causes the stability problem. Did the proposed method suffer from the stability issue? How did the authors handle this issue? (Importance weight helps alleviate extrapolation error regarding RL agent, not explicitly guaranteeing the stability of GAN). What are the update periods of D, G, and the auxiliary generator?\n\n2. To improve the quality of the paper, it is a good idea to show the visual difference of the coverage/distribution with and without the auxiliary generator to support the claimed in-support optimization.\n\n3. It looks like there is a gap converting $\\min_\\pi \\log(1-D(s,a))$ in Eq 2 and 13 to $\\max_\\pi \\log D(s,a)$ without any explanation. In fact, this is not supported by the claimed mathematical proofs. Did the authors also conducted experiment using $\\min_\\pi \\log(1-D(s,a))$ in Eq 13?\n\n4. Is there any theoretical insight of adopting importance weight in Eq 11?\n\n5. Why did the authors use only CQL and IQL for the baselines in Tables 1 and 2?\n\n6.  Did the authors tune key parameters in CQL and IQL for the noisy/biased dataset? (e.g., \\tau in IQL) If yes, what are the values?\n\n7. Why were the model-based offline RL methods not considered in Table 3?\n\n8. For clarity, does 'without (auxiliary generator)' in Table 4 use importance weight?\n", "limitations": "The authors did not address a potential societal impact of their work.\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n  1) The authors tried to explain the proposed method with mathematical justifications.\n  2) The authors provide a reasonable ablation study.\n\nCons\n  1) Several parts of the paper should be clarified. (Please see the below Questions.)\n  2) Additional experiments are required. (Please see the below Questions.) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KlSRhbj47KL", "reviewer": "Reviewer_nQaE", "summary": "This paper proposes a novel GAN-based approach to tackle the distributional shift issue in offline RL. The authors noticed that the conventional single generation GAN framework requires the generator to both maximize the expected return and match the dataset distribution, which is challenging and allows the discriminator dominate the generator. To address this problem, the authors proposed to add an auxiliary generator to aid the distributional matching. The main generator will be trained to both match the data distribution and maximize the expected return while the auxiliary generator focuses on learning the \"remainder\" of the data distribution that the main generator does not learn well. The authors did extensive analysis and experiments with good ablation studies to demonstrate the effectiveness of the proposed method. ", "questions": "Have the authors encountered challenges when training the dual generator GAN? Are any special tricks required to stabilize the GAN training?  \n", "limitations": "The authors didn't provide a discussion on the limitations and potential negative societal impact of their work. Please mention the offline evaluation challenge to allow the audience understand that the offline learned RL policy cannot be easily deployed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. The proposed dual-generator is well motivated and overall writing is easy to follow.\n2. The proposed method outperforms strong offline RL baselines based on distribution constraints.\n3. The ablation study provides a nice way to understand the effect of the auxiliary generator. \n\n### Weakness\n1. The experiments are performed on relatively simple navigation and locomotion tasks. It would be nice to see results on more complex manipulation tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Gvr-1Zt_s2", "reviewer": "Reviewer_Y2tj", "summary": "This paper proposes a new method for performing offline reinforcement learning in the face of low-quality data via modelling two separate generator networks - one capturing the policy and maximizing return, and the other auxiliary network capturing the adjustments needed to fool the discriminator network - and together with a value function weight, they empirically show that their method surpasses previous methods in the setting of noisy, biased data. ", "questions": "* What is the effect of adding only bias in the Antmaze setting?\n* What do the poor actions in the auxiliary network look like for the AntMaze without the policy network? I am surprised it performed so poorly given that the performance in other domains seem decent.\n* Are there settings for the bias/noisy data in the real world which are analogous where a similar biasing/noising effect plays out?", "limitations": "By far the greatest improvements lie in the setting of the bias, noisy data, but the setting is unfortunately quite artificial with hardcoded biases and noises and producing results in a setting which naturally induces biases/noise in the action would be more convincing. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Theoretical insight as to why prior work in modelling with GANs fail in Theorem 4.1\n+ Simple yet theoretically convincing modifications to GANs to enable the training to perform well \n+ Thorough ablations done with evidence on the boost in performance due to the auxiliary generator network and value function weighting\n- The experimental setting of the adding noise and bias to the AntMaze seems artificial, which is unfortunate because it underlies the primary result of the paper\n- Standard deviation reported only for a portion of the results in the Appendix\n- Some awkward phrasing made the paper hard to read at points", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept