Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
Authors: Harshit Sikchi
Qinqing Zheng
Amy Zhang
Scott Niekum
Venue: ICLR 2024 spotlight (2024)
Keywords: Robot Learning
Offline Imitation Learning
Offline Reinforcement Learning
Deep Reinforcement Learning
Primary area: reinforcement learning
TL;DR: A unification of RL and IL methods through the lens of duality that allows us to propose new methods for discriminator-free imitation learning and stable offline reinforcement learning.

Abstract: The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as *dual RL*, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method $f$-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and $f$-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks.

Published as a conference paper at ICLR 2024
DUAL RL: UNIFICATION AND NEW METHODS FOR REINFORCEMENT AND IMITATION LEARNING
Harshit Sikchi', Qinqing Zheng? 2, Amy Zhang 2,, Scott 2
1 The University of Texas at Austin, 2 FAIR, Meta Al 3 University of Massachusetts Amherst [hsikchi] @utexas.ee edu
ABSTRACT
The goal of reinforcement learning (RL) is find policy that maximizes the expected cumulative return. It has been shown that this objective can be represented optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several ttate-of-the-ar offline RL and offline imitation learning (IL) algorithms as instances of dual RI. approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline II., our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, propose new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL. in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project page (code and videos): hari-sikchi, wrrf--ikcci.gitub..ioom github.io/dual-li
INTRODUCTION
A number of deep Reinforcement Learning (RL) algorithms optimize a regularized policy learning objective using approximate dynamic programming (ADP) [Bertsekas and Tsitsiklis, 1995]. Popular off-policy temporal difference algorithms spanning both imitation learning [Kostrikov et al., 2018, Ni et al., 2021] and RL [Haarnoja et al., 2018, Janner al., 2019, Sikchi al., 2022b, Hafner et al., 2023] exemplify this class. As we discuss in Section 3, one way to develop a principled off-policy algorithm to ensure unbiased estimation of the on-policy policy gradient using off-policy data [Nachum and Dai, 2020]. Unfortunately, many classical off-policy algorithms do not guarantee this property, resulting in issues like training instability and over-estimation of the value function [Fu et al., 2019, Fujimoto al., 2018, Baird, 1995]. To obtain high learning performance, these algorithms require that most data be nearly on- on-policy, otherwise require special algorithmic treatments (c.g., importance sampling [Precup et al., 2001], layer normalization [Ball et al., 2023], prioritized sampling (Vecerik et al., 2017]) to avoid the aforementioned issues. Recently, there have been developments leading to new off-policy algorithms with improved performance for RL [Kumar et al., 2020, Garg et al., 1, Kostrikov et al., 2021] and IL [Zhu et al., 2020, Ma et al., 2022, Garg et al., 2021, Florence al., 2022]. These methods are derived via a variety of mathematical tools and attribute their success to different aspects. It remains an open question if we can inspect these algorithms under a unified framework understand their limitations. and subsequently propose better methods.
In this work, we consider a specific formulation for RL that writes the performance of a policy as a convex program with linear constraints [Manne, 1960]. This convex program can be converted into unconstrained forms using Lagrangian duality, which is more amenable for numerical optimization. We refer to the class of approaches that admit the dual formulations as Dual RL. Dual RL approaches naturally provide unbiased estimation of the on-policy policy gradient using off-policy data, in principled way. They avoid explicit importance sampling that leads to high variance and ensures training stability and convergence [Tsitsiklis and Van Roy, 1996]. Related approaches in this space have often been referred to as DICE (Distribution Correction Estimation) methods in previous literature [Nachum al., 2019, Kostrikov et al., 2019, Lee et al., 2021, Ma et al., 2022, Zhang et al.,

---

Published as a conference paper at ICLR 2024
Method
dual-Q/V
Gradient
Ohjective
Off-Policy Data
AlgaeDICE, GenDICE, COL
Q
semi
regularized RL
Atbirary
RL
OpEDiCE
full
regularized RL
Athimary
XQI. REPS, f-DVI. VIP. GoFAR Logistic Q-keaiing
V
semi
regularized RI.
Arbirmry
V
full
regularized RL.
Arbizary
QF
full
regularized RL.
x
1Qizam, IRC
Q
semi
Dr(o" I
Expert-only
OPOLO, OPIRI
Q
semi
Dailo" Io
Arbitary
SMODICE
V
full
Duile" Ie")
Athirary
DemoDICE, LobsDICE
V
full
Dellp" p²)) + aDailo" le")
Atbiuray
I'IL.
QV
full
De 1p
ReCOIL-Q
Q
full
Dy (omin I (
Arhirary
ReCOIL-V
V
full
Py (P
Arbimury
Table I: A number of recent works can be studied together under the unified umbrella of dual-RL. These methods are instantiations of dual-RL with choice of update strategy, objective, constraints, and their ability to handle off- ooflppllyy data. Bold names correspond the methods proposed in the paper and Italic names correspond to methods that aren't 't yet known to be dual-approaches.
2020]. We note that the linear programming formulation for the RL objective has been used and studied in Manne [1960], Denardo [1970], de Ghellinck and Eppen [1967], Borkar [1988], Malek al. [2014] and the general duality framework for regularized RL was first introduced in Nachum and Dai [2020].
Our first contribution is extend the work of Nachum and Dai [2020] and show that many recent algorithms in deep RL and IL can all be viewed as different instantiations of dual problems for regularized policy optimization. see Table 1 for the complete list. These algorithms have been motivated from a variety of perspectives and differing derivations. For example, XQL [Garg et al., 2023] focuses on introducing Gumbel regression into RI., CQL (Kumar et al., 2020] aims at learning pessimistic Q function, IQLearn [Garg et al., 2021] and OPOLO [Zhu et al., 2020] use the change of variables for IL, and IBC [Florence et al., 2022] uses a contrastive loss for imitation learning, but as we show all can actually be derived from the dual formulation.
Second, the dual unification in IL reveals an important shortcoming of prior methods that learn to imitate the expert by leveraging arbitrary off policy data. Prior work [Ma et al., 2022, Zhu et al., 2020, Kim et al., 2022b] imposes a coverage assumption (the suboptimal data covers the visitations of the expert data) and learn a density ratio between suboptimal and expert data via discriminator to use it for downstream learning. In an offline setting, with limited data and coverage, learning a density ratio between suboptimal and expert can be challenging, and the inaccuracies of the discriminator can compound in downstream RL, negatively affecting resulting policy performance. We show that by a simple modification to the dual formulation, we can get away from this limitation. In Section 5, we present ReCOIL, simple, theoretically principled, and discriminator free imitation learning method from arbitrary off-policy data. We empirically demonstrate the failure of previous IL methods based on the coverage assumption in number of MuJoCo environments and show substantial performance improvements of ReCOII. II in Section
Third, the presented unification also provides us with a useful tool to examine the limitation for recent offline RL method, XQL |Garg et al., 2023]. XQL's success was originally attributed to better modeling of Bellman errors using Gumbel regression. On the other hand, XQL also suffers from training instability, also caused by Gumbel regression. By situating the implicit policy improvement algorithms like XQL in the dual RL framework, in Section 6 implicit algorithms f-Dual V Learning (J-DVL), which successfully addresses the training instability issue. The empirical experiments the D4RL benchmarks establish the superior performance of f-DVL, see Section 7.
able to propose family of
2 RELATED WORK
Off-Policy Methods for IL Imitation learning has benefited greatly from using off-policy data to improve learning performance [Kostrikov et al., 2018, Agarwal et al., 2020, Zhu et al., 2020, Ni et al., 2021, Sikchi et al., 2022a]. Often, replacing the on-policy expectation common in most Inverse RL formulations [Ziebart et al., 2008 Swamy et al., 2021] by expectation under off-policy samples, which is unprincipled, has led to gains in sample efficiency [Kostrikov et al., 2018]. Previous works have proposed a solution in the dual RL space for principled off off-policy imitation but is based
'hhee methods different regularizer. More details in Appendix C.4.
2

---

Published as a conference paper at ICLR 2024
on restrictive coverage assumption [Ma et al., 2022, Zhu et al., 2020, Kim et al., 2022b] which requires estimating a density ratio using a discriminator and further limit themselves to matching a particular f f-divergence. In this work, we eliminate this assumption and allow for generalization to all f-divergences, presenting a principled off-policy diseriminator-fiee approach to imitation.
Off-Policy Methods for RL Off- Off-policy RL methods promise a way utilize data collected by arbitrary behavior policies to aid in learning an optimal policy and thus are advantageous over on-policy methods. This promise falls short, as previous off-policy algorithms are plagued with a number of issues such as overestimation of the value function, training instability, and various biases [Thrun and Schwartz, 1993, Fu et al., 2019, Fujimoto et al., 2018, Kumar et al., 2019]. A common cause for a number of these issues is distribution mismatch. As we shall discuss later, the RL objective requires on-policy samples but is often estimated by off-policy samples in practice. Prior works have proposed fixing the distribution mismatch by using importance weights [Precup, 2000], which can lead to high variance policy gradients or ignoring the distribution mismatch completely [Haarnoja et al., 2018, Fujimoto et al., 2018, Sun et al., 2022, Ji et al., 2023, Chen et al., 2021b]. Unfortunately, these approaches do not carry over well to the offline setting. For example, when deploying the policy online, the overestimation bias feedback, which is infeasible for offline RL.A number of solutions exist for controlling overestimation in prior work- forverfdieeeeeence regularization et al., 2019], support regularization (Singh et al., 2022], implicit maximization [Kostrikov et al., 2021), },, Gumbel regression [Garg et al., 2023] and learning a function that penalizes OOD actions [Kumar et al., 2020]. Dual-RL methods consider a regularized RL setting suitable for offline RL as well as fixing the distribution mismatch issue in principled way.
be corrected by the environment
the training distribution [Wu et al., 2019, Fujimoto
3 PRELIMINARIES
We consider an infinite horizon discounted Markov Decision Process denoted by the tuple M AS, p. r, do), where S is the state space, A is the action space, p is the transition probability function, r S x A - IR is the reward function, 7 E 1) is the discount factor, and do is the distribution of initial state sg. Let ((AA) denote the probability simplex supported on A. The goal of RL is to find policy
S - A(A) that maximizes the expected return: E. ( 7'r(st, ].)] where we use E,, to denote the expectation under the distribution induced by a ~ T( (1)3)), 8141 ~ p(.()e,at) at). We also define the discounted state-action visitation distribution d' d= a) (1 ())(( 2 P(s
s/T).. The unique stationary policy that induces a visitation d(s, a) given by T (als) = d(s, a)/ E. d(s. a). We will use do and d= denote the visitation distributions of the behavior policy of the offline dataset and the expert policy, respectively. f-divergences are denoted by D, and measure the distance between two distributions. For a more formal overview of the above concepts, refer to Appendix B.1.
Value Functions and Bellman Operators Let V": S - be the state value function of T. V" the expected return when starting from and following т: V=(s) E, E 7'r(s,, at)lsg s]. Similarly, let Q"
R be the state-action value function of 9, such that (* a) = E, - 7 ('r ) |0p = s, ag = =]] Let V* and Q" denote the value functions corresponding to an optimal policy n* Let T, be the Bellman operator with policy x and reward function r such that 7," (F. r(s,a) a) + pe.)) att (Q(s', aa)]. We also define the Bellman operator for the state value function (s, = r(s, + (p
S x A
3.1 REINFOR LEARNING VIA LAGRANGIAN DUALITY
RL optimizes the expected return of policy. We consider the linear programming formulation of the expected return [Manne, 1960], to which we can apply Lagrangian duality or Fenchel-Rockfellee duality to obtain corresponding constraint-free problems. We now review this framework, first introduced by Nachum and Dai [2020]'. Consider the regularized policy learning problem
max J(T] = (rr.s.))[(ss a)] - -aD,(d" (s, a) || (ssss)))
(1)
where Dy (d (s, a) || (s a)) is conservatism regularizer that encourages the visitation distribution of to stay close to some distribution dO, and is temperature parameter that balances the expected return and the conservatism. An interesting fact is that J(TT) can be rewritten as a convex problem that searches for a visitation distribution that satisfies the Bellman-flow constraints. We refer this
We Lagrangian duality instead of Penchel-Rockfelee duality for case of exposition.
3

---

Published as a conference paper at ICLR 2024
form as primal-Q:
primal-9 max J(r = max max [e(o,))((ss - ad,(d(s, d)
(2)
s.t ((((s,)) a) = (1 (s) (als) + > Ev d(s', (()((s) As Vse a A].
We can convert this to an unconstrained problem with dual variables Q(s. defined for 8, a S x A by applying Lagrangian duality and the convex conjugate, giving us the dual formulation: dual-Q max, ming(1 aattt) a)] + aB [f* ([T, Q(s,a)) (3) where f is the convex conjugate of f. In fact, one can note that Problem (2) is overconstrained the constraints already determine the unique solution d", rendering the inner maximization w.r.t unnecessary. Therefore,
can relax the constraints to obtain another problem with the same optimal solution 7* and d², which we call primal- - below:
pr primal-V
max Ed(s,a))r(( a)] - aDy(d(s, || (s) dz0
(4)
s.t EEe d(x,a) a) (1-7)do(s) + Lis d(s', , p(ssssssssss ,d), As S.
Similarly, we consider the Lagrangian dual of (4), with dual variables (s) defined for all S: dual-V min (1 [V(]))))_VV(())) (a.)))) [1 ((TV(s, -[())]
(5)
where f is variant of f defined as (z) = max(0, f ( f(ma modification is cope with the nonnegativity constraint d(s, 7 in primal-v. Note that in both cases for dual-g -Q and dual-V, v, the optimal solution is the same as their primal formulations due to strong convexity. See Appendix B. for a detailed review, connections between Fenchel and Lagrangian duality, and discussion of computing π from V* for the dual-V formulation.
Such
Remarks. The dual formulations have a few appealing properties. (a) They allow us transform constrained cetrriutiionmmacccin problems into unconstrained forms w.r.t previously logged data. (b) One can show that the gradient of dual-Q w.r.t æ, when Q optimized for the inner problem, is the on on-policy policy gradient computed by off-policy data [Nachum and Dai, 2020]. This property is key to relieving the instability or divergence issue many off-policy learning algorithms [Thrun and Schwartz, 1993, Fu et al., 2019, Fujimoto et al., 2018) 8).
4 A UNIFIED PERSPECTIVE ON RL AND IL THROUGH DUALITY
In this section, we discuss how a number of recent RL and IL algorithms can be cast as dual-RL methods. We restrict ourselves to demonstrating this equivalence and IL settings from Table whose shortcomings we study via the unified viewpoint. In later sections, present approaches for addressing these shortcomings in both RL and IL. For the interested reader, a complete discussion on Table 1, particularly how algorithms like implicit behavior cloning, CQL and OPOLO can be cast dual-RL., be found in Appendix C. We further discuss the extension of dual-RL formulation to the online setting in Appendix E. All proofs are deferred to the appendix.
subset of methods in offfine RL
4.1 DUAL FORMULATION FOR EXISTING IMITATION LEARNING ALGOIIHTM
We first consider the standard imitation learning setup where the agent is given set of expert demonstrations, i.e. sate-action trajectories, and does not have access to environment reward. We consider two possible offline II. settings - 1) only expert demonstrations are available and 2) we additionally have access to suboptimal transitions from the environment. Intuitively, these suboptimal transitions should aid in better matching the expert behavior.
Offline IL with Expert Data Only Imitation learning, occupancy matching [Ghasemipour et al., 2020] is a direct consequence of the regularized RL problem (Eq. 1) when the reward is set to be 0 uniformly across the state-action space and the regularization distribution and dº are set to be the expert visitation distribution d= The corresponding dua -Q under these conditions simplifies to:
dual-Q max, min@(1 - 7) - [[Q(s, [)]s + aB [f* ([T, Q(s, [)]] (6) Interestingly, this reduction directly leads us IQLearn [Garg al., 2021], which was derived using a change of variables in the form of an inverse backup operator.
Proposition 1. eQlearn Gaar et al., 2021] is an instance of dual 1 using the semi semi-gradient update rule with a (soft) Bellman operator, where r(s, a) = 0 E S, a A do = d=
For overview of semi-gradient vs full- full-gradient methods please refer to Appendix B.1.7
4

---

Published as a conference paper at ICLR 2024
b. Offline IL with Additional Suboptimal Data Unfortumately, the dual-RL formulations above offer no way to naturally incorporate additional suboptimal data ds. To remedy this, prior methods have relied on careful selection of the f-divergence and coverage assumption to craft an off-policy objective [Zhu et al., 2020, Hoshino et al., 2022, Ma et al., 2022, Kim et al., 2022b,a]. More precisely, under the coverage assumption that the suboptimal data visitation covers the expert visitation (d > 0 wherever de > 0) [Ma et al., 2022], and with the KL divergence, we obtain the following simplification for the imitation objective: (s,
log d(s,a)
E d(,a) log d(s,,)) + log d= 0) ds(s,o)
DKL(d(s,a) || ds(,))) B(sa-
(= de (s.a)
d(s
log d (s.
+KL(d((sa)) [/d0///d@ (s,
(e,a)) and correspondingly The final objective resembles primal-Q when r(s,a) - log (s,a)) we obtain the following dual-Q problem using Eq. 3:
dual-Q (al.) mitl (1 ())))))) [Q [Q(s, + (E E. (T Q(s.))(s (7) Q(s
where Tyimt denote Bellman operator under the psendo- reward function u (s. a) log [5(,,0] This objective also allows us cast IL method OPOLO [Zhu et al., 2020] in the dual-g framework (see Appendix C.3). The pseudo-reward is a logarithmic density ratio learned using a discriminator and is later used for policy learning by optimizing Equation 7. Density ratio learning is difficult in a limited data regime as well as when the expert and suboptimal data share low coverage, and errors in learned discriminators can cascade for RL training and deteriorate the performance of output policy. We show how simple modification to the imitation objective can allow us to relax the coverage assumption and propose discriminator-fret free IL. method that learns performant policies from arbitrary suboptimal data, see Section 5.
d (s.a)
4.2 DUAL FORMULATION FOR EXISTING REINFORCEMEN, LEARNING ALGORIHTM
Now that we have seen how IL can be understood as a special case of the full regularized RL objective, consider the full objective in Eq. (1). Regularized policy learning, in its various forms [Nachum et al., 2019, Wu et al., 2019], is a natural objective for offline RL algorithms, preventing the policy from incorrectly deviating outto--dittribution by regularizing against the offline data visitation. However, implicit policy improvement (XQL [Garg et al., 2023], IQL. [Kostrikov et al., 2021J). one of the most successful classes of offline RL methods which uses in-distribution samples to pessimistically estimate the greedy improvement to the Q-fuccioon, has evaded connections to regularized policy optimization. Proposition 2 shows, perhaps surprisingly, that XQL can be cast as a dual of regularized policy learning, concretely as a dual-V problem.
Proposition 2. XQL is an instance of dual-V under the semi- grai-graddent update rule, where the f-divergence is the reverse Dallback-Lieblee divergence, and dº is the offline visitation distribution.
The success of XQL was attributed to the property that Gumbel distribution better models the Bellman errors [Garg et al., 2023]. Despite its decent performance, XQL is prone to training instability (see e.g., Figure 2), since the Gumbel loss is an exponential function that can produce large gradients during training. Situating XQL in the dual-RL. framework allows us to propose a solution to the training instability problem, a new insight we discuss in Section 6.
consequence of unification in RL: Offline RL can be broadly categorized in three approaches: 1) regularized policy learning. 2) pessimistic value learning e.g. CQL [Kumar et al., 2020], ATAC [Cheng et al., 2022] and 3) implicit policy improvement algorithms (e.g. XQL). The latter two frameworks have seemingly been exceptions to the regularized policy learning formulation (e.g. Eq. (1)). In Proposition 4, we show that with an appropriate choice of f-divergence, CQL and ATAC can be cast as a dual-g problem. Overall, our results (Proposition and Proposition 2) are the first, to our knowledge, to bring together the latter two approaches, pessimistic value learning and implicit policy improvement as dual approaches to regularized policy learning.
RECOIL: IMITATION LEARNING FROM ARBITRARY EXPERIENCE
As demonstrated in Section 1.1, previous off- coficpllcy IL methods often rely on the coverage assumption and train discriminator between the demonstration and the offline data to obtain a pseudo- pimi We propose Rllaxed Coverage for Off Off-policy Imitation Learning (ReCOIL), an off-polic IL. algorithm that relaxes the coverage assumption and eliminates the need for the discriminator. To
5

---

Published as a conference paper at ICLR 2024
achieve this, we consider alternative way leverage suboptimal data for imitation: matching two mixture distributions d Bd(ss + (1 3)d (s,a) and d := 3d a) (1 -3)d (s, a), where 3 E (0, 1) is a fixed hyperparameter. We consider the following problem in primal-aee form: primal-Q
max - (dmmii((, || d (8.a))
d(s,a)
s.t Vs E S, A, (1-)))))))
=
+ +
d(s', a' d')p(s "asssss,a")<<(" (a)s(a (8)
This is valid imitation learning formulation [Ghasemipour et al., 2020] since the global maximum of the objective is attained at d d irrespective of the suboptimal data distribution dd. The primal formulation (Eq. 8) deters offline learning, as it requires sampling from d to estimate the f-divergence. We thus consider its dual formulation that allows us to derive an off off-policy objective that only requires samples from the offline data. We term this formulation and associated approach ReCOIL.
Theorem 1. (ReCOIL objective) The dual -2 problem the mixture distribution matching objective in Eq. 8 is given by:
max, ming 8(1 ([Bs.=[Q(s.a) a. (f (s) Q(s,a)]] (1 a/ ((x Q(s.a]] and recovers the same optimal policy n* as Eq. 8 since strong duality holds from Slater's conditions.
(9)
In other words, imitation learning can be solved by optimizing the unconstrained problem ReCOIL with arbitrary off-policy data, without the coverage assumption. Besides, as opposed to many previous algorithms, ReCOIL uses the Bellman operator To which does not need the pseudo-rewan making it discriminator-ffee Although the pseudo-reward is not needed for training, ReCOIL allows for recovering the reward function using the learned Q*, which corresponds to the intent of the expert. That is, r(s, a.) @*(s., a) Tox (Q* a)). Moreover, our method is generic to incorporate any f-divergence. We also present the dual form for ReCOIL in Appendix D but defer its investigation for future work.
iimit
A Bellman Consistent Energy- -ased Model (EBM) View for ReCOIL Instantiating ReCOIL with x² Divergence, we present simplified objective (complete derivation in Appendix D.3) to: max, ming stais
ai] (Qs,, 0.25 5. (s.0) [[0Q(s' Q(s.a)) (10)
Rellman convistincy
One that ReCoIL learns score function Q whose expected value is low over the suboptimal distribution but
Algorithm 1: ReCOIL (offline, x²)
high over the expert distribution, while ensuring that Q is Bellman consistent over the mixture. The Bellman consistency is crucial propagate the information of how to recover when the policy makes a mistake. The Q value can be interpreted as a score as is not representative of any expected return, and thus ReCOIL is an energy- based model with Bellman consistency. Figure 6 in the appendix illustrates this intuition.
Initialize Q+, Vo, and Tg. mixing ratio B, conservatism т,
temperature
2: DS = (s,a, s') be suboptimal dataset
DE = (s, a, s') be expert dataset.
for l 1.T iterations do
Train Qa using ming L(d):
6. Train Ve using mine [(0)
Practical Algorithm In Algorithm we consider three parameterized functions Qs(s,)) a), Va(s) and Tg. Furthermore, we rely on Pearson x2 divergence (Eq. 10)
Update Te via max. M(
end for
as it has been shown to lead to stable learning for the imitation setting [Garg et al., 2021]. Our practical algorithm uses a semi- update that results in minimizing the following loss for Qx:
L())= (( (Qas) a}] TT [Qa(., 0.25 R. =.. ((VV ()(s (11)
Naively maximizing Q over in Eq. 10 can result in the selection of out-of-distribution actions
in the offline setting. To prevent extrapolation error in the offline setting, we rely on an implicit maximizer [Garg al., 2023] that estimates the maximum over the Q-function conservatively with in-distribution samples.
I(0)= ( (ex][( a) - Vo(s))/T) + (Qo(s, a) Vo(s))/7)].
(12)
Finally, the policy is extracted via advantage-weipe
regression [Peters and Schaal, 2007]:
M(b}) max E, Ea (s.a) [exp(a(( (s a) Vo(s)) log (als))
(13)
6 -DVV BETTER IMPLICIT MAXIMIZERS FOR OFFLINE RL
Proposition 2 shows that XQL is particular dual-V -v problem where the Gumbel loss is the conjugate f corresponding to reverse KI. divergence. This insight allows us extend XQL by

---

Published as a conference paper at ICLR 2024
choosing different f-divergences, where the conjugate functions amenable to optimization. We further show that the proposed methods enjoy both improved performance and better training stability in Section 7.
Implicit policy improvement algorithms iterate two steps alternately: 1) regress Q(s, a) to + a(s.a)
(s') for transition a, s') and 2) estimate V(s) maX Q(s, a). The learned Q.
V functions can be used to extract a policy. As for the dual-V formulation, see Appendix B. 1.1.6. Step 1) is akin the policy evaluation step of generalized policy iteration (GPI), and step 2) acts like the policy improvement step without explicitly learning a policy *(s) arg max, Q(s, a). The crux is to conservatively estimate the maximum of Q in step 2.
Consider rewriting of dual-V V with the temperature parameter À and a chosen surrogate function f that extends the domain of f R. We discuss the need for surrogate function below.
min(1 - , [V(s)] + (oa) [fp (Q(s, a) V(s))
(14)
V
where ((s,a) a) denotes ooto-rrrdeeetrrfr variable of distribution D. Eq (14) can be considered as a special instance of the following problem: min(1 A)v + AEE D (x -e--v) v)],
+ 7 E. p(s' (s))) Let be random
(15)
where æ is analogous to Q and v analogous to V. As opposed handcrafted choices [Kostrikov et al., 2021, Garg et al., 2023], we show through Proposition 3 below that objective (15) naturally gives rise family of implicit maximizers that estimates sup=-D I as
Proposition 3. Let æ be a real-valued random variable such that Pr(z > x*) = O. Let Vy be the solution of Problem (15). It holds that UX, & Ux2- YO < \ < x2 1. Further, lima--1 UX x*.
Figure 5 provides an illustration for Proposition 3. We propose family of maximizers associated with different f-divergences and apply them to dual-V. We call the resulting methods f-DVL. (Dual-V Learning).
Practical Considerations and Algorithm A practical issue for optimizing Eq 14 is that fe not well defined over the entire domain of R. To remedy this, we consider an extension of f on R that leads to the surrogate function denoted by f for (1) Total
Algorithm 2: f-DVL (Under Stochastic Dynamics)
I: Initialize Qs. Vo, Tei temperature a, weight X
Let D = (s, a, r. )') be offline dataset
Variation: fzx = 2/1z 1l. (1) = max(y, (y. 0). (2) Pearson x² divergence: f(x) (a 1)2, fg(y)=
3: for 1 = 1..T iterations do
Train Qs by minimizing:
max(3y² + y. 0). We defer the derivation of these surrogates to Appendix F.3. Recall that XQL. uses the implicit maximizer associated with reverse KI. divergence, where f is exponential. Compared with XQL, our f& functions are low-order polynomials and are thus stable for optimization. Algorithm 2 details the steps for FDVVL.
Fea.'= ((Q.lp(( - (r(s,a) +VV((9))).
5: Train Vo by minimizing Eq 14
with surrogate ftt
6: Update nh by maximizing: En,o-peei@Rste. (100)) (ela))
2: end for
7 EXPERIMENTS
Our experiments aim to answer the following four questions. IL: 1) How does ReCOIL perform and compare with previous offline IL methods? 2) Can ReCOIL accurately estimate the policy visitation distribution d' and the reward function/intent of the expert? RL: 3) How does fDVL perform and compare with previous offline RL methods? 4) Is the training of f-DVL more stable than XQL?
In order circumvent the intricacies associated with exploration and direct our attention towards the intrinsic nature of dual RL formulation, we focus on the offline setting in this section, although the approaches can also be applied to online settings. We consider the locomotion and manipulation tasks from the D4RL benchmark [Fu et al., 2020]. and report the results in Section 7.1 and 7.2, respectively. For each algorithm, we train instances with different seeds and report their average return and standard derivation. Complete experiment details can be found in Appendix F.
7.1 OFFLINE IL
Benchmark Comparisons For every task, our agent is given expert demonstration and a set of suboptimal transitions, both extracted from the D4RL datasets. We follow the construction
7

---

Published as a conference paper at ICLR 2024
SMOOUTT
12j Perfect Coverage
DicrIE Dicrfbuttonn Gap
Export btatc
Vittoriin Steottuttion
wAra: Lier Radior
00L
0.108
0:154
0.228
0.128
0.200
Truts
0.400
loteen Diat busion
Kerations
Expert trajectory Learned reward function Learned function
(b)
Figure I: (a) [Left] shows an MDP that starts at the leftmost state and transitions to one of the five absorbing
states on the right. Under the given expert and replay/offline visitation we study if prespecified policy's s visitation can be inferred whose ground truth visitation is known [Right] shows MSE error plots with policy's ground truth visitation where ReCOIL perfectly infers d" whereas method that only relies on expert data or the replay data with the coverage assumption fails. Results averaged over 100 seeds. More details in Appendix G.3 (b) Recovered R and V* on simple grid-world environment by ReCOIL.
of suboptimal dataset in SMODICE [Ma et al., 2022]. For locomotion tasks, the suboptimal dataset consists of 1 million transitions of the random or medium D4RL datasets and 200 expert demonstrations, which we label as 'random+xxpert' and 'medium+cxpert', respectively. We also consider suboptimal datasets mixed with only 30 expert demonstrations, which called random+ few-expert and medi www-ffe-eepeeeeee to simulate a more difficult setting, Similarly, we construct datasets for the manipulation tasks. More details in Appendix F.2.
Suboptimal Datase:
Env
RCE
ORIL
SMODICE
BC (xaly expert data)
BC (full dassett)
10 Leamn (oflinee)
ReCOIL
Expert
randome
halftheetah
hupper
33.61
64.19-11.00 11.06
73.93=11.00 1.00
30.49-3.55
101661-7.69
$0.16-7.30
4.52:1.42
2.2-001
2.25-0.00
5.66+4.83
1.35 +2.19
7.8347.99
105, 3.23
$0.20-6.61
111.33
38.83
expert
walker24
20:90-26 26, 80
2,86+3.39
105.86-3.49 3.47
0186-0661
0.1-005
0.57++0.09
162.16-7.119
106.92
sui
981 14.05
1266711266
126.78
51715.43
30.66:1.35
42 23.230005
126.74:4463
rundom+
halftheetuh
hupper
25.31-18.97
42.04-13.76
2.84-5.52
2.28-0062
60.11-18.22
4.84+3.833
0.3-0035
2,34-0001
3.+00.44
1.14
97.85 17.89
111.33
fos-espet
50
15.81
25:41
3911:303
25.36:2.66
130.75
medimm
halfchortah
65. 06.14:111 13.82
58.71 34,06
61.08 7661
54.56-0.38
59 08
39.74+3..22
16.09:
-1.99-0222
42.452 0.42
3.225-3.71
25 25.67:30033
92 24.00
58.5 16.73
$1.15:2.44
2.84
88.81
capari
walker2a
96.24 14,04
7.1977.00
2.62-0533
1.436:182
12.7623622
59 50.374-30.14
105. 54: 1.81
106.92 92
uni
$6.14: 56.14:38.99
102.74-663
7.86
95.47:10.37
37. 37.171:41155
120.36:7.60
130.75
modhum
halfchectah
hupper
66.15:33.16
61.14:18 18.31
15:40:15115
43.24:0075
77.61:706
36.45::112
-1550006
1.3
56.8725.11
00.21-0006
11:05::00550 59
26.27:200334
50,01 10.36
75.96-4.5
111.38
88.83
few-espet
walker2a
55.28:34.90
66176676
6:00-6.99
2.02-0.22
70.42:2.86
73.5042.55
91.25-177 17.63
106.92
ani
67.95236.7
81.51136618
51 518588618
10451100
61652667
38 121.9035
110.381
10.96
190055
peu
19.00:11.40
100000.0
0.36::0.11
11.0
2.
95.444445
clooedrespent
fommer
dooe
403100001
0:2:0005
0221000
96.77-17.00
100.44
202010001
00172004
14.60
17812591
-3.561222
-2.0022
1385510000
90.6.125
14 16229128.22
205722...90
106.42
humunecapet
femmer
00,521005
5002564
10.000.011
18820.70
-0.001 0.11
40.0220.99
00050005
01820.14 14
1057111 1:22
1226112485
$3221138 38
5.627.79
125,19:3.29
104.7019.55
125.71
101 94
relocate
0.2-0.10
0.29H0.01
0,16:0.04
0.13-0111
$1.19-7.35
0.04-0222
91.95 2.89
118.39
puninlerager
kinchen
6.67529.4
0.0010000
19.162 1.17
2,5.50
45.51187
0.00000
68.0.5.70
25.0
misadecspert
kischen
1.65
0,00:0.00
42:5:2004
222388
42.1-1.12
0.060.0
$2
73.0
Table 2: The normalized return obtained by different offline II. methods trained on the D4RL, suboptimal datasets with 1 expert trajectory. Methods with avg. perf within the std-dev of the top performing method is highlighted.
We compare ReCOIL against recent offline IL methods RCE [Eysenbach al., 2021]. IQLearn [Garg et al.. 2021], SMODICE [Ma et al., 2022], ORIL [Zolna et al., 2020] and behavior cloning. We do not compare to DEMODICE [Kim et al., 2022b] and ValueDICE [Kostrikov et al., 2019] as SMODICE was shown to outperform DEMODICE in Ma et al. [2022] and IQLearn was shown to outperform ValueDICE in [Garg et al., 2021] on the same environments. Both SMODICE and ORIL require learning a discriminator, and SMODICE relies on the coverage assumption. RCE also uses a recursive discriminator to test the proximity of the policy visitations to successful examples. In contrast, ReCOIL is discriminator-fin free and does not need this coverage assumption. Table 2 shows that ReCOIL strongly outperforms the baselines in most environments. SMODICE exhibits poor performance in cases when the combined offline dataset has few expert samples feenxeet-ewweepeet) or where the discriminator can easily overfit (kigh-dimensional environments like dextrous manipulation).
Estimation of the Policy Visitation Distribution and Reward Recovery Correctly estimating a given policy's visitation distribution d= is key to testing its closeness to the expert visitation. d" can be computed via Eq (49) (appendix). Figure la and Figure 11 show that ReCOIL can estimate d' more accurately than SMODICE [Ma et al., 2022] which relies on coverage assumption and IQLearn [Garg et al., 2021] which only utilizes expert data. This empirically validates our hypothesis that a learned discriminator with low coverage can lead to poor performance of the downstream policy. Further, Figure lb shows the reward function recovered by ReCOIL for a simple grid-world task. For Hopper and Walker, we respectively observe a Pearson correlation of 0.98 and 0.92 between the recovered reward with the ground truth. See more details in Appendix G.3 and 1.
8

---

Published as a conference paper at ICLR 2024
Ditasel
10SRC
TDN+BC
CQI.
IQI.
XQL40)
(-DYL (-DVL((TV)
hullchhetaa m2
12.6
12.5
12.6
48.3
44.0
47.4
41.1
47.7
47.5
hopper-mediurr-y
52.9
56.9
67.6
59.3
58.5
66.3
68.5
63.0
64.1
walker224mssiummm2
75.3
75.0
74.0
83.7
72.5
78.3
N1.4
81.5
halffcheetub melimmeelll
36.6
40.6
36.6
44.6
45.5
44.2
44.1
42.9
44.7
wwmer-meliurt--ppla
18.1
75.9
82.7
60.9
95.0
94.7
95.1
901.7
98.0
Solker/dd-meilum--e repluun-----2222222
2600
62.5
77.2
52.1
hulfcherttbb mediim-exppprii-
55.2
92,9
86.8
90.7
91.6
86.7
90.8
89.3
91.2
hopper medium experti<2
52.5
110.9
107.6
98.0
105,4
91.5
94.0
105.8
93.3
medium malker26-medumm
expurt-v2
107.5
109,0
108.1
110.1
108.8
109.6
110.1
110.1
109.6
mntrane-umzze-0
54.6
62.8
59.2
78.6
74.0
87.5
47.7
83.7
87.7
antmaaz--mmnn
45.6
50.2
53.0
71.4
84.0
62.2
51.7
50.4
48.4
antmaze- paaeess--iiiiipp--eeee-
0.0
10.6
61.2
71.2
31.2
$6.7
71,0
mntmaze-mllium -iiee------ee-t)
0.0
antmazz large-pay--00
0.0
0.0
0.0
0.2
15.8
39.6
10.7
36.0
41.7
antmaae- large divensevel
0.0
6.0
0.0
0.0
14.9
47.5
31.28
44.5
39.3
kitchen-compllte-vi
65.0
62.5
kilchen Silchen-prtial--
38.0
49.8
46.3
48.6
53.8
70.0
kitchen-mixee---0
51.5
$1.0
51.0
40.4
53.75
$2.5
Table 3: The normalized return of offline RL methods on D4RL tasks. XQL(() denotes the results obtained
under the standard evaluation protocol. Results aggregated over 7 seeds. Highlighted results are within one performance point of the best- perf-oenfo algorithm.
7.2 OFFLINE RL.
Benchmark Comparison Table 3 shows that J-DVL outperforms XQL and other prior offline RL. methods [Chen al., 2021a, Kumar et al., 2019, 2020, Kostrikov et al., 2021, Fujimoto and Gu, 2021] on broad range of continuous control tasks. We note an inconsistency between our reproduced XQL results and the results reported in the original paper: their results were reported by taking the best average return during training opposed to the standard practice of taking the average of the last iterate performance across different seeds at 1 million gradient steps. Such inconsistency can be validated by comparing their training plots and reported results (Fig 11 and Table in Garg et al. [2023]). XQL(r) shows the results for XQL under the standard evaluation protocol.
antmaze modium diverso-vO
antmaze aage play
Paestons
C4
XOL
d12OVL
TYow
Figure 2: XQL training diverges due to the numerical instability of its loss function. f-DVI. fixes this problem by using more well-behaved f-divergences.
Training Stability As pointed out by the authors, the exponential loss function of XQL causes numerical instabilities during optimization. As discussed in Section 6, this is a by- -yoouuctt of reverse KL divergence. Fig. 2 confirms that this is fixed by f-DVL by using other f-divergences with more stable loss functions. Additionally, Fig. 13 demonstrates that f-DVL is competitive to XQL and SAC in the online setting as well. See Appendix for additional experimental details.
7.3 ADDITIONAL EXPERIMENTS
We conduct additional experiments in Appendix G. We further demonstrate a) when incorporating off-policy data in online training, traditional ADP-based methods can suffer from the over-estimation of value functions, and the performance gain is limited, whereas dual-RL methods can leverage the same data to achieve better performance (Figure 3 and Appendix G.I); b) the reward functions learned by ReCoIL are of high quality (Appendix G. G.);;; c) the hyperparameter ablation for DDVL (Appendix G.8) and qualitative results for ReCOIL (Appendix G.4).
Hammer-vo vū (Expert) [4:20] pan.
Hammer-v0 (Expert)
Bucrage Return
SAC (ro additional d2ts)
SAG + polloy Data
SACID
dual RL (igoeDICE)
Figure 3: Augmenting SAC with expert data at the start of training destabilizes value function learning (r), but dual- RL approaches can make effective use of the additional data to learn performant policy (I).
8 CONCLUSION
Our work unifies a significant number of recent developments in RL and IL. The insights gleaned from the unification allow us to identify key gaps and subsequently improve performance and training stability in imitation learning and reinforcement learning. Leveraging this unification, we propose 1) family of stable offline RL methods f-DVL relying on implicit value function maximization, 2) ReCOIL, a general off-policy IL method to learn from arbitrary suboptimal data while being discriminator-fiee We show that f-DVI and ReCOIL both outperform previous methods in online/offline RL and offline IL domains, respectively. We demonstrate that Dual-RL algorithms have great potential for developing performant algorithms and warrant further study. We direct interested readers to read our Appendix containing more insights into Dual-RL and potential directions for improvements and new algorithms.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Ahmed Touati, Ben Eysenbach, Siddhant Agarwal, and ICLR reviewers for valuable feedback this work. This work has taken place in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University Massachusetts Amherst and Machine Intelligence through Recision-making and Interaction (MIDI) Lab at The University of Texas at Austin. SCALAR research is supported part by the NSF (IS-2323384) AFOSR (FA9550-20-1-0077), and ARO (78372-CS, I991NN-1--)-0333), and the Center for AI Safety (CAIS). This research was also sponsored by the Army Research Office under Cooperative Agreement Number N911NF-19-2-032 HS and AZ are funded in part by sponsored research agreement with Cisco Systems Inc. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
A. Agarwal, N. Jiang, S. M. Kakade, and W. Sun. Reinforcement learning: Theory and algorithms.
CS Dept., UW Seattle, Seattle, WA, USA. Tech. Rep, pages 10-4, 2019. 18
S. Agarwal, H. Sikchi, C. Gulino. E. Wilkinson, and S. Gautam. Imitative planning using conditional normalizing flow. arXiv preprini arXiv:2007 2020. 2
F. Al-Hafez, D. Tateo, O. Arenz, G. Zhao, and J. Peters. Ls-iq; Implicit reward regularization for inverse reinforcement learning. arXiv preprint arXiv:2303.00599, 2023, 28, 35, 36
Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.
P. J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data. arXiv preprint arXiv:2302.02948, 2023.
J. Bas-Serrano, S. Curi, A. Krause, and G. Neu. Logistic q-learning. In International Conference on Artificial Intelligence and Statistics, pages 3610-3618. PMLR, 2021. 30
D. Bertsekas and N. Tsitsiklis. Neuro-dynamic programming: overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume 1, pages 560-564. IEEE, 1995.
V. S. Borkar. A convex analytic approach to markov decision processes. Probability Theory and Related Fields, 78(4))583-602, 1988.
L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021a. 9, 17
X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021b.
C.-A. Cheng, T. Xie, N. Jiang, and A. Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878.
PMLR, 2022. 24, 25
B. Dai, He, Pan, B. Boots, L. Song. Learning from conditional distributions via dual embeddings. In Artificial Intelligence and Statistics, pages 1458-1461 PMLR, 2017. 20
G. T. de Ghellinck and G. D. Eppen. Linear programming solutions for separable markovian decision problems. Management Science, 13(5):371-394, 1967.
E. V. Denardo. On linear programming in markov decision problem. Management Science, 16(5): 281-288, 1970.
Emmons, B. Eysenbach, Kostrikov, and S. Levine. Rvs: What essential for offline via supervised learning? arXiv preprint arXiv:2112. 10751, 2021. 17
B. Eysenbach, S. Levine, and R. R. Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Processing Systems, 34:
11541-11552, 2021. 37
Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, Downs, A. Wong, Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pages 158-168. PMLR, 2022. 1, 2, 28, 29
Fu, A. Kumar, M. Soh, and S. Levine. Diagnosing bottlenecks deep q-lenrnnn algorithms. In International Conference Machine Learning, pages 2021-2030. PMLR, 2019. 1,
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. 2020. 7, 36
S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems,
2021. 9,37

---

Published as a conference paper at ICLR 2024
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in octor-critic methods. In Interaational conference on machine learning, pages 1587-1596. PMLR, 2018. 1,3, 4, 27
S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without exploration. In International conference machine learning, pages 2052-2062. PMLR, 2019. 3
D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-4000 4039, 2021. 1, 2, 4, 6, 8, 24, 27, 28, 35, 37
D. Garg, Hejna, M. Geist, and S. Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint wwXiv:2301.02332. 2023. 2, 3, 5, 6, 7, 9, 25, 26, 35, 36, 38, 39
S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020. 4. 6
A. Gleave, M. Dennis, S. Legg. S. Russell, and Leike. Quantifying differences in reward functions. arXiv preprint arXiv:2006. /3900, 2020. 45
Z. Goldfeld. Ece 5630: Information theory for data transmission, security and machine learning, lecture Duality for f-divergences. 27,
T. Haarnoja, H. Tang, P. Abbeel, and Levine. Reinforcement learning with deep energy-based policies. In International conference muchine learning, pages 1352-1361. PMLR, 2017. 19
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018. 3, 3,,9, 40
D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.041000 2023.
J. Ho and Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565 4573, 2016. 37
H. Hoshino, K. Ota, A. Kanezaki, and R. Yokota. Opirl: Sample efficient off off-policy inverse reinforcement learning via distribution matching. In 2022 International Conference on Robotics and Automation (ICRA), pages 448-454. IEEE, 2022. 3,
M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
T. Ji, Y. Luo, F. Sun, X. Zhan, J. Zhang, and H. Xu. Seizing serendipity: Exploiting the value of past success in off-policy actor-critic. arXiv preprint 1XXv:23006028655 2023.
G.-H. Kim, Lee, Y. Jang, H. Yang, and K.-E. Kim. Lobsdice: offline imitation learning from observation via stationary distribution correction estimation. arXiv preprint arX1v:2202.13536, 2022a.
G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022b. 2, 2, 3, 37
Kostrikov, K. K. Agrawal, D. Dwibedi, Levine, and Tompson. Discriminator-actor--cnn Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02925. 2018. 1,2
I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv: 1912.0502,, 2019. 8,
I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arxiv: 16.06169, 2021. 1,3,5 7,9, 35, 36,36,39
A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019. 3,9
12

---

Published as a conference paper at ICLR 2024
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191, 2020. 1, 3,5,9 24, 27
J. Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pages 6120-6130. PMLR, 2021. 1,27
J. Lei Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. ArXiv e-prints, pages arXiv-1607, 2016. 36, 39
S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643: 2020. 32
Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433, 2022. 1, 2, 3, 5, 8, 22, 29, 30, 36, 37
A. Malek, Y. Abbasi- Yadkori, and P. Bartlett. Linear programming for large-scale markov decision problems. In International Conference on Machine Learning, pages 496-504. PMLR, 2014.
A. Manne. Linear programming and sequential decisions. Management Science, 6(3):259-267, 1960. 1, 2, 3, 18
Mehta and S. Meyn. Q-learning and pontryagin's minimum principle. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, pages 3598-3605. IEEE, 2009. 30
O. Nachum and B. Dai. Reinforcement learning via fenchel-rockafellaa duality. arXiv preprint arXiv:2001.01866. 2020. 1, 2, 3, 4, 17, 18, 19, 20, 22, 41
O. Nachum, B. Dai, I. Kostrikov, Y. Chow, L. Li, and Schuurmans, Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019. 5, 27, 40
A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093:3 2020. 27, 40
M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning, arXiv preprint arXiv:2303.05479 2023. 35
T. Ni, H. Sikchi, Y. Wang, T. Gupta, I.. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021. 1, 2,2
J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007. 6
A. Picard- Weinde-WWeelllae and B. Guedj. On change of measure inequalities for f-divergences. arXiv preprint arXiv:2202.05556, 2022a. 27
A. Picard-Weibel and B. Guedj. On change of measure inequalities for f -divergences. 2022b. 37
D. Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000. 3
D. Precup, R. S. Sutton, and S. Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pages 417 417-424, 2001.
M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. 18
A. Rényi. On measures of entropy and information. In Proceedings the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics, pages 547-561. University of Califoria Press, 1961. 17
13

---

Published as a conference paper at ICLR 2024
J. Schmidhuber. Reinforcement learning upside down: Don't predict rewards just map them to actions. arXiv preprint arXiv:1912.02877 2019. 17
J. Schulman, S. Levine, Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015. 17
Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, arXiv preprint arXiv; 2017, 17
H. Sikchi, A. Saran, W. Goo, and S. Niekum. A ranking game for imitation learning, arXiv preprint arXiv:2202.0348// 03 2022a. 2636
H. Sikchi, W. Zhou, and Held. Learning off-policy with online planning. Conference on Robot Learning, pages 1622-1633. PMLR, 2022b.
A. Singh, A. Kumar, Q. Vuong, Y. Chebotar, and S. Levine. Offline rl with realistic datasets:
Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052. 2022.
S. P. Singh and R. C. Yee. An upper bound on the loss from approximate optimal-value functions. Machine Learning, 16:227-233 1994. 34
H. Sun, L. Han, R. Yang, X. Ma, J. Guo, and B. Zhou. Optimistic curiosity exploration and conservative exploitation with linear reward shaping. arXiv preprint arXiv:2209.07288 2022. 3
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018. 23, 26
G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021. 2
S. Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, volume 255, page 263. Hillsdale, NJ, 1993. 3,4
E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012. 36
J. Tsitsiklis and B. Van Roy. An analysis temporal difference learning with function approximation. Rep.
demporal-differecce Lab. Inf. Decis. Syst. Massachusetts Inst. Technol. Tech. Rep, 1996.
I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, J. Jiao, et al. Jump-start reinforcement learning. arXiv preprint urXiv:2204.02372. 2022. 40
M. Vecerik, Hester, Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and
M. Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems 2017.
with sparse rewards. arXiv preprint 1707
L. Viano, A. Kamoutsi, G. Neu, I. Krawezuk, and V. Cevher. Proximal point imitation learning. arXiv preprint arXiv:2209. 10968, 2022. 30
Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rXiv:1911.11386 2019. 5,
R. Zhang, B. Dai, L. Li, and D. Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09071 2020.
Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In international conference on machine learning, pages 27042-27059, PMLR, 2022. 17
Z. Zhu, K. Lin, B. Dai, and J. Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33: 1402--1241 2020. 1, 2, 3, 5,
B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1493 1438. Chicago, II., USA, 2008.
14

---

Published as a conference paper at ICLR 2024
K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang, Y. Aytar, M. Denil, N. de Freitas, and S. Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13885, 2020. 8, 37
15

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Limitations and Negative Societal Impacts
17
B Dual Reinforcement Learning
17
B.1
A Review of Dual-RL
17
B.1.1
Convex conjugates and f-divergence
17
B.1.2
An Overview of Reinforcement Learning via Lagrangian Duality
B.1.3
Deriving dual-Q
B.1.4
Deriving dual-V
21
B.1.5
Discussion Dual Formulations
22
B.1.6
How to recover the optimal policy in dual-V?
23
B.1.7
Semi-gradient and Full-gradient.
23
unified perspective on RL and algorithms through duality
24
C.1
Dual Connections Reinforcement Learning..
24
C.1.1 f-DVL: A family of implicit policy improvement algorithms for RL
26
C.1.2 Connections of CQL to AlgaeDICE and XQL to OptiDICE
27
C.2 Dual Connections to Imitation Learning
27
C.2.1 Offline imitation learning with expert data only
27
Off-policy imitation learning (under coverage assumption)
29
C.4 Logistic Q-learning and P'IL as dual-QV methods
D ReCOIL: Off-policy imitation learning without the coverage assumption D.1 ReCOIL-V.
30
32
D.2 Suboptimality Bound for ReCO1 L--
32
D.2.1 Approximation Error of the Imitation Learning Objective
33
D.2.2 Performance Bound Learned Policy
34
D.3 ReCOIL. with x" divergence
35
E Taking dual-RI. from offline to online setting
35
F
Implementation and Experiment Details
36
Offline ReCOIL algorithm and implementation details
36
Numerical Stability:
36
F2 Offline Imitation Learning Experiments
36
E3
Calculation of f for f-DVL under practical considerations
37
x divergence
37
Total Variation
37
Practical Choice of C
38
E4
Online and Offline RL: f-DV Algorithm and imp eemelmttttaion details
38
Rewriting of dual-V using temperature parameter À instead o::
38
Online RI. Experiments with f-DVI.
39
Compute
39
G Additional Experimental Results
39
G.I Why Dual-RL Methods are Better Alternative to Traditional Off-Policy Algorithms
39
G.2 Training Curves for ReCOT on MuloCo tasks
40
G.3 Does ReCOIL Allow for Better Estimation of Agent Visitation Distribution?
40
G.4 ReCOIL: Qualitative Comparison with a Baseline
43
G.5 Evaluation of f-DVI. for Online RI.
43
16

---

Published as a conference paper at ICLR 2024
G.6 Training Curves for /-DVL MuJoCo Tasks (Offline)
43
G.7 f-DVL: Complete Offline RL Results
43
G.8 Sensitivity of f-DVI. (offline) with varying MuloCo tasks
44
G.9 Sensitivity of f-DVL (online) with varying À on MuloCo tasks
44
G.10 Recovering Reward functions from ReCOIL
44
A LIMITATIONS AND NEGATIVE SOCIETAL IMPACTS
Notably, none of the supervised learning approaches, like Upside-Down RL /Schmidhuber, 2019], DT [Chen et al., 2021a, Zheng et al., 2022] and RvS [Emmons et al., 2021], can be cast as instances of this framework. The main reason is that there is no return optimization term in the supervised learning loss for those approaches. Second, our framework is constrained to the regularized RL setting where the regularization coefficient a >> O (see Eq. ). Finally, our framework considers a static regularization distribution dº, and we believe that regularization with dynamically changing distributions, like those in on- on-policy methods (PPO [Schulman et al., 2017], TRPO [Schulman et al., 2015], etc) requires a more careful theoretical treatment. Another limitation of the paper is the assumption that the expert demonstrations used in the imitation learning process are always of high quality and provide the desired behavior. In practice, obtaining high-quality demonstrations can be challenging, especially in complex environments where the behavior of the expert is not always clear. The performance the proposed approach could be limited in cases where the expert demonstrations are of poor quality where the behavior of the expert does not correspond to the desired behavior.
Limitations:
Negative Societal Impacts: As machine learning algorithms continue to grow in sophistication, it is important to consider the potential risks and harms associated with their use. One such area of concern imitation learning, which involves training model imitate a desired behavior by providing with examples of that behavior. However, this approach can be problematic if the demonstration data includes harmful behaviors, whether intentional or not. Even in cases where the demonstration data is of high quality and desirable behavior is learned, the algorithm may still fall short of providing sufficient guarantees of performance. In high-stakes domains, the use of such algorithms without appropriate safety checks on learned behaviors could lead to serious consequences. As such, it is crucial carefully consider the potential risks and benefits of imitation learning, and to develop strategies for ensuring safe and effective use of these algorithms in real-world application
B DUAL REINFORCEMENT LEARNING
B.1 A REVIEW OF DUAL-RL
In this section, we aim to give a self-contained review for Dual Reinforcement Learning. For a more thorough read, refer to [Nachum and Dai, 2020]. Our proofs will take different approach than [Nachum and Dai, 2020] which we believe leads to easier exposition and understanding.
B. CONVEX CONJUGATES AND f- FDDIVERGENCE
We first review the basics of duality in reinforcement learning. Let f R4 E be convex function. The convex conjugate f* R. R of f is defined by:
f*yy) sup Izy [zy f())].
(16)
The convex conjugates have the important property that f is also convex and the convex conjugate of f* retrieves back the original function We also note important relation regarding f and f": (f*) -f where the notation denotes first derivative.
Going forward, we would be dealing extensively with f-divergences. Informally, f-divergences [Rényi, 1961] are measure of distance between probability distributions. Here's a more formal definition:
Let P and Q be two probability distributions over space Z such that P is absolutely continuous
with respect to Q S. For a function : R that is a convex lower semi-continuous and /(1) = 0.
'et denote the random variahle. For any measurable set 7 Z, Q(2 Z) 0 implics P(z Z) 0.
17

---

Published as a conference paper at ICLR 2024
Divergence Name
Generator f(x)
Conjugate (v)
Reverse KL
x loggæ0
e(v-1)
Squared Hellinger
(Vr - 1)2
Pearson x²
(x - 1)2
y + 4
Total Variation
1 1|
y y y E 1- 2: E otherwise oo
Jensen-Shannon
(x + 1) log( + + log
log (2 - eτ)
Table 4: List of common f f-divergences.
the f-divergence of from Q is
P(2)
Dy(P = EE~ f
(17)
Q(2)
Table lists some common f-divergences with their generator functions f and the conjugate functions
f*
B.1.2 AN OVERVIEW REINFORCEMENT LEARNING VIA LAGRANGIAN DUALITY
In this section, we give a more detailed review than we are able to in the main text due to space constraints. We consider RL problems with their average return considered in the form of convex program with linear constraints [Manne, 1960], to which we apply Lagrangian duality to obtain corresponding constraint-free problems. This framework was first introduced in the work of Nachum and Dai [2020], which obtains the same formulations as ours via Benchel-Rockfelee duality. Here we use Lagrangian duality for its simplicity and popularity.
Consider the following regularized policy learning problem
max J(x) = E.t (s..) (s.a)))) [r(s, a)] - adD(("(ssaa) || dl
(18)
where D, (d
(s. a) || (s, a)) is conservatism regularizer that encourages the visitation distribution of to stay close to some distribution d,, and is temperature parameter that balances the expected return and the conservatism.
An interesting fact is that J(z) can be rewritten convex problem that searches for achievable visitation distribution that satisfies the Bellman flow constraints:
J(TT) max [ (((sss))) a)] - aD)(((s, || d (s)
(19)
+ + EE d(8', a') (p( s|s (a'dd') d')m(als), Vs S, A. Applying Lagrangian duality and convex conjugate (16) to this problem, we can convert it to an unconstrained problem with dual variables Q(s, a) defined for all a
s.t d(x, = (1 - ()))))))
A: min(1 E( [((s, a)] + (E(.)))-a [f ([T, a) ((s,a)]/a)],
(20)
where I* the convex conjugate of f. We defer the derivation the next section. As problem (19) is convex, strong duality holds and problems (19) and (20) have the same optimal objective value up constant scaling". We refer to the nested policy learning problem where J(T) is of form (19) as pr mrimall00 and the joint problem with scaled J (TT) of form (20) as dual-Q.
primal-8 Q
max [J( in the form Eq. (2)],
(21)
dual Q max, (1 - 7)Be-)) [Q(s, a) + aB (IT, a(s,
(22)
In fact, problem (19) overconstrained the maximization w.r.t is unnecessary, as for fixed x the
[S] x [A| equality constraints already uniquely determine solution d" [Puterman, 2014]. Let n²* d be the optimal policy and corresponding visitation distribution. In fact, we can relax the constraints to get another problem [Agarwal et al., 2019] with the same optimal solution d", which we call
"For our derivations, we will consider smooth extensions of TV conjugate function 'We scaled the dual problem by 1/a for derivation simplicity.
18

---

Published as a conference paper at ICLR 2024
primal-V below: below:
primal-Y
max R - aD) d=0
(s))a)
s.t E as ())os) + d(s', , d')p(sss', d'), Vs
(23)
Comparing with problem (19), the constraints are relaxed and there is no policy ı in this formulation. In fact, as opposed to primal-S Q, which needs to solve nested inner problems, primal VV solves a single problem to obtain d² from which we can recover πT via Eq. (24)%:
T(als) = d" (s, a)/ E d' (8. a).
(24)
Similarly, consider the Lagrangian dual of (23), with dual variables V(s) defined for all E S: dual-V V (1 - [V() (s. [F (LTV(s,
([s))]/a]] (25)
where 1 a variant of f defined in Eq. (47). Such modification to cope with the nonnegativity constraint d(s,a) > 0 in primal-V. This constraint is ignored in primal-Q because the constraints of the inner problem (19) already uniquely identify the solution. See Appendix B. B.1.4 for the derivation. As before, strong duality holds here (up a factor of 1/a), and we can compute the optimal policy n** after obtaining V* We discuss this in detail in Appendix B.1.6.
Remark The above formulations generalizes to the popular MaxEnt RL framework, where the objective J(T) contains an extra policy entropy regularizer. One only needs to replace the Bellman operator T, by its soft variant: T7 a) = r(8, a) + >yE,',''' (Q (Q(s', - log π (a'[s')].
Remark 2. We derive the dual problems via the Lagrangian duality. Taking the primal-9 problem as an example, the key step which bridges its Lagrangian dual problem ming max, I(Q, d) and the final formulation dual-Q is that the maximizer d* of the inner problem has closed form solution. Equivalently. we can rewrite the inner problem maxa L(Q, d) via the convex conjugate (32), which eliminates the variable d. The Fenchel- Rockerfeller duality provides an alternative way to directly reach the same formulation, where one first rewrites the linear constraints as part of the objective using the Dirac delta function [Nachum and Dai, 2020].
Remark 3. The dual formulations have a few appealing properties. (a) They allow us to transform constrained distribution-matte matchiin problems, w.r.t previously logged data, into unconstrained forms. (b) One can show that the gradient of dua 1-Q w.r.f I, when Q is optimized for the inner problem, is the on- on-policy policy gradient computed by off-policy data. This key property relieves the instability or divergence issue in off-po learning. (c) The dual framework can be extended to the max-entropy RL setting, where J(T) consists of additional entropy regularization, by replacing Bellman-opprator with their soft Bellman counterparts [Haarnoja et al., 2017].
B.1.3 DERIVING DUAL-Q
We again consider the RL problem as maximization of convex program for estimating policy performance J(x) by considering optimization over achievable state-action visitations (i.e max, (T)):
max
aDy(d(s,a) max di -aa (s, d30
(26)
s.t d(s,a) (1 (1- (()))))(()).T((())) y Σ d(s' a'
(27)
where allows us to weigh policy improvement against conservatism from staying close to the state-action distribution d., We will assume that d0 has non-zero coverage over S x A for the derivation below.
A careful reader may notice that the inner problem is overconstrained and overparameterized. The solution to the inner maximization problem with respect is uniquely determined by the [S] x |A]] linear constraints, and the nonnegativity constraint d > 0 is not necessary. Moreover, given fixed policy म, the solution of the inner problem is its visitation distribution d=
'Eq. (24) can be easily computed for discrete actions, yet it is difficult for continuous actions. While our analysis focuses on the tabular case, we discuss two methods for recovering x* for continuous actions in Appendix 1.
19

---

Published as a conference paper at ICLR 2024
The constraints of the inner problem known the Bellman flow equations that an achievable stationary state-action distribution must satisfy. The next question is how can we solve it? Here is where Lagrangian duality comes into play. First, form the Lagrangian dual of our original optimization problem, transforming our constrained optimization into an unconstrained form. This introduces additional optimization variables the Lagrange multipliers Q.
As mentioned before, we can discard the nonnegativity constraint d 3 0 as the other constraints imply unique solution for d. Focusing on the inner optimization problem, optimize the Lagrangian dual problem:
min max Re..-d(eea)l(ssss - (D)(d(s, 1l (s,a)) ((.a) d
Q (oos:((11=))0)) (1--))d(().=((lb)) + adddd)pp (()())))) -dsdd((s)) d(s,a) 6,0
d.a
where Q(s, a) are the Lagrange multipliers associated with the equality constraints. We can now do some simple algebraic manipulation to further simplify it:
min max dssnddsss) a)] aDy(d(ssa) d (s)a)) ((
((s.a) (1- (1-7)do(ss (1- + Σ d(s' a')p(sls'' - d(s,a) (28)
min max(1 - 7/) (1oss))))) ((s,a)
+ Eg r(s,a)+ +y
aDy(d(s, (s,
s'
(29)
where we swap the maximum and minimum the last step as strong duality holds for this problem. This is equivalent to solving the following scaled objective (scaled by 1/a).
(1-2) (att))
min max ((s.a) d
+ Eg End (r(,,a)+ + -=wll//soodddsill1000000
- Dy(((s, (s,
s'aa
(30)
(1 (1-1) Baatss)aaaasl
min Q(s.a)
+ Esa (*(((s,a)) o( Q(s a))/a)
(31)
8'.0'
where we applied the convex conjugate (Eq. (16) in the last step. To this more clearly, let a)) = (r(s,a) + + >Ev, p(8'|s, a))a))( ((()) (0(((((('')) a') Q(s, a)) /a. Then, under mild
conditions that the interchangeability principle [Dai et al., 2017] is satisfied, and d has sufficient support over A [Nachum and Dai, 2020], holds that
max Exa aly(s, a)] Dy(d(s,a) d (s)a))
(32)
(s, y a) - (d(ssa)) ( ]
d(s,a) a)
d(s.a)
max
(33)
Eg ()*((s))))))
(34)
We have transformed the problem of computing J(T) to solving Eq. (31). Finally, the policy optimization problem max, (TT) is reduced to solving the following min-max optimization problem, which we will refer to as dual-Q:
max, ming (1-r] atla
E,a-so (()(s,)))>E... 2ie Q(+ ())
(35)
Table 4 lists the corresponding convex conjugates f* for common f-divergences.
20

---

Published as a conference paper at ICLR 2024
In the case of deterministic policy and deterministic dynamics, the above-obtained optimization takes a simpler form:
(1 (1-7) B (oo())
max min x(a]a) ((3,a) a
+ B sa ( + Q ((()
(36)
Now, we have seen how we can transform a regularized RL problem into its dual-Q form which uses Lagrange variables in the form of state-action functions. Interestingly, we can go further to transform the regularized RL problem into Lagrange variables (V) that only depend on the state, and in doing so we also get rid of the two-player nature (min-max optimization) in the dual-Q.
B.1.4 DERIVING DUAL-V
One important constraint we have not discussed so far is that the variable we are optimizing must be nonnegative. This constraint uniquely determined by the constraints. Nonetheless, is important we consider this constraint for primal-V and derive the correct dual problem.
needed for pri 1 -Q, as for the inner problem (2), the solution
In primal-V/ we formulate the visitation constraints to depend solely on states rather than state-action pairs. Note that doing this does not change the solution ** for the regularized RL problem (Eq (18)). We consider a = 1 for the sake of exposition. Interested readers can derive the result for a * 1 as in the dual-Q case above. Recall the formulation of primal-V:
max (r(s,))r(ss (s, a)] - [d(((s, a) ||d0(s (s.a) 430
Σ d(ssaa) do)-)s + Σ d(s'' d''(ss' d').
(37)
s'a"'
As before, we construct the Lagrangian dual to this problem. Note that our constraints now solely depend on s.
min max E. F,, - ( (l
V(s) da0
v(s) (1-7))0(s) + Σ dis al (()spp)sss
d((s,a)
(38)
aEA
Using similar algebraic manipulations we used to obtain dual-Q Section B.1.3, we have
min max E [. - d (s)
V(s) d(a.)=0
Eg a(),a) + p(s()sss.)) v(s)) Dy(d(s,a) (s)
(39)
min max (1 - (l
(s) d(s.))00
E Egadd r(s,a) + + [p(6ls,,)V(()) v(s) Dy(d(s,a) (s,
(40)
min max
(1 Bed [V(s)]
V(s) d(s.))20
E R
d(s.a) To [s.a] (r(s,a) + [16))b,))66-- v(s)) E Bsaando
[s(
d(s.a)
(41)
Let w(8 a) d(s.a)
(s,a) and óv a) = r(s, 7 7E a() V(s) denote the TD emror.
The last equation becomes
min max
(1-7) (1-7) (V) [s)] Eo a)( a])] Ex [f a))]- (42)
((s) w(s,a}30 We now direct the attention to the inner maximization problem and derive a closed-form solution for Consider the Lagrangian dual problem of it:
Σ X(s. 8,0 where the parameters X(s, a) for all E S and a E A are the Lagrange multipliers. Since strong duality holds, we use the KKT constraints to find the optimal solutions w (s, a) and x* ():
min max Eya-do [w(s, (5v (s) - X w(s.a)
(43)
21

---

Published as a conference paper at ICLR 2024
1. Primal feasibility w* w*,s Vs,a
2. Dual feasibility (s 2 Vsa
Stationarity
(s, a)) + (v + vs,a
Complementary Slackness w*( (8, (5saa))5 ())* = V8,a
Using stationarity we have the following:
d a) f'(w" (s) (s,
(s) a) X* Vs.0
(44)
Now using complementary slackness only two cases are possible w² (s, > or (s, Combining both cases we arrive at the following solution for this constrained optimization:
> 0.
w* (s, a) max (0, f (óv(sss
(45)
We refer to the resulting function after plugging the solution for w back in Eq. (42) and refer to the closed form solution for d in second and third term as fp.
I (8v a)) = w (8, (8) a)) f(w"((, Plugging in w*, a) from Eq. (45) to Eq. (46), we get:
(46)
( a)) max (0, fr (s,(s) (ōv (s, a)) - (max (o. (ov
(47)
Note that we get the original conjugate ! back if we do not consider the nonnegativity constraints:
(s) (óv(s, (ōv a)) - (ōv a)))
(48)
Finally, we have the following optimization to solve for dual -V when considering the nonnegativity constraints:
dual-V: miny(s) 7) (( (fv((sss
Some works e.g. SMODICE [Ma et al., 2022], ignore the nonnegativity constraints and use the corresponding dual- - formulation
dual-V (w/o nonneg. constraints): miny (1 (1) [V(s)] (esss)))++aaaaddddeele of
[V()))
B.1.5 DISCUSSION ON DUAL FORMULATIONS
In summary, we have two dual formulations for regularized policy learning:
dual 1 max, ming(1 7) ml + E0 [f (r(s,a) + E, al -((ss))
(l((ss))) a)]
and
dual-V: miny(s) - E,, (V) + a
(fv
The above derivations for dual primal RL formulation dual-Q and dual-V brings out some important observations
dual-Q and dual-v present off-pooicy policy optimization solutions for regularized RL problems which requires sampling transitions only from the off-policy distribution the policy state-action visitation is being regularized against. The gradient with respect to policy when dl is optimized dual -Q can shown to be equivalent to the on-policy policy gradient under regularized Q-function (see Section 5.1 from [Nachum and Dai, 2020]).
The above property allows to solve not only RL problems but also imitation problems by setting the reward function to be zero everywhere and do to the expert dataset, and also offline RI. problems where want to maximize reward with the constraint
22

---

Published as a conference paper at ICLR 2024
that our state-action visitation should not deviate too much from the replay buffer (d replay-buffer).
dual-V formulation presents a way to solve the RL problem using a single optimization rather than min-max optimization of the primal maaa--- -Q or standard RI. formulation. dual 1baa1 -V implicitly subsumes greedy policy maximization.
B.1.6 HOW TO RECOVER THE OPTIMAL POLICY IN DUAL-V?
In the above derivations for dual- and dual- we leveraged the fact that the closed form solution for optimizing Eq. (16) w.r.t d is known. The value of d for which Eq. (32) is maximized can be found by setting the gradient zero (stationary point) leading to:
d (s.a) d (s.a)
max 0, (J)) ()(saa)
(49)
This ratio can be utilized two different ways to recover the optimal policy:
Method 1: Maximum likelihood on expert visitation distribution
Policy learning can be written as maximizing the likelihood of optimal actions under the optimal state-action visitation:
max E, =s [log TT(a|s]]
(50)
Using importance sampling we can rewrite the optimization above in form suitable for optimization:
d (*,
max R do
d0 log log=o(als) = max E (x,, a) log [o(a|s)]
(51)
This way of policy learning is similar weighted behavior cloning or advantage- advantage-weighted regression, but suffers from the issue that policy is not optimized at state-actions where the offline dataset d0 has no coverage but d* >
Method 2: Reverse KL matching on offline data distribution (Information Projection)
To allow the policy be optimized at all that states the offline dataset actions outside the dataset we consider an alternate objective:
min DKL (d (s)To(als) (s)
(52)
0
The objective can be expanded as follows:
min DRI(d (s) (s)l))))))
|| d (<ss)<(als))
(53)
[a(a
min E (2)
log T (als)
(54)
(s) (s)=
min E. (
log (+(ass)d* (s)
(55)
min E. ( log ma(ass) log (* (s. a)) log d (s)
(56)
(als)
(0(s)
min E. (s) (og(TT((ass)) - (alss) log(w" a))]
(57)
This method recovers the optimal policy at the states present in the dataset but has the added complexity of learning another policy (als). One way of obtaining (a(s)) (s) is by behavior cloning the replay buffer.
B.1.7 SEMI-GRADIENT AND FULL- TULL-GRADIENT
RL algorithms often learn via Bellman backups which minimize an error of the form L(0) = "'
a) (r(s,a) a) + 7l [Qo [ through the entire objective, whereas semi-gradient methods do not differentiate through the Qa(s', aa) term in the bootstrapping target and is generally implemented through a stop-graud doo-ggraieent operator. Note that the semi- aami-graddent update still changes the value of the bootstrapping target when used the next iteration as is updated. Semi- grmieerdd optimization is a common choice in deep RL and often enables significantly faster learning [Sutton and Barto, 2018).
d')) Full stochastic gradient differentiates
23

---

Published as a conference paper at ICLR 2024
C A UNIFIED PERSPECTIVE ON RL AND IL ALGORITHMS THROUGH DUALITY
Figure 4 shows an alternate viewpoint on the landscape of dualRL methods and highlights the gaps in algorithms that we able to successfully address in this work. Now we discuss in detail how recent algorithms can be unified via duality viewpoint for both RL and IL.
The Dual-RL Landscape
Deep RL
Imitation Learning
(offline and online)
Offline (only expert data)
offlcy (Online/Offline) with coverage assumption
officy (Online/Offline with arbitrary data
IQLearn
dualQ 2 2-player gamee
AlgaeDICE, (CQL)
(Implicii
OPOLO
ReCOIL-Q
Behavior
Cloning)
dualV 1-player optimization
OptIDICE (XQU,
ILLea
SMODICE
ReCOIL-V
f-DVL
No discriminator learning needed.
Discriminator
No discri minator learning needed.
learning
Figure 4: We show that number of prior methods can be understood as special case of the dual RL framework. Based this framework, we also propose new methods addressing the shortcomings of previous works (boxed green).
C.I DUAL CONNECTIONS TO REINFORCEMENT LEARNING
We begin by showing reducing popular offline RL class of methods: pessimistic value learning (CQL [Kumar et al., 2020], ATAC [Cheng et al., 2022)) and implicit policy improvement (XQL [Garg al., 2021]) the dual-9 1 and dual-V framework respectively. Then, show how the dual l-V framework under a semi-gradient update rule leads to a family of offline RL algorithms that do not sample OOD actions.
Proposition 4. CQL an instance of dual under the semi eemi-gradient update rule, where the f-divergence is the Pearson x" divergence, and d is the offline visitation distribution.
Proof. We show that CQL [Kumar al., 2020] and ATAC [Cheng et al., 2022], popular offline RI. methods are a special case of dua 1 -Q for offline RL. Consider the x² f-divergence with the
generator function f (t - 1)2. The dual function f is given by f - f). With this f-divergence the dual-O optimization can be simplified as:
(1 (( (Q(, + E Rg y(s (,,,,,,,8')2 v(8,a,r,8') 4
(58)
a
7) E do (al))) [Q(s,
y(s,,ar,,'))
y(s,
LQ(s a)] E Exaano
Eg
E
(59)
402
Let's simplify the first two terms:
1
a (1 7) ((a))) xtt)) + Esaado r(8,,)+ + [1661ssa011(111100660 -
(60)
(1 - B( Esa-d0 p(s' [/ (aa-dol((ss0)) + t
s'aa
(61)
1
(1 7) E [((ss) + (
E IQ(s, a))
(62)
24

---

Published as a conference paper at ICLR 2024
(1 7) Σ + do -o [Q(s,
(63)
a
2,0
7) Σ (n(3) .Q) E Ro [0(8,))
(64)
a
7)
Σ (1()) + [(s p(ls'','')d ,d)) E Eg [Q(s,)]]
a
(65)
Σ
(do(s) + p (() Eo
E, Be [r(8,a)]
(r(
8.0
8'a
(66)
Σ ((())((a))((ssa) 10
+ ( (r
(67)
1 B a-t 10 a)] - E. (Q(s,
(68)
where pd denotes the policy transition operator, PE denotes the adjoint policy transition operator. Removing constant terms (Eq. (61)) with respect to optimization variables we end up with the following form for dual-Q:
y(s,a,r,s') 4a
E a0-d),arTIQ(ss00
E Bo [Q(s,)]] [Q(s,aa
E B,ando
(69)
reduce Q at 0OD actions
increase Q at in-distribution actions
minimize Bollman Ervor
Hence the dual-Q optimization reduces to:
Es [ a)]] + E ga as,,a,r,8')
max [E a~ - [Q(s,
(70)
Our proposition assumes semi-gradient update, ie the gradients are not backpropogated through Q. The bootstrapping target is regularly
the bootstrapping target Q(s', ((s)) when updating updated with the most recent parameters. Thus, maximization with respect to policy just amounts maximizing the first term Ey-do ann-dd_a--1((66 [Q(s, a)]. This update equation matches the unregularized CQL objective (Equation 3 in [Kumar et al., 2020]) and the ATAC objective (Equation in [Cheng al., 2022] when / 0.25). One of they key differences between CQL and ATAC is the use of optimization strategy CQL uses Gradient Descent Ascent whereas ATAC uses Stackelberg formulation.
Proposition 2. XQL is an instunce of dual-V under the semi- graidgradient update rule, where the [-divergence is the reverse Kullbuck-Liebler divergence, and d is the offline visitation distribution.
Proof. We show that the Extreme Q-Learning [Garg al., 2023] framework for offline and online RL is a special case of the dual framework, specifically the dual -V using the semi-gradient update rule.
Consider setting the fddivergence to be the KL divergence in the dual-V framework, the regularization distribution and the initial state distribution to be the replay buffer distribution (d dl and dg d The conjugate of the generating function for KL divergence is given by (*(t)= c1
min(1 (Bdocc)(((s)) E Egn-dR fr ((rissa) + ( /a V(*)
(71)
25

---

Published as a conference paper at ICLR 2024
([risso) + ( -=(661s.))(( -
expl(
la /a-1)
min(1 - Bdsss) + sawdd V(s)
(72)
popular approach for stable optimization in temporal difference learning the semi-gradient update rule which has been studied in previous works [Sutton and Barto, 2018]. In this update strategy, we fix the targets for the temporal difference backup. The target in the above optimization given by: (ss, a) = r(s, + + Σ [
(73)
The update equation for V is now given by:
min - Be (B))))))))+alll [exp( ((Q(s,a) v())) 1)] V(s)
(74)
where hat denotes the stop- goopggradient operation. We approximate this target by using mean-squared regression with the single sample unbiased estimate as follows:
min as [(Q(s, (r(s. ( Q
(75)
The procedure (alternating Eq. (74) and Eq. (75)) is now equivalent to the Extreme- learning and is special case of the dua 1-V framework.
C.I.1
f-DVL: A FAMILY OF IMPLICIT POLICY IMPROVEMENT ALGORIHHM FOR RL
0.4
0.4
04
04
0.2
02
02
02
9060
AL87380
2,0/5000
7,07500
2,07200
0.
80
00
80
Figure 5: Illustration of a family of implicit maximizers corresponding to different f fdiivergences. The underlying data distribution is a truncated Gaussian TN with mean O, variance I and truncation range (-2, 2). We sample 10000 data points from TN and compute the solution ex of Problem (15). As X the solution vx becomes more accurate estimation for the supremum of the random variahle
Proposition 3. Let be a real-valued random variable such that Pr(x > x*) 0. Let ux be the solution of Problem (15), It holds that UX & Uxp VO < X < X2 1. Further, lim UX = x*
Proof. The behavior of dual-V (Equation 5) when f is reverse KL to serve as an implicit maximizer was established in [Garg et al., 2023]. In this section we consider other divergences from Table under the rewriting of dual-V in terms of temperature À and surrogate extention for J (defined below). We analyze the behavior for the following optimization of interest.
min(1 - X)E2=D[1] AR (x -)]
(76)
f is given by (using the definition in Eq. (47):
f (t) ff (max(f" (()) (t).
+ max (f (t).
(77)
Accordingly, the function f* admits two different behaviors given by:
-(- (0)+ +f- (*)),, if (1)
(78)
-()).
otherwise
where f is the convex conjugate of of f-divergence. We consider all f-divergences for which f* is strictly increasing in Rt, and note that TV divergence will need special treatment as (f")-1 is not well defined. Some properties of note are f' and (f"'-1 is non-decreasing, f(0+) > 0 and f*(x)= > 0 Vx > 0 for the divergences we consider. A key limitation of formulating an optimization objective with forms of f-divergences is their domain restriction of R. t. First, note as a result of restriction of f to + that f R+ + 10) for some / E IR and as a consequence (f')-1 : (1) R Since objective function depends (f')-((x) being well defined
26

---

Published as a conference paper at ICLR 2024
on R. We consider extension that preserves the non-decreasing property of (f" such that
(x) = 0 for æ E (-xc, J]. We also know from above that for x >= 1 (f') (z) >= 0 as
is dencescc....... We define the surrogate f to be a particular extension for f with / = 0.
Similar extensions can be found in prior work [Picard- Weiiel and Guedj. 2022a, Goldfeld).
We analyze the second term in Eq. (76). It can be expanded as follows:
A (x:((")+(aauj)))) f(z) (x - vvde - A
(a--))) 1(0)((x)dr
(79)
From the properties of f, we use the fact that (f²)- v) > 0 when z - v > 0 or equivalently
x > v.
{ (x A f(0)p(z)dz
(80)
Jasy
The first term in the above equation decreases monotonically and the second term increases monotonically (thus the combined terms decrease) as e increases until v x* (supremum of the support of the distribution) after which the equation assumes a constant value of -)((0.
Going back to our original optimization in Eq. (76), the first term decreases monotonically with v. As X 1, the minimization of the second term takes precedence, with increasing until saturation (v = x²). We can go further to characterize the effect À on solution VY of the equation. The solution of the optimization can be written in closed form as (using stationarity):
(1 X) = Ez-D
(81) Using the fact that f is non-decreasing, we can show that the right-hand term in the equation above increases/stays the same as decreases. This in turn implies that for all X1, A2 such that x & A2 we have that E)2 < Vx2-
1 (x (a- v)
A
TV divergence require a special treatment as (f*)- is not defined. We construct f by noting that f* for TV exists even if (f')-1 does not. A concise proof can be found in Example 8.1 from Goldfeld. Thus, for TV we consider a smooth extension in R by using f*(x) (xx = x. For Squared Hellinger and RKL, f is discontinuous. The problem can be addressed by considering random variable ~ D upper bounded by and In 2 for Hellinger and RKL respectively. This can be ensured by rescaling rewards so that the maximum reward is y and (1 7) In(2) for hellinger and RKL respectively. Appendix F.3 outlines the derivation of surrogate implicit maximizers that we use in practice.
C.1.2 CONNECTIONS OF CQL TO ALGAEDICE AND XQL TO OPTIDICE
Kumar et al. [2020] shows that CQL outperforms a family of behavior-reguulriized offline RL methods [Fujimoto et al., 2018, Wu et al., 2019, Nair et al., 2020]. which solve different forms of primaal -Q using approximate dynamic programming. The above result indicates that CQL's better performance is likely due the choice of f f-divergence and more amenable optimization afforded by the dual formulation. Moreover, the same dual-Q formulation has been previously studied for online RL in AlgaeDICE [Nachum al., 2019], and proposition 4 suggests that CQL is an offline version of AIga.DICE.
We also highlight that the full-gradient variant of the dual framework for offline RL has been studied extensively in OptiDCE [Lee et al., 2021] and proposition 2 highlights that XQL is a special case OptiDICE with a semi-gradient update rule.
C.2 DUAL CONNECTIONS TO IMITATION LEARNING
This section outlines the reduction of a number of algorithms for Imitation Learning to the dual framework. Most prior methods can either take into account expert-only data for imitation whereas the other methods which do imitation from arbitrary offline data are limited by their assumptions and the form of /-divergence they optimize for. We walk through explaining how prior methods can be derived through the unified framework and also why they are limited.
C.2.1 OFFLINE IMITATION LEARNING WITH EXPERT DATA ONLY
We saw in Section 4.1, how using the dual-Q framework directly led to a reduction of IQ-Leamn [Garg et al., 2021] as part of the dual framework. This accomplished by simple
27

---

Published as a conference paper at ICLR 2024
setting the reward function to be uniformly and setting the regularization distribution to the expert. Garg et al. [2021] uses this method in the online imitation learning setting as well by incorporating the replay data additional regularization which suggest is unprincipled, also pointed out by others [Al-Hafez et al., 2023] (as only expert data samples can be leveraged in the above optimization) and provide fix in the Section 5. In this section, show how the same approach directly lead to another method for learning to imitate from expert-only data avoiding the alternating min-max optimization of IQ-Learn.
IV-Learn: A new method for offline imitation learning: Analogous to dual 1 - Q (offline imitation), we can leverage the dual-V (offline imitation) setting which avoids the min-max optimization given by:
IV-Learn or dual-V (offliine imitation from expert-only data): min - (v) + B. [f ((ToV(s,a) v(s))]/a)]
(82)
V(*)
We propose dual-V (offline imitation) to be a new method arising out of this framework which we leave for future exploration. This work primarily focuses on imitation learning from general off-policy data.
Proofs for this section:
Corollary 1. IBC [Florence al., 2022] is an instance of dual- using the full- full-gradient update rule, where r(s,a) = 0 Vs E S, E A, d = d=, and the f-divergence is the total variation distance.
Eq. (6) suggests that intuitively IQ-Learn trains an energy-based model in the form Q where it pushes down the Q-values for actions predicted by current policy and pushes up the Q-values at the expert state- action pairs. This becomes more clear when the divergence chosen to be Total-Variation (f* IT), IQ-Learn for VariatVai divergence reduces to:
(1 - ( Es
Σ p(s'ls, (( Q(s, a) 6'.a'
(83)
(1 7) (116))))). + B Beaane [pdd
8'
E d aQ)
(84)
First, we simplify the initial two terms:
> En661sammdd)))
(1 (4), +,, Ba
(85)
= (1 - x) Σ (() + ddl Emstsaadddsees
(86)
8,a
8.0
(1 - 7) Σ ((((s)
ΣΣ (ss.ppppsss
(87)
8;a
L' 8,a
v) Σ dd((s) + Σ T(d's))Q(s a') Z d' dss
(88)
6,a
8'.a'
v) Σ de(ss) Σ 7(dl)s)(((s,))))) (s.
(89)
28

---

Published as a conference paper at ICLR 2024
= (1 (1-1) [ [d(s)))())))) + + ((s,))
a) Σ ('p(ssss)))
(90)
Σ
d ((ddd)p((((sdd
(91)
8:0
Σ [=(als)((s.0 a)
(1 (1- 7d)(s) + Σ
a (K
(92)
8,8
s/aa
Σ ((a)s)Q(ssa))) (s) 8,8
(93)
where the last step is due to the steady state property of the MDP (Bellman flow constraint).
Therefore (Q-Leam/dual-O) for offline imitation (in the special case of TV divergence) simplifies to (from Eq. (84)):
(1 Ed ((tss dE ? Σ p(s'ls, [llssis.))add))))(
E Es de [Q(s,
a)]
(94)
min (eg(e) a)] 2 [Q(s,a)] Q
(95)
The update gradient w.r.t for the above optimization matches the gradient update of infoNCE objective in Implicit Behavior Cloning [Florence et al., 2022] with as the energy-based model.
C.3 OFF-POLICY IMITATION LEARNING (UNDER COVERAGE ASSUMPTION)
Directly utilizing the dual-RL framework for imitation has its limitation as we see in the previous section we cannot leverage off- off-co suboptimal data. We first show that easy to see why choosing the f f-divergence to reverse KL makes it possible get off-policy objective for imitation learning in the dual framework. We start with the primal -Q for imitation learning under the reverse KL-divergence regularization (r(8, a) = 0 and do = d)):
d(s.a)=0,x (als) - -KL (d(s, a) || (s,a))
s.t d(s,a) = (1 (1--7) (1--))o(s) + yo(als) d(s',e d'ssssss's s'aa'
(96)
Under the assumption that the suboptimal data visitation (denoted by d covers the expert visitation (ds > wherever d0> [Ma et al., 2022], which we refer to as the coverage assumption, the reverse KL divergence can be expanded as follows:
log d(s,0) a)
E (( log (s,a) d ( dF a) d3 ds(s,
Dkl(d(s,a) || as) B d(s,a)
d (s)
(97)
log d(s.a)
d (s)
B(a
log (F(
(98)
75 (8,0)
( a)
=Baaavd(saa) Hence the primal-9 can now be written as:
log d (s.
DRL(((ssa)) (|
(s,
(99)
ds ($(x,0)
max ((s.)))0,7<a(s) Be
log (s,aa)
- (dl(d a) ds
(100)
s.t d(s, (1s
Σ d(s' a p(ssss p d'lss)
(101)
Now, in the optimization above the first term resembles the reward function and the second term resembles the divergence constraint with a new distribution dd (s, a) in the original regularized RL primal (Eq. (26)). Hence we can obtain respective dual 1 1 and dual-V in the setting for off off-policy
4 (s.a) and the regularization imitation learning using the reward function as pimit (s, a) log (5,a)
distribution as d (x, a). Using T.m. and T,ma to denote backup operators under a new reward function y iimit we have
29

---

Published as a conference paper at ICLR 2024
Input
ReCOIL
Output
Offine Data
Expert Data
Expert state action score
Learned score function
Learned Policy
+
Replay score
+
Nomsaspert argnick,Qts.0 4)
Bellman consistency
Figure 6: Recipe for ReCOIL:: Learn Bellman consistent EBM A model which increases the score of expert transitions, and decreases the score of replay transitions while maintaining Bellman consistency throughout.
dual-Q for off- off-policy imitation (coverage assumption)
max min (1 (. (e)))
[Q a)] + a (T, Q(s a))]
(102)
*(als) Q(..a)
This choice of KL divergence leads
reduction of another methods, OPOLO [Zhu al., 2020] and OPIRL [Hoshino et al., 2022] for off off-policy imitation learning to dualq which we formalize in
the proposition below:
Proposition 5. OPOLO (Zhu et al., 2020] and OPIRL [Hoshino al., 2022] are instances of dual-Q using the semi-gradient update rule, where r(s,a) 0 VS d d and the f-divergence set to the reverse KL divergence.
Proof. Proof is sketched in the above section, ic. Eq. (102) is the update equation for OPOLO.
Analogously have dual-v for off-policy imitation (coverage assumption):
min 7) (V) [s]]+ Ex [
( (T, (s, a) V(s))]
(103)
(*)
We note that the dua 1 v framework for off-polic imitation learning under coverage assumptions was studied the imitation learning work SMODICE [Ma et al., 2022].
C.4 LOGISTIC L-EARNNING AND P'IL AS DUAL- QV METHODS
Logistic Q-learning and Proximal Point Imitation Learning (P²IL) uses a modified primal for regularized policy optimization:
max (r(e,,)(((ssa)) - ()(((dsss))) lI (s, HI( d=0
s.t d(s,a)= (10)==11-0)))) (1-7)do(s)) m(als)> Σ p(s a's''a')pdl a')ssss's'
(104)
and a(s.a) = p(s,a)
(105)
where H(p(s ((l(s,)))ln (s,a)) M(s a) log (als)
(a)() denotes the conditional relative entropy and
nO is another offline distribution state-action transitions potentially the same as d0, The optimization is overparameterized (setting μ d). This trick popularized via [Mehta and Meyn, 2009] and leads to unbiased estimators and better downstream data driven algorithms. We call these methods dual-Qu 1-Q their dual requires estimating both Q and V shown in [Viano et al., 2022, Bas-Serrano et al., 2021]
D RECOIL: OFF-POLICY IMITATION LEARNING WITHOUT THE COVERAGE ASSUMPTION
Understanding the limitations of existing imitation learning methods in the dual framework, we now proceed to derive our proposed method for imitation learning with arbitrary (off- policy) data. The derivation for the dual-Q setting is shown below. dual-V derivation proceeds analogously. Theorem 1. (ReCOIL objective) The dual - problem the mixture distribution matching objective in Eq. given by:
max, ming =Q(s,,) + E, g. (TT Q(s, (1- 33) [Te Q(s,a)) (9) and recovers the optimal policy * Eq. 8 since strong duality holds from Slater's conditions.
30

---

Published as a conference paper at ICLR 2024
Proof. Let's define two mixture distributions that going leverage to formulate the imitation learning problem: d (8,a) a) Bd((s a) (1 3)d (s, a) and d'iis (s, a) 3d (8. a) + (1
3)d (s, a). (s, is a mixture between the current agent's visitation distribution with suboptimal transition dataset obtained from a mixture of arbitrary policies and d a) the mixture between the expert's is visitation distribution with arbitrary experience from the offline transition dataset. Minimizing the divergence between these visitation distributions still solves the imitation learning problem, i.c d = d We again start with the new modified pr primal- under this mixture divergence regularization:
max - -Dy (d (8, ()( a) || F..S d(s.)) ((
(s, (8.
d(s,a) (1 7)do(s). + rm(als) Σ di(s', d(ss'd')psssssdd
Using the same algebraic machinery of duality before (Section B.1.3) get an unconstrained tractable optimization problem, we obtain:
max min -Dy(d (s, d.E. (s.a)) a))
3,030 ((s.a)
8 ((s,a)) (1- (1-7)do(s) +y Σ d(sss - d((s,))
(106)
8,0
d'a''
min B(1 (1l)))/aco)))a)1(((sss
a.d30 Q(s.a)
+BE a
Dy(dmix (5, d (s, a))
(107)
max min
B(1 ((11))BBahhhaa
a.d30 Q(s.a)
+ HE and
+ (1 Ba
(1 - -B
[x66)s0)mdd11))0 -((sa) DD(d (s, a) d (s,a))
a))
(108)
Strong duality allows us to swap the order of max and minq in order arrive the following result:
Imitation from Arbitrary data (dualQ)
max min max 3(1 7) () =(als) ((5.a) dz0
+ E Bsaadd (s.a)
Σ p(s [mllls.amm((1))0) Q(s,))
(s, d (s, a))
-(1 - (1- Ba
? 1 -(()))
(109)
We can ignore the constraints (d > 0) as the primal-Q is overparameterized and the constraints uniquely determine the distribution Therefore, ignoring this constraint (d > 0) results in the following dual-optimization for imitation from arbitrary data.
31

---

Published as a conference paper at ICLR 2024
max min 3(1 - 7) x(ops) ((a.a)
+ aa-d (
(1 3)
Q(s,a)
(110)
D.1 RECOIL-V
similar derivation can done in V-space to obtain an analogous result for ReCOIL-V, although extra care has to be taken to ensure the non-negativity constraints similar to proof for in section B.1.4.
primal-V
max d(s,a)=0
- (s, a) || 2
s.t L d(s a) (1-7)d((s) (ed d(s', a' (')p(sls's a'), Vs E S.
(111)
The dual of primal-ly form for mixture distribution matching is given by: Rec011-V miny A(1 - B... (V]]] (1 V[+)]] - a (s;a)) v(+)].
(112)
D.2 SUBOPTIMALITY BOUND FOR RECOIL-V
Theorem 2 (Suboptimality Bound for Offline ReCOIL). Let S denote the joint support of ds and d Let r(8, a) Vss) (s, a) be the pseudo-reward implied by ReCOIL and Rmaax max,,a r(s, a). Let D5 =(d |ra ((s, a) S > 1 1- 8} be the set of visitation distributions that have 1- coverage of SJ, Let nž be the best policy over all policies whose visitation distribution is in D&. Let g(d, V) = (1- - Ed,to))) the imitation learning objective, and hVV) = max g(d, V). Suppose that we can solve ReCOIL
+ V a) - V(s)] - (d(((s a) || dd (s, a)) be
with the constraint d Da, h is n-strongly convex in V and 3 1, then the output policy ß satisfies that J(mz) J(A) < 1- 2288
We provide a suboptimality bound by analyzing ReCoIL-V in this section. An important note is that we consider the setting
1, which implies that we study the behavior of the optimization when 3 is number close to 1 and not exactly 1. This allows us to incorporate suboptimal data in off- off-policy imitation learning setting.
Recall that ReCOIL-V admits dual-v form (112). When deriving dua 1 1-V, there is one step (Eq. (41)) where we assumed the importance sampling is exact, i.c., (TV(s, v(s))]
(113) However, this assumption does not hold in general and is not practical, because do and d might have different support. The gap between the two terms greatly affects the performance of dual RL approaches. We shall bound the approximation error introduced by importance sampling for ReCOIL-V in Section D.2. I, and then bound the suboptimality of the learned policy in Section D.2.2, under mild conditions. This analysis also results in the suboptimality bound of IV-Learn and IO-Learn methods.
B(s.,)-a(TV(sss)) V(s)] (s.a))do
d(s.a)
(x.0)))
Let S denote the joint support of d and d= Letr(s, = V(s) ToV (s, a) be the pseudo-reward implied by ReCoIL and Rmax max, |r(s,a),. Let D, {d Pra ((s, a) S > 1 - 8} be the set of visitation distributions that have 1 8 coverage of sd, where Pra ((s, a) 5y) is the probabily that (s, a) lies in S when sampling (s, a) from d.
We make the following assumptions for our proof:
Assumption 1 We consider imitation learning under the constraint d E Da. This is similar to pessimism assumption when learning from fixed datasets in offline RL Levine et al. [2020].
Assumption 2 The hyperparameter (3 for defining d (8, a) and F.S (s, a) goes to 1: B
32

---

Published as a conference paper at ICLR 2024
Assumption 3 The function h(V) defined in Section D.2.2 is k-strongly convex.
For Assumption 1, ReCoIL-Vis is able to find a policy under the visitation constraint as a result of a combination of implicit maximization, which prevents overestimation and thus choosing OOD action, and weighted behavior cloning (Advantage-weighted regression), which keeps the output policy close to the dataset policy.
D.2.1 APPROXIMATION ERROR OF THE IMITATION LEARNING OBJECTIVE
= The imitation learning problem can be written in the Lagrangian form of primal -l where r(s,
0 everywhere:
min max (1 d)) [V()]] + (u a) - Dy(d(s. || (s, (114)
-
de
where we have constraint d D, due to Assumption 1. RecoIL-V optimizes a surrogate objective of Problem (114). To derive ReCoIL-V, consider the corresponding primal-v in its Lagrangian form
min max (1 (o)))) [V()) Eu VV()] (y (s, V deD; Rewriting the second term, we obtain
|| (s, (s,a)) a))-
(115)
min max (1 - Edsss) 3
B B. (2,a) (o)
V de Ds
1-8 Ess ToV V(s)] 3
Dy D (s, a) a
(s,
(116)
Now we approximate the second term via importance sampling, which leads to
d (s, a)
min max (1 7) (c) [V()) (V)())) dEDs
Beaaddii (s.a)
B Cha-dtis
(ToV V(s))
d (s,a)
di (8.
)
1- Egs (To((s, V(s)].
E mis (s,a)
f(
(117)
E.S (s,a)
By expanding (s, Bdl a) (1 3)d (s,a), we obtain
min max (1 Edo(s) [V{)] + 1 deDa
d ( a)
3 B aa (e,20)
(ToV (((s)
V(s))
E.S
(s, a)
E R (2,a)
f
ds (s
min
)
3
Eds (ToV - -(s)],
(118)
g (s, a)
3
This can be further simplified
d(s,a) (nToV(s,,) v(s)) -vs))
min max (1 B)) [V)] + d. (s.a) deDs
E.S
(s,
d (s,
E 2 (s,0)
f
(119)
E.S (s,
where we used the fact
ds (s, (TV
E E. E. (s,a)
4s (oV V(s)]
J.. mix (s.a)
as the support of E.S (8, a) contains the support of ds.
Let gdd, V) and (d, V) be the objective functions of Problem (114) and (119). g(d, V) is the original IL objective sampling) of g(d, V) used by ReCOIL-V. To simplify the analysis, we consider the case when mixture ratio 3 - 1 (Assumption 2), that the approximation error of the objective function reduces to the approximation error of importance sampling. That is,
want to solve, and GRecOII (d, V) is approximation (with importance
lg(d, V) GRecorL(d. V)l EaLToV V(s)] E (s,a)
d(e.a)
(s.0) (ToV(s, - V(s)) (120)
5.3 15.3
33

---

Published as a conference paper at ICLR 2024
For any visitation distribution E Ds. it holds that
(al(ToV(ss [[s)) E (s,a)
Maaa)
(s, V(+))
(x.a)
$ R To (s, a)- V()) & max 8 (T, a) - V(s) & m where St is the support of d. and the second inequality follows from the definition of Dj. As consequence, we can bound the approximation error
(121)
CReCUIL
max g(d, V) lim TRecOLL (d, V) 8-1
& &
(122)
de ds
Similarly, one can show that for IV- Learn, we have
Ig(d, V) - (dv V)l - Edl (V V(s)] B.
d(s,a) d (x,0)
(ToV v(s))
(123)
Let S be the support of d=, Unlike ReCOIL-V, V, the objective of IVLearn suffers from the following worst-case estimation error
(s.((TV((s.a) V(s))] - EnE
d(e.a) (3,a)
(T) (VV)))
< B (e,0)ss VV a) - V(s) < max (sV(ss - -V(s) & Rmaxx
(124)
and consequently
(VVLaarn
g(d lim Diviearo (d, V) x Rimax- deDa.V
max
(125)
B-1
We note that the same approximation error bounds hold similarly for IQLearn as that of IVLearn. Thus ReCOIL has smaller upper bound for the approximation error than IQLearn which we will see in the next sections leads to a better performance guarantee than TQLearn.
D.2.2 PERFORMANCE BOUND OF THE LEARNED POLICY
Recall that CRecOIL denotes the approximation error of the objective function by ReCOIL-V:
ÉReCOIL
max g(d V) lim G(d, V) dED,,V
(126)
Let h(V) ma (dd, V) and T(VV) max pim O(d, V). It directly follows from Eq. (126) that
(n(V) h(V) & LeRecorI. VV.
(127)
We note that max, od, V) (without the d E D3 constraint) is the standard dual-v form for imitation learning, but ((V)) here defined as the same optimization under constrained set d E Dg.
Let V. arg miny T((V) and V
arg miny h(V). We are interested in bounding the gap
h(V) h(v*). It holds that h(V) h(V*)
n(V) - 7(()) R(V) h(V*)
(128)
= A(V - P((P)= + 7(V - T((V*) + -((V*) - M(V*)
(129)
$ 2ERECOIL + 0 + 2EReCOIL
(130)
4ERECOLL
(131)
where the inequality follows from Eq. (127) and the fact V = arg migg h((V).
As a consequence, we have
4ERecOII > A(V) - h(V*)
(132)
> ((V*) + (V* V)Vb(V*) + K f - - -N(V*)
(133)
V* V12
(134)
where the second inequality comes from the fact that the function k(V) is k-strongly convex (Assumption 3) and Vh(V*) It directly follows that
V | -
V 2 CROCOLL-
(135)
Let T be the policy that acts greedily with value function V*, which is an optimal policy over all policies whose visitation distribution is within Dg. Let & denote the policy that acts greedily with value function V. ie.. the output policy of ReCOIL-V. We then use the results in Singh and Yee
34

---

Published as a conference paper at ICLR 2024
[1994] to bound the performance gap between mg and I::
28 28
J
(136)
K
The above results demonstrate that ReCOTI able to leverage suboptimal data with an approximate in-distribution policy improvement and results in a policy close to the best policy with visitation almost iu-pup of the dataset.
D.3 RECOIL WITH x² DIVERGENCE
In this section, derive the objective for ReCOIL under the chosen x² divergence. with the core ReCOIL objective for Q-update:
max min ((1-2)
[To Q ((s,,))(((s,a))
(137)
Under the x² divergence (f* = x² 2), we can simplify the Recoil objective as follows. Let
((),z)) t (Q(s,, (002(''''))) d')], then:
3(1-7) 3 max min
+ (1- [1-=))Ean-t - (1-3) 1: -(s,a))) (138)
The last term of the ReCOIL objective cancels, simplifying to:
max min ((1-2) [Q 4).25 a. (0
Q(s,
(139)
Finally, rearranging terms we get:
max min 11
Q
(140)
Equivalently:
0.25 3 E E.3 [
max min
[aa - -0
(141)
Substituting the initial distribution the dataset distribution dg d (similar [Kostrikov al., 2019] and common practice in off-policy RL), and combining the first two terms which decrease Q values mixture of dataset states(offline and expert) under the current policy we obtain the intuitive definition of ReCOIL from the paper which indicates to increase Q at expert-state actions and decrease Q at dataset states under current policy. But this can lead to unbounded Q functions. Finally, Eq. 141 in the practical algorithm (Algorithm 1) implements max, by performing implicit maximization.
E TAKING DUAL-RL FROM OFFLINE TO ONLINE SETTING
Imitation Learning Problem (7) naturally extends to online II, as suboptimal data does not need to be static-it it can be the replay buffer during online training. The corresponding algorithms generalize as well, since their key component estimating the Q" function using off-policy data. It is worth noting that d is dynamically changing for online IL. In contrast, Eq. (6) cannot be extended to online IL. Garg et al. [2021] uses IQLearn in the online setting where they add additional regularization using bellman backups on d Our results suggest this be unprincipled (also pointed out by Al-Hafez et al. [2023]), as only expert data samples can be leveraged in this formulation.
Reinforcement Learning Again, all the above-discussed offline methods naturally extend online settings [Kostrikov et al., 2, Garg et al., 2023, Nakamoto et al., 2023], as their off-policy nature extends beyond the offline setup. Our analysis still holds, where the regularization distribution do becomes the visitation distribution of the replay buffer d It is worth noting that d is dynamically changing over the course of training.
35

---

Published as a conference paper at ICLR 2024
F IMPLEMENTATION AND EXPERIMENT DETAILS
F.1 OFFLINE IL: RECOIL ALGORITHM AND IMPLEMENTATION DETAILS
The algorithm for ReCOIL can be found in Algorithm 1. We base the ReCOIL implementation on the official implementation of XQL [Garg et al., 2023] and IQL [Kostrikov et al., 2021]. Our network architecture mimics theirs and uses the same data preprocessing techniques.
In our set of environments, keep the same hyper- hyper-parameters (except À parameter that intuitively controls the pessimism or the upper expectile of Q function) across tasks locomotion, adroit manipulation, and kitchen manipulation. For each environment, the values of are searched between 12.5,5,10]. We keep a constant batch size of 256 across all environments. For all tasks we average mean returns over 10 evaluation trajectories and 7 random seeds. We add Layer Normalization [Lei Ba et al., 2016] to the value networks for all environments, Full hyper-parae parameerrsssssss we used for experiments are given in Table 5. Although there might better alternatives for implicit maximization, we found the implicit maximizer from [Garg et al., 2023] to be especially performant in the imitation setting. For policy update, using Advantage weighted regression, we use the temperature to be 3 for MuJoCo locomotion environments and to be 0.5 for kitchen environments. The resembles prior work [Kostrikov et al., 2021].
Numerical Stability: In practice naive implementation of ReCOIL update for Qa in equation 11 suffers from numerical instability to learning Q Q-functions that are unbounded and since our objective maximizes Q-values at expert distribution, the Q values can be arbitrarily large without any grounding. The equation for Q-update from ReCOIL is given by: [()) ((Fag (=t))) [Qp(s, [)]] (((te (+.)) [Q a)]) + E s. (..0) [(TVi(s')) Qo(s, a))
(142)
To avoid this numerical instability we make small modification to the objective, that upper bounds Q-function regression target as follows:
( 3(E stale: (aaaiiiaaaaaaoo... a)] Eastr.i Qmae))]] [+]] E Ban " ( Qu(s,a))
(143)
Such modifications inspired by [Sikchi al., 2022a, Al-Hafez al., 2023] which have found that bounding targets can make learning significantly more stable.
Hyperparameters for proposed off-policy imitation learning method ReCOIL are shown in Table
Hyperparameter
Value
Policy learning rate
3e-4
Value learning rate
3e-4
f-divergence
x²
max-clip (loss clipping)
MLP layers
(256,256)
LR decay schedule
cosine
Qmaz
200
Table 5: Hyperparameters for ReCOIL.
F.2 OFFLINE IMITATION LEARNING EXPERIMENTS
Environments: For the offline imitation learning experiments focus 10 locomotion and manipulation environments from the MuloCo physics engine [Todorov et al., 2012]. These environments include Hopper, Walker2d, HalfCheetah, Ant, Kitchen, Pen, Door, Hammer, and Relocate. The MuloCo environments used this work are licensed under CC BY 4.0 and the datasets used from D4RL are also licensed under Apache 2.0.
Suboptimal Datasets: For the offline imitation learning task, we utilize offline datasets consisting of environment interactions from the D4RL framework [Fu et al., 2020]. Specifically, we construct suboptimal datasets following the composition approach introduced in SMODICE [Ma et al., 2022]. meadoo-ffww-eppert 'mediun+expert" and
The suboptimal datasets, denoted as 'random+expert', "medium=fewwexpert combine expert trajectories with low-quality trajectories obtained from the
36

---

Published as a conference paper at ICLR 2024
"random-v2" and "medium-v2" datasets, respectively. For locomotion tasks, the 'x+expert' dataset (where x is 'random' or 'medium') ) contains a mixture of some number of expert trajectories ( 200) and = million transitions from the "x" dataset. The 'xfew=expert' dataset is similar to x*expert." but with only 30 expert trajectories included. For manipulation environments we consider only 30 expert trajectories mixed with the complete 'x' dataset of transitions obtained from D4RL.
To enable imitation learning, an offline expert dataset is required. In this work, we use 1 expert trajectory obtained from the "expert-v2" dataset for each respective environment.
Expert Dataset:
Baselines: To benchmark and analyze the performance of our proposed methods for offiin imitation learning with suboptimal data, we consider four representative baselines in this work: SMODICE [Ma et al., 2022], RCE [Eysenbach et al., 2021]. ORIL [Zolna et al., 2020], and IQLearn [Garg et al., 2021]. We exclude DEMODICE [Kim al., 2022b] from the comparison, as SMODICE has been shown to be competitive [Ma et al., 2022]. SMODICE is an imitation learning method based on the dual framework, assuming restrictive coverage. ORIL adapts the generative adversarial imitation learning (GAIL) [Ho and Ermon, 2016] algorithm to the offline setting, employing offline RL algorithm for policy optimization. The RCE baseline combines RCE, an online example-based RL method proposed by Eysenbach et al. [2021]. J. RCE also uses recursive discriminator to test the proximity the policy visitations to successful examples. [Eysenbach et al., 2021]. with TD3-BC [Fujimoto and Gu, 2021]. Both ORIL and RCE utilize state-action based discriminator similar to SMODICE, and TD3-BC serves as the offline RI. algorithm. All the compared approaches only have access the expert state-action trajectory.
The open-source implementations of the baselines SMODICE, RCE, and ORIL provided by the authors [Ma et al., 2022] are employed in our experiments. We the hyperparameters provided by the authors, which are consistent with those used in the original SMODICE paper [Ma et al., 2022], for all the MuJoCo locomotion and manipulation environments.
F.3 CALCULATION OF f FOR f-DVL UNDER PRACTICAL CONSIDERATIONS
The main practical consideration when optimizing føt is that the function is not well defined in IR. We extend the domain of fo from a semi-closed interval [L oo) for certain e IR the set of real numbers R. Such extension and the behavior of f is described in the proof of proposition 3, but we will discuss it in more detail in this section.
Let start from Eq.47. Here, the domain of f is the same as f'11. Recall that f only admits a domain of R [0, v0). As a consequence, the function f'= has a limited domain [4, o0) for certain / E IR (To see this, first note that f' is non-decreasing as f is convex; further, since the domain of f is bounded from below, the range of f' is also bounded from below.). The behavior of f* 00) ~ R,, is then described as:
(f)(x). (x) C= otherwise
if (z)
Next, we extend the domain of f* to R. We use fo to denote the extended function. A natural choice is to take Eq 78 and extend it to TR. We also note that similar extensions can be found in prior work [Goldfeld, Picard-Weit and Guedj, 2022b).
x² divergence In practice, using the definition of f for x2 that still maintains the property of implicit maximization for Equation 76 and concisely write:
smoother surrogate objective
f* max(C, C, x² ///4 1/4+2)) 27).
(144)
Total Variation For the special case of total variation divergence, note that the convex conjugate f* (y) exists and is given by f (y)=y if y E [- 1/2 1 otherwise oc even if f'-1 does not exist. A concise proof can be found in Example 8.1 from [Goldfeld]. The reason is basically that a
closed-form solution for convex conjugate does not exist as equation 45 our paper no longer follows (f" is not invertible). We recover f² for total-variation divergence using the definition in
Eq 78 similar to x2, follows:
if 0.5 > x> 0
00, (x)
if 0.5
(145)
C = -f(), otherwise
In practice we use a smooth extension of ftt for TV divergence given by max(-f(0),
37

---

Published as a conference paper at ICLR 2024
Algorithm 3: FDVVI (Under Stochastic Dynamics)
1: Initialize Qo Vo, and Ty, temperature a, weight
Let D (8, r, 8') be data from (offline) or replay buffer (online)
3. for = 1..T' iterations do
Train Qo using ming L()):
[()) -' (r(s(()))))))))(())-)----)))))))
Train Vo using ming [0)
(1 X)E -- [Vo(+)] aD mmax (e(),a) Vo(s).0]]
300
(1 - X)B a [V[(s)] 2o max (u((. - V((s)) 0.5(Q.(s.))) (660)))))
[6())).0) x²
(1 - X/E [Vo(e)] AE, cxp a) Vo(a))] -]]
RKL/XLL
Update Ty via max M(1)):
M(1 = E., Ea (l log (3|a)]
(148)
end for
Practical Choice of C' While Eq 78 suggests setting C' dependent on the corresponding f-divergence,
found single value of C' 0 to be robust choice across our experiments. A comparison between using f = { f(0) if x < -4, ///4////// + otherwise]) and f = max(0, x² /4- x)) for x² divergence can be found in Table below. Choosing C = 0 instead of - f(0) led to performance improvements.
Dataset
F-DVL maxx0. x)) f-DVL (TV) /-DVL ft 100) -4.4 otherwise))
Balicheetali mallchestal-mccuun22 mediumm-2
47.7
11.5
16.19
medium-22
63.0
64.1
78.66
walker medium x2
80.0
81.5
76.85
halfcheetahh medium replay-v2
42.91
hopper medium meclim--eplay--2?
90.7
98.0
97.73
walker 2dd-ediiu---epaae replay-12
52.1
68.7
73,5
hulfchhesah-mmolinnee mefi-mm--i-ieett.....
89.3
91.2
80.1
hogper- mediim----mm-e hopeer-medium-ssge expert-V2
105.8
93.3
94.5
109.5
kischen complete-v0
67.3
65.71
67.14
kiichen-puse kiichen-purial-O purial--0
58.8
70.0
48.2
kischen-mixal-v0
53,75
52,5
52.4
Table The normalized return of offline RL methods on D4RL tasks. Shows comparison of setting the cutoff' constant for f to C = (0( vs C =
F.4 ONLINE AND OFFLINE RL: f-DVL ALGORITHM AND IMPLEMENTATION DETAILS
Rewriting of dual-V using temperature parameter À instead of a: We found rewriting dualV using temperature parameter instead of a to be particularly useful in reducing the number of hyperparameters to tune in order obtain strong learning performance. We replace the temperature parameter from a to X. Notice that our initial dual-V formulation used the temperature parameter follows:
dual-V min (1 - ao
(+)) (V
+ aB(a,))-ao [rp (LTV V(s))
(146)
V
The temperature parameter a captures the tradeoff between the first term which seeks to minimize V the second term which seeks to maximize V and set it to the maximum value possible when taking various actions from that state onwards. Depending on different ʃ generator functions we would require tuning this parameter as has a non-linear dependence on the entire optimization problem through the function f. Instead we consider simpler objective, that we observe to empirically reduce hyperparameter tuning significantly by trading off linear between the first term and the second term using parameter A. This modification is used in all of our experiments for RL and II.
dual-V (rewritten) min (1 [V(s)] ao [f," (ITV(s, V(s))))) (147)
Algorithm F.4 gives the algorithm for f-DVL. This section provides additional offline RL experimences along with complete hyper-parameter and implementation details. Figure shows learning curves for all the emvironments. f-DVL exhibits fast convergence as XQL but avoids the numerical instability of XQL with one hyperparameter across each set of environments. We base our implementation of f f-DVL off the official implementation of XQL [Garg al., 2023] and IQL from
Offline RL:
38

---

Published as a conference paper at ICLR 2024
Eov
Lambda X
Batch Size
Y_updates
hulfchertah medium- v2
0.7
256
hogper-medium-22 medium-v2
0.7
256
walker2d-mediue
256
hulfcheetah-meee 2eddiuu---pllyy--222
0.7
256
hopper- pnper-mediom-rpphin
walker2d-medium Sedd-medium--a -idud--pppla--2222
0.7
256
hullcheetah medium expert-v2
0.7
256
hopper-medium-er medium expert-v2
0.7
256
walker2d-medium meddim--miium-xxpe--2222
0.7
256
antmaze-ume mumaz---aaz--0
256
antmaae- umuze diverse-vU
0.8
256
antmaze- modium--paa--0
256
medium- medium-diverse--u
0.8
256
antmaze large play vO
0.8
256
antmare- natmare-largeee lige------v-------- diverse-v0
256
hitbcn--oomplete-v
0.8
256
kitchen mixed-vi vO
0.8
256
pen-
lummer huuerr-uun-
0.8
256
door human vO
0.8
256
relocate-human-v0
256
pen-cloncd-vO
0.8
256
hummer--loned--ee
doar human vO
0.8
256
telecate-human-v0
Table Offline Hyperparameters used for f-DVI. Lambda the value that controls the strength of implicit maximizer. V-updates gives the number of value updates per Q updates.
Kostrikov et al. [2021]. Our network architecture mimics theirs and uses the same data preprocessing techniques.
In our set of environments, we keep the same hyper- parammerrettrr across sets of tasks locomotion, adroit manipulation, kitchen-manipulation and antmaze. Contrary to XQI, we find no need to use tricks like gradient clipping to stabilize learning. For each set of environment, the values of were tuned via hyper-parameter sweeps over fixed set of values [0.65, 0.7, 0.75, 0.8, 0.9]. We keep a constant batch size 256 across all environments. For MuJoCo locomotion tasks we average mean returns over 10 evaluation trajectories and 7 random seeds. For the AntMaze tasks, we average over 1000 evaluation trajectories. We add Layer Normalization [Lei Ba et al., 2016] to the value the
networks for all environments. For policy update, using Advantage weighted regression, temperature a to be 3 for MuloCo locomotion environments and to be 0.5 for kitchen environments. The resembles prior work [Kostrikov al., 2021]. Full hyper-parameten used for experiments are given in Table 7.
F.5 ONLINE RL EXPERIMENTS WITH fDVV
Online RL: We base the implementation of SAC on pytorch sac and XQL [Garg al., 2023], Like in offline experiments, hyper- parameters were left as default except for X, which we tuned between [0.6, 0.7, 0.8] and found a single value to work best across all environments. This was in contrast to XQL's finding which required environment different hyperparameter. Also, opposed to XQL we required no clipping of the loss function. We test our method on 7 random seeds for each environment.
Compute We ran all our experiments on machine with AMD EPYC 7J13 64-Core Processor and NVIDIA A100 with a GPU memory consumption of <1000 MB per experiment. Our offline RL and IL experiments for locomotion tasks take 10-20 min and the online IL experiments took around 5-6 hours for million timesteps.
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 WHY DUAL-RL METHODS ARE A BETTER ALTERNATIVE TO TRADITIONAL OFF-POLICY ALGORITHMS
Our experimental evaluation aims to illustrate the benefits of the dual RL framework and analyze our proposed method for off off-policy imitation learning. In the RL setting, we first present a case study on the failure of ADP-based methods like SAC [Haamooa et al., 2018] to make the most when bootstrapped with additional (helpful) data. This setting is what motivates the use of off-policy
39

---

Published as a conference paper at ICLR 2024
Hyperparameter
Value
Policy updates pol
Policy learning rate
3e-4
Value learning rate
3e-4
MLP layers
(256,256)
LR decay schedule
cosine
Table 8: Common hyperparameters for for f-VVL.
Hyperparameter
Value
Batch Size
1024
Learning Rate
0.0001
Critic Freq
1
Actor Freq
1
Actor and Critic Arch
1024, 1024
Buffer Size
1,000,000
Actor Noise
Auto-tuned
Target Noise
Table 9: Hyperparamettess for SAC.
algorithms in the first place and is invaluable in domains like robotics Uchendu et al. [2022], Nair et al. [2020]. Our results validate the benefit of utilizing the dual RI. framework for off-policy learning.
The limitations of classical off-policy algorithms: In this section, we test the sensitivity of an ADP method (SAC [Haarnoja al., 2018)) vs dual-RL methods in the case when we initialize the replay buffer of both styles of off-policy algorithms with expert or human demonstrated trajectories. At the beginning of training, each learning agent is provided with expert or human-deroonstrated trajectories for completing the task. We add 1000 transitions from this dataset to the replay buffer for the off- aff-policy algorithm to bootstrap from. SAC is able to leverage this helpful data and shows improved performance in Hopper-v2, where the action dimension is small. As the action dimension increases, the instability of SAC becomes more apparent (see SAC+off policy data and SACI) plots in Figure 7). We hypothesize that this failure in the online RL setting primarily due to the training instabilities caused by TD-backups resulting in overestimation in regions where the agent's 's current policy does visit. In Figure 8, we observe that overestimation indeed happens in environments with larger action dimensions and these overestimations take longer to get corrected and in the process destabilize the training.
Figure shows that the dual-RL method (AlgaeDICE) is able to leverage off- off-policy data to increase learning performance without any signs of destabilization. This can be attributed to the distribution correction estimation property of dual RL methods which updates the current policy using the corrected on-policy policy visitation [Nachum et al., 2019]. Note, that we set the temperature a to low value (0.001) to disentangle the effect of pessimism which is an alternate way to avoid overestimation.
G.2 TRAINING CURVES FOR RECOIL ON MuJoCo TASKS
We show learning curves for ReCOIL in Figure for locomotion tasks and Figure 10 for manipulation tasks below. ReCOIL training curves are reasonably stable while also being performant, especially in the manipulation setting where other methods completely fail.
G.3 DOES RECOIL ALLOW FOR BETTER ESTIMATION OF AGENT VISITATION DISTRIBUTION?
In this section, we consider the experiment of visitation estimation for a prespecified policy given expert data and suboptimal/replay data. Our experiments are tabular, so we can have an accurate estimate of the visitation of the policy by running rollouts in the MDP. We call this estimate the ground-truth policy visitation. We will estimate the accuracy of dual-RI. methods to estimate visitation density by measuing MSE error against the ground truth agent/policy visitation. The property of Dual-RL methods to implicitly estimate density ratios (eg. Eq 49) has been studied
40

---

Published as a conference paper at ICLR 2024
Hopper-v2 [A:3]
Walker2d-v2 [A:6]
Pen-vO (Expert) [A:24]
Average Reward
4000
6000
4000
4000
2000
1030
166
146
Hammer-vO (Expert) [A:26]
Pen-vO (Human) [A:24]
Hammer-v0 (Human) [A:26]
20000
Average Reward
15000
4030
15000
10000
10000
2030
2000
1030
Timesteps
1c6
Timesteps
128
Timesteps dualQ RL (AlgaeDICE)
SAC (no additional data)
SAC off-policy Data
SACID
Figure 7: Despite the promise of off policy methods, current methods based on ADP such as SAC fail when the dimension of action space, denoted by A, increases even when helpful data is added to their replay buffer. On other hand, dual-Q methods are able to leverage off-policy data to increase their learning performance
Pen-vo (Expert)
Hammer-v0 (Expert)
Average Value
20
8.0
030
1:00
145
105
SAC (no additional data)
SAC off-policy Data
SACfD
dualQ (AlgaeDICE)
Figure 8: SAC and SACID suffer from overestimation when off- off-policy data is added the replay buffer. We hypothesize this to cause instabilities during training while dualQ has no overestimation.
before [Nachum and Dai, 2020]. We explain below how to extract visitation density ratios for any policy π with ReCOIL. First, we discuss the setup for the experiment.
We consider two settings in our experiments: (1) a 2-timestep MDP where agent states from state so and transitions to one of the states {81, 82, $3, 84, $5} (from left to right in Figure la) which are absorbing. In this setting, the replay buffer perfectly covers the unknown ground truth agent visitation. (2) 2-D gridworld (Figure 11) where the agent can move cardinally and the replay buffer distribution does not cover the unknown ground truth agent visitation. In both environments we have access to policy at training time the task is to estimate this policy's 's visitation using the offline dataset of expert and suboptimal quality. Both figures also demonstrate the ground truth policy visitation that is used to compute the mean-sq evaluation loss and test the quality of our predictions.
We consider the proposed ReCOIL method and investigate its ability to estimate distribution ratios correctly. We consider the inner optimization for ReCOIL-B Q:
min B( - ((1-7)EEaaee (onnntt) @(+.a)
f* Σ p(s'ls ((s(ls..))(dd1)))0 - -
e.a min
- -(1-8) (1-3)
1-B)) En(dls, - ((s,a)
(149)
41

---

Published as a conference paper at ICLR 2024
Hopper-22 random-axpart
HalCheelsh-122 random-expert
Walker2--22 randome export
Ant-2 randommospert
Average Reward
70-
randomm03 expert
randome50 expert
rendomt expert
random expeett
Average Reward
medium sopert
medium mesportt
medikam rexpert
Average Reward
16
114
2.5
Timestops
105
Timestops
Timestaps
Timestops
RCE
SMODICE
ReCOIL
Figure Learning curves ReCOIL showing that outperforms baselines in setting learning imitate from diverse offline data. The results are averaged over 7 seeds
Pen-vB hument expert
DoaryD
Hammer-vo humantexpert
Relocate-vo human rexpert
humannexpert
Average Reward
23
doned rexpert
cloned rexpert
doned eexpert
cloned rexpert
Average Reward
Timestops
105
Timestaps
100
Timestops
Timestops
RCE
ORIL
SMODICE
ReCOIL
Figure 10: Learning curves for ReCOIL showing that it outperforms baselines the setting of learning imitate from diverse offline data. The results are averaged over seeds
The following holds for the inner optimization for ReCOTL-Q when Q is optimized:
3dd (s, a) + (8, fr pp(Esssss (aaa)))))(('')(((((((()()))))) - Q(s,)))= (s. + (1 ())-))
(150)
Thus, given the visitation distribution of the replay buffer R expert d and the policy the inner optimization implicitly learns the distribution ratio in Eq 150, allowing us to infer agent visitation d"
Our results (Figure la and 11) demonstrate that in the perfect coverage setting, ReCOTL is able infer the agent policy visitation perfectly, and in the case of imperfect coverage is able to significantly outperform other methods (IQLeamn and SMODICE). Note that we modify SMODICE with Q objective instead of objective to incorporate state-action expert data rather than relying on state-only expert data. IQLeam does not leverage replay data information, relying only on expert d (s.a) arising from its coverage data to infer agent visitation. SMODICE's reward function log (F(aa))
42

---

Published as a conference paper at ICLR 2024
assumption is ill-defined in parts of state space where the expert has support leading to poor
downstream density ratio estimation.
Our results on the 2-D gridworld environment that demonstrate the failures of a method that either do not utilize all available suboptimal data (IQ-Leam) or relies on a coverage assumption (SMODICE). We saw that ReCCIL is able to perfectly infer the agent's s visitation when the replay buffer covers agent ground truth visitation perfectly (Fig la) and here we see that ReCOIL is able to outperform baselines when the replay buffer has imperfect coverage over the agent's ' ground truth visitation (Fig 11). In this task, the agent starts at (0,0) which is the top-left comer. The agent can only move in cardinal directions with deterministic dynamics. The agent has access two sources of off- pfllic data expert visitation and replay visitation. The problem is to estimate the agent's visitation distribution given access to the agent's policy using all the available transition data. IQLearn and SMODICE predict an agent's 's visitation that wildly differs from Agent's ground truth visitation distribution. While ReCOIL is not perfect as the coverage of the offline data is limited, we can estimate some visitation which is qualitatively very similar to the agent's 's ground truth visitation. Imperfect Coverage
Expert visitation
Repley visitation
Agent ground truth visitation
Estimated agent visitation with expert data only (IQUeam))
Estimated Agent Visitation with coverage assumption (SMODICE)
Estimated agent visitation (ReCOIL)
Replay buffer consists of data that visits near the initial state (0,0), a setting commonly observed when training RL agents. We estimate the agent's policy visitation and observe ReCOIL to outperform both methods which rely expert data only use the replay data with coverage assumption
Figure 11:
G.4 RECOIL: QUALITATIVE COMPARISON WITH A BASELINE
In Figure 12, we investigate qualitatively why other baselines fail where ReCOTL succeeds in high-dimensional tasks. A surprising finding is that the baseline we consider 'SMODICE" almost learns imitate. It follows nearly the same actions as expert but makes small mistakes along the way- - eg. gripping the hammer too loose' or "picking up the ball at a slightly wrong location'. SMODICE is unable to recover from such mistakes and ends up having low performance. ReCOIL, on the other hand, learns a performant task-solving policy from the same data.
G.5 EVALUATION OF f-DVL FOR ONLINE RL
Fig 13 shows that f-DVL is competitive to performant off-policy RL methods in the online RL benchmarks.
G.6 TRAINING CURVES FOR f-DVL ON MuJoCo TASKS (OFFLINE)
Figure 14 shows the learning curves during training for f-DVL. f-DVL able to leverage low-order conjugate f f-divergences to give offline RL algorithms that more stable compared to XQL. XQL frequently crashes in the antmaze environment.
G.7 fDVVL: COMPLETE OFFLINE RL RESULTS
Table 10 and Table 11 show complete results for benchmarking f-DVL on MuJoCo D4RL environments. Here we also show the author-reported results for XQL and the reproduced results (XQL(r)) using the metric of taking the average of the last iterate performance across seeds.
43

---

Published as a conference paper at ICLR 2024
SMODICE
ReCOTL
SMODICE
ReCOIL.
Figure 12: Errors compound in imitation learning and recovery is of crucial importance. Figure demonstrate how SMODICE 'almost' imitates, ligures out roughly what actions to but does realise once it made a mistake. In Hammer environment, it grips the hammer too loose causing to get thrown away and for relocate picks up just beside the ball missing the original task the expert intended to solve.
Walce-Run
Hoppwe-Hop
Queetsppe-RRun
Chretsh-llun
Averge Reward
3/C
XOL
6n2 DVL
TV TV.
Figure 13: Online RL: f-DVL is competitive to SAC and XQL, particularly for Hopper-Hop and Quadruped-Rus tasks.
G.8 SENSITIVITY OF f-DVL (OFFLINE) WITH VARYING À ON MUJoCo TASKS
We ablate the temperature parameter, À for offline RL experiments using f-DVL in Figure 16 and Figure 15. The temperature À controls the strength of KL penalization between the learned policy and the dataset behavior policy, and small À is beneficial for datasets with lots of random noisy actions. In contrast, high À favors more expert-like datasets. We observe that significantly less hyperparameter tuning is required compared XQL single temperature value works well across broad range of experiments.
G.9 SENSITIVITY OF f-DVL (ONLINE) WITH VARYING À ON MuJoCo TASKS
We ablate the temperature parameter for online RL experiments using f-DVL Figure 18 (chi-square) and Figure 17 (TV). We observe that significantly less hyperparameter tuning is required compared to XQL as a single temperature value works well across a broad range of experiments.
G.10 RECOVERING REWARD FUNCTIONS FROM RECOIL
We study the quality of reward functions recovered from ReCOIL using the hopper-medium-expee and Walker2d-ng
Walker2d-medium medium-ennt expert datasets and the setup described in Section 7.1. For all trajectories
in this dataset, we calculate the ground truth return (sum of rewards) and the predicted cumulative
44

---

Published as a conference paper at ICLR 2024
Average Reward
antmaze-umaze-vV
entiaaz-uuazze-dvvrre-o0
antmmze-medduumppaa--o
100
75
50
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
Average Reward
ortmaaz-mmdduumdiivrree--e
untmaze-large-play--a
antmazze-argg-dilerseww
76
26
20
20
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
Average Reward
kitchen-complete-v0
kitchen-partial-vO
kitchen-mixed-vO
100
50
0.0
02
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
1e8
Timesteps
1e6
Timesteps
1e6
XOL
ADVL (crk-square)
ADMM (lota variston)
Figure 14: Learning curves for f-DVL showing that it is able to leverage low-order conjugate f f-divergences to give offline RL algorithms that more stable compared XQL. The results averaged over 7 seeds
Table 10: Averaged normalized reproduced results with author's implementation. We do not highlight XQL due to incorrect evaluation strategy used the work. Standard deviations our proposed offline RL methods can be found below.
MuJoCo locomotion and Ant Maze tasks. XQL(r) denotes the
Dataset
BC
10RBC
DT
TD3+BC
CQL
IQL
XQL
XQL())
f-DVL x²
f-DVL TV
hullcheetah medium-v2
42.6
42.5
42.6
48.3
44.0
47.4
47.7
47.4
0..7
0.25
hopper medium-v2 v2 valker2d-mediume
52.9
56.9
67.6
59.3
58.5
66.3
71.1
68.5
63.0
64.1-133.46
75.3
75.0
74.0
83.7
72.5
78.3
81.5
81.4
80.0=6.75
51.3434477
36.6
44.6
45.5
44.2
44.8
44.1
42.9=1.79
44.7+:0.44
hopper-meed medium replay-v2
18.1
75.9
82.7
60.9
95.0
94.7
97.3
95.1
90.7
98.0144.22
walker2d medium replay
73.9
75.9
58.0
68.711..20
halfcheetah- mediom-expert.....
55.2
92.9
86.8
90.7
91.6
86,7
89.8
90.8
89.3+22.2
91.2-+2.27
hopper medium expert
52.5
110.9
107.6
98.0
105.4
91.5
107.1
94.0
105.815.79
93.3 14.00
walker24 2d 2e--2dd--d-mm--m medium-experttv
107.5
109.0
108.1
110.1
108.8
109.6
110.1
110.1
110.1+0.29
109.6++
antmaze umaze v0)
54.6
62.8
59.2
78.6
74.0
87.5
87.2
47.7
63.71.5.00
87.7
AntMaze
antmaze
98.9
mmmaze-mediun-plaas
0.0
5.4
0.0
10.6
61.2
71.2
73.5
31.2
56.7+13.82
71.0+
suumazz-mediu--diveee
0.0
9.8
0.0
53.7
70.0
67.8
0.0
48.2
60.247..9
antmave-large-play
0.0
0.0
15.8
39.6
10.7
36.0=5.82
41.7+9945
autmaz-largo-divrss--d
0.0
6.0
0.0
0.0
14.9
47.5
47.3
31.28
44.5=7.66
39.3++1..4
Frankan
kitchen-complete-w
62.5
72.5
56.7
67.5
61.377.95
kitchen-partiall-0)
38.0
49.8
46.3
73.8
48.6
58.8=9.00
70.0 1.82
kilcban-mixed-vv
51.5
51.0
51.0
54.6
40.4
53.75:55.32
$2.5
reward using ReCOIL. The scatter plot in figure 19 shows the correlation between predicted rewards. We that ReCOIL is an IRL method and suffers from the reward ambiguity problems rest of the IRL methods- we can only expect reward function that induces an optimal policy whose visitation is close an expert and cannot guarantee that recover the expert's exact reward function. To test the quality of rewards functions output by IRL methods, Pearson correlation is the accurate metric and metrics like EPIC [Gleave et al., 2020] might be used instead.
45

---

Published as a conference paper at ICLR 2024
Table 11: Evaluation Adroit tasks from D4RL.XQL-C (r) denotes the reproduced results with author's implementation.
Dataset
BC
BRAC-p
BEAR
Onestep RL
CQL
IQL
XQL
XQL()
/-DVL (x2)
f f-DVL (TV)
pen bumaner-uum--vO
85.5
1.2
0.3
0.3
4.4
1.4
1.4
1.8
door human vO
0.3
00.
9.9
4.3
11.5
6.63
6.77
relogate-human-x0
-0.3
-0.3
0.2
0.1
0.17
0.12
pon-eloned-v0
37
1.6
26.5
60.0
39.2
37.3
38.6
25.25
36.1
38.1
kemmee--cooued-v0
door cloned-vv vO
0.0
-0.1
00.1
0.4
0.4
1.6
5.9
0.69
0.45
0.87
relocate-cloned
-0.3
Average Reward
hopper-mmdium-expprrtt
haitcheetah--mdiim--eegrrrt---
walker2d-medium-expeee
50
50
0.0
0.4
0.6
1.0
0.2
0.4
0.8
0.2
0.6
0.8
1.0
1e8
Average Reward
hopper-medium--2
hallchhestah-medium-22
walker2d-medluum-22
80
40
75
50
40
20
25
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
166
1e5
166
Average Reward
antmaze-umazze-v0
antmaze-uuaaze-divesseeve
antmaze-meddiumppaayyy
80
60
80
40
40
26
20
20
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1e6
1e8
166
Average Reward
kilchen-complete-v00
kitchen-partial-vo
kitchen-mixed-v0
60
60
40
40
40
20
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Timesteps
1e6
Timesteps
1e8
Timesteps
166
TVAVLL (A=0.7)
(A-0.75)
TVDVL (F0.8
TVADV (A=0.85)
Figure 15: Offline RI.: Ablating the temperature parameter for f-VVI. (Total variation). The plot shows the effect of temperature parameters on learning performance.
3.79
Learned reward
1.2
60-
1.5
100
400
1256
1959
7008
196
900
W30
2000
3000
GT reward
GT reward
46
Hopper Pearson correlation: 0.98
Walker2d
Pearson correlation: 0.92

---

Published as a conference paper at ICLR 2024
Average Reward
hopper-meddum-exxper--
helicheeta--mediim-eeeenrz--
walleez2immediium-exprrtz/
100
50
50
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1e6
166
Average Reward
hopper-medium-v2
hallchheetah-meiuumv22
walker22-medium-22
50
75
40
50
40
20
25
10
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1e6
1e6
1e6
Average Reward
antmaze-umazee-v0
antmaze-umaaze-divereewvo
antmaze-meddum-ppayy--0
80
75
60
50
40
26
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
166
1e6
166
Average Reward
kitchen-complete-v0
kitchen-partial-v0
kitchen-mixed-vD
100
80
60
40
40
20
20
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.8
0.0
0.2
0.6
1.0
Timesteps
1e6
Timesteps
1e6
Timesteps
1e6
(A=0.7)
CHICVL (A=0.76)
(4:0.56)
Figure 16: Offline RL: Ablating the temperature parameter for f-DVL (Chi-square). The plot shows the effect of temperature parameters on learning performance.
Walker Warker-Rnn
Hopper-Hop
Quadrrued-FFun
Cheelsh-Rui Run
Average Reward
80
Avorage
190
02
10
0.5
Timeslaps
TV-VL (Ar0.85)
TV- (2-07)
TV-VL (A-075)
TV-0VL(1-00)
Figure 17: Online RL: Ablating the temperature parameter for f-DVL (Total variation). The plot shows the effect of temperature parameters learning performance.
47

---

Published as a conference paper at ICLR 2024
WalerrRun
HopperHHop
Qusdruped-Run
Cheelali-Rn
Average Reward
330
Timesteps
TVDVL (A-0.55)
DV (1r0.))
DV-DVL (V-077775
TV-DVL(A-0.))
Figure 18: Online RL: Ablating the temperature parameter for f-DVL (Chi-square). The plot shows the effect
temperature parameters on learning performance.
48


Reviews:
[{"review_id": "xqhzQb8KPW", "reviewer": "Reviewer_NyfP", "summary": "This paper proposes a unified dual RL framework that connects several recent offline RL and IL methods. New algorithms called ReCOIL (for IL) and f-DVL (for RL) are presented that aim to address limitations of prior approaches. Experiments across locomotion, manipulation and navigation tasks generally demonstrate improved or comparable performance to baselines.", "questions": "Overall, I am moderately positive about this submission. The theoretical framework is clean and impactful. The empirical results are reasonably strong but could use tighter presentation and analysis. \n\nI have the following specific questions:\n\n- In Eq. 10, is the reward assumed to be zero in the Bellman consistency term?\n- Can you expand on how ReCOIL estimates the policy visitation and reward for the results in Sec 7.1? Also clarify if Fig 1 and Fig 10 use OPOLO or SMODICE for comparison.\n- What do the distributions D and d^O refer to in Eq. 12 and 13? Are they both d_{mix}^{E,S}?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The theoretical dual RL formulation provides a common lens to view recent offline RL/IL methods. This is a useful conceptual contribution.\n- The methods are evaluated extensively on a diverse set of tasks using standardized benchmarks. Implementation details are clearly described.\n- For offline IL, ReCOIL relaxes assumptions like coverage and avoids instability of density ratio estimation. The results showcase strong performance on imitation tasks.\n- The modifications in f-DVL seem to improve training stability compared to XQL, a prior state-of-the-art in offline RL.", "weaknesses": "- While f-DVL outperforms XQL, its gains over other offline RL methods like IQL are marginal. The gains are not as significant as claimed over the full spectrum of baselines.\n- The presentation of empirical results could be improved by using standardized metrics, showing confidence intervals, and increasing clarity around performance highlights.\n- There are open questions around design choices, estimation procedures, and other technical details that warrant clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1m8ugE1drc", "reviewer": "Reviewer_7FxJ", "summary": "The paper presents a new approach in offline IL, adopting the f-MAX objective function instead of the commonly utilized SMODICE objective. Unlike f-MAX, which employs GAIL for optimization, this work leverages the DICE technique to derive a dual form and subsequently applies an IQL-style algorithm for resolution. Additionally, the framework exhibits versatility, with potential applications in both the online and offline RL settings.", "questions": "Your work is commendable, but improvements in presentation and accuracy are needed. My specific questions and suggestions are as follows:\n\n1. XQL's instability and heavy reliance on hyperparameters are noticeable, with subpar performance in both online and offline RL under unsuitable conditions. As an alternative example, it might be more suitable to reference a more general and effective implicit policy improvement algorithm, like IQL[1]. I suggest you use a more general and effective implicit policy improvement algorithm, IQL, as an example.\n\n2. In the last paragraph of Section 4.1, it is not suitable to say, \"This insight also allows us to cast IL method OPOLO .....\" as the objective function here originates from SMODICE. Referencing \"this objective function\" or similar expressions would be more appropriate.\n\n3. In the first paragraph of Section 4.2, it is strange to use \"Unfortunately\" just because XQL is not the policy regularization style. \n\n4. Figure 1 (a) is quite hard to follow. Please refine the caption to explain what the floats and the colors mean.\n\n5. The first paragraph of Section 7.1 lacks supportive evidence for this claim. Incorporating illustrative figures and evaluations would substantiate this statement.\n\n6. On page 36, typo in \"We base the implementation of SAC off pytorch_sac ...\"\n\n7. On page 37, I disagree with your claim that \"Our experiments with the popular off-policy method SAC [Haarnoja et al., 2018] reveal its brittleness to off-policy data\". SAC is not brittle to off-policy data. Filling the replay buffer arbitrarily with expert data at the beginning is not fair for an ADP-style algorithm. Please rewrite this paragraph.\n\n8. On page 40, the statement \"f-DVL leads to noticeable improvements even in the online RL benchmarks\" appears overstated:\n\n   * The baselines are too few.\n\n   * The performance improvement is not noticeable in 75% tasks you conducted experiments in.\n\n   If you would like to claim noticeable,  please include more baselines (PPO[2], TD3[3], AlgeaDICE[4], and some other more recent algorithms, e.g., RRS[5], BAC[6], REDQ[7]) and more benchmark tasks. And please broaden the discussion of related works in the online RL setting beyond SAC.\n\n9. Some crucial experimental outcomes, currently in Appendix H, should be integrated into the main paper for better accessibility and coherence, especially those results referenced and analyzed within the main paper.\n\n10. Better not to claim \"New\" in the title directly. Just saying \"A unified framework\" is ok.\n\nBy addressing these points, your work could gain greater clarity and impact. And then, I would like to raise the score.\n\n[1] Kostrikov I, Nair A, Levine S. Offline Reinforcement Learning with Implicit Q-Learning[C]//International Conference on Learning Representations. 2021.\n\n[2] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n\n[3] S Fujimoto, H van Hoof, and D Meger. Addressing function approximation error in actor-critic methods. Proceedings of Machine Learning Research, 80:1587–1596, 2018.\n\n[4] Nachum O, Dai B, Kostrikov I, et al. Algaedice: Policy gradient from arbitrary experience[J]. arXiv preprint arXiv:1912.02074, 2019.\n\n[5] Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Optimistic curiosity exploration and conservative exploitation with linear reward shaping. In Advances in Neural Information Processing Systems, 2022.\n\n[6] Ji T, Luo Y, Sun F, et al. Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic[J]. arXiv preprint arXiv:2306.02865, 2023.\n\n[7] Chen X, Wang C, Zhou Z, et al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model[C]//International Conference on Learning Representations. 2021.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a thorough and insightful analysis of recent developments within the dual-RL paradigm.\n2. The proposed framework demonstrates adaptability across IL, offline RL, and online RL settings.\n3. By eliminating the discriminator in offline IL, the paper potentially enhances stability and reliability in this domain.", "weaknesses": "1. The paper's structure appears ill-suited for a conference format, dedicating 3.5 pages to prior knowledge while relegating significant analyses and experimental results (pertaining to online and offline RL) to the Appendix.\n2. There are instances of overclaims and problematic expressions that require rectification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ay1PjMg1lD", "reviewer": "Reviewer_JKJA", "summary": "The paper proposes a framework called dual RL based on the LP formulation of RL. The author shows dual RL can be used to derive offline RL and offline IL algorithms, providing a unification of existing approaches. Based on this insight, they propose new offline IL and RL algorithms (ReCOIL and f-DVL), which remove existing assumptions. These algorithms are tested in the D4RL benchmarks empirically and achieve competent performance compared with other baselines.", "questions": "1. Proposition 4: The authors write that CQL can be cast as a dual-Q problem with a right choice of f-divergence (Pearson). However, in the proof of Proposition 4 in Appendix D.1, Eq (70) actually corresponds to the objective of ATAC [Cheng et al. 2022], not CQL. See Eq (1) in [Cheng et al. 2022], which is exactly the same Eq (70) here. To my understanding, the main difference between CQL and ATAC is that ATAC takes a maxmin formulation, whereas CQL doesn't, though they have a similar objective for learning Q. The dual-Q formulation here is a maxmin problem. The authors should update Proposition 4 to make the connection to ATAC, which would be more appropriate and precise.\n\n2. Bug in the proof of Theorem 1. There is a minor bug in the proof of Theorem 1. In the block starting \"Imitation from Arbitrary data (dualQ)\" after Eq (108), max_d is moved inside min_Q, whereas max_d is outside min_Q in Eq (108). So Eq (109) is not just a summary, and that \"we summarize the result of the derivation so far\" can be misleading. But this can be easily fixed though, by applying strong duality before moving forward. Another minor suggestion: I think it would be good to write out the proof Eq (112) for completeness.\n\n3. Validity Eq (10). In the proof, the author assume d_0 = d_E. I think that this is a fairly strong assumption, as it implies that the initial state at the test time would be drawn from the same distribution of the expert. This is rarely the case in practice, and in my experience it's not a common assumption in offline or off-policy RL. The authors should highlight this assumption in the main paper, or the authors should present Eq (140) as the main result. A minor bug in the proof is the derivation misses a 1/4 factor in front of the square loss.\n\n4. Bug in Proof of Proposition 3. There're a few issues in the derivation, and one is major. \n    a. f', (f')^-1 are not strictly increasing. it's only non-decreasing. e.g. see f(t) = | t-1|. \n    b. why is f*(x) = -f(0) for x<0? \n    c. why is f(0+)>0?\n    d. You write (f')^{-1}(t) >0 when t>0 and 0 otherwise. What does \"0 otherwise\" mean?\n    e. (Major issue). From Eq (79) to Eq (80), it doesn't use \"(f')^{-1}(t) >0 when t>0\" (though the paper writes so). It uses the other way (f')^{-1}(t) >0  --> t>0 and likewise (f')^{-1}(t) <0  --> t<0. But they are not true. See e.g. f(t) = | t-1| again. This error step prevents me from validating the remaining part. \n\n5. The experiment design of offline IL can be a bit confusing. It says the agent is given 1 expert demonstration. But the suboptimal transition dataset actually contains 200 expert demonstrations, though I understand they're not marked as expert demonstrations to the algorithm. I think a more convincing experiment is to completely remove the expert demonstrations in the suboptimal data. Table 2 writes 1000 expert transitions. Is that the same as one expert trajectory?\n\n6. I don't know how to interpret Fig 1. Can you explain more?\n\nMinor: \n\n1. This sentence is confusing to read. \"Unfortunately, one of the most successful classes of offline RL methods .. has evaded connections to regularized policy optimization. Proposition 2 shows, perhaps surprisingly, that XQL can be cast as a dual of regularized policy learning, concretely as a dual-V problem.\"\n\n2. I don't understand the connection between the sentence \"To prevent extrapolation error in the offline setting, we rely on an implicit maximizer [Garg et al., 2023] that estimates the maximum over the Q-function conservatively with in-distribution samples.\" and Eq (12)\n\nReference: \nCheng, Ching-An, et al. \"Adversarially trained actor critic for offline reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed dual RL framework based on the duality of a regularized RL problem is quite clean. The authors provide good background information and detailed derivations, which makes the paper self-contained and easy to follow. The dual RL framework makes some connections of existing works and suggests potential generalization (which the authors adopt) to design new algorithms. The empirical results are promising. Overall, this is an insightful paper.", "weaknesses": "There are some technical parts, which I think are not fully correct (see below). If they're indeed wrong, then the contribution of this paper would be compromised. The current writing is also quite dense as well, though the details are provided in the appendix. It would be better if the main paper is more readable. Lastly, in some sense, this paper lacks novelty. The unifying perspective and connection are interesting, but at the end the resulting algorithms can be viewed as minor tweaks from existing ones. Nonetheless, I think this last point on novelty is minor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TH5rp7zdRI", "reviewer": "Reviewer_aLZ7", "summary": "This paper unifies a number of recent methods in offline RL and IL through the lens of regularized policy learning and Lagrange duality. Specifically, the paper views the regularized policy learning problem as a convex optimization problem with equality constraints on the state-action visitation distribution. Then, deriving the dual version of two equivalent primal problems, each named dual-Q and dual-V, the authors show that many recent offline IL and RL methods such as IQLearn and XQL, are specific instance of those problems with a particular choice of the f-divergence as the regularizer. Based on this analysis, the authors propose two algorithms for fixing shortcomings of previous algorithms. For offline IL, ReCOIL is presented as a new dual-Q algorithm that can leverage suboptimal demonstration data and that does not require the coverage assumption needed in prior work. For offline RL, f-DVL is proposed as a new dual-V algorithm as an extension of XQL, which results in more stable training by choosing a different f-divergence. The proposed methods are evaluated on the locomotion and manipulation tasks from the D4RL benchmark, and baseline comparisons show superior performance of ReCOIL and f-DVL on various tasks.", "questions": "* ~~I do not see how one can directly derive equation (14) from equation (5), as a rewriting of dual-V with the temperature parameter $\\lambda$. Can you elaborate on the derivation? I believe that the discount factor $\\gamma$ is determined by the problem specification and is not a hyperparameter that can be tuned arbitrarily.~~\n\n* ~~In equations (32) to (34), can you elaborate on why the interchangeability principle holds in this specific case? In other words, can you explain why $\\max$ and $\\mathbb{E}$ can be exchanged?~~\n\n* ~~All the analysis presented in the paper implicitly assumes discrete state and action spaces (which I think is good for readers in the RL community to follow the math). Do the results naturally extend to continuous spaces? I would be surprised if they did not, but I bet we need more sophisticated tools from real-analysis and probability theory to prove the same results.~~", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The major novel contribution of the paper is the theory which shows that various existing offline IL and RL methods can be derived from a Lagrange dual formulation of a single regularized policy learning problem. This itself has a huge consequence. For instance, showing that pessimistic value learning such as CQL is essentially a dual RL problem brings about a new insight into analyzing their properties and deficiencies. The self-contained review as well as derivations of key theoretical results in Appendix C has a core value in the paper, where the math is relatively easy to follow with the basic knowledge of convex optimization. This analysis itself paves a new way to developing many practical algorithms for both offline IL and RL, and has a significant potential impact on future research in this domain.", "weaknesses": "On the other hand, I have found a handful of math errors throughout the paper, especially in Appendix C, which critically degrades the quality of such a theory-driven paper. While many of them can be simple typos, some may actually require careful consideration. I will list them below.\n\n* ~~For the state-action visitation distribution $d(s,a)$ to satisfy the Bellman-flow constraint in the form of equation (2), I think that the definition of $d$ has to involve the $(1 - \\gamma)$ correction term: $d(s, a) = (1 - \\gamma) \\pi(a \\mid s) \\sum_{t=0}^{\\infty} \\gamma^t P(s_t = s \\mid \\pi)$.~~\n\n* ~~In equation (9) on p.6 and p.28, the conjugate of the f-divergence should be just $f^*$, not $f^*_p$ because the positivity constraint on the visitation distribution is not necessary in the dual-Q formulation.~~\n\n* ~~In equation (29) you have the term $\\gamma \\sum_{s'} p(s' \\mid s, a) \\pi(a' \\mid s') Q(s', a')$. I believe that the sum must be taken over both $s'$ and $a'$.~~\n\n * There is a typo in equation (35). The second term inside $f^*$ should be $\\gamma \\sum_{s', a'} p(s' \\mid s, a) \\pi(a' \\mid s') Q(s', a')$, not $\\gamma \\sum_{s'} p(s' \\mid s, a) \\pi(a \\mid s') Q(s', a')$.\n\n~~* There are two typos in equation (38). First, the expectation in the first term needs to be over both $(s, a) \\sim d(s, a)$, not $s \\sim d(s, a)$. Second, I believe that the last term has to be $\\sum_a d(s, a)$, not just $d(s, a)$.~~\n\n* ~~The Lagrangian stationarity condition as stated on p.20 does not seem to be consistent with equation (43). Taking the derivative of the Lagrangian with respect to the primal variables, I suppose that the stationarity is $d^o(s, a) \\delta_V(s, a) -d^o(s, a) f'(w^*(s, a)) + \\lambda^*(s, a) = 0 ~\\forall s \\in S, a \\in A$. Also, there seems to be an implicit assumption that $d^o(s, a)$ is non-zero everywhere, and the degenerate case of $d^o(s, a) = 0$ is not discussed carefully. It seems that the Lagrangian stationarity is not informative enough to uniquely determine the relationship between $f'(w^*(s, a))$ and $\\delta_V(s, a)$ in the degenerate case.~~\n\n* ~~In equation (110), $\\alpha$ should be replaced by $\\beta$.~~\n\n\nRegarding the empirical results in Section 7, the quality of presentation has much room for improvement, as described below.\n* It is unclear what the left-most figure in Figure 1(a) represents and how it is related to the squared distribution gap plot right next to it.\n\n* In Table 2, it is unclear how the authors determined to highlight some of the entries with blue bold texts as an indication of the most-performant policy. For instance, In the hopper task with the medium few-expert dataset, the RCE seems to have the highest mean but not highlighted in blue.\n\n* ~~Similarity in Table 3, it is unclear what rules were applied to highlight entries in blue. For example, IQL on the antmaze-large-diverse-v0 dataset had a larger return than f-DVL, but both IQL and f-DVL are highlighted.~~\n\n* ~~Moreover, I do not see a huge performance gap between f-DVL and prior methods, and it is hard to judge whether the proposed RL methods empirically outperform existing ones (partially because there is no standard deviation information in the table). In fact, a recent study [1] finds that just reporting the mean and the standard deviation of different RL policies is insufficient, and suggests reporting confidence intervals or performance profiles as more reliable performance measures. The authors should consider leveraging some statistical tools like [1] in comparing policies, especially in situations like Table 3 where many methods have similar average returns.~~\n\n[1] Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc Bellemare. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)