An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning
Authors: Haoran Xu
Shuozhe Li
Harshit Sikchi
Scott Niekum
Amy Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: imitation learning
offline RL
deep RL
dual RL
Primary area: reinforcement learning
TL;DR: An optimal discriminator-weighted imitation view of solving Reinforcement Learning.

Abstract: We introduce Iterative Dual Reinforcement Learning (IDRL), a new method that takes an optimal discriminator-weighted imitation view of solving RL. Our method is motivated by a simple experiment in which we find training a discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weighted behavior cloning gives strong results on various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, we find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero-weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets.

Published as a conference paper at ICLR 2025
AN OPTIMAL DISCRIMINATOR WEIGHTED IMITATION PERSPECTIVE FOR REINFORCEMENT LEARNING
Haoran Xu"; Shuozhe Li Harshit Sikehi', Scott Niekum², Amy Zhang!
1 University of Texas at Austin, UMass Amherst, Meta AI
ABSTRACT
We introduce Iterative Dual Reinforcement Learning (IDRL), new method that takes an optimal discriminator weighted imitation view of solving RL. Our method is motivated by simple experiment in which we find training discriminator using the offline dataset plus an additional expert dataset and then performing discriminator-weigeee
behavior cloning gives strong results various types of datasets. That optimal discriminator weight is quite similar to the learned visitation distribution ratio in Dual-RL, however, find that current Dual-RL methods do not correctly estimate that ratio. In IDRL, we propose a correction method to iteratively approach the optimal visitation distribution ratio in the offline dataset given no addtional expert dataset. During each iteration, IDRL removes zero- weight suboptimal transitions using the learned ratio from the previous iteration and runs Dual-RL on the remaining subdataset. This can be seen as replacing the behavior visitation distribution with the optimized visitation distribution from the previous iteration, which theoretically gives a curriculum of improved visitation distribution ratios that are closer to the optimal discriminator weight. We verify the effectiveness of IDRL on various kinds of offline datasets, including D4RL datasets and more realistic corrupted demonstrations. IDRL beats strong Primal-RL and Dual-RL baselines in terms of both performance and stability, on all datasets. Project page at wtthttps:// Pattps://Eyanxher ryanxhr...../////////// github. wotthub.i//nn /DRDL
INTRODUCTION
Offline reinforcement learning (RL) has attracted attention as it enables learning policies by utilizing only pre-collected data. It is a promising area for bringing RI. into real-world domains, such as industrial control (Zhan et al. (022) and robotics (Kalashnikov al., (021). In such scenarios. arbitrary exploration with untrained policies is costly or dangerous, but sufficient prior data is available. The major challenge of offline RL is the distributional shift issue (Levine al. 2020). i.e., the optimized policy's 's distribution is different from the offline data distribution, which means the value function may have extrapolation errors to unseen actions produced by the optimized policy Fujiioooo et al., 2018). To deal with this challenge, most offline RL algorithms build on the primal form of RL (i.e., maximizing a value function respect to the policy, which needs to alternate between policy evaluation and policy improvement), and impose an additional behavior constraint either by policy constraint, which constrains the policy to be close to the behavior policy using some distance function (Wu et al. 2019 Fujimoto & Gu 2021); by value regularization, which directly modifies the value function to be pessimistic (Kumar et al. 2020) Kostrikov et al.) 2021a); or by uncentainty estimation, which guides policy optimization to low-uncertainty regions (An et al., 2021 Yu et al.) 2020). However, in Primal-RL based methods, the policy may still output out-of-distribttion (OOD) actions if the policy constraint weight is not set properly, which will cause either inaccurate or over-constrained value estimation that results in suboptimal behavior.
Notably, recently there's 's also another line of in-sample based offline RL algorithms (Xu et al. 2023 Mao et al. 2024a b). which can be derived from the dual formulation of RL (i.c., maximizing the reward respect to the policy's 's visitation distribution) (Sikchi al. 2023b). Different from the primal counterpart, Dual-RL based methods isolate the learning of value function and policy; the value function can be learned using only dataset actions, which brings minimal estimation errors. To extract
"Equal contribution. Correspondence Haoran Xu.

---

Published as a conference paper at ICLR 2025
Iterative Dual-RL: Dual-RL with dataset refinement iterations
Original Cataret
Fibered Dataent
Nerroll cooateoddare rettre
d[(,,) 0
dg
Mapos
Achmassa
(a) Framework of Iterative Dual-RL
Results of Optimal-DWWE
Figure 1: (a) Illustration of our proposed IDRL framework. IDRL breaks the regularization barrier by performing imitation learning on seratively-refinee dataset. it can solve hard tasks where previous behavior behavior-regularize regularized offline RL can not do. For example, previous methods will fail tinding the shortest path from blue point to red point while crossing yellow point, to non- con-uniform data coverage at different state. (b) Mean scores of optimal discriminator-weiihteel hehavior cloning (Optimal-DWWB. on D4RI, Mujoco-(m.m-t,me (m.m-r.m-e). Antmaze- (all) and Kitchen-(all (all) datasets. We train a discriminator d on offline dataset 'D and an additional expert dataset Dg, we then use wa(s, a)
(T...) T-d[s.n] to do weighted-Be on D if wa(s, a) > 6. We compare it with SOTA
(a.o))) d(s.a)
Primal-RR RL method ReBRAC Tarasov et al. 2024 and SOTA Dual-RL RL method ODICE (Mao et al.
2024a
the implicit policy contained in the value function, one often uses the visitation distribution ratio between the optimized and behavior policy to do weighted behavior cloning (BC) (Peng et al. 2019). Although Dual-RL methods suffer minimal OOD errors by using in-sample learning, they often lag behind in performance of primal-Re RL methods. One conventional opinion using weighted -BC style policy extraction causes poor generalization combined with its mode-covering nature (Park) (et al. (024). In this paper, we aim to investigate pitfalls and push the limit of Dual-RL methods. We start from surprising finding: learning the optimal visitation distribution ratio using "gold" discriminator on the offfine dataset and an additional expert dataset, and using this ratio as a filtering weight in weighted-BC outperforms all classical offfine RL methods various datasets, shown in Figure I Although this axcriminattorweiihte BC is an oracle baseline as the expert datasets are unavailable, it points to the fact that Dual-RL methods, as virtue of learning visitation distribution ratios, could faciliate this kind of dataset filtering and weighting.
Given this potential, what is missing in current Dual-RL methods? 1) We show that current Dual-RL methods are unable to accurately estimate the state-action visitation ratio as result of gradient update strategies they use for minimizing the Bellman residual. We find using a semi-gradien update in Dual-RL changes the fixed-point of optimization, learning an action distribution ratio rather than a visitation distribution ratio. Using the action distribution ratio to learn policies forces learning potentially suboptimal actions at states never visited by an expert, leading to poor generalization during weighted-BB -C policy extraction. 2) Irrespective of the gradient update strategy used, Dual-RL. methods learn regularized optimal visitation distribution w.r.t offline data distribution (i.e., visitation that maximizes performance while staying close to offline data), as opposed to learning the optimal visitation distribution ratio. Our key contribution in this paper is to address both the limitations above and present a new offline RL algorithm that is able to break the regularization barrier. First, we propose new objective for accurately learning the visitation distribution ratio using a two-stage procedure. After getting the action distribution ratio using semi-gradient Dual-RI, we adopt an off-policy evaluation approach that recovers the correct state-action visitation distribution ratio using the action distribution ratio. This allows decomposing the original tasks of obtaining visitation distribution ratio into two distinct, complementary, and easy solve tasks. Second, we show that the regularized optimal visitation ratio has a sparse form (some weights are zero) (Xu et al. 2023). This property can be used to refine the offline dataset iteratively and gradually learn an optimal visitation ratio that moves closer to the optimal discriminator weight and can be used for learning better policy by weighted behavior cloning. Our theoretical results justify our method by deriving a smaller optimality lower bound due to this curriculum-refinee dataset filtering.
We term our method iterative Dual-RL (IDRL), IDRL iteratively filters out suboptimal transitions and then performs imitation learning on the remaining subdataset that is within close to the optimal visitation distribution. An overview of our method can be found in Figure This learning paradigm also has connections some recent success in using deep generative models (DGMs) in offline RL algorithms (Mao et al. 2024b) Haasenn et al, (023). These methods first fit the complex, multi-modal behavior distribution using the strong expressivity of DGMs, and then
2

---

Published as a conference paper at ICLR 2025
conditioned or guided generation to unearth optimal actions. Compared to these generative methods, our method bypasses the costly two-step procedure and uses lightweight behavior cloning to directly fit the "optimal" distribution. Empirically, kinds of offline datasets, including D4RI, benchmark datasets and corrupted demonstrations that is heteroskedastic and occur more often in real-world scenarios. On D4RL datasets, IDRL matches or surpasses both Primal-RL and Dual-RL baselines. On corrupted demonstrations, IDRL outperforms strong support-base and reweighting-basee -baiid approaches. Under all settings, IDRL achieves better stability and robustness during inference time compared to all other baselines.
verify the efffectiveness of IDRL on different
2 PRELIMINARIES
We consider the RL problem presented as a Markov Decision Process (Sutton et al. 1998). which specified by tuple M = (S, A, P, 7)). r,),
Here S and A state and action space, P(s [ss a) and do denote transition dynamics and initial state distribution, r(s, a) and 7 represent reward function and discount factor, respectively. The goal of RL is find a policy T(aas) which maximizes expected return (Tr) EI_:=0 >" = r(st, at)]. Offline RI. considers the setting where interaction with the environment is prohibited, and one needs to learn the optimal ı from a static dataset DD {&;; 0g Tú, & f= We denote the empirical behavior policy of DD as μr, which represents the conditional distribution p(als) observed in the dataset.
Value functions and visitation distributions Let V S and Q S A R be the state and state-action value function of æ, where (s) = E, ('r(8), a,)],a = s] and Q" (8. a) E I2 (rr(st, 55) = s, 3,00 = a]. The visitation distribution d" is defined as d' (s, = (1 n) Et= 7 Pr (st = s, a, = a so do. Vt, a, ~ I , 8141 P (st, as)))))), which measures how likely π is to encounter 8,a when interacting with M. averaging over time via 7-discounting. Let V*, Q" and d' denote the value functions and visitation distribution corresponding to the regularized optimal policy x². We denote df as the visitation distribution of the true optimal policy =. We denote the empirical visitation distribution of μ as dº Let 7," be the Bellman operator with policy and reward such that 7, T,, a) = r(s, a) + ((l.a, [Q(s', a')]. We also define two Bellman operators T, and T for the state value function, where T (s, r(s, a) + ?Beee(()a.)) [V()) and TV(s, a) = P(.lea)) (s')).
2.1 PRIMAL AND DUAL RL
Interestingly, the value of policy, J(x), may be expressed in two ways, where the primal form uses the value function while the dual form uses the visitation distribution:
Primal-RI. (1 7) -=00~=000000 [Q" (So: ao)] (TT) B( [r(s), a))- Dual-RL
In Primal-RI., there are typically two steps, policy evaluation and policy improvement. In policy evaluation, the Q-values are estimated by finding the fixed point of the Bellman recurrence (Watkins)
& Dayan 1992 Sutton et al. 2008 Mnih et al.) 2013). i.e., minimizing the squared single-step Bellman difference using off poficy data D as ming J(Q) ! ( [(T Q Q)(s, (sssa)?? a)? In policy improvement, the policy и be optimized by max, EEND, (+) (Q(s, a)] using policy gradient (Sutton et al.) 1999). In practice, the policy evaluation and improvement step are alternated till convergence to the optimal solution Q* and π². In the offline setting, prevent overestimation arising due to the distribution shift issue, one often imposes a policy constraint (action-level) to the policy improvement step (Kumar et al., 2019 Fujimoto & Gu (021); to the policy evaluation step
(Kumar et al. 2020 An et 2021); or to both steps (Fakoor et al. 2021 Tarasov et al. 2024).
Dual-RL, also known as Distribution Correction Estimation (DICE) (Nachum Dail 2020 Sikchi et al. 2023b), was first used to ensure unbiased estimation of the on-policy policy gradient using off-policy data. Dual-RL. RI. incorporates
((T) with a visitation constraint term (state-action level)
(d(d"Ilad B( [f( d" (s.a) )} where f(x is convex function, i.e., finding x² satisfying
= arg max E( [r(s, a)) (dF
This regularized learning objective is generally intractable due to the dependency on d= (s, a). especially under the offline setting. However, by imposing the Bellman-flow constraint (Manne
3

---

Published as a conference paper at ICLR 2025
1960).
Easa d((s (1-7)do(s)) + Es.a d(3, a))(s) a) on states and applying Lagrangian duality and convex conjugate, its dual problem has the following tractable unconstrained form:
min (1 - -] [v()) + (B(aa))arr)) [
( (T,V - ((s))/a))
(2)
V
where /== = max (0, (x)) (x) / (max (o, (x))) Note that objective 2 can be calculated
solely with (s, a, 8') sample from D. which enables in-sample learning and mitigating the distribution shift issue in offline RL by not querying dunction-approximator
out-of-distributiinn actions. In principle, Dual-RL serves as a better off-policy (by getting unbiased on-policy policy gradient estimation) and offline algorithm (no OOD actions in training value functions) than Primal-RL, however, in next sections we show several limitations that prevent leveraging their full potential.
2.2 LIMITATIONS OF CURRENT DUAL-RL METHONS
Incorrect distribution ratio estimation Notice that the nonlinear term in objective ( contains the Bellman residual term T, (s, -ss). In -siduual-gadient update, both V('') and (v(s) would contribute to the gradient update, which causes slow convergence and a gradient conflict issue (Baird 1995 Zhang et al. (001b)), especially in the offline setting where the gradient on (s))) is crucial for implicit maximization to find the best action (Mao al. 2024a). To remedy this, one often uses prevalent semi comi-gradient technique in RL that estimates T.V (s, a) with Q(s, a). The update of Q(s, a) and (s) in semi-gradiens Dual-RL RL methods are as follows (see Appendix A. for details).
min E (e V(s) + af (1Q(s, - ((s)]/a)
(3)
min (ss [(r(s, + +VV(s') a)), &
(4)
Note that 1 1 is omitted and the initial state distribution do is replaced with dataset distribution d' stabilize learning (Kostrikov et al. 2020 This learning objective of V can be viewed as an implicit maximizer to estimate sUPo>u((lls) ((,aa) (Xu et al. 2023 Sikchi et al. 2023b), the first term pushes down V-values while the second term pushes up -vaauues - >0, O. In addition, update also helps alleviate biased gradient estimation caused by single-sampe estimation of Tú in objective (2). However, as we show later in Section using semi-gu -radientt update changes the fixed-point of the learning objective, preventing us from learning the true visitation distribution ratio.
because fo is non-linear and usually cannot cover all possible s', using the semi-gradies
Data-regularized policy extraction One intriguing property of Dual-RL is that, although the learning objective doesn't 't contain the policy, it actually learns an implicit optimal policy through the visitation distribution ratio between the optimized and behavior policy π²" =a(ass) x [0(s.a) p(ass)
(s.aa)
One way to extract an explicit policy is minimizing the KL divergence (Boyd et al. 2004) between n
and π² which is equivalent to performing ddvantage-weightne behavior cloning (Peng et al. 2019)
using the visitation distribution ratio as shown:
d' ds(ss a)
w" (s, = d (P a)
max ((Q).) vs))
(5)
π = arg max (e..))ad")) [w" (s,a) log {a]s))]
(6)
The explicit policy is parametrized as a unimodal Gaussian distribution in order to compute log T(a s) (Haarnoja et al.) 2018). 8). However, this may cause some mode-covering behavior and deviation from the optimal mode some suboptimal transitions are assigned non-zero weight (Ke et al., 2019). One way to solve this problem is using more expressive generative models like diffusion models (Ho) al, (220) [Song et al., 2020) fit the multimodal behavior distribution and select the optimal mode using guided sampling (Mao et al.) (2024b). However, the success depends on both accurate behavior modeling and correct sampling guidance, and also bring additional training/evaluation burden.
ITERATIVE DUAL-RL: TOWARDS OPTIMAL DISCRIMINATOR WEIGHTTD BEHAVIOR CLONING
In this section, we introduce our proposed approach, Iterative Dual-RL (IDRL). IDRL presents principled solution to fix the aforementioned two limitations in Dual-RL. We start by deriving
4

---

Published as a conference paper at ICLR 2025
the result that semi-gradient Dual-RL learns the action distribution ratio between the optimized and behavior policy; we then propose a correction to recover the true visitation distribution ratio based on the obtained action distribution ratio. Leveraging the learned visitation distribution ratios, we present an iterative self-distillation method to approach the optimal visitation distribution ratio. Theoretical analysis shows how using iterations help Dual-RI. get better imitation result by adopting crricullu-reffince dataset filtering. Finally, we use a motivating toycase to validate the usefulness of IDRL in successfully extracting the optimal visitation distribution in the dataset.
3.1 TRUE VISITATION DISTRIBUTION ESTIMATION
First, show that semi-ge aami-gradient Dual-RL changes the fixed point of optimization and gives solution that learns the action distribution ratio between the optimized and behavior policy rather than the expected visitation distribution ratio.
Proposition 1.
Semi Semi-gradient Dual-RL only learns w" (als)
="(els) instead of w² (s, (e,0) p(ala)
(s.o)
The result can obtained by setting the derivative of objective
3 w.r.t (s)) to zero, where we have (s, (s, V (s))/a)
Vs dp, , + Ea-yll ma ((") (z)(()) max(0, (1))))))
0, and leveraging the fact that
There are two main shortcomings of learning an action-distribuune ratio. (1) Action distribution ratio does not reason whether states are ever visited by the optimal policy (are 'good'), therefore may assign positive weights to actions at bad states. Under the function approximation setting where states may share similar representations, this is more likely to cause incorrect generalization to suboptimal patterns generated by 'bad" states. This problem is especially severe when the dataset is dominated by a large portion of suboptimal data. (2) Iteratively filtering the dataset with action ratios can result in a fragmented dataset as some weights can be zero. For instance, if transition is assigned
weight, the preceeding transition in trajectory is fragmented and does not have backup target. Deep RI. algorithms are not well suited to learn from such fragmented trajectories as they will overestimate the value of missing transitions.
We reframe as an off-policy evaluation (OPE) problem (Uchara et al. 2022) to recover the state-action visitation distribution ratio given the action distribution ratio. Inspired by techniques from Nachum & Dai 2020), we construct and solve the following constrained optimization problem
max -(d) s.t. d(s) = (1 - 7)do(s) + Σ d($)T (3) 420
3.8
In this optimization problem, the [|] constraints uniquely determine d = d' while h serves function that could make the optimization problem more approachable and stable when applying duality; common choices of h include h(d) or ldd) == (d)l
The above objective 77 requires fitting an explicit policy π². We leverage the fact that x (alss) can be estimated via importance sampling w² (a(s) [[(ass to propose a way learn visitation ratios while only ever sampling from in Theorem [ Theorem presents a tractable OPE objective by applying
duality (Rockafellar 1970 Eq. under h(d) == (d)
Theorem 1.
Given an action distribution ratio w* (als), we can recover its corresponding state visitation distribution ratio w" (s) as
(s) max (f.( (Harup (773 w (²"ss)= := (
(8)
where U is the optimal solution of the dual form of (7) as following,
min E (3 [($)) TU,(s, a)) + +E Exnar S (Baw to (alss) (T0ss))))
U
Obtaining an unbiased estimator: In objective the first term easy to estimate, however, an unbiased estimation of the second term along with w (s) is generally non-trivial due to the expectation inside a non-linear function f; or (f')-1 We show that Lemma can be used to obtain an unbiased estimate for the second term above.
5

---

Published as a conference paper at ICLR 2025
Lemma 1. Given random variable X and its corresponding distribution P(X), for any convex func- tion f()). the following problem is convex and the optimal solution is y² = (f') (E ((x) (g()))
min Bew
g(x) y)-
The proof follows by setting the derivative of the objective w.r.t y to zero. Substituting P(z) as p and g(x:) as w" (a)s)(TU((s a) U()). we can get the following result:
Theorem 2. The unbiased estimate of (f") [w" (als) (+)]]) is the optimal solution W (s) of the following optimization problem:
min ((ea)-a ((((()) (F (((()) W
W(s)]
This learning objective provides an unbiased estimator by only using (s, a, 8') samples from the dataset, as two expectations that cause biased gradient estimation (Ba-y and E. in TU) are all linear. Note that this objective is convex with respect to W, which indicates guarantee of convergence to W' under mild assumptions.
Using the fact that (z))(x) = max(0, (()')(-(--) (x)), we are ready to present a surrogate objective for the second term in Eq. ( that admits the same gradient w.r.t U, by using the following equation.
E Eand (fF)
(1)) (Ea-u w (als) (TU(s.a) (U())))) Ea [ w0 (als) (TU(s, (TUl..)) U(5))
-V)Bea,))ad max(0, (*)))) w' (als) (TU(s,a) (+))
Finally with the reductions shown above, learning Eq.(9) in an unbiased way using sample- based estimation amounts solving the following two optimization problems jointly:
min E (0 ((((()) ( - " ('(als - Uss) W(s)]
(10)
W
min E (x.))=a [ss)) (U()) a)) + ((.a)aaa ma (()) - -w
(11)
U
In (TT) the learning objective introduces sparsity in learning the value function (Xu et al. 2023).
The first term pushes down value residuals while the second term pushes up value residuals if w (s. a) > 0. This makes the residuals of (s, a) under the regularized optimal visitation distribuion to be high. Note that the learning of U and W introduces no extra hyperparameters that are prevalent in offline RL. After getting the optimal U and W*, we can obtain the corrected state-action visitation distribution ratio as:
d' (s) w" (s) = ws(s wn (als) = max (0. (V)
(12)
d (s,aa
3.2 LEARN THE OPTIMAL DISCRIMINATOR WEIGHT BY ITERATIVE SELF-DISTIL DISTILLATIION
Although we get the correct visitation distribution ratio in Dual-RL, that ratio is not the true optimal visitation distribution ratio (i.e. d² 7 dB) as it is regularized by the offline data distribution. We thus propose an iterative oif-distilllation way to break the regularization barrier, our key insight is that can leverage the learned (regularized) optimal visitation distribution ratio after one iteration of Dual-RL to refine the offline dataset to a new one, which will be used for another iteration of Dual- -L. This procedure can be repeated several times to approach the true optimal visitation distribution ratio.
More specifically, we extend Dual-RL. to M iterations (M 2 1) and use x² -ivieeggee as Dj. After the i-th iteration of Dual-RL, we select support of dataset D; that has non-zero probability mass in optimal regularized visitation (wā( (s, a) > 0), and run the i + 1-th iteration of Dual-RL. The visitation distribution ratio at the last iteration w (s,a) a) is used for learning the policy by weighted behavior cloming. We term this iterative method as iterative Du (IDRL). the pseudo-code is presented in Algorithm|] []] IDRL uses multiple iterations of Dual-RL break the regularization barrier that impedes offline RL to learn performant policies by selecting support of offline dataset without suffering overestimation. Note that breaking the regularization barrier is hard with offline Primal-RL RL algorithms as they do not reason about visitation distributions, thus making it impossible to refine the offline dataset used for regularization.

---

Published as a conference paper at ICLR 2025
(2) Offline dataset d
(b) WBC using wf s)
(c) WBC using (s.
(d) WBC using (a(as)
(e) WBC using (5
Fini laration
Sooard larrsiion
Figure 2: IDRL in grid- world domain. The initial state (blue) and the goal state (red) define the task, with the green arrows representing remaining transitions from dataset. The opacity the green arrows demotes magnitude of the weights from the respective distribution ratios. Red arrows depict the trajectories generated by policies obtained through weighted-BC with the estimated distribution ratios. (a) Original dataset. (b) and (d) show the results after filtering the dataset based the learned policy ratio (Uncorrected Dual RL visitation ratio) w(als) s) in the first and second iterations, respectively. (c) and (e) demonstrate using the state-action visitation distribution ratio w(s, a). which is computed by combining w waass) and w(s) (IDRL. correction). This process reveals that the method progressively focuses on the most relevant transitions, enabling the recovery of a near-optimal visitation distribution ratio after 2 iterations.
subsequent filtering
Algorithm 1 Iterative Dual-RL (IDRL)
The filtering procedure in IDRL can be approx- imately viewed as replacing each iteration's 's be-
1: Initialize value functions Q Voz. Uwi Wez, pol- icy network To, require and dataset Dj D
havior visitation distribution to a better one (i.e.,
(. 2, .M do 2: for =1
the regularized optimal visitation distribution at this iteration): D (d]l()) = (d] d 1) In- tuitively, this could boost performance improve-
1........ .N
4:
Sample transitions r...) Dr Update Qo and Vas by 4 and
5:
ment, especially on suboptimal datasets. For- mally, we give a theoretical analysis of IDRL based on previous analysis of behavior cloning,
6:
end for
Get action ratio wn (als) by Bq
8:
for t - 1,2, ... .22 do
we aim to 1) give the behavior cloning perfor-
Sample transitions (s, a. Dr Update Us, and W by and 11: end for
9:
mance bound of IDRL, and 2) analyze how iter-
10:
12
ations will influence this bound.
Get state-action ratio we wn a) by Eq. and
Theorem 3. Given horizon length H and the dataset sample size of 'D as ND. the behavior cloning performance bound of IDRL iteration k is given by
Dx+1 = {(s, ((s,arr,s) E Dr. wa () >0}
13: end for
14: Learn by Eq, 6 using Dar and wsr(s,
S
= ( (ND... + NN DDDa- max, Wh+ (s)
Theorem 4. We have V(D++1) 2 V(Dx) after the k-tth iteration of IDRL.
Theorem 3 is derived by framing weighted-BC as selecting expert data Da++1 1 from suboptimal data
Dy using discriminator-weighted behavior cloning (Xu et al. 2022b Li et al. 2024). From Theorem 3 we can see that if Da+1 closer to the expert distribution, (D++1) increases, however, ND decreases, so the second term may also increase. This hints a trade-off by adjusting Dx+1 to maximize the behavior cloning performance bound. Theorem 4 shows that IDRL uses iterations efficiently find the optimal choice of Dj by ensuring a monotonic improvement over V(Dx)- This result is important as the above property is missing from classical offline RL; decreasing may be expected produce similar empirical results like Theorem 3 however, decreasing a does not have a monotonic improvement guarantee like Theorem and suffers from overestimation in practice.
IDRL in gridworld toycase We demonstrate the effectiveness of IDRL through simple grid- world experiment. The environment consists of continuous state and action space. However, we collect offline data in discrete manner. We use discrete behavior policy that takes action randomly from (t, 1, +, +) to collect 500 transitions. Note that the weighted-BB BC policy is learned to be continuous with unimodal gaussian distribution. We use this toycase to mimic the scenario in offline continuous control problem under function approximation, where similar states share the same representation in latent space one latent state may have multiple behavior actions. In this case
7

---

Published as a conference paper at ICLR 2025
Table 1: Averaged normalized scores of IDRL against other baselines. We report the average normalized scores the of training with standard deviation across random seeds. highlight the top score (integer-leve) IDRL matches or outperforms previous SOTA Primal-RL and Dual-RL RL methods on almost all tasks.
Primal (Max-Q)
Dual (Weighted-BS
DARL Dutaset
TD3+BC
CQL
ReBRAC
Diftusion-QL
X%6BBC
IQL
SQL
O-DICE
IDRL (ours)
halfcheetah-m
48.3
44.0
47.4
48.3
58.4 +0.1
hopper-m
59.3
58.5
102.2 +1.0
66.3
75.7
434
99.5 +0044
walker2d-m
83.7
72.5
82.5 =3.6
+009
72.5
84.2 -4.6
84,9
92.8 +0.3
halfcheetah-m--r
44.6
45.5
51,0=0.8 =0.8
47.8
+0.3
40.6
44.2
44.8 =0.7
44.0 =0.3
58.0 +0.3
hopper m-r
60.9
95.0
98.1 15.3
101.3
1.0.6
95.2
99.7 13.3
1:2.7
101.7 10.7
walker2d-m-4
81.8
77.3 47.9
95.5
76.1
81.2-3.8
83.6-4.11
84.4 40.4
halfcheetah-mme
90.7
90.7
101.1 +5.2
96.8
86.7
94.0 -044
98.1 +0.2
hopper-m-e
98.0
105.4
107.0 =6.4
111.1 +1.3
110.9
101.5
110.8 =2.2
110.8 +0.6
111.0 4:0.4
walker2d-m-o
110.1
109.6
111.6 -0.3
110.1 +:033
109.0
110.6
110.0
=0.8
110.0 =0.2
113.9
+:01
antmaze-u
78.6
84.8
97.8 11.0
85.5 +19
62.8
85.5
92.2 =14
94.1 =6.6
99.5
+0.3
antmazz-u-d
71.4
43.4
88.3
133.0
66.7 +:.0
50.2
66.7
74.0 =2.3
79.5 #3.3
88.5
+12
antmaze-m-p
10.6
65.2
84.0 =4.2
72.2 453
5.4
72.2
80.2 =3.7
86.0=1.6
+1.1
antmaze-m-d
54.0
76.3 13.55
71.0 #3.2
9.8
71.0
79.1 44.2
82.7
13.9
antmaze-I-p
38.4
60.4
4-26.1
39.6 4-4.5
0.0
39.6
53.2 +4.88
55.9 +3399
60.3
433.4
antmaze-I-d
31.6
54.4 +:2.11
47.5 +:4.4
6.0
47.5
52.3
=5.2
54.6 +:4.8
54.2
433.8
kitchen-c
43.8
77.2 +8.3
84.0 +-7.4
33.8
61.4
76.4
-8.7
75.0 +66.6
80.5
+-3.8
kitchen-p
49.8
68.5 10.66
60.5 +669
33.9
46.1
72.5
=74
72.8 +:4.3
74.3
+44.4
kitchen-m
51.0
55,3 =9.2
62.6 +:5.1
47,5
52.8
67.4
=5.4
65.8
#3.6
67.8
+2.1
assigning positive weights to suboptimal transitions will deteriorate the weighted-BC result. Figure walks through how applying IDRL subsequently filters the offline dataset towards optimal visitation thus learning an optimal policy, while using only action ratios lead to suboptimal performance.
EXPERIMENTS
In this section, we present empirical evaluations of IDRL on different kinds of offline datasets. We first evaluate IDRL on D4RL. benchmark offline RI. datasets (Fu et al., 2020) and compare against several SOTA Primal-RI. and Dual-RI. baseline algorithms. However, D4RL. datasets are collected from policies that enable methods with strong distribution constraints to already have strong performance. As consequence, we also find that IDRL only needs or two iterations to achieve best performance. To further show the benefits of IDRL, we test IDRL on more realistic datasets where more iterations are needed find the optimal visitation distribution. We choose corrupted demonstrations where few expert demonstrations are mixed with a large portion of random data (Xu
2022b Hong et al. 2023b). Experimental details are shown in Appendix C al.
4.1 RESULTS ON OFFLINE RL
We first evaluate IDRL on the D4RL benchmark (Fu ct al. 2020) and compare it with several related algorithms. For the evaluation tasks, we select Mujoco locomotion tasks, Antmaze navigation tasks and Kitchen tasks which require both locomotion and navigation. While Mujoco tasks are popular in offline RL, Antmaze and Kitchen tasks are more challenging due to their stronger need for selecting optimal parts of different trajectories to perform stitching. For baseline algorithms, we selected state-of-the-art methods not only from Primal-RL methods (that select best policy modes given by Max-Q), but Dual-RL RL methods (weighted-BC). Primal- RL baselines includes TD3+BC (Fujimoto
Gul Gul[00TT (02T). CQL (Kumar et al., (020)) ReBRAC (Tarasov et al.] (024) and Diffusion- QL (Wang et al. 2023) which has strong policy expressivity by using diffusion models. Dual-R baselines include IQL. (Kostrikov et al. 2021bb. SQL (Xu et al. 2023) and O-DICE (Mao et al.) (024a). The results of IQL and SQL reflect the performance of using semi-gradient update as they apply action-level behavior constraints. Notably, O-DICE is recently proposed algorithm that stands out among various Dual-RL methods. Although it claims to achieve the correct vate-action visitation distribution ratio, it introduces another hyperparameter 77 which is hard to tune in practice. We also include the results of using X%-BC, which is a filtered version of BC that runs behavior cloning on only the top X% high-return trajectories in the dataset. This comparision is show the necessity of behavior
8

---

Published as a conference paper at ICLR 2025
cloning based on transition-wist selection rather than trajectory-wise We select the best results by spanning X in {2, 5, 10).
The results shown in Table it can be seen that IDRL matches outperforms all previous SOTA algorithms on almost all D4RL tasks. This suggests that IDRL can effectively learn a strong policy from the dataset, even in challenging Antmaze and Kitchen tasks that require RL to "stitch" good trajectories. The large performance boost between IDRL and previous Dual-RL. methods, especially on sub-optimal datasets that contain less few near-optimal trajectories. demonstrates the benefits of iteratively finding the optimal visitation distribution ratio. The consistently better performance compared with IDRL and Primal-Re baselines demonstrates the essentiality of in-sample value learning, which causes fewer overestimation errors. All these results reveal the power of using an optimal discriminator-weigee weighted imitation view of solving offline RI..
To show both components are necessary in IDRL, perform ablation study where compare the performance of IDRL with (IDRL w/ M = 1). and the ablation that uses w= == (a|s) to filter datasets (IDRL w/ w" (a)s)). It can be shown in Table 2 that using more iterations boosts the performance by finding the best support of the dataset to do imitation learning, as discussed in Theorem 3 This also validates the importance of using the correct state-action visitation distribution ratio as using action distribution ratio will get significantly worse results due to overestimation errors caused by fragmented trajectories. This ablation study justifies the benefits of both improvements to Dual-RI..
Ablation study
Table 2: Ablation study of IDRL.
Ablation
Mujaco
Antmaze
Kitchen
IDRL
90.8
80.3
74.8
IDRL w/ M IDRL w/ (a]s))
73.2
68.8
70.5
56.8
51.3
67.8
4.2 RESULTS ON CORRUPTED DEMONSTRATIONS
In this experiment, we aim to demon- strate the effectiveness of IDRL in recovering the true visitation distri-
hulfCrontah
Walker2d
H0ppor
ANI
Normalize return
bution from datasets that contain a significant amount of noisy sub-
80
80
80
60
60
optimal transitions. The experiment involves creating "mixed" dataset by combining random and expert pol- icy generated transitions from Mujoco datasets D4RL at varying expert ra- tios, thereby simulating dataset col- lected with low-per oow-prforming behavior policies. The purpose of this experi- ment is to tackle a common challenge in offline RL algorithms: their tendency anchor the learned policy to the dataset's behavior policy. As shown by previous work (Hong et al., [2033]]b) while this anchoring works well when the behavior policy is high-performing. it becomes problematic datasets dominated by low-performing trajectories. In such heteroskedastic and corrupted datasets, where only few trajectories are high- performing, most offline RL algorithms struggle to recover the optimal policy because they heavily rely on the behavior policy present in the data. By applying IDRL iteratively to these mixed datasets, we aim to remove suboptimal transitions, gradually refining the dataset to better represent the optimal policy. Compare with the weighted sampling strategy proposed by previous paper (Hong al.) 2023a]b), iterative application of IDRL can exploit and combine the rare, high-performing transitions while filtering out suboptimal transitions, effectively learning near-true visitation distribution ratio that enables the recovery of an optimal policy, even in challenging, noisy offline settings.
20
20
Expert ratio
Figure 3: Performance of naive BC, Advantage Weighted (AW), Keensity-Weighee initialized with AW (DW+AW) and IDRL on mixed datasets created by combining varying percentages of expert transitions with random transitions from D4RI. Mujoco datasets. For IDRL, this table shows results using three iterations.
RELATED WORK
Primal-RL and Dual-RL approaches: Offline RL requires trading off between unconstrained policy improvement and staying 'close" the offline data distribution, choosing actions arbitrarily can lead to requiring evaluation of policies that go beyond the support of offline dataset and have incorrect evaluation especially with overestimation bias (Fujimoto al. 2018) present in Bellman

---

Published as a conference paper at ICLR 2025
optimality backups. To tackle this distributional shift problem, most classical/primal offline RL meth- ods augment existing off- policy RL methods with an action-level behavior regularization term that prevents the learned policy from deviating too far from the dataset policy. Action-leve level regularization
can appear explicitly as divergence penalties (Wu et al., 2019; Kumar et al. 2019; Xu et al. 2021
Fujimoto & Gu| 2021; Cheng et al. 2023: Li et al.] 2023), implicitly through weighted behavior cloning (Wang et al. 2020; Nair et al. 22220) or more directly through careful parameterization of the policy (Fjjimoto et al. 2019; Zhou al. 2020). Another way apply action-level regularization
is by learning pessimistic value functions that encourage staying near the behavioral distribution and
being pessimistic about OOD actions (Kumar et al., 2020 Kostrikov et al.) 2021a] Xu et al.) [022c]
2023:
Wang et al. 2023 Niu et al. (022). ). Several works also incorporate action-level regu arization through the use of uncertainty (An et al. 2021 Bai et al. 2021) or distance function (Li et al. 2022a).
Another line of methods, on the contrary, imposes action-level regularization by performing form of imitation lcarning on the dataset. When the dataset is high-quality (contains high return trajectories), can simply clone filter dataset actions to extract useful transitions (Xu et al.] 2022b Chen et al., 2020; Zhang et al.) 2023 Zheng et al. 2024), or directly filter individual transitions based on how advantageous they could be under the behavior policy and then clone them (Brandfonbrener et al. 2021 Xu et al. (022a)). Prior attempts to include state-action level behavior regularization (Li et al. 2022b Zhang et al.| (022) have required computationally costly extra steps of model-based OOD state detection (Li et al. 2022b] Zhang et al., 2022). Most offline RL methods rely on unimodal Gaussian policy that suffer from mode-covering behavior during policy optimization potentially outputting suboptimal policies. The use of expressive generative models allows us to move past this limitation (Hansen-Fstree
al., 2023; Mao et al. 2024b) by their ability to match value functions modes accurately, albeit at a high computational cost and slow inference speed.
Unlike the above Primal-RL approaches. Dual RL approaches reason in the visitation space for maximizing returns and directly regularize the state-action visitation divergence with the offline dataset. As a by-product, these methods also promise to learn the ratio of optimal regularized visitation distribution with offline data visitation. Numerous works have shown their utility in off- policy evaluation (Nachum et al. [201 9a] Zhang et al.] 2019a| 2020). offline policy selection (Yang) 2020), off-policy RL (Nachum al. [019b] Lee et al.) 2021 [Sikchi et al., 2023b), safe RL (Lee 2022). GCRL Ma et al. (2022bb:: (Sikchi et al. (2023a), and imitation learning (Kostrikov et al.
et al.
et al.
2020
Zhu et al. 2020 Garg et al.) 2021] Kim al. 2021 Ma al. 2022a Sikchi et 2024).
Connection between offline RL and imitation learning: Offline RL and imitation learning share deep connections. Rashidinejad et al. (202T) studies the connection theoretically from the lens of data composition to analyze optimality rates; (Sikchi et al. (2023b) shows that imitation learning becomes a special case of RL with reward set to zero in the dual RL framework. Other works (Kumar et al., 2022) have empirically evaluated when imitation learning in the form of behavior cloning is preferred over offline RL. Our work, motivated by the simple experiment that having access to expert data is sufficient to outperform current offline RL methods, proposes a smooth and principled interpolation that refines offline datasets in offlin RL to be closer to expert state-action visitation distribution. This
is opposed to prior works that iteratively relax policy regularization (Li et al. 2023 Hu al. 2023 Liu et al. 2024) which only consider actions, potentially learning suboptimal actions at states never visited by the expert and lacking a principled reduction to expert visitation distribution.
CONCLUSION AND LIMITATIONS
In this paper, we provide an optimal Motivated by a simple experiment that finds the effectiveness of optimal discrininator-weigheed behavior cloning, we build on the result of Dual-RL and propose iterative Dual-RL (IDRL) that aims to fix two pitfalls of current Dual-RL RL methods. IDRL iteratively filters out suboptimal transitions and extracts a policy with weighted behavior cloning on that subdataset. We give both theoretical and empirical justification for our approach. IDRI. achieves SOTA results on various kinds of datasets, including D4RL benchmark datasets and noisy demonstrations where all other offline RL methods fail. One limitation of IDRL is the increase of training time due to running multiple iterations. Another limitation is that IDRL may suffer from generalization issues when the data size is small. One future work is extend IDRL to the online setting where Dual- RL naturally serves a principled off- off-policy method.
weweiiinnnttt-weihhhhe imitation view of solving offline RL.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by NSF 2340651, NSF 2402650, DARPA HRO0J 12490431, and ARO W91 INF-24-1-0193, The authors would like to thank the ICLR reviewers for their constructive feedback.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Proc. of NeurIPS, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang. Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driver offline reinforcement learning. In Proc. of ICLR, 2021.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. Machine Learning Proceedings 1995, pp. 30-37, 1995.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. university press, 2004.
Cambridge
David Brandfoonrenerr, William F Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Proc. of NeurIPS, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith W. Ross. BAIL: best-action imitation learning for batch deep reinforcement learning. In Proc. of NeurIPS, 2020.
Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang. Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. In Thirty- Conference on Neural Information Processing Systems, 2023.
Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continu- ous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems, 34:1 10111120-1110 1273, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv preprint, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. ArXiv preprint, 2021.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proc. of ICML, pp. 1582-1591, 2018
Scott Fujimoto, David Meger, and Doina Precup. Off-pooicy deep reinforcement learning without exploration. In Proc. of ICML, pp. 2052-2062, 2019.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 0028-4039, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proc. of ICML, pp. 1856-1865, 2018.
Philippe Hansen-Estruch. Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning as an actor-critic method with diffusion policies. arXiv preprint MrXiv:2304.10573. 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
11

---

Published as a conference paper at ICLR 2025
Zhang- Zhang-Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Hamesssing mixed offline reinforcement learning datasets via trajectory weighting, arXiv preprint vrXiv22306.13083 2023a.
Zhang- Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Of- fline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023b.
Xiaohan Hu, Yi Ma, Chenjun Xiao, Yan Zheng, and Jianye Hao. Iteratively refined behavior regularization for offline reinforcement learning. arXiv preprint arXiv: 2306.05726, 2023.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. ArXiv preprint, 2021.
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjiban Choudhury, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. ArXiv preprint, 2019.
Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang. Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In Proc. ICL,, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of ICLR, 2015.
Uya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In Proc. of ICLR, 2020.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In Proc. of ICML, pp. 5774-5783, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. ArXiv preprint, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Proc. of NeurlPS, pp. 11761-11771, 2019. 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine Conservative q-learning for offline reinforcement learning. In Proc. of NeurlPS, 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? arXiv preprint arXiv:2204.05616 2022.
Jongmin Lee. Wonscok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In Proc. of ICML, pp. 6120- 6130, 2021.
Jongmin Lee, Cosmin Paduraru, Daniel J Mankowitz, Nicolas Heess, Doina Precup, Kee-Eung Kim, and Arthur Guez. Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation. arXiv preprint 2ttti:::2204089551 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv preprint, 2020.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022a.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning. arXiv preprint arXiv:2305. 15669, 2023.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Dealing with the unknown: Pessimistic offline reinforcement leaming. In Conference Robot Learning, pp. 1455-1464. PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2025
Ziniu Li, Tian Xu, Zeyu Qin, Yang Yu, and Zhi-Quan Luo. Imitation learning from imperfection. Theoretical justifications and algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Shirong Liu, Chenjia Bai, Zixian Guo, Hao Zhang. Gaurav Sharma, and Yang Liu. Selfbe: Self behavior cloning for offline reinforcement learning. arXiv preprint arXiv:2408. 02100, 2024.
Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching. arXiv preprint arXiv:2202.02433. 2022a.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via f-advantage regression. arXiv preprint arXiv:2206.03023. 2022b.
Alan S Manne. Linear programming and sequential decisions. Management Science, 6(3):259 1960.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via arthogonal-gradingg update. arXiv preprint 4xXx:22402.003488 2024a.
Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109 2024b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529-533 2015.
Ofir Nachum and Bo Dai. Reinforcement learning via rockhel-r duality, arXiv preprint arXiv:2001.01866, 2020.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Proc. of NeurlPS, pp. 2315-2325, 2019a.
Ofrr Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. ArXiv preprint, 2019b.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. ArXiv preprint, 2020.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, Xianyuan Zhan, et al. When to trust reinforcement learning. Advances in
your simulator: Dynamics- aware hybrid Neural Information Processing Systems, 15:36599-36612. 2022.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.0 09329, 2024.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Davantage-weigheee regression: Simple and scalable off-policy reinforcement learning. ArXiv preprint, 2019.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: tale of pessimism. Advances Neural Information Processing Systems, 34:11702-11716, 2021.
R Tyrrell Rockafellar. Convex analysis. Princeton university press, 1970.
Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Nickum. Score models for offline goal-conditioned reinforcement learning, arXiv preprint arXiv:2311.02013. 2023a.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint 23X2v:230..08500, 2023b.
Harshit Sikchi, Caleb Chuck, Amy Zhang, and Scott Nickum. A dual approach to imitation learning from observations with offline datasets. arXiv preprint arXiv:2406. 08800, 2024.
13

---

Published as a conference paper at ICLR 2025
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. 2020.
Richard S Sutton, Andrew G Barto, al. Introduction reinforcement learning. MIT press Cambridge, 1998.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances neural information processing systems, 12, 1999.
Richard S Sutton, Hamid Maci, and Csaba Szepesvári. A convergent o(n) canporal-differrnce algo- rithm for off-policy learning with linear function approximation. Advances in neural information processing systems, 21, 2008.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Masatoshi Uchara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation rein- forcement learning. arXiv preprint arXiv: 2212.06355, 2022.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh Merel, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y. Siegel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Proc. of NeurlPS, 2020.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 1992.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv preprint, 2019.
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offline reinforcement learning with soft behavior regularization. ArXiv preprint, 2021.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 35, pp. 4085-4098, 2022a.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weightedd offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp- 14725-24744 PMLR, 2022b.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu, Constraints penalized q-learning for safe offline rein- forcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.
2022c.
Haoran Xu, Li Jiang. Jianxiong Li, Zhuoran Yang. Zhaoran Wang, Vietor Wai Kin Chan, and Xianyuan Zhan. Offline with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303.1 15810, 2023.
Mengjiao Yang. Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems, 33:6551-6561, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. In Proc. of NeurlPS, 2020.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin. and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 4680-4683 2022.
14

---

Published as a conference paper at ICLR 2025
Hongchang Zhang, Jianzhun Shao, Yuhang Jiang. Shuncheng He, Guanwen Zhang, and Xiangyang Ji. State deviation correction for offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 9022-99999 2030, 2022.
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, and Dacheng Tao. Saformer: A conditional sequence modeling approach to offiie safe reinforcement learning. arXiv preprint arXiv:2301.12000 2023.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In International Conference on Learning Representations, 2019a.
Shangtong Zhang, Wendelin Boehmer, and Shimon Whiteson. Deep residual reinforcement learning.
arXiv preprint 1rXivv 00072, 2019b.
Shangtong Zhang, Bo Liu, and Shimon Whiteson. Gradientdice: Rethinking generalized offline estimation of stationary values. International Conference on Machine Learning, pp. 11194- 11203. PMLR, 2020.
Yinan Zheng. Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with reasibility-guiddes diffusion model. In The Twelfth International Conference Learning Representations, 2024.
Wenxuan Zhou, Sujay Bajracharya, and David Held. Latent action space for offline reinforcement learning. In Conference on Robot Learning, 2020.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33:12402- 12413, 2020.
15

---

Published as a conference paper at ICLR 2025
A AN EXTENDED INTRODUCTION OF DUAL-RL METHODS
A.1 DERIVATION OF DUAL-RL
Dual-RL algorithms consider the following regularized RL problem as a convex programming problems with Bellman-flow constraints and apply Fenchel-Rockfellet duality or Lagrangian duality to solve The regularization term aims imposing visitation distribution constraints (Nachum &
Dai 2020
Lee et al. 2021 Mao et al. 2024a).
max (s.a)=ar [r(s.)]] - (s.a))ls
Dy(d (s. (s, a)) the f-divergence which is defined with Dy - EuEQ [s P(w) Q(w)
Directly solving '² is impossible because it's intractable calculate (s, a). However, one can change the optimization variable from π to d' because of the bijection existing between them. Then with the assistance Bellman-flov constraints, we can obtain an optimization problem with respect to d:
-a. (s,a)) (s..a))) d20
s.t. Σ = + Σ d(s', a' a') (o',a")
aEA
Note that the feasible region has be {d Vs E S,a A, A,,(s, a) 2 0) because should be non-negative to ensure a valid corresponding policy. After applying Lagrangian duality, we can get the following optimization target following (Lee al. 2021):
min max E ((s.)) V(+) d20
(d(ss
(( +y
d(s' d(s''a))pssls',a)) - Ed(s,a)) (sr,ar)
Σ
= min max(1 - [Vsss))
((e) =20
E. Eaand
+ Ex(s)s, aB an [))]
Here we denote w(s, a) as (6.0) for simplicity. By incorporating the non-negative constraint
d(s.))
and again solving the constraint problem with Lagrangian duality, we can derive the optimal solution w" (8,a) for the inner problem and thus reduce the bilevel optimization problem the following optimization problem:
[ ((8,a) (T((((,))) + E, p(8'1s, (V -V())
min(1 (V( aB V(s)
a
Here f a variant of f's convex conjugate as fØ max (0, fx-) (x) - f (max (
A.2 DIFFERENT DUAL.-RL METHODS
Note that in the offline RL setting, the inner summation Σ. p(s'|s, ((s()s (s')) is usually intractable because of limited data samples. To handle this issue and increase training stability, semi- eradiett Dual-RL methods usually additional network Q(s,a) a) to fit r(s, + Σ. p(s' (s'),, by
optimizing the following MSE objective:
min E )- (r((s + - Q(s,
In doing so, the optimization objective in A. can be replaced with:
min V
Eg-ar [[1 (x)) + B(s.))aa [a (IQ(s, - V(s)) /a)]-
(13)
Also note that to increase the diversity of samples, one often extends the distribution of initial state do dd by treating every state in trajectory initial state (Kostrikov al. 2020).
16

---

Published as a conference paper at ICLR 2025
Recently, another Dual-RL method O-DICE (Mao et al. 2024a) proposed improve semi- gradient Dual-RL. O-DICE uses orthogonal-gratte gradient update to Dual-RL, which adds projected backward gradient to semi cemi-gradient. The projected backward gradient can be written
VoV(s) VoV(s') (s)
= =
[+)22
Intuitively, the projected backward gradient will not interfere with the forward (semi) gradient while still retaining information from the backward gradient. After getting the projected backward gradient, O-DICE adds it the forward gradient term, resulting new gradient flow as
Vos (fV(s, -
(f*)(r + +V(() V())) (7 V V(8) VoV(s))
where O-DICE uses one more hyperparameter 0 to control the strength of the projected backward gradient against the forward gradient. In theory, 7 needs to be large enough to guarantee a convergence.
B PROOFS
Lemma 2. (1 - -s) (V(()) = ((s.))= (V(s) - aV)
Proof. The proof can be simply obtained by using the definition of state visitation distribution de at timestep t following the behavior policy μ..
(1 Esndo [V()] = (1 -)) Σ
> E Eends ((1)))((1-))
Σ
Eande [V(s
1=0
t=0
50
(1 11-7) Σ
7 Bgnds [V(s) -TByNT(A,)) [V(')]]
1=0
ds (Vs)) TV(ss
Theorem 1. Given an action distribution ratio w (a(ss,, we can recover its corresponding state visitation distribution ratio w" (s)
w (8) (s)
(( (als (r))))
where U is the optimal solution of the dual form of as following.
min do U(s)) (s, a)] + E Eangr f Buun [onem[[orr (als) (TU(s, -
U
Proof. We first reframe as
max g(-Ad) h(d)
where Ad) corresponds to the linear constraints with respect the adjoint Bellman operator,
= ((1-7)46) and A == T. - I.
When applying Raakallla duality, the linear operator is transformed to its adjoint
A, 7 T and is used an argument to the Fenchel conjugate h, (.) Ex of h. At the same time, g is replaced by its Fenchel conjugate g-(:) (1 Eu, [-].
The dual problem is therefore given by ( g- (+( + h- (A,
min (1 7) E,gddo (x)) + Eg-ar [J (Bang (als) (TU(s, a) - U())))
We can get Theorem by replacing (1 Bando [s)] with E (o (U(s) (U(s, a)) using Lemma 2.
17

---

Published as a conference paper at ICLR 2025
Lemma 1. Given random variable X and its corresponding distribution P(X), for any convex func- tion f(),, the following problem is convex and the optimal solution is y" = (f*) (E (Em=)P(x))((0]]
min Bex - g(x) wl.
Proof. Taking the derivative of T(y) with respect y gives:
L(v) = P(x) (f'(w) g(x)]
dy
Setting the derivative to zero for optimality:
('(w") P(x) (g(x))
Since f is convex, f' invertible, and the optimal y" is:
y" = (8" (ExxP(x)(((t)))
Theorem 3. Given horizon length H and the dataset sample size of 'D as ND, the behavior cloning performance bound of IDRL at iteration ie given by
V (T) = V (Dx+1) - -O(N.,, + ND. - Dat1/ max, 1
S
Proof. The proof is highly built on the Theorem 3 in Li al. (2024), which gives the performance bound doing imitation learning on expert dataset De plus a supplementary dataset Ds. If one uses the visitation distribution ratio d a)/d (s,a) to do weighted behavior cloning, the imitation gap bound is given by
[V V (SW = =o ( 1S|H2
(14)
NE + Ns/u
(c.=
our case, we can view IDRL at iteration as doing imitation learning on "expert" dataset Dx41 plus supplementary dataset Dx+1 Dx. and we have the learned visitation distribution ratio w (8) (s)/ (s). Put these results in can get Theorem 3.
where μ = (aX(s.))ESX(M)] dg (x. (w))
Theorem 4. We have V(D++1) 2 (Dz) after the k-th iteration of IDRL.
Proof. Assuming the reward function is bounded, i.e, r(s,a) E [0, Rmax). Note that V(D)
Eg (e,))rrss a)] and because d the solution
dz = arg max (d(a,))(((ss a)]
d
so we have
E
(t) = max E -
2 [rs -
Using these inequality and positivity of fddivergence, it follows that:
((D++1) (s.o) (r a)) 2 Eap(s, (r(s, a)] + (s.
2 B (rss, a)] V(Dx)
18

---

Published as a conference paper at ICLR 2025
C
EXPERIMENTAL DETAILS
For the Pearson x² choose in practice, the corresponding f. (f')
has the following form:
((x) - - 1)2; (w) y( + 1); (R)
R + 2
We apply trick from Sikchi et al. (2023b) that rewrites objective from
min [1 (s)(s) + (f((Q(s,a) - v(s))
where E (0, 1) trades off linearly between the first term and the second term. This trick makes hyperparameter tuning easier as has nonlinear dependence through the non-linear function fü
Toycase experimental details The dataset consists of 500 transitions collected random policy, resulting in discretized state transitions. The weighted-BC policy, learned from filtered transitions, outputs continuous actions, illustrating the effect weighted-BC on each transition. Both the policy and value networks are 3-layer MLPs with 256 hidden units. We used the Adam optimizer (Kingma Ba 2015) with a learning rate of x 10-4 -4, The was set to 0.6 to both iterations, each running for 10"h steps. The first 500k steps of each iteration dedicated to learning the action ratio, followed by the remaining steps to optimize for the state-action distribution ratio.
discrete
D4RL datasets experimental details For all tasks, we conducted our algorithm for 2 iterations with 10t steps for each iterations. First 500k steps of each iteration are dedicated learn the action ratio, followed by the remaining steps to optimize for the state-action distribution ratio. In Mujoco locomotion tasks, we computed the average mean returns over evaluations every 5 10* training steps, across 7 different seeds. For Antmaze and Kitchen tasks, calculated the average 50 evaluations every 5 10a training steps, also across 7 seeds. Following previous research, we standardized the returns by dividing the difference in returns between the best and worst trajectories in MuJoCo tasks. In AntMaze tasks, we subtracted 3 from the rewards to better fit with our algorithm.
For the network, we use 3-yayer MLP with 256 hidden units and Adam optimizer (Kingma & Ba 2015) with a learning rate x 10-4 for both policy and value functions in all tasks. We also use target network with soft update weight 5 10 for Q-function.
We implemented IDRL using PyTorch and ran it on all datasets. We followed the same reporting methods mentioned earlier. Baseline results for other methods were directly sourced from their respective papers. In IDRL, we have one parameters: A. Because larger À indicates a stronger ability search for optimal actions, we select a larger if the value of doesn't diverge. The values of À for all datasets are listed in Table
Table 3: À used in IDRL
Dataset
halicheetal-meciinn-2
0.5
hopper-medium--2
0.6
0.5
halicheethh-mmddiin-eeeaaa--
0.6
hopper-medium-rrpllay
0.6
walker226-mcdiuu--xppayyy
0.6
halichhetal-mmediin--epprrr//
0.5
hopper-meddium--xxpe-t-
0.5
walker2blmmedium--erprrr-2
0.5
antmaze-umaze-v2
0.6
autmaaz-umazze-diveree--y
autmazze-mcdiummplaay-77
0.7
autmazee-nedium--divvreee--
0.7
antmazz-large-llayyv>
0.7
autmazelarge-diverse.->>
0.8
19

---

Published as a conference paper at ICLR 2025
Corrupted demonstrations experimental details In this experiment setting, we introduce the noisy dataset by mixing the expert and random dataset at varying expert ratios using Mujoco locomotion datasets, thereby simulating dataset collected with most low-perforning behavior policies. The number total transitions of the noisy dataset is 1, 000, 000. We provide details in Table
Table Noisy dataset of MuloCo locomotion tasks with different expert ratios.
Eav
Expert ratio
Total transitions
Expert transitions
Random transitions
1%
1,000,000
10,000
990,000
Walker2d
5%
1,000,000
50,000
950,000
10%
1,000,000
100,000
900,000
1%
1,000,0000
10,000
990,000
Halfcheetah
5%
1,000,000
50,000
950,000
1,000,000
100,000
900,000
1%
1,000,000
10,000
990,000
Hopper
5%
1,000,000
50,000
950,000
10%
1,000,000
100,000
900,000
D ETHICS STATEMENT
While IDRL has positive social impacts by helping to solve various practical data-driven decision making tasks, such as robotics, healthcare, and industrial control, it important acknowledge ledge that some potential negative impacts also exist. If crucial transitions in the dataset are filtered out, the trained model could behave unpredictably even dangerously in certain situations.
surplry
Eio Pichectiin
000
Figure Learning curves IDRL. on D4RI. Mujaco Iocomotion datasets.
20

---

Published as a conference paper at ICLR 2025
Figure 5: Learning curves of IDRL. on D4RI. Antmaze datasets.
21


Reviews:
[{"review_id": "rwlaoUx0nL", "reviewer": "Reviewer_DfDQ", "summary": "The paper presents a new framework, Iterative Dual-RL (IDRL), which utilizes an optimal discriminator-weighted imitation approach to enhance offline reinforcement learning (RL). This method iteratively refines the dataset to approximate the optimal visitation distribution by filtering out suboptimal transitions, thus aiming to overcome limitations of previous Dual-RL methods. IDRL is evaluated on D4RL benchmarks and several corrupted datasets, showing promising improvements in stability and performance over existing offline RL methods.", "questions": "1.\tThe reviewer believes the author should provide more explanation on how the additional variance introduced by the training of the U and W networks affects the overall stability of the algorithm. \n2.\tIn Algorithm 1, the reviewer wonders whether the W network is updated based on (12) rather than (10) on line 10? \n3.\tThe reviewer is confused about the value of M used in the experiments, and considers further clarification is needed here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe motivation of this paper is interesting and meaningful, which tries to combine offline RL with expert datasets.\n2.\tThe proposed IDRL offers a novel discriminator-weighted imitation view that extends Dual-RL to better handle offline datasets by iteratively optimizing the dataset. \n3.\tDetailed theoretical derivations and empirical validations make the methodology clear and support the proposed approach’s effectiveness. \n4.\tThe empirical results show that IDRL outperforms Primal-RL and existing Dual-RL methods on various benchmarks, indicating IDRL’s superior policy performance and dataset filtering effectiveness.", "weaknesses": "1.\tThis paper misses some literature in RL trained with weighted loss, such as EDP, and QVPO [1, 2].\n\n[1] Kang B, Ma X, Du C, et al. Efficient diffusion policies for offline reinforcement learning[J]. Advances in Neural Information Processing Systems, 2024, 36.\n\n[2] Ding S, Hu K, Zhang Z, et al. Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization[J]. arXiv preprint arXiv:2405.16173, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FiMDZzQS8n", "reviewer": "Reviewer_oApW", "summary": "This paper presents IDRL (iterative Dual RL), an algorithm for dual reinforcement learning that aims to solve two issues in current dual RL methods -- the semi gradient update and data regularized policy extraction. IDRL is a method which iteratively refines the dataset based on a trained discriminator. The paper proves both a theoretical iterative update guarantee and empirically shows that this method has superior performance compared to primal RL and dual RL offline methods.", "questions": "- How does this policy generalize to states that were filtered out?\n- How does filtering the dataset in round $k$, change the approximation of previously removed s,a pairs in later rounds?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This paper does a good job with outlining the main issues with current dual RL algorithms and provides a theoretically grounded solution.\n- While the idea is simple, it is well explained and well founded.\n- The proposed method also has strong empirical results.", "weaknesses": "- It is unclear whether this method will suffer from poor generalization to other states which may have been ignored during dataset filtering.\n- Further, this method seems to be computationally more expensive compared to other methods. It would be nice if this was discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ts1uwTSAsp", "reviewer": "Reviewer_DNBG", "summary": "The authors point out that current Dual-RL methods incorrectly estimate the visitation distribution ratio. As a remedy, they propose a method to recover the true visitation distribution ratio by solving an OPE problem using Fenchel-Rockafellar duality. Additionally, they introduce a method to iteratively refine the offline dataset using the learned distribution ratio. They theoretically analyze the performance bound and the monotonic improvement property of the filtering procedure. The authors perform experiments on a gridworld toy case and the D4RL benchmarks to validate their claims.", "questions": "1. Equation 12 shows that $w^{*}(s, a) = w^{*}(s) * w^{*}(a | s)$, which implies that state-action pairs filtered by $w^{*}(a | s)$ would also be filtered by $w^{*}(s, a)$. If $w^{*}(a | s)$ produces fragmented trajectories during dataset refinement, the trajectories produced using $w^{*}(s, a)$ will only be more fragmented. Also, from looking at Figure 2(e), it appears that using $w^{*}(s, a)$ produces incomplete trajectories as well. How does correcting the visitation distribution address the fragmented trajectory problem?\n2. Line 240 states that Deep RL algorithms are prone to overestimation errors caused by fragmented trajectories. Is this conclusion based on a previous study? To the best of my knowledge, the \"stitching\" challenge (which is a task design factor of D4RL) requires offline RL algorithms to assemble sub-trajectories in order to solve a task [1].\n3. (Line 263, 283) Which equation are you referring to? I assume it is Equation 9?\n4. Does the \"IDRL w/ $w^{*}(a | s)$\" result in Table 2 apply the iterative refinement procedure? If so, does iterative refinement contribute negatively with $w^{*}(a |s )$? Without distribution correction, one might expect the algorithm to produce results similar to conventional Dual-RL methods (e.g., IQL). However, the average score in Table 2 seems to be significantly worse (56.8 vs. 77.8 of IQL on Mujoco). A more detailed ablation study may help.\n\n[1] Fu, Justin, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. 2020. “D4RL: Datasets for Deep Data-Driven Reinforcement Learning.” _arXiv Preprint arXiv:2004.07219_. [http://arxiv.org/abs/2004.07219](http://arxiv.org/abs/2004.07219).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work theoretically demonstrates that semi-gradient Dual-RL only learns an action-distribution ratio, and derives a method for recovering the full state-action visitation ratio with tractable objectives.\n2. The proposed iterative filtering procedure is supported by theoretical analysis and empirical evaluations.", "weaknesses": "1. There should be a comparison of compute costs (e.g., run time, memory usage), given the substantial amount of modifications introduced (e.g., additional updates and iterative dataset refinement).\n2. The proof of Theorem 1 lacks clarity for readers not familiar with Fenchel-Rockafellar duality, as the authors have omitted some details (e.g., solving for $w^{*}(s)$). A more detailed explanation would be helpful.\n3. Line 240 states that Deep RL algorithms are prone to overestimation errors caused by fragmented trajectories. And the authors claim that the proposed method avoids this issue (Line 448-449). However, this fragmentation effect does not seem to be supported by any theoretical/empirical analysis in the paper or in a previous work. Please cite relevant texts if any.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPRcuP6y6y", "reviewer": "Reviewer_msFL", "summary": "This paper proposes Iterative Dual-RL (IDRL), a new algorithm for solving offline RL. The paper claims that an iterative \"filtering weight\" for imitation learning outperforms other offline RL methods. This point can be understood with the well-known dual formulation of RL, and by iterative self-distillation, the authors argue that RL can gradually correct state-action distribution between train and expert datasets. To validate this claim, the authors have provided theoretical justification for IDRL techniques and experimental results to support their claims.", "questions": "1. Is the denoising process of diffusion-based offline RL (such as Diffusion-QL) similar to filtering datasets? How is IDRL conceptually different?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written; the core contribution is straightforward to understand and supported by the theoretical arguments (I did not fully read the proof line-by-line). \n2. This paper offers a novel perspective on offline RL. The authors successfully demonstrated that combining the dual formation of RL and imitation learning algorithms brings synergy to solve various tasks.\n3. The paper contains realistic imitation learning experiments with corrupted datasets.", "weaknesses": "1. I believe the performance of IDRL is not stellar. Considering that the algorithm requires multiple dataset filtering steps, the performance gains from extra computation might not necessarily suggest the significance of the results.\n2. Since it works with filtering, the algorithm might fail in scarcity of data. In MuJoCo, single demonstration imitation learning is a standard setting. In this case, I suspect IDRL's performance will converge with (single demonstration) behavior cloning performance.\n3. An analysis scalability (such as computation costs) of various offline RL tasks should be reported and experimentally validated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)