Mind the Gap: Offline Policy Optimization for Imperfect Rewards
Authors: Jianxiong Li
Xiao Hu
Haoran Xu
Jingjing Liu
Xianyuan Zhan
Qing-Shan Jia
Ya-Qin Zhang
Venue: ICLR 2023 poster (2023)
Keywords: offline policy optimization
imperfect rewards
reward gap

Abstract: Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, \textit{RGM (Reward Gap Minimization)}, which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves a pessimistic RL problem with the corrected rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions.  Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong or inconsistent rewards against expert preference and retrieve useful information from biased rewards. Code is available at https://github.com/Facebear-ljx/RGM.

Published as a conference paper at ICLR 2023
MIND THE GAP: OFFLINE POLICY OPTIMIZATION FOR IMPERFECT REWARDS
Jianxiong Li Xiao Hu Haoran Xun,Jingjing Liu Xianyuan Zhan -0t Qing-Shan Jia & Ya-Qin Zhang
Tsinghua University, Beijing, China o Shanghai Artificial Intelligence Laboratory, Shanghai, China (1i-jx21, hu-x21 @mails ts sainghua edu whanxianyuaneeeee wtw.ennaairrrriiuaaaaaaaa edu
ABSTRACT
Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously diffi- cult to design. many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, RGM (Reward Gap Mininization), which can smartly handle diverse types of imperfect rewards. RGM is formulated as bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves pessimistic RL problem with the corrected rewards. By exploiting the du- ality of the lower layer, we derive a tractable algorithm that enables sampled-base based learning without any online interactions. Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong inconsistent rewards against expert preference and retrieve useful information from biased re- wards. Code is available at https: g hubb
1iii-1xiGGMMM /RGM.
INTRODUCTION
Reward plays an imperative role in every reinforcement learning (RL) problem. It encodes the desired task behaviors, serving as guiding signal to incentivize agents to learn and solve a given task. As widely recognized in RL studies, desirable reward function should not only define the task the agent learns to solve, but also offers the "bread crumbs" that allow the agent to efficiently learn to solve the task (Abel et al., 2021; Singh et al., 2009; Sorg, 2011)
However, due to task complexity and human cognitive biases (Hadfield-Menell et al., 2017), accurately describing a complex task using numerical rewards often difficult or impossible (Abel et al., 2021; Li et al., 2019). In most practical settings, the rewards are typically "imperfect" and hard to be fixed through reward tuning when online interactions are costly or dangerous (Zhan et al., 2022). Such imperfect rewards are widespread in real-worlde applications and appear in forms such as partially correct rewards, sparse rewards, mismatched rewards from other tasks, and completely incorrect rewards (see Figure 1 for intuitive illustration ). These rewards either fail to incentivize agents to learn correct behaviors or cannot provide effective signals to speed up the learning process. Consequently, it is of great importance and practical value to devise versatile method that can perform robust offline policy optimization under diverse settings of imperfect rewards.
Reward shaping (Ng al., 1999) the most common approach to tackling imperfect rewards, but it requires tremendous human efforts and numerous online evaluations. Another possible avenue is imitation learning (IL) (Pomerleau, 1988; Kostrikov et al., 2019) or offline inverse reinforcement learning methods (IRL) (Jarboui & Perchet, 2021), by directly imitating or deriving new rewards from expert behaviors. However, these methods heavily depend on the quantity and quality of expert demonstrations and offline datasets, which are often beyond reach in practice. Another key challenge how to precisely measure the discrepancy between the given reward in the data and the true reward
"Equal contribution.
Cooresponnence Xianyuan Zhan, Qing-Shan Jia and Ya-Qin Zhang

---

Published as a conference paper at ICLR 2023
Win++1 Lose:1
Reward for read pertially
oyclng
Only tinray rewerds are soalatre the end of eploode
Muli-cas training data
larget tank
Rewand for got compately incomeet for
boving
Partially correct reward
Sparse reward
Multi task data sharing
Completely incorrect
Figure 1: Diverse settings of imperfect rewards.
of the task. As evaluating the learned policy's behavior under a specific reward function through environment interactions becomes impossible under the offline setting, let alone revising the reward.
In this paper, we investigate the challenge of learning effective offline RL. policies under imperfect rewards, when environment interactions are not possible. We first formally define the relative gap between the given and perfect rewards based on state-action visitation distribution matching (referred as reward gap), and formulate the problem as a bi-level optimization problem. In the upper layer, the imperfect rewards are adjusted by reward correction term, which is learned by minimizing the reward gap toward expert behaviors. In the lower layer, we solve pessimistic RL problem obtain the optimized policy under the corrected rewards. By exploiting Lagrangian duality of the lower-level problem, the overall optimization procedure can be tractably solved in a fully-offline manner without any online interactions. We call this approach Reward Gap Minimization (RGM). Compared to existing methods, RGM can: 1) evaluate and minimize the reward gap without any online interactions; 2) eliminate the strong dependency on human efforts and numerous expert demonstrations; and 3) handle diverse types of reward settings (e.g., perfect, partially correct, sparse, multi-task data sharing, incorrect) in unified framework for reliable offline policy optimization.
Through extensive experiments on D4RL datasets (Fu et al., 2020), sparse reward tasks, multi-task data sharing tasks and discrete-space navigation task, we demonstrate that RGM can achieve superior performance across diverse settings of imperfect rewards. Furthermore, we show that RGM effectively corrects wrong/incoonistent rewards against expert preference and effectively retrieves useful information from biased rewards, making it an ideal tool for practical applications where reward functions are difficult to design.
2 RELATED WORK ON DIFFERENT REWARD SETTINGS
We here briefly summarize relevant methodological approaches that handle different types of rewards.
Perfect rewards. Directly applying offline RI. algorithms is natural choice when rewards are assumed to be perfect for the given task (Fujimoto et al., 2019; Kumar et al., 2019; 2020; Xu et al. 2021; Fujimoto & Gu, 2021; Kostrikov et al., 2021a;b; Xu et al., 2022a; Li et al., 2023; Lee et al., 2021; Bai et al., 2021; Xu et al., 2023). However, specifying perfect rewards requires deep understanding of the task and domain expertise. Even given the perfect rewards, some offline RL methods still need to shift the rewards to achieve the best performance (Kostrikov et al., 202 la; Kumar al., 2020), which is shown be equivalent engineering the initialization of Q-function estimation that encourages conservative exploitation under offline learning (Sun et al., 2022).
Partially correct rewards. Reward shaping is the most common approach to handle partially correct rewards, by modifying the original reward function to incorporate task-specific domain knowledge (Dorigo & Colombetti, 1994; Randløv & Alström, 1998; Ng al., 1999; Marom & Rosman, 2018; Wu & Lin, 2018). However, these approaches follow trial-and-erros paradigm and require tremendous human efforts. Recent approaches such as population- based method (Jaderberg et al., 2019), optimal reward framework (Chentanez et al., 2004; Sorg et al., 2010; Zheng et al., 2018 8) and automatic reward shaping (Hu et al., 2020; Devidze et al., 2021; Marthi, 2007) can automatically shape the rewards when online interaction is allowed. However, to the best of the authors' knowledge, no reward shaping or correction mechanism exists for offline policy optimization. Researchers have to discard the given imperfect rewards and resort other stopgaps like offline II. under offline settings.
Sparse rewards. Sparse rewards can be seen as a special case of partially correct rewards. The key challenge of offline policy optimization for sparse rewards is how to effectively prooppro the sparse signals to stitch up suboptimal trajectories (Levine et al., 2020). Recent works (Kostrikov et al., 2021b; Kumar et al., 2020) use reward shaping densify the sparse rewards for better performance.
2

---

Published as a conference paper at ICLR 2023
However, reward shaping requires online evaluation and tuning, which is not applicable in the offline setting. Currently, few mechanisms are specifically designed for offline RL to handle sparse rewards.
Imperfect rewards in multi-task data sharing. Sharing data across different tasks can potentially enhance offline RL performance on a target task by utilizing additional data from other relevant tasks. As the goals of other relevant tasks different from that of the target task, the rewards designed for other tasks are naturally imperfect for solving the target task. Since directly sharing datasets from other tasks exacerbates the distribution shift in offline RL (Yu et al., 2021; Bai et al., 2023), prior work such as CDS (Yu et al., 2021) shares data relevant to the target task based on learned Q-values, but it requires access to the functional form of the reward for relabling. CDS+UDS (Yu al., 2022) directly set the shared rewards to zero without reward relabeling to reduce the bias in the shared rewards, but it cannot completely remedy the reward bias.
Completely incorrect rewards. When rewards are believed to be totally wrong or missing, re- searchers typically adopt offline imitation learning (IL.) methods. These methods directly mimic the expert from demonstrations without the presence of reward signal. Among these approaches, behavior cloning (BC) (Pomerleau, 1988; Florence et al., 2022) is the simplest one, but is vulnerable to covariate shift and compounding errors (Rajaraman et al., 2020). Recent works tackle this problem via distribution matching (Jarboui & Perchet, 2021; Kostrikov et al., 2019; Kim et al., 2021; Ma et al., 2022) or using discriminator measure the optimal level of the data and further guide policy learning (Zolna et al., 2020; Xu et al., 2022b; Zhang et al., 2022). These approaches all have strong requirements the size and coverage of the expert datasets, and only try to imitate the expert rather than improve beyond the policies in data via RL based on the underlying reward of the task.
3 PRELIMINARIES
Markoy decision process under imperfect rewards. We consider the typical Markov Decision Process (MDP) setting (Puterman, 2014), which defined by a tuple M := AS, T, Ho, 7). S and A represent the state and action space, r : S x A + R is the perfect reward function, T S x A + (SS) is the transition dynamics which represents the probability T (s 18, as) of the transition from state st state $141 by executing action a₂ at timestep t. Mo A(S) is the distribution of the initial state sp. and 7 E (0, 1) is the discount factor.
The perfect reward function r(s, a) encodes the desired behaviors of the task. But in most cases, only have access to an imperfect human-designed reward function f(s, a), which may not align well with the target task. This leads to biased MDP M == (S, A, T, Moo 7) compared to the original MDP M. To remedy the adverse effects of imperfect reward signals, existing offline policy learning studies (Zolna et al., 2020; Xu et al., 2022b; Ma et al., 2022; Kim et al., 2021; Jarboui & Perchet,
N
E 2021) introduce additional expert demonstrations DE = {sf a a .f..... ....) (4)
to provide extra
imn
information on the desired policy behaviors. We follow a similar setup, but only consume very limited expert demonstrations. In our offline policy optimization setting, we are given pre-collected
that is generated by an unknown behavior policy T and dataset D = { (No, 4o, Fo, .....)))))) ==0
annotated with imperfect rewards F. We aim to learn an effective policy म, S + A(A) to capture the optimized agent behavior in M rather than M using both D and very small expert dataset E
Reinforcement learning. Given a MDP and the reward function r(s, a). the goal of RL to find an optimized policy गद to maximize the expected cumulative discount reward: , = arg max(1
E 7'r (St, ar) |so ~ po(+), a, ~ T, (-1st): 82411 T (-|s,, a.). This optimization objective can be equivalently written into the following succinet form (Puterman, 2014; Nachum et al., 2019b) by defining the normalized discounted state-action visitation distribution dd a) (in the rest of the paper, we omit "normalized discounted satte-action" for brevity unless otherwise specified):
arg max ((s.))arr-(( a)]
(1)
2
(s = (1 71)7)
Σ 1=U
y Pr s, a. also po(:) T (-1ss), 8141 T a-))
3

---

Published as a conference paper at ICLR 2023
This RL objective is not directly applicable to offline setting, is no longer possible to sample from dd via online interactions, and serious distributional shift (Kumar et al., 2019) may occur without proper data-related regularization when learning from offline datasets. To tackle these problems, several recent works (Nachum et al., 2019h 9b; Nachum & Dai, 2020; Lee et al., 2021) incorporate a regularizer into Eq. (1) formulate pessimistic RL framework that is solvable in the offline setting:
arg max ((..a))dT- (r(s, a)] aD (d (lo))
where dd is the visitation distribution of dataset D, D.D(G (-11-) represents some statistical discrepancy measures and a > 0 controls the strength of the regularization.
4 REWARD GAP MINIMIZATION
To handle diverse imperfect reward settings, three challenges have to be tackled:
1) Measure the gap between the given rewards and the underlying unknown perfect rewards;
2) Unify different reward settings and bridge the reward gap:
3) Perform offline policy optimization using an integrated framework.
Our solution to these challenges is Reward Gap Minimization (RGM). We formally define the reward gap in the perspective of visitation distribution matching and introduce a correction term to correct the problematic rewards. Then, we model RGM as a bi-level optimization problem, with the upper layer minimizing the reward gap and the lower layer solving a pessimistic RL problem. To derive tractable algorithm, we leverage Lagrangian duality to eliminate the requirement for online samples.
4.1 DEFINITION OF REWARD GAP
As observed in recent literature, some tasks cannot be captured by a numerical Markovian reward function (Abel et al., 2021). Hence, learning an explicit proxy of the perfect reward function and comparing it to the given rewards is unlikely the best option to characterize the reward gap. In this study, we define the reward gap based on the outcome of the learned agent behavior, i.e., from the perspective of visitation distribution matching.
Definition 1. (Reward gap) Given an arbitrary reward rewardfunction function i(s, a) and the visitation distribution d' of the optimal policy induced from the perfect rewards r, the reward gap between i and is:
Dj (d=alld))
where Dy (pl(g) = Ezzg (1(3 is the f-divergence between distributions p and q and d= ((2)
represents the visitation distribution induced by nTp. which is derived using Eq. (2) with f.
Note that d= is unobtainable since the perfect reward function unknown. We can alternatively use the visitation distribution d induced by unknown n in expert demonstrations DE to approximate d'. Next, we discuss how to adjust f to minimize the reward gap.
4.2 BI-LEVEL OPTIMIZATION
Reward correction. In our study, we consider f(s, a) == F(s, + Ar(s, o, F), where Ar(s, a, f) is learnable reward correction term that is correlated with the given imperfect rewards F in D. The introduction of Ar(s, a, i) enables us to exploit useful information within the partially correct rewards, while also correcting the wrong or inconsistent reward signals. We can further use it to construct a bi-level optimization formulation for RGM, where the upper-level problem optimizes the reward correction term to minimize the f-divergence between d and d and the lower-level problem solves मे as the optimal policy of a pessimistic RL problem with the corrected rewards:
Ar arg min Dy (d= (1d=)
(4)
Ar
s.t.
arg max max (e.0) (f(s,a)) - aD, (d=)
(5)
x
4

---

Published as a conference paper at ICLR 2023
The above bi-level optimization formulation poses several technical difficulties, stemming from the complexity of deriving d' from x as well as the requirement of online samples from d which is impossible under the offline setting. In the following, we present reformulations for both lower and upper-level problems, which leads to tractable form and an casy-to-implement algorithm.
Reformulation the lower-level problem. We first reformulate the lower-level problem by exploiting duality and the Bellman flow constraint (Puterman, 2014).
Definition 2. (Bellman flow constraint) Let T.d(s) - E T(s|s, a)d(s, a) denote the transpose (or adjoint) transition operator, the Bellman flow constraint for the visitation distribution d(s,, (s) is:
Σ - -(-.)))=(1-))003) (Ts
(6)
(d((s,) a) 2 satisfies the Bellman flow constraint, then d(s, a) is feasible and there a one-to-one correspondence between d and the related policy i.c. d is the only visitation distribution for policy (aass = Ee d(s,a) while
the only policy whose visitation distribution is d (for detailed proof see Puterman (2014)). Then, the lower level problem Eq. (5) can be re-written to a constraint maximization problem w.r.t d in place of np!
d(a.0)
d
arg arg max ((s,a)=a((sssa)) , (dlldP) d(s,aa) = (11-))ooss) + 7Txd(s), e
d20
(7)
The Lagrange dual problem of Eq. (7) as follow:
min max ((na) a)] aD, (dlla)) VV(s) (1
+ Ed(s,a)
V(a) 420
where (s)) Lagrange multipliers. Note that the primal problem Eq. (7) convex w.r.t. and under a mild assumption (see Assumption in Appendix A.2), the Slater's condition (Boyd et al., 2004) holds, which means by strong duality, we can solve the original primal problem by solving Eq. (8), After rearranging the terms, Eq. (8) can be equivalently written as the following form (see Lemma 2 in Appendix A.2 for detailed deduction):
min max(1 Bo [V)) + E ((saa)a + +TV(s, V(s)] aD, (dlld) (e) d20
(9)
in which aV - Vs') denotes the transition operator. Next, by exploiting the Fenchel conjugate, we can further transform the minimax problem Eq. (9) into a tractable single-level unconstrained minimization problem (see Proposition I in Appendix A.2 for detailed derivation), which eliminates the requirement of online samples:
min(1 - Bo-pu (V + a B(o,a)=aD Sa( (. + +l,,,)) (VV V(s)
V(s))
(10)
a
where f. the Fenchel conjugate of f. In the above formulation, the Lagrange multipliers V(s) can be equivalently perceived as some sort of state-value function, which can be learned and opti- mized via a parameterized neural network, similar to the treatment used in the DICF-family of RI. algorithms (Nachum et al., 2019a; Nachum & Dai, 2020).
Reformulation of the upper-level problem. Using the property Fenchel conjugate, the optimal d' and V* from the lower level problem satisfy the following nice relationship (see Proposition 2 in Appendix A.3 for details):
d (s,
-1 (s) V=))
(f(s,a) (f(5,,)+ 77V
(11)
d (s,
Plugging the above equation into Eq. (5), we can obtain new objective for the upper-level problem:
Ar" arg min D, (f- f TTV - d (l1d
(12)
Ar
5

---

Published as a conference paper at ICLR 2023
RGM 1 Dieriminator Leaningg
Input
(10)
Output
wrseeteeeeeeerr with
Stodhartic Upper Level Problem Minnize the reward gap
segmin Beu In (X
(N.o)
/alna)) an
Lower Level Problem: Solve pessimistic RL offiine setting
V1 eppand 78 Viol oo [A (Mea) vf VM
Policy Estraction
Figure 2: Illustration of the reformulated bi-level optimization problem.
(++TVV~-V** as g. By expanding the f-divergence, we have:
For simplicity, we denote Si
Dy (d (dgglde) = (aa)-a
(s,a))(s, (s,
dd d (P (s,
(s
(13)
The above objective involves computing the distribution ratio w(s, 0) & d d a) (s,(/) tabular case, we can empirically estimate wi(s, a) = Daaaee ((==,.=a)//""
a). In the EGaDje 103=s,a=)/NN But in the continuous state-action settings, estimating the distribution ratio w using only samples from dd and d becomes a challenge. Inspired by previous studies (Goodfellow al., 2020; Ma al., 2022), we instead train a discriminator / S x A - (0, 1) to infer if (s, a) samples are from D or not:
h" arg min (s.a)aa (og(h(ss,))) + (ss,a)=== [log(1 h(s,a))]
(14)
where the optimal discriminator (8, a) (=(xaa)+ dF (Goodfellow et al., 2020). We can optimize the above objective to obtain the optimal h², and further recover w(s, a) 1/h (s, a)
(s.a)
Finally, combining the reformulations, the final tractable form of the original bi-level optimization problem Eq. (4)-(5) is given as follows:
Ar - min Ba w(s,a) / s(:: (fls.a) +TVV) VV*(s) / a
or
(15)
S.L. (a) arg min (1 Bo [V()) (a0) [5. (A(s,a)+ + TTV(s, (TV(.)) (((s) V(s))
V(m)
a
Policy extraction. With the learned reward correction term Ar(s, a, F). we can in principle use existing offline RI algorithms to learn the policy with the corrected rewards. However, this implicates additional policy evaluation and policy improvement steps. A more elegant way is to extract the policy through weighted BC as follows, which is substantially more robust and less expensive:
[d (s,a) a)
"² arg min E
(s.a)~a=; [log (a (a]s)) arg min E (s,a)dd
log m(als)
(16)
d (s,
where (e.e) can be calculated from Eq. (11).
2e (s.a)
4.3 PRACTICAL IMPLEMENTATION
In our implementation, we use stochastic first-order two-timescale optimization technique (Borkar, 1997), which has been successfully applied in several RL algorithms (Hong et al., 2020; Cheng et al., 2022), to solve bi-level optimization problems. Specifically, we make the gradient update step size of the upper layer much smaller than the one of the lower layer (see Figure for RGM framework. Refer to Appendix B for additional implementation details of RGM).
EXPERIMENTS
In this section, present empirical evaluations of RGM under diverse imperfect reward settings, including partially correct rewards, completely incorrect rewards, sparse rewards, and multi-task data sharing setting Robomimic (Mandlekar et al., 2021), D4RL-v2 (Fu al., 2020) and dataset of a grid-world navigation task. As D4RL MuJoCo tasks are deterministic, we use only one expert trajectory assist the reward correction and policy learning for these tasks.

---

Published as a conference paper at ICLR 2023
Table I: Average normalized scores of RGM compared with offline IL and RL baselines on D4RL datasets.
The scores are from the final 10 evaluations with 5 seeds. (T), (P) and (C) mean policy optimization with true rewards. partially correct rewards and completely incorrect rewards, respectively. "-r"."-m" "rr."-m".-mm-""" .--m"r,, and "-m-e" are short for random, medium, medium-replay, and medium-expert, respectively. We obtain the results by running author-provided open-source code, and some scores are reported from TD3+BC and IQI. papers. For each dataset, the top scores under partially correct rewards are marked in blue.
DMKL Dataset
Offinee IL
Offiine
KG/(T///C
BC
DWBC
SMODICE
TD34BC (T/P/C)
(QL/T/P/C)
CQL PP//CC
hopper
4.9
23.9
5.9
13.
0.4
7.9
1.3
0.7
8.3
1.7
0.0
29.7
21.2
25,9
halfchhetab r
0.2
2.0
2,6
11.0
-17.1
-11.6
11.2
2.2
2.2
20.0
-0.4
-38.4
0.2
0.2
0.2
w
1.7
68.3
0.2
1.6
2.0
5.9
0.3
0.3
8.3
0.1
-0.0
3.9
7.7
0.1
hopper m
52.9
59.3
13.7
34.0
58.5
11.2
56.2
55.5
54.6
halfchectab m
42.6
8.2
42.9
48.3
35.
1.2
47.4
42.0
35.4
44.0
43.5
4.1
40.4
40.7
40.3
walker2d m
75.3
18.8
1.0
83.7
30.1
172
78.3
68.9
22.0
72.5
71.1
3.5
73.3
72.3
38.4
bopger-m-r
18.1
20.4
60.9
23.5
16.3
94.7
0.7
95,0
0.0
59.1
44.5
halfcheetub-m-r
9.2
37.1
31.8
45.5
37.7
37.8
29.8
walker2d-m-r
26.0
56.6
41.1
81.8
7,8
-0.7
73.8
4.9
-0.2
77.2
17.4
-0.0
46.3
48.6
46.1
hopper-m-e
52.5
16.5
75,4
98.0
50.8
22.3
91.5
49.3
13.6
105.4
68.3
11.6
106.1
87.1
66.0
helfchoetab-m-ee
0.0
88.2
35.3
1.9
35.8
81.5
784
walker2d-m-e
107.5
54.3
29.8
110.1
44.7
6.8
109.6
108.3
20.9
108.8
75,4
16.6
108.3
108.8
108.8
Monn Score
39.5
33.2
58.2
24.6
7.7
59.8
32.0
14.0
60.5
52.0
419
5.1 COMPARATIVE RESULTS
Comparisons for partially correct rewards. We train RGM and SOTA offline RL methods (TD3+BC (Fujimoto & Gu, 2021), IQL (Kostrikov et al., 2021b) and CQL (Kumar et al., 2020)) under partially correct! rewards and report their performances evaluated based on the perfect rewards" in Table 1. Table 1 shows that RGM surpasses offline RL methods under partially correct rewards' by a large margin and achieves similar performance to offline RL policies that are trained on perfect rewards. This shows a remarkable advantage of RGM it can alleviate severe performance degradation when perfect rewards are unattainable and hence removes the restrictive requirements on perfect rewards, which can be particularly useful for a wide range of real-world scenarios.
Comparisons for completely incorrect rewards. When rewards are believed to be completely incorrect, one generally resorts to IL methods. We compare RGM with BC and SOTA offline IL methods (DWBC (Xu et al., 2022b) and SMODICE (Ma et al., 2022)) that can learn from mixed- quality data. Only offline IL methods are considered as baselines, because other existing methods that tackle incorrect rewards can only be applied in the online settings (see Section 2 for discussions).
In our setting, we train offline IL baselines using the original D4RL dataset D, which may not cover enough expert trajectories. However, DWBC and SMODICE both build on the strong assumption that DD already covers a large proportion of expert datasets, which is a rare case in real scenarios. As result, Table 1 shows that these two methods suffer from inferior performance when the restrictive requirements on the quality and state-action space coverage of expert data are not satisfied, RGM, however, performs well when nearly no expert trajectories are contained in the offline dataset, because RGM is optimizing an RL objective that relaxes the requirements on the quality of the dataset.
To further illustrate the superiority of RGM. we compare RGM against DWBC and SMODICE under their settings by adding 100~200 expert trajectories into D. Results show that RGM can still outperform SOTA offline IL. methods by a large margin (see Table 8 in Appendix D).
Comparisons for sparse rewards. We evaluate RGM against BC and offline RL methods TD3+BC, CQL and IQL on Robomimic (Mandlekar et al., 2021) Lift and Can tasks. We also evaluate on the well-known extremely difficult AntMaze tasks. We report the average max success rate as the evaluation metric in Table (See Appendix C.2 for task descriptions and experimental setups).
'The signs of 50% D4RL rewards are flipped and henee only half rewards give correct learning signals. We regard the original D4RL rewards as perfect since we evaluate the policies terms of these rewards, w which can be perceived as solving the tasks encoded in the original D4RL rewards. All sign of the original rewards flipped.
7

---

Published as a conference paper at ICLR 2023
Table 2 shows that the offline RL baselines fail miserably on AntMaze tasks*, as sparse rewards hard to back-propagee propagate through a very long horizon (Re 1K steps), while RGM can correctly provide dense signals to guide the ant navigate to the destination. For Robomimic Lift and Can tasks, RGM again outperforms existing methods, while other methods can also achieve reason- able performance. We suspect that these offline datasets may already contain near-optimal trajectories as BC can achieve reasonable performance. Moreover, the planning horizon of both tasks are relatively short (Rs 150 steps), thus is relatively simple for offline RI. to pack-prop the sparse signals.
Table 2: Results on sparse reward tasks.
Dataset
BC
TD3+BC
CQL.
IQL
RGM
Antmare-m-p
0
0
0
13.7
Lif MG
65.3
87.3
64.0
56.0
90.3
Can
Extension to multi-task data sharing. We highlight that RGM can also perform well in the offline multi-task data sharing tasks (Yu et al., 2021), which utilize datasets from other rele- vant tasks to enhance the offline RL performance on target task. Prior works either require the functional form of rewards to be known for re- labeling (Yu al., 2021) or partially correct the reward biases (Yu et al., 2022). In contrast, RGM systematically corrects the reward biases with- out reward relabelling, using just one expert trajectory from the target task. To demonstrate the efficacy of RGM compared to SOTA multi-task data sharing algorithms CDS (Yu al., 2021) and CDS+UDS (Yu et al., 2022), we conduct experiments in multi-task Walker (Stand, Walk, Run, Flip) and Quadruped (Walk, Run, Roll-Fast, Jump) domains built on DeepMind Control Suite (Tassa et al., 2018). For each task, use TD3 (Fujimoto et al., 2018) to collect three types of datasets (expert, medium, replay), and share the replay dataset of the relevant task with the medium dataset of the target task. For RGM, we only draw the experimental results in Figure 3, which shows that RGM substantially outperforms CDS and CDS+UDS (see Appendix C.3 and D.5 for more experiment details and results).
domath Mad--aeaaan pareetto
GOD
4000
200
CDS
COS-UDD
RGM
CDS
CDS-UDS
RGG
Figure Results on multi-task data sharing tasks.
expert trajectory for the discriminator training. We report
5.2 INVESTIGATIONS REWARD CORRECTION
Benefits of learned rewards. We investigate the potential benefits of the learned rewards via demonstrative experiments in 8x8 grid world environment. We observe the learned rewards in RGM enjoy three desirable properties that are unlikely to be provided other existing methods: 1) encode long horizon information: 2) correct wrong rewards against expert preference; and 3) retrieve useful information from existing rewards, as shown in Figure 4.
OM
800
300
(a) Expert demonstration
(b) Results of zero
(c) Results of partially correct F
Figure 4: Learned rewards and optimal distribution d=è trained on two types of imperfect rewards F. The opacity each square represents the value of marginal state distribution a= (s). The opacity of the arrow shows the learned reward F, where the darkest arrw points to the direction of the highest reward. The expert starts from follows the path and arrow to reach goal D. f (b) +1 +10 the goal and is zero at other states. F in (c) falsely punishes the agent X and correctly punishes the RL agent fire marks A
NNte that in IQI. and CQL. papers, they turn the original sparse rewards into dense rewards by applying the reward subtraction trick (minus every reward, so the reward becomes negative except the goal).
8

---

Published as a conference paper at ICLR 2023
CrU
Performance drop
Performance drop
0-
-30-
Performance
-30
-60
Whong recenss
-80
manrds
Whang
-90
Oarmwards
-90
Ourmeands
HC-F
H-m
HC m
W-m
H-n--
HC-m-r
Wm
Hmmo
HC-m- W-m-
H+
HC-1
W
HC m
W-m
Hum-
HC-m--
W
H-me
HC
Wime
H-r
HC-F
W-
H-m
m
Wim
H-m-
HC-m-r
W-me
Hmmo
HC-ie
Wen-
Figure 5: Performance drop of normalized returns of SOTA offline RL methods on D4RL datasets under perfect
RGM corrected rewards, wrong rewards are the partially correct rewards as Table H: Hopper; IC:
HalfCheetah; W: Walker2d.
Correcton term dr
expert
Per www.ergaaaaaee Ant
3.8
2-
3.3
0-
Comsuttion teres
60
Perfect
oon OORTOC
Incomrett
0.4
3,0
Steps pxt.ff
Training Steps (=10%)
Perfect
(a) .earning curve of Ar
(b) Effect of i on
(c) Effect of f on Ar
Figure 6: Experiments on learned rewards in hopper-m-r task. The superscript denotes the mean value of mini-batch samples. The subscript "E" and "O" denote the value on expert and non-expert data. In (b)(c), large îg to AFg - Año indicate that expert and non-expert data are clearly distinguishable according to the learned rewards, and small values mean the opposite.
Specifically, Figure 4b shows that the learned rewards not only recover correct learning signals on the path of the expert, but also generalize well regions not covered by expert data. In most locations, the agent can navigate to the destination by simply maximizing the one-step reward, meaning that the learned rewards encode long- morizoo information. Moreover, Figure 4c shows that the learned rewards can avoid the dangerous fire locations by retrieving useful information provided in imperfect meanwhile correcting the wrong rewards against expert preference.
Offline RL with corrected rewards. The learned corrected rewards f obtained by RGM can also be used in other offline RL approaches. To be mentioned, the corrected rewards are optimized based on the specific in Eq. (5), hence may not be optimal to other offline RL methods. Nevertheless, Figure shows that the corrected rewards can largely remedy the negative effects the partially correct rewards and even surpass perfect rewards in some datasets.
Additionally, we investigate the learned rewards in high- dimensional continuous control tasks by inspecting the learning process of both the reward correction term Ar and the final learned rewards f. Figure 6a shows that the reward correction term Ar initially cannot distinguish expert and non-expert data well, but adapts and converges quickly. After few training steps, Ar can correctly reward expert data and punish non-expert data very well. We also perform ablations the effect of diverse types of imperfect rewards that a perfect is beneficial to enlarge reward differences on expert and non-expert samples, and incorrect i can be counterproductive. Nevertheless, RGM can largely correct the wrong rewards and produce reasonable learning signals. Similar effects are also observed on Ar, as Figure 6c shows.
Ablations on learned rewards.
Ar and F. Figure 6b shows
DISCUSSION AND CONCLUSION
In this paper, we propose RGM (Reward Gap Minimization), a unified offline policy optimization approach applicable diverse settings of imperfect rewards. RGM is formulated as bi-level optimization problem, which achieves reward correction and simultaneous policy learning in a fully offline paradigm. Extensive experiments and illustrative examples show that RGM can perform robust policy optimization under imperfect rewards. Several desirable properties are also identified in the corrected rewards learned by RGM. One limitation of RGM is the need for small expert dataset, which may not be easily accessible in some applications. However, RGM relaxes the strong dependencies on online reward tuming and tedious human efforts, which renders it powerful tool to solve many real-world problems.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by funding from Ha,mo.AL, and National Natural Science Foundation of China under Grant 62 62125304, 62073182. The authors would also like to thank the anonymous reviewers for their feedback on the manuscripts.
REFERENCES
David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34:7799-7812. 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhen Wang, Zhaoran Wang, Bin Zhao, and Xuelong Li. Uncertainty-based multi-task data sharing for offline reinforcement learning, 2023, URL
https:///ppezeet
nwt forum? id=ulVje (e--uVv66cc22.....
Vivek S Borkar. Stochastic approximation with two time scales. Systems Control Letters, 29(5):
291-294, 1997.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, ICML 2022. 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 3852-3878. PMLR, 2022.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning, Advances in neural information processing systems, 17, 2004.
Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual embeddings. In Artificial Intelligence and Statistics, pp. 1458-1467. PMLR, 2017.
Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Explicable reward design for reinforcement learning agents. Advances in Neural Information Processing Systems, 34: 201 80118-200 101111,, 2021.
Marco Dorigo and Marco Colombetti. Robot shaping: Developing autonomous agents through learning. Artificial intelligence, 71(2):321-370, 1994.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on Robot Learning, pp. 158-168. PMLR, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 1444 2020.
10

---

Published as a conference paper at ICLR 2023
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Internationall conference machine learning. pp. 1861-1870. PMLR, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Mingyi Hong, Hoi-To Wai, Zhaoran Wang. and Zhuoran Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint a 2020.
Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang. Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Advances in Neural Information Processing Systems, 33:15931-15944 2020.
Max Jaderberg, Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance 3d multiplayer games with population-basee reinforcement learning. Science, 3644(444)):559-866 865, 2019.
Firas Jarboui and Vianney Perchet. Offline inverse reinforcement learning. arXiv:2/06.0500 05068. 2021.
arXiv preprint
Geon-Hyeong Kim, Scokin Sco, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok
Yang. and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. International Conference on Learning Representations, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off- aff-policy distribution matching. In International Conference on Learning Representations, 2019.
llya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 13:1179-1191, 2020.
Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning, pp. 6120-6130. PMLR, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In International Conference on Learning Representations, 2023. URL https: htttttts:///ppeeeee /ttp///ppp///e/e///////p/e wwt.eeewweeeeeeeeeeeeeeeeeeeeeee net/forumm
id=1M07TC7cuuis
Xiao Li, Zachary Serlin, Guang Yang, and Calin Belta. A formal methods approach to interpretable reinforcement learning for robotic planning. Science Robotics, 4(37):eaay6276, 2019.
Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference Machine Learning, pp. 14639-14663. PMLR, 2022.

---

Published as a conference paper at ICLR 2023
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martín. What matters in learning from offline human demonstrations for robot manipulation. In 5th Annual Conference Robot Learning, 2021.
Ofir Marom and Benjamin Rosman. Belief reward shaping in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Bhaskara Marthi. Automatic shaping and decomposition reward functions. In Proceedings of the 24th International Conference on Machine learning, pp. 601-608, 2007.
Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellaa duality. arXiv preprint arXiv:2001.01866, 2020.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Bchavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv: wwXiv:/9/2.02077 2019b.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Icml, volume 99, pp. 278-287, 1999.
Art B. Owen. Monte Carlo theory, methods and examples. 2013.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914-2922 2020.
Jette Randløy and Prehen Alstrøm. Learning drive bicycle using reinforcement learning and shaping. In ICMI,, volume 98, pp. 463-471. Citeseer, 1998.
R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pp. 2601-2606. Cognitive Science Society, 2009.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.0322 032.8888. 2021.
Jonathan Sorg, Richard Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010.
Jonathan Daniel Sorg. The optimal reward problem: Designing effective reward for bounded agents. PhD thesis, University of Michigan, 2011.
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploiting reward shifting in value-based deep rl. In Advances in Neural Information Processing Systems, 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: /8/1:0/800.0069. 2018.
Yueh-Hur Hua Wu and Shou-De Lin. A low-cost ethics shaping approach for designing reinforcement learning agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
12

---

Published as a conference paper at ICLR 2023
Haoran Xu, Xianyuan Zhan, Jianxiong Li, and Honglei Yin. Offliee reinforcement learning with soft behavior regularization. arXiv preprint arXiv:2110.07395, 2021.
Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for
offline reinforcement learning. Advances in Neural Information Processing Systems, 2022a.
Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator seseriminatorwweighteed offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pp. 24725 22742.. PMLR, 2022b.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan Zhan. Sparse q-learning: Offine reinforcement learning with implicit value regularization. In International Conference on Learning Representations, 2023. URL /ttp:rsv///peeevieeee Eorum? ? id=ueYYgo2pSSU.
net/ net/Eorum?
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Sergey Levine, and Chelsea Finn. Conservative data sharing for multi-task offline reinforcement learning. Advances in Neural Information Processing Systems, 34:11501-11516, 2021.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine How to leverage unlabeled data in offline reinforcement learning. In International Conference Machine Learning, pp. 25611-22111 1-26655, PMLR, 2022.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Wenjia Zhang, Haoran Xu, Haoyi Niu, Peng Cheng, Ming Li, Heming Zhang, Guyue Zhou, and Xianyuan Zhan. diseriminator-gudder model-based offline imitation learning. In Conference on Robot Learning. 2022.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in Neural Information Processing Systems, 31, 2018.
Konrad ZoIna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13880 2020.
13

---

Published as a conference paper at ICLR 2023
A PROOFS
A. BACKGROUND
We begin by briefly introducing the Fenchel conjugate (also known as convex conjugate Legen- dre Fenchel transformation):
Definition 3. (Fenchel conjugate) Fenchel conjugate fa of f at y is:
real Hilbert space X, if function f(x) is proper, then the
[+(U) sup(y" -xx)))
(17)
sEA
where the domain the f+(y) given by:
dom f+
y
sup #Edom/
(y (5-x- f(x)
(18)
If is convex and lower semi-continuous. well, we have the duality fx) f(x). Furthermore, f is also differentiable, then the maximizer x" of (y) satisfies:
x f1)
(19)
Next, present the interchangeabiiltyy principle, which plays key role in Proposition 1. Lemma 1. (Interchungeability principle) Let & be random vuriable on E and assume for any Ç E E, function g(:,)) E) is proper and upper semi-continuous concuve function, Then
Eg
max g(u, WER
max u())E(())
(ELg(((EE),
(20)
where ((E) = (u(-)= R} the entire space of functions defined on support E.
Proof. Please refer to (Dai et al., 2017; Rockafellar & Wets, 2009).
A.2 PROOF OF TRACTABLE TRANSFORMATION OF THE LOWER- PROBLEM
We start our proof from the original bi-level optimization problem Eq. (4) and Eq. (5). Using the Bellman flow constraint for Eq. (5) yields:
Ar" arg min Dj (dmilldE)
s.t. d = arg max ((s.))a(((s,a)) -DD (dlld)
(21)
d20
s.t. E = =1 (1)
Assumption 1. There exists at least one d such that:
dd(s,a) a) = (1-7)uo((s) + 7T,d(s), d(s) > 0, V
(22)
We note that this assumption is mild since when every state is reachable from the initial state distribution, the assumption is satisfied, which is common in practice.
Slater's theorem (Boyd et al., 2004) states that strong duality holds, if the optimization problem is strictly feasible (Slater's condition holds) and the problem is convex. So under Assumption I with the fact that the lower level problem is convex w.r.t. d, the strong duality holds, which means that the above lower level problem can be re-written as the following form:
min max ((s,))= -aD, (dld²)
EVss) (1 (1-2)u0()) 7)o())))))))))))))))))))) +T.d((s)- Ed(s,a))
(23)
V() 420
14

---

Published as a conference paper at ICLR 2023
Lemma 2. The minimax problem:
min max (fss))alf(ss a)] - -DD (dlld) + + V(s) (1 - + - Σ _d((saa) (24)
V(a) 420
can be equivalently written as:
min max (1 7) (o (V + Beaasaa +7V(s)))) -V(s)) aDy, (dlld)) (25)
((a) d20
Proof.
(((,a)_ aD, (dlaP) + V(1)
V())
(1 ())())(() + +Td((s) Ed(s,a)
= aD, (dlld²)
Σ V())
(1 7)) + Σ
T(sl,,
Ed(s,a)
d(s,a)
8.8
[ds - aD, (dlldP)
+ +1 -1-1 7) Σ (o(s)[(ss) + Σ d(s,ū)
((s)3 a((s)))((s))
5.0
5,0
5,a
Σ ((s,)) - ,D, (dlld)) + (1 (1->) > (s(s)V(ss) + Σ d(s, Σ T(s((s, - (((s)))((s))
=11 -1-7)) Σ (s)s)V((s)+ Ed(s,a) F(s,a) + 5 T(sls,.))Vs)) V(s)
-aD, (dlla²)
2.a
(1 - 7 [V()]] + E (f( + 7TV(s, V(s)]] aD, (dlld)
(26)
Proposition 1. The minimax problem:
min max (f(ss a)] - aD, (dlle) + Vv()) (1-2)00(s++777 Σ -d((s,)) (27)
V()) 420
shares the same optimal value as the following minimization problem:
min(1 - Barppa + a B(s fa( ((s, + ( v(s)) -
(28)
where f. is the Fenchel conjugate function of f with dom {u 0}
Proof.
Using Lemma 2, this minimax problem can re-written as:
min max(1 - Es [V(s)] V(a) d20
+ (V(ss) +VV V(s)] ao (dlld")
-DD
(29)
Next,
minmax(1 1)) (V()) Ei (f(s () (1
(dlae)
V(e)
=min(1 En
max E (..a) (F(s, + +7V V(+)) ax) (dle²)
V())
#20
(30)
((s,a) + TVV(s V(s) D, (dld))
= min(( - 7) (V(s)
max E (o,)mdd
V(a)
d20
15

---

Published as a conference paper at ICLR 2023
L in the last step reduces to:
a(s,a) + (s) - DD, (dllde)
max (a.a)al d20
d(s,a) (f(s,0) + -TV(s, - v(5)]
d(s,a)
max R (a.))dd)))
TB (aue
f
(220
(8,
a
(
a(s,a) (f(s,a) 7TV(s, v()))
d(s,a)
max R (s.a)ad 420
d (s)
(s)
(31)
d(s,a) (f(s, + -V())
d(s, a)
d(s.a)20 dl (s
max
d (s,
d(s,a)
d(s,a) a) dP (s.
a aa (
as u(s,a) -5
(a.a) 20
=a [s. (v(s,)))
where y(s, a) f(s.a) +VV the third step follows the interchangeability principle (Lemma and the last step comes from the Fenchel conjugate of convex function f
a
Using this result, we finally yield the tractable lower-level problem Eq. (10).
A.3 PROOF OF TRACTABLE TRANSFORMATION OF THE LEVEL PROBLEM
Proposition 2. The original upper-level problem
min Dj (dmilla
(32)
Ar
can be equivalently written as:
min Dj (Sx ( V* ))
d l|de
(33)
Ar
a
where d is the optimal state-action visitation distribution of Eq. (7)
Proof. By the property Eq. (19), the maximizer
d(s.a) (2,0)
of f. (v(ss in Eq. (31) satisfies
d(s.0) d (P(,,))
(f(s.a) (VV V(s)
(34)
a
Given V*, have:
(s) (
(35)
dp (s.
Substituting this result into the original upper-level problem completes the proof.
Next, we denote f (AtVTV*-V* as g. By expanding the f-divergence, we have the upper-level objective:
(P((s ]
Dy (d (d glld²) R (
(36)
dF
(F
23(3, (
((ssa)
g(s, g(s,a)
(37)
()(),
w(s)a))
(38)
w(s8a)
'dom {u: 2 0} and is convex, f+(v) f(()) when y f()).
16

---

Published as a conference paper at ICLR 2023
where the distribution ratio w(s, é dss
Finally, by combining proposition and proposition 2, the original bi-level optimization problem Eq.
(4)-(5) is rewritten equivalently as follows:
(ssa))dd (o(ss,)) (1: ((V (5)
(
Ar arg min Ar
a /wv(s,
s.t. (*()) arg min (1 7) (oe + a (sa)ddd f. - xTV(saa) V(s) V(a)
a
(39)
B IMPLEMENTATION DETAILS OF RGM
B.1 RGM WITH KI-DIVERGENCE
In this section, we introduce the implementation details of RGM. For KI--divergence, we have f()) x log x and Fenchel conjugate (x) = ex-1 However, this exponential form is numerically unstable and prone to value explosion in practice. We address this issue by using the fact that the conjugate the negative entropy function, restricted to the probability simplex, the log-sum-exp function (Boyd et al., 2004), i.e., D (v) log Erng lexp (z)) Then, the optimization problem of RGM with KL divergence is
Softmax Adv(Ar,
(1og dl(s, + log Softmax ( a)
Addv(Ar, V*)
min E (a Ar
(Adv(Ar,V)
s.t. V' = arg minn(1 - B [V()) + a log E (s,a)ad
exp
(40)
where, () (1,----()))=-oo0---rrr+77++++777
and
(((..)) log (B(0.o) can be obtained by training discriminator log ][(oma]
log h-- - 1) using Eq. (14)
in continuous MDPs. The importance ratio used to extract the policy is
d=i (s, a) Softmax (s,
F Ar TTV V (s)
3 (s, (), d (s.
(41)
B.1.1 OPTIMIZE WITHOUT SUM-EXP
Note that in the upper level objective Eq. (40), we need to calculate a log-sum-exp value in the de- nominator of the log(Softmax) term, where log (eettmax(Adv(A)) V*)/a)) Adv(Ar, V*)/a- log > L exp(Adv(Ar) Ar, V*)/a). In low-dimensional discrete state-action space, we can easily get this value via summing over the overall space. In high-dimensional continuous MDPs, however, pretty difficult retrieve the value because requires integration
the entire space. CQL (Ku- mar et al., 2020) approximates this value via importance sampling but requires additional samples from the entire state-action space. There are some other methods like Markov Chain Monte Carlo (MCMC) or Score Match (SM) (Song & Kingma, 2021) that can approximate the update gradient but bring additional computation costs and suffer from some technical issues.
Fortunately, we can subtly circumvent the log-sum-ews exp term by optimizing the upper bound of the original upper-level problem using the following inequality (Boyd al., 2004):
max (11, (I1, U1,...,.. Fn) & max -.. In & log Σ exp (2:) E
(42)
where max(x1,,... the max value in a mini-batch which is sampled from {2), ) z.EB For simplicity, we denote ..,, } as max Substituting Eq. (42) into the leper- 2,E8
2.EB
problem of Eq. (40), get the upper bound of the original upper-level optimization objective:
17

---

Published as a conference paper at ICLR 2023
Adv(Ar.VV)
log d (s, d (s,
Adv(An,V)) a
Adv(An, Vo)
Upper(40) (a
Softmax
log E exp
s,aESKA
AAdv(Ar, Vo) a
(a. log (o,a)
Adv(Ar,
max Adv(Ar,
$ (a
Softmax
a Adv(An, V*) max
Adv(An, V*) d (s, (s,
Adr(Ar, Vo)
< (a
log
a
(43)
where Upper(40) denotes the upper level objective in Eq. (40).
Replacing Eq. (43) the upper level objective in Eq. (40), obtain the final optimization problem:
Adv(ar,
(108 (s, Adv(Ar, de (s,
Ady V*)
min () Ar
exp
+
max [Adv(Ar, V*
s.t. V = arr min B [s)] + a log EE (s
exp
(44)
We practically utilize the same mini-batch B as that of SGD gradient update step to calculate max Adv(An) }. Note that the exp term the upper-level problem prone to value explosion
in practice, we clip the exp value to (-oo, 100] like IQL (Kostrikov et al., 2021b) does to improve training stability.
When extracting the policy, we can ignore the annoying sum-exp term in the denominator of Softmax and get the following ratio, because it does not influence the direction of gradients update the policy.
(s, (s, a) = d (s,a)
A+ TTV* (+) = (8,
x exp
(45)
However, using (45) only get unnormalized distribution ratio instead of exact one. We resort to self-normalized importance sampling (Owen, 2013) to obtain a normalized ratio:
S (s, a)
(8, ()
(46)
(0,0)aa ( 11 (s,a))
B.2 RGM WITH X DIVERGEENC
Additionally, we can also implement RGM using -ivddivegeence. For Xi=-diergence, we have f(z) = {x - 1)² with dom f = {x : 2 03 and its Fenchel conjugate is fz(r) = 1/2 (x 1)a and fx = max (0, x + 1). Then, the optimization objective of RGM with x² divergence is
min E (AA)-AD Ar
d (s) (d.
max 0, Adv(Ar, V*)
+1
(s. de (3,
-1)
(47)
Adv(Ar. 27
s.t V = arg min(1 - >) E [V())] + L (aanaao
The importance ratio used to extract the policy is:
S (s. a) = d (s. ds
mmxx 00 F F+ Ar TTV* (* +1
(48)
For RGM with KL-divergence, the upper layer contains exponential term exp( Adv(8r.V*) which may pose numerical instability. For RGM with x² divergence, f. (x:) max (0, 1) and so the
'On account of the state-action visitation distribution 2 0
18

---

Published as a conference paper at ICLR 2023
gradient vanishes when + < 0, which makes the policy learning slow even fail. In practice, follow the criteria from SMODICE (Ma et al., 2022) by monitoring the initial policy loss to choose the types of f-divergence.
B.3 RGM HYPERPARAMETERS AND PSEUDOCODE
For continuous MDPs with high dimensional state-action spaces, we implement RGM by parameter- izing /,, Ar Vo and Tw using deep neural networks with parameter Φ, and w, respectively. We implement RGM based on two-time scale first-order stochastic gradient update, where the reward correction term is updated much slower than the Lagrangian multiplier V. We choose the cosine annealing learning rate schedule of the reward correction term and policy network stabilize the training process. To make the reward correction term comparable w.r.t the original imperfect rewards, we normalize the imperfect rewards to standard Gaussian distribution NO0 1) and strict the output o to --3,3 3] by Tanh function. The conclusive hyperparameters can be found in Table 3.
range of Aro
Table 3: The hyperparameters of RGM with deep neural networks
Hyperparameter
Value
Reward correction hidden dim
256
Reward correction layers
Reward correction activation function
ReLU
Discriminator hidden dim
512
Discriminator layers
3
Discriminator activation function
Tanh
Architecture
hidden dim
256
hidden layers
V activation function
ReLU
Policy hidden dim
256
Policy hidden layers
Policy activation function
ReLU
Optimizer
Adam (Kingma Ba, 2015)
Reward correction learning rate lo 3c-7
Reward correction learning rate schedule
cosine annealing
Discriminator learning rate 1-
lc-3
RGM Hyperparameters
V, learning rate
3e-4
Policy learning rate I Policy learning rate schedule
3e-4
cosine annealing
Ve gradient L2- regularization Discount factor
lc-4
0.99
x² for Robomimic tasks
f-divergence
KI. for other tasks
4 for maalker2d-medium-rrplaa
0.5 for other D4RI. tasks
for Antmaze tasks
for Lift and Can tasks
0.3 for Quadruped wwalupeddwalk-- Quadruped-jumg and
for the others in multi-task data sharing experiments
The pseudocode of RGM with deep neural networks can be found in Algorithm 1. We run RGM on one RTX 3080Ti GPU with about 1h30min training time apply 1M gradient steps.
We report the wall-clock training time of RGM compared with SOTA offline RL methods as well as SOTA offline IL methods that can learn from mixed quality data in Table 4. RGM is as efficient as most baselines but has an additional ability to combat the negative impacts of imperfect rewards.
Table 4: Wall-clock run time comparison of RGM and other baselines
BC
DWBC
SMODICE
TD3+BC
CQL
IQL
RGM(ours)
30min
2h40min
2h20min
45min
4h30min
1h30min
2h30min
19

---

Published as a conference paper at ICLR 2023
Algorithm 1 RGM (KL-divergence) with Deep Neural Networks
Input: One Expert demonstration DE, offline Dataset D, set D <- DE U.D Initialize 7, 6, 0, w. Discriminator learning
Train h, using D and D using Eq. (14).
for & 0, 1, 2, NV do
Sample mini-batch transitions (s, a, F, s¹) ~
// Reward Gap Minimization Bi-level optimization
Update Vo. Arg using Eq. (44) with lo < lo / Pollicy extraction
Update The based on Eq. (16) and Eq. (46)
end for
C EXPERIMENTAL DETAILS
In this section, we introduce the detailed experimental setups in our paper.
C.I D4RL EXPERIMENTS
Task Descriptions. The D4RL (Fu et al., 2020) tasks we try to solve include Hopper, Halfcheetah and Walker2d. For these tasks, RL policies need control the robots move the forward (right) direction by applying torques on the joints.
Hopper
(b) Halfcheetah
(c) Walker2d
Figure 7: D4RL MuJoCo tasks
Dataset composition. The D4RL (Fu al., 2020) datasets that used in this paper contain types of datasets: random: roll out a random policy for 1M steps. expert: roll out an expert policy that trained with SAC (Haarnoja al., 2018) for 1M steps. medium: roll out medium policy that achieves 1/3 the performance of the expert for 1M steps. medium-replay. replay buffer of SAC agent that trained the performance of the medium policy. medium-expert: equally mixed dataset combines medium and expert data. We sample only one trajectory from the expert dataset to serve as the expert demonstration DE The other datasets are treated as non-expert datasets D.
Table 5: Dataset compositions for D4RL Experiments
Task
State Dim
Expert Dataset
Number of Trajectories Expert Data Size
Hopper
11
hopper-expert-v2 halfcheetah-experre
1000
Halfcheetah
17
1000
Walker2d
walker2d-expert-v2
1000
Imperfect rewards. We assume the original rewards in D4RL datasets are perfect, since evaluate the policy performance based on the perfect reward function in the original gym environment during evaluation. We randomly flip the sign of 50% original rewards to construct partially correct rewards, where half rewards can provide correct learning signals while the other half cannot. We flip all signs of the original rewards to construct completely incorrect rewards.
20

---

Published as a conference paper at ICLR 2023
C.2 SPARSE REWARD EXPERIMENTS
Task descriptions. The Robomimic (Mandlekar et al., 2021) tasks we try to solve include Lift and Can. For the Lift task, RL policy needs to control 7-DOF robot randomly located at a table. For the Can task, RL policy needs to control a 7-DOF robot arm to learn pick a can that is randomly located table and place it in specific location.
learn lift cube that is
(a) Lift
(b) Can
Figure 8: Robomimic tasks
The AntMaze tasks we try to solve include AntMaze medium tasks, where an ant not only needs to learn to walk but also navigates from the goal to the destination in medium-size maze. This task is extremely difficult due to the non-markovian and nixed-quality offline dataset, the stochastic property of environments, and the high dimensional state-action space (Fu et al., 2020).
E
Figure 9: AntMaze medium task.
Robomimic dataset composition. The Robomimic (Mandlekar et al., 2021) datasets that we used in this paper contain types of datasets: PH (Proficient- Humbnn):: datasets collected single, experienced human operator. MH MMuti--ummnn, datasets are collected by human operators of varying proficiency. MG (Machine-Generatecl Generated); datasets are collected by first training SAC on the Lift and Can task, taking agent checkpoints that are saved regularly during training, and collecting 300 rollout trajectories from each checkpoint. We treat PH dataset the expert dataset since the environment is stochastic, thus only one expert trajectory is difficult to capture the expert distribution. We use MG datasets as the large potentially suboptimal dataset rather than MH datasets since MH datasets are non non-markovian and thus are hard to be solved by modern offline RL methods (Mandlekar al., 2021), which is not the main challenge we try to solve.
Table 6: Dataset compositions for Robomimic Datasets
Task
State Dim
Expert Dataset
Expert Size Non-expert Dataset
Non expert Size
Lift
19
Lift-PH
9666
Lift-MG
225K
Can
23
Can-PH
23207
Can-MG
585K
AntMaze dataset composition. trajectories (which may be suboptimal) that are collected by training IQL with dense rewards. We set the original D4RL Antmaze- Antmaze-medium plan---laly22 and manmmaze-edium-diivee v2 datasets as non-expert datasets.
The expert dataset of RGM is composed of 30 successful
21

---

Published as a conference paper at ICLR 2023
C.3 MULTI-TASK DATA SHARING EXPERIMENTS
Task descriptions. The multi-task data sharing experiments contain domains with 4 tasks per domain built on DeepMind Control Suite (Tassa et al., 2018). The immediate rewards in the 8 tasks are all in the unit interval, r(s, a) E [0. 1]. (a) For Walker (Stand, Walk, Run, Flip) domain, the agent needs control biped in 2D vertical plane master four different locomotion skills. The observation space is 24 dimensional, and the action space is 6 dimensional. The episode length is 1000. (b) For Quadruped (Walk, Run, Roll-Fast, Jump) domain, the agent needs to control quadruped within a 3D space to master four different moving skills. The observation space is 78 dimensional, and the action space 12 dimensional. The episode length set to 1000.
Walker-stand
(b) Walker-walk
(c) Walker-run
(d) Walker-fipp
(c) Quadruped-war
(f) Quadruped-run
(
Quadruped-jump
(h) Quadruped-roll fast
Figure 10: Different tasks in Walker and Quadruped domain
Dataset composition. We take the same rule of dataset generation and similar task settings as the work (Bai et al., 2023). For each task, we utilize TD3 (Fujimoto et al., 2018) to collect three types of datasets (expert, medium, replay). The expert dataset contains only one expert episode, the medium dataset contains 1000 episodes of interactions, and the replay dataset contains 2000 episodes of interactions. For Walker (Stand, Walk, Run, Flip) domain, the Stand task is set to the target task, relevant tasks. For Quadruped (Walk, Run, Jump, Roll-Fast) domain, the Walk
and the others task is set to the target task, and the others are relevant tasks. We conduct two-task data sharing experiments, in which share the replay dataset of the relevant task with the medium dataset of the target task.
C.4 GRID WORLD EXPERIMENTS
Dataset composition. The offline dataset D we use in grid world experiments consists of 1000 trajectories generated by completely random policy (Figure 11 (b)). There are two settings of imperfect rewards F: (i) F = +10 when reaching the goal while = anywhere else. (ii) (Figure 11 (c)) F +10 when reaching the goal, f -10 when encountering the fire (true fire fake fire), F = 0 everywhere else. The expert demonstration dataset D consists of only one expert demonstration (Figure 11 (a)).
D ADDITIONAL RESULTS
In this section, we provide additional comparative and ablation results RGM against baseline methods.
22

---

Published as a conference paper at ICLR 2023
S
(a) Expert demonstration
(b) Empirical distribution 7
Imperfect rewards F
Figure 11: (a) only one expert demonstration path, which starts from follows path and arrow
reach the goal 0 (b) The empirical distribution heatmap of offiinc dataset Do, which consists of trajectories generated by random policy starting from The darker the color is, the more frequently the agent passes. (c) Illustration imperfect rewards. Agent gets i when reaching everywhere else.
+10 when reaching
D.1 ADDITIONAL COMPARISON TO OFFLINE IL
Recall that DWBC (Xu et al., 2022b) and SMODICE (Ma et al., 2022) all assume the offline dataset already covers lot of expert trajectories, which more restrictive compared the requirement
of RGM. Therefore, we further demonstrate the superiority of RGM compared to these offline IL methods by evaluating RGM under the same settings of DWBC and SMODICE. We combine the original D4RL dataset with 200 or 100 expert trajectories as the offline dataset DD see Table 7 for descriptions the expert trajectories. The comparisons under these dataset configurations can be found in Table 8. We can observe from Table 8 that RGM still outperforms existing SOTA offline IL methods under their settings.
Table 7: The details about the expert data that used construct the non-expert dataset in offline II. settings.
Task
State Dim
Expert Dataset
Number of Trajectories
Expert Data Size
Hopper
11
hopper-expert-v2
200
193430
Halfcheetah
17
balfcheetab-expert
200
199800
Walker2d
walker2d-expert-22
100
99900
D.2 EXPERIMENTS SAMPLING FROM DISCOUNTED DISTRIBUTIONS
We also implemented the discounted visitation distribution sampling in RGM. This is done by augmenting the D4RL datasets that adds the timestep of each (s, a) pair in an episode. When performing sampling in (q.(14- and calculating the gradient, datasets and then multiply the gradient by 7'_ Empirically, we found that the performance of the discounted visitation distribution version is not better than the sampling distribution version of RGM. Figure 12 and Table show that RGM (sampling distribution) surpasses RGM (discounted visitation distribution) in most cases with lower variance, while the latter wins by a slight margin in only few cases.
sample (s, a,a,t) f) in the D4RL
D.3 EXPERIMENTS ON NOISY PARTIALLY CORRECT REWARDS
We add i.i.d Gaussian noises with different standard deviation a to original D4RL rewards to construct noisy imperfect rewards with different degrees of imperfection. We set = 1 to construct partially correct rewards and = 10 as largely incorrect rewards, see Table 10 for detailed results.
Table 10 shows that RGM under perfect rewards slightly outperforms RGM with partially correct rewards, indicating that RGM can largely remedy the negative impacts caused by reward noises with 1. Meanwhile, the highly noisy rewards 10) surely impact the performance, but its mean
23

---

Published as a conference paper at ICLR 2023
Table 8: Average normalized scores of RGM compared with SOTA offline IL methods that can learn from mixed quality data under their settings. The notation "-w.e" stands for the mixed dataset that combines the original D4RL dataset with some expert trajectories. The scores are taken over the final 10 evaluations with 5 random seeds. We obtain the results via ruining author-provided open-source codes. RGM achieves highest scores in 12 tasks.
Dataset
BC
DWBC
SMODICE
RGM (Ours)
hopper-r-w.e
2.8
59.5.130.3
108.715.4
110.4 11.2
halfcheetah-r-w.e
0.2
3.31177
11.5
57.61..4
walker2d-r-w.c
1.2
81.8100.6
102.01:9.9
109.2 #0.2
hopper-m-w.c
54.9
39.0+:22.5
54.5
66.1 +9.7
halfcheetah-m-ww.e
41.2
8.5+-9.2
55.4 +1099
50.5+7.9
walker2d-m-w.ee
62.1
56.11
6.5.1..7
79.2 +12.4
hopper-mm-sw.ee
23.4
23.1117.11
$3.01127.5
58.6 127.0
halfchectah-mm-rww.e
24.2
1.+112
84.9 +7.2
65.4++15.1
walker2d-m-r-ww.e
21.8
85.5 1.33.6
8.9
71.7.L32.6
hopper-mme-w.e
51.2
40.01:22.5
71.415..5
89.1 13.5
halfcheetah-mme-wwe
61.7
1.2+0.5
87.2 +1.9
76.8
walcer2d-m-o-w.c
103.2
76.8
14.1#2.0
105.8 8.66
Mean Score
37.8
39.71177.5
61.319.0
78.3 12.0
hopper-random--2
hooper-medlum-22
hoppe--medimm-rellyyy..
nooger-medum-expertc
Nomalized Return
Nomalized Return
Nomalized Return
Nomalized Return
100
100
100
102
60
80
38
Undioonurte
8.00
8.50
8.00
8.0
Training Steps lx10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
hallcheetth-randddm-2
hallcheetah-medium22
halfcheetah expert-v2
Nomalzed Return
Nomalized Return
Nomalzed Return
Nomalired Return
100
100
100
60
80
60
60
Undiacoutted
Flocsunted
8.30
0.25
0.00
0.76
1.0
8.00
6.00
0.26
0.50
0.76
8.00
0.75
1.0
Training Steps lx10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10*)
random
walker2d medium-dimmmmi---- replay
walker Meleerddmmedimmmn expertev2
Nomalizd Return
Nomalized Return
Nomalbzd Raturn
Nomalized Return
10
Undioouurad
Dazzuriied
8.00
0.25
0.00
0.76
1.00
8.00
0.75
8.00
0.26
0.50
0.76
1.00
8.0
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 12: Experiments on sampling from discounted and undiscounted distributions
score is 45.0, which is still considerably higher than other Offline RL and IL methods under partially correct rewards with the largest mean value 35.5 as shown in Table I.
D.4 ABLATIONS ON NUMBER EXPERT TRAJECTORIES
We add the ablations on the number of expert trajectories in E FN for RGM, SMODICE and DWBC. Table 11, 12 and 13 show that RGM achieves better performance than offline II. methods
24

---

Published as a conference paper at ICLR 2023
Table 9: Normalized scores of RGM sampling from discounted distribution and undiscounted distribution
Dataset
RGM (Discounted)
RGM (Undiscounted)
hopper-r
19.8+0.2
21.2 #0.4
halfcheetah-rr
0.2 #0.0
0.2 #0.0
walker2d-r
1,2+1.7
7.7 #3.3
hopper-m
51.1+4.9
55.5 +1.0
halfcheetah-m
40.3+1.6
40.7 #1.4
walker2d-m
62,2+22.5
72.3 +10.7
hopper-m-r
13.
59.1 +15.3
halfcheetah-m--r
34.5+4.5
37.8 42.6
walker2d-m-r
34.3+11.0
48.6 #3.6
hopper-m-e
65.3+19.5
87.1 +10.7
halfcheetahh-m-e
87.3 +7.8
81,5+0.8
walker2d-ni m-e
108.440.6
108.8 40.4
Mean score
45.7+7.2
52.0 +4.2
Table 10: Normalized scores of RGM on different degrees noisy datasets.
Dataset
RGM(T)
RGM (a
RGM (a 10)
hopper-r
29.6
8.5
9.8
halfchectah-r
0.2
0.2
walker2d-r
3.9
hopper-m
56.2
$2.0
47.9
halfchcetah-m
40.4
41.2
38.4
walker2d-m
73.3
71.9
72
hopper-m-r
60.3
58.0
40.0
halfchcetah-m-rr
37.9
38.3
28.1
walker2d-m-r
46.3
42.5
43.8
hopper-m-e
106.1
82.0
82.8
halfcheetah--ig
85.6
88.7
69.1
walker2d-m-e
109.2
108.2
108.5
Mean score
54.1
45.0
designed for mixed-quality data (DWBC and SMODICE). is found that RGM also enjoys higher level of performance gains when the amount of expert data is increased.
D.5 EXPERIMENTS ON MULTI-TASK DATA SHARING
We present concrete results of the multi-task data sharing experiment. Table 14 shows the evaluated scores multi-task data sharing, which are illustrated in Fig. 3.
D.6 ADDITIONAL LEARNING CURVES RGM
We present the learning curves of RGM compared with offline IL and RL baselines D4RL datasets related to the results presented Table 1.
D.7 ILL.SSRAATVVE EXAMPLE FOR THE NON-TABULAR SCENARIOS
The results the 8x8 grid world experiments in Section 5.2 and Appendix C.4 illustrate the potential benefits of the learned rewards in the tabular case. In this subsection, consider one-dimensional
25

---

+3)), then we keep Published as a conference paper at ICLR 2023
Table 11: Normalized scores RGM offline II. baselines when De contains expert trajectories.
Dataset DWBC (N
10)
SMODICE (N 10)
RGM (N 10)
hopper-r
52.5
1.3
30.8
halfcheetah-rr
-0.3
2.1
0.2
walker2d-r
96.2
0.3
6.1
hopper-m
31.1
53,8
54,5
halfcheetah-m
5.0
40.9
41.4
walker2d-m
22.4
3.3
72.9
hopper-m-r
37,4
33.2
55.5
halfchectah-m--r
3,9
36.7
34.9
walker2d-mn m-r
90.7
43.1
hopper-m-e
85.0
89.2
halfcheetah-m--e
86.6
79.4
walker2d-m-e
46.3
14.1
109.0
Mean score
35.6
32.7
51.4
Table 12: Normalized scores of RGM and offline IL baselines when D contains 40 expert trajectories.
Dataset
DWBC (N 40)
SMODICE (N 40)
RGM (N 40)
hopper-r
54.6
67.2
36.9
halfcheetah-r
8.8
14.8
18.7
walker2d-r
78.7
92.9
-0.1
hopper-m
13.5
54.2
57.0
halfcheetah-n
5.6
44.5
40.9
walker2d-m
16.9
3.5
74.3
hopper-m-r
54.3
47.2
54.3
halfcheetah-m-rr
46.1
54.6
46.1
walker2d-m-r
87.8
35.7
61.2
hopper-m-c
34.5
75.9
92.4
halfcheetah-m--e
3.4
85.1
84.7
walker2d-m-e
57.2
108.6
Mean score
38.5
44.9
56.5
random walk task in the non-tabular case and provide the visualization of the learned corrected rewards In this task, the state space is a straight line from [0, +3] and the agent can move at cach step in the range of [-0.5, 0.5]. If the agent goes beyond the edge (s
at the edge (s = 0 or s = +33). The agent needs to start from state s = O and reach the destination located = 3 as fast possible. The expert dataset DE consists of one trajectory where the expert takes action = 0.5 at every state. The offline dataset consists of 1000 trajectories generated by completely random policy where the agent takes action uniformly from [-0.5, 0.5] at every state. The sparse rewards 7 = +10 is set when reaching the destination while F = 0 anywhere else. The visualization of learned rewards f each state-action pair is shown in Figure 14.
E DISCUSSION ON THE APPLICABILITY TO ONLINE SETTINGS
should be noted that the proposed RGM framework can also be applied to the online setting. This be achieved by simply setting 0 in Eq. (4-5), and we have the bi-level objective of the online
26

---

Published as a conference paper at ICLR 2023
Table 13: Normalized scores RGM offline II. baselines when De contains expert trajectories.
Dataset DWBC (N 80)
SMODICE (N
80)
RGM (Nv 80)
hopper-r
65.1
92.7
47.3
halfcheetah-rr
2.3
48.1
40.4
walker2d-r
86.1
98.8
109.1
hopper-m
8.8
53,4
59.7
halfchoetah-m
6.7
51.2
42.5
walker2d-m
36.5
2.9
73.4
hopper-m-r
35.3
46,6
66.2
halfchectah-m--r
36.1
59.6
54.0
walker2d-m-r
85.8
32.8
64.9
hopper-m-e
83.7
97.1
halfcheetah-m--e
87.4
83,9
walker2d- m-e
64.5
43.7
108.6
Mean score
37.5
58.4
70.6
Table 14: Evaluated scores multi-task data sharing.
Domain
Dataset
CDS
CDS+UDS
RGM
Walker
stand-medium + walk-replay 486.147.2
415.3+44.8
753.3+107.6
Walker
stand-medium run-replay
455.8+21.9
440.0=8.4
620.0+31.0
Walker
stand-medium flip-replay
492.4+19.0
371.2+123.0
745.6*100.
Quadruped
walk-medium run-replay
527.6+213.0
155.7=53.0
900.0=448.6
Quadruped
walk- roll fast-replay
476.2+45.1
139.1
393.2+37.3
Quadruped
walk- medium jump-replay
138.5
521.9+236.9
490.9: 1 19.4
Mean score
495.3479.1
390.5492.8
667.0+74.1
version of RGM:
Ar* arg min Dy (d ((al(a)))
(49)
πे arg max ((s.a)=ars [fs
Since we could get online samples from d in the online setting, so we don't have to eliminate d= One can use the existing popular online RL algorithms to solve the lower-level problem, while leveraging the online samples from d= to solve the upper-level problem. Hence the online version of reduced and simplified version of the original RGM. The idea of the reward correction has not been changed the online setting. which illustrates that to some extent, proposed RGM is unified policy optimization method for imperfect rewards.
RGM can be perceived
27

---

Published as a conference paper at ICLR 2023
hopper-random-v2
hooper-medlum-22
hogper-medlum-raeee w2
hopper: www.--mmddlummmm um-expert-22
Nomalized keturn
Nomalized Return
100
Nomalized Roturn
Nomalized Return
10:
98-
75
P8
75
18
8.50
0.25
0.00
0.76
1.0
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.25
0.80
0.75
1.00
Training Steps (x10%
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
hallcheetab-randdone
hallcheetah-medium-22
halfcheetah- anichertuhmmmiliim-reeeaaaa
hellchrettahrmediumeen 22
Nomalized keturn
Nomalized Return
Nomalized Retur
100
100
79-
75
22
Nomalized
8.50
0.25
0.50
0.7
100
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
025
0.50
0.75
1.00
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10
Training Steps (x10%)
walker2d-ranoom-22
wallser2d-medium--22
walker2d-mediumt expertt-x22
Nomalized keturn
Nomalized Return
Nomalized Roturn
Nomalized Return
1oc
10c
10:
76-
76-
0.25
0.50
0.75
1.00
0.50
0.75
0.25
0.50
0.75
1.00
0.25
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Training Steps (x10%)
Figure 13: Learning curves of RGM trained D4RL datasets under imperfect rewards.
One-dimensional random walk task
0:8
0.0
0.5
1.5
2.0
02
0.05
Learned reward
Action
0.04
High
Action
0.2
Distribution
-02
001
-02
-044
-0.4
25
2.5
State
State
Empirical distribution of
Visualization of learned rewards
Figure 14: (a) The empirical distribution of offline dataset D in a continuous ine-dimensional random walk task. Most states the offline dataset are distributed near the starting point. (b) At each state (at each vertical line), the learned reward gets larger value when the action gets closer to 0.5. The expert data has contain states (s 0.0, 0.5. 1.0, 1.5, 2.0, 2.5, 3.0), but the learned rewards can still generalize well in the state space even in regions that are not covered by the expert data. Similar to the 8x8 grid world experiment, we can successfully navigate to the destination by only maximizing per-step reward f, which means that the learned rewards also encode long-horizon information.
28


Reviews:
[{"review_id": "Iysb4wXTQcJ", "reviewer": "Reviewer_gH6T", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the offline RL setting with imperfect rewards. That is, a known portion of the trajectories contain possibly corrupted rewards. The approach proposed to tackle this problem is Reward Gap Minimization (RGM), which finds the reward function (correction) which, when used to solve for the optimal policy, agrees with the expert trajectory distribution. This requires a dual-optimization problem to be solved and a number of reformulations are provided to get a practical form. The approach is empirically tested, with some analysis and ablations, in three continuous control problems (hopper/half-cheetah/hopper).", "summary_of_the_review": "Good and clear paper about a principled and practical way to deal with imperfect rewards in offline RL. I think the paper has enough empirical evidence as it is, but I do wonder whether the approach can be scaled to more complex settings (larger datasets, more difficult/higher-dim task).\n", "strength_and_weaknesses": "The right context and background is provided to describe the approach. The proposed RGM formulation is intriguing and seems natural mathematically, even if intuitively it seems like a challenging optimization objective. \n\nAs far as I could tell, all derivations are correct and they are explained clearly. The experimental section is generally well designed, with some good illustrative examples in Fig 3 and a number of useful experiments to establish the validity of the method.\n\nSome questions and comments:\n\n- The classification problem for h seems hard in general. As if I understand correctly, it amounts to detecting whether an action is optimal or not. Couldn’t this by itself, if learned perfectly, be used to generate the optimal/expert policy.\n- It would be useful to have some notion of the computational cost of the method (e.g. as a function of the dataset size) and how that compares to the perfect reward case.\n- It would be nice to visualize how well the reward is recovered in the non-tabular scenarios.\n- The fact that RGM does better than other offline RL methods even with perfect rewards seems a bit surprising (sec 5.1). Why should this be expected?\n- It seems a bit unfair to evaluate offline IL approaches that expect expert data on these mixed datasets. Are there other methods that can deal naturally with the B (imperfect reward) condition?   \n- The RGM formulation makes no assumption on the way the rewards might be corrupted (versus for example some noise process etc.) It would be good to comment on the pros/cons of that. \n- Could you clarify how the approach relates to \"Generative Adversarial Imitation Learning\"?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FCwoGQ-RPlR", "reviewer": "Reviewer_KWzN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new unified offline policy optimization approach, named RGM (Reward gap minimization), that can handle imperfect rewards. RGM has two layers: (1) upper layer learns reward correction term by minimizing the reward gap towards expert behaviors, (2) lower layer solves a pessimistic RL problem under the corrected rewards. More precisely, the authors use f-divergence matching to learn the correction term, and OptiDICE to solve the pessimistic RL problem. Then, policy extraction is used to extract optimal policy from the learned stationary distribution. Finally, this paper provides an empirical study on D4RL datasets.", "summary_of_the_review": "I think this paper proposes a novel algorithm to handle imperfect rewards. However, I have some questions and I believe they are important in evaluating this paper. Thus, I vote to weakly reject until these questions are resolved.", "strength_and_weaknesses": "## **Strength**\n- RGM uses bi-level optimization and this approach is very interesting.\n## **Weakness**\nI have following questions and concerns:\n### **Theory**\n- RGM does not consider the degree of imperfection of rewards. Theoretically, it seems that the degree has no effect on the performance of RGM - it is weird to me. Intuitively, if rewards are perfect, RGM should perform better than offline IL algorithms and comparable to offline RL algorithms, and if rewards are too imperfect, RGM should perform poorly. Consequently, it is questionable whether the imperfection of rewards affects RGM performance.\n- (Related to the previous question) Suppose that we can obtain optimal values for each optimization problem. In this ideal case, RGM obtains the optimal reward correction term, optimal stationary distribution, and optimal policy. Then, what is the main difference between RGM and existing offline IL algorithms in theory? I think RGM is equivalent to DemoDICE (if f-divergence is KL divergence) and SMODICE if the max / min can be computed exactly.\n- In footnote 4, page 8, the authors claim that “SMODICE and DemoDICE treat $\\log\\frac{d^E}{d^D}$ as rewards, which can only provide correct signals when $d^E>0$.“. However, in thes upper level problem (learning reward correction term), RGM also uses $w:=\\frac{d^E}{d^D}$, especially in Eq. (17) – therefore, in my opinion, learnable reward correction term also can provide correct signals when $d^E>0$.\n### **Experiments**\n- In the beginning of the paper, the authors claim that the proposed methods can handle all three types of rewards (perfect, partially correct, incorrect). However, in the experiments consisting of D4RL dataset, there is only one imperfect reward setting (50% of rewards are sign-flipped). Thus, I think there should be additional empirical studies on (1) perfect and incorrect rewards (2) other types of imperfect rewards (e.g. adding i.i.d. gaussian noise for each reward).\n- There should be RGM (T) in Table 1.\n- What is your intuition about imperfect rewards used in the experiments in this paper? I do not think flipping the signs is a natural setting.\n- In addition, I am curious about the ablation study on performance of RGM, offline RL algorithms, and offline IL algorithms according to (1) the degree of imperfection of rewards (including perfect rewards and completely incorrect reward) (2) the number of expert trajectories in $D^E$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8EhRUR4rWp", "reviewer": "Reviewer_1KgB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose an approach to address the problem of imperfect reward under offline settings. \n\nThe main idea is to formulate the problem as a bi-level optimization problem, where the upper problem is to minimize the gap between the stationary distribution of the optimal policy defined by the imperfect reward, and the distribution of the expert data. The lower optimization problem is to use the optimized reward function to obtain a new policy under the offline RL settings. To track the bi-level optimization problem, the authors leverage the fenchel duality to obtain a tractable optimization solution. \n\nEmpirically the authors demonstrate that their approach can obtain better performance when the reward function is imperfect. ", "summary_of_the_review": "overall I think the paper is not ready in terms of the following two main reasons:\n\n- There are fundamental issues of their reward gap and followed up derivations. \n- The experiments they conducted I think is not the right settings to demonstrate the advantage of their approach. In most of the scenarios that perfect reward functions are hard to obtain, such as sparse reward settings, they did nothing on that. \n\nAs a result, I recommend a major revision of the paper.  \n\nFor authors' rebuttal message: I know I am quite strict about your definition and derivation details, but please correct me if there is anything wrong of my points. I am very happy to modify scores if I miss anything important. \n\n----\n\nTypos:\n- Line under Eq(11), it should be $TV(s)$, not $TV(s,a)$", "strength_and_weaknesses": "**Strength**\n- The authors conducted detailed empirical experiments to demonstrate the advantage of their approach under the offline settings. \n- The derivations in the paper are quite comprehensive, which are good for authors to follow in detail, and check out how the results are derived.  \n\n\n**Weakness**\n1. I believe there are some fundamental technical faults in the basic formulation of the paper, regardless of the superior empirical performance.  Here are some technical flaws I am concerned about: \n      - In Equation (5) & (6), you define the gap between the current reward function and the optimal reward function as $$D_{f}( d^{\\pi^{*}}_{\\hat{r}} | d^{E}) $$\n\n     According to your definition, $ d^{\\pi^{*}}_{\\hat{r}}$ is the  state-action visitation distribution defined in Equation (4). In Equation (4), I believe the discounted visitation distribution is defined together with $\\gamma$ and initial distribution. As I can infer from your following derivation, the stationary distribution of the expert data $d^{E}$ you used is actually the sampling distribution, which is not related to the discounted factor $\\gamma$ and initial distribution. If so, how could you actually minimize the gap? They are two different kinds of stationary distribution. Let me illustrate a more intuitive case, suppose $\\gamma$ is very small, close to zero (e.g., $\\gamma=0.01$), your expert distribution is define on the whole trajectory (unrelated to $\\gamma$), while the stationary distribution of the policy obtained by the reward function $\\hat{r}$ is actually close to the initial state distribution and the policy distribution, then your reward gap definition is totally problematic. \n\n  Similar thing happens when you try to estimate Eq (16). The density ratio you estimated is the undiscounted ($\\gamma=1$ or sampling) distribution, while in your whole derivations $d$ is the discounted distribution. I feel it is not correct to match the discounted stationary distribution to the undiscounted stationary (sampling) distribution, unless you proof that it is okay to do that, and you can convince me the previous example I illustrated is wrong. \n\n3. For estimating Eq(13) or Eq(43), I am wondering how would actually deal with double sampling? your choice of conjugate function does now allow you to obtain unbiased estimation since there is an expectation in $TV^{*}(s)$? Do you assume the transitions and policy is deterministic? If so, I do not see any related assumptions in your paper. \n\n4. The whole algorithm is too complex, I feel it may require lots of efforts and hyperparameter tunning to make it work. \n\n5. Why you want to conduct the exepriments in the offline setting? I believe the imperfect problem issue exists in all settings of RL. \n\n6. And also, how do you define **imperfect** and **perfect** reward in the mujoco environments? the mujoco reward functions you used are human designed, why would you think they are perfect. One case I can think about is that in the sparse reward settings (or multi-goal RL), the perfect reward is that you would achieve higher success rate when testing. So I don't think mujoco is the perfect environments to demonstrate your claim, since there is no definition of perfect reward here (correct me if I am wrong, this is my personal thoughts).\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster