Score Models for Offline Goal-Conditioned Reinforcement Learning
Authors: Harshit Sikchi
Rohan Chitnis
Ahmed Touati
Alborz Geramifard
Amy Zhang
Scott Niekum
Venue: ICLR 2024 poster (2024)
Keywords: Robot Learning
Goal-Conditioned Reinforcement Learning
Deep Reinforcement Learning
Primary area: reinforcement learning
TL;DR: A discriminator-free occupancy matching approach for performant offline goal-conditioned reinforcement learning.

Abstract: Offline Goal-Conditioned Reinforcement Learning (GCRL) is tasked with learning to achieve multiple goals in an environment purely from offline datasets using sparse reward functions. Offline GCRL is pivotal for developing generalist agents capable of leveraging pre-existing datasets to learn diverse and reusable skills without hand-engineering reward functions. However, contemporary approaches to GCRL based on supervised learning and contrastive learning are often suboptimal in the offline setting. An alternative perspective on GCRL optimizes for occupancy matching, but necessitates learning a discriminator, which subsequently serves as a pseudo-reward for downstream RL. Inaccuracies in the learned discriminator can cascade, negatively influencing the resulting policy. We present a novel approach to GCRL under a new lens of mixture-distribution matching, leading to our discriminator-free method: SMORe. The key insight is combining the occupancy matching perspective of GCRL with a convex dual formulation to derive a learning objective that can better leverage suboptimal offline data. SMORe learns *scores* or unnormalized densities representing the importance of taking an action at a state for reaching a particular goal. SMORe is principled and our extensive experiments on the fully offline GCRL benchmark composed of robot manipulation and locomotion tasks, including high-dimensional observations, show that SMORe can outperform state-of-the-art baselines by a significant margin.

Published as a conference paper at ICLR 2024
SCORE MODELS FOR OFFLINE GOAL-CONDITIONED REINFORCEMENT LEARNING
Harshit Sikchi Rohan Chitnis o, Ahmed Touati & Alborz Geramifard Amy Zhang "ecan" Niekum" e University of Texas at Austin, Meta AI, UMass Amherst
ABSTRACT
Offline Goal-Conditioeed Reinforcement Learning (GCRL) is tasked with learning to achieve multiple goals in an environment purely from offline datasets using sparse reward functions. Offlin GCRL is pivotal for developing generalist agents capable of leveraging pre-existing datasets to learn diverse and reusable skills without hand-engineering reward functions. However, contemporary approaches to GCRL based on supervised learning and contrastive learning are often suboptimal in the offline setting. An alternative perspective GCRL optimizes for occupancy matching, but necessitates learning a discriminator, which subsequently serves as pseudo-reward for downstream RL. Inaccuracies in the learned discriminator can cascade, negatively influencing the resulting policy. We present a novel ap- proach to GCRL under a new lens ciittureddistribution matching, leading our discriminator-fii free method: SMORe. The key insight is combining the occupancy matching perspective of GCRL with convex dual formulation to derive a learning objective that can better leverage suboptimal offline data. SMORe learns scores unnormalized densities representing the importance of taking an action at state for reaching particular goal. SMORe is principled and our extensive experiments on the fully offline GCRL benchmark composed of robot manipulation and locomotion tasks, including high -ish-dimensoonal observations, show that SMORe outperform state-of- satte-of-the-assd tt-the-art art baselines by a significant margin.
INTRODUCTION
A generalist agent will require vast repertoire of skills, and large amounts of offline pre-collected data offer a way to learn useful skills without any environmental interaction. Many subfields of machine learning like vision and NL.P have enjoyed great success by designing objectives to learn a general model from large and diverse datasets. In robot learning, offline interaction data has become more prominent in the recent past (Ebert et al., 2021), with the scale of the datasets growing consistently (Walke et al., 2023; Padalkar et al., 2023). Goal-conditioned reinforcement learning (GCRL) offers principled way to acquire a variety of useful skills without the prohibitively difficult process of hand-engineering variety of goals in the environment. The rewards are sparse and coul-conditioned: when the agent's ss state is proximal to the goal and 0 otherwise. However, the benefit of not requiring the designer to hand-engineer dense reward functions can also be curse, because learning from sparse rewards is difficult. Driving progress in fundamental offline GCRL algorithms thus becomes an important aspect of moving towards performant generalist agents whose skills scale with data.
reward functions. In GCRL, the agent learns policy to accomplish a
Despite recent progress in developing methods for goal-reaching in the online setting (where envi- ronment interactions are allowed), a number of these methods are either suboptimal in the offline setting suffer from learning difficulties. Prior GCRL algorithms can largely be classified into one of three categories: iterated behavior cloning, RL with sparse rewards, and contrastive learning. Iterated behavior cloning goal-conditioned supervised learning approaches (Ghosh et al., 2019; Yang et al., 2019) have been shown to be provably suboptimal (Eysenbach et al., 2022a) for GCRL. Modifying single-task RL methods (Silver et al., 2014; Kostrikov et al., 2021) for GCRL with 0-1 reward implies learning a Q-function that predicts the discounted probability of goal reaching, which
$ Work done partially during an internship at Meta AL Correspondence to hsikchi @ udexas.edu Equal Contribution. Project page (Code and Videos): hari-sikchi, www.chi.gtthu.hhie www.gggiiohhhooorr.......oo io/smore/

---

Published as a conference paper at ICLR 2024
makes it essentially density model. Modeling density directly is hard problem, insight which has prompted the development of methods (Eysenbach et al., 2020) that learn density-ratio instead of densities, classification is an easier problem than density estimation. Contrastive RL approaches to GCRL (Rysenbach et al,, 2020; 2022b; Zheng et al., 2023) aim to do precisely this and are the main methods to enjoy success for applying GCRL in high-dimensional observation spaces. However, when dealing with offline datasets, contrastive RL approaches (Eysenbach et al., 2022b; Zheng et al., 2023) are suboptimal, as they only learn policy that a greedy improvement over the Q-function of the data generation policy. This begs the question: How can we derive a performant GCRL method that learns near-optimal policies from offline datasets of suboptimal quality?
In this work, we leverage the underexplored insight of formulating GCRL as an occupancy matching problem. Occupancy matching between the joint state-action-gol goaaionnngollllooo visitation distribution induced by the current policy and the distribution over state-actions that transition to goals can be shown to be equivalent to optimizing a max-entropy GCRL objective. Occupancy matching has been studied extensively in imitation learning (Ghasemipour et al., 2020) and often requires learning discriminator and using the learned discriminator for downstream policy learning through RL. Indeed, prior GCRL work (Ma et al., 2022) explores similar insight. Unfortunately, errors in learned discriminators can compound and adversely affect the learned policy's 's performance, especially in the offiine setting where these errors cannot be corrected with further interaction with the environment.
Going beyond the shortcomings of the previous methods, our proposed method combines the insight of formulating GCRL an occupancy matching problem along with an efficient, discriminator-free dual formulation that learns from offline data. The resulting algorithm SMORe forgoes learning density functions or classifiers, but instead learns unnormalized densities or scores that allow to produce near-optimal goal-reaching policies. The scores are learned via a Bellman-regulaarized contrastive procedure that makes our method a desirable candidate for GCRL with -igeesimesii observations, avoiding the need for density modeling. Our experiments represent wide variety of goal-reaching environments consisting of robotic arms, anthropomorphic hands, and locomotion environments. We lay out the following contributions: 1) on the extended offline GCRL, benchmark, our results demonstrate that SMORe significantly outperforms prior methods in the offline GCRL setting. 2) In line with our hypothesis, discriminator-fiee training makes SMORe particularly robust to decreasing goal-coverage in the offline dataset, a property we demonstrate in the experiments. 3) We test SMORe for zero-shot GCRL on prior benchmark (Zheng et al., 2023) for high dimensional vision-based GCRL where contrastive RL approaches are the only class of GCRL methods that have been successful, and show improved performance over other ofate-of-the-art baselines.
2 PROBLEM FORMULATION
We consider an infinite horizon discounted Markov Decision Process denoted by the tuple M (S, A. p, r, Y, du), where is the state space, .A is the action space, p is the transition probability function, r S x A- IR is the reward function, 7 (0, 1) is the discount factor, and do the initial state distribution. We constrain ourselves to the goal-conditioned RL setting, where we additionally assume a goal space G where states in S are mapped the goal space using a known mapping: 6 : S - G. The reward function r(s, a) in GCRL is sparse and also depends on the goal. A goal conditioned policy π S G A(A) outputs a distribution over actions in given state conditioned on goal. Given a distribution over desired evaluation goals qrest (g), the objective of goal-conditioned RL is to find a policy xg that maximizes the expected discounted return:
J [Tg := Eg (o)))==o=do
Σ 7 7rr((s, g) t=
We denote by pT the transition operator induced by the policy Ta defined as PT S(s, a, g) = E., (p..)( "r,ls"'''',_' () [S(s', a',g)]. for any score function S S x Ax - R. We use d3 (s. g) to denote the discounted goal-conditioned state-action occupancy distribution of Tg. i. i.e ( g) (1 - (a|s, .) >; [ Pr(s; g do)]. which represents the expected discounted time spent in each state-action pair by the policy Ty conditioned on the goal g. For complete generality, in GCRL, the distribution of goals the policy is trained on often differs from the test goal distribution. To
make this distinction clear we define the training distribution rarnn a uniform measure over goals
We use the subscript g to make the policy'ss conditioning on g explicit.
2

---

Published as a conference paper at ICLR 2024
we desire to learn to optimally reach during training. We write d= as g) (g) a [g) as the joint A state-action- -gaal occupancy distribution must satisfy the Bellman flow constraint in order for it to be a valid occupancy distribution for some stationary policy Tg. Vss E S, A, A.geg 9:
-oaal visitation distribution of the policy Ng under the training goal distribution.
d(s8, a,g) = (1 - -d ( 8.9) + 7 Ep(s
p(s s'.a'
a' gg) 8,9);
(2)
where do(s,
(5, g) do (())))=re)n(g)) Finally, given d we can express the learning objective for the GCRL agent under the training goal distribution as ( = - ().. [r(s, a, 9.].
In this work, we focus on the offline setup where the agent cannot interact with the environment
M and instead has access to a offline dataset of 'D = {ri) where each trajectory 7(0)
(1) .r(") { (f(;.gg))))))) with s dg- The trajectories are usually relabelled with the ())aingg) (s( oy
during learning. We denote the joint state-action-goal distribution of the offline dataset DD as p(s, o,
3 SCORE-MODELS FOR OFFLINE GOAL CONDITIONED REINFORCEMENT LEARNING
In this section, we introduce our method in two parts: First, we build up the equivalence of the GCRL objective to the occupancy matching problem in Section 3.1, and then we derive a discriminator freee dual objective for solving the occupancy matching problem using off-policy data in Section 3.2. Finally, we present the algorithm for SMORe under practical considerations in Section 3.3.
3.1 GCRL AS AN OCCUPANCY MATCHING PROBLEM
Define goal-transition distribution a(s,a,9) in stochastic MDP as q(s, a) oc q² (g,ii [T [=)(s]=g]- Intuitively, the distribution has probability mass on each transition that leads goal. We formulate the GCRL problem as an occupancy matching problem by searching for the policy Ty that minimizes the discrepancy between its state-action-goal occupancy distribution and the goal-transition distribution q(s, a.))
Occupancy matchi ng probl em: (d (s, (ss,a,))ll( g)] a, a,g)),
where D denotes f-divergence with generator function f. Note that the q distribution is potentially unachievable by any goal-conditioned policy Ty- Firstly, it does not account for the initial transient phase that the policy must navigate reach the desired goal. Secondly, even if we consider only the stationary regime (when y - 1), it may not be dynamically possible for the policy to continuously remain at the goal and rather necessitate cycling around the goal. However, in Proposition that the occupancy matching in Eq. 3 offers a principled objective since it forms a lower bound to the max-entropy GCRL problem.
show
Proposition 1. Consider a stochastic MDP, a stochastic policy n, and a sparse reward function r(s, a, Ey--p--ee pl-)p [[(((s)) 9, 9, =reing)) (g) > +0] where II is indicator function. Define soft goal transition distribution to be q(s.a,g) g) a expla r( (,,g). The following bounds hold for any f-divergence that upper bounds Kl-divergence (eg. x², Jensen- Shaenn--annon)::
jerain (Tg) -H(d"))> > (dd (s, a, (q(s, a)))) + C,
1 -==(("))~~>><<>>
(4)
where H. denotes the entropy, is temperature parameter and C is the partition function for R(s,0.9), Furthermore, the bound is light when J is the KL-divergence.
Ma et al. (2022) (in Proposition 4.1) presented similar result connecting state-goal distribution matching (i. DxL (d" ()ll9(.. g))) to GCRL objective and Proposition 1 extends their results to goal-transition distribution matching. Matching action-free distributions necessitates constructing a loose lower bound that is tractable to optimize. By considering goal-transition distributions we sidestep constructing loose lower bound and instead directly obtain tractable distribution matching objective (Ghasemipour et al., 2020; Kostrikov et al., 2019) that is tight under KL-divergence.
How does converting a GCRL objective to imitation learning objective make learning easier? Esti- mating the f-divergence still requires estimating the joint policy visitation probabilities d d a, g). which itself presents challenging problem. We show in the following section that we can leverage
We will use "occupancy" and "visitation" interchangeably.
3

---

Published as a conference paper at ICLR 2024
SMORe Mixture occupancy matching with dual objective
Current policy
Goal transition distribution
Input
Output
B
Minimize dual
f ddiveereence
8
B
Optimal Policy %
Offline Data
Offiin Data
Figure Illustration the SMORc objective where 1- 9: SMORc matches a mixture distribution of current policy and offline data to a mixture of the goal-tranae transition distribution and offline data in order to find the optimal goal reaching policy.
convex duality to transform the imitation learning problem into an off-policy optimization problem, removing the need to sample from d= a, g) whilst being able to leverage offline data collected from arbitrary sources.
3.2 SMORE: A DUAL FORMULATION FOR OCCUPANCY MATCHING
The previous section establishes GCRL an occupancy matching problem (Eq. 3) but provides way to use offline data whose joint visitation distribution is given by p(s, g). To leverage offline data to learn performant goal-reaching policies, we consider surrogate objective to the occupancy matching learning problem by matching mixture distributions:
min D (Mi (d' Mixx({, p)).
(5)
where for any two distributions f41 and #2, Mi x K22 denotes the mixture distribution with coefficient 3 (0, 1] defined as Mi (2 M2) 3p1 + B) Proposition 2 (in appendix) shows the matching mixture distribution provably maximizes lower bound to the Lagrangian relaxation of the max-entropy GCRL objective subject to a dataset regularization constraint. We can rewrite the mixture occupancy matching objective as convex program with linear constraints (Manne, 1960; Nachum and Dai, 2020):
max - DD,(MMM x p)][Mixs((.)) ()) a..d
s.t d(s a) (1- (a|s) + + E.ve d(s', a', g)p(s|s', (ilis,a'))tt a')x(a'(s'.g)) E. (6) illustration of this objective can be found Figure 1. Effectively, we have simply rewritten Eq. 5 into an equivalent problem by considering an arbitrary probability distribution d(s, a, g) in the optimization objective, only to later constrain to be a valid probability distribution induced by some policy Fy using the Bellman-flow construints. The motivation behind this construction of the primal form is that we have made computing the Lagrangian-dee with linear constraints. Theorem shows that can leverage tools from convex duality to obtain an unconstrained dual problem that does not require computing a (s, a, g) or sampling from it, while effectively leveraging offline data.
easier as this objective is convex
Theorem The dual problem the primal occupancy matching objective (Equation 6) given by: max min B1 - Edo. [S(s, a, 9)] R (1 pm, S(s, a, g) S(s, a, ())
(7)
Ry S
- (1- 1 a. S(s, a)),,
where f is conjugate function of f and S is the Lagrange dual variable defined as S : S x A x G - R. Moreover, as strong duality holds from Slater's conditions the primal and dual share the same optimal solution σ. for any offline transition distribution p.
To our knowledge, the closest prior works to our proposed method are GoFAR (Ma et al., 2022) and Dual-RL (Sikchi et al., 2023). GoFAR considers the special case of KL-divergence for the imitation formulation and derives a dual objective that requires learning the density ratio p(s.g) in the ((*.g)
Note that Eq. 5 shares the same global optima as the previous occupancy matching objective at (s, a, q(s, a, when is an achievable visitation under some policy and recovers the original objective in Eq. 3 when
4

---

Published as a conference paper at ICLR 2024
form of discriminator and using this pseudo-reward. This leads to compounding errors in the downstream RL optimization when learning the density ratio is challenging, c.g. in the case of low coverage between p(s. a, g) and q(s, a. g). We show this phenomenon experimentally in Section 4.3. Dual-RI. (Sikchi et al., 2023) uses convex duality for matching visitation distribution of realizable expert demonstrations and does not deal with the GCRL setting. Our contribution is novel method for GCRL that is discriminator-free applicable for a number of f-divergences, and robust to low coverage of goals in the offline dataset.
Sampling from the goal-transition distribution: Goal relabelling is an effective technique to address reward sparsity by widening the training goal distribution qº r**an (g). It utilizes knowledge about reaching other goals, possibly unrelated to test goals, to help in reaching the test distribution of goals qt glest(g (g). the most general case, qrrain (g) can set to a uniform distribution over goals corresponding to all the states in the offine data. A common method, Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) chooses a training goal distribution that depends on the current sampled state from the offline dataset as well as the data-collecting policies. In this setting, the sampling distribution used for training Eq 7, p(8, a, 9), can no longer be factorized into p(s, s, a) and ra (g), as goals conditionally dependent
acttinccions. However, our formulation can naturally account for learning from such relabelled data as the SMORe objective in Eq 7 is derived considering the joint distribution p(s, a, g). In this setting, we construct our goal transition distribution q(8, a, g) as the uniform distribution over all transitions that lead to the goals selected by the HER procedure in practice. this amounts to first selecting g through HER and then selecting {s, af that transitions to the selected goal from the offline dataset to get a sample {s, a, g) from goal transition distribution. We emphasize that relabelling does not change the test distribution of goals, which is an immutable property of the environment.
3.3 PRACTICAL ALGORIIIM
To devise a stable learning algorithm we consider the Pearson X² divergence. Pearson x" divergence has been found to lead to distribution matching objectives that are stable to train as a result of a smooth quadratic generator funetion f (Garg et al., 2021; AI-Hafez et al., 2023; Sikchi et al., 2023).
Our dual formulation SMORe simplifies to the following objective:
Decrease score transitions under current policy
maxa, ming A(1 - (..)) (p LS (E(...)) (is.a.))-[[((.....) [S(s, 9.9
(S(s, g)l + B
,'99)
+0.25 (isss ((7S(s', a(s
(8)
Incresse soore at the proposed goal transition distribution
Smoochess//Rllmmn regularization
Equation 8 suggests a contrastive procedure, maximizing the score at the goal-transition dis- tribution and minimizing the score at the offline data distribution under the current policy with Bellman regularization. The Bellman regularization has the interpretation of discourag- ing neighboring
values from deviating far and smoothing the score landscape. Instantiat- ing with KL divergence results in an objective with similar intuition while resembling an In- foNCE (Oord et al., 2018) objective. Although Propositions and suggest that KL divergence gives an objective that is a tighter bound to the GCRL objective, prior work has found KI di- vergence to be unstable in practice (Sikchi et al., 2023; Garg et al., 2023) for dual optimization.
It important to note that S-function is not grounded to any rewards and does not serve as probability density of reaching goals, but is rather a score func-
Algorithm 1: SMORe
1: Init Sp, My. and ne
tion learned via a gellman-regularizan contrastive learning procedure.
2: Params: expectile т, mixture ratio B, temperature
3: Let = م = {(s, a, &' he an offline dataset and g be goal-transition distribution
We now derive practical approach for SMORe in the offline GCRL setting. We use parameterized func- tions: So(s, a, g). Ms.(s.g), To(als.9) g). The offline learning regime necessitates measures to constrain the learning policy to the offline data support in order to prevent overestimation due to maximizing Tg in Eq. over potentially out-of-distributio actions. In- spired by prior work (Kostrikov et al., 2021), we use implicit maximization to constrain the learning algorithm to learn expectiles using the observed empirical samples. More concretely, expectile
4:
for 1 = 1..7 iterations do
5:
Train Sa, via Eq. 10
6: Train M. via Eq. 9
7: Update To via Eq. 11
8: end for
5

---

Published as a conference paper at ICLR 2024
regression:
min [(1b) ((sa))>p(2((MMss.9))) Sa(s. a)g))..
where Lī(n) = - I(u 0)/u². Intuitively, this step implements the maximization w.r.t n by using
expectile regression. With the above practical considerations, our objective for learning Sy reduces to:
(s min L(Ø) = (e (. (ss))
-l4a.))-1155( a, 9)] (.ag://...l.........a.a.a. (asng))-wraaaa -
a,(s,a.9))*]
(10)
where we have the offline data distribution our initial state distribution. Finally, the policy is extracted via advintage-weighted regression that learns in-distribution actions maximizing the score S(s, a. g):
min L())= := Brs.s)-peesp((((S((ss... Ma(s, log(To(a|ss,))
(11)
where a is the temperature parameter. Algorithm details the practical implementation.
EXPERIMENTS
Our experiments study the effectiveness of proposed GCRL algorithm SMORe on set of simulated benchmarks against other GCRL methods that employ behavior cloning, RL with sparse reward, and contrastive learning. We also analyze if SMORe robust to environment stochasticity number of prior methods are based on an assumption of deterministic dynamics. Then, we study the discriminator-fii free nature of SMORe is indeed able to prevent performance degradation in the face of low expert coverage in offline data. Finally, we analyze if SMORe's score-modeling approach helps SMORe scale to vision-based manipulation offline GCRL benchmark, as density model- ing and discriminator learning become increasingly difficult with tigh-dimensional observations. Hyperparameter ablations can be found in Appendix D.
4.1 EXPERIMENTAL SETUP
Our experiments will a suite of simulated goal-conditioned tasks extending the tasks from previous work (Ma et al., 2022; Plappert et al., 2018). In particular we consider the following enviromments: Reacher, Robotic arm environments [SawyerReach, SawyerDoor, chReach FetchPick, FetchPush, FetchSlide], Anthropomorphic hand environment HandReach and Locomotion environments Che Vel-re, AntTgtVel-me Ant Tasks all environments are specified by sparse reward function. Depending on whether task involves object manipulation, the goal distribution is defined over valid configurations in robot or object space. The offline dataset for manipulation tasks consists transitions collected by random policy mixture of 90% random policy and 10% expert policy. For locomotion tasks, generate dataset using the D4RL benchmark (Fu et al., 2020), combining a random or medium dataset with 30 episodes of expert data. Note that the policies used collect the expert locomotion datasets have different objective than the tasks here, which are to achieve and maintain a particular desired velocity.
[CheetahTgt
4.2 OFFLINE GOAL-CONDITIONED RI. BENCHMARK
Baselines. We compare to state-ofart offline GCRL algorithms, consisting both regression-based and actor-critic methods. The eccupancy-matchiin based methods are: (1) GoFar (Ma et al., 2022), which derives a dual objective for GCRL based coverage assumption. The behavior cloning based methods are: (1) GCSL (Ghosh et al., 2019), which incorporates hindsight relabeling in conjunction with behavior cloning to clone actions that lead to a specified goal, and (2) WGCSL (Yang et al., 2022), which improves upon GCSL by incorporating discount factor and advantage weighting into the supervised policy learning update. Contrastive RL (Eysenbach et al., 2022b) generalizes C- learning (Eysenbach et al., 2020) and represents contrastive GCRL approaches. The RL with sparse reward methods are (1) IQL (Kostrikov et al., 2021) where we use a state-of-the ofthe-of-the-att offline RL method repurposed for GCRL along with HER (Andrychowicze al., 2017) 7) goal sampling, and (2)

---

Published as a conference paper at ICLR 2024
Task
Occupancy Matching SMORe GoFAR
Behavior cloning WGCSL.
Contrastive RL
RL+sparse reward
GCSL.
CRL
AM
IQL
Reacher (+)
28.40
19.74
17.57
15.87::::
16.44
23.26 10.4
11.70 1::4
SawyerReach (+)
37.67 +0.12
15. t063
15. 45
14.25+
22.32
23.34
35.18
SawyerDoor (+)
18.941nm
20.01
20188-10m
12.96:14
22.12 10.
25.52
1:14
FetchReach (+)
35,08
28.2 28.2
21.9
20.91 27
30.07
30.1
34,43
FetchPiick (+)
26.47 + 1.34
19.7 257
9.84 + 258
1.58
0.42:to29
8,94
+: 3m
16.8 t: 1.80
PetchPush (x)
26.83+ 1:21
18.2
14.7
13.4
2,40
14.0
22.40
6.74
FetchSlide
4.99 0:40
2.47 + 1/4
2.73 + 1/44
1.75 +: L3
0.0
1.46 +: 1.38
4,80 t: 1.99
HandRReach (+)
18.68
3,3
11.5
5.26
1.37
0.0100
1.35
CheetahT VeelaahT (x)
136.71 + 10.91
0.0+
0.0+ 0e
95. 98 1977
0.0
0.0+
100,38+ 38 1:32
CheetahTgtVel-ee (x)
60.01
39,00
0.0+ 60
0.0+t 06
11.56 13,47
0.0
0,0+
0,0+
AntT-tVel-m-- Vel
154.95:
168.27: 9.58
0.0:t: o8
164.544t 7.0
0.0.000
0.01
148.17 1 5.4
126.22.1 14:00
74.36.1 1890
0.0.1 o.
104.95 6:00
0.0.1oM
0.0. o0
3,06 1 2.44
Table I: Discounted Return for the offline GCRL benchmark. Results are averaged over 10 sceds. 'm-e' and 'r-c' stands for medium-expee mixture and random-ex expert mixture respectively. (+) denotes statistically significant improvements.
Medi (AM) (Chebotar et al., 2021), which incorporates conservative Q-Learning (Kumar ct al., 2020) as well as goal-chaining on top of an actor-critic method.
The results for all baselines are tuned individually, particularly the best HER ratio searched among (0.2, 0.5, 0.8, 1.0} for each task. SMORe shares the same network architecture for baselines and uses mixture ratio of 3 0.5. Each method is trained for 10 seeds. Complete architecture and hyperparameter table as well as additional training details are provided in Appendix C.
Table reports the discounted return obtained by the learned policy with a sparse binary task reward. (+) denotes statistically significant improvement over the second best method under a Mann- Whitney U
FetchReach: Robustness to noise
Discounted Return
3.0
test with a significance level of 0.05. This metric allows us to compare the algorithms on a finer scale to understand which methods reach the goal as fast as possible and stay in the goal region thereafter for the longest time. Additional results on metrics like success rate and final distance to goal can be found in the appendix. These additional metrics do not take into consideration how precisely and consistently goal is being reached. In Table 1, we see that SMORe enjoys nigh-performance gain consistently across all tasks in the extended offline GCRL benchmark.
25
20
1.
GaFAR
CRI
W92S7
Noise Level
Figure 2: SMORe is robust in stochastic environ- ments. With increasing noise, SMCRe still outper- forms prior methods.
Robustness to environment stochasticity: We consider noisy version of the FetchReach environment in this experiment. Gaussian zero-mean noise is added before executing an action to generate different variants of the environment with standard deviations of {0.5, 1.0, 1.5). Datasets for these environments obtained from prior work (Ma et al., 2022). As we see in Figure 2, SMORe robust to stochasticity in the environment, outperforming baselines in terms discounted return. Behavior cloning based approaches assume deterministic dynamics and are therefore over-optimistic in stochastic environments.
4.3 ROBUSTNESS OF OCCUPANCY-MATCHINS METHODS TO DECREASING EXPERT COVERAGE
We posit that the discriminator-frre nature of SMORe makes it more robust to decreasing goal coverage, as it does not suffer from cascading errors stemming from learned discriminator. In this section, set out to test this hypothesis by decreasing the amount of expert data in the offline goal-reaching dataset. We compare with GoFAR in Table 2 due to the similarity between methods and GoFAR's restrictive assumption coverage of expert data in the suboptimal dataset. Comparison against all the baselines can be found in Appendix D.
Our hypothesis holds true as we in Table 2, the performance of the discriminator-baste method GoFar rapidly decays as expert data is decreased in the offline dataset 28.4% with 2.5% and 36.15 15% with 1% expert data(i.e. optimal policy's coverage) respectively. SMORe shows much slower decay
7

---

Published as a conference paper at ICLR 2024
expert duta
expert
expert
SMORe
GoFAR
SMORe
GoFAR
SMORe
GoFAR
12.20
SawyerReach
36.35 Los7
13.20 1::3
36.74 c.2
11.57 11ze
35.44 oz7
9,34 I.IT
SawyerDoor
32.82:tuss
19.54
11:00
FotchReach
36.00+ om
27.66
t: 0.55
35.58 947
27.84 012
35.97 + 0.5
28.01 +
FetchPick
26.43+
16.21
26.174
3.21
15.38
0.31
TetchPush
23.81.1 o:7
18.2 7m
22.75_111
5.17 1: mm
19.04 279
4,23. 3.6
FetchSlide
4.05
1.08
3.11
0.96
3,50#
0.86
Average Performance
25.98
16.18
23,99
11.49
21.81
10.33
Avg. Perf. Drup
-7.6%
-8.4%
116%
16.155
Table 2: Discounted Return for the offline GCRL benchmark with 5%, 2.5% and 1% expert data in offline datasct. Results are averaged over 10 seeds.
in performance, 7.6% with 2.5% and 16% with 1% expert data, attesting to the method's 's robustness under decreasing expert coverage in the offline dataset.
4.4 OFFLINE GCRL IMAGE OBSERVATIONS
SMORe provides effective algorithm for offline GCRL in high-dimensional observation spaces by learning unnormalized scores using a contrastive procedure as opposed to prior works that learn normalized densities (Eysenbach et al., 2020) which are difficult learn or density ratios (Eysenbach et al., 2022b; Zheng et al., 2023) which do not optimize for the optimal goal-conditioned policy in the offline GCRL setting. Similar prior work (Bysenbach al., 2022b), we consider the following structure in S-function parameterization to learn performant and generalizable policies: S(s, g) ((s, a) (g)).. The S-function can be interpreted the similarity between the two representations given by é and e). Our network architecture for both representations is similar Zheng al. (2023) and is kept the same across baselines ensure fair comparison of the underlying GCRL method.
We use the offline GCRL benchmark from (Zheng al., 2021) which learns goal-reaching policies from an image-obsse coserroobsrvvtiion dataset of 250K transitions with the horizon ranging from 50-100. The benchmark adds another layer of complexity by testing on goals absent from the dataset dataset contains primitive behaviors like picking up objects and pushing drawers but no behavior that completes the compound task we consider from the initial state. The observations and goals are 48x48x3 RGB images.
Baselines We compare to the best performing GCRL algorithms from Section as well as a recent state-of of-he-of work, stable contrastive RL Zheng et al. (2023). Stable contrastive RL features a number of improvements over contrastive RL by changing design decisions in neural network architecture, layer normalization, and data augmentation. Since objective is compare the quality of the underlying GCRL. algorithm, we keep these design decisions consistent across the board.
Results Figure 3 shows the success rate on a variety of unseen tasks for all the methods. SMORe achieves highest success rates across all the tasks, even for the most challenging task of pick, place
SMORe GCBC
Stable Contrastive
GC-IQL
Success Rate
t 0
t=
f= 0
t=
0.3
0.6
10.4
0.2
pick
blcc open drawer
pick placoldraweri
goal
goal
goal
goal
placetuble)
Evaluation on simulated manipulation tasks with image observations. The left image shows the starting state the top and the goal the bottom for evaluation tasks. The error bars show the standard deviation with random seeds. SMORe is competitive or outperforms prior methods all the tasks we considered.
Figure 3:
8

---

Published as a conference paper at ICLR 2024
and closing the drawer. We note that our results differ from Zheng et al. (2023) for the baselines as we apply the same design decisions for all methods whereas Zheng et al. (2023) focuses on ablating design decisions.
RELATED WORKS
Offline Goal Conditioned Reinforcement Learning. Learning to achieve goals in the enviromment optimally forms the basis of goal-condition RL problems. Studies in cognitive science (Molinaro and Collins, 2023) underscore the importance goal-achieving plays in human development. Offline GCRL approaches are typically catered to designing learning algorithms for addressing the sparsity of reward function in the offline setting. One of the most successful techniques in this setting has been hindsight relabelling. Hindsight--xxperience relabelling (HER) (Kaelbling, 1993; Andrychowicz al., 2017) suggests relabelling any experience with some commanded goal to the goal that was actually achieved in order to leverage generalization. HER has been investigated in the setting of learning from demonstrations (Ding et al., 2019) and exploration (Fang al., 2019) to validate its effectiveness. A number of prior works (Ghosh et al., 2019; Yang et al., 2019; Chen et al., 2020; Ding et al., 2019; Lynch et al., 2020; Paster et al., 2020; Srivastava al., 2019; Hejna et al., 2023) have investigated using goal-conditioned behavior cloning, a strategy that uses relabelling to learn goal-conditioned policies, as a way to learn performant policies. Eysenbach et al. (2022a) shows that this line of work has limitation of learning suboptimal policies that do not consistently improve over the policy that collected the dataset. The simplest strategy of applying single-task RL to the problem of multi-task goal reaching requires learning a Q-function which represents normalized densities over the state-action space. Contrastive RI. (Eysenbach et al., 2022b; 2020; Zheng et al., 2023) emerged as another alternative for GCRL which relabels trajectories and, rather than use that relabelling to learn policies, learns a Q-function using a contrastive procedure. While these approaches learn optimal policies in the online setting, they fall behind in the offline setting where they only learn a policy that greedily improves over the Q-function of the data collecting policy, Our work learns optimal policies by presenting an off- aff-policy objective that solves GCRL and furthermore learns scores (or unnormalized densities) that alleviate the learning challenges of normalized density estimation.
Our approach is inspired by the distribution matching ap- proach (Ghasemipour et al., 2020; Ni et al., 2021; Sikchi et al., 2022; Swamy et al., 2021; Sikchi et al., 2023) prominent in imitation learning. Ghasemipour et al. (2020); Ni et al. (2021) 1) takes the problem of imitating an expert demonstrator in the environment and converts it into problem of distribution matching between the current policy's state-action visitation distribution and the expert policy's visitation distribution. Indeed, prior work f-PG (Agarwal et al., 2024) proposes distribution matching approach to GCRL but is restricted to the on on-policy setting. Another, prior work (Ma et al., 2022) creates one such distribution matching problem and presents new optimization problem for GCRI. in the form of an off-policy dual (Nachum and Dai, 2020; Sikchi et al., 2023). Such an off-policy dual is very appealing for the offline RL setup, as optimizing for this dual only requires sampling from the offline data distribution. A limitation of their dual construction is the fact that they require learning a discriminator and use that discriminator as the pseudo-rewars for solving the GCRL objective. Our approach presents a new construction for GCRL as distribution matching problem along with a dual construction that leads more performant discriminator free off policy approach for GCRL.
Distribution matching.
6 CONCLUSION
Prior work in performant online goal-conditioned RL often relies on iterated behavior cloning or contrastive RI. However, these approaches are suboptimal for the offline setting. Existing methods specifically derived for offline GCRL require learning discriminator and using it as a pseudo-reward, enabling compounding errors that make the resulting policy ineffective. We present an occupancy- matching approach to offline GCRL that provably optimizes a lower bound to the regularized GCRL objective. Our method is diseriminator- free, applicable to a number of f-divergences, and learns unnormalized scores over actions at a state reach the goal. We show that these positive aspects of our algorithm allow us to empirically outperform prior methods, stay robust under decreasing goal coverage, and scale high-dimensional observation space for GCRL.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank the MIDI lab members and ICLR reviewers for valuable feedback on this work. This work has taken place in the Safe, Correct, and Aligned Learning and Robotics Lab (SCALAR) at The University of Massachusetts Amherst and Machine Intelligence through Decision-making and Interaction (MIDI) Lab The University of Texas Austin. SCALAR research supported in part by the NSF (IS-2323384), AFOSR (FA9550-20-1-0007), and ARO (78372-CS, W911NE- 19-2-0333). and the Center for AI Safety (CAIS). This research was also sponsored by the Army Research Office under Cooperative Agreement Number 1911NN-11-2-055
HS and AZ are funded in part by a sponsored research agreement with Cisco Systems Inc. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
A. Agarwal, N. Jiang. S. M. Kakade, and W. Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pages 10-4, 2019.
S. Agarwal, I. Durugkar, P. Stone, and A. Zhang. f-policy gradients: A general framework for goal-conditioned rl using f-divergences. Advances in Neural Information Processing Systems, 36, 2024.
F. AI-Hafez, D. Tateo, O. Arenz, G. Zhao, and J. Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. arXiv preprint arXiv:2303. 005999 2023.
M. Andrychowicz, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov. J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.0770 2021.
L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self-supervieed reward regression. arXiv preprint arXiv:2010.11723, 2020.
Y. Ding, C. Florensa, P. Abbeel, and M. Phiclipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
F. Ebert, Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Danilidiss, C. Finn, and S. Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109, 13396, 2021.
B. Eysenbach, R. Salakhutdinov, and S. Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909 2020.
B. Eysenbach, S. Udatha, R. R. Salakhutdinov, and S. Levine. Imitating past successes can be very suboptimal. Advances in Neural Information Processing Systems, 35:6047-6059, 2022a.
B. Eysenbach, T. Zhang, S. Levine, and R. R. Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603- 2022b.
M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang. Curriculum- hindsight experience replay. Advances in neural information processing systems, 32, 2019.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiy preprint wxXiv:2004002//9 2020.
D. Garg, S. Chakraborty, C. Cundy, J. Song, and Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 4039, 2021.
10

---

Published as a conference paper at ICLR 2024
D. Garg, J. Hejna, M. Geist, and S. Ermon. Extreme q-leaming: Maxent rl without entropy. arXiv preprint arXiv:2301.022 023288, 2023.
S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
D. Ghosh, A. Gupta, A. Reddy, J. Fu, Devin, B. Eysenbach, and Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 2019.
J. Hejna, J. Gao, and D. Sadigh. Distance weighted supervised learning for offline interaction data. arXiv preprint arXiv:2304. 13774, 2023.
L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. Citeseer, 1993.
I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv: 2019.
I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2 0606666, 2021.
A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 179-1199,, 2020.
C. Lynch, M. Khansari, Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1 1111-11332 PMLR, 2020.
Y. J. Ma, J. Yan, D. Jayaraman, and O. Bastani. How far i'll go: Offline goal-conditioned reinforce- ment learning via f-advantage regression. arXiv preprint arXhv:2206.03023, 2022.
A. Manne. Linear programming and sequential decisions. Management Science, 6(3):259-267, 1960.
G. Molinaro and A. G. Collins. A goal-centric outlook on learning. Trends in Cognitive Sciences, 2023.
O. Nachum and B. Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01 01 0166, 2020.
Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021.
A. v. d. Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03748, 2018
A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023.
K. Paster, S. A. Mcllraith, and J. Ba. Planning from pixels using inverse dynamics models. arXiv preprint ar arXiv:2012.02419, 2020.
M. Plappert, M. Andrychowicz. A. Ray, McGrew, B. Baker, G. Powell, J. Schneider, J. To- bin, M. Chociej, P. Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 180.00000000944. 2018.
A. Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume I: Contributions to the Theory of Statistics, pages 547-561. University of California Press, 1961.
H. Sikchi, Saran, W. Goo, and S. Niekum. ranking game for imitation learning. arXiv preprint 22X/0:2202.03481, 2022.
H. Sikchi, Q. Zheng. A. Zhang, and S. Niekum. Dual rl: Unification and new methods for rein- forcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning. 2023.
11

---

Published as a conference paper at ICLR 2024
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pages 387-395. Pmlr, 2014.
R. K. Srivastava, P. Shyam, F. Mutz, W. Jaskowski, and J. Schmidhuber. Training agents using upside-down reinforcement learning. arXiv preprint arXiv:1912.02877, 2019.
G. Swamy, S. Choudhury, J. Bagnell, and Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021.
H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng. T. Zhao, Hansen- Estrucc,, Q. Vuong, A. He, et al. Bridgedata v2: dataset for robot learning at scale. arXiv preprint arXiv:2308. /2952, 2023.
Y. Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning. arXiv preprint
2019.
H. Xu, L. Jiang, Li, Z. Yang, Z. Wang, W. K. Chan, and Zhan. Offline with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXiv:2303./588 2023.
C. Yang. X. Ma, W. Huang, F. Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. arXiv preprint 191v:4111. 2019.
R. Yang. Y. Lu, W. Li, H. Sun, M. Fang, Y. Du, X. Li, L. Han, and C. Zhang. Rethinking goal- conditioned supervised learning and its connection offline rl. arXiv preprint arXiv:2202.04478, 2022.
C. Zheng, B. Eysenbach, H. Walke, P. Yin, K. Fang, R. Salakhutdinov, and S. Levine. Stabilizing contrastive rl: Techniques for offline goal reaching. arXiv preprint vrXiv:2306.03340 2023.
L. Zheng, Fiez, Z. Alumbaugh, B. Chasnov, and L. J. Ratliff. Stackelberg actor-critic: Game- theoretic reinforcement learning algorithms. arXiv preprint 11Xi::2109. 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 THEORY
In this section, we first show the equivalence of the GCRL problem and the distribution-matchee objective of imitation learning. Then, we show how the mixture distribution objective relates to offline GCRL objective. Finally, we derive the dual objective for mixture distribution matching that leads to our method SMORe.
A.1.1 REDUCTION OF GCRL TO DISTRIBUTION MATCHING
Proposition 1. Consider stochastic MDP, stochastic policy 9,, and sparse reward function r(s,a, a(s,a,g) = -pt-s(-)s.a) [D(((s') = 9.9 ( (g) ) > 0)] where 1 is an indicator function. Define a soft goal transition distribution to q(s, a, g) a exp(a r(s, a, g)). The following bounds hold any f-divergence that upper bounds KL-divergence (eg. x², Jensen-Shunnon):
1 jtrain (Tg) + --( Dd a, (allg((s,)) a. + a where H. denotes the entropy. a temperature parameter and is the partition function for M.s...)) Furthermore, the bound is tight when f is the KL-divergence.
(4)
a
Proof. This proof adapted from Ma al. (2022) for goal transition distributions and state-action distributions. Let Z [e ds da dg and > be the temperatue parameter. Note that q(s, (. = (s.a.)) where r is defined in the proposition, strictly generalizes the original definition
q(s, g. grran (()es/=()))))))((((((
9)] and recovers it when co. Starting with the true
GCRL objective:
aJ (m²) = Egro [al(s, (0((s, a.g) g)]
(12)
= Egro log (((,,a.))))
(13)
B( d= d= 2)
Egr
log(
(14)
(s
log( ((s,a,g) 2) + Exro Earollog ]
Egr
(15)
= DKL ( (s, g)lll(s) a,g)) - =(dd00) )) + log ())
(16)
Rearranging terms we get:
= - (s. g)) + a
((Tg) + a
H(d)))
(17)
For any f-divergence that upper bounds the KL divergence we have:
J(T)) + - H(dd))) = (s.. +C 1
( +
(18)
A (dataset) regularized GCRL objective: Define regularized objective for GCRL as follows:
Joffline ④ 11 (((...))) ag a, [.]].
(19)
The second term in the objective (s.a.,...)) [p(s, a, a.g)] above is maximized when the policy visitation places more probability mass on the most visited transitions in the dataset. To see why this is, consider two probability distributions represented as vectors dF and p with individuals elements of the vector indexed by i:
(d', & max
(20)
The equality holds only when d places probability mass all state-action-goal tuples which are most visited in the offline dataset p. The first term maximizes the true GCRL objective while the second term prefers staying close to transitions that most frequently observed in the offline dataset. constraint of (d",, p) > 8 implies that the agent visitation places atleast half of the probability mass iite-action-gol tuples whose average visitation in offline dataset is greater than equal
13

---

Published as a conference paper at ICLR 2024
to 1 1 8. With weights a1 and ag. the objective above reflects an lagrangian relaxation to this constraint. Thus the above offline objective presents an alternative offline objective when compared to the classical offline RL objectives Wu et al. (2019); Nachum and Dai (2020).
Proposition 2 derives the connection between the dataset regularized GCRL objective and SMORe:
Proposition 2. Consider stochastic MDP, a stochastic policy n, and a sparse reward function r(s, a) -xt-p()s.aa) [[(((s') = g, q frasn (g) > 0)] where is an indicator function, define a soft goal transition distribution be q(s, a, g) expla r(s,a,)) a.))) the following bounds hold for any f-divergence that upper bounds KL-divergence (eg. x². Jensen-Shannony
log (liftine(ty + H(Mixx xaliixx(ddd p) a, 9)) + + = -Dy(Mixgl p) a, () g)llMii)M (), s, a, 9)
(21)
where H denotes the entropy, or is a lemperature parameter, on = 3. 02 3(1 3)2 and C is positive constant. Furthermore, the bound is light when f is the Kl-divergence.
Proof. We first consider the following two objectives for GCRL and show that they are equivalent. This reduction will later help in proving connection to mixture occupancy matching, We consider w.l.o.g. Here are two objectives we consider:
( EBa= [r(s, a g)
(22)
J'(T) = E e (s,...)))
(23)
In GCRL reward functions are sparse and binary. We show the equivalence of first two objectives in find the optimal goal conditioned policy via two arguments. First, notice that the rewards for goal transition states for objective JJ JJ((T) is and 1 for all other transitions. This is in contrast to J(T) which considers reward function at goal transitions states and otherwise. Under our assumption of infinite horizon discounted MDP, we can translate the rewards while keeping the optimal policy 1 goal transitions states and 0 otherwise. Further can scale the rewards by 1/(e- 1) and recover and MDP with same optimal policy that has reward of 1 goal-transition states and 0 otherwise. This concludes the equivalence maximizing J'(TT as an alternative to (TT) while recovering the same optimal policy.
same in MDP considered by J'(xz)
We now consider a regularized (pessimistic/offline)) GCRL problem with the shifted reward functions e(ss...)) that maximizes the reward while ensuring the policy visitation stays close offline data visitation in cosine similarity.
ai Egr e ((.as))) agBar (sagg))s a, a,g)]
(24)
With particular instantiation of hyperparameters we show that the fointin((T)) objective can be simplified to an equivalent objective J' J' f(ne(() by setting ai = B² and 02 = 3(1 - B) where Z the partition function for ((s..g) over entire S Ax G.
foffeine -Exing(d,o
Berlea.s) + (1- (-))ppss,,,,)1
(25)
Joffline -xxxx(d, Be + (1 3) 9)
(26)
B ((
++3(1
(27)
+ (1 -1- =) (( (1 ()p(ss..g).21)
(28)
(29)
B ((
+ +1111 (11-)))EE=ll(s,a..pp
(30)
Josf (T) + C"
(31)
Now that we have shown J'f fline = (offline(T)) and hence solving the same optimization problem, we proceed to derive connections with mixture occupancy matching which follows through an application of lensen's 's inequality:
14

---

Published as a conference paper at ICLR 2024
log Jof flime log (xixx(d. Belbaa) + (
a.)
(32)
(dixx(d,))e..sss log(Be
(33)
(34)
g) + (1 (1-3))((,,,)p))l log -DDxi[Mixx(C (s, a, g) p)
(35)
(MMixs(d, p) (s, a, g)) + log Z
(36)
For any f-divergence that upperbounds the KL divergence since Z we have:
p), log foi + H(Mixs(d, p).,, 2- (b
(37)
Further simplifying using Eq 31: log fline (T)+ -M (4MMixx(4 - a,
-
(38)
Optimizing the mixture distribution matching objective of SMORe maximizes a variant of of- fline/dataset regularized GCRL objective where the entropy for distribution Mi Mixa(d, p) a, g) is jointly maximized. Therefore we have shown that the minimizing discrepancy of mixture distribu- tion occupancy maximizes lower bounds offline variant of maxent GCRL objective.
A.2 CONVEX CONJUGATES AND f F-DIVERGENCES
We first review the basics of duality in reinforcement learning. Let f R4 IR be convex function. The convex conjugate f R. of f defined by:
(*) supxeB, [xy fx)]].
(39)
The convex conjugates have the important property that f* is also convex and the convex conjugate of f* retrieves back the original function f. We also note an important relation regarding f and fx: (f*) (f')-1, where the notation denotes first derivative.
Going forward, we would be dealing extensively with f-divergences. Informally, f- divergences (Rényi, 1961) are measure of distance between two probability distributions. Here's more formal definition:
Let P and Q two probability distributions over space such that absolutely continuous with respect Q* For function f: R4 IR that is convex lower semi-continuous and f(1) 0. the f-divergence of P from Q is
(le) ((z)
Dy = Ez-q
(Q()
(40)
Table lists some common f-divergences with their generator functions f and the conjugate functions f+.
A.3 SMORE: DUAL OBJECTIVE FOR OFFLINE GOAL CONDITIONED REINFORCEMENT LEARNING
In this section, derive the dual objective for solving the multi-task occupancy problem formulation for GCRL. First, we derive the original variant of SMORe for the GCRL problem and later derive the action-free SMORe variant for the interested readers.
*Let denote the random variahle. For any measurable 7 Q(2 Z) 0 implics P(z Z) 0.
15

---

Published as a conference paper at ICLR 2024
Divergence Name
Generator (a) x: log x
Conjugate (")
KL (Reverse)
e(v-1)
Squared Hellinger
(Vz - 1)2
Pearson x2
(x 1)2
y +
Total Variation
# 1|
y y E [- 2. = otherwise dc
Jensen-Shannon
(x + 1) log( 2 + log
- log (2 - -2
Table 3: List of common f f-divergences.
Theorem 1. The dual problem to the primal occupancy matching objective (Equation 6) is given by: max min B(1 Edo (S,,, 9)] + [f S(s, a) S(s, a.s.a,91 a.g)) (7)
2g 5
- (1- (1= S(s, a.g),
where f is conjugate function of f and the Lagrange dual variable defined S : R. Moreover, as strong duality holds from Slater's conditions the primal and dual share the same optimal solution n for any offline transition distribution p.
G
Proof. Recall that: Mi Mixa(d,pp) (s, a,g) 3d(s, a, 8q(s, a.
a, g) and Mix a,g) :-
3)p(s, a, g). Mixs(d, (s,, a. denotes the mixture between the current agent's joint-goal visitation distribution with an offline transition dataset potentially suboptimal and Mixa(g (g. (s. (s, a. the mixture between the expert's visitation distribution with arbitrary experience from the offline transition dataset. Minimizing the divergence between these visitation distributions still solves the occupancy problem, d = q when y achievable. We start with the primal formulation from Eq 6 for mixture divergence regularization:
max ((s.a.g)30,,(al))
-D, (M)xxxx () a. g) Mixx(q, (), a.
s.t d(s, g) = (1 (1->))p(() 7) 9) (als, 9) (T((,) Σ d(d', a' g)p(s(s .').
Applying Lagrangian duality and convex conjugate (39) to this problem, we can convert it to an unconstrained problem with dual variables S(ss a. defined for all s, a E 8 Ax G: max min -Dy(Mixp(d, p) a, Mix (s, as,
x.d20 8(s.a.))
+ Σ a(s,agg) (a- (1-7)do(s,))) v
Σ d(s', d' g)psssss 3)) d(s, a,g) 2.
(41)
max min (lan))) x.d20 ((s.ag))
+ E..a.gd
y
a()
(42)
- -D,(Mix (s, Mixx (
(43)
max min 3(1 (Bac ddd00s(s,a.g
JE gdd
2 S(s,a.g)
s'
(44)
16

---

Published as a conference paper at ICLR 2024
+ (1 3) R
7 [no6)e.nn((1))))1 S(8,
s/,a'
- - 3) s
Σ
S(s,a,g)
(45)
- (Mixxid (s. a. MiMxx(( 0.))
(46)
Now using the fact that strong duality holds in this problem we can swap the inner max and min resulting in:
max min S(s,,g) Mins (xx.
max
8(1 (()
+ Bgaggd
Σ pdlllssam(((1)))sd.d.dd s(s,0,g)
?
+ (1 - 3) 3)aa
7
2 S(s,a,9)
staa'
(1 3) S 7
S(s,a,g)
(47)
DD(MMixs(d, (). pl8....9)llMx((pp(ss..... (IMxx((e))(s....)))
(48)
(49)
We can now apply the convex conjugate (Eq. (39) definition to obtain a closed form for the inner maximization problem simplifying to:
min B(1 (.l
(als_g) S(a.a.))
E, Baa.g-MxooaaDeaa f* Eolllss..goomollssssss... -(ssa)))))s(ss.)))
[ S(s,
(50)
This completes our derivation of the SMORe objective. Since strong duality holds (objective convex, constraints linear and feasible), SMORe and the primal mixture occupancy matching share the same global optima xg.
A.4 ACTION-FREEE SMORE: DUAL-V OBJECTIVE FOR OFFLINE GOAL CONDITIONED REINFORCEMENT LEARNING
The primal problem in Equation 6 is over-constrained. The objective determines the visitation distribution uniquely under a fixed policy. It turns out we can further relax this constraint to get an objective that results in the same optimal solution (Agarwal et al., 2019) n by rewriting our primal formulation as:
max DD (), a, (ixx,(a.p)(s.a,g) d(s,a.g)a)
d(s,a,g) (1 (1-7))o0ss 7 Σ d(s',a' g)p(s)s'.')) e'.a'
(51)
[S(s's))] [00]-S(sg))) S(s,g). The action-free dual problem to the multi-task mixture occupancy matching objective (Equation 51) is given by:
Theorem 2.
Let y(s, a, -Es-p(()a.a) (
min 3(1 (e S(a.g)
+ E BBr.a.gg-Kixafeppls..... [max (0, (o'
(v(s, a, (, a,) /max (0, (v(s, a, g)))
(1 B,).s...g p(ss'ss,a)ss S(s.9)
17

---

Published as a conference paper at ICLR 2024
where S is the lagrange dual variable defined S S x G - R. Moreover, strong duality holds from Slater's conditions the primal and dual share the same optimal solution zy for any offline transition distribution d
Proof. Proceeding as before and applying Lagrangian duality and convex conjugate (39) to this problem, we can convert an unconstrained problem with dual variables (s,, defined for all 8, E S 9:
max min - (M,(Mixg((,)) p00
d=0 S(s.g)
Σ (() (1 (1- + +y Σ d(s'.dd.)ppssss Ed(s,a.g)
(52)
max min (1 7)
d S(s.g)
+ E. a.
7 M 5(s,g)
5(s,9)
(53)
Dy (M(MMxx(( p1
(54)
= max min 3(1 7) EAscs
d30 S(s.9)
+ agd
? Σ p(s'|s, 2((')ss S(0,9)
+ (1 -
E)(0)),,0)(($510) -
(1 3)
Ep(dlss S(s,g)
(55)
(M ( a,, M (s., a,))
(56)
Now using the fact that strong duality holds in this problem can swap the inner max and min resulting in:
min S(s.9) six ( ())
max
3(1 F)o (S(s,))) (5(.
+ 3E B.dd
? ()) S(s.g)
+ (1 F)
Ep( S(s,g)
(1 5) E
Ep(s'(s S(s.g)
(57)
(M ( a, (s,
(58)
Unlike previous case where constraints uniquely define valid d for any given in this case need to take into account the hidden constraint d > 0 or equivalently Mixs(d, p) a, g) > To incorporate the non-negativity constraints consider the inner maximization separately and derive a closed-form solution that adheres to the non-negativity constraints. Recall y(s, a. = E [S (S(s'9)) .)]] S(s,g).
18

---

Published as a conference paper at ICLR 2024
E (asaga Mixe (..
max
Ep(d S(s,g)
(Myxxi p), () a.g))
We can now construct the Lagrangian dual to incorporate the constraint Mi Mixa(d, (s, (, > 0 its equivalent form w(s, a. > 0 and obtain the following where w - Mixg(d,p)(s.a.g)
mixa(g-p)ls.agg)
max B ae a, g)y(g))(s)s)))) a. 9) Bfuss w(s,o.) A35
Σ X(w(s, 6.0.8
(59)
Since strong duality holds, we can use the KKT constraints to find the solutions w² (s, a, g) and X* (* a. g).
1. Primal feasibility: w= os, 3 0 Vs,a
2. Dual feasibility: X*
0 ssa
3. Stationarity: Mix (1 v(s,a,g) (*, a, V
4. Complementary Slackness:
(1) ss
Using stationarity we have the following:
f'(w" (s, a, g)) = y(s, a, + >" vs
(60)
Now using complementary slackness, only two cases are possible w* (s, g) 0 or (* a, Combining both cases we arrive at the following solution for this constrained optimization:
w* (s, a) max (D, (v(s,,,g))
(61)
Using the optimal closed-form solution (w") for Mi Mix (), a, g) of the inner optimization in Eq. (57) we obtain
min B(1 7) [S(ss) ((s.a)
E xs [max (0. (v(s, (oss..)) v(s,a,g) S(s,g)
of (max (o,
(y(s, a, 9))))]
(1 -a
(62)
For deterministic dynamics, this reduces the action-free SMORe objective: min B(1 - Bdn (S( (8)
S(s.a)
(v(s, mmax (v(s,") v(s,a,g) - - -11- B)-))Booo
E B.
f (max (0, (v(s, a, g)))
S(s,))
(63)
where y(s, a, = 7S(s' S(s,g)
Note that we longer need actions in the offline dataset learn optimal goal conditioned score function. This score function can be used to learn presentation in action-free datasets as well as for transfer of value function across differing cediii-modalities where agents share the same observation space (eg. images as observations).
19

---

Published as a conference paper at ICLR 2024
B SMORE ALGORITHMIC DETAILS
B.I SMORE WITH COMMON /DIVEERGENCES
KL divergence
We consider the reverse KL divergence and start with the general SMORe objective:
max min 3(1 7).... [S(s, a, 9.9)]= (ww.g-/tss//pplaaaalg P> S(s, - S(s, a. g))l
59 S
3) -...) S ((,, a(s,a,9) (64)
Plugging in the conjugate J for reverse KI. divergence we get:
max min 3(1 -7 (S(s, a, g)1 -,,,,, + [ Mas.gggiiixggggaasssaa. ( ( (
(1 3) /e S(s a. S(s, a)g (65)
Using the telescoping sum for the last term in the objective above, can simplify it follows:
max min B(1 -))... [S(s, (5) a, E. Ma..gg-iiaall (t
Fo
+ (1 de g) (66)
With the initial state distribution dg set to the offline dataset distribution p. and Since our initial state distribution is the same offline data distribution, we get:
max min 3(1 - 7) E,,, [S(s, + E. Bes. ( ((om) F,
+ +1- (1-58)E[[5(s,a,))) (67)
Collecting terms together we get:
max min [,, - S)S a.g) + ())e(((1))((,,)))) a, g)g)
Z9
(68)
The objective for SMORe with reverse KL divergence pushes down the "score" of offline dataset transitions selectively (without pushing down score of the goal-transition distribution) while minimiz- ing the term resembling bellman regularization that also encourages increasing score the mixture dataset jointly over the offline dataset as well as the goal transition distribution.
b. Pearson chi-squared divergence
We consider the Pearson x² and start with the general SMORe objective:
max min ((1 7) E,,,,,, [S(s, a, g)1 E. # (aPSsss S(s, a, g))
Fe
(1 -) S(s 3(s,a,9) (69)
With the initial state distribution do set to the offline dataset distribution p. and plugging in the conjugate f* for Pearson x2 divergence we get:
max min Bd, [S(s, g)) aar. p (") S(s,,, 2g S +E
-1 - S(s, a.s. (70)
20

---

Published as a conference paper at ICLR 2024
Using the fact that Mix Mix.(q. Bq(s, + (10 we can further simplify the above equation to:
max min B(1-7) ())) htw.::/00............aaaaaaaaaa. a(s,a,9) - S(s, a. =)
f (asagg-l(11PP..(sssss S(s,a, (71)
Collecting terms together we get:
PT (y max min B(1 -) ( [5(s,a,g)] + gagga 11
BE - g)) s. (0.0)(.. [(> (ts
S(s, ()))))
(72)
Observing the equation above, we note that the first two terms decrease score at offline data distribution as well as the goal transition distribution when actions are sampled according to the policy mg- Simultaneously the third term pushes score up for the {s, a, g) tuples that are sampled from goal transition distribution. Finally the last term encouraged enforces a bellman regularization enforcing smoothness is the scores of neighbouring states.
C SMORE EXPERIMENTAL DETAILS
C.I TASKS WITH OBSERVATIONS AS STATES
Environments: For the offline GCRL experiments we consider the benchmark used in prior work GoFar and extend it with locomotion tasks. For the manipulations tasks we consider the Fetch environment and a dextrous shadow hand environment. Fetch environments (Plappert et al., 2018) consists of manipulator with seven degrees of freedom along with parallel gripper. The set of environments get a sparse reward of 1 when the goal is within cm and 0 otherwise. The action space 4 dimensional (3 dimension cartesian control + dimension gripper control). The shadow hand is 24 DOF manipulator with 20-dimensional action space. The goal is 15-dimension specifying the position for each of the five fingers. The tolerance for goal reaching is cm. For the locomotion environments, the task is to achieve particular velocity in the x direction and stay at the velocity. For HalfCheetah, the target velocity set to 1 11.0 and for Ant the target velocity is 5.0. For locomotion environments, the tolerance for goal reaching if 0.5. The MuloCo environments used in this work are licensed under CC BY 4.0.
We use existing datasets from the offline GCRL benchmark used in (Ma et al., 2022) for all manipulation tasks except Reacher, SawyerReach, and SawyerDoor. For Reacher, SawyerReach, and SawyerDoor comprised on x% random data and (100-x)% expert data depending on the coverage over goals reached in individual datasets. We create our own datasets for locomotion by using trandom/medium/neddum replay' data as our offline (suboptimal) data combined with 30 trajectories from corresponding 'expert' datasets. The datasets used from D4RL licensed under Apache 2.0.
Offline Datasets:
existing datasets from (Yang et al., 2022). These datasets are
Baselines: To benchmark and analyze the performance of our proposed methods for offline imitation learning with suboptimal data, we consider the following representative baselines in this work: GoFAR (Ma et al., 2022), WGCSL (Yang et al., 2022), GCSL (Ghosh et al., 2019), and Actionable Models (Chebotar al., 2021), Contrastive RL (Eysenbach et al., 2020) and GC-IQL Kostrikov et al. (2021). GoFAR is dual occupancy matching approach to GCRL that formulates it as a weighted regression problem. WGCSL and GSCL use goal-conditioned behavior cloning with goal relabelling as the base algorithms and WGCL uses weights to learn improved policy over GCSL. Actionable models uses conservative learning with goal chaining to learn geal-rea behaviours using offline datasets. Contrastive RL treats GCRL classification problem contrastive goals that achieved in trajectory from random goals. Finally, GC-IQL extends the single task offline RL algorithm IQL to GCRL.
The open-source implementations of the baselines GoFAR, WGCSL, GCSL, Actionable models, Contrastive RL and IQL are provided by the authors (Ma et al., 2022) and employed in our experiments. We use the hyperparameters provided by the authors, which are consistent with those used in the original GoFAR paper, for all the MuloCo locomotion and manipulation environments. We implement
21

---

Published as a conference paper at ICLR 2024
contrastive learning using the code from Contrastive RL repository. GC-IQL is implemented using code from author's s implementation found here.
Architecture and ayperparameters For the baselines, works that were tuned on the same set of tasks and datasets. Implementation for SMORe shares the same network architecture as baselines. GoFAR additionally requires training discriminator. For all experiments, all methods are trained for 10 seeds with each training run. Fetch manipulation (except Push) tasks are trained for maximum of 400k minibatch updates of size 512 whereas all other environments training is done for 1M minibatch updates. The expectile parameter was searched over [0.65 0.7,0.8,0.851 For the results shown in table 1, Fetch and Sawyer environments use = 0.8, Locomotion and Adroit hand environments use T = 0.7. In general, the HER ratio is searched over (0.2,0.5.0 0.0.88888. 1.0) for all methods and the best one was selected. HER ratio of 0.8 gave best performance across all tasks for SMORe.
tuned hyperparameters from previous
The architectures and hyperparameters for all methods are reported in Table 4.
Hyperparameter
Value
Policy updates pol
1
Policy learning rate
3e-4
Value learning rate
3e-4
MLP layers
(256,256)
LR decay schedule
cosine
Discount factor
0.99
LR decay schedule
cosine
Batch Size
512
Mixture ratio 3
0.5
Policy temperature (a)
3.0
Table 4: Hyperparameters for SMORe.
C.2 TASKS WITH OBSERVATIONS AS IMAGES
Hyperpuramelers
Values
hatch size
2048
number of training epochs
300
number of training iterations per epoch
1000
Horizon
image encoder architecture
3-layer CNN
policy network architecture
(1024, 4)
critic/scrr network architecture
(1024, 4) ML.P
weight initialization for final layers of critic and policy
UNIF 10 10-12]
policy std deviation
0.15
representation dimension
data augmentation
random cropping
discount factor
0.99
leamning
3e-4
Table 5: Hyperparameters for image-observation GCRL from Zheng et al. (2023).
Tasks and dataset Our experiments use a suite of simulated goal-conditioned control tasks based on prior work Zheng et al. (2023). The observations and goals are 48 x 48 x 3 RGB images. The evaluation tasks require multi tualii-trajectoryy stitching whereas the dataset contains trajectories solving only parts of the evaluation tasks.
22

---

Published as a conference paper at ICLR 2024
Task
Behavior cloning
Contrastive RL
RL+sparse reward
WGCSL
GCSL
CRL
AM
IQL
Reacher
15.30 t:o.s
14.01 10.3
16.62 2:009
23.68
8.86
SawyerReach
14.06
12.05
23.03
23.37
36.19
SawyerDoor
16.79
18.29 1..44
12.26 13344
16.63 10.76
29.31
0.88
FetchPick
6.87:t 0.77
6.54 + 1.85
0.21 a29
0.45 :: 0.32
15,24
1.27
FetchPush
10.62::: 0.8
12.38
= 1.10
3.60 + 0.59
2.74
+ 0.7
19.95
+: 1.9
FetchSlide
2.62.1 1.15
2.03
1. 0.00
0.411 0.03
0.31
0.31
3.25
1.02
Table Discounted Return for the offline GCRL henchmark with 5% expert data. Results averaged 10 seeds.
Task
Behavior cloning WGCSL
Contrastive RL
Rp+sparse reward AM
GCSL.
CRL
IQL
Reacher
13.03
12.17 1os
19.63 1309
24.78 10.2
4.44
0,70
SawyerReach
11.455
11.34
25.35 108
25.19
35.73
0.22
SawyerDoor
16.79+0.9
13.20:t0::
14.78
16.59
16.87
FetchPick
4.39
4.99 +
0.21
0.24
11.79
FetchPush
8.01
8.04
3.60
2.02
19.66
FetchSlide
2.33
2.37 1:
0.44
0.45 ::
1.83
Table 7: Discounted Return for the offline GCRL benchmark with 2.5% expert data. Results are averaged over 10 seeds.
In the simulation, we employed an offline manipulation dataset comprising near-optimal examples of basic action sequences, including the initiation of drawer movement, the displacement of blocks, and the grasping of items. The demonstrations exhibit variations in length, ranging between to 100 horizon, while the offline dataset contains total of 250,000 state transitions in its entirety. It is important to note that the offline trajectories do not depict a complete progression from the initial condition the final objective. For the purposes of evaluation, et al. (2023), against which we compare the success rates in realizing these specified objectives.
consider 4 tasks similar to Zheng
Baseline and SMORe implementations. We use the open-source implementation of Stable- contrastive RL the same design decisions and implement SMORe, GC-IQL that code- base. We use the same hyperparameters as the stable-contrastive RI. implementation for the shared hyperparameters. The hyperparameters for SMORe were kept the same in Table 4.
D ADDITIONAL EXPERIMENTS
D.1 RESULTS ON OFFLINE GCRL BENCHMARK WITH VARYING EXPERT COVERAGE IN OFFLINE DATASET
We ablate the effect of dataset quality the performance of an offline GCRL method in this sections.
Table 6, 7, 8 show performance of all methods with 5%, 2.5% and 1% expert data in the offline dataset respectively.
D.2 SUCCESS RATE AND FINAL DISTANCE GOAL MANIPULATION TASKS
Table 10 and Table 11 reports the success rate and final distance goal metrics manipulation tasks.
D.3 ROBUSTNESS OF MIXTURE DISTRIBUTION PARAMETER
We find that SMORe is quite robust to the mixture distribution parameter 3 except in the environment FetchPush where 0.5 is the most performant. Table 9 shows this result empirically.
23

---

Published as a conference paper at ICLR 2024
Task
Behavior cloning
Contrastive RL
RL+sparse reward
WGCSL
GCSL
CRL
AM
IQL
Reacher
13.56t000
12.27
17.94
34.8
4.28
SawyerReach
10.71 1o6
11.79.t1.46
25.61
25.54
31.31
SawyerDuor
15.18 toa
11.89
10.26#461
18.04#18
17.11
FetchPick
1.89
3.30
0.42
0.41 +
7.90
FetchPush
6.44
6,43
1.69
2.63+:
7.11
FetchSlide
1.77
1.11
0.0
0.10 +
0.80
Table 8: Discounted Return for the offline GCRL benchmark with 1% expert data. Results are averaged over 10 sccds.
D.4 HOW MUCH DOES HER CONTRIBUTE TO THE PERFORMANCE IMPROVEMENTS OF SMORE?
GoFAR demonstrated improved performance without relying on HER. The authors also demonstrated that HER is detrimental GoFAR's performance. In this section, we aim to conduct similar stufy and see how much HER contributed to SMORe's performance. Table 12 shows that HER gives SMORe small performance boost and show that SMORe still able to outperform GoFAR without HER.
D.5 COMPARISON TO VARIANTS OF GOFAR
GoFAR formulates GCRL as an occupancy matching problem, but it is also suggested that using a discriminator optional. Without a discriminator, GoFAR reduces sparse reward RL problem. Table 13 shows that GoFAR achieves poor performance when a reward function is substituted in place on discriminator. We also study the performance benefits we obtain are due the offline learning strategy we used from IQL. We modify GoFAR with discriminator reward to use expectile loss for value learning and AWR for policy learning. Results in Table 13 shows that no performance gains were observed.
D.6 OFFLINE GCRL WITH PURELY SUBOPTIMAL DATA
In this experiment, we study offline GCRL from purely suboptimal datasets. Except FetchReach, these datasets provide very sparse coverage of goals expected to reach in evaluation. Table 14 shows the robustness of SMORe even in the setting of poor quality offline data.
D.7 COMPARISON WITH IN- SAMPLE LEARNING METHODS
In-sample learning methods perform value improvement using bellman backups without OOD action sampling. This makes them particularly suitable candidate for offline setting. We compare against a number of recent in in-sample learning methods, IQL (Kostrikov et al., 2021), SQL/EQL Xu et al. (2023) and XQL (Garg al., 2023). Table 15 compares SMORe to in in-sample learning methods adapted to GCRL.
D.8 ABLATING COMPONENTS OF SMORF FOR OFFLINE SETTING
In offline setting, it is well known that bellman backups suffer from overestimation and results in poor policy performance. We validate the utility of the components used in this work in Table 16: expectile loss function and constrained policy optimization with AWR.
24

---

Published as a conference paper at ICLR 2024
Task
3 0.5
0.7
| 0.8
8 0.9
FetchReach
35.08 0.44
36.57 = 0.20
36.59: 0.30
36.30 0.30
FetchPick
26.47 0.34
27.04 081
27.43: 097
27.89
1: 1.19
PetchPush
26.83 = 121
16.20:1 111
11.501 119
13.85 535
FetchSlide
4.991
3.76
3,43
4.10
Table 9: Discounted Return for the offline GCRL benchmark with varying mixture coefficients in offline dataset. Results are averaged over 10 seeds.
Task
Occupancy Matching SMORe
Behavior cloning
Contrastive RL
RL+sparse reward
GoFAR
WGCSL
GCSL
CRL
AM
IQL
Reacher
tu07
0.90+am
0.97
0.92 20,0
0.76
1.0ta1
0.26 +
RawyerReach
0,98 0.000
0.75:1a0s
1.0
0.98
0.98
1.0
0.81 1: 0.01
SooyerDoor
0.875 0.000
0.5
U.78 20.0
0.5 12
0.22
0,3-to.11
0.84 + 0.06
FetchReach
1.0+ ou
1.0:00 0.0
1,0+ 0.
0.98 + 0.8
1.0 au
1,0 1.0
1.0 = uo
FetchPick
0.925 0.045
0.84 0.9
0.54 0.16
0.54 1: 0.20
1:
FetchPush
0,90
0.88 0.9
0.76
0.72 4 0.15
0.06:
0.67
0,65
FetchSlide
0.315
0.18
0.18 0.14
0.17::0000
0.0 ::
0.11
0.26 0057
HanddRach
0.47
0.40 1. 0.20
0.25 023
0,047.L 0.10
0,0. 00
0.0 d 0.0
0.0 d 00
Table Success Rate for offline GCRL benchmark with 10% expert data. Results are averaged over 10 seeds.
Task
Occupancy Matching SMORe
Behavior cloning
Contrastive RI.
Rl-ssparse reward
GoFAR
WGCSL
GCSL
CRL
AM
IQL
Reacher
0.02
0.03
0.011
0.016 1000
10.0
0.013:am
0,12: 3005
SawyerReach
0.008 0.008110.04 1o.00
0.04
0.004:1000
0.00
0.01 1om
0.01 1:00
0.053 1 0004
SawyerDoor
0.02
0.18
0.011:t0m
0.0172001
0.14:o0.7
0.06
0.019 +
FetchReach
0,004+ 00012
0.018 0.0o
0.007 00043
0.008 + 000
0.007 +: 0000
0,007 + 0.01
0.002-t 000
FetchPick
0,04= 0018
0.036
0013
0.094-: 0.43
0.108:1 0.06
0.25 auzz
0.04+ a.2
0.04+ 0012
FetchPush
0.033-2
0.041
0.0424:
0.15+
0.07=0.009
0.05+
FetchSlide
0.09= 0,12
0.12 0.02
0.173 0.04
0.204:t 0.51
0.42 0,05
0.198 0.299
0.09 2013
HandReach
0.024 0.009
0012
0.038 o015
0.08
0.037 +0.004
+
Table 11: Final distance to goal for the offline GCRL. benchmark with 10% expert data. Results are averaged
over 10 seeds.
Task
SMORe
SMORe w/o HER
GoFAR
FetchReach
35.08 + 0.54
34.86 1.
28.2+t061
SawyerReach
37.674 0.12
3 0.36
15.34-10.64
SawyerDoor
31.48+ 0.46
31.53+ 062
18.94-tu.11
FetchPick
26.474
25.72
19.7
FetchPush
26.83
25.62
18.2
FetchSlide
4.99- 0.40
4.09 0.33
2.47
Table 12: Performance gains using HER (resampling ratio=0.8) SMORe. All other hyperparameters are kept the same between SMORe and SMORe w/o HER.
Task
SMORe
GeFAR (discriminator)
GoFAR (spansc roward)
r-GFAR reward)
GoFAR (expectile loss+AWR)
FetchReach
26.90:n-1
SawyerReach
37.67 a11
15.34
0.34=axs
16.17::3v
SawyerDoor
31,48 a.
18 18.94-205
10, 36.26
22:47::115
PetchPick
26.474.2
19.7
17.4
0.25
18.461
FetchPush
18.2
17.4
17.39
FenchSlide
4.99 0.0
2,47214
5.13
0.29
3.50g130
Table 13: Ablation comparison with GoFAR: a) sparse binary reward is used in place of a learned discriminator in GoFAR b) The policy and value update is replaced by AWR and expectile loss respectively. The main difference with SMORe remains the of discriminator for training. indicates these environments were not considered in (Ma al., 2022). Our reproduced results (denoted by r-) with the official code base binary results not match up to the reported results.
25

---

Published as a conference paper at ICLR 2024
Tisk
SMORe
GoFAR
WGCSL
GC-IQI
FetchReach
35.01
27.37
21.65:06::
23.72.11.18
SawyerReuch
36.264:
5.89-2136
7.27
33.08
SawyerDoor
20.28.1: 2.65
13.33
13.81
16.05 14.9
FetchPick
0.61
0.0220
1.31
FetchPush
6.39.t 0.88
4.23
4.27133
2.63:1
FetchSlide
0.42=00.1
0.059:10.0
0.93
0.75:ta.s
Table 14: Discounted Return for the offiine GCRL benchmark with 0% expert data. Results averaged over 10 seeds.
Task
SMORe
GC-IQL
GC-SQL
GC-BQL
GC-XQL
FetchReach
35.08 t 0.54
34.43t 1.00
35.67
29.23
33.94
SawyerReach
37.67 0.12
35.18 020
37.10#124
10.
32.88+225
Sooy
31.481 0.6
25.524
27.96
3.57
5.85-421
FetchPick
26.47-1 0:34
16.8:t
18.35
1.31
1.31:1.0
FetchPush
26.83 + 1.21
22.40:b 0:74
17.19 1:256
2.64-1130
3.79
FetchStide
4.99 0.40
4.80
1.59
4.68 3:332
0.06
0.36:10.52
Table 15: Comparison of SMORe in-sample RL methods IQL (Kostrikov al., 202 (Xu al.,
2023), XQL (Garg et al., 2023) that learn from sparse rewards. in EQL denotes the learning diverged. observed IQL to be the most stable alternative compared to SQL, EQL and XQL. SQL, EQL and XQL were implemented using author's ss official codebase
Task
SMORe
SMORe (w/o AWR)
SMORe (w/ AWR and Expectile loss)
FetchReuch
35.08 : 05a
0.30+bo.9
0.10-b0.1
SawyerReach
36.26 1: 0.9
2953
SawyerDoor
5.06
2.119
FetchPick
26.47#
1,79-o04
1.77#151
FetchPush
26.83 +: 1.21
4.60-11.1
2.69:t1M
FetchSlide
4.99tt
0.22:t033
0.50
Table 16: Ablating practical components of SMORe. Without adapting for offline setting we consider in this work by using in sample maximization constrained policy optimization using AWR the performance degrades expected. Without wsaappl-maxiiieee value function explodes in the offline setting and using policy that maximizes Q can often select OOD actions leading poor performance.
26


Reviews:
[{"review_id": "nC71faUsvU", "reviewer": "Reviewer_Wf6E", "summary": "The paper introduces a novel approach for offline goal-conditioned reinforcement learning, known as SMORe, which is derived from a mixture-distribution matching perspective and eliminates the need for learning a discriminator. The paper demonstrates that SMORe outperforms state-of-the-art baselines in various robot manipulation and locomotion tasks, including high-dimensional observations.", "questions": "Why do the WGCSL and GCSL methods have similar performance in Table 7-10, while show a large difference in Table 1 for the following four environments: CheetahTgtVel-m-e, CheetahTgtVel-r-e, AntTgtVel-m-e and AntTgtVel-r-e? What factors could explain this discrepancy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and clearly motivates the problem of offline GCRL. It uses convex duality theory to derive a dual optimization problem that can use offline data to learn score functions and policies, which provides a rigorous theoretical analysis of the proposed method. It also presents extensive empirical results that demonstrate the effectiveness and robustness of SMORe on challenging benchmarks.", "weaknesses": "One weakness of the paper is that the claim of being discriminator-free is somewhat overclaiming. From Eq. 12, the S-function can be seen as a Q-function and the M-function can be seen as a V-function. In this case, although the framework does not have an explicit discriminator, the S-function actually plays the role of a discriminator. What's more, the proposed method requires two networks, S and M, to learn the optimal policy, while many works only require a network (such as contrastive RL). This implies that the proposed method has more parameters and computational complexity.\nAnother weakness is that it is unclear how much of the performance gain comes from each component. A possible suggestion is to add an experiment without using expectile regression and AWR, as well as experiments that show how they work separately. This would clarify the role of each part in the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMuQJ57V33", "reviewer": "Reviewer_V1Bw", "summary": "This paper proposes a method for offline goal-conditioned RL, integrating occupancy matching with a convex dual formulation so that the learning objective is converted for better leveraging on suboptimal offline data. Instead of behavior cloning, contrastive RL, or RL with sparse reward, the proposed method is built upon the direction of occupancy matching but without learning an additional discriminator. The proposed method is supported by theorems and evaluated with comprehensive simulation experiments.", "questions": "* Is the \"0.25\" in equation 9 a fixed number or a kind of coefficient that can be tuned? \n\n* Any reason why you make the mixture of random/medium and expert data following 4.1 EXPERIMENTAL SETUP?  I thought that there is already a pipeline for collecting random, medium, and expert sub-dataset?\n\n* Can you explain why SMORe has an even higher discounted return in Figure 2 under 0.5 noise level compared with 0 noise? In addition, I am interested in the variance of Figure 2.\n\n* Is there any insight or analysis into why pick and place tasks in Figure 3 are relatively difficult for other baselines compared with the remaining tasks while the proposed method can have a significant improvement in pick and place tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper is well-written, and the concepts are clearly and concisely explained\n* The authors provide theoretical contributions.\n* The paper's contributions are supported by empirical analyses on a range of benchmarks, demonstrating the advantage of using SMORe for suboptimal offline data, especially the evaluation of robustness and high-dimensional observation space.", "weaknesses": "* Could you please elaborate more about the technical differences between SMORe and GoFAR? My understanding is that the main difference is whether the training involved a discriminator. Does any other difference in details improve the novelty of SMORe?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PtAfWzue1X", "reviewer": "Reviewer_ojhF", "summary": "This paper proposes an offline GCRL algorithm from the occupancy matching perspective. The problem setting basically follows GoFAR [1], using a similar DICE-based construction. The difference is that GoFAR is formulated as a V-DICE (learn V and perform goal-conditioned state occupancy matching) and does not use HER; while this paper adopts Q-DICE (learn both Q and $\\pi$ by solving a max-min problem, and performs goal-conditioned state-action occupancy matching) and uses HER to generate desirable goal-reaching data ($q(s,a,g)$). The tricky part is that the final practical algorithm is essentially a goal-conditioned version of in-sample learning algorithm which bears lots of similarities with methods like IQL[2], SQL/EQL[3], and XQL[4]. Although such in-sample learning algorithms like SQL/EQL and XQL have been show to have some connection with DICE-based methods, there are some important distinctions. There are some noticeable theoretical gaps here. The proposed method has a number of strange design choices and the algorithm development is not very principled in several places. See the following strengths and weaknesses for detailed comments.", "questions": "- Please report the hyperparameter $\\alpha$ values in your experiments.\n- How will the proposed method perform if not tuned individually for each task? Such as using 1~3 sets of hyperparameters. \n- All datasets in the experiments are mixed with some expert data. How will the proposed method perform if only uses sub-optimal data samples?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed practical algorithm provides a reasonable approach to combine goal-conditioning and in-sample learning offline RL.\n- The performance is good in a number of GCRL tasks. The experiments are comprehensive.\n- Provide experiments on vision-based tasks.", "weaknesses": "There are several key weaknesses in the paper.\n- The problem setting and Section 3.1 largely follow GoFAR[1] with minor changes. A key motivation from the authors is that methods like GOFAR need an unstable discriminator-based construction. However, this does not really hold. As in the GoFAR paper[1], their authors clearly mentioned in the paper that the discriminator can be bypassed by using the reward in the dataset.\n- Although using HER to generate augmented goal-transition samples could potentially improve distribution coverage and may contribute to certain level performance improvement. However, as discussed in GoFAR, using HER could also lead to sensitive hyperparameter tuning and suffer from hindsight bias.\n- The biggest problem of this paper is the gap between theoretical derivation and the practical algorithm. Based on the augmented samples from HER, the proposed method constructs a goal-conditioned Q-DICE objective which needs to solve a max-$\\pi$ and min-$S$ (analogous to Q function in typical Q-DICE algorithm like AlgaeDICE[5]). This actually caused some stability issues due to extracting $\\pi$ through the max-min optimization problem. Hence the practical algorithm directly jumps to an in-sample learning framework which is similar to IQL[2], SQL/EQL[3], and XQL[4]. Although there are some connections between the DICE-based method and previous in-sample learning methods, there are also some distinctions. An apparent difference is that the DICE-based method requires minimizing $S$ (analogous to Q in other Q-DICE methods) in the first term of Eq.(9) using samples from initial states $d_0$, while in in-sample learning algorithms, this can be sampled from the whole dataset $\\mathcal{D}$ (Eq.(10)). Second, DICE-based method only learn Q (similar to S in this paper) or V, while the previous in-sample learning algorithms learn both Q and V. In my opinion, the paper starts with a DICE formulation and goes a long way to turn it into a non-DICE algorithm. If the authors check the SQL/EQL[3] paper, a more straightforward approach will be starting with its implicit value regularization framework and designing a proper, goal-conditioned regularization function $f$, which will provide a neat and more principled algorithm.\n- The proposed algorithm has many hyperparameters, e.g. $\\tau$ in Eq.(10), $\\beta$ in Eq.(11), and $\\alpha$ in Eq.(12). The paper conducts heavy tuning to obtain the best performance. First of all, for an offline RL algorithm, introducing too many hyperparameters and requiring heavy parameter tuning is an extremely bad practice. In practical offline RL applications, it is almost impossible to evaluate or tune model parameters given restricted access to the real environment. In practice, no one will use an offline RL algorithm if it needs careful hyperparameter tuning to achieve good performance.\n\n\n**References:**\n\n[1] Ma, J. Y., et al. Offline goal-conditioned reinforcement learning via $ f $-advantage regression. NeurIPS 2022.\n\n[2] Kostrikov I, Nair A, Levine S. Offline Reinforcement Learning with Implicit Q-Learning ICLR 2022.\n\n[3] Xu, H., et al. Offline RL with no OOD actions: In-sample learning via implicit value regularization. ICLR 2023.\n\n[4] Garg, D., Hejna, J., Geist, M., & Ermon, S. Extreme Q-Learning: MaxEnt RL without Entropy. ICLR 2023.\n\n[5] Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., & Schuurmans, D. Algaedice: Policy gradient from arbitrary experience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kiMTqmIRtD", "reviewer": "Reviewer_9ANq", "summary": "This paper formulates the offline goal conditioned reinforcement leanring (GCRL) problem as an occupancy matching problem and leverages the popular DICE methods to transform this problem into an objective that can be exclusively trained on offline dataset. Specifically, they adopts ValueDICE[1]-like method for this reformulation. The authors argue that by doing so, the offline GCRL problem can be solved without necessitating a well-trained discriminator. Instead, it learns unnormalized densities or scores that allow it to produce optimal goal-reaching policies, greatly enhancing the training results. In addition, this paper does not directly optimize the objectives derived from the DICE reformulation (Eq. 9), but adopts IQL[2] to optimize some surrogate objectives (Eq. 10-12). From my perspective, this approach mitigates the side-effect of residual learning in Eq. 9 since the surrogate objective in Eq. 11 is optimized via semi-gradient, which is believed to enjoy better performances than residual learning. This paper conducts some experiments on low-dimensional and image-based tasks to support the effectiveness of the proposed method, and also carries out some ablation stuidies on stochastic environments with varied noise levels.\n\n[1] Imitation Learning via Off-Policy Distribution Matching, ICLR 2020\n\n[2] Offline Reinforcement Learning with Implicit Q-Learning, ICLR 2022", "questions": "Please refer to weakness for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Despite GoFAR[3] already employing DICE methods to address offline GCRL problem, this paper is the first to eliminate the need for an additional discriminator.\n2. The experimental results are promising, outperforming baselines (especially the most related GoFAR[3]) across a wide range of evaluation settings.\n\n[3] How far i'll go: offline goal-conditioned reinforcement learning via f-advantage regression, NeurIPS 2022.", "weaknesses": "## Problem formulation\n1. The target goal-transition distribution $q(s,a,g)$ defined in Section 3.1 might not be a valid discounted visitation distribution that fulfills the Bellman-flow constraint. Therefore, this suggests that the  occupancy matching problem in Eq. 4 may not be appropriately formulated.\n\n## Experiment\n2. This paper optimizes surrogate objectives using IQL[2] rather than directly solving the objective derived from DICE reformulation, but does not provide explainations for this choice. Moreover, this paper does not conduct enough ablation studies on this aspect. It raises a question about the performance of GoFAR using this same technique. In my view, optimizing the surrogate objectives in Eq. 10-12 can mitigate the side-effect of residual learning in Eq. 9. Therefore, GoFAR may also obtain large performance gains using the IQL tricks.\n3. Given that GoFAR does not employ IQL surrogate objectives, I cannot ensure the comparison in Figure 2 is fair. It would be more fair if the authors could also apply this technique to GoFAR and compare it against GoFAR using this approach.\n\n## Others\n4. Some potential overclaims. Section 3.1 is essentially an extension of the conclusion of GoFAR[3] from state-occupancy matching to state-action-occupancy matching. The authors should provide more discussions on the relationships with GoFAR. Although they have made statements like \"Proposition 1 extends the insights of formulating GCRL as an imitation learning problem from Ma et al. for goal-transition distributions when matching state-action-goal visitations\", this similarity should be made clearer.\n5. The Problem Formulation section (Section 2) is largely similar to the one in GoFAR[3] paper, with only minor rewording. In my view, this section should be reorganized or rewording a lot to avoid potential plagiarism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)