Offline Goal-Conditioned Reinforcement Learning via $f$-Advantage Regression
Authors: Yecheng Jason Ma
Jason Yan
Dinesh Jayaraman
Osbert Bastani
Venue: NeurIPS 2022 Accept (2022)
Keywords: offline reinforcement learning
goal-conditioned reinforcement learning
deep reinforcement learning

Abstract: Offline goal-conditioned reinforcement learning (GCRL) promises general-purpose skill learning in the form of reaching diverse goals from purely offline datasets. We propose $\textbf{Go}$al-conditioned $f$-$\textbf{A}$dvantage $\textbf{R}$egression (GoFAR), a novel regression-based offline GCRL algorithm derived from a state-occupancy matching perspective; the key intuition is that the goal-reaching task can be formulated as a state-occupancy matching problem between a dynamics-abiding imitator agent and an expert agent that directly teleports to the goal. In contrast to prior approaches, GoFAR does not require any hindsight relabeling and enjoys uninterleaved optimization for its value and policy networks. These distinct features confer GoFAR with much better offline performance and stability as well as statistical performance guarantee that is unattainable for prior methods. Furthermore, we demonstrate that GoFAR's training objectives can be re-purposed to learn an agent-independent goal-conditioned planner from purely offline source-domain data, which enables zero-shot transfer to new target domains. Through extensive experiments, we validate GoFAR's effectiveness in various problem settings and tasks, significantly outperforming prior state-of-art. Notably, on a real robotic dexterous manipulation task, while no other method makes meaningful progress, GoFAR acquires complex manipulation behavior that successfully accomplishes diverse goals.

(GCRL) aims to learn repertoire of skills in the form of reaching distinct goals. Offfine GCRL O 47 is particularly promising because it enables learning general goal-reaching policies from purely offline interaction datasets without any 25,, which can be expensive in the real-world. As offline datasets contain diverse goals and become increasingly prevalent 9 19 6 policies learned this way can acquire a large set of useful primitives for downstream tasks 30) A central challenge in GCRL is the sparsity reward signal 3]; without any additional knowledge about the environment, an agent at state typically only accrues positive binary reward when the state lies within the goal neighborhood. This sparse reward problem exacerbated in the offline setting, which the agent cannot explore the environment to discover more informative states about desired goals. Therefore, designing and Offline Goal-Conditioned Reinforcement Learning via f-Advantapp Regression
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, Osbert Bastani
University of Pennsylvania
{jasonyma, jasyan, dineshj, obastami) udu
https:/ (ttps://jasonma2000 github Eetthub.io/GoFie 1o/GGFAR/
Abstract
Offline goal-conditioned reinforcement learning (GCRL) promises general-purpont skill learning the form reaching diverse goals from purely offline datasets. We propose Goal-conditioned f-Advantage Regression (GoFAR), novel regression- based offline GCRL algorithm derived from state-occupancy matching perspec- tive; the key intuition is that the goal-reaching task can be formulated as state- occupancy matching problem between dynamics-abiding imitator agent and an expert agent that directly teleports to the goal. In contrast to prior approaches, GoFAR does not require any hindsight relabeling and enjoys uninterleaved opti- mization for its value and policy networks. These distinct features confer GoFAR with much better offline performance and stability as well as statistical performance guarantee that is unattainable for prior methods. Furthermore, that GoFAR's training objectives can be re-purposed learn an agent- ieeen-independentt goal-conditioned planner from purely offline source-domain data, which enables zero-shot transfer to new target domains. Through extensive experiments, we validate GoFAR's effectiveness in various problem settings and tasks, significantly outperforming prior state-of-art. Notably, on a real robotic dexterous manipulation task, while no other method makes meaningful progress, GoFAR acquires complex manipulation behavior that successfully accomplishes diverse goals.
demonstrate
Introduction
Goal-conditioned reinforcement learning
18
43
39
environment interaction [28
effective offline GCRL algorithm a concrete yet challenging path towards scalable policy learning.
In this paper, we present a novel offline GCRL algorithm, Coal-conditioned f-Advantage Regression (GoFAR), first casting GCRL as a state- occupeccy matching [27] 32) problem and then deriving regression-bess based policy objective. In particular, GoFAR begins with the following goal-conditioned state-matching objective:
min DRL. (d (a: (p((s))
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Inputs
Goal-Conditioned f- Regression (GoFAR)
Output
Offline data do
Train (Dual) Value Function
Learned
behavior xt'
minn1 y) g)l + (.agg-e
2. Train Policy via Advantage Regression
min = "
Figure 1: GoFAR schematic illustration.
where d' (s:g) is the conliionn -cete-ocy distribution of policy т and p(s;g) is the distribution of states that satisfy a particular goal g. This objective casts goal-conditioned offline RL as an imitation learning problem: a dynamics-abiding agent imitates as well as possible an expert who can teleport to the goal in one step; see Figure 2 Posing GCRL this way is mathematically principled, as we show that this objective is equivalent to probabilistic interpretation of GCRL that additionally encourages maximizing state entropy. More importantly, this objectives enables us to extend a state-of-art offline imitation learning algorithm [32 to the goal-conditioned setting and admits elegant optimization using purely offline data by considering an f-divergence regularized objective and exploiting ideas from convex duality theory [4] 4. In particular, we obtain the dual optimal value function V* from single unconstrained optimization problem, using which we construct the optimal advantage function (we refer to this f-advantage) that serves importance weighting for a regression based policy training objective; see Figure for a schematic illustration.
There are several distinct features to our ap- proach. First and foremost is GoFAR's s lack of goal relabeling. Hindsight goal relabeling [3] (known as HER in the literature) relabels tra-
Dynamics-abiding imitator
jectory goals to be states that were actually achieved instead of the originally commanded goals. This heuristic is critical for alleviating the sparse-reward problem in prior GCRL methods, but is unnecessary for GoFAR. GoFAR updates its policy as if all data is already coming from the optimal goal-conditioned policy and therefore does not need to perform any hindsight goal relabeling. GoFAR's 's relabeling-fee training is of significant practical benefits. First, it enables more stable and simpler training by avoiding sensitive hyperparameter tuning associated with HER that cannot be easily performed offline (48] Second, hindsight relabeling suffers from hindsight bias in stochastic environments 44 as achieved goals may be accomplished due to noise; this is exacerbated in the offline setting due to inability collect more data. By bypassing hindsight relabeling, GoFAR promises to be more robust and scalable.
"Teleporting" expert
Figure 2: GCRL can be thought of imitating an expert agent that can teleport to goals.
Second, as suggested in Figure | GoFAR's value and policy training steps completely uninter- leaved: we do not need to update the policy until the value function has converged. This confers greater offline training stability [23 32 compared to prior works, which mostly involve alternating updates to a critic Q-network and policy network. Furthermore, it enables an algorithmic reduction of GoFAR to weighted regression [7], which allows us to obtain strong finite- sample statistical guarantee on GoFAR's 's performance; prior regression- based GCRL approaches 14] 47 this reduction and obtain much weaker theoretical guarantees.
do not enjoy
The key insight lies in observing that, under mild assumption, GoFAR's 's dual value function objective does not depend on the action and can learn from state-only offline data. This enables learning a goal-centric value function and thereby near-optimal coal-conditioned planner that is capable of zero-shot transferring to new domains the same task that shares the goal space. In our experiments, we illustrate that GoFAR planner can indeed plan effective subgoals in new target domain and enable a low-level controller to reach distant goals that it is not designed for.
Finally, we show that GoFAR can also used to learn a goal-conditioned planner 36 5
We extensively evaluate GoFAR on a variety of offline GCRL enviromments of varying task complexity and dataset composition, and show that it outperforms baselines in settings. Notably, GoFAR is more robust to stochastic environments than methods that depend on hindsight relabeling. We additionally demonstrate that GoFAR learns complex manipulation behavior on a real-world high- dimensional dexterous manipulation task [2) for which most basclines fail to make any progress.

---

Finally, we showcase GoFAR's planning capability in cross-robot zero-shot transfer task. To summarize, our contributions are:
1. GoFAR, a novel offline GCRI. algorithm derived from goal-conditioned state-matching
2. Detailed technical derivation and analysis of GoFAR's distinct features: (1) relabeling-fiee
(2) uninterleaved optimization, and (3) planning capability
3. Extensive experimental evaluation of GoFAR, validating its empirical gains and capabilities
2 Related Work
Offline Cond-Conditooned Reinforcement Learning, One core challenge of (offline) GCRL [8] 43 the sparse-reward nature of goal-reaching tasks. A popular strategy hindsight goal relabeling (or HER), which relabels off- off-policy transitions with future goals they have achieved instead of the original commanded goals. Existing offline GCRL algorithms 6 adapt HER-based online GCRL algorithms to the offline setting by incorporating additional components conducive to offline training. [6] builds on actor-critic style GCRL algorithms [3] by adding conservative Q-learning [24] as well as goal-chaining which expands the pool of candidate relabeling goals to the entire offline dataset.
Besides actor-critic methods, goal-conditioned behavior cloning, coupled with HER, has been shown to be a simple and effective method [1]. 47 improves upon (14 by incorporating discount-factor and advantage function weighting [37
offline GCRL. Diverging from existing literature, GoFAR does not tackle offline GCRL by adapting online algorithm; instead, it approaches offline GCRL from a novel perspective of state-occupancy matching and derives an elegant algorithm from the dual formulation that is naturally suited for the offline setting and carries many distinct properties that make it more stable, scalable, and versatile.
38 and shows improved performance
Cacte-Occupacyy Matching, State occupancy matching objectives have been shown to be effective for learning from observations
K2 49). exploration [27]]. as well as matching hand-designed target state distribution [3] occupancy matching, in general, has been explored in the imitation learning
literature [1
45
13 20 Our strategy of estimating the state-occupancy ratio related DICE
techniques for offline policy optimization [33 35 26 [2] The closest work to ours is [32] which
proposed f f-divergence state-occupancy matching for offline imitation learning. Our work shows that such saat--occupancy matching approach can be applied to GCRL. Algorithmically, GoFAR differs from [32) in that it does not require separate dataset of expert demonstrations; achieves this by modifying the discriminator, and in some settings can even be used without a discriminator, whereas discriminator is always required by (32). Furthermore, we derive several new algorithmic properties (Section 4.4) that are novel and particularly advantageous for offline GCRL.
3 Problem Formulation
Goal-Conditioned Reinforcement Learning. We consider an infinite-horizon Markov decision process (MDP) [40] M = (S, A, R,T,
rewards r(s, a). stochastic transitions ; 7) No, with state space S, action space A, deterministic ~ T(s, a). initial state distribution po(s), and discount factor (0, 1]. A policy π S + ((A) outputs distribution over actions use in given state. In goal-conditioned RL, the MDP additionally assumes a goal space G == {Ø(s) E state-to-goal mapping
S}, where the 5 - G is known. Now, the reward function r(s: 9 well the policy π (a s. g) depend on the commanded goal E G. Given a distribution over desired goals p(g), the objective of coal-conditioned RL is to find a policy æ that maximizes the discounted return:
((T) Eg-plg) (ssg (18),00))
Σ L=
(2)
of is The goal-conditioned state-action occupancy distribution d= (8, () S x G - 1f
d"(ssa (1-.a:))((111) /Pr(s: s, s, = a so ~ no. ~ ( $1+1 ~ T(ss, ac)
Σ
(3)
which captures the relative frequency of state-action visitations for policy conditioned on goal g. The state-occupancy distribution then marginalizes over actions: d (x: = Σ. (s, (s,a; q).). Then,
In GCRL, the reward function customarily does not depend action.
3

---

follows that raa s. (a.ag)) A state-action occupancy distribution must satisfy the Bellman
(=:))
flow constraint in order for be an occupancy distribution for some stationary policy
Σ d(s, (1 ()) +Y
> T(s (, ā:g),
(4)
We write (s. g) = pl Finally, given d², we can express the objective function (2) as J(T) 1-y (o.g)>a (a.a) (r(s;
dd (s;
the joint goal-state density induced by p(g) and the policy
Offline GCRL. In offline GCRL, the agent cannot interact with the environment M; instead, it is equipped with static dataset of logged transitions D == {Ti) where each trajectory -00 (s0 (8) :TO { ...',"..gg))) with =/"" ~ μí and g is the commanded goal of the trajectory. Note that trajectories need not to be generated from a goal-directed agent, which case g can be randomly drawn from p(g). We denote the empirical goal-conditioned state-action occupancies of D as d (s, a; g).
Goal-Conditioned f f-Advantage Regression
In this section, we introduce Goal-conditioned f-Advantage Regression (GoFAR), We first derive the algorithm in full (Section 4.1). then delve deep into its several appealing properties (Section 4.2 4.4
4.1 Algorithm
We first show that goal-conditioned state-occupancy matching is a mathematically principled approach for solving general GCRL problems, formalizing the teleportation intuition in the introduction.
Proposition 4.1. Given any g)s g). for each g in the support of p(g), define p(s; g) = Z(a) where
7(g) = (r(ss) ds is the normalizing constant. Then, the following equality holds:
(Dx ()))(sppps:)))) 9)) +C (1- + (dd (s) (s)
(5)
where (TT) is the GCRL objective (Eq. (2) with reward g(::)) and C == Lg- [log [g))).
See Appendix 1. for proof. This proposition states that, for any choice of reward r(s; g). solving the GCRL problem with maximum state-entropy regularization is equivalent to optimizing for the goal-
conditioned state-occupancy matching objective with target distribution p(=;g) == -(() Now, the
c'(Sip)
key challenge with optimizing this objective offline is that we cannot sample from d' (s: To address
this issue, following [32], we first derive an offline lower bound involving an f-divergence (see Appendix for for definition) regularization term, which subsequently enables solving this optimization problem via its dual using purely offline data:
Proposition 4.2. Assum for all g in support of p(g). Vs, d (s; 0(fp(s;;)) > O. Then, for any f-divergence that upper bounds the KL-divergence,
log p(s:g) do (s:g) - g))d g))g))
(ds.g)
(6)
The RHS of 66 can be understood as an f-divergence regularized GCRL objective with reward p(e:s)
function R(s;g) log [5:g) (we use capital R to differentiate user-chosen reward R from the environment reward r). Intuitively, this reward encourages visiting states that occur more often the "expert" state distribution p(s; g) than in the offline dataset, and the f-divergence regularization then ensures that the learned policy is supported by the offline dataset. This choice of reward function can be estimated in practice by training discriminator [15] S x G + (0, 1) using the offline data: (#:a) [log 1 c(s,g)]]
min Ey-p(g) [Ep(s:g) [log c(s, + (
We can in fact obtain a looser lower bound that does not require training discriminator (see B for derivation):
- DKL (<:g))))))))) 2 E (s.g)) (o.0) [logp(s; g)] (d a ())
(8)
2 This assumption is only needed in our technical derivation to avoid division- by-zooo issue.

---

Because p(s;g) & c'(ss)), we may substitute R(s;g) := r(s;g) (when the offline dataset contains reward labels) for log p(s;g) and bypass having to train discriminator for reward.
Now, for either choice of lower bound, we may pose the optimization problem with respect to valid choices of state-action occupancies directly, introducing the Bellman flow constraint 4
((s.))=d(s.n)) (r(s:g) - (d(((s,)))))))))))))))s)) (s, a; g)lld (s.ag))
max d(s.a:9)20
(9)
(P)
S.t. Σ Ed(s,agg) (1 ()) + T(s d.
This reformulation does not solve the fundamental problem that p still requires sampling from d(s;g); however, it has now written the problem in a way amenable to simplification using tools from convex analysis. Now, we show that its dual problem can be reduced to an unconstrained minimization problem over the dual variables which serve the role of value function; importantly, the optimal solution to the dual problem can be used to directly retrieve the optimal primal solution: Proposition 4.3. The dual problem to 9 is
(D) min (1-7))( [V g)] +B (o [ffe (R(s;g) + TTV(s, a; V(s; g)),, V(s.g)20
(10)
where So denotes the convex conjugate function of f. Vssg) is the Lagrangian vector, and TV (2 = E. Ex V (s'::)) Given the optimal V*, the primal optimal d' satisfies:
d' (s, (: d (s, g. (R(s:) TTV* (s, gz V (x: Vs S, E AggEG (11)
A proof is given in Appendix B Crucially, as neither expectation in (TO) depends on samples from d, this objective can be estimated entirely using offline data, making suitable for offline GCRL. For tabular MDPs, we show that for suitable choice of f-divergence, the optimal V in fact admits closed-form solutions; see Appendix for details. the continuous control setting, we can optimize I0 using stochastic gradient descent (SGD) by parameterizing V using a deep neural network.
Then, once have obtained the optimal (resp. converged) V*, we propose learning the policy via the following supervised regression update:
max Ey-rig) ((e.)))d((ssa3) [ff + +TV g) (*(ss g)) log T(a s,
(12)
We see that the regression weights are the first-order derivatives of the convex conjugate of f evaluated the dual optimal advantage, R(s; g) + +TV' a:
V*(s: g); we refer this weighting term as f-advantage. Hence, we name our overall method Goal-conditioned f f-Advantage Regression (GoFAR); an abbreviated high-level pseudocode is provided in Algorithm [ and a detailed version is provided in Appendix C In practice, we implement GoFAR with xivdeeerg,,ee, choice of f-divergence that is stable for off-policy optimization
26 see Appendix[C for details.
Algorithm 1 -onaalCondteoeeed f f-Advantage Regression (Abbreviated); 3-disjoint steps
Note that (12) forgos directly minimiz- ing ( offline, which has been found to suffer from training instability [21. Instead, it naturally incorporates the primal-dual optimal solutions in re- gression loss. Now, we will show that this policy objective has several theo- retical and practical benefits for offline GCRL that make it particularly appealing.
(Optionally) Train discriminator based reward (7)
2: Train (optimal) dual value function V*(ssg g) (10)
Train policy æ via f-Advantage Regression (2)
4.2 Optimal Goal-Weighting Property
We show that optimizing 12)
automatically obtains the optimal geig-weigtting distribution. That is, GoFAR trains its policy as if all the data is coming from the optimal goal-conditioned policy for (9). In particular, this property is achieved without any explicit hindsight relabeling (see Appendix Alror technical definition). mechanism that prior works heavily depend on. To this end, we first define plg 8, a) to be the conditional distribution of goals in the offline dataset conditioned on state-action pair (s, a). Then, according to Bayes rule, have that
p(g s, = (s, az g) g)p(g) + p(g) = p,
(13)
(8
5

---

Using this equality, we can rewrite the policy objective (12 as follows:
min ((e.a) ( goppllss.a [C' 9) +,TV"(s,,;;) ( (a 1,99] (14)
min -E (a B Malaaa [o s,g)]
(15)
where
d' (s. a: g)
pig s. x plg s. (f.
p(g s,
(16)
d (s, a:g)
Thus, we see that GoFAR's f f-advantage weighting scheme is equivalent to performing supervised policy regression where goals are sampled from p(g s.0). Now, combining (T6) and Bayes rule gives
d' as)
d (s, a) s, & d as.g) s,
pla (s.a) = (lg))
(17)
p(g)
Thus, we can replace the nested expectations in (15) and obtain that GoFAR policy update amounts to supervised regression of the state-action occupancy distribution of the optimal policy to the regularized GCRL problem (
GoPAR = min Eggupts) ((e.a))dd (s.agg) [log (a 8,99]]
(18)
This derivation makes clear GoFAR's connection with the hindsight goal relabeling mechanism S that is ubiquitous in GCRL: GoFAR automatically performs the optimal goal-weightig policy update without any explicit goal relabeling. Furthermore, our derivation also suggests why hindsight relabel- ing sub-optimal without further assumption on the reward function [10]: heuristically chooses p(g s, a) to be the empirical trajectory-wiss future achieved goal distribution (i.e., HER), which generally does not coincide with the goals that the optimal policy would reach; see Appendix for further discussion.
4.3 Uninterleaved Optimization and Performance Guarantee
An additional algorithmic advantage of GoFAR is that it disentangles the optimization of the value network and the policy network. This can be observed by noting that GoFAR's s advantage term (II) is computed using V*, the optimal solution of the dual problem. This has the practical significance of disentangling the value-function update (10p from the policy update (12). as we do not need to optimize the latter until the former has converged. This disentanglement is in sharp contrast to prior GCRL works [47 6
I 33, which typically involve alternating updates to the critic Q-network and the policy network, training procedure that has found to be unstable in the offline setting 23 This is unavoidable for prior works because their advantage functions are estimated using the Q-estimate of the current policy, whereas our advantage term naturally falls out from primal-dual optimality.
The uninterleaved and relabeling-free free nature of GoFAR also allows us to derive strong performance guarantees. Because V is fixed in (12), this policy objective amounts
weighted supervised learning problem. Therefore, we can extend and adapt mature theoretical results for analyzing Behavior Cloning [ 42 46 well finite sample error guarantees for weighted regression (7] to obtain statistical guarantees on GoFAR's performance with respect to the optimal π for (
Theorem 4.1. Assume suPs.a.g (s.ag))) S M and suplr(s, g)l S Rmax. Consider policy class
d' d'(s,a:))
II
{S > ((A)} such that x² E II. Then, for any 8 (0, 1), with probability at least
GoFAR 18] will return a policy ft such that:
2Rmax
In (1181//88 N
--- V*
(19)
Notably, the error shrinks as the size of the offline data N increases, requiring no dependency on access to data from the "expert" distribution d'. This provides a theoretical basis for GoFAR's empirical scalability as we are guaranteed to obtain good results when the offline data becomes more expansive. contrast, prior regression-batt based GCRL methods cannot be easily reduced to a simple weighted regression with respect to the desired goal distribution p(g). so they only obtain weaker results under stronger assumptions on the offline data (e.g, full state-space coverage) as well as the policy; see Appendix EK for discussion.

---

Table 1: Discounted Return, averaged over 10 seeds.
Task
Supervised Learning
Actor-Critic AM
GoFAR (Ouns)
WGCSL
GCSL
DDPG
FetchReach
28.2 061
21.91 2.13 (1.0)
20.91 = (1.0)
30.1 0.32 (0.5)
29.8 f 0.59 (0.2)
FetchPick
19.7 2.7
9.84 1 258 (1.0)
8.94 - (1.0)
18.4 151 (0.5)
16.8 + (0.)
FetchPush (+)
18.2 300
14.7 1 268 (1.0)
13.4 (1.0)
14.0 281 (0.5)
12.5 - (0.)
FetchSlide
2.47
2.73 164 (1.0)
1.75 1.110)
1.46 = 1.38 (0.5)
1.08 = (0.5)
HandReach (*)
11.3 + 5:66
5.97 + 484 (1.0)
1.37 = 221 (1.0)
0. = 0u (0.5)
0.81 # L.S 173 (0.5)
"ClawTurn (+)
9.34 + 3.15
0.0 : 00 (1.0)
0.0 + 0. (1.0)
2.82 1 (1.0)
0.01 0. (0.2)
Average Rank
1.5
3
4.17
2.83
4
4.4
Goal-Conditioned Planning
Next, we show that GoFAR can be used to learn a goal-conditioned planner that supports zero- shot transfer to other domains of the same task. The key insight is that GoFAR's value function objective (IO) does not depend on actions assuming deterministic transitions. This is because given a transition (s, a. s', 9) do (V a; g) (s':g)) g). so we can rewrite the second term in (10) as (9)==0 [f+ (R(s;)) g) + (V
Vss; g))l, which does not depend on the action. This property enables us learn a goal-cenine centric value function that independent of the agent's state space because the agent's actions are not relevant. Specifically, propose the following objective:
:9>=do
min (1 (1=)))\(sa)=wooodo (V V()(s).9)20
-o) [f+ (R(s;g) + (V(((s)) V(o(s),g))
(20) We can think of this objective as learning value function with the inductive bias that the first operation transforms the state input to the goal space via Φ. Since is now independent agent, use f-advantage regression to instead learn goal-conditioned planner:
max LE g(o) (',,'' gi-do (R.((((s))) + +VV (b(s');g) V (o(s):g)) log (ó(s') ((5), 9))
(21) where π now outputs the next subgoal d(s') conditioned on the current achieved goal ((s) and the desired goal In our experiments, we show how this planner can achieve hierarchical control through zero-shot transfer subgoal plans new target domain.
Experiments
We pose the following questions and provide affirmative answers in our experiments:
1. Is GoFAR effective for offline GCRL? What components are important for performance?
2. Is GoFAR more robust to stochastic environments than hindsight relabeling methods?
3. Can GoFAR be applied to real-robotics system?
4. Can GoFAR learn goal-conditioned planner for zero-shot cross-embodiment transfer?
5.1 Offline GCRL
Tasks. In our simulation experiments, consider six distinct environments. They in- clude four robot manipulation environments 39
FetchReach, Anddlaec FetchPush, and FetchSlide, and dexterous manipu- lation environments: HandReach (39], and D' D'ClawTurn 22] All tasks use sparse reward, and their respective goal distributions are de- fined over valid configurations in either the robot space or object space, depending on whether the task involves object manipulation. The offline dataset for each task is collected by either a random policy or a mixture of 90% random policy and 10% expert policy [32 21, depending on whether random data provides enough coverage of the desired goal distribution. The first five tasks and their datasets are taken from [47]. See Appendix for dataset details and Appendix F for detailed task descriptions and figures.
Figure 3: D'Claw (left);
trans- fer source (middle) and target (right) domains.

---

Algorithms. We compare to state-of art offline GCRL algorithms, consisting of both regression- and actor-critic methods. The regression-basee methods are: (1) GCSL [4. which incorporates hindsight relabeling in conjunction with behavior cloning to clone actions that lead to a specified goal, and (2) WGCSL [47], which improves upon GCSL, by incorporating discount factor and advantage weighting into the supervised policy learning update. The actor-critic methods are (1)
DDPG [3], which adapts DDPG [29] to the goal-conditioned setting by incorporating hindsight relabeling, and (2) ActionableMode (AM) [6], which incorporates conservative Q-Learning [24] as well as goal-chaining on top of an actor-critic method.
We use tuned hyperparameters for each baseline on all tasks; in particular, we search the best HER ratio from {0.2.0.3 1.0} for each baseline on each task separately and report the best-periie performennnnngggg one. For GoFAR, we use identical hyperparameters as WGCSL for the shared network components and do not tune further. We train each method for 10 seeds, and each training run uses 400k minibatch updates of size 512. Complete architecture and hyperparameter table as well as additional training details are provided in Appendix G
Evaluations and Results. We report the discounted return using the sparse binary task reward. This metric rewards algorithms that reach goals as fast as possible and stay in the goal region thereafter. In Appendix H. we also report the final success rate using the same sparse binary criterion. Because the task reward binary, these metrics do not take account into how precisely a goal is being reached. Therefore, we additionally report the final distance to goal in Appendix H
The full discounted return results are shown in Table T (*) indicates statistically significant improve- ment over the second best method under a two-sample 1-test. The fraction inside () indicates the best HER rate for each baseline. As shown, GoFAR attains the best overall performance across six tasks, and the results are statistically significant on three tasks, including the two more challenging dexterous manipulation tasks. In Table i in Appendix H we find GoFAR is also superior on the final distance metric. In other words, GoFAR reaches the goals fastest and most precisely.
Ablations. Recall that GoFAR achieves optimal goal-weighting and does not require the HER heuris- tie that has been key to prior GCRL approaches. We now experimentally confirm that the baselines are not performant without HER. Conversely, investigate whether HER can help GoFAR. Additionally, we evaluate GoFAR using the sparse binary reward, which implements the looser lower bound in Finally, replace x >ivvevrgeeece with KL to understand whether the f-divergence lower bound in (6) is necessary. The results are shown in Figure [ (task-specffic breakdown is included in Appendix H... Compared to their original performances (col- ored dash lines), baselines without HER suffer severely, especially GCSL and WGCSL: both these methods in fact require 100% relabeling and rely solely on HER for positive learning signals. In sharp contrast, GoFAR also regression-based method, yet performs very well without HER and its performance is unaffected by adding HER. These findings are consistent with the theoretical results of Section[4.2] GoFAR (Binary) performs slightly worse than GoFAR. which is to be expected due the looser bound; however, is still better than all baselines, which use the same binary reward and are aided by HER. Again, this ablation highlights the merit of our optimization approach. Finally, GoFAR (KL. suffers from numerical instability and fails to learn, showing that our general f-divergence lower bound (6) is not only of theoretical value but also practically significant.
GoFAR Ablations and Baselines withou: Hindsight Relabeling
Averaye Discourited Return
AM
3DPG HERR
CCSL -ERR
WGCSL HERR
GuFAR
GuFAR
GuFAR (Birary)
GuFAR (KL)
HEER
+ HER
Figure 4: Offline GCRL ablation studies. While GoFAR is robust to hindsight relabeling, removing it is highly detri- mental to all baselines.
5.2 Robustness in Stochastic Offline GCRL Settings
For this experiment, we create noisy variants of the FetchReach environment by adding varying levels of white Gaussian noise to policy action outputs. FetchReach is well-suited for this experiment because all baselines attain satisfactory performance and even outperform GoFAR in Table I Further-

---

more, to isolate the effect of noisy actions, we use GoFAR with binary reward so that all methods now use the same reward input and perform comparatively without added noise. We consider zero-mean Gaussian noise of value from 0.5, 1, and 1.5. For each noise level, we re-collect offline data by executing random actions in the respective noisy environment and train on these noisy offline random data.
The average discounted returns over different noise levels are illustrated in F As shown, noise level 0.5, while GoFAR's performance is not effected, all baselines already exhibit degraded performance. In particular, DDPG sharply degrades, underperforming GoFAR de- spite better original performance. At 1.0, the
FetchReach with Noise
Discounted Return
26
15
CoFAR
10
DDPG
gap continues to widen, and AM notably col- lapses, highlighting its sensitivity to noise. This expected as AM's relabeling mechanism also samples future goals from other trajectories and labels the selected goals with their current Q- value estimates; this procedure becomes highly noisy when the dataset already exhibits high stochasticity, thus contributing to the instability observed in this experiment. It is only at noise level 1.5 that GoFAR's 's performance degradation becomes comparable to those of other methods. Thus, GoFAR's relabeling-free free optimization indeed confers greater robustness to environmental stochasticity; in contrast, all baselines suffer from "false positive" relabeled goals due such disturbances.
AM
0.0
0.5
1.0
1.5
Figure 5: Stochastic environment evaluation. Go- FAR is more robust to stochastic environments due to its lack of hindsight goal relabeling.
5.3 Real-World Robotic Dexterous Manipulation
Now, evaluate real D'Claw (2) dexterous manipulation robot (Figure 3 (left)). The task is to
have the tri-finger robot rotate the valve to a specified goal angle from its initial angle; see Appendix F for detail. The offline dataset contains 400K transitions, collected by executing purely random actions in the environment, which provides sufficient coverage of the goal space. Compared to the simulated D' Claw, additional challenge the inherent stochasticity on a real robot system due to imperfect actuation and hardware wear and tear. Given our conclusion in the stochastic environment experiment above, should expect GoFAR perform better.
The training curve of final distances averaged
D'ClawTurn (Real)
over 3 seeds is shown in Figure 6 We that GoFAR exhibits smooth progress over training and significantly outperforms all baselines. At the end of training, GoFAR is able to rotate the valve on average within 10 degrees of the specified goal angle. In contrast, most baselines do not make any progress this difficult task and learns policies that do not manage to turn the valve at all. AM is the only baseline that learns non-trive non-trivial policy; however, its training fluctuates significantly, and the learned policy often executes extreme actions during evaluation that fail reach the desired goal. We provide qualitative analysis Appendix H.3 and videos are included in the supplement.
Final Distance
0.8
0.6
0.4
0.2
10000
20000
30000
40000
50000
60000
70000
Gradient Updates
GoFAR
WGCSL
GCSL
DDPG
Figure Real-robot results.
5.4 Zero-Shot Transfer Across Robots
For this experiment, we consider zero-shot trans- fer from the FetchPush task (source domain) to an equivalent planar object pushing task using the 7-DOF Franka Panda robot [1]: the environ- ments are illustrated in Figure 3, In the target do- main, we assume a pre-programmed learned low-level goal-rean coal-reaching controller Tlo (a. s, s,
Table 2: Zero-shot transfer results.
Method
Success Rate
GoFAR Planner (Oracle)
84%
GoFAR Hierarchical Controller Low-Level Controller
37%
19%

---

that can push objects over short distances but fails for longer tasks. For these distant goals, we train the goal-conditioned GoFAR value function V ()(())): g) and planner (()(s') ((s).g) according
to (20) and (2T). set goals for Tlw- We use the same offline FetchPush data in Section 5.1 We
compare two approaches: (1) naively using the Low-Level Controller, and (2) using the GoFAR planner to guide the low-level controller (GoFAR R Hierarchical Controller). See Appendix G.4
for additional details. We report the success rate over 100 random distant test goals Table 2 We
see that naively commanding the low-level controller with these distant goals indeed results in very low success rate. When the controller is augmented with GoFAR planner, the success rate nearly doubles. Note that our planner itself is much superior: if all subgoals Planner (Oracle) achieves 84% success. See Appendix G.4 for additional experimental details. These experiments validate GoFAR's s ability to shot transferring subgoal plans to enhance the capabability of low-level controllers. Note that we have no access to any data from the target domain, and this planning capability naturally emerges from training objectives and does not require any change the algorithm. We show qualitative results in Appendix HH.4 and videos the supplement.
perfectly reached, GoFAR
Conclusion
We have presented GoFAR, a novel pearession-based offline GCRL algorithm derived from state- occupancy matching perspective. GoFAR is relabeling-fre and enjoys uninterlcaved training, making both theoretically and practically advantageous compared to prior state-of-art. Furthermore, GoFAR supports training a goal-conditioned planner with promising zero-shot transfer capability. Through extensive experiments, we have validated the practical utility of GoFAR in various challenging problem settings, showing significant improvement over prior state-of-art. We believe that GoFAR's strong theoretical foundation and empirical performance make it an ideal candidate for scalable skill learning in the real world.
Acknowlegement
We thank Edward Hu and Oleh Rybkin for their feedback on an earlier draft of the paper. Kun Huang for assistance the real-robot experiment setup, and Selina Li for assistance on graphic design. This work is funded in part by an Amazon Research Award, gift funding from NEC Laboratories America, NSF Award 1CF-1910769, NSF Award CCF-19178S3 and ARO Award W911NFF-0- 1-0080. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

References
[1] Alckh Agarwal, Nan Jiang, and Sham M Kakade. Reinforcement learning: algorithms. 2019.
Theory and
[2] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Confer- ence robot learning, pages 1300-1313. PMLR, 2020.
[3] Marcin Andrychowicz, Filip Wolski, Alex Ray. Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances neural information processing systems, 30, 2017.
[4] Stephen Boyd, Stephen Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[5] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pages 1430-1440. PMLR, 2021.
[6] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.0772: 2021.
[7] Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. Advances in neural information processing systems, 23, 2010.
[8] Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distributions via dual embeddings, 2016.
[9] Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. arXiv preprint arXiv: 09/0. 1215, 2019.
[10] Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information processing systems, 33:14783-14795, 2020.
[11] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909 2020.
[12] Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. Multi-gon reinforcement learning environments for simulated franka emika panda robot. arXiv preprint arXiv:2106. 33687, 2021.
[13] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimiza- tion perspective on imitation learning methods, 2019.
[14] Dibya Ghosh, Abhishek Gupta. Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
[15] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
[16] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
[17] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[18] Leslie Pack Kaelbling. Learning achieve goals. Citeseer.
[19] Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint 1:Xiv:2104.082/11 2021.
[20] Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srini- vasa. Imitation learning f-divergence minimization, 2020.
11

---

[21] Geon-Hyeooe Kim, Seokin Seo, Jongmin Lee. Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In International Conference Learning Representations, 2022.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[23] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.009/4 2019.
[24] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.00 0.7.799900.4779, 2020.
[25] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pages 45-73. Springer, 2012.
[26] Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offine policy optimization via stationary distribution correction estimation. arXiv preprint arXiv:2106. 10783, 2021.
[27] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1996.00574. 2019.
[28] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01645 2020.
[29] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.0297, 2015.
[30] Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pages 1113-1132. PMLR, 2020.
[31] Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
[32] Yecheng Jason Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Smodice: Versatile offline imitation learning via state occupancy matching, arXiv preprint arXiv:2202.02243 2022.
[33] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. arXiv preprint arXiv: 1906.04333 2019.
[34] Ofir Nachum and Bo Dai. Reinforcement learning via fenchel--occafelllar duality, 2020.
[35] Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Al- gaedice: Policy gradient from arbitrary experience. arXiv preprint drXiv:191..02072 2019.
[36] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
[37] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintage-weiighted regression: Simple and scalable ofl-p reinforcement learning. arXiv preprint 19X0v:1910.000 2019.
[38] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty- Fourth AAAI Conference on Artificial Intelligence, 2010.
[39] Matthia Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.00446. 2018
[40] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming.
John Wiley & Sons, 2014.
[41] R. Tyrrell Rockafellar. Convex analysis. Princeton Mathematical Series. Princeton University Press, Princeton, N. J., 1970.
[42] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
12

---

[43] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. Francis Bach and David Blei, editors. Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.
[44] Yannick Schroecker and Charles Lee Isbell. Universal value density estimation for imitation learning and goal-conditioned reinforcement learning, 2021.
[45] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXiv:
2018.
[46] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances Neural Information Processing Systems, 13:15737-15749, 2020.
[47] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking coal-conditoone supervised learning and its connection to offline RL. In International Conference on Learning Representations, 2022.
[48] Siyuan Zhang and Nan Jiang. Towards hyperparameter--an free policy selection for offline rein- forcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[49] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. Advances in Neural Information Processing Systems, 33, 2020.
[50] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] We include the
code and data for our simulation experiments in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix G
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix H
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix G
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [Yes]
We include videos in our supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ybVV7mDZTzB", "reviewer": "Reviewer_U8M8", "summary": "The paper presents an approach for offline goal-conditioned RL that is based on state-occupancy matching. In particular, the paper outlines a procedure for a goal-conditioned state occupancy optimization in an offline setting by deriving a lower bound on the optimization objective such that it does not require online sampling. While doing so, the algorithm incorporates f-divergence based dual optimization and decouples value function from actor policy training, which improves stability of training. In contrast to prior works, the method does not require hindsight relabeling, which further stabilizes the training. The algorithm is compared to multiple recent GCRL baselines and it is shown to outperform them on several simulated robotic manipulation tasks as well as in a dexterous real-world D’Claw environment.", "questions": "Please refer to Strengths And Weaknesses for a list of questions.", "limitations": "The authors discussed limitations of their work. The proposed approach does not lead to any obvious negative societal impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents an approach for offline goal-conditioned RL that is based on state-occupancy matching. In particular, the paper outlines a procedure for a goal-conditioned state occupancy optimization in an offline setting by deriving a lower bound on the optimization objective such that it does not require online sampling. While doing so, the algorithm incorporates f-divergence based dual optimization and decouples value function from actor policy training, which improves stability of training. In contrast to prior works, the method does not require hindsight relabeling, which further stabilizes the training. The algorithm is compared to multiple recent GCRL baselines and it is shown to outperform them on several simulated robotic manipulation tasks as well as in a dexterous real-world D’Claw environment.\n\nStrengths\n- Efficiently training goal-conditioned policies from offline data promises to enable training on large and diverse robotic datasets.\n- The paper is well-written, and straightforward to understand and follow. Mathematical derivations are coherent.\n- The method is shown to work on a variety of interesting manipulation robotic tasks, including a real world dexterous manipulation task.\n- The paper provides source code of the method implementation.\n\nCriticism\n- It would be interesting to see an evaluation of the method on higher-dimensional and especially articulated visual goal tasks to better understand the scalability of the approach. \n- It would be interesting to see an evaluation on tasks that require stitching of multiple episodes to accomplish a task, to better understand the ability of the method to combine experiences from different trajectories.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ao84CqYgQGP", "reviewer": "Reviewer_PYXz", "summary": "This work presents a method for *offline* goal-conditioned policy learning, from offline data that *does not contain actions or rewards*, using only a specified target goal distribution p(g). The approach, named GoFAR, is formulated as a state-occupancy matching objective, and draws upon a similar derivation as SMODICE [1], with additional modifications to be applicable to the goal-conditioned setting. As a result, GoFAR inherits the advantages of SMODICE. Experiments study various components of the GoFAR method, provide real-world robotic results, and demonstrate cross-embodiment transfer.\n\nReferences:\n\n[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching\n", "questions": "- Why is this called goal-conditioned RL? The objective being discussed throughout the paper, i.e. objective (1), is a state-occupancy matching objective that does not have any rewards involved. Also the title does not reflect that the method is directed towards the setting where the offline dataset contains states only (no rewards or actions).\n- I don't see how the teleportation described in the abstract is relevant to this work? Do you mean because p(g) is a distribution of goal states?\n- I think in this work the authors need to discuss SMODICE [1] more thoroughly as the method is very closely related. Furthermore, a number of \"advantages\" of GoFAR that are listed are due to the use of SMODICE-like methodology, rather than a direct contribution of this work. Such aspects should be more clearly discussed. Also in Section 4.1, it would be good if the manuscript noted that the derivations draw upon similar derivations in prior DICE lines of work.\n- For state-occupancy matching related work it would be good to cite [2] as they also present results with state-marginal/occupancy matching, and present results for target state distributions that were not generated by an agent (e.g. hand designed target state distributions).\n- Equation (7): It is not clear to me why this is a looser bound than (5). Do you have a derivation for why this would be a lowerbound to (5)?\n- Equation (11): You should explain why you are proposing to learn the policy using equation 11. Namely the fact that the \"f-advantage\" is equal (at least theoretically) to the density ratio $\\frac{d^*}{d^O}$.\n- Line 192: Can you please elaborate on how Bayes rule leads to that equality? Also, I don't have an intuition for what p(g | s, a) is? You explained initially that p(g) is a \"desired goal distribution\", but I don't understand what p(g | s, a) would mean?\n- Section 5.1: It would be good to note which methods are state-only, which methods are state and action, and which methods use rewards. Despite the appendix, it would also be good to at least briefly mention what are the task rewards (e.g. dense or sparse), how the goal distributions are defined, etc.\n- Section 5.1: Instead of DDPG, why did you not opt for an offline RL method, given that online RL algorithms tend to perform poorly in offline settings?\n- Section 5.1: Please report standard deviation of results, even if it's only 3 random seeds. It is very challenging to understand the significance of your results without error metrics.\n- Section 5.1: Can you elaborate on how the sparse binary reward is used with equation (7) to implement GoFAR (binary)?\n- Baselines: I think a potentially unfair advantage that GoFAR has compared to the other approaches is that it is being given $p(g)$, the distribution of desired goal states. In contrast for example, HER is relabeling any future state as a goal state and performing updates using all those possible relabeld \"goals\". I think important alternative baselines would be: 1) use $p(g)$ in HER as well. For example, only do HER relabeling using goals that are in the support of $p(g)$. 2) Perform goal-conditioned offline RL, e.g. using CQL or any favorite offline RL method, where the input to the policy / value function is (s, g) / (s, a, g), where g ~ p(g), and the reward is relabeled as r(s, a, g).\n- In the appendix G you don't describe how the $\\tilde{O}$ relabeled distribution is obtained for each method, e.g. GCSL.\n- Section 5.2: Without errorbars, it is not possible to evaluate the significance of the results.\n\nWith regards to aspects that would directly help me better evaluate and improve my score:\n- Addressing my concern regarding baselines, either verbally convincing me that I am mistaken in my thought process / assumptions, or implementing the two suggested baselines and demonstrating that they would not resolve the poorer performance of the baselines you tried.\n- Highlighting exactly what you believe are the core contributions of *your work*, not aspects of your work that are due to the use of SMODICE, etc. What can GoFAR do that prior methods cannot (e.g. type of data, type of available information, etc.)? What is the first impressive result you would show a person? If I am choosing an algorithm for my use-case, why would I opt for GoFAR as opposed to the other methods (currently I am not convinced solely by numerical results since there is a lack of reporting of standard deviation / error bars across random seeds, as well as my concern regarding baselines discussed above)?\n\nReferences:\n\n[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching\n\n[2] A Divergence Minimization Perspective on Imitation Learning Methods\n\n", "limitations": "Authors have not highlighted the limitations / potential failure cases / future work for GoFAR.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n- I would not necessarily consider this work very original. As the authors point out, goal-conditioned reinforcement / imitation learning (as well as their state-only variants) are important well-studied problems. Furthermore, the GoFAR approach appears to be an extension of SMODICE [1] to the goal-conditioned setting.\n\nQuality:\n- I believe the quality of the paper is good. The writing is largely clear, and the experiments try to cover various aspects of the approach. However, there are a number of questions / concerns noted below.\n\nClarity:\n- The paper is generally well-written\n\nSignificance:\n- I think the significance can be better understood after authors response to my questions / concerns below.\n\nReferences:\n\n[1] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AD3zw0crhM", "reviewer": "Reviewer_D96G", "summary": "The authors propose a mechanism for performing offline multi-goal reinforcement learning without hindsight goal relabelling, by attempting to mimic the state-occupancy distribution of an agent that teleports directly to the goal. This avoids HER's well-known hindsight bias, and allows GoFAR to perform better on stochastic environments that are challenging for methods that employ hindsight goal relabelling. They find that this method outperforms offline versions of DDPG with HER and several other offline GCRL algorithms on several experiments, including real robot experiments. Additionally, they use the same method to derive a goal-based planner that allows for zero shot transfer to systems with new dynamics, as long as they share the same goal space and reward structure. They additionally prove several finite-sample guarantees for the method.", "questions": "\nFor the fetch robot results, I'd like to see training curves shown with error bars for statistical significance. Statistical significance is reported for these experiments, but only in the appendix, which hides that fact that the improvement over other methods is not statistically significant for several tasks. The lack of improvement on these tasks is not in and of itself a problem, but it should be easier for a casual reader to see what is significant. I'd also like to see success rates for these methods -- it's more informative and intuitive than reward alone. \n\nI'd also like to see the goal-conditioned planner elaborated on a bit more, if space allows. For instance, what does a policy over goals mean exactly, and what does it look like? How do the formal guarantees given earlier in the paper translate to a policy that outputs goals", "limitations": "There seem to be no significant negative social impacts, and no obvious limitations other than those that are inherited from GCRL and offline RL. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$\\textbf{Originality: }$\nThe method is fairly original. It uses similar algorithm to SMODICE, but adapts it to a goal-conditioned setting. The two most unique insights here seem to be \n1) given a goal, the \"optimal\" teleporting agent's behavior is known $\\textit{a priori}$, which means no expert demonstration is needed -- the learning objective can be derived analytically, while still retaining the strong theoretical guarantees from imitation learning. \n2) the derived algorithm can be transformed into a planner over intermediate goals. This allows a formulation that does not depend on actions, but instead plans over the goal space, allowing zero-shot generalization to novel dynamics, as long as the goal-space and task remain the same. \nThese are both interesting and creative insights. \n\n$\\textbf{Quality: }$\nThe math results generally look solid, and provide some nice guarantees on convergence. \n\nThe experimental results are slightly more mixed. For several experiments, including the standard fetch experiments, no measures of statistical significance are reported in the main paper, and are only shown in the appendix. Additionally, success rates are not reported for these experiments. Broadly, the results for the Fetch environments  indicate that the method compares favorably against other offline methods, but still falls significantly short of the performance achieved by online methods. A real-robot experiment shows that the method is able to learn a task in which it manipulates a knob, while other methods fail. \n\nMost interesting is the goal planner experiment, in which the authors are able to significantly improve the performance of a weak policy by learning a sequence of goals that may be zero-shot transferred across different domains. They find that using the planer to guide the weak policy significantly improves the success rate.\n\n$\\textbf{Clarity: }$\nAlthough the math can be a tad difficult to get through in places, I found it easier to follow than many other DICE-based papers. Broadly speaking, the core ideas are clear and well-articulated. \n\n$\\textbf{Significance: }$\nThe proposed method solves the bias problem in the offline setting and provides valuable convergence guarantees. Additionally it provides good experimental results and proposes an interesting new approach to zero shot cross-domain transfer. The one downside is that it's a bit niche, by virtue of sitting at the intersection of two subfields, goal conditioned RL and offline RL. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3xuCF7FNXGY", "reviewer": "Reviewer_gkGs", "summary": "This paper proposes a new regression-based offline GCRL algorithm from the perspective of state-occupancy matching. Compared to previous methods, GoFAR does not require any hindsight relabeling. GoFAR also decouples the training process of value and policy updating. These features make GoFAR enjoy the benefits of better performance, training stability, and zero-shot transfer to new target domains.  ", "questions": "1. Regarding the first concern listed above, why we can choose $r(s; g) = p(s; g)$ in the general offline RL setting? If not, can we still cast offline goal-conditioned RL as a state-occupancy matching problem?\n2. Can Proposition 1 be derived from the aspect of variational inference? It seems that Proposition 1 can be derived from variational inference with fewer lemmas and assumptions. \n3. Rewards signal: What’s the difference between discriminator-based reward and dataset-based reward? Could you add some ablations experiments using these two types of reward in the model?\n", "limitations": "The main limitation of this work is stated in my concerns regarding the reformulation of offline goal-conditioned RL as an imitation learning problem. If the authors can provide convincing responses to the above questions (especially Question 1), I’d like to raise my score. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The biggest contribution of this paper is casting offline goal-conditioned RL as a state-occupancy matching problem. After this formulation, GoFAR reuses techniques from existing offline imitation work [1][2] (see below) to solve the state-occupancy matching problem by regarding the data that accomplish the desired goal as expert data, and the whole offline data as supplemental data.\nHowever, I do have several concerns about this reformulation of offline goal-conditioned RL.\n\n1. The objective of goal-conditioned RL is to find a policy that maximizes the cumulative reward, while state-occupancy matching is only mimicking the expert. Suppose: 1) the reward is not sparse reward (not the case that only gets 1 reward when accomplishing the goal) and 2) the dataset contains successful but not optimal trajectories that accomplish the goal g (i.e., the cumulative reward is low). In this case, mimicking these trajectories will only get suboptimal policies.\n2. In Appendix E.1, the authors give a theoretical connection between Goal-Conditioned State-Matching and Probabilistic GCRL, however, this builds on the premise that we can choose $r(s; g) = p(s; g)$. Why is it a valid choice under RL (i.e., have the freedom to choose $r$ as $p$)? It is reasonable under the imitation learning setting but is problematic under the RL setting. As the reward function is given by the problem and $p(s;g)$ is given by the offline data.\n\nAside from the concerns, the additional strengths and weaknesses of the paper are listed as follows\n\n**Strengths:**\n\n* This paper provides a novel view to solve the offline GCRL problem by adopting state-occupancy matching. It avoids hindsight relabeling, improves training stability, and owns the zero-transfer ability.\n* The authors connect well with probabilistic GCRL and HER.\n* The proposed method performs well on offline GCRL tasks and the authors conducted comprehensive experiments to show the advantage of GoFAR.\n\n**Weaknesses:**\n* The learning objective of GoFAR, Eq. (5), is a lower bound of $KL(d^{\\pi}, p)$, rather than the exact KL divergence, and objective (7) is an even looser lower bound.\n* The benchmark experiments are not sufficient. Could you test your algorithm on the offline GCRL benchmark introduced in [3]?\n\t\n[1] Kim et al, DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations. ICLR, 2022.\n\n[2] Ma et al, Smodice: Versatile offline imitation learning via state occupancy matching. ICML, 2022.\n\n[3] Yang et al, Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL. ICLR, 2022. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept