Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL
Authors: Rui Yang
Yiming Lu
Wenzhe Li
Hao Sun
Meng Fang
Yali Du
Xiu Li
Lei Han
Chongjie Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Goal-conditioned reinforcement learning
offline reinforcement learning
goal-conditioned supervised learning

Abstract: Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL --- optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best-advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.

Published as a conference paper at ICLR 2022
RETHINKING GOAL-CONDITIONED SUPERVISED LEARNING AND ITS CONNECTION TO OFFLINE RL
Rui Yang', Yiming Lu', Wenzhe Li', Hao Sun?, Meng Fang', Yali Du', Xiu Li", Lei Han Chongjie Zhang "Tsinghua University, Universitty of Cambridge, Eindhoven University of Technology, 'King's College London, Teeneett Robotics X
{yangrui19, luym19, 1wz21 femallmmmlsss.... www.ggghaaeuu...uaeae en, h87890cam. ac. uk m. fang@tue nl, yali. yali.du@kcl ac. 1i Li.xiuβsz tsinghuu edu. cn Ixhan@tencent.com chongj eât eitsinghua edu. cn
ABSTRACT
Solving goal-conditioned tasks with sparse rewards using sulf-supe learn- ing is promising because of its simplicity and stability over current reinforcement learning (RI.) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating gelf-generated experiences. In this paper, we revisit the theoretical property of GCSL- optimizing lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The pro- posed method is named Weighted GCSL (WGCSL), in which introduce advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) best- advantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and of- fline settings. To evaluate algorithms in the offline goal-conditioned RL. setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consis- tently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.
INTRODUCTION
Reinforcement learning (RL) enables automatic skill learning and has achieved great success in various tasks (Mnih et al. 2015 Lillicrap et al.; [0016] Haarnoja et al.) 2018 Vinyals et al.) 2019). Recently, goal-conditioned RL is gaining attention from the community, as encourages agents to reach multiple goals and learn general policies (Schaul et al., 2015). Previous advanced works for goal-conditioned RL consist of a variety of approaches based on hindsight experience re- play (Andrychowicz et al.] [017]] Fang et al.) 2019), exploration (Florensa et al. 2018 Ren et al.) P et al. (020), and imitation learning (Sun et al., 2019, Ding et al. [011 Sun et al. [0200] Ghosh et al. 2021). However, these goal-conditioned methods need intense online interaction with the environment, which could be costly and dangerous for real-world applications.
Pitis
A direct solution to learn goal-conditioned policy from offline data is through imitation learning. For instance, goal-conditioned supervised learning (GCSL) (Ghosh et al. 2021 iteratively relabels collected trajectories and imitates them directly. is substantially simple and stable, as it does not quire expert demonstrations or value estimation. Theoretically, GCSL guarantees to optimize over a lower bound of the objective for goal reaching problem. Although hindsight relabeling (Andrychow- icz et al. 2017) with future reached states can be optimal under certain conditions (Eysenbach et al.] 2020). would generate non-optimal experiences in more general offline goal-conditioned RL. set-
"Corresponding Authors

---

Published as a conference paper at ICLR 2022
ting, discussed in Appendix [B.1 As result, GCSL suffers from the same issue other behavior cloning methods by assigning uniform weights for all experiences and results in suboptimal policies.
To this end, leveraging the simplicity and stability of GCSL, we generalize it to the offline goal-conditioned RL setting and propose an effective and theoretically grounded method, named Weighted GCSL (WGCSL). We first revisit the theoretical foundation of GCSL by additionally con- sidering discounted rewards, and this allows us to obtain a weighted supervised learning objective. To learn better policy from the offline dataset and promote learning efficiency, we introduce more general weighting scheme by considering the importance of different relabeling goals and the expected return estimated with value function. Theoretically, the discounted weight for relabeling goals contributes to optimizing a tighter lower bound than GCSL. Based on the discounted weight, we show that additionally re-weighting with an exponential function over advantage value guar- antees monotonic policy improvement for goal-conditioned RL. Moreover, the introduced weights build a natural connection between WGCSL and offline RI., making WGCSL available for both online and offline setting. Another major challenge in goal-conditioned RL is the multi-modality problem (Lynch et al. 2020), i.c., there are generally many valid trajectories from a state to goal, which can present multiple counteracting action labels and even impede learning- To tackle the challenge, we further introduce the best-advantage weight under the general weighting scheme, which contributes the asymptotic performance when the data is multi-modal. Although WGCSL has the cost of learning the value function, we empirically show that it is worth for its remarkable improvement. For the evaluation of offline goal-conditioned RL algorithms, we provide public benchmark and offline datasets including a set of challenging multi-goal manipulation tasks with a robotics arm or an anthropomorphic hand. Experiments conducted in the introduced benchmark show that WGCSL significantly outperforms other tfate-ofthhe-e art baselines in the fully offline goal- conditioned settings, especially in the difficult anthropomorphic hand task and when learning from randomly collected datasets with sparse rewards.
PRELIMINARIES
Markoy Decision Process and offiine RI. 'Te RL problem can be described as a Markov De- cision Process (MDP), denoted by a tuple AS, P, r, 7), where S and A are the state and action spaces; P describes the transition probability as S x A
+ [0, 1]; r: S x A + I is the reward function and E (0, 1] is the discount factor; π S A denotes the policy, and an optimal policy n' satisfies π = arg max, E -n (1-ss))) (>P(-(s,.a)) 1"= r(st, az)], where p(s1) is the distribution of initial states. For offline RL problems, the agent can only access static dataset D, and is not allowed to interact with the environment during the training process. The offline data can be collected by some unknown policies.
Goal-conditioned RL Goal-conditioned RL further considers goal space G. The policy π : S x g - A and the reward function r S x 9 x A - IR are both conditioned on goal g G. The agent learns to maximize the expected discounted cumulative return J(x) = Ei -"=1r(s,,, at,g)] where p(g) is the distribution over goals g. We study goal gond-condi RL with sparse rewards, and the reward function is typically defined as:
Egr-pi)) -p((s))))))))))))))))""))))))) (xc)
r r(st, at g) = (1.
[lx(s;) - 9112 < some threshold
0. otherwise
where Φ: : S -> G is mapping from states to goals. For theoretical analysis in Section
3 we consider
MDPs with discrete state/goal space, such that the reward function can be written as r(st, az,g) g) = 1 1(((s)) g). which provides positive signal only when 8; achieves g. The value function V
S x G + R is defined as V (s) = (aamrani11P(((pss [ (rrrrs:s az, g)ss s].
Goal-conditioned Supervised Learning Different with goal- coal-conditioned RL that maximizes discounted cumulative return, GCSL considers the goal-reaching problem, i.e.. maximizing the last-step reward (yy<p())) [r(ST, ar,g)] for trajectories of horizon T. GCSL iterates be- tween two process, relabeling and imitating. Trajectories in the replay buffer are relabeled with
Code and offline dataset are available at https://ggthaaa.ooo/////guuuuuus
2

---

Published as a conference paper at ICLR 2022
Of
Bibabe
Value Function Approtimation
Weighted seper Loening
Ri 0.g
Seu 0,g" - (lG))
Fa U.S
P2 1. =
min (1/ ro ).)))))))))))))))))))))))))))))) (04
4g 0.g
Sample
1y 0,g 1622
R.00 m 0,5
N, 1,9 = pkloy)
mas.F (1 Sareer/rllggollee logng(ss14. expeup (4(4, (6.1)) =200)
Figure I: Diagram of WGCSL in the offline goal-condi conditind RL setting, WGCSL samples data from the offline dataset and relabels them with hindsight goals. After that, WGCSL learns a value function and updates the policy via a weighted supervised scheme.
hindsight methods (Kaelbling 1993
Andrychowicz et al. 2017 to form a relabeled dataset
Drelabel {(St, at. as. where ((s)) for 2 denotes the relabeled goal. The policy is optimized through supervised learning to mimic those relabeled transitions: arg max, E ""laac"""""Petttt log ( s'.
WEIGHTED GOAL-CONDITIONED SUPERVISED LEARNING
In this section, we will revisit GCSL and introduce the general weighting scheme which generalizes GCSL to offline goal-conditioned RL.
3.1 REVISITING GOAL-CONDITIONED SUPERVISED LEARNING
As an imitation learning method, GCSL sticks the agent's policy the relabeled data distribu- tion, therefore it naturally alleviates the problem of out-of-distribution actions. Besides, it has the potential to reach any goal in the offline dataset with hindsight relabeling and the gemeralization ability of neural networks. Despite its advantages, GCSL has major disadvantage for offline goal- conditioned RI, i.e., it only considers the last step reward r(ST, ar, g) and generally results in suboptimal policies.
A Motivating Example We provide training results of GCSL
PointReach task Figure 2 The ob- jective of the task is to move point from the starting position to the desired goal as quickly as possible. The offline dataset is collected using a random policy. As shown in Figure 2 GCSL learns a suboptimal policy which detours to reach goals. This is because GCSL only considers the last step reward r(St, .r, g). As long the trajectories reaching the goal at the end of the episode, they are all considered equally to GCSL. To improve the learned policy, straightforward way is to evaluate the importance of samples or trajecto- ries for policy learning using importance weights. comparison, Weighted GCSL (WGCSL), which uses a novel weighting scheme, learns the optimal policy in the PointReach task. We will derive the formulation of WGCSL the following analysis.
Achined Geal of WOCOL
Act and Seen WCCC
Frstton
Achevve Gan leGco.
Acteree Guak elsics.
Sharing Poutton
Guating Noello
Oveidd
Figure 2: Visualization of the trajectories generated by GCSL (blue) and WGCSL (orange) in the PointReach task.
Connection between Coal-Conditioned RL and SL
GCSL has been proved to optimize a lower bound
the goal-reaching objective (Ghosh et al. 2021) which
different from the goal-conditioned RL objective that optimizes the cumulative return. In this sec- tion, we will introduce a surrogate function for goal-conditioned RL and then derive the connection between goal-conditioned RI. and weighted goal-conditioned supervised learning (WGCSL).. For
3

---

Published as a conference paper at ICLR 2022
overall consistency, provide the formulation of WGCSL as below:
JwGCsL(T) [gpp
[rr log no (ar ((s)))},
(I)
where p(g) is the distribution of desired goals and refers to the data collecting policy. The weight we, is subscripted by both the time step & and the index i of the relabeled goal. When Wt, 1. JWGCSL reduces to GCSL (Ghosh et al. 2021). and we denote the corresponding objective as JGCSL for comparison convenience. Note that in Eq. ((s,), implies that any future visited states after can be relabeled as goals.
Theorem 1. Assume a finite-horizon discrete MDP, stochastic discrete policy x which selects
actions with non-zero probability and a sparse reward function r(st, at,g) = 1[0(s)) = g), where the state-to-goal mapping and 1[0(st) g] is an indicator function. Given trajectories T factor ($1, ",,'.., ST, ar) and factor E (0, 1), let the weight We,i = - ft [1. E [t. T]. then the following bounds hold:
Jourr (T) 2 T Jwacsz JoCSL(TT)
where (sur((T) = -1 E log ((aess) 9) Et= 7 1(0(s:) - =a] is a surro-
gate function of J(x)).
We defer the proof to Appendix B.2 where we also show that under mild conditions, (1) Jsurr a lower bound of log J, and (2) Jourr shares the same gradient direction with J at Ty. Theorem D reveals the connection between the goal-conditioned RL objective and the WGCSL/GCSL ob- jective. Meanwhile, it suggests that GCSL with the discount weight y- is a tighter lower bound compared to the unweighted version. We name this weight as discounted relabeling weight (DRW) as it intuitively assigns smaller weights on longer trajectories reaching the same relabeled goal.
3.2 A MORE GENERAL WEIGHTTNGG SCHEME
DRW can be viewed as a weight evaluating the importance of relabeled goals. In this subsection, consider more general weighting scheme using a weight function measuring the importance of state-action-goal combination, revealed by the following corollary.
Corollary 1. Suppose function h(ss, a, g) 2 1 over the state-action-goal combination. Given tra- jectory т = ($1, 01, 88, 0.),, let Wi,i = 7² (1=h(s ar ((si)), E [1,T], E [t, then the following bound holds:
Juurr (T) 2 T (WGCSL(T).
The proof are provided in Appendix B.3 Naturally, the Q-value in RL with some constant shift is candidate of the function family h a, g). Inspired by prior offline RL approaches (Wang et al. 2018 Peng et al., 22119] Nair et al., 2020), we choose h[s, a, g) = exp(A(s, a, g) + C), an exponen- tial function over the goal-conditioned advantage, which we refer as goal-conditioned exponential advantage weight (GEAW). The constant is used to ensure that h(ss, a. g) 2 1. The intuition is that GEAW assigns larger weight for samples with higher values. In addition, the exponential ad- vantage weighted formulation has been demonstrated to be a closed-form solution of an offline RI. problem, where the learned policy is constrained stay close the behavior policy (Wang al.) 2018). Compared to DRW, GEAW evaluates the importance of state-action-go samples, taking advantage of the universal value function (Schaul et al. 2015)..
To tackle the multi-modality challenge in goal-conditioned RL (Lynch et al. 2020), we further introduce the best-advantage weight (BAW) based on the learned value function. BAW has the following form:
(1, A(st, at, ((s:)) > À
c(A(s,, at, ((s:)))
Eminy otherwise
where À is a threshold, and 6min a small positive value. Implementation details of BAW can be our implementation, A gradually increases. Therefore, BAW gradually leads
found in Section the policy toward the modal with the highest return rather than sticks to position between multiple modals, especially when fitting using a Gaussian policy. Combining all three introduced weights, we have the weighting form: Wt - " (xp(A(ss, (st, at, o(s:)) + C)- e(A(s,, a, (()))). The constant C' only introduces a coefficient independent t, for all transitions, therefore, we simply omit it for further analysis.
4

---

Published as a conference paper at ICLR 2022
3.3 POLICY IMPROVEMENT VIA WGCSL
We formally show that combining the three introduced weights, WGCSL can consistently improve the policy learned from the offline dataset. First, we assume there exists policy Trelabel that can produce the relabeled experiences Dretabel- GCSL essentially equivalent to imitating Trelabed:
arg ming ( [o) arg max, B.Dretunct [lo me (a. St, d(s))))]
Following (Wang et al. 2018), solving WGCSL is equivalent to imitating new goal-conditioned policy ग defined as:
[ ((s;)) = (a o(s;)) exp(A(ss. ae d(s))) at,
Then, we have the following proposition:
Proposition 1. Under certain conditions, it is uniformly as good as better than Trelabel- That is, Vst, ((s.) E Drelabel. 2 t, V (s,, 2 V (st, ((s)), where Drelabel contains the experiences after goal relabeling.
The proof can be found in Appendix B.4 formly non-worse policy using the relabeled data Delabell. To be rigorous, we also need to discuss the relationship between the relabeled return and the original return. In fact, there is a monotonic improvement for the relabeled return over the original one when relabeling with inverse RL (Ey- senbach et al. (2220). Below, we provide a simpler relabeling strategy which also offers monotonic value improvement.
Proposition 1 Timplies that WGCSL can generate a uni-
Proposition 2. Assume the state space be perfectly mapped the goal space, Vst E S, o(s)) G, and the datasel has sufficient coverage. We define a relabeling strategy for (8t, 9) as
g -'= ( (d(()),) Utt, if ((s)) + g, vi 2t,
otherwise.
We also assume that when g' + g. the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g' in the original dataset D. Then, Reelabel is uniformly as good as or better than Tb, i.e., V8,, 9, D.V.Da. Vaealball (st, 2 (**( g), where my is the behavior policy forming the dataset D.
The proof is provided in Appendix B.5 In Proposition 2 the relabeling strategy is slightly different from the strategy of relabeling with random future states (Andrychowicze. et al. 2017). When there is no successful future states, the relabeling strategy in Proposition randomly samples from the future visited states for relabeling, otherwise it keeps the original goal. For random datasets, there are rare successful states and the relabeling strategy in Proposition / acts similarly to relabeling with ran- dom future states. Combining Propositions] [ [ we know that under certain assumptions, after performing relabeling and weighted supervised learning, the policy can be monotonically improved. In fully offline settings, WGCSL is able to learn a better policy than the policy learned by GCSL and the behavior policy generating the offline dataset.
ALGORITHM
In this section, we summarize our proposed algorithm. Denote the relabeled dataset as Dratabet = (st, ('.r'''g), [1, = <(s:), r/ r(st, a, ((s:)), 2 I, and we maximize the following WGCSL objective based on the relabeled data
(w Ess (l.) wyy log ma (ar|s,, ((+:)),,
(3)
where the weight wi, is composed of three parts conforming to the general weighting scheme
W > 1-t exPetip (A(s), ae o(s,)) c(A(st, a, ((s))))
(1) 1- is the discounted relabeling weight (DRW) as introduced in Section 3.1
(2) (A a₂, ((s)))) = clip(cxp(A(s) (A(t, a, 0, is the goal-conditioned exponen- tial advantage weight (GEAW) with a clipped range (0, MM. where M is some positive number for numerical stability. In experiments, M is set as 10 for all the tasks.
5

---

Published as a conference paper at ICLR 2022
(a)
(b)
(c)
(d)
(c)
(f)
g
(h)
(i)
Figure 3: Goal-conditioned tasks: (a) PointReach, (b) PointRooms, (c) Reacher, (d) SawyerReach, (e) SawyerDoor, (f) FetchReach, (g) FetchPush, (h) FetchSlide, (i) FetchPick, (i) HandReach.
(3) ((A(st, de, ((s;))) is the best-advantage weight (BAW) with the form in Eq. 2 In experi- ments, À N percentile of recent advantage values and emin set 0.05. Motivated by curriculum learning (Bengio et al. 2009). N gradually increases from 0 to 80, i.c., we learn from all the experiences the beginning stage and converge to learning from samples with the top 20% advantage values.
The advantage value in Eq. 3 can be estimated by A(s,, at ag.g') r(s ae. ae.g') + -V(Ss+,))) V V8s:gg) and we learn a Q-value function ((+t. a',g') by minimizing the TD error
LTD as r(s 1(1 (((55 (((+))9)). Q(si, ar, I'
where relabel original goals 9 as g' (s,) with probability of 80% and remain the original goal unchanged for the rest 20% time. For simplicity, in our experiments, once it falls into the 80% probability of goal relabeling. there exist successful future states in the trajectory or not, to avoid parsing all the future states as
use purely relabeling with random future states, matter whether
discussed in Appendix B.5 Q refers to the target network which is slowly updated to stabilize training. The same value function training method is adopted in (Andrychowicz et al.) [2017 Fang iet al. 2019). The value function V(s,,g) is estimated using Q-value function as V(st. (st. (((+++1')) g'), The overall framework of WGCSL is presented in Figure and the entire algorithm is provided in Appendix C
5 EXPERIMENTS
In our experiments, we demonstrate our proposed method on a diverse collection of continuous sparse-reward coal-conditioned tasks.
5.1 ENVIRONMENTS AND EXPERIMENTAL SETTINGS
As shown in Figure the introduced benchmark includes two point-based environments, and eight simulated robot environments. all tasks, the rewards are sparse and binary: the agent receives a reward of +1 if it achieves desired goal and reward of 0 otherwise. More details about those environments are provided in Appendix[F] For the evaluation of offline goal-condition
algorithms, collect two types of offline dataset, namely 'random' and 'expert', following prior offline RL work (Fu et al. 2020). Each dataset has the sample size of x 106 for hard tasks (FerchPush, FetchSlide, FetchPick and HandReach) and 1 x 10 for other relatively easy tasks. The 'random' dataset is collected by a uniform random policy, while the 'expert' dataset is collected by the final policy trained using online HER. We also add Gaussian noise with zero mean and 0.2 standard deviation to increase the diversity of the 'expert' dataset. This may lead to the consequence that the deterministic policy learned by behavior cloning achieves a higher average return than that in the dataset, which has also been observed in previous works (Kumar et al., 2019 Fujimoto al.] 2019). More details about the two datasets are provided Appendix [ Beyond the offline setting, we also evaluate our method in the online setting and the results are reported in Appendix E.8
As for the implementation of GCSL and WGCSL, we utilize the Diagonal Gaussian policy with a mean vector Te and constant variance o² for continuous control. In the offline setting, does not need to be defined explicitly because we evaluate the policy without randomness. The loss function of GCSL can be rewritten as: LCCSL = (s (ao(s,,))-Pettabae [ll (o) ((s:)) a l1 Our implemented WGCSL learns a policy to minimize the following loss
Lwaost. Ee ((s)))).d.)))))))) ((..)) D) [we, ( ((s))) arl13)

---

Published as a conference paper at ICLR 2022
WSCSL
GCSL
Goal MARWIL
Goal BC
Goal BGQ
Goal COL --.
HER -... Averge Reaum of Catase!
FetchPush expert
FetchSiide expert
FetchPick expert
HandReeac expert
30
Averagg Retum
30
00
10
10
20020 Training Steps
20020 Trainig Steps
Training Steps
Training Steps
FaldhP.ah-vaadom
Side-random
FilkePPic
HandRoach-anddom
3.0
2.5
Aeerage 70
2.0
0
10000
20000
20000
40000
10200
20000
30000
40000
10000
20000 Trainig Steps
30000
40000
10000
20000 30000
40000
Training Steps
Training Stops
Training Stops
Figure 4: Performance expert (top row) and random (bottom row) offline dataset of four simulated manipulation tasks. Results are averaged over random seeds and the shaded region represents the standard deviation.
Table I: Average return of algorithms on the offline goal-conditioned benchmark. g-MARWIL, g- BC, g-BCQ and g-CQL, are short for Goal MARWIL, Goal BC, Goal BCQ, Goal CQL, respectively. The suffix "-e" and "-r"" are short for expert and random datasets, respectively.
Task Name
WGCSL
GCSL
g-MARWIL
g-BC
g-BCQ
g-CQL
HER
PointReach-e
44.40 10.11
39.27
10.8
42.95 10.15
39.36
1048
40.42 L057
39.75 1033
24.68 L3.07
PointRooms-e
36.15 =085
33.05
-0.54
36.02 #0:57
33.17 #052
32.37
#1.77
30.05
=038
12.41 #8.59
Reacher-e
40.57 -0000
36.42
20.30
38.89 10
35.72
20.77
39.57
=008
42,23
=012
8.27 24.33
SawyerReach-e
40.12 =0.29
33.65
=0.38
37.42 #0.31
32,91 #031
39.49
#0.33
19.33
=0.45
26.48
=6623
SawyerDoor-e
42.81 =:1.3
35.67
=0.00
40.03 #0.16
35.03
=0220
40.13
=0075
45,86 =0111
44.09
=0065
FetchReach-e
46.33 =0004
41.72
=0.31
45.01 10.11
42.03
=025
35.18
=3.09
1.03 =0.26
46.73
=00144
FetchPush-e
39.11 =:0117
28.56
0.99
37.42
1022
31.56
1001
3.62 1099
5.76 +0.83
31.53 1:047
FetchSlide-e
10.73 11.00
3.05
#0.42
4.55 11.79
0.84 #0.35
0.12
=0.30
0.86 =00388
2,86 #:2.40
FetchPick-e
34.37 10.5t
25.22
10.85
34,56 10.54
31.75 11.19
1.46
1.0.9
3.23
12.52
24.79 14.49
HendReach-e
26.73 _1.20
0,57
10.88
0.81 11.59
0.06 10.00
0.04
10.04
0.00
10.00
0.05 .0.00
PointReach-r
44.30
1021
30,80
11.4
7.67 11.99
1.37 :1009
1.78
100.4
1.52
110.6
45.17 1:0.33
PointRooms-r
35.52 =0.00
24.10
00.81
4.67 +:0.00
1.43
=:.188
1.61
=:0.17
1.29
=:0.37
36.16 1:1.16
Reacher-r
41.12 =0.11
22.52
=0.77
15.35 #1.95
1.66 =0.30
2.52
=0028
2.54
#0.17
34, #88.2
SawyerReach-r
41.05
_0.19
14.86
13.27
11,30 12.12
0.58 1121
1,36 100.4
1.18
.00.9
39.27 1.216
SawyerDoor-r
36.82
_3320
25.86
11:12
25.33
11:46
3.73 1083
9.82 11.08
4,36
10.86
28. 11.9
FetchReach-rr
46.50 =0.00
38.26
24.24
30.86 18.40
0.84 20311
0.19
2004
0.97
20.33
47.01 -007
FetchPush-r
11.48 =1.00
5.01 10.64
3.01 10.71
3.14 2025
3.60
2042
3.67 20:5
5.65 20664
FetchSlide-r
2.34 +0.25
1.47 #0.12
0.19 +0.11
0.25
=0.33
-0.29
0.15 +0.00
0.59 +0.30
FetchPick-r
8.06 +0.71
3.05 +0.42
2.01 +0.46
1.84
=0.17
=0.58
1.73
=0.17
3.91 11.29
HandReach-r
5.23 +0.55
3,96 #0.81
0.00
+0.00
0.00
=000
0.00
=000
0.00
=0000
0.00 =0.01
where w defined in Eq 4 The policy networks of GCSL and WGCSL are 3-layer MLP with 256 units each layer and relu activation. Besides, WGCSL learns a value network with the same structure except for the input and output layers. The batch size is 512 for 4 harder tasks from FetchPush to HandReach. We use Adam optimizer with learning rate of 14-4, For all the experiments we repeat for different random seeds and report the average evaluation performance with standard deviation.
128 for the first 6 tasks, and
5.2 EXPERIMENTAL RESULTS
In the offline goal-conditioned setting, we compare WGCSL with GCSL, HER and modi- fied versions of most related offline works such goal-conditioned MARWIL. (Wang al.)

---

Published as a conference paper at ICLR 2022
WGGSL
GCSL + BAW
GCSL + GEAI
GCSL DAW
GCSL
PoimiReacch-raddom
FelchhResc-randdm
SawyerFoschreapen
Hand Reachheecpeee
45
40
Avarage Ruturn
35
-0
20
30
25
200
2500
500
5000
30000
Traiting Steps
Training Stepe
Triining Steps
Triining Stups
Figure Ablation studies of WGCSL in the offline setting.
2018)
(Goal MARWIL), goal-conditioned behavior cloning (Bain & Sammut 1995 (Goal BC), goal-conditioned Batch-Constrained Q-Learming (Fujimoto al.) 2019) (Goal BCQ), and goal- conditioned Conservative Q-learning (Kumar et al. [220]] (Goal CQL). The results are presented in Table and Figure Additional comparison results with more baselines and results on 10% dataset provided in Appendix E
Expert Datasets The top row of Figure reports the performance of different algorithms using the expert datasets of four hard tasks. In all these tasks, WGCSL outperforms other baselines in terms of both learning efficiency and final performance. GCSL converges very slowly but can finally reach beyond the average return of expert dataset in FetchSlide and FetchPick tasks. Goal MARWIL is more efficient than GCSL and achieves the second best performance. However, none of the baseline algorithms succeeds in the HandReach task, which might be due to high-dimensional states, goals, actions and the noise used to collect the dataset. In addition, HER itself has the ability handle offline dataset as the relabeled data is far off-policy (Plappert et al., 2018). But we can conclude from Table [ that the performance of HER is unstable and inconsistent across different tasks. In contrast, learning curves of WGCSL are consistently stable. It is also verified that techniques used in WGCSL are effective to generate better policies than original and relabeled datasets offline goal-conditioned RL setting.
Random Datasets The bottom row of Figure Shows the training results of different methods on random datasets. We can observe that WGCSL outperforms other baselines by large margin. Base- lines such as Goal MARWIL, Goal BC, Goal BCQ and Goal CQL can hardly learn to reach goals with the random datasets. GCSL and HER can learn relatively good policies in some tasks, which proves the importance of goal relabeling. Interestingly, in tasks such as Reacher, SawyerReach and FetchReach, results of WGCSL and HER on the random dataset exceeds that of the expert dataset. The possible reason is that the data coverage of random dataset could be larger than the expert dataset these tasks, and the agent can learn to reach more goals goal relabeling.
Ablation Studies We also include ablation stud- ies for offline experiments in Figure In our abla- tion studies, we investigate the effect of the three proposed weights, i.c., DRW, GEAW and BAW, by applying them to GCSL respectively. The re- sults demonstrate that all three weights are effec- tive compared to the plain GCSL, while GEAW and BAW show larger benefits. This is reasonable since GEAW and BAW learn additional value net- works while DRW does not. Moreover, the results suggest that the learned policy can be improved by combining these weights.
HardfReach aesort
FurdPReach-eapet
WaOS COPS
Trannin Stape
Traiiin Stops
Figure 6: Average estimated value (left) and performance (right) of WGCSL, HER and DDPG. The TD target is clipped to [o, 50), thus the maximum estimated value is around 50.
Value Estimation Off-policy RL algorithms are prone to distributional shift and suffer from value overestimation problem on with -igh-dimensional states and actions. As shown in Figure 6 DDPG and HER exhibit large estimated values during offline training. In contrast, WGCSL has a more robust value approximation through weighted supervised learning, thus achieving higher returns in the difficult HandReach task.
actions. Such problem can be exacerbated in tasks
8

---

Published as a conference paper at ICLR 2022
6 RELATED WORK
Solving goal-conditioned RL task is an important yet challenging domain in reinforcement learning. In those tasks, agents are required achieve multiple goals rather than a single task (Schaul al., 2013). Such setting puts much burden on the generalization ability of the learned policy when deal- ing with unseen goals. Another challenge arises conlicond RL is the sparse reward (Plap- pert et al. 2018). HER (Andrychowicz et al.) (017) tackles the sparse reward issue via relabeling the failed rollouts as successful ones to increase the proportion of successful trails during learning. HER is then extended to deal with demonstration (Nair et al.) 2018: Ding et al. 2019) and dy- namic goals (Fang et al. (2018) and combined with curriculum learning for trade-off (Fang et al., 2019). Eysenbach et al. (2020) and Li et al. (2020) further improved the goal sampling strategy from the perspective of inverse RL.. Zhao et al. (2021) encouraged agents
to control their states to reach goals by optimizing mutual information objective without external rewards. Different from prior works, we consider the offline setting, where the agent cannot access the environment collect data in the training phase.
Our work has strong connections to imitation learning (IL). Behavior cloning is one of the simplest IL methods that learns policy predicting the expert actions (Bain & Sammut [995 [Bojarkii al. 2016). Inverse RL (gg et al.) (2000) extracts a reward function from demonstrations and then uses the reward function to train policies. Several works also leverage self-imitation improve the stability and efficiency of RL without expert experience. Self-imitation learning (Oh et al. 2018) imitates the agent's 's past advantageous decisions and indirectly drives exploration, |Sun al. (2019) introduced a dynamic programming method to learn policies progressively via self-imitation. Lynch et al. (2020)
employed VAE to jointly learn latent plan representation and goal-conditioned policy from replay without the need for rewards. Our work is most relevant GCSL (Ghosh et al., 2021), which iteratively relabels and imitates its own collected experiences. The difference is that we further generalize GCSL to the offline goal-conditioned RL setting via weighted supervised learning.
Our work belongs to the scope of offline RL. Learning policies from static offline datasets is chal- lenging for general off off-policy RL algorithms due to error accumulation with distributional shift (Fu- jimoto et al. 2019) Kumar et al., 2019). To address such issue, a series of algorithms are proposed constrain the policy updates to avoid excessive deviation from the data distribution (Wang al.]
2018
Fujimoto et al.] 2019 Wu et al. 2019: Yang et al. 2021). MARWIL (Wang et 20188) can learn better policy than the behavior policy of the dataset with theoretical guarantees. In addition to policy regularization, other works leverage techniques such as value underestimation (Kumar et al.)
Yu et al., 2020), robust value estimation (Agarwal et al.) 2020), data augmentation (Sinha & Garg 2021; Wang et al., 2021), and Expectile V-Learning (Ma et al. 2021) to handle distribution shift. As offline RL can easily overfit to one task, generalizing offline RL to unseen tasks can be even more challenging Li et al., 2021). Li et al. (2019) tackled the challenge by leveraging the triplet loss for robust task inference. Our method has similar exponential advantage weighted form as MARWIL, however, we tackle goal-conditioned tasks and introduces general weighting scheme for WGCSL. A most recent work (Chebotar et al., (222I) also addresses the offline goal-conditioned problem through underestimating the value of unseen actions. The difference is that WGCSL is weighted supervised learning method without explicitly value underestimation. Therefore, WGCSL. substantially simpler, more stable, and more applicable for real-world tasks.
2020
CONCLUSION
In this paper, we have proposed novel algorithm, Weighted GGoal-Conditioee) Supervised Learn- ing (WGCSL), to solve offline goal-conditioned RL problems with sparse rewards, by first revis- iting GCSL and then generalizing it to offline goal-conditioned RL. We further derive theoretical guarantees to show that WGCSL can generate monotonically improved policies under certain con- ditions. Experiments on an introduced benchmark demonstrate that WGCSL outperforms current offline goal-conditioned approaches by great margin in terms of learning efficiency and convergent performance. For future directions, we are interested in building more stable and efficient offline goal-conditioned RL algorithms based on WGCSL for real-world applications.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was partly supported by the Science and Technology Innovation 2030-Key Project under Grant 20212 in part by Science and Technology Innovation 2030 "New Generation Artificial Intelligence" Major Project (No. 2018AAA0100900 and National Natural Science Foun- dation of China (No. 2021 (325500) Part of this work was done when Rui Yang and Hao Sun worked as interns at Tencent Robotics X Lab.
ETHICS AND REPRODUCIBILITY STATEMENTS
Our work provides a way to learn general policy from offiin data. Learning policy for general purpose one of the fundamental challenges in artificial general intelligence (AGI). The proposed offline goal-conditioned algorithm can provide new insights into training agents with diverse skills, but currently we cannot foresee any negative social impact of our work. The algorithm is eval- uated on simulated robot environments, thus our experiments would not suffer from discrimina- tion/bias/fairness concerns. The details of experimental settings and additional results are also pro- vided in the Appendix. All the implementation code and offline dataset are available in our code link: hutp:///sttubbb.ooom//gguuuuuu/t
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-114. PMLR, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- tion Processing Systems, 34, 2021.
Marcin Andrychowicz. Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-500 5058, 2017.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15. pp. 103-129, 1995.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp- 41-48, 2009.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprint 1rXiv:1604.07311 2016.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. 2021.
Yiming Ding, Carlos Florensa, Mariano Phiclipp, and Pieter Abbeel. Goal-conditioned imitation learning. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 15324-15335. 2019.
Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ R. Salakhutdinov. Rewriting history with inverse RL: hindsight inference for policy improvement. In Advances in Neural Information Processing Systems, 2020.
Meng Fang, Cheng Zhou, Bei Shi, Boqing Gong, Jia Xu, and Tong Zhang, Dher: Hindsight perience replay for dynamic goals. In International Conference on Learning Representatiinss 2018.
10

---

Published as a conference paper at ICLR 2022
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculum-guider hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 2600-12613, 2019.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pp. 1515-1528, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysen- bach, and Sergey Levine. Learning reach goals via iterated supervised learning. Interna- tional Conference on Learning Representations, 2021. URL https:: nett/
6yNJ
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290. 2018.
Leslie Pack Kaelbling. Learning achieve goals. In International Joint Conference on Artificial Intelligence, pp. 1094-1099, 1993.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. In Learning Motor Skills, pp. 83-117. Springer, 2014.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Sys- tems, pp. 11761-1177,, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. 2020.
Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning.
Advances in Neural Information Processing Systems, 33, 2020.
Jiachen Li, Quan Vuong, Shuang Liu, Minghua Liu, Kamil Ciosek, Henrik Iskov Christensen, and Hao Su. Multi-task batch reinforcement learning with metric learning. arXiv e-prints, pp. arxiv- 1909, 2019.
Lanqing Li, Rui Yang, and Dijun Luo. FOCAL: efficient fully-offline meta-reinforrcemet learning via distance metric learning and behavior regularization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on Robot Learning, pp. 1113- 1132. PMLR, 2020.
Xiaoteng Ma, Yiqin Yang, Hao Hu, Qihan Liu, Jun Yang, Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint arXiv:2110.09796, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep rein- forcement learning. Nature, 5 518(7540):529, 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforeement learning with demonstrations. In 2018 IEEE International Conference Robotics and Automation, pp. 2299. IEFF, 2018.
11

---

Published as a conference paper at ICLR 2022
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.093359, 2020.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume I, pp. 2. 2000.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. arXiv preprint 1806:08806..55533 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantago-weighted regression: Simple and scalable pollcy reinforcement learning. arXiv preprint arXiv:1910.00117,, 2019.
Silviu Pitís, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761, 2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob Megrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, and Peter Welinder. Multi-goal reinforcement learning: Challenging robotics environments and request for research. 2018.
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. Advances in Neural Information Processing Systems, 32: 1248555--3496, 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pp. 1312-1320, 2015.
Samarth Sinha and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforce- ment learning. arXiv preprint arXiv:2 2021.
Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hind- sight inverse dynamics. In Advances in Neural Information Processing Systems, pp. 10265- 10275, 2019.
Hao Sun, Xinyu Pan, Bo Dai, Dahua Lin, and Bolei Zhou. Evolutionary stochastic policy distillation. arXiv preprint 2rXhv:2004.12909 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
Jianhao Wang. Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Neural Information Processing Systems, 34, 2021.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Informatio Processing Systems, pp- 6288 6297, 2018
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 2019.
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. arXiv preprint arXiv: 00400000000 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-141422 2020.
Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information- state- control for intrinsically motivated reinforcement learning. In International Conference on Learn- ing Representations, 2021.
12

---

Published as a conference paper at ICLR 2022
A A COMPREHENSIVE SUMMARY FOR WGCSL
WGCSL is strongly efficient in the offline goal-conditioned RL due the following advantages:
(1) WGCSL learns a universal value function (Schaul et al. 2015) which is more generalizable than the value function learned in single task, thus alleviating the overfitting problem in offline RL.
(2) WGCSL utilizes goal relabeling which contributes to augment huge amount of data (²) times) for offline training and meanwhile alleviates the sparse reward issue.
(3) WGCSL leverages weighted supervised learning to reweight the action distribution in the dataset, avoiding the out-of-distribution action problem in offline RI.
In the three weights, GEAW keeps the learned policy close to the offline dataset implicitly and BAW handles the multi-modality problem in offline goal-conditioned RL. WGCSL utilizes the three weights to learn a better policy with theoretical guarantees.
B THEORETICAL RESULTS
In our analysis, we will assume finite-horizon MDPs with discrete state and goal spaces.
B. NON-OPTIMALITY RELABELING STRATEGY
Given a trajectory T {$1, aq...., Sy, ). we show that under general offline goal-conditioned RL setting with the sparse reward function r(st, ax) 1 g]. relabeling with future states is not optimal. This a direct result from Eq. of (Eysenbach et al. 2020), which has proven that the optimal relabelling strategy is
((]T)) x p(g) exp Σ 1 r(s a:) log
((g)) p(g) exp
t-1 l[0(st) -gl log Z(g)).
This is different from simply relabeling with the last state as 7(g/T) 1[g ST] in Section 4.1 of (Eysenbach et al. 2020) and relabeling with future states as ((g]s;, at) ( 1[g = st t
B.2 PROOF OF THEOREM I
Theorem. Assume finite-horizon discrete MDP, stochastic discrete policy which selects ac- tions with non-zero probability and a sparse reward function r(st, a,,g) g) 1[0(s:) = g]. where à is the state-to-goal mapping and 1[p(s:) g] is indicator function. Given trajectory ($), a,.... SS. ST) and discount factor ? E (0, 1], let the weight We,; = = [1,T], [t, factoun
then the following bounds hold:
Jaurr (T) 2 T JwGCSL(T) 2 T (GCsL(
where
1
Jour (Tt) T Barp(a)
Σ log n (aclst,))
Σ 1-1 1[((s:)
i=t
surrogate function of ((T).
13

---

Published as a conference paper at ICLR 2022
Proof.
T log (os g)
1 T
Jsurr
E gopp))))
1 1[0(s;) = g
i=t
1
E gg
log (agl88) 9)
Σ
1 =al
T
log (als, 1(0(8,) - 99
2 T (ygrp()) (1)) [t,2]] [ i- log Tog st, o(s))) 1(((8)) = 9] (s:)) - T JWGCSL(T) (5) T (ggp(()) (1) [log (ag (s:)) & T JCCSL(T).
T Bgwp(o)) (1))) lo log
holds because of the non-positive logarithmic function function used in Appendix B.1 (Ghosh et al. (202)). Jaurr (T) further considers the discounted cu- mulative return. The theorem shows that WGCSL optimizss compared to GCSL.
Eq.
Compared to the surrogate
tighter lower bound of Jaure (T))
Remark. While Theorem bounds the surrogate function Jourr with JwacSl and JoCSL, here we bridge the gap between Juur and the coal-conditioned RL objective
We first show that log J is an upper bound of J in addition to KL divergence and constant. Let R(T) شس= - -'-1__1_____010 [(((;) g] and C(g) f (() and assume C(g)) 0. Since practically is sampled from offline dataset, we can find Cm min, C(g). Similarly to (Kober & Peterss 20144) we have following results using Jensen's inequality and absorbing constants into
"constant':
log J(x) = log Ex-p(n) / π T(Tl))
E Egrpio) C(g) /
To(T|g) (000--((l)))))))) (TT))
C(g) nb(T]g)
Popploil / (T(T)))((T))
(log (T(g)
log T (T(TT)))) constant
C(g)
2 E
log (F(g)
1-1 1[((s,) gll + constant
C(g)
T
T
2 Cm
log Taal/81, g)
Σ
1-1 1[((8)) g]] + constant
T
-Jurr (TT) + (gg=p(9).
log (ggT(al Σ
1[0(s;) = 9]] constant
Cm
Cm
T
T
Jeurr (T)
EE
Ey (
log (ac 9) constant
Cm
Cm
T
T
J Juurr (T)
(o (a, st.g)) + constant
Om
Cm
T.
2
- -Juur (
(a, g) logno(a,)s (t.))) constant
Om
Cm
T Jeurr (T))
T
BBgg (1) + constant.
C.
Cm
Naturally, in offline setting, explicitly or implicitly bound the KL divergence between T. and via policy constraints or (weighted) supervised learning. Therefore, we can regard Jeury as a lower bound of J.
14

---

Published as a conference paper at ICLR 2022
Furthermore, We show that Jsurr (T) shares the same gradient with J(t) when my after scaling with T.
((T) Bgrppl))rrattl Σx (10(s)) =ss
VT(TT) = Byrp(s)
Vao(Tlg)
1-1 l[((s;) = gldt
1. (b((l9)
T (T
Σ
i-1 [[((s;) = d
= pg
T
V Voog
Σ xt_ ((6(s))
T
Vlog nb (
81,
Σ 2-- 1[6(8;) - gl
= (Tb)..
Therefore, taking gradient update with proper step size on Jearr can also improve the policy on the objective J. In our algorithm, the exponential weight poses an implicit constraint on the policy update similar (Wang al.) 2018 Nair et al. 2020). With proper constraint, optimizing on the Jeurr equivalent as optimizing on the goal-conditioned objective J. Intuitively, Jaurr maximizes the likelihood for sttt-acctio pairs with large returns, and therefore it can lead to policy improvement based on the offline dataset.
B.3 PROOF OF COROLLARY
Corollary. Suppose function h(s, a, g) 2 over the state-action-goal combination. Given tra-
jectory ($1, 01,-.- ,8,, aT), let Wi,i 7² hh=,, ar ((s.)), [1,T], [T, then the following bounds hold:
Juury n
JwCCsI(T)
Proof. Similarly to the proof of Theorem
we have
T
(eurr(T) T B -egpl)))
log (a (1.
Σ
1-1 ([b(s)) =
i=t
T
2 E -ggplll)
log ( (a st. g)
Σ
7 h ar o(s,)) l[((s;)
T
i=t
T
hn
h((s, au o(s.) log o(s:))
A T JwGcsL (T).
B.4 PROOF OF PROPOSITION
Proposition. Under certain conditions, म is uniformly as good or better than Trelahel- That is, Vst, o(s;) E Dretabel and > f, (*t,,, ((s))) 2 Viretabel (3t, ②(8:)), where Dretobet contains the experiences after goal relabeling.
Proof. We assume the advantage value for the behavior policy of the relabeled data, i.c., Trelabel- We consider the compound state å = (st, d(st)) from the relabeled distribution Dretabel. When 8 is fixed (the index of i, t are fixed), -i is constant and we have:
f(8, A(8, az)) clip( A(6, at). -cc, log M) + log(>"=") log(e(A($,, C(()
15

---

Published as a conference paper at ICLR 2022
f(.-) is monotonically non-decreasing function for fixed 8, where C(s) is normalizing factor, M is the positive maximum exponential advantage value, and ((A(3,00))))) az)) is also a non-decreasing function introduced in Section 3.2 with minimum Emin > 0. Leveraging the Proposition 1 in MAR-
WIL (Wang et al. 2018), we can conclude that ($) V (8).
Remark. The above proposition assumes the advantage value is for Rrelabel. However, em- pirically find that using advantage value of the learned policy π (the value function is learned to minimize Ltp in Section 4) works better in offline goal-conditioned RL setting. As analyzed in prior works (Nair et al.] 2020 Yang et al. 2021), using the value function of x is equivalent to solving the following problem:
T arg max (g) AA (s, a, 9)], s.t.
()) (-)s, S
In addition, for offline goal-conditioned RL, learning policies with inter-trajectory information is important for reaching diverse goals. Therefore, using Q(5++1, ( to compute TD target helps value estimation across trajectories compared with Q Trelabel (8 g'). Moreover, ı is trained via weighted supervised learning to avoid the OOD action problem.
B.5 PROOF OF PROPOSITION
Proposition. Assume the state space can be perfectly mapped the goal space, Vs, S, ((8)) G, and the datasel has sufficient coverage. We define a relabeling strategy for (st, as
g' -'-1 { Ul, if ((()) 9. otherwise.
We also assume that when g' # the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g' in the original dataset D. Then, Trelabel is uniformly as good as or better than nu, i.e., Vss, 9, D, V ($1,g) 2 (**(st,)) where is the behavior policy forming the dataset D.
We assume that the data of D and Dretabet are sufficient and cover the full state and goal space, i.e., U (si,9)) S x G. Note that every trajectory has only a single goal, and does not have to reach the goal within the trajectory. Therefore, we can partition D into two subsets DP (positive set) and D (negative set):
Proof. Let = (T, (si, ai 1 1=
DP = f = (8i, ai T E T, ((;)) -gsh.
D" = {r = (si, ai 93 1 Vs; M(s:) #g;h:
where DP contains trajectories that can reach the goal and n contains trajectories that cannot reach the goal. Let D(st, g) = {T T (st, as g) begin with } and similarly we define the partition
DP g) = = (st, a;, g) begin with 8t, 3s; E r, ((x)) = g).
D"(, = fr = (si, g) begin with s,, Vs, E T, ((s;) + g}.
Hence, the value F can be written
T
(*b g) = Eyr
16b(8)) = =l] 2 E
1=1
(s 9)
D (st,
E Bew
[R]]
B, By-DD((0)))
((tt, g)1
[ g)l
(8 9)
E Br [T]].
D( g)l
Given + = g) E D, we relabel the future trajectory Tralabel (8is ai "is.aig''2 g' for value
estimation following the strategy
g' (((s;). -,,)), if > (, A
9.
otherwise
16

---

Published as a conference paper at ICLR 2022
and when g' 9. the strategy only accepts Trelabel with higher return than any trajectory in D(s,, Let Dretabel(S1, g) = (Frelabel (8t, at [ [Trelbee begin with 8t } be the relabeled trajectories with g as goal. Hence, the value Vreetebel can be written
Virrllbal (5c, g) LE DrrtaaareDDeaaaaa [R]riitee].
For any Trelabel Drelabel g), we enumerate the situation of its counterpart without relabeling.
1. DP(st, 9). Following the relabeling strategy, have g' g. such that Trelabel
2. T (s g). then T has another goal 9 + g. but it reaches 9 at some intermediate state. It holds that R Rivelabet > 0 = RT Following the relabeling strategy, we further have Rerrlabell 2 R VT' D(st, g). which guarantees improvements expected value of all trajectories starting from St with goal g.
Therefore, we have
VReelabel (st,g) DrrntaharDDatrarae raiaab
(F
[R]]
[s 9)1
E DP [ V (8t:
D(st: g)l
Remark. The relabeling strategy of Proposition 2 requires complete scan of future states of tra- jectories, which is costly. Empirically we find that relabeling with random future states as HER has comparable performance, and is very simple and efficient to implement. We provide the comparison between the relabeling strategy of Proposition 2 (called WGCSL-slow-red and the relabeling strategy that used in practice in Table
Table Average Final Performance Different Relabeling Strategies.
WGCSL
WGCSL-sloo-relabe
PointRooms-exper
36.15 (#0.85)
37.08 (#0.56)
FetchReach-expert
46.33 (#0.04)
46.15 (#0.10)
FetchPush-expert
(+-017)
(+-0288)
FetchSlide-expert
(+:1099
(=1.71)
FetchPick-expert
34.37 (#0.51)
33.99 (+1.81)
HandReach-expert
(+1.200
28.61 (+1.8
PointRooms-rande
35.52 (+0080)
36.56 (#1.28)
FetchReach--radoo
46.50 (#0.09)
46.61 (#0.07)
FetchPush-randaes
11.48 (+1.03)
11.64
Fetchhlidd-random
2.34 (+0.28)
1.81 (10.41)
FetchPick-randon
8.06 (+00.71)
6.48 (+2.24)
HandReach--andom
5.23 (#0.55)
5.29 (+17))
C ALGORITHM
We summarize our proposed approach WGCSL into Algorithm
D OFFLINE SETTINGS
Offline Datasets In the offline setting, only use collected datasets train agents, without additional interaction with the environment. The difference between the offline goal-conditioned RL dataset and the general offline RL dataset is that it also needs to save the original desired goals and the achieved goals ((s,). We follow previous works (Fujimoto et al. 2019 Fu et al. 2020) to introduce two different types of datasets:
17

---

Published as a conference paper at ICLR 2022
Algorithm 1: Weighted Goal-Conditioned Supervised Learning
Given offline dataset D, the best advantage percentile threshold N; Initialize policy No and value function Q: Initialize Huss-In-FFrst-Out queue B = (} to save recent advantage values;
for training step 2. ..,, M do
Sample mini-batch from the offline dataset: {(sE, 9, (TT) ) D;
Relabel the mini-batee using future goals:
{[s;, a,, ((,,), r(tt, ae, ((s:)),, f Drelabel:
Update value function Q to minimize TD error LTD on the relabeled mini-batch; Estimate advantage value A(st, (t, ((8))) using value function Q;
Update queue with the estimated advantage values;
Get the N percentile advantage value from B to compute the BAW;
Update policy xp to maximize the objective Eq. with the relabeled mini-batch:
JWGCSL (s. (al.aar.ss)-Detaeeeeea Dretaaelly log Te (e (s:))) (A(ss, as ((ss)) ((s))))
end for
Table 3: Average Return of Different Offline Datasets
Dataset
PointReach
PointRooms
Reacher
SawyerReach
SawyerDoor
Random
1.33
1.32
1.25
2.26
4.30
Expert
32.22
29.11
27.56
30.93
27.01
Dataset
FetchReach 0.71
FetchPush 3.19
FetchSlide 0.16
FetchPick
HandReach
Random
1.76
0.00
Expert
36.69
31.35
1.58
17.44
0.50
Random dataset: using a uniformly random policy to collect trajectories.
Expert dataset: using the final online policy trained by HER with additional Gaussian noise (zero mean and 0.2 standard deviation) and random actions (with probability 0.3) following previous work (Plappert et al., 2018) for data collection.
For the dataset of six easy tasks (PointReach, PointRooms, Reacher, SawyerReach, SawyerDoor, FetchReach), we save dict of x 10a transitions (2000 trajectories) containing observations, ac- tions, desired goals and achieved goals. Regarding the other four hard tasks (FetchPush. FetchSlide, FetchPick, HandReach), we store 2 x 106 transitions (40000 trajectories) in the offline dataset. The rewards can be computed by leveraging the corresponding interface (namely rewarr)) of the environment. The average returns of the two types of datasets are shown in Table Generally, average returns of the expert datasets are close to that of the optimal policy, while average returns of the random datasets are close to O.
Baseline Introduction In the offline goal-conditioned setting, we denote the offline dataset as D and consider the following baselines:
GCSL: GCSL (Ghosh et al., 2021) without online interaction, only using offline samples after relabeling (denoted Dretabel) to maximize
JOCSL (Tt) EE ((s)) [log (ayls ((+)))]
Goal MARWIL: conlicondd MARWIL (Wang 2018), using offline samples to maximize
1
JAARWIL = E (o. -))
T (a,lo g)pp( A(se, ge))
18

---

Published as a conference paper at ICLR 2022
To fairly compare with method, Goal MARWIL is implemented with actor-critic style as AWAC (Nair et al. 2020) and WGCSL, therefore it can also be treated as 'Goal AWAC'. Besides, also clip the exponential advantage value as WGCSL for numerical stability.
Goal AWR: goal-conditioned AWR Peng al.
2019 using offline samples to maximize
JAwn(w) F)(e....))) (ar exp(
3 (Rt V(s,, 9)))];
where Rs -i=0 7'rz4i. Different from the original AWR that uses TD(X) or Monte TD loss learn value function as
Carlo return RG train the value function, WGCSL. Empirically, we find TD target works better than Monte Carlo in our setting.
Goal BC: goal-conditioned behavior cloning (Bain & Sammut 1995). using offline sam- ples maximize
(TC( E a)) log (a g))].
Goal BCQ: goal-conditioned Baach-Constrained Q-Learning (Fujimoto et al. 2019), which we use the official codebase and concatenate observations, desired goals and achieved goals as states. The objective function of Goal BCQ is as follows:
JBCQ (E) ((e...)-Daa>G_.ll (e... (Q(;, a + E(s,, a, 9, +).g),
where {(s,, a. g, 4) the goal-conditioned perturbation model, which outputs an adjust- ment to an action within the range [-b, $). G.. is the VAE fitted on the behavior policy of the offline dataset. The policy is optimized through optimizing The Q function is learned using the Clipped Double Q-learning in (Fujimoto et al. 2019), and the only difference is that the Q function also includes goal g as input.
Goal CQL: goal-conditioned Conservative Q- Q-Learning (Kumar et al. 2020). which we also use the official codebase and concatenate observations, desired goals and achieved goals as states. The objective function of Goal CQL as follows:
Jcqu(T) (s.gg)> -.a==( (s..)) [Q(s,, a, g) log n (a)s
which jointly optimizes estimated return and policy entropy. The Q function of CQL is learned by minimizing the following loss:
Leql =aB(s.g) log Σ exp((((s), a,g))
((ss,,
E (( [Q(ss, a., - B (( a, g))
where Tb is the behavior policy for the offline dataset and 8 is the Bellman operator.
HER: Hindsight Experience Replay (Andrychowicz et al. 2017). using offline samples after relabeling to maximize
JHER(TT) = ("" [Q π ((:.g)).g))
where g' = =0(x), i 2 1 and the Q function is learned by minimizing TD error:
LTD B(s.or. (l"e) (rx + (Q( T (8 g'),g'))))))) - Q(+t ar g'))
Implementation of Baselines For fair comparison, all the implementation of WGCSL, GCSL, Goal MARWIL, Goal AWR, Goal BC, and HER use the Diagnal Gaussian policy with mean vec- (s, g) and non-zero constant variance o², where is set 0.2. WGCSL, Goal MARWIL, Goal AWR, and HER share the same network structure, i.e., the value function and the policy net- work (along with their target networks) are both 3-layer MLPs with 256 units each layer and relu activation. Besides, GCSL and Goal BC both keep single policy network without target network and value function. We also normalize the observations and goals with estimated mean and stan- dard deviation for WGCSL and other baselines, which is helpful for offline multi-goal manipulation tasks. The parameter 3 in Goal MARWIL and Goal AWR are set as by default. As regards to Goal BCQ and Goal CQL, we use the same batch size as other algorithms and other parameters are set as default (we find that default parameters perform equally well, if not better than others, in the grid search of critical parameters). All other syper-parameter of different algorithms kept the same for each task, such as batch size (128 for the first 6 tasks and 512 for harder taks), discount factor y 0.98, and Adam optimizer with learning rate x 10-4
19

---

Published as a conference paper at ICLR 2022
Implementation of WGCSL The basic implementation is introduced as above. To compute the best-advantage weight for WGCSL, we keep Ferst-In-First-oo queue B of size 5 x 10 to store recent caleulated advantage values, and the percentile threshold N gradually increases from 0 to 80. For each training step, N increases by 0.01 (for 4 harder tasks from FetchPush to HandReach) or 0.15 (for other tasks). The maximum clipping upper bound M for WGCSL is set 10. Though DRW brings improvements for most of the tasks on expert and random datasets, we find DRW slightly affects the performance of harder tasks on random datasets, which may be because in the random datasets we need more timesteps to obtain enough coverage of achieved goals, especially for harder tasks. Tuning ? for DRW trade-offs between the value and the effective coverage of relabeling goals, and excluding DRW is equivalent to setting 7 = 1 for DRW. Therefore, we only use GEAW and BAW for these tasks on random datasets.
For evaluation, we evaluate all the algorithms for 100 episodes without ran- domness. We repeat for 5 different random seeds and report the average evaluation performance with standard deviation.
Evaluation Settings
Poar:eadh- rupert
Po repooe-eepet
Sawyer Reachreppd
FeshhReach-wcoed
Reachen-spart
Avenag Ra.3
23
20
10
10
100
200
-00
200
1000
2000
1000
2000
5303
15000
15000
20000
Suayy
FatenSk
HandReacheeser:
30
Averaga Ratunn
23
10
Tnaring Shace
Trainin
Trening Siepe
Trainig Stapte
Trarkngg Sieps
(a)
Reart
PonFRoomsee candora
Sawyy Reach
FotahReach-eeddm
Reachat-andion
Avaragg Retum
30
20
12
10
200
1000
2000
1000
2000
19000 10000 20000
FandiP Jahvandom
Fatch-5 de mondom
HawssRaschranddmm
12
Avarage Raturn
2.0
1.5
2030 "0EE 1993) 1200d Training Steco
12030 20933 S00EE
4322)
10506 2315 2093) 400te Traikin Stope
1380 3009) Sa0e 4303)
10000 2200 3000J 4100e ring Stope
Train ng Stops
Traking Steps
(b)
Figure 7: Full average returns of different algorithms on (a) expert and (b) random datasets.
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, provide more empirical results in both offline and online goal-conditioned RL setting, including average return, final distance of the offline setting, results on small (10%) dataset, and the results of online experiments.
20

---

Published as a conference paper at ICLR 2022
E. FULL OFFLINE RESULTS
We provide the average return curves of all tasks in Figure In these results, WGCSL performs con- sistently well across ten environments, especially when handling hard tasks (FetchPush, FetchSlide, FetchPick, HandReach) and learning from the random offline dataset. Other conclusions have been discussed in Section 5.2 Comparing AWR with MARWIL, the major difference is using Monte Carlo return to replace the TD estimation for learning the policy, Hence, we can find that TD boot- strap learns more efficiently in weighted supervised leamiin, which similar to the conclusion of prior work (Nair et al. 2020). In addition, we present all final performance of different algorithms Table E.9
PoinRoadh export
Po export
Sawyer Reach export
FexhhReach export
Reaoher expert
Avongg Rud.m
30
30
20
20
10
100
201
193
200
1093
2000
1000
2000
9303
1906
19993
200ce
Average Rebum
10.0
20
20
20
010
Training Stece
Training Stepe
Traiing
Teanling
(a)
Point Beacii-andomm
Suerr Fbeecheea rdbm
Feichileecs-eerdtoo
Reacchee-asdom
40
Avarage
23
20
10
100
200
150
200
1000
2000
1000
2000
5303
12000
13800
20000
FondhP .sh rondom
FetchSt
Fonch
HandResch
Destngg Ret.m
85
6.0
90030
1003) Truriing Blass
Training Stose
Trainng Sxpps
Training Steps
Tranknng Biops
(b)
Figure 8: Comparison of WGCSL, Actionable Models, HER, DDPG offline (a) expert and (b) random datasets.
E.2 COMPARISON RESULTS WITH ACTIONABLE MODELS, HER AND DDPG
In this subsection, we make comparison with Actionable Models (AM) Chebotar al. 202 which applies the conservative estimation idea similar to (Kumar et al. 2020) into offline goal- conditioned RL. There some different settings between AM and ours. For example, consider cumulative return of the maximum horizon, ie.. the agent doesn't terminate the interaction after reaching the goal. To fairly compare with AM, we implement AM based HER. Specifically,
21

---

Published as a conference paper at ICLR 2022
revise the value update
CAM El s1+))) ) Dr Ur' T ( - Q az g'))
(( ( (Q(,, g')
In the formula, AM tries to minimize TD error and minimize unseen Q values at the same time. The action a is sampled according to exponential Q values. In our implementation, we sample 20 actions from the action space and compute their exponential Q value in order to sample a.
The results are shown in Figure 8 We can observe that DDPG and HER are unstable during train- ing, with large variance and performance drops at the end. In both expert and random dataset, HER outperforms DDPG, which demonstrates the benefits of goal relabeling for offline goal-conditioned RL. Besides, Actionable Model (AM) achieves higher final performance than HER and DDPG es- pecially in expert dataset, which indicates the conservative estimation technique is useful for offline goal-conditioned RL. Moreover, WGCSL is more efficient than AM, especially in harder tasks with large action space. Intuitively, WGCSL promotes the probability of visited actions, while AM re- strains the probability of unvisited actions. For tasks with large action space, conservative strategy can be ineffective compared to WGCSL
Expert Datisel
Random Dalaset
1.0
GCSL
Godl MARALL
God APA
Goal BC
Godl BCG
0.6
0.6
Godl CO.
Nomealan Londedddd
Normatted
HER
OCBL
Oool MAWWL
Oool AWR
Gool 90
0.2
Goal DCG
0.2
Goal COL
HEN
0.0
0.4
Training Process
Training Process
(a)
(b)
Figure 9: Normalized score across 10 tasks on (a) expert and (b) random datasets. The black dotted line refers to the normalized score of offline dataset. The value are normalized by dividing by the maximum final return of each task in Table [ Note that the value can exceed as the final return can be lower than the return during training. The x-axis represents the training process scaled to for different tasks.
E.3 NORMALIZED SCORE ACROSS 10 TASKS
For clarity, report the normalized score across 10 tasks to compare different algorithms in Fig- ure | We can observe the significant improvement of WGCSL over other baselines. We take an insight into the impact of the goal relabeling technique. For expert datasets, goal relabeling is not dominant factor, e.g., the learning curve of Goal BC and GCSL. are very close. In random datasets, goal relabeling is essential to achieve high performance. In addition to goal relabeling, WGCSL contains other techniques to jointly promote the performance in offline goal-conditioned RL tasks. Other conclusions are the same as prior subsections (Appendix E. and Appendix E.2)
E.4 COMPARISON OF TRAINING TIME
We further analyze the computation cost of the four most effective algorithms in the random offline setting, i.c., WGCSL, GCSL, HER and Actionable Models (AM). The results are shown in Figure 10 from which we can conclude that GCSL needs the minimal training time, while WGCSL and HER require similar cost. It needs to be emphasized that WGCSL has only a small amount of additional training time over GCSL, but it achieves significantly higher performance. Therefore, we believe that the cost of learning the value function on top of GCSL is worthwhile. Moreover, AM requires the largest computation among these algorithms, which is mainly because it needs to
22

---

Published as a conference paper at ICLR 2022
FetchPush
FetchSlide
FetchPick
HandReach
Training Time (min)
100
100
100
100
60
60
60
20
GC3L
WG2SL
HER
GC51
HER
(CCRL
VOCSL
HER
WICCA
AM
Figure 10: Training time of GCSL, WGCSL, HER and Actionable Models (AM) on harder tasks.
PIWEESL
POWGCSL
AM
AM
DDPG
DDPG
HER
Goal
Goal
Goal BC
Goal AWR
Goal AWR
Goal MARWIL
Goal MARWIL
GCSL
GCSI
WGCSL
WGCSL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
(a) Expert Dataset
(h) Random Dataset
Figure 11: Average probability of improvement offline (a) expert and (b) random datasets. Each figure shows the probability of improvement of WGCSL compared to all other algorithms. The interval estimates are based stratified bootstrap with independent sampling with 2000 bootstrap re-samples.
sample several actions and compute their exponential Q value for constraining the large Q value. These procedures make AM the most time-consuming algorithm. In contrast, WGCSL can achieve higher performance with less training time. The training time would vary across different platforms, and we use single GPU (Tesla P100 PCIe 16GB) and 5 cpu cores (Intel Xeon E5-2680 v4 @ 2.40GHz) to run 5 random seeds parallelly.
E.5 AVERAGE PROBABILITY OF IMPROVEMENT
In this subsection, adopt the average probability of improvement (Agarwal al. 2021 robust metric to measure how likely it is for one algorithm to outperform another on a randomly selected task. The results are reported Figure I1] As shown in the results, WGCSL robustly outperforms other baselines in both expert and random datasets. For instance, WGCSL is 100% better than baselines on the expert dataset, and 6 baselines on the random dataset. Note that the most two effective and robust algorithms on both random and expert datasets are WGCSL and Actionable Models (AM), which are specifically designed for offline goal-conditioned RI.. Comparing the two algorithms, WGCSL outperforms AM with probability of 75% on the expert dataset and 62% on the random dataset.
E.6
FINAL DISTANCES IN THE OFFLINE SETTING
GCSL (Ghosh et al. 2021 uses the average final distance to the desired goal measure for goal-reaching problems. We also provide average final distances in the offline setting to evaluate the learned goal-conditioned policy in Figure 12 From the results substantially outperforms other baselines in both random and expert dataset. Although GCSL learns suboptimal policy, it also achieves relatively consistent final distance across different environments
conclude that WGCSL
23

---

Published as a conference paper at ICLR 2022
Model
Poir/Bwach-eppe
Sussyer flecchequedd
Pwzzh Teech-eecowd
Desccheneoper
020
Trral Dr MAREO
8.15
0.3
0.10
8.10
2000
Sowyy oxport
oxpen
Fin Distance
064
020
02
0.19
0.1
6:2
840
6.0
6.65
8.00
18300
20000
10000
20003
50000
40300
10000
20006
30000 40000
10000
20000
20000
43030
20000
30000
(a)
Point nendoi
Po Pantoouneerrddr
Suere yer Tbbencheeadddrmm
Feichilesc--arrdrm
$.00
Fin Dleance
8.15
0.15
0.11
8.0
100
200
100
200
1003
2000
1000
2000
5303
12000
13000
20000
Door
FenchPush random
FetchSl de random
FoschP
HandResch
0.26
0.25
029
Piodl sttroo
830
0.29
2.40
0.32
2030
10000 20033 50066
10066 2006
1003)
<00c0
2000
43033
3300) arece
Training Stoso
Trainng Sxpss
Traniig Blass
Trainin Stops
Traning Blops
Figure 12: Average final distance of different algorithms offline (a) expert and (b) random datasets.
and offline datasets. Besides, Goal MARWIL and Goal BC are two effective baselines when learning from the expert dataset, but they do not perform well on the random dataset. In addition, HER and DDPG learn unstable results mainly because of bootstrapping from outto--distribution actions.
E.7 RESULTS ON 10% OFFLINE DATASET
To evaluate the impact of dataset size the performance, select 10% of samples (i.c., 2 1 transitions) from the offline dataset of hard tasks (FetchPush, FetchSlide, FetchPick, HandReach). The results are reported in Figure 13 and Figure [4] From the results, only 10% offline data WGCSL consistently achieves the best performance out of these algorithms. In Figure [4] we can see that WGCSL is very robust the dataset size the expert dataset, while the performance is affected more by the small data size on the random dataset. The probable reason that in the offline setting, the number of samples influences the data coverage greatly and learning generalizable value function from small set of data is very difficult.
conclude that with
E.8 ONLINE EXPERIMENTS
We introduce implementation details and dyper-parameters the online setting in this subsection. As GCSL is originally proposed for online goal-conditioned RL, WGCSL also be applied to online goal-conditioned setting. In the online training setting, the agent iterates between collecting trajectories from the environment and training with samples from the replay buffer. We conduct
24

---

or 4 mini-batches Published as a conference paper at ICLR 2022
WECSL
GCSL
Goa MARNIL
Soal BC
HER
DOPG
Actonable Models
FathhPush-
FatzhSlidf-ccpprt
FelccPPck-Oxpert
HandRRachh-oxpar
Avarage Reltum
20
20
0000
20020
30000
40300
10020
20000
30000
40000
10000
20000
30000
42000
10000
20000
30020
40000
Training Steps
Training
Training
Training Stepe
FeichPusshranddm
FotchSlide-rande random
FelchPPok-random
HandRRacch-aandee
1.25
Average Return
1
2.5
0.50
2.0
1.5
0.00
1.0
10000
20000
20000
40200
10000
20003
30003
40000
10000
20003
30000
42000
10000 20000
30000
40300
Training Steps
Training Stops
Training Steps
Traiiing Steps
(b)
Figure 13: Average return of 10% offline (a) expert and (b) random dataset.
FelchPPst--epart
FulchSSidd-sspert
FatchPPssoappet
HanddReach-eqpeett
date
&
(a)
FutchPush--nddae
Fetch Side-sanamm
FutchPiieeendoom
HandRoach-anniim
2
WOOELOO O00L NAYAIL
DCPO
WOOEL OOOL MATTINL
HER DOPO
WOOEL MARK
HOT DOPO
WOOOL OCOL NARIA BC
DOPO
(b)
Figure 14: Final performance comparison of full dataset and 10% dataset. Top row is the result on expert dataset, and the bottom row is the result on random dataset.
experiments on the first six taks, i.c., PointReach (here we call Poin22DarrgeEvv) PointRooms SawyerReach, FetchReach, Reacher and SawyerDoor. The
(here we call Sain2D-FourRoomae buffer size for first four tasks is set as 1 and 10a for the other two tasks. Before training starts, we first randomly sample 20 trajectories to initialize the buffer. The policies of WGCSL and GCSL are both Diagonal Gaussian policy with mean vector ( g) parameterized as and a non- zero constant variance o², The standard deviation is set as 0.2. All the policy networks and value networks are the same as that used in the offline setting. When interacting with the environments, additional random actions for exploration with probability of 0.3. After data collection, we sample mini-batch from the replay buffer and perform goal relabeling for WGCSL and GCSL. Then we train the agent for mini-batch (Point2DLarge)
25

---

Published as a conference paper at ICLR 2022
for other tasks. The batch size, optimizer, learning rate keeps the maximum clipping upper bound M for WGCSL is set as 10. We also utilize target nets for WGCSL to stabilize training, and the polyak average coefficient for target nets 0.9. With respect to the evaluation phase, we use the deterministic policy [ g) without randomness. After each epoch, we evaluate the policy for 100 episodes and compute the average success rate. For all the experiments we repeat for 10 different random seeds and report the mean performance with standard deviation. In this subsection, we report the results of WGCSL with the discounted relabeling weight (DRW) and the condiiond exponential advantage weight (GEAW), though the performance can be improved with the best- advaatga weight (BAW).
the offline setting. The
The empirical results regarding the average success rate of the online setting are shown in Figure 15 From Figure 15 we can conclude that WGCSI. achieves better success rate and converges faster than other baselines in most of the environments except SawyerReach. Therefore, WGCSL is verified to be highly efficient both online and offline goal-conditioend RI., and we hope could be a stepping stone for future real-world RL applications.
WGCSL(OUrs)
GCSL
TD3+HER
TD3
PPO
Point2DLargeEn-v11
PointD-FFurRoom-v1
SawyerReachxXZZEnwvv1
1.0
1.0
Mean Success Rate
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
50
200
100
200
150
FetchReech-v11
Reacher-v2
SawyerDoor-v00
1.0
1.0
1.0
Mean Success Raie
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.0
0.0
0.0
50
100
150
200
250
600
1000 Episodes
1600
2000
0
500
1000
1500
2000
Episodes
Episodes
Figure 15: Mean success rate of online experiments with standard deviation (shaded) across 10 random seeds.
Paintt0 FuuRoont
SaeysRoodkWYEewr
Roahhvt
Reachane
Sakys Dupruv0
30
28
20
93
130
203
100
239
190
123
280
500
1300
1500
2000
500
1000
1521
2000
Figure 16: Study of the impact of clipping exponential advantage in WGCSL in the online setting. Results are averaged across 10 random seeds.
E.9 THE IMPACT OF CLIPPING FOR EXPONENTIAL ADVANTAGE WEIGHTS
We study the impact of clipping in WGCSL's exponential advantage weights by comparing with the following variants:
WGCSL w/o clip: clipping weights performed.
26

---

Published as a conference paper at ICLR 2022
WGCSL clip 5: clipping the exponential advantage weights to (0, 5].
WGCSL clip 10: clipping the exponential advantage weights (0, 10), which is used in our paper by default.
From the results in Figure [6 we can observe that without clipping the advantage estimation can harm the performance in Reacher and SawyerDoor. Therefore, we perform weight clipping by default.
F DETAILS OF ENVIRONMENTS
All of the ten tasks have continuous state space, action space and goal space. The maximum episode horizon is set as 50. In this section, we introduce these tasks in detail.
PointReach PointReach taken from the open-source environment multi-world and it requires the blue point to reach the green circle. The state space [-5,5] [-5,5] has two dimensions representing Cartesian coordinates of the blue point, and the action space [-1, 1] x [-1, 1] also has two dimensions meaning the horizontal and vertical displacement. The goal space is the same as state space, which means ((s) = s. The bule point and the green circle are randomly initialized in the state space. The allowable error of reaching goals is the radius of the target circle and is set as
1. The reward function is defined as:
r(xxy, a, (xx) = 1[1lxxy gxv e).
The PointRooms environment is built on multi-world. The state space, the action space, the goal space, and the reward function are the same as PointReach. The difference that there are four rooms separated by walls.
PointRooms
Reacher The Reacher environment is revised from OpenAl Gym (Brockman et al. 2016). States
are I--dimensional, which indicate the angles, the positions, and the velocity the joints. Actions are 2-dimensional and they control the movement of two joints. Goals are 2-dimensional vectors representing the expected XY position. And the state-to-gon toag mapping Ø(s) = s(-3 -1], where the last three dimensions are the XYZ position of the end-effect. The reward function is defined as:
r(xxy, a, gxy) 1[]sxx -xx < c).
where the allowable error is set as 0.05.
SawyerReach The SawyerReach environment is taken from multi-world. The Sawyer robot aims to reach a target position with its end-effector. The observation space is 3-dimensional, representing the 3D Cartesian position the end-effector. Correspondingly, the goal space 3-dimensional and describes the expected position, and the state-to-gov mapping is ((s) has dimensions describing the next position of the end-effector. The reward function is defined as:
Besides, the action space
(,xxy,,0,xy22 = 1[[|sxyz gXY 2 <)),
where the allowable error 0.06.
SawyerDoor The SawyerDoor environment is revised from multi-world. The Sawyer robot is re- quired open the door desired angle. The state space (4-dimensional) consists of the Cartesian coordinates of the Sawyer end-effector and the door's angle. As in the SawyerReach task, the action space 3-dimensional and controls the position of the end-effector. The desired goals are uniform from 0 to 0.83 radians. And the reward function is defined as:
r(8, a, Jongie) 1(b((s) Janglel S <e),
where the allowable error is set as 0.06.
27

---

Published as a conference paper at ICLR 2022
FetchReach The FetchReach environment is taken from OpenAl Gym (Brockman et al. 2016 Plappert et al. 2018). this environment, a 7-DoF robotic arm is expected to touch a desired loca- tion with its two-finger gripper. The state space is 10-dimensional, including the gripper's 's position and linear velocities. The action space is 4-dimensional, which represents the gripper's 's movements and its status about the opening and closing. Moreover, the goals are 3-dimensional vectors repre- senting the target places of the gripper. The state-to-goal mapping is o(s) s(0 3]. because the first 3 dimensions of the state describe the position of the gripper. The allowable error in FetchReach is c 0.05. The reward function is defined as:
rss, a, 9xYz) = 1(]l((s) (xx2212se)
FetchPush Similar FetchReach, the FetchPush environment taken from OpenAl Gym where a 7-DoF robotic arm is controlled to move a box to the desired location. The state space is 25- dimensional, including the gripper's 's position, linear velocities, and the box's position, rotation, lin- ear and angular velocities. The action space is also 4-dimensional as in FetchReach. Different from FetchReach, the 3-dimensional goals and achieved goals represent the target and current position of the box. The state-to-goal mapping is (ss) = s|3 6]. because the 3 dimensions of state describe the position of the box. The allowable error in FetchPush c = 0.05. The reward function is defined
r(s, a, 9XYZ) = ([])(s) gxvz 11 c).
FetchSlide The FetchSlide environment is like FetchPush where 7-DoF robotic arm controlled to slide box to the desired location. The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush. The difference is that the target position is outside of the robot's 's reach thus it has to hit the box and makes the box slide and then stop at the desired position.
FetchPick Like FetchPush and FetchSlide environments, the FetchPick environment controls 7-DoF robotic arm to pick up a box to the desired location. The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same FetchPush. The difference is that the target position may be on the table or in the air, thus the agent needs to pick the box with its gripper.
HandReach In HandReach environment, the agent is required to control 24-DoF anthropomor- phic hand and use its fingers to reach the target place. Observations is 63-dimensional containing two 24-dimensional vectors about positions and velocities of the joints, and additional 15 dimen- sions indicating current state fingertips. Actions are 20-dimensional vectors that control the non-coupled joints of the hand. Goals have 15 dimensions for HandReach representing the tar- get Cartesian positions of each fingertip. The reward function is the same as Fetch tasks except the state-to-goallmappiig (ø- s[-15 15 :)) and the allowable threshold (e 0.01).
28

---

Published as a conference paper at ICLR 2022
Table 4: Average final performance of all algorithms with standard deviation over random seeds.
HandRRach cauuuRRaadrrdddddo random
RiccPick--ran AichPicc-randdom candom
FetchSlide- MnchhSiide-anddon madd-nndoo
Pusis -andom
FotchReacl--ne Radd-RRcc--aadddo
Sn SmnyerDoor random
SuwyerRennt
Reacher-ean seceeee-raaaeee senercaddo
Rooms- Kooms-anddm
Point
Poin:Reach Reach random
HondRood exppert
Hichrickk-eese expert
FetchPick
dedchiiid--opprr expert
PetchSiide expert
PutchPusi exper
FetehRende poneeesee
Atreyur aenur exseett
Reacher-esee expee deteeeteadeerr
Poin Rooms com expert
Point Pount Reael--xpon
Taok
randon
madim
5,23 (0.55)
8.06 -0.71)
2.34 2 (-0 28)
(1 (1)5)
(6.50(::000 (00000
46.50
36.82 (13 3.200
41.05 (10.19)
2 (0.11)
41.12 (200000)
35 35.52 (355(12000
44.30 (E2224
26.73 [1: (.20)
34,37 (1:05:1)
10.73 (:: 10017)
/
39.11
22.81
40.12 10223
MLST 1800110
M6.1
41400
3,96 (0.1)
3.05 (2022
1.47 (-0.12)
020.60
38 322
86
25 00077
1486
22
22
2110
00,80
(- 22
0.57
25.23
3.05
22
111
31
12
06
00
0,00 E10-6
2.01
0.19 280.7
8086
0.33
11330
15.55
1667
3667
081
345
1.55
1699
2000
72
1200000
[20 11)
21
(1)
16
00
0.0
(00
1110
0.00
1.64
0.13
385 [E
231
5.5
3390
144
0.13
0.1
225
16.
1:1000000
(10
111
300
13)
(09
23
000
(V
(20
00000
1.5 (10 155
0.25
2.44
000
100
00
15
55
22
3
06
11
200
1
SWYRE 92
1000
125
115
185
0000-
(200
3 95
6 01
00
1100
08
(100
155
600
(11
101
1:2
0.55
+ 21
00
2:
300
055
09- 2010
1E5000 70.3
OFF
198
100
654
(E5
001
236
10000
33
000
20
11133
2000
000000
138 0
10000
13
101
113
0,000
699
039 10
1025
38
200
10
28
88
T7 17 02222220000
20000
1.2
000
200
0,00
5000 (20000
1772
11
A600
2100000
55
160
100000
1022
29


Reviews:
[{"review_id": "kIbUb5ABIgt", "reviewer": "Reviewer_FPSj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThe paper is well written and the notation very easy to follow.  There are a couple of typos/weird phrase constructions in the paper that could warrant a bit of proofreading.  I put some examples at the end of this section.\nThe idea, although somewhat an obvious followup to GCSL, is well explained and analyzed, and experimental results are convincing.  \n\nCons:\nOne of the advantages of GCSL is that it is a purely supervised method, which greatly enhances the simplicity and stability.  The proposed approach uses a Q-function to derive the advantage function A, which requires a bootstrap-based method to be applied to an offline dataset.  I would greatly appreciate a bit of commenting on this in the paper, as this is well-known to not be trivial to get working.  Is the method relying on the goal-conditioned aspect to compensate for over-estimation issues?  \n\nAnother topic I would have like to see more discussion on is the difference between HER and WGCSL.  If I understand correctly, HER is able to train in the offline setting due to goal-conditioned aspects of the problem, but why would WGCSL fundamentally work better than a purely value-based method?  \n\nOverall I vote to accept this paper, but would appreciate responses from the authors on my questions.\n\nSection: 'Expert Datasets': 'Tabel 1' -> 'Table 1'.  'in offline' -> 'in the offline'\n'Random Datasets' section: 'relatively well policy'  -> 'relatively good policy'.  'results of the random' -> 'results on the random'.", "summary_of_the_paper": "This paper proposes an extension of GCSL where the goal-conditioned BC loss is weighted by a variable that correlates with the number of steps necessary to achieve the desired goal.  The advantage of this approach is that sub-optimal trajectories to a particular goal will get downweighted to the benefit of more direct trajectories.  This effectively adds a policy improvement step to GCSL with the underlying objective function being that of arriving at goal states as fast as possible.  \n\nResults on goal-conditioned tasks show that the proposed approach (WGCSL) performs consistently better than GCSL and in some cases significantly better (HandReach-expert).", "summary_of_the_review": "A somewhat obvious extension to GCSL (online goal-conditioned BC), but that is well-explained and shows convincing experimental results.  I vote for acceptance, and believe this provides a potentially new baseline for offline goal-based methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GvFIMSnJdF", "reviewer": "Reviewer_fo1Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper proposes a weighted GCSL approach that appears to be motivated from optimizing a better lower bound (Theorem 1) and results in large empirical gains (although statistical uncertainty in results isn’t reported).\n- The empirical experiments are thorough with ablations and show the benefit of the proposed WCGSL.\n- The paper is generally positioned well wrt prior work but more discussion of some closely related work is needed for clarity.\n\nWeaknesses:\n\n- I found the paper to be somewhat unclear to read. Some examples:\n  - The main contributions are presented very succinctly in Section 3.2. This makes the (1) goal-conditioned advantage re-weighting and (2) best-advantage weight hard to comprehend and understand the motivation behind such schemes.\n  - Notations are overloaded heavily, for example, the advantage and value functions are used without any superscript for the policy which was confusing. For example, I didn’t understand what value function is being talked about in Proposition 2.\n  - Results from Wang et. al (2018) and Haarnoja et al (2017) are directly used without clearly explaining them to the reader. \nMulti-modality challenge is used in intro and section 3.2 without introducing what is the challenge (it only becomes clear in the last part of sec 3.2) \n  - The motivating example was confusing – if we only have a sparse 0/1 reward function for goal reaching (as defined in the preliminaries), how are we claiming one trajectory to be better than another? Under this reward function, both the trajectories achieve the same return.\n\n- Theoretical presentation of some results don’t add much value. Examples:\n  - Theorem 2  trivially follows from Theorem 1 following a 2 line proof and calling it a theorem is unwarranted. \n  - Proposition 2 is presented in the paper but empirical experiments simply avoid its relabeling strategy and implications. This made me wonder about the relevance of that proposition for this paper.\n\n\n- The empirical reporting and rigor of the results should be improved.\n   - No error bars are reported in Table 1 but as shown by Agarwal et al. (2021), it seems critical to report statistical uncertainty in deep RL results. Furthermore, the large number of entries in Table 1 makes it slightly overwhelming to read, maybe presenting it visually using performance profiles might be more informative.  \n  - The learning curves are only shown selectively for tasks where WGCSL outperforms other methods. Instead, curves showing aggregate scores across all tasks such as mean scores or interquartile mean (less prone to outliers)  for random and expert datasets with confidence intervals (CIs) might be a better way to summarize performance across tasks.  \n  - Adding average probability of improvement of WGCSL over other methods as done by Cong et al (2021) would further make the results more statistically reliable. The rliable library would be quite useful for reporting results with CIs and aggregate metrics: https://github.com/google-research/rliable\n  - No std are shown for some of the methods in Figure 5.\n  - Why are the default hyper parameters tuned for behavior regularization methods such as CQL and BCQ – it seems unlikely that the default values from D4RL/Mujoco tasks would work for the tasks presented here. Thus, are the comparisons fair?\n\nOther minor comments:\n - Typo: GSCL → GCSL (under expert datasets in experimental section)\n\n\nReferences:\n\n[1] Haarnoja, T., Tang, H., Abbeel, P. and Levine, S., 2017, July. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (pp. 1352-1361). PMLR.\n\n[2] Wang, Q., Xiong, J., Han, L., Sun, P., Liu, H. and Zhang, T., 2018, December. Exponentially Weighted Imitation Learning for Batched Historical Data. In NeurIPS.\n\n[3] Agarwal, R., Schwarzer, M., Castro, P.S., Courville, A. and Bellemare, M.G., 2021. Deep reinforcement learning at the edge of the statistical precipice. In NeurIPS.\n", "summary_of_the_paper": "The paper proposes weighted goal-conditioned supervised learning (WGCSL) as a tighter lower bound to optimize than the typical GCSL objective. Then the paper combines a bunch of prior approaches / heuristics to set these weights and shows improvements on a new offline RL benchmark containing various simulated robotic manipulation tasks. ", "summary_of_the_review": "While the empirical results of the proposed approach looks promising, as pointed in the weaknesses, I have concerns about clarity of the paper, presentation of the theory and empirical rigor. If the authors can address these issues, I'd be willing to re-assess the paper.\n\n--- After rebuttal --- Updated my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DlWKGI7UkV6", "reviewer": "Reviewer_i8oK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance**: On the one hand, the empirical results of the paper are quite strong. On the other hand, by introducing a value function, the paper significantly increases the complexity of the method, and simplicity is arguably the most important feature of goal-conditioned behavior cloning methods.\n\n**Originality**: The proposed method is novel, to the best of my knowledge. While both $\\gamma$-discounting and advantage weighting have been studied in a number of prior works (as noted in this paper), I am unaware of prior work that applies them to goal-conditioned behavior cloning methods.\n\n**Clarity**: The paper is clear and readable.\n* I was confused about the difference between GoalBC, Goal BCQ, and Goal CQL. For example, is Goal-CQL designed so that Q function reflects the *marginal* data collection policy or the *goal-conditioned* data collection policy? (i.e., how is hindsight relabeling combined with CQL)?\n\n**Correctness**: I have a few concerns about the correctness of the paper.\n* In the second to last line of the proof, if we replaced the weights $\\gamma^{i-t}$ with 0, we'd get a useless objective, but would the bound be made *even* tighter?\n* I'm unsure if the main result is actually useful. In particular, Theorem 1 shows that the proposed method optimizes a tighter bound on $J_{surr}$, but we actually want a tighter bound on $J$ (defined in Sec. 2).\n* Proposition 1, \"good as or better than\" -- According to what criterion?\n* Proposition 2: I think this result is a bit misleading, because it is really a result about the dataset, not the policy; it is not a \"policy improvement\" result, as indicated by the subsection header.\n\n**Minor comments**\n\n* Abstract: It's not clear what problem is being solved. I'd recommend making this more clear in the first 1-3 sentences.\n* \"can we learn goal-conditioned policies from offline data?\" -- [1] already does this.\n* \"hindsight relabeling would generate non-optimal experiences\" -- I don't this this is correct: [2] prove that hindsight relabeling is equivalent to inferring a task assuming that the experience were optimal.\n* \"extremely sparse\" -> \"sparse\"\n* \"last step reward\" -- Clarify what this means.\n* Eq. 2, $w_{i,t}$: This should be defined before the equation.\n* I'd recommend including advantage weighting methods (e.g., [3]) as an additional baseline. This should be trivial to implement because the proposed method already includes a form of advantage weighting.\n* Ethics Statement, \"there are concerns that AGI might go out of human control in the future\" -- The ethics statement lacks nuance. I'd recommend that it be revised to more carefully address the specific ethical implications of this paper.\n\n\n[1] Lynch, Corey, et al. \"Learning latent plans from play.\" Conference on Robot Learning. PMLR, 2020.\n\n[2] Eysenbach, Benjamin, et al. \"Rewriting history with inverse rl: Hindsight inference for policy improvement.\" arXiv preprint arXiv:2002.11089 (2020).\n\n[3] Peng, Xue Bin, et al. \"Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.\" arXiv preprint arXiv:1910.00177 (2019).", "summary_of_the_paper": "The proposed method proposes a method for goal-conditioned RL that can be interpreted as a weighted version of prior work.These weights resemble a combination of discounting and advantage weighting. The paper provides theory arguing that these weights cause the proposed method to optimize a tighter lower bound than prior work. Experiments show that it outperforms the prior work, which does not include weights on each training example.", "summary_of_the_review": "My main concerns are about the clarity and correctness of the paper (the empirical results are already great!). If the paper were revised to address these concerns, I'd be inclined to change my vote.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a method that improves goal-conditioned supervised learning (GCSL) by iteratively re-weighting the experience by a variable that correlates with the number of steps till the desired goal. The reviewers mention that the authors focus on an important problem, their method is simple and the empirical results are significant. However, they do point several flaws of the paper, the main ones being questionable theoretical claims and the clarity of the presentation. After an extensive discussion, most reviewers agree that the paper should be accepted but I do encourage the authors to take into account the comments by the reviewers for the final version of the paper and make the theory more clear.
Decision: Accept (Poster)