Year: 2022

Abstract:
Solving goal-conditioned tasks with sparse rewards using self-supervised learning is promising because of its simplicity and stability over current reinforcement learning (RL) algorithms. A recent work, called Goal-Conditioned Supervised Learning (GCSL), provides a new learning framework by iteratively relabeling and imitating self-generated experiences. In this paper, we revisit the theoretical property of GCSL — optimizing a lower bound of the goal reaching objective, and extend GCSL as a novel offline goal-conditioned RL algorithm. The proposed method is named Weighted GCSL (WGCSL), in which we introduce an advanced compound weight consisting of three parts (1) discounted weight for goal relabeling, (2) goal-conditioned exponential advantage weight, and (3) bestadvantage weight. Theoretically, WGCSL is proved to optimize an equivalent lower bound of the goal-conditioned RL objective and generates monotonically improved policies via an iterated scheme. The monotonic property holds for any behavior policies, and therefore WGCSL can be applied to both online and offline settings. To evaluate algorithms in the offline goal-conditioned RL setting, we provide a benchmark including a range of point and simulated robot domains. Experiments in the introduced benchmark demonstrate that WGCSL can consistently outperform GCSL and existing state-of-the-art offline methods in the fully offline goal-conditioned setting.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) enables automatic skill learning and has achieved great success in various tasks (Mnih et al., 2015; Lillicrap et al., 2016; Haarnoja et al., 2018; Vinyals et al., 2019). Recently, goal-conditioned RL is gaining attention from the community, as it encourages agents to reach multiple goals and learn general policies (Schaul et al., 2015). Previous advanced works for goal-conditioned RL consist of a variety of approaches based on hindsight experience replay (Andrychowicz et al., 2017; Fang et al., 2019), exploration (Florensa et al., 2018; Ren et al., 2019; Pitis et al., 2020), and imitation learning (Sun et al., 2019; Ding et al., 2019; Sun et al., 2020; Ghosh et al., 2021). However, these goal-conditioned methods need intense online interaction with the environment, which could be costly and dangerous for real-world applications.
A direct solution to learn goal-conditioned policy from offline data is through imitation learning. For instance, goal-conditioned supervised learning (GCSL) (Ghosh et al., 2021) iteratively relabels collected trajectories and imitates them directly. It is substantially simple and stable, as it does not require expert demonstrations or value estimation. Theoretically, GCSL guarantees to optimize over a lower bound of the objective for goal reaching problem. Although hindsight relabeling (Andrychowicz et al., 2017) with future reached states can be optimal under certain conditions (Eysenbach et al., 2020), it would generate non-optimal experiences in more general offline goal-conditioned RL set-
∗Corresponding Authors
ting, as discussed in Appendix B.1. As a result, GCSL suffers from the same issue as other behavior cloning methods by assigning uniform weights for all experiences and results in suboptimal policies.
To this end, leveraging the simplicity and stability of GCSL, we generalize it to the offline goal-conditioned RL setting and propose an effective and theoretically grounded method, named Weighted GCSL (WGCSL). We first revisit the theoretical foundation of GCSL by additionally considering discounted rewards, and this allows us to obtain a weighted supervised learning objective. To learn a better policy from the offline dataset and promote learning efficiency, we introduce a more general weighting scheme by considering the importance of different relabeling goals and the expected return estimated with a value function. Theoretically, the discounted weight for relabeling goals contributes to optimizing a tighter lower bound than GCSL. Based on the discounted weight, we show that additionally re-weighting with an exponential function over advantage value guarantees monotonic policy improvement for goal-conditioned RL. Moreover, the introduced weights build a natural connection between WGCSL and offline RL, making WGCSL available for both online and offline setting. Another major challenge in goal-conditioned RL is the multi-modality problem (Lynch et al., 2020), i.e., there are generally many valid trajectories from a state to a goal, which can present multiple counteracting action labels and even impede learning. To tackle the challenge, we further introduce the best-advantage weight under the general weighting scheme, which contributes to the asymptotic performance when the data is multi-modal. Although WGCSL has the cost of learning the value function, we empirically show that it is worth for its remarkable improvement. For the evaluation of offline goal-conditioned RL algorithms, we provide a public benchmark and offline datasets including a set of challenging multi-goal manipulation tasks with a robotics arm or an anthropomorphic hand. Experiments1 conducted in the introduced benchmark show that WGCSL significantly outperforms other state-of-the-art baselines in the fully offline goalconditioned settings, especially in the difficult anthropomorphic hand task and when learning from randomly collected datasets with sparse rewards.

heading: 2 PRELIMINARIES
text: Markov Decision Process and offline RL The RL problem can be described as a Markov Decision Process (MDP), denoted by a tuple (S,A,P, r, γ), where S and A are the state and action spaces; P describes the transition probability as S ×A× S → [0, 1]; r : S ×A → R is the reward function and γ ∈ (0, 1] is the discount factor; π : S → A denotes the policy, and an optimal policy π∗ satisfies π∗ = arg maxπ Es1∼ρ(s1),at∼π(·|st),st+1∼P(·|st,at)[ ∑∞ t=1 γ
t−1r(st, at)], where ρ(s1) is the distribution of initial states. For offline RL problems, the agent can only access a static dataset D, and is not allowed to interact with the environment during the training process. The offline data can be collected by some unknown policies.
Goal-conditioned RL Goal-conditioned RL further considers a goal space G. The policy π : S × G → A and the reward function r : S × G × A → R are both conditioned on goal g ∈ G. The agent learns to maximize the expected discounted cumulative return J(π) = Eg∼p(g),s1∼ρ(s1),at∼π(·|st),st+1∼P(·|st,at) [∑∞ t=1 γ t−1r(st, at, g) ] , where p(g) is the distribution over goals g. We study goal-conditioned RL with sparse rewards, and the reward function is typically defined as:
r(st, at, g) = { 1, ||φ(st)− g||22 < some threshold 0, otherwise ,
where φ : S → G is a mapping from states to goals. For theoretical analysis in Section 3, we consider MDPs with discrete state/goal space, such that the reward function can be written as r(st, at, g) = 1[φ(st) = g], which provides a positive signal only when st achieves g. The value function V : S × G → R is defined as V π(s, g) = Eat∼π,st+1∼P(·|st,at) [∑∞ t=1 γ t−1r(st, at, g)|s1 = s ] .
Goal-conditioned Supervised Learning Different with goal-conditioned RL that maximizes discounted cumulative return, GCSL considers the goal-reaching problem, i.e., maximizing the last-step reward Eg∼p(g),τ∼π [ r(sT , aT , g) ] for trajectories τ of horizon T . GCSL iterates between two process, relabeling and imitating. Trajectories in the replay buffer are relabeled with
1Code and offline dataset are available at https://github.com/YangRui2015/AWGCSL
hindsight methods (Kaelbling, 1993; Andrychowicz et al., 2017) to form a relabeled dataset Drelabel = {(st, at, g′)}, where g′ = φ(si) for i ≥ t denotes the relabeled goal. The policy is optimized through supervised learning to mimic those relabeled transitions: θ = arg maxθ E(st,at,g′)∼Drelabel log πθ(at|st, g′).

heading: 3 WEIGHTED GOAL-CONDITIONED SUPERVISED LEARNING
text: In this section, we will revisit GCSL and introduce the general weighting scheme which generalizes GCSL to offline goal-conditioned RL.

heading: 3.1 REVISITING GOAL-CONDITIONED SUPERVISED LEARNING
text: As an imitation learning method, GCSL sticks the agent’s policy to the relabeled data distribution, therefore it naturally alleviates the problem of out-of-distribution actions. Besides, it has the potential to reach any goal in the offline dataset with hindsight relabeling and the generalization ability of neural networks. Despite its advantages, GCSL has a major disadvantage for offline goalconditioned RL, i.e., it only considers the last step reward r(sT , aT , g) and generally results in suboptimal policies.
A Motivating Example We provide training results of GCSL in the PointReach task in Figure 2. The objective of the task is to move a point from the starting position to the desired goal as quickly as possible. The offline dataset is collected using a random policy. As shown in Figure 2, GCSL learns a suboptimal policy which detours to reach goals. This is because GCSL only considers the last step reward r(sT , aT , g). As long as the trajectories reaching the goal at the end of the episode, they are all considered equally to GCSL. To improve the learned policy, a straightforward way is to evaluate the importance of samples or trajectories for policy learning using importance weights. As a comparison, Weighted GCSL (WGCSL), which uses a novel weighting scheme, learns the optimal policy in the PointReach task. We will derive the formulation of WGCSL in the following analysis.

heading: Connection between Goal-Conditioned RL and SL
text: GCSL has been proved to optimize a lower bound on the goal-reaching objective (Ghosh et al., 2021), which
is different from the goal-conditioned RL objective that optimizes the cumulative return. In this section, we will introduce a surrogate function for goal-conditioned RL and then derive the connection between goal-conditioned RL and weighted goal-conditioned supervised learning (WGCSL). For
overall consistency, we provide the formulation of WGCSL as below: JWGCSL(π) = Eg∼p(g),τ∼πb(·|g),t∼[1,T ],i∼[t,T ] [ wt,i log πθ(at|st, φ(si)) ] , (1)
where p(g) is the distribution of desired goals and πb refers to the data collecting policy. The weight wt,i is subscripted by both the time step t and the index i of the relabeled goal. When wt,i = 1, JWGCSL reduces to GCSL (Ghosh et al., 2021), and we denote the corresponding objective as JGCSL for comparison convenience. Note that in Eq. 1, φ(si), i ≥ t implies that any future visited states after st can be relabeled as goals. Theorem 1. Assume a finite-horizon discrete MDP, a stochastic discrete policy π which selects actions with non-zero probability and a sparse reward function r(st, at, g) = 1[φ(st) = g], where φ is the state-to-goal mapping and 1[φ(st) = g] is an indicator function. Given trajectories τ = (s1, a1, · · · , sT , aT ) and discount factor γ ∈ (0, 1], let the weight wt,i = γi−t, t ∈ [1, T ], i ∈ [t, T ], then the following bounds hold:
Jsurr(π) ≥ T · JWGCSL(π) ≥ T · JGCSL(π), where Jsurr(π) = 1T Eg∼p(g),τ∼πb(·|g) [∑T t=1 log π(at|st, g) ∑T i=t γ i−1 · 1[φ(si) = g] ] is a surro-
gate function of J(π).
We defer the proof to Appendix B.2, where we also show that under mild conditions, (1) Jsurr is a lower bound of log J , and (2) Jsurr shares the same gradient direction with J at πb. Theorem 1 reveals the connection between the goal-conditioned RL objective and the WGCSL/GCSL objective. Meanwhile, it suggests that GCSL with the discount weight γi−t is a tighter lower bound compared to the unweighted version. We name this weight as discounted relabeling weight (DRW) as it intuitively assigns smaller weights on longer trajectories reaching the same relabeled goal.

heading: 3.2 A MORE GENERAL WEIGHTING SCHEME
text: DRW can be viewed as a weight evaluating the importance of relabeled goals. In this subsection, we consider a more general weighting scheme using a weight function measuring the importance of state-action-goal combination, revealed by the following corollary. Corollary 1. Suppose function h(s, a, g) ≥ 1 over the state-action-goal combination. Given trajectory τ = (s1, a1, · · · , sT , aT ), let wt,i = γi−th(st, at, φ(si)), t ∈ [1, T ], i ∈ [t, T ], then the following bound holds:
Jsurr(π) ≥ T · JWGCSL(π).
The proof are provided in Appendix B.3. Naturally, the Q-value in RL with some constant shift is a candidate of the function family h(s, a, g). Inspired by prior offline RL approaches (Wang et al., 2018; Peng et al., 2019; Nair et al., 2020), we choose h(s, a, g) = exp(A(s, a, g)+C), an exponential function over the goal-conditioned advantage, which we refer to as goal-conditioned exponential advantage weight (GEAW). The constant C is used to ensure that h(s, a, g) ≥ 1. The intuition is that GEAW assigns larger weight for samples with higher values. In addition, the exponential advantage weighted formulation has been demonstrated to be a closed-form solution of an offline RL problem, where the learned policy is constrained to stay close to the behavior policy (Wang et al., 2018). Compared to DRW, GEAW evaluates the importance of state-action-goal samples, taking advantage of the universal value function (Schaul et al., 2015).
To tackle the multi-modality challenge in goal-conditioned RL (Lynch et al., 2020), we further introduce the best-advantage weight (BAW) based on the learned value function. BAW has the following form:
(A(st, at, φ(si))) =
{ 1, A(st, at, φ(si)) > Â
min, otherwise (2)
where Â is a threshold, and min is a small positive value. Implementation details of BAW can be found in Section 4. In our implementation, Â gradually increases. Therefore, BAW gradually leads the policy toward the modal with the highest return rather than sticks to a position between multiple modals, especially when fitting using a Gaussian policy. Combining all three introduced weights, we have the weighting form: wt,i = γi−t exp(A(st, at, φ(si)) + C) · (A(st, at, φ(si))). The constant C only introduces a coefficient independent of t, i for all transitions, therefore, we simply omit it for further analysis.

heading: 3.3 POLICY IMPROVEMENT VIA WGCSL
text: We formally show that combining the three introduced weights, WGCSL can consistently improve the policy learned from the offline dataset. First, we assume there exists a policy πrelabel that can produce the relabeled experiences Drelabel. GCSL is essentially equivalent to imitating πrelabel:
arg minθDKL(πrelabel||πθ) = arg maxθEDrelabel [log πθ(at|st, φ(si)))].
Following (Wang et al., 2018), solving WGCSL is equivalent to imitating a new goal-conditioned policy π̃ defined as:
π̃(at|st, φ(si)) = γi−tπrelabel(at|st, φ(si)) · exp(A(st, at, φ(si))) · (A(st, at, φ(si))),
Then, we have the following proposition: Proposition 1. Under certain conditions, π̃ is uniformly as good as or better than πrelabel. That is, ∀st, φ(si) ∈ Drelabel, i ≥ t, V π̃(st, φ(si)) ≥ V πrelabel(st, φ(si)), where Drelabel contains the experiences after goal relabeling.
The proof can be found in Appendix B.4. Proposition 1 implies that WGCSL can generate a uniformly non-worse policy using the relabeled data Drelabel. To be rigorous, we also need to discuss the relationship between the relabeled return and the original return. In fact, there is a monotonic improvement for the relabeled return over the original one when relabeling with inverse RL (Eysenbach et al., 2020). Below, we provide a simpler relabeling strategy which also offers monotonic value improvement. Proposition 2. Assume the state space can be perfectly mapped to the goal space, ∀st ∈ S, φ(st) ∈ G, and the dataset has sufficient coverage. We define a relabeling strategy for (st, g) as
g′ = { φ(si), i ∼ U [t, T ], if φ(si) 6= g,∀i ≥ t, g, otherwise.
We also assume that when g′ 6= g, the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g′ in the original dataset D. Then, πrelabel is uniformly as good as or better than πb, i.e., ∀st, g,∈ D,V πrelabel(st, g) ≥ V πb(st, g), where πb is the behavior policy forming the dataset D.
The proof is provided in Appendix B.5. In Proposition 2, the relabeling strategy is slightly different from the strategy of relabeling with random future states (Andrychowicz et al., 2017). When there is no successful future states, the relabeling strategy in Proposition 2 randomly samples from the future visited states for relabeling, otherwise it keeps the original goal. For random datasets, there are rare successful states and the relabeling strategy in Proposition 2 acts similarly to relabeling with random future states. Combining Propositions 1 and 2, we know that under certain assumptions, after performing relabeling and weighted supervised learning, the policy can be monotonically improved. In fully offline settings, WGCSL is able to learn a better policy than the policy learned by GCSL and the behavior policy generating the offline dataset.

heading: 4 ALGORITHM
text: In this section, we summarize our proposed algorithm. Denote the relabeled dataset as Drelabel = (st, at, r ′ t, g ′), t ∈ [1, T ], g′ = φ(si), r′t = r(st, at, φ(si)), i ≥ t, and we maximize the following WGCSL objective based on the relabeled data
ĴWGCSL(π) = E(st,at,φ(si))∼Drelabel [ wt,i · log πθ(at|st, φ(si)) ] , (3)
where the weight wt,i is composed of three parts conforming to the general weighting scheme
wt,i = γ i−t · expclip(A(st, at, φ(si))) · (A(st, at, φ(si))) (4)
(1) γi−t is the discounted relabeling weight (DRW) as introduced in Section 3.1.
(2) expclip(A(st, at, φ(si))) = clip(exp(A(st, at, φ(si))), 0,M) is the goal-conditioned exponential advantage weight (GEAW) with a clipped range (0,M ], where M is some positive number for numerical stability. In our experiments, M is set as 10 for all the tasks.
(3) (A(st, at, φ(si))) is the best-advantage weight (BAW) with the form in Eq. 2. In our experiments, Â is set as N percentile of recent advantage values and min is set as 0.05. Motivated by curriculum learning (Bengio et al., 2009), N gradually increases from 0 to 80, i.e., we learn from all the experiences in the beginning stage and converge to learning from samples with the top 20% advantage values.
The advantage value in Eq. 3 can be estimated by A(st, at, g′) = r(st, at, g′) + γV (st+1, g′) − V (st, g ′), and we learn a Q-value function Q(st, at, g′) by minimizing the TD error
LTD = E(st,at,g′,r′t,st+1)∼Drelabel [(r ′ t + γQ̂(st+1, π(st+1, g ′), g′))−Q(st, at, g′))2], where we relabel original goals g as g′ = φ(si) with a probability of 80% and remain the original goal unchanged for the rest 20% time. For simplicity, in our experiments, once it falls into the 80% probability of goal relabeling, we use purely relabeling with random future states, no matter whether there exist successful future states in the trajectory or not, to avoid parsing all the future states as discussed in Appendix B.5. Q̂ refers to the target network which is slowly updated to stabilize training. The same value function training method is adopted in (Andrychowicz et al., 2017; Fang et al., 2019). The value function V (st, g′) is estimated using Q-value function as V (st, g′) = Q(st, π(st+1, g ′), g′). The entire algorithm is provided in Appendix C.

heading: 5 EXPERIMENTS
text: In our experiments, we demonstrate our proposed method on a diverse collection of continuous sparse-reward goal-conditioned tasks.

heading: 5.1 ENVIRONMENTS AND EXPERIMENTAL SETTINGS
text: As shown in Figure 3, the introduced benchmark includes two point-based environments, and eight simulated robot environments. In all tasks, the rewards are sparse and binary: the agent receives a reward of +1 if it achieves a desired goal and a reward of 0 otherwise. More details about those environments are provided in Appendix F. For the evaluation of offline goal-conditioned algorithms, we collect two types of offline dataset, namely ’random’ and ’expert’, following prior offline RL work (Fu et al., 2020). Each dataset has the sample size of 2 × 106 for hard tasks (FetchPush, FetchSlide, FetchPick and HandReach) and 1 × 105 for other relatively easy tasks. The ’random’ dataset is collected by a uniform random policy, while the ’expert’ dataset is collected by the final policy trained using online HER. We also add Gaussian noise with zero mean and 0.2 standard deviation to increase the diversity of the ’expert’ dataset. This may lead to the consequence that the deterministic policy learned by behavior cloning achieves a higher average return than that in the dataset, which has also been observed in previous works (Kumar et al., 2019; Fujimoto et al., 2019). More details about the two datasets are provided in Appendix D. Beyond the offline setting, we also evaluate our method in the online setting and the results are reported in Appendix E.8.
As for the implementation of GCSL and WGCSL, we utilize the Diagonal Gaussian policy with a mean vector πθ(st, g) and a constant variance σ2 for continuous control. In the offline setting, σ does not need to be defined explicitly because we evaluate the policy without randomness. The loss function of GCSL can be rewritten as:
LGCSL = E(st,at,φ(si))∼Drelabel [‖πθ(st, φ(si))− at‖ 2 2].
Our implemented WGCSL learns a policy to minimize the following loss
LWGCSL = E(st,at,φ(si))∼Drelabel [wt,i · ‖πθ(st, φ(si))− at‖ 2 2].
where wt,i is defined in Eq 4. The policy networks of GCSL and WGCSL are 3-layer MLP with 256 units each layer and relu activation. Besides, WGCSL learns a value network with the same structure except for the input and output layers. The batch size is set as 128 for the first 6 tasks, and 512 for 4 harder tasks from FetchPush to HandReach. We use Adam optimizer with a learning rate of 5 × 10−4. For all the experiments we repeat for 5 different random seeds and report the average evaluation performance with standard deviation.

heading: 5.2 EXPERIMENTAL RESULTS
text: In the offline goal-conditioned setting, we compare WGCSL with GCSL, HER and modified versions of most related offline works such as goal-conditioned MARWIL (Wang et al.,
2018) (Goal MARWIL), goal-conditioned behavior cloning (Bain & Sammut, 1995) (Goal BC), goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019) (Goal BCQ), and goalconditioned Conservative Q-learning (Kumar et al., 2020) (Goal CQL). The results are presented in Table 1 and Figure 4. Additional comparison results with more baselines and results on 10% dataset are provided in Appendix E.
Expert Datasets The top row of Figure 4 reports the performance of different algorithms using the expert datasets of four hard tasks. In all these tasks, WGCSL outperforms other baselines in terms of both learning efficiency and final performance. GCSL converges very slowly but can finally reach beyond the average return of expert dataset in FetchSlide and FetchPick tasks. Goal MARWIL is more efficient than GCSL and achieves the second best performance. However, none of the baseline algorithms succeeds in the HandReach task, which might be due to high-dimensional states, goals, actions and the noise used to collect the dataset. In addition, HER itself has the ability to handle offline dataset as the relabeled data is far off-policy (Plappert et al., 2018). But we can conclude from Table 1 that the performance of HER is unstable and inconsistent across different tasks. In contrast, learning curves of WGCSL are consistently stable. It is also verified that techniques used in WGCSL are effective to generate better policies than original and relabeled datasets in offline goal-conditioned RL setting.
Random Datasets The bottom row of Figure 4 shows the training results of different methods on random datasets. We can observe that WGCSL outperforms other baselines by a large margin. Baselines such as Goal MARWIL, Goal BC, Goal BCQ and Goal CQL can hardly learn to reach goals with the random datasets. GCSL and HER can learn relatively good policies in some tasks, which proves the importance of goal relabeling. Interestingly, in tasks such as Reacher, SawyerReach and FetchReach, results of WGCSL and HER on the random dataset exceeds that of the expert dataset. The possible reason is that the data coverage of random dataset could be larger than the expert dataset in these tasks, and the agent can learn to reach more goals via goal relabeling.
Ablation Studies We also include ablation studies for offline experiments in Figure 5. In our ablation studies, we investigate the effect of the three proposed weights, i.e., DRW, GEAW and BAW, by applying them to GCSL respectively. The results demonstrate that all three weights are effective compared to the plain GCSL, while GEAW and BAW show larger benefits. This is reasonable since GEAW and BAW learn additional value networks while DRW does not. Moreover, the results suggest that the learned policy can be improved by combining these weights.
Value Estimation Off-policy RL algorithms are prone to distributional shift and suffer from value overestimation problem on out-of-distribution actions. Such problem can be exacerbated in tasks with high-dimensional states and actions. As shown in Figure 6, DDPG and HER exhibit large estimated values during offline training. In contrast, WGCSL has a more robust value approximation through weighted supervised learning, thus achieving higher returns in the difficult HandReach task.

heading: 6 RELATED WORK
text: Solving goal-conditioned RL task is an important yet challenging domain in reinforcement learning. In those tasks, agents are required to achieve multiple goals rather than a single task (Schaul et al., 2015). Such setting puts much burden on the generalization ability of the learned policy when dealing with unseen goals. Another challenge arises in goal-conditioned RL is the sparse reward (Plappert et al., 2018). HER (Andrychowicz et al., 2017) tackles the sparse reward issue via relabeling the failed rollouts as successful ones to increase the proportion of successful trails during learning. HER is then extended to deal with demonstration (Nair et al., 2018; Ding et al., 2019) and dynamic goals (Fang et al., 2018), and combined with curriculum learning for exploration-exploitation trade-off (Fang et al., 2019). Eysenbach et al. (2020) and Li et al. (2020) further improved the goal sampling strategy from the perspective of inverse RL. Zhao et al. (2021) encouraged agents to control their states to reach goals by optimizing a mutual information objective without external rewards. Different from prior works, we consider the offline setting, where the agent cannot access the environment to collect data in the training phase.
Our work has strong connections to imitation learning (IL). Behavior cloning is one of the simplest IL methods that learns a policy predicting the expert actions (Bain & Sammut, 1995; Bojarski et al., 2016). Inverse RL (Ng et al., 2000) extracts a reward function from demonstrations and then uses the reward function to train policies. Several works also leverage self-imitation to improve the stability and efficiency of RL without expert experience. Self-imitation learning (Oh et al., 2018) imitates the agent’s past advantageous decisions and indirectly drives exploration. Sun et al. (2019) introduced a dynamic programming method to learn policies progressively via self-imitation. Lynch et al. (2020) employed VAE to jointly learn a latent plan representation and goal-conditioned policy from replay without the need for rewards. Our work is most relevant to GCSL (Ghosh et al., 2021), which iteratively relabels and imitates its own collected experiences. The difference is that we further generalize GCSL to the offline goal-conditioned RL setting via weighted supervised learning.
Our work belongs to the scope of offline RL. Learning policies from static offline datasets is challenging for general off-policy RL algorithms due to error accumulation with distributional shift (Fujimoto et al., 2019; Kumar et al., 2019). To address such issue, a series of algorithms are proposed to constrain the policy updates to avoid excessive deviation from the data distribution (Wang et al., 2018; Fujimoto et al., 2019; Wu et al., 2019; Yang et al., 2021). MARWIL (Wang et al., 2018) can learn a better policy than the behavior policy of the dataset with theoretical guarantees. In addition to policy regularization, other works leverage techniques such as value underestimation (Kumar et al., 2020; Yu et al., 2020), robust value estimation (Agarwal et al., 2020), data augmentation (Sinha & Garg, 2021; Wang et al., 2021), and Expectile V -Learning (Ma et al., 2021) to handle distribution shift. As offline RL can easily overfit to one task, generalizing offline RL to unseen tasks can be even more challenging (Li et al., 2021). Li et al. (2019) tackled the challenge by leveraging the triplet loss for robust task inference. Our method has similar exponential advantage weighted form as MARWIL, however, we tackle goal-conditioned tasks and introduces a general weighting scheme for WGCSL. A most recent work (Chebotar et al., 2021) also addresses the offline goal-conditioned problem through underestimating the value of unseen actions. The difference is that WGCSL is a weighted supervised learning method without explicitly value underestimation. Therefore, WGCSL is substantially simpler, more stable, and more applicable for real-world tasks.

heading: 7 CONCLUSION
text: In this paper, we have proposed a novel algorithm, Weighted Goal-Conditioned Supervised Learning (WGCSL), to solve offline goal-conditioned RL problems with sparse rewards, by first revisiting GCSL and then generalizing it to offline goal-conditioned RL. We further derive theoretical guarantees to show that WGCSL can generate monotonically improved policies under certain conditions. Experiments on an introduced benchmark demonstrate that WGCSL outperforms current offline goal-conditioned approaches by a great margin in terms of learning efficiency and convergent performance. For future directions, we are interested in building more stable and efficient offline goal-conditioned RL algorithms based on WGCSL for real-world applications.

heading: ACKNOWLEDGMENTS
text: This work was partly supported by the Science and Technology Innovation 2030-Key Project under Grant 2021ZD0201404, in part by Science and Technology Innovation 2030 – “New Generation Artificial Intelligence” Major Project (No. 2018AAA0100904) and National Natural Science Foundation of China (No. 20211300509). Part of this work was done when Rui Yang and Hao Sun worked as interns at Tencent Robotics X Lab.

heading: ETHICS AND REPRODUCIBILITY STATEMENTS
text: Our work provides a way to learn general policy from offline data. Learning policy for general purpose is one of the fundamental challenges in artificial general intelligence (AGI). The proposed offline goal-conditioned algorithm can provide new insights into training agents with diverse skills, but currently we cannot foresee any negative social impact of our work. The algorithm is evaluated on simulated robot environments, thus our experiments would not suffer from discrimination/bias/fairness concerns. The details of experimental settings and additional results are also provided in the Appendix. All the implementation code and offline dataset are available in our code link: https://github.com/YangRui2015/AWGCSL.

heading: A A COMPREHENSIVE SUMMARY FOR WGCSL
text: WGCSL is strongly efficient in the offline goal-conditioned RL due to the following advantages:
(1) WGCSL learns a universal value function (Schaul et al., 2015) which is more generalizable than the value function learned in a single task, thus alleviating the overfitting problem in offline RL.
(2) WGCSL utilizes goal relabeling which contributes to augment huge amount of data ( ( T 2 ) times) for offline training and meanwhile alleviates the sparse reward issue.
(3) WGCSL leverages weighted supervised learning to reweight the action distribution in the dataset, avoiding the out-of-distribution action problem in offline RL.
(4) In the three weights, GEAW keeps the learned policy close to the offline dataset implicitly and BAW handles the multi-modality problem in offline goal-conditioned RL. WGCSL utilizes the three weights to learn a better policy with theoretical guarantees.

heading: B THEORETICAL RESULTS
text: In our analysis, we will assume finite-horizon MDPs with discrete state and goal spaces.

heading: B.1 NON-OPTIMALITY OF RELABELING STRATEGY
text: Given a trajectory τ = {s1, a1, . . . , sT , aT }, we show that under general offline goal-conditioned RL setting with the sparse reward function r(st, at) = 1[φ(st) = g], relabeling with future states is not optimal. This is a direct result from Eq. 7 of (Eysenbach et al., 2020), which has proven that the optimal relabelling strategy is
q(g|τ) ∝ p(g) exp ( T∑ t=1 γt−1r(st, at)− logZ(g) ) = p(g) exp ( T∑ t=1 γt−1 · 1[φ(st) = g]− logZ(g) ) .
This is different from simply relabeling with the last state as q(g|τ) = 1[g = sT ] in Section 4.1 of (Eysenbach et al., 2020) and relabeling with future states as q(g|st, at) = 1(T−t+1)1[g = si], t ∈ [1, T ], i ∈ [t, T ].

heading: B.2 PROOF OF THEOREM 1
text: Theorem. Assume a finite-horizon discrete MDP, a stochastic discrete policy π which selects actions with non-zero probability and a sparse reward function r(st, at, g) = 1[φ(st) = g], where φ is the state-to-goal mapping and 1[φ(st) = g] is an indicator function. Given trajectory τ = (s1, a1, · · · , sT , aT ) and discount factor γ ∈ (0, 1], let the weight wt,i = γi−t, t ∈ [1, T ], i ∈ [t, T ], then the following bounds hold:
Jsurr(π) ≥ T · JWGCSL(π) ≥ T · JGCSL(π),
where
Jsurr(π) = 1
T · Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 log π(at|st, g) T∑ i=t γi−1 · 1[φ(si) = g] ]
is a surrogate function of J(π).

heading: Proof.
text: Jsurr(π) = 1
T · Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 log π(at|st, g) T∑ i=t γi−1 · 1[φ(si) = g] ]
≥ 1 T · Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 log π(at|st, g) T∑ i=t γi−t · 1[φ(si) = g] ]
= Eg∼p(g),τ∼πb(·|g),t∼[1,T ] [ T∑ i=t γi−t log π(at|st, φ(si)) · 1[φ(si) = g] ] ≥ T · Eg∼p(g),τ∼πb(·|g),t∼[1,T ],i∼[t,T ] [ γi−t log π(at|st, φ(si)) · 1[φ(si) = g]
] ≥ T · Eg∼p(g),τ∼πb(·|g),t∼[1,T ],i∼[t,T ] [ γi−t log π(at|st, φ(si)) ] , T · JWGCSL(π) (5)
≥ T · Eg∼p(g),τ∼πb(·|g),t∼[1,T ],i∼[t,T ] [log π(at|st, φ(si))] , T · JGCSL(π). Eq. 5 holds because of the non-positive logarithmic function over π. Compared to the surrogate function used in Appendix B.1 of (Ghosh et al., 2021), Jsurr(π) further considers the discounted cumulative return. The theorem shows that WGCSL optimizes over a tighter lower bound of Jsurr(π) compared to GCSL.
Remark. While Theorem 1 bounds the surrogate function Jsurr with JWGCSL and JGCSL, here we bridge the gap between Jsurr and the goal-conditioned RL objective J .
We first show that log J is an upper bound of Jsurr in addition to KL divergence and a constant. Let R(τ) = ∑T i=1 γ i−1 · 1[φ(si) = g] and C(g) = ∫ πb(τ |g)R(τ)dτ , and assume C(g) > 0. Since practically g is sampled from offline dataset, we can find Cm = ming C(g). Similarly to (Kober & Peters, 2014), we have following results using Jensen’s inequality and absorbing constants into ’constant’:
log J(π) = logEg∼p(g)[ ∫ π(τ |g)R(τ)dτ ]
≥ Eg∼p(g)[logC(g) · ∫
πb(τ |g) C(g) · πb(τ |g) π(τ |g)R(τ)dτ ]
≥ Eg∼p(g)[ ∫ πb(τ |g)R(τ)
C(g) (log π(τ |g)− log πb(τ |g))dτ ] + constant
≥ Eg∼p(g),τ∼πb(·|g)[ 1
C(g) log π(τ |g) T∑ i=1 γi−1 · 1[φ(si) = g]] + constant
≥ 1 Cm · Eg∼p(g),τ∼πb(·|g)[ T∑ t=1 log π(at|st, g) T∑ i=1 γi−1 · 1[φ(si) = g]] + constant
= T
Cm · Jsurr(π) +
1
Cm · Eg∼p(g),τ∼πb(·|g)[ T∑ t=1 log π(at|st, g) t−1∑ i=1 γi−1 · 1[φ(si) = g]] + constant
≥ T Cm · Jsurr(π) + T Cm · Eg∼p(g),τ∼πb(·|g)[ T∑ t=1 log π(at|st, g)] + constant
= T
Cm · Jsurr(π) +
T 2 Cm · Eg∼p(g),(st,at)∼πb(·|g)[log π(at|st, g)] + constant
= T
Cm · Jsurr(π) +
T 2 Cm · Eg∼p(g),(st,at)∼πb(·|g)[log π(at|st, g)− log πb(at|st, g)] + constant
= T
Cm · Jsurr(π)−
T 2 Cm · Eg∼p(g),st∼πb(·|g)[DKL(πb(·|st, g)‖π(·|st, g))] + constant.
Naturally, in offline setting, we explicitly or implicitly bound the KL divergence between πb and π via policy constraints or (weighted) supervised learning. Therefore, we can regard Jsurr as a lower bound of log J .
Furthermore, We show that Jsurr(π) shares the same gradient with J(π) when π = πb after scaling with 1T .
J(π) = Eg∼p(g),τ∼π(·|g) [ T∑ i=1 γi−1 · 1[φ(si) = g] ] ,
∇J(πb) = Eg∼p(g) [∫ τ ∇πb(τ |g) T∑ i=1 γi−1 · 1[φ(si) = g]dτ ]
= Eg∼p(g) [∫ τ πb(τ |g)∇ log πb(τ |g) T∑ i=1 γi−1 · 1[φ(si) = g]dτ ]
= Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 ∇ log πb(at|st, g) T∑ i=1 γi−1 · 1[φ(si) = g] ]
= Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 ∇ log πb(at|st, g) T∑ i=t γi−1 · 1[φ(si) = g] ] = T · ∇Jsurr(πb).
Therefore, taking a gradient update with a proper step size on Jsurr can also improve the policy on the objective J . In our algorithm, the exponential weight poses an implicit constraint on the policy update similar to (Wang et al., 2018; Nair et al., 2020). With a proper constraint, optimizing on the Jsurr is equivalent as optimizing on the goal-conditioned objective J . Intuitively, Jsurr maximizes the likelihood for state-action pairs with large returns, and therefore it can lead to policy improvement based on the offline dataset.

heading: B.3 PROOF OF COROLLARY 1
text: Corollary. Suppose function h(s, a, g) ≥ 1 over the state-action-goal combination. Given trajectory τ = (s1, a1, · · · , sT , aT ), let wt,i = γi−th(st, at, φ(si)), t ∈ [1, T ], i ∈ [t, T ], then the following bounds hold:
Jsurr(π) ≥ T · JWGCSL(π).
Proof. Similarly to the proof of Theorem 1, we have
Jsurr(π) = 1
T Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 log π(at|st, g) T∑ i=t γi−1 · 1[φ(si) = g] ]
≥ 1 T Eg∼p(g),τ∼πb(·|g) [ T∑ t=1 log π(at|st, g) T∑ i=t γi−th(st, at, φ(si)) · 1[φ(si) = g] ] ≥ T · Eg∼p(g),τ∼πb(·|g),t∼[1,T ],i∼[t,T ] [ γi−th(st, at, φ(si)) log π(at|st, φ(si))
] , T · JWGCSL(π).

heading: B.4 PROOF OF PROPOSITION 1
text: Proposition. Under certain conditions, π̃ is uniformly as good as or better than πrelabel. That is, ∀st, φ(si) ∈ Drelabel and i ≥ t, V π̃(st, φ(si)) ≥ V πrelabel(st, φ(si)), where Drelabel contains the experiences after goal relabeling.
Proof. We assume the advantage value is for the behavior policy of the relabeled data, i.e., πrelabel. We consider the compound state ŝ = (st, φ(si)) from the relabeled distribution Drelabel. When ŝ is fixed (the index of i, t are fixed), γi−t is a constant and we have:
f(ŝ, A(ŝ, at)) = clip(A(ŝ, at),−∞, logM) + log(γi−t) + log( (A(ŝ, at))) + C(ŝ)
f(ŝ, ·) is a monotonically non-decreasing function for fixed ŝ, where C(ŝ) is a normalizing factor, M is the positive maximum exponential advantage value, and (A(ŝ, at)) is also a non-decreasing function introduced in Section 3.2 with minimum min > 0. Leveraging the Proposition 1 in MARWIL (Wang et al., 2018), we can conclude that V π̃(ŝ) ≥ V πrelabel(ŝ).
Remark. The above proposition assumes the advantage value is for πrelabel. However, we empirically find that using advantage value of the learned policy π (the value function is learned to minimize LTD in Section 4) works better in offline goal-conditioned RL setting. As analyzed in prior works (Nair et al., 2020; Yang et al., 2021), using the value function of π is equivalent to solving the following problem:
πk+1 = arg max π
Eg∼p(g),a∼π(·|s,g)[Aπk(s, a, g)], s.t. DKL(π(·|s, g)‖πrelabel(·|s, g)) ≤ .
In addition, for offline goal-conditioned RL, learning policies with inter-trajectory information is important for reaching diverse goals. Therefore, usingQ(st+1, π(st+1, g′), g′) to compute TD target helps value estimation across trajectories compared with Q(st+1, πrelabel(st+1, g′), g′). Moreover, π is trained via weighted supervised learning to avoid the OOD action problem.

heading: B.5 PROOF OF PROPOSITION 2
text: Proposition. Assume the state space can be perfectly mapped to the goal space, ∀st ∈ S, φ(st) ∈ G, and the dataset has sufficient coverage. We define a relabeling strategy for (st, g) as
g′ = { φ(si), i ∼ U [t, T ], if φ(si) 6= g,∀i ≥ t, g, otherwise.
We also assume that when g′ 6= g, the strategy only accepts relabeled trajectories with higher future returns than any trajectory with goal g′ in the original dataset D. Then, πrelabel is uniformly as good as or better than πb, i.e., ∀st, g,∈ D,V πrelabel(st, g) ≥ V πb(st, g), where πb is the behavior policy forming the dataset D.
Proof. LetD = {τj = (si, ai, gj)Ti=1}Nj=1. We assume that the data ofD andDrelabel are sufficient and cover the full state and goal space, i.e., ⋃ (si, gj) = S × G. Note that every trajectory has only a single goal, and it does not have to reach the goal within the trajectory. Therefore, we can partition D into two subsets Dp (positive set) and Dn (negative set):
Dp = {τ = (si, ai, gj)Ti=1|∃si ∈ τ, φ(si) = gj}, Dn = {τ = (si, ai, gj)Ti=1|∀si ∈ τ, φ(si) 6= gj},
whereDp contains trajectories that can reach the goal andDn contains trajectories that cannot reach the goal. Let D(st, g) = {τ = (si, ai, g)Ti=t|τ begin with st}, and similarly we define the partition
Dp(st, g) = {τ = (si, ai, g)Ti=t|τ begin with st,∃si ∈ τ, φ(si) = g}, Dn(st, g) = {τ = (si, ai, g)Ti=t|τ begin with st,∀si ∈ τ, φ(si) 6= g}.
Hence, the value V πb can be written as
V πb(st, g) = Eτ∼D(st,g)[ T∑ j=t γj−t · 1[φ(sj) = g]] , Eτ∼D(st,g)[R τ t ]
= |Dp(st, g)| |D(st, g)| Eτ∼Dp(st,g)[R τ t ] + |Dn(st, g)| |D(st, g)| Eτ∼Dn(st,g)[R τ t ] = |Dp(st, g)| |D(st, g)| Eτ∼Dp(st,g)[R τ t ].
Given τ = (si, ai, g)Ti=t ∈ D, we relabel the future trajectory as τrelabel = (si, ai, g′)Ti=t for value estimation following the strategy
g′ = { φ(sj), j ∼ U [t, T ], if ∀j ≥ t, φ(sj) 6= g g, otherwise ,
and when g′ 6= g, the strategy only accepts τrelabel with higher return than any trajectory in D(st, g
′). Let Drelabel(st, g) = {τrelabel = (si, ai, g)Ti=t|τrelabel begin with st} be the relabeled trajectories with g as goal. Hence, the value V πrelabel can be written as
V πrelabel(st, g) = Eτrelabel∼Drelabel(st,g)[R τrelabel t ].
For any τrelabel ∈ Drelabel(st, g), we enumerate the situation of its counterpart τ without relabeling.
1. τ ∈ Dp(st, g). Following the relabeling strategy, we have g′ = g, such that τrelabel = τ .
2. τ /∈ Dp(st, g), then τ has another goal g̃ 6= g, but it reaches g at some intermediate state. It holds that Rτrelabelt > 0 = R τ t . Following the relabeling strategy, we further have
Rτrelabelt ≥ Rτ ′ t ,∀τ ′ ∈ D(st, g), which guarantees improvements on expected value of all trajectories starting from st with goal g.
Therefore, we have
V πrelabel(st, g) = Eτrelabel∼Drelabel(st,g)[R τrelabel t ]
≥ Eτ∼Dp(st,g)[R τ t ] ≥ |D p(st, g)| |D(st, g)| Eτ∼Dp(st,g)[R τ t ] = V πb(st, g).
Remark. The relabeling strategy of Proposition 2 requires a complete scan of future states of trajectories, which is costly. Empirically we find that relabeling with random future states as HER has comparable performance, and is very simple and efficient to implement. We provide the comparison between the relabeling strategy of Proposition 2 (called WGCSL-slow-relabel) and the relabeling strategy that we used in practice in Table 2.

heading: C ALGORITHM
text: We summarize our proposed approach WGCSL into Algorithm 1.

heading: D OFFLINE SETTINGS
text: Offline Datasets In the offline setting, we only use collected datasets to train agents, without additional interaction with the environment. The difference between the offline goal-conditioned RL dataset and the general offline RL dataset is that it also needs to save the original desired goals g and the achieved goals φ(st). We follow previous works (Fujimoto et al., 2019; Fu et al., 2020) to introduce two different types of datasets:
Algorithm 1: Weighted Goal-Conditioned Supervised Learning Given offline dataset D, the best advantage percentile threshold N ; Initialize policy πθ and value function Q; Initialize a First-In-First-Out queue B = {} to save recent advantage values; for training step = 1, 2, . . . ,M do
Sample a mini-batch from the offline dataset: {(st, at, g, rt)} ∼ D; Relabel the mini-batch using future goals: {(st, at, φ(si), r(st, at, φ(si))), i ≥ t} ∼ Drelabel; Update value function Q to minimize TD error LTD on the relabeled mini-batch; Estimate advantage value A(st, at, φ(si)) using value function Q; Update queue B with the estimated advantage values; Get the N percentile advantage value Â from B to compute the BAW; Update policy πθ to maximize the objective in Eq. 3 with the relabeled mini-batch:
JWGCSL = E(st,at,si)∼Drelabel [γ i−t log πθ(at|st, φ(si))expclip(A(st, at, φ(si)))· (A(st, at, φ(si)))]
end for

heading: Dataset PointReach PointRooms Reacher SawyerReach SawyerDoor
text: For the dataset of six easy tasks (PointReach, PointRooms, Reacher, SawyerReach, SawyerDoor, FetchReach), we save a dict of 1 × 105 transitions (2000 trajectories) containing observations, actions, desired goals and achieved goals. Regarding the other four hard tasks (FetchPush, FetchSlide, FetchPick, HandReach), we store 2× 106 transitions (40000 trajectories) in the offline dataset. The rewards can be computed by leveraging the corresponding interface (namely env.compute reward) of the environment. The average returns of the two types of datasets are shown in Table 3. Generally, average returns of the expert datasets are close to that of the optimal policy, while average returns of the random datasets are close to 0.
Baseline Introduction In the offline goal-conditioned setting, we denote the offline dataset as D and consider the following baselines:
• GCSL: GCSL (Ghosh et al., 2021) without online interaction, only using offline samples after relabeling (denoted as Drelabel) to maximize
JGCSL(π) = E(st,at,φ(si))∼Drelabel [log π(at|st, φ(si))].
• Goal MARWIL: goal-conditioned MARWIL (Wang et al., 2018), using offline samples to maximize
JMARWIL = E(st,at,g)∼D[log π(at|st, g) exp( 1
β A(st, at, g))].
To fairly compare with our method, Goal MARWIL is implemented with actor-critic style as AWAC (Nair et al., 2020) and WGCSL, therefore it can also be treated as ’Goal AWAC’. Besides, we also clip the exponential advantage value as WGCSL for numerical stability.
• Goal AWR: goal-conditioned AWR (Peng et al., 2019), using offline samples to maximize
JAWR(π) = E(st,at,g)∼D[log π(at|st, g) exp( 1
β (Rt − V (st, g)))],
where Rt = ∑T−t i=0 γ
irt+i. Different from the original AWR that uses TD(λ) or Monte Carlo return Rt to train the value function, we use TD loss to learn value function as WGCSL. Empirically, we find TD target works better than Monte Carlo in our setting.
• Goal BC: goal-conditioned behavior cloning (Bain & Sammut, 1995), using offline samples to maximize
JBC(π) = E(st,at,g)∼D[log π(at|st, g)].
• Goal BCQ: goal-conditioned Batch-Constrained Q-Learning (Fujimoto et al., 2019), which we use the official codebase and concatenate observations, desired goals and achieved goals as states. The objective function of Goal BCQ is as follows:
JBCQ(ξ) = E(st,g)∼D,a∼Gw(st,g)[Q(st, a+ ξ(st, a, g,Φ), g)],
where ξ(st, a, g,Φ) is the goal-conditioned perturbation model, which outputs an adjustment to an action awithin the range [−Φ,Φ]. Gw is the VAE fitted on the behavior policy of the offline dataset. The policy is optimized through optimizing ξ. TheQ function is learned using the Clipped Double Q-learning in (Fujimoto et al., 2019), and the only difference is that the Q function also includes goal g as input.
• Goal CQL: goal-conditioned Conservative Q-Learning (Kumar et al., 2020), which we also use the official codebase and concatenate observations, desired goals and achieved goals as states. The objective function of Goal CQL is as follows:
JCQL(π) = E(st,g)∼D,a∼π(·|st,g)[Q(st, a, g)− log π(a|st, g)],
which jointly optimizes estimated return and policy entropy. The Q function of CQL is learned by minimizing the following loss:
LCQL =αE(st,g)∼D [ log ∑ a exp(Q(st, a, g))− Ea∼πb(·|st,g)[Q(st, a, g)] ]
+ 1
2 E(st,at,st+1,g)∼D
[ (Q(st, at, g)− BπQ(st, at, g))2 ] ,
where πb is the behavior policy for the offline dataset and Bπ is the Bellman operator.
• HER: Hindsight Experience Replay (Andrychowicz et al., 2017), using offline samples after relabeling to maximize
JHER(π) = E(st,at,g′)∼Drelabel [Q(st, π(st, g ′), g′)],
where g′ = φ(si), i ≥ t and the Q function is learned by minimizing TD error:
LTD = E(st,at,st+1,g′)∼Drelabel(rt + γQ(st+1, π(st+1, g ′), g′)−Q(st, at, g′))2.
Implementation of Baselines For fair comparison, all the implementation of WGCSL, GCSL, Goal MARWIL, Goal AWR, Goal BC, and HER use the Diagnal Gaussian policy with a mean vector πθ(s, g) and a non-zero constant variance σ2, where σ is set as 0.2. WGCSL, Goal MARWIL, Goal AWR, and HER share the same network structure, i.e., the value function and the policy network (along with their target networks) are both 3-layer MLPs with 256 units each layer and relu activation. Besides, GCSL and Goal BC both keep a single policy network without target network and value function. We also normalize the observations and goals with estimated mean and standard deviation for WGCSL and other baselines, which is helpful for offline multi-goal manipulation tasks. The parameter β in Goal MARWIL and Goal AWR are set as 1 by default. As regards to Goal BCQ and Goal CQL, we use the same batch size as other algorithms and other parameters are set as default (we find that default parameters perform equally well, if not better than others, in the grid search of critical parameters). All other hyper-parameters of different algorithms are kept the same for each task, such as batch size (128 for the first 6 tasks and 512 for 4 harder taks), discount factor γ = 0.98, and Adam optimizer with learning rate 5× 10−4.
Implementation of WGCSL The basic implementation is introduced as above. To compute the best-advantage weight for WGCSL, we keep a First-In-First-Out queue B of size 5 × 104 to store recent calculated advantage values, and the percentile threshold N gradually increases from 0 to 80. For each training step, N increases by 0.01 (for 4 harder tasks from FetchPush to HandReach) or 0.15 (for other tasks). The maximum clipping upper bound M for WGCSL is set as 10. Though DRW brings improvements for most of the tasks on expert and random datasets, we find DRW slightly affects the performance of harder tasks on random datasets, which may be because in the random datasets we need more timesteps to obtain enough coverage of achieved goals, especially for harder tasks. Tuning γ for DRW trade-offs between the value and the effective coverage of relabeling goals, and excluding DRW is equivalent to setting γ = 1 for DRW. Therefore, we only use GEAW and BAW for these tasks on random datasets.
Evaluation Settings For evaluation, we evaluate all the algorithms for 100 episodes without randomness. We repeat for 5 different random seeds and report the average evaluation performance with standard deviation.

heading: E ADDITIONAL EXPERIMENTAL RESULTS
text: In this section, we provide more empirical results in both offline and online goal-conditioned RL setting, including average return, final distance of the offline setting, results on small (10%) dataset, and the results of online experiments.

heading: E.1 FULL OFFLINE RESULTS
text: We provide the average return curves of all tasks in Figure 7. In these results, WGCSL performs consistently well across ten environments, especially when handling hard tasks (FetchPush, FetchSlide, FetchPick, HandReach) and learning from the random offline dataset. Other conclusions have been discussed in Section 5.2. Comparing AWR with MARWIL, the major difference is using Monte Carlo return to replace the TD estimation for learning the policy. Hence, we can find that TD bootstrap learns more efficiently in weighted supervised learning, which is similar to the conclusion of prior work (Nair et al., 2020). In addition, we present all final performance of different algorithms in Table E.9.

heading: E.2 COMPARISON RESULTS WITH ACTIONABLE MODELS, HER AND DDPG
text: In this subsection, we make comparison with Actionable Models (AM) (Chebotar et al., 2021), which applies the conservative estimation idea similar to (Kumar et al., 2020) into offline goalconditioned RL. There are some different settings between AM and ours. For example, we consider cumulative return of the maximum horizon, i.e., the agent doesn’t terminate the interaction after reaching the goal. To fairly compare with AM, we implement AM based on HER. Specifically, we
revise the value update as:
LAM = E(st,at,g′,r′t,st+1)∼Drelabel [(r ′ t + γQ̂(st+1, π(st+1, g ′), g′))−Q(st, at, g′))2] + E(st,g′)∼Drelabel,a∼exp(Q)[Q(st, a, g ′)]
In the formula, AM tries to minimize TD error and minimize unseen Q values at the same time. The action a is sampled according to exponential Q values. In our implementation, we sample 20 actions from the action space and compute their exponential Q value in order to sample a.
The results are shown in Figure 8. We can observe that DDPG and HER are unstable during training, with large variance and performance drops at the end. In both expert and random dataset, HER outperforms DDPG, which demonstrates the benefits of goal relabeling for offline goal-conditioned RL. Besides, Actionable Model (AM) achieves higher final performance than HER and DDPG especially in expert dataset, which indicates the conservative estimation technique is useful for offline goal-conditioned RL. Moreover, WGCSL is more efficient than AM, especially in harder tasks with large action space. Intuitively, WGCSL promotes the probability of visited actions, while AM restrains the probability of unvisited actions. For tasks with large action space, conservative strategy can be ineffective compared to WGCSL.

heading: E.3 NORMALIZED SCORE ACROSS 10 TASKS
text: For clarity, we report the normalized score across 10 tasks to compare different algorithms in Figure 9. We can observe the significant improvement of WGCSL over other baselines. We take an insight into the impact of the goal relabeling technique. For expert datasets, goal relabeling is not a dominant factor, e.g., the learning curve of Goal BC and GCSL are very close. In random datasets, goal relabeling is essential to achieve a high performance. In addition to goal relabeling, WGCSL contains other techniques to jointly promote the performance in offline goal-conditioned RL tasks. Other conclusions are the same as prior subsections (Appendix E.1 and Appendix E.2).

heading: E.4 COMPARISON OF TRAINING TIME
text: We further analyze the computation cost of the four most effective algorithms in the random offline setting, i.e., WGCSL, GCSL, HER and Actionable Models (AM). The results are shown in Figure 10, from which we can conclude that GCSL needs the minimal training time, while WGCSL and HER require similar cost. It needs to be emphasized that WGCSL has only a small amount of additional training time over GCSL, but it achieves significantly higher performance. Therefore, we believe that the cost of learning the value function on top of GCSL is worthwhile. Moreover, AM requires the largest computation among these algorithms, which is mainly because it needs to
sample several actions and compute their exponential Q value for constraining the large Q value. These procedures make AM the most time-consuming algorithm. In contrast, WGCSL can achieve higher performance with less training time. The training time would vary across different platforms, and we use one single GPU (Tesla P100 PCIe 16GB) and 5 cpu cores (Intel Xeon E5-2680 v4 @ 2.40GHz) to run 5 random seeds parallelly.

heading: E.5 AVERAGE PROBABILITY OF IMPROVEMENT
text: In this subsection, we adopt the average probability of improvement (Agarwal et al., 2021), a robust metric to measure how likely it is for one algorithm to outperform another on a randomly selected task. The results are reported in Figure 11. As shown in the results, WGCSL robustly outperforms other baselines in both expert and random datasets. For instance, WGCSL is 100% better than 4 baselines on the expert dataset, and 6 baselines on the random dataset. Note that the most two effective and robust algorithms on both random and expert datasets are WGCSL and Actionable Models (AM), which are specifically designed for offline goal-conditioned RL. Comparing the two algorithms, WGCSL outperforms AM with a probability of 75% on the expert dataset and 62% on the random dataset.

heading: E.6 FINAL DISTANCES IN THE OFFLINE SETTING
text: GCSL (Ghosh et al., 2021) uses the average final distance to the desired goal as a measure for goal-reaching problems. We also provide average final distances in the offline setting to evaluate the learned goal-conditioned policy in Figure 12. From the results we can conclude that WGCSL substantially outperforms other baselines in both random and expert dataset. Although GCSL learns a suboptimal policy, it also achieves relatively consistent final distance across different environments
and offline datasets. Besides, Goal MARWIL and Goal BC are two effective baselines when learning from the expert dataset, but they do not perform well on the random dataset. In addition, HER and DDPG learn unstable results mainly because of bootstrapping from out-of-distribution actions.
E.7 RESULTS ON 10% OFFLINE DATASET
To evaluate the impact of dataset size on the performance, we select 10% of samples (i.e., 2 × 105 transitions) from the offline dataset of 4 hard tasks (FetchPush, FetchSlide, FetchPick, HandReach). The results are reported in Figure 13 and Figure 14. From the results, we can conclude that with only 10% offline data WGCSL consistently achieves the best performance out of these algorithms. In Figure 14, we can see that WGCSL is very robust to the dataset size on the expert dataset, while the performance is affected more by the small data size on the random dataset. The probable reason is that in the offline setting, the number of samples influences the data coverage greatly and learning generalizable value function from a small set of data is very difficult.

heading: E.8 ONLINE EXPERIMENTS
text: We introduce implementation details and hyper-parameters of the online setting in this subsection. As GCSL is originally proposed for online goal-conditioned RL, WGCSL can also be applied to online goal-conditioned setting. In the online training setting, the agent iterates between collecting trajectories from the environment and training with samples from the replay buffer. We conduct
experiments on the first six taks, i.e., PointReach (here we call Point2DLargeEnv), PointRooms (here we call Point2D-FourRooms), SawyerReach, FetchReach, Reacher and SawyerDoor. The buffer size for first four tasks is set as 1× 104, and 5× 104 for the other two tasks. Before training starts, we first randomly sample 20 trajectories to initialize the buffer. The policies of WGCSL and GCSL are both Diagonal Gaussian policy with a mean vector πθ(s, g) parameterized as θ and a nonzero constant variance σ2. The standard deviation σ is set as 0.2. All the policy networks and value networks are the same as that used in the offline setting. When interacting with the environments, we use additional random actions for exploration with a probability of 0.3. After data collection, we sample a mini-batch from the replay buffer and perform goal relabeling for WGCSL and GCSL. Then we train the agent for 1 mini-batch (Point2DLarge, Point2D-FourRooms) or 4 mini-batches
for other tasks. The batch size, optimizer, learning rate keeps the same as the offline setting. The maximum clipping upper bound M for WGCSL is set as 10. We also utilize target nets for WGCSL to stabilize training, and the polyak average coefficient for target nets are 0.9. With respect to the evaluation phase, we use the deterministic policy πθ(st, g) without randomness. After each epoch, we evaluate the policy for 100 episodes and compute the average success rate. For all the experiments we repeat for 10 different random seeds and report the mean performance with standard deviation. In this subsection, we report the results of WGCSL with the discounted relabeling weight (DRW) and the goal-conditioned exponential advantage weight (GEAW), though the performance can be improved with the best-advantage weight (BAW).
The empirical results regarding the average success rate of the online setting are shown in Figure 15. From Figure 15 we can conclude that WGCSL achieves better success rate and converges faster than other baselines in most of the environments except SawyerReach. Therefore, WGCSL is verified to be highly efficient in both online and offline goal-conditioend RL, and we hope it could be a stepping stone for future real-world RL applications.

heading: E.9 THE IMPACT OF CLIPPING FOR EXPONENTIAL ADVANTAGE WEIGHTS
text: We study the impact of clipping in WGCSL’s exponential advantage weights by comparing with the following variants:
• WGCSL w/o clip: no clipping weights performed.
• WGCSL clip 5: clipping the exponential advantage weights to (0, 5].
• WGCSL clip 10: clipping the exponential advantage weights to (0, 10], which is used in our paper by default.
From the results in Figure 16 we can observe that without clipping the advantage estimation can harm the performance in Reacher and SawyerDoor. Therefore, we perform weight clipping by default.

heading: F DETAILS OF ENVIRONMENTS
text: All of the ten tasks have continuous state space, action space and goal space. The maximum episode horizon is set as 50. In this section, we introduce these tasks in detail.
PointReach PointReach is taken from the open-source environment multi-world2 and it requires the blue point to reach the green circle. The state space [−5, 5] × [−5, 5] has two dimensions representing Cartesian coordinates of the blue point, and the action space [−1, 1]× [−1, 1] also has two dimensions meaning the horizontal and vertical displacement. The goal space is the same as state space, which means φ(s) = s. The bule point and the green circle are randomly initialized in the state space. The allowable error of reaching goals is the radius of the target circle and is set as 1. The reward function is defined as:
r(sXY , a, gXY ) = 1(‖sXY − gXY ‖22 ≤ ).
PointRooms The PointRooms environment is built on multi-world. The state space, the action space, the goal space, and the reward function are the same as PointReach. The difference is that there are four rooms separated by walls.
Reacher The Reacher environment is revised from OpenAI Gym (Brockman et al., 2016). States are 11-dimensional, which indicate the angles, the positions, and the velocity of the joints. Actions are 2-dimensional and they control the movement of two joints. Goals are 2-dimensional vectors representing the expected XY position. And the state-to-goal mapping is φ(s) = s[−3 : −1], where the last three dimensions are the XYZ position of the end-effect. The reward function is defined as:
r(sXY , a, gXY ) = 1(‖sXY − gXY ‖22 ≤ ),
where the allowable error is set as 0.05.
SawyerReach The SawyerReach environment is taken from multi-world. The Sawyer robot aims to reach a target position with its end-effector. The observation space is 3-dimensional, representing the 3D Cartesian position of the end-effector. Correspondingly, the goal space is 3-dimensional and describes the expected position, and the state-to-goal mapping is φ(s) = s. Besides, the action space has 3 dimensions describing the next position of the end-effector. The reward function is defined as:
r(sXY Z , a, gXY Z) = 1(‖sXY Z − gXY Z‖22 ≤ ),
where the allowable error = 0.06.
SawyerDoor The SawyerDoor environment is revised from multi-world. The Sawyer robot is required to open the door to a desired angle. The state space (4-dimensional) consists of the Cartesian coordinates of the Sawyer end-effector and the door’s angle. As in the SawyerReach task, the action space is 3-dimensional and controls the position of the end-effector. The desired goals are uniform from 0 to 0.83 radians. And the reward function is defined as:
r(s, a, gangle) = 1(|φ(s)− gangle| ≤ ),
where the allowable error is set as 0.06.
2https://github.com/vitchyr/multiworld
FetchReach The FetchReach environment is taken from OpenAI Gym (Brockman et al., 2016; Plappert et al., 2018). In this environment, a 7-DoF robotic arm is expected to touch a desired location with its two-finger gripper. The state space is 10-dimensional, including the gripper’s position and linear velocities. The action space is 4-dimensional, which represents the gripper’s movements and its status about the opening and closing. Moreover, the goals are 3-dimensional vectors representing the target places of the gripper. The state-to-goal mapping is φ(s) = s[0 : 3], because the first 3 dimensions of the state describe the position of the gripper. The allowable error in FetchReach is = 0.05. The reward function is defined as:
r(s, a, gXY Z) = 1(‖φ(s)− gXY Z‖22 ≤ ).
FetchPush Similar to FetchReach, the FetchPush environment is taken from OpenAI Gym where a 7-DoF robotic arm is controlled to move a box to the desired location. The state space is 25- dimensional, including the gripper’s position, linear velocities, and the box’s position, rotation, linear and angular velocities. The action space is also 4-dimensional as in FetchReach. Different from FetchReach, the 3-dimensional goals and achieved goals represent the target and current position of the box. The state-to-goal mapping is φ(s) = s[3 : 6], because the 3 dimensions of state describe the position of the box. The allowable error in FetchPush is = 0.05. The reward function is defined as:
r(s, a, gXY Z) = 1(‖φ(s)− gXY Z‖22 ≤ ).
FetchSlide The FetchSlide environment is like FetchPush where a 7-DoF robotic arm is controlled to slide a box to the desired location. The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush. The difference is that the target position is outside of the robot’s reach thus it has to hit the box and makes the box slide and then stop at the desired position.
FetchPick Like FetchPush and FetchSlide environments, the FetchPick environment controls a 7-DoF robotic arm to pick up a box to the desired location. The state space, the action space, the state-to-goal mapping, the reward function and the allowable error are all the same as FetchPush. The difference is that the target position may be on the table or in the air, thus the agent needs to pick the box with its gripper.
HandReach In HandReach environment, the agent is required to control a 24-DoF anthropomorphic hand and use its fingers to reach the target place. Observations is 63-dimensional containing two 24-dimensional vectors about positions and velocities of the joints, and additional 15 dimensions indicating current state of fingertips. Actions are 20-dimensional vectors that control the non-coupled joints of the hand. Goals have 15 dimensions for HandReach representing the target Cartesian positions of each fingertip. The reward function is the same as Fetch tasks except the state-to-goal-mapping (φ = s[−15 :]) and the allowable threshold ( = 0.01).

Review summaries:
The proposed method proposes a method for goal-conditioned RL that can be interpreted as a weighted version of prior work.These weights resemble a combination of discounting and advantage weighting. The paper provides theory arguing that these weights cause the proposed method to optimize a tighter lower bound than prior work. Experiments show that it outperforms the prior work, which does not include weights on each training example.

This paper proposes an extension of GCSL where the goal-conditioned BC loss is weighted by a variable that correlates with the number of steps necessary to achieve the desired goal.  The advantage of this approach is that sub-optimal trajectories to a particular goal will get downweighted to the benefit of more direct trajectories.  This effectively adds a policy improvement step to GCSL with the underlying objective function being that of arriving at goal states as fast as possible.    Results on goal-conditioned tasks show that the proposed approach (WGCSL) performs consistently better than GCSL and in some cases significantly better (HandReach-expert).