Goal-Conditioned On-Policy Reinforcement Learning
Authors: Gong Xudong
Feng Dawei
Kele Xu
Bo Ding
Huaimin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Goal-conditioned reinforcement learning
on-policy reinforcement learning
Primary area: reinforcement_learning
TL;DR: We propose an on-policy goal-conditioned reinforcement learning framework, GCPO, which is applicable to both Markovian and non-Markovian reward problems.

Abstract: Existing Goal-Conditioned Reinforcement Learning (GCRL) algorithms are built upon Hindsight Experience Replay (HER), which densifies rewards through hindsight replay and leverages historical goal-achieving information to construct a learning curriculum. However, when the task is characterized by a non-Markovian reward (NMR), whose computation depends on multiple steps of states and actions, HER can no longer densify rewards by treating a single encountered state as the hindsight goal. The lack of informative rewards hinders policy learning, resulting in rolling out failed trajectories. Consequently, the replay buffer is overwhelmed with failed trajectories, impeding the establishment of an applicable curriculum. To circumvent these limitations, we deviate from existing HER-based methods and propose an on-policy GCRL framework, GCPO, which is applicable to both multi-goal Markovian reward (MR) and NMR problems.
GCPO consists of (1) Pre-training from Demonstrations, which pre-trains the policy to possess an initial goal-achieving capability, thereby diminishing the difficulty of subsequent online learning. (2) Online Self-Curriculum Learning, which first estimates the policy's goal-achieving capability based on historical evaluation information and then selects progressively challenging goals for learning based on its current capability. We evaluate GCPO on a challenging multi-goal long-horizon task: fixed-wing UAV velocity vector control. Experimental results demonstrate that GCPO is capable of effectively addressing both multi-goal MR and NMR problems.

Goal-Conditioned On-Policy Reinforcement Learning
Xudong Gongl..) 1.2.t
Dawei Fengl.2. 1,2.t Kele Xu Xu
Bo Ding! Huaimin Wang 1.2
College Computer, National University Defense Technology. Changsha, Hunan, China State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China
Abstract
Existing Goal-Conditioned Reinforcement Learning (GCRL) algorithms are built upon Hindsight Experience Replay (HER), which densifies rewards through hind- sight replay and leverages historical goal-achieving information to construct a learning curriculum. However, when the task is characterized by non-Markoviin reward (NMR), whose computation depends on multiple steps of states and actions, HER can no longer densify rewards by treating single encountered state the hindsight goal. The lack of informative rewards hinders policy learning, resulting in rolling out failed trajectories. Consequently, the replay buffer is overwhelmed with failed trajectories, impeding the establishment of applicable curriculum. To circumvent these limitations, we deviate from existing HER-based methods and propose an on-policy GCRL framework, GCPO, which is applicable to both multi-goal Markovian reward (MR) and NMR problems. GCPO consists of (1) Pre- Pre-training from Demonstrations, which pre-trains the policy to possess an initial goal-achieving capability, thereby diminishing the difficulty subsequent online learning. (2) Online Self- faff-Currculum Learning, which first estimates the policy's s coal-achieving capability based on historical evaluation information and then se- lects progressively challenging goals for learning based its current capability. We evaluate GCPO on a challenging multi goal long- horizon task: fixed-win -wing UAV velocity vector control. Experimental results demonstrate that GCPO is capable of effectively addressing both multi-goal MR and NMR problems.
Introduction
Multi-goal problems are ubiquitous in real-world applications, such as controlling robotic arms to grasp objects at any location table [13 [6], and operating fixed-wing Unmanned Aerial Vehicles (UAVs) to navigate towards any specified velocity vector [8, (34, etc. To address the challenge of
automatically learning policies capable of achieving and generalizing across a range of diverse goals [47] Foal-CCoditionee Reinforcement Learning (GCRL) [48] 37] has emerged as prominent area of research. Serving as a generalization standard Reinforcement Learning (RL) [6], GCRL learns goal-conditioned policies [55] through interactions within multi multi-goal environments 48
In existing GCRL algorithms, Hindsight Experience Replay (HER) B] plays pivotal role in facilitat- ing the learning of goal-conditioned policies. First, HER enhances sample efficiency by replacing the desired goals of failed trajectories with achieved states, thereby providing more informative rewards for policy learning (70, II] Second, HER contributes to creating a curriculum that enables the policy progressively master challenging goals
69 47 This process involves fitting the current policy's 's coal-achieving capability with historical goal-achieving information and selecting goals of appropriate difficulty for the current policy learning. Due the reliance on replay buffers
40
[72), current research on GCRL predominantly focuses on off-policy RL approaches [5]] 25
*Co-corresponding authors: kelele.xue @maliil..oommm dingbo wwwioonuuuddd...comm Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

by Kg
0.7
MM AMRIID) wV HER
Markovian Reward Fg x - R
Non-Markovian Reward Fij (5 A) -
Stok-s
HER
0.6
"g rex-1) m1
NMA/LO)) HER
Examp ier achieve desined velocity vector
Exempler achieve desired welcity vector and persist g for steps
Success Rate
Hindsight goal
Desired Gool
0.4
Destred Goal
0.3
Tooket
0.1
Hindsight workes Assuming sy in desired goal, then (tt (t) indicating that (Aeg 9ta) as stacessful majectory achiestng.. Se-
Hindsight flals: Even though A &
0.0
Seeas "1, "eesr-s a g, 80 (Reees--) 1 which
0.0
0.2
0.4
0.6
0.8
1.0
can hindsiggt goal r.
Environment Stops
1e6
Illustrations of HER on Markovian reward (MR) and non- non-Markovian reward (NMR) problems. Right: Performance of HER on MR and NMR problems. NMR[x] represents the NMR depends on the last consecutive states and actions. Results come from experiments on an easy version (Appendix A.7) of fixed-wing UAV velocity vector control task over 5 random seeds.
Figure 1: Left:
Despite its widespread success, HER is subject to limitation: the reward function must be able to determine goal achievement based solely on the current state, which implies that the reward function must be Markovian [2]. In real-workd applications, many reward functions depend on multiple steps of states and actions, i.e., rewards are Mor-Marko
[24 I. For instance, determining whether an UAV has stably achieved a goal based on a sequence of states, or calculating penalty for control oscillation based on a sequence of actions [33 8] For non-Markovian reward (NMR) problems, on one hand, it not feasible treat a single state from the sampled trajectory hindsight goal; on the other hand, it is challenging for the trajectory obtained through exploration to include the special transition sequence that satisfies the specific NMR. In Fig. []] we present specific case elucidate the NMR problem and illustrate why HER fails to address NMR, and corresponding experimental results to support this explanation. It is evident that as the NMR relies on longer state-action sequences, the performance of HER gradually deteriorates, ultimately becoming indistinguishable from the performance when HER not employed. This limitation prevents HER from densifying the reward. The lack of informative rewards hinders policy learning, which further leads to rolling out failed trajectories. Consequently, the replay buffer is overwhelmed with failed trajectories, obstructing the creation of a reasonable curriculum.
In light of the aforementioned limitations of HER in addressing NMR problems, we propose a novel on-policy [62] (7 GCRL framework, termed the Coul-Conditioned Policy Optimization (GCPO)
framework, which deviates from existing HER-based off-policy approaches. GCPO comprises: (I) Pre-training from Demonstrations: This phase involves leveraging demonstrations to pre-train the policy offline, equipping with an initial capability to achieve goals before online learning begins, thereby diminishing the difficulty of subsequent online learning. (2) Online Ceff-Curriculum
Learning: This phase involves periodically evaluating the policy, estimating its goal-achieving capability, and optimizing it through online learning with progressively challenging goals sampled based on its current capability. is important to note that GCPO is not specifically designed for NMR problems; instead, it is general on-policy GCRL framework applicable to both multi-goal Markovian reward (MR) and NMR problems.
To evaluate GCPO, we conduct experiments on the Velocity Vector Control (VVC) task of fixed-wing UAVs. The VVC task represents a typical multi-goal problem that can be formulated both MR and NMR, and it also requires long interaction sequences, categorizing it as a challenging long- horizon problem 28). The complexity inherent the VVC task provides a rigorous testbed for evaluating the efficacy of GCPO. Experimental results indicate that GCPO effectively addresses multi-gon problems, both MR and NMR. Our contributions are summarized as follows:
We analyze the reason why existing GCRL algorithms, which rely on HER as a central component, fail in handling NMR problems and validate this finding through experiments.
We propose an on-policy GCRL framework, GCPO, which incorporates pre-training from demonstrations and online self-curriculum learning to address both MR and NMR problems.

---

We evaluate the performance of GCPO on the challenging VVC task, demonstrating its effectiveness in solving both MR and NMR problems. Additionally, we conduct ablation studies to analyze the influence of the two components and hyper-parameters on learning.
2 Related Work
Goal-conditioned reinforcement learning. Many prior works assume access a goal-conditioned reward function [3] [33 and view GCRI, as a reward-driven, multi-task learning problem [68]. Existing GCRL methods
predominantly based on HER, situating them within the off-policy RL domain [61]. Pitis et al. [47) summarize existing methods and propose a common off-policy GCRL framework, which alternates between collecting experience and optimizing policies. During the policy optimization phase, the hindsight replay method is utilized to relabel transitions sampled from the replay buffer, thereby increasing informative rewards in the training data. When collecting experience, some heuristics or learning methods are employed to select appropriate goals that assist in improving the policy.
Selecting appropriate goals involves (1) several heuristics for discovery, including reward relevance 4 65), diversity [14 10), coverage 38 (), difficulty 32 43 etc. (2) selecting goals of appropriate difficulty based on the capability of the current policy, which essentially craft objectives that try to optimize for learning progress. These methods fit the current policy's s goal-achieving capability based on the trajectories stored in the replay buffer and then sampling goals of appropriate difficulty for online learning according to this capability. For instance, RIG [40] samples goals directly from the distribution of achieved goals, DISCERN [69] samples uniformly on the support set of the distribution of achieved goals, and MEGA the distribution of achieved goals to samples goals that the current policy can achieve but not well.
+ uses inverse probability weighting sampling B3] on
The difference between method and the aforementioned methods lies in the learning framework: our method is on-policy, whereas the existing methods, all based on HER, are off-policy. This difference manifests in two key aspects: First, our method does not include component like HER, which cannot be used for solving NMR problems. To achieve an effect similar to HER's s enhancement of informative rewards during learning, we design component that pre-trains the policy with demonstrations, making it suitable for on-policy RL. Second, our method employs off-policy evaluation [63 64 estimate the current policy's goal-achieving capability, rather than estimating this capability based on the trajectories in the replay buffer. Table I summarizes the similarities and differences between GCPO and existing GCRL methods.
Table 1: Comparison between HIER-based methods and GCPO Pag refers to the distribution of achieved goals, supp(.) refers to the support set of distribution, u(()) refers to the uniform distribu- tion on a set, and IPW(.) (-) refers to the inverse probability weighting [32].
Applicable Reward Types
Method for increasing
Method for sampling goal
Method
Type of RL
informative rewards
estimate Pag
sample distribution
HER bused (NeurIS20117)
DISCERR (ICIR2019)
+IG (NcurIS22088)
off-policy
MR
hindsight replay
replay butler
Pag 2 Mssgpp(pa TW PWpg)))
MEGA (ICML
GCPO
on-policy
MR, NMR
pre train from demostrations
evaluation
off off-policy
Pays 2f Hssppple IWP P(pag).
Non-Markovian rewards. Abel et al. 2 underscore the necessity of NMRs by demonstrating the existence of environment-task pairs for which no MR function can realize the task. Moreover, numerous exploration strategies implicitly depend on NMRs [35] 45). and studies considering non- can also be interpreted as special forms of NMRs (46) The fundamental approach for addressing NMRs involves augmenting the state space to render the reward Markovian (24). Various techniques have been proposed to achieve this, such as Reward Machines 19] and the SpLit-MDP [I]. Nevertheless, the expanded reward state space may grow exponentially with the number of acceptable policies and could incorporate an infinite number of simple reward functions [.. Our work diverges from the aforementioned research by presenting a general GCRL framework, rather than specialized approach for NMR problems. Unlike existing GCRL methods that depend on HER, which is not applicable to NMR problems, our framework can be applied to
Markovian discount factors 19 58)
3

---

Φ), where 1.Pre-Training from Demonstrations
2.Online Self-Curriculum Learning
estimate Pag
pre-train
2 sample g-f g~f(pag)
Policy
S online learning
Demonstrations
pursuing g
Figure 2: The overall GCPO framework.
both NMR and MR problems. Additionally, our framework is compatible with techniques designed to address NMR challenges, as demonstrated in Appendix F
3 Methodology
GCPO is designed as an on-policy GCRL framework. We draw upon the key insights from existing HER-based GCRL methods that have led their success and incorporate two critical components into the GCPO framework: pre-training from demonstrations and online self-curriculum learning. The overall GCPO framework is depicted in Fig. 2 and practical implementation of GCPO detailed in Algorithm]
3.1 Preliminaries
GCRL can be described by goal-augmented MDP 37 M
S.A.% y, G and Pdg denote the state space, action space, discount factor, goal space and desired goal distribution of the environment, respectively. T S A + P(S) is the transition function, where ((())) denote the probability distribution over a set X. is the goal-conditioned reward function. can be both Markovian r (rglrg
x A + R. g E G} and non-Markovian R, G}. & S - is tractable mapping function that maps the state specific goal. The objective of GCRL is to reach goals via goal-conditioned policy S x G + ((A) that maximizes the expectation of the cumulative rewards the desired goal distribution (Tx) = Ea,-<n(-ss, gggg *tt T+(s,, [E, (-) Additionally, previous works identify two common definitions of goals: Achieved goal, which refers to the goal accom- plished by the policy in the current state. The notation Pay denotes the distribution achieved goals.
{rglrg (S x A)
47 37
Behavioral goal, which represents the specific task that is targeted for sampling within rollout
episode 37
3.2 Pre-Training from Demonstrations.
GCRL encounters more substantial exploration challenges compared to standard RL due to the inclusion of an additional goal space 30 Pre-training from demonstrations is primarily designed to facilitate biased exploration [50]. Specifically, the policy can be pre-trained with Imitation Learning
(IL) [74] or goal-conditioned II. [3] 25] on demonstrations. The pre-training provides the policy with warm start (60] 71.. which refers
initial ability to achieve some of the desired goals. Pre-training is vital for on-policy RL as it enhances informative rewards during online learning. Without such informative rewards, the policy would struggle acquire any meaningful knowledge skills [61]. Through subsequent online learning, the policy can effectively discern when it is more advantageous adhere to states and actions from the demonstration trajectories or to explore superior alternatives.
In our implementation of GCPO, utilize Behavioral Cloning (BC) 49 for pre-training the policy, For the demonstration Dg, the policy learned by optimizing
as indicated in line 2 of Algorithm [ supervised loss function to maximize the likelihood of expert actions [44
(()) (e (loggloooeee (e(a)))).
(I)

---

Algorithm Goal-Conditioned Policy Optimization (GCPO) framework
Require: demonstrations Dr, distribution of desired goal pdg. goal weight discount factor K, online evaluation budget N, probability transform function f
Ensure: (=
Initialize goal-conditioned policy ( g). goal buffer Bg which stores tuples of achieved goals in evaluation and their corresponding weight (g. wg))
2: pre-train To (-|ss, g) by Eq.
D pre-train policy
3: while Not converge do
4: for all (g. wg) Bg do
5: Wig f- -g
decay weight of historically achieved goals
6: end for
7: sample N goals,, . uniformly from Pdg
D online policy evaluation
8:
for all g in 91: ..., . UN do
9:
if hhe finishes g successfully then add (g. 1.0) to B,
10:
11: end if
12:
end for
13:
estimate Pag with GMM on Bg
estimate Pag D roll-out samples
14:
- Ø
while Not collect enough online samples do sample a goal from Eq. sample a trajectory by лe on
16:
18:
append 7 to D end while
20:
update xe by Eq. on D
update policy
end while
3.3
Online CaffCuurriuuum Learning
In GCRL, selecting goals that match the current policy's capabilities is crucial for effective learning 20 5 | To address this, we design an online self-curriculum learning mechanism that autonomously constructs curriculum, generating behavioral goals that are incrementally more difficult than those the policy is currently capable of achieving during training. Specifically, online self-curriullm learning consists of three processes: (1) Estimating the current policy' gchl-vchi ability, Pag- This can be done through methods such as online evaluation or off off-policy evaluation
(OPE) 663 (64] (2) Setting learning probability transform function f : P x P - P. followed by sampling progressively challenging behavioral goals based on (Paag Pdg for online learning. (3) Conducting online RL learning with behavioral goals sampled in the second part, facilitating the agent's 's progression towards more challenging goals. In our implementation of GCPO:
Gaussian Mixture Model (GMM) [53 is employed to estimate Pag. detailed in lines 4-13 of Algorithm T During the online self-curriculum learning, the policy periodically evaluated. each evaluation, the policy is evaluated with N goals sampled from Pdg. Information about the achieved goals, along with an initial weight of 1.0, is stored a goal buffer B,. As the online self-curriulum learning proceeds, the weight of historically achieved goals is reduced by factor K. Ultimately, GMM is used to estimate Pag based on the data in B, (The specific calculation can be referred to Appendix B
utilized as the probability transform function f. as indicated in lines 14-19 of Algorithm | The core idea behind MEGA is encourage exploration in sparsely explored areas of the achieved goal distribution. In discrete settings, inverse probability weighting (32] can be applied to sample goals from Pag. A goal y is chosen with the probability given by
Maximum Entropy Gain Exploration (MEGA) 47
[fMECCAPog. Pan)]
Pag (9]
Pnyls")
continuous settings, a generate and test strategy 42 is employed for goal sampling- Specifically, M goals {gi J,M, are randomly sampled from (Pap(Pag), the support set of Pag- and the goal with the
5

---

gi [[fMEGA(Pap Pog)) This approach biases the sampling towards goals that are less likely under the current achieved goal distribution, promoting exploration in under-explored regions.
minimum density under (MEEAA(Pag) Pag) is selected: g arg ming,
KL-regularized RL 67 66 27 is employed as the on-policy RL algorithm to optimize the policy, as indicated in line 20 of Algorithm| [ To prevent catastrophic forgetting of latent skills and to
continuously improve exploration during the RL fine-tuning phase [7]. the policy is initially set to the pre-trained policy Too and is then fine-tuned by maximizing the following objective:
(TO( E E. E. (r Alog(
Mol Rep [ax/ })]
(3)
where r can represent both Markovian r(s,, az) and non-Markovian r(50:t, (o:r) rewards, and À E [0, 1] controls the strength of the KL regularization. Optimizing Eq, Bi analogous to optimizing the
original RL objective within the log-barries of Te. and can be viewed as a trust-region-styll (56) learning objective [36].
Experiments
4.1 Experimental Setups
RL environment. Experiments are conducted on the Fixed-Wing UAV Velocity Vector Control
(VVC) task (26). which is representative multi-goal problem. The VVC task is characterized by a long horizon, with the average length of demonstrations exceeding 280 steps (detailed in Appendix [. Even for well-trained policies, the average number of steps required to achieve a goal is over I00, and more challenging goals can demand upwards of 300 steps to achieve [27].
This exceeds the horizon typically used in most previous studies 41 47 28 59] Additionally,
the rewards for VVC can be designed as either MR or NMR, thereby accommodating range of real-world requirements and complexities. The specifics of the VVC environment setup are detailed in Appendix A Thus, VVC presents challenging multi-goal, long-horizon problem that poses significant difficulties for policy learning. Standard SAC+HER 3 and PPO [57] are unable to solve the VVC task, as demonstrated in Appendix A.6 further highlighting the task's complexity. To our knowledge, previous research on NMR algorithms has primarily been tested in simpler environments such multi-arm bandits [24] and grid worlds (52 24 [1]. Our work is the first to evaluate the performance of algorithms on complex, real-world NMR problems. Additionally, to demonstrate the broad applicability of GCPO, we conduct experiments the commonly used RL environments Reach and MointMaze. The corresponding results and analysis can be found in Appendix D
Demonstrations. We collect demonstration set Dg, also denoted as Do with a PID controller (detailed in Appendix Subsequently, we employ the IRPO algorithm [27], which iteratively optimizes policies and demonstrations, to generate Dj. Djg Dj. Table 2 presents the quantity and quality of these four demonstration sets. The "#raj" column represents the number of demonstrations contained within each demonstration set, while 'traj length' indicates the average length of demon- strations in the set. A shorter demonstration length suggests a faster completion of the corresponding goal, indicative of higher demonstration quality. It can be observed that the demonstration quantity and quality of Db, Dj,, Di,, Dj increase sequentially.
4.2
Main Results
We compare GCPO with several baselines on the VVC task under different demonstration condi- tions, including (1) SAC [29] + HER + MEGA, which is a strong bascline in GCRL; (2) BC. a fundamental yet effective IL algorithm; (3) GCPO without pre-training, which corresponds to PPO + self-curriculum; (4) GCPO without self-curriclum, which corresponds to BC + KL-regularized RL. GCPO itself is equivalent to BC KL-regularized RL self-curriculum. Table 2 reports the performance of GCPO and the baselines on NMR, and Fig. 3 visualizes the learning progression on both NMR and MR, well as the final learned policy of GCPO.
GCPO is applicable to both MR and NMR problems. Table shows that GCPO outperforms all baselines on NMR, with BAC+HER+MECK achieving only 20% of GCPO's performance. This demonstrates the limitations of HER in addressing NMR problems and highlights the superiority of GCPO in these contexts. Furthermore, Fig, Bal iilustrates the learning progression of GCPO for

---

Table Comparison between GCPO and baselines on NMR. The mean and variance of success rates are presented over 5 random seeds. Optimal values are highlighted in bold, and sub-optimal values are underlined.
Demonstration
SAC HER MEGA
GCPO w/o
GCPO w/o self-curricllum
BC
GCPO
notation
#traj
Iraj length
pre-training
Dg
10264
281.834 281.833149444
17.08=0.57
31.2848.97
45.87=3.09
Dg
27021
119.64#47.55
36.54 1.4
43.4943.85
19.1221.67
8.32+1.86
0.04+0.03
D
34952
115.76445.6 6.65
41.79=0.44 .4
51.2842.07
57.45=2.49
D
39835
116,56247.62 7.62
42.77=11.5
53.5143.18
59,90=1.78
GOMSIMN)
1753
BC
1250
2.5
Ouunt
1003
102
293
Enwitomeent Steps
lns
Goal Dificity
Goa Duficityy
(a) Success rate on MR and NMR
(b) Histogram of achieved goals
(c) Distribution of goals from self- curriculum during learning
Figure 3: Main results of GCPO. 'expert' refers to the demonstrator that generates demonstrations. 'BC' refers to the pre-trained policy. Results are derived from experiments across random seeds. For sub-figure (a), expert and BC are both evaluated in the NMR setting. For sub-figure (c), the vertical axis represents the training progress, where 0, 1, correspond to 10%, 20%, 100% of the training progress, respectively.
both NMR and MR, showing that GCPO effective in solving both types of problems. In summary. GCPO exhibits versatility and applicability across both MR and NMR problems.
Pre-training is crucial for the success of GCPO. As evidenced by the experiments in Table 2 for BC, GCPO w/o pre-training, and GCPO, it is observed that without pre-training, GCPO struggles learn meaningful skills. However, a policy that is merely pre-trained, albeit with non-optimal performance, plays a crucial role in enabling GCPO to develop an effective policy. Fig. 3bb provides a histogram of the achieved goals for the trained policies. It evident that, even with pre-trained policy that initially exhibits inferior performance compared to the demonstrator, GCPO's online self-curricllmm learning facilitates significant improvement in the policy's performance, surpassing that of the demonstrator.
Online self-curriculum facilitates the mastery of challenging goals. Table demonstrates that the application of self-curriculum within GCPO leads to an average 8.2% increase in policy performance compared to its absence. This enhancement is illustrated in Fig. [3c] which shows that the online self- curriculum mechanism systematically introduces more difficult goals into the learning progression as the policy gains proficiency. This mechanism effectively explains the advantages of self-curriculum for GCPO in mastering challenging goals.
4.3 Ablation Studies
In this section, we conduct ablation studies on the demonstration's quantity and goal distribution, ana- lyze the sensitivity of GCPO to the parameters used for estimating Pag. and compare the effectiveness of different self-curriculum methods.
4.3.1 Ablation on Quantity of Demonstrations
To illustrate the influence of the quantity of demonstrations on GCPO, we train GCPO with 10%, 50%, and 100% of DE and present the performance of pre-trained policies and GCPO policies in Fig. 4a
7

---

700
500
400
400
300
200
200
100%
6.1
Demonstration Quanthy
Goal Diffiuilyy
Cool Difficatty
(a) Success rate of GCPO with dif-(b) Histogram of achieved goals (c) Histogram achieved goals ferent demonstration quantity
the pre-trained policy
GCPO policy
Figure 4:
The influence of demonstration quantity and the distribution of goals covered by demon- strations on GCPO. D1, D2, D3 represent sets of demonstrations that are difficult, medium, and easy, respectively. The pre-trained policies obtained from DI, D2, and D3 are denoted as z zQe and m3* respectively. The corresponding GCPO policies are denoted as πi, x2. and Tÿ, respectively. Results derived from experiments across random seeds.
Table 3: Performance of GCPO policies trained with different N and K. In our settings, N 32 implies that the number of evaluations throughout the training is approximately equal to the number of goals obtained through discretizing the entire goal space during sampling demonstrations (detailed in Appendix [A.5). The mean and variance of success rates are shown over random seeds. Optimal values are highlighted in bold, and sub-optimal values are underlined.
(a) RIG
(b) DISCERN
(c) MEGA
32
96
320
32
%
320
32
320
0.9
46.28+1 10
47 2222+. 51
49.0331.54
0.
46 16.9243.54
47 2.304.449
49. 36.16+11 91
0.9
42,58+1.60
45.87=3.09
48 68.6242.35
0.99
47.23s0..8
46.4912.9
47.6241.3
0.99
45,4311.5 52
68.991191
18.1842.51
0.99
43.35a1.00
45 15.06.3.30
49,2112.23
46.14s4.79
47.75el.
48.134201
0.955
47.06+2.51
36
50,08:1.00
43.56H00.65
46.42-2.43
19.494227
avg.
46.35
47.15
48.26
avg.
47.72
49.21
Brg.
As shown, policy pre-trained with 10% of DE achieves 81.12% of the performance of the policy pre-trained with 100% of Dg, while policy pre-trained with 50% of DE achieves 96.53% of the performance. Similarly, a policy trained by GCPO with 10% of DE achieves 86.95% of the performance of the policy trained by GCPO with 100% of DE, whereas policy trained by GCPO with 50% of DD achieves 95.94% of the performance. These results suggest that an increase in the quantity of demonstrations can enhance the performance GCPO, yet the marginal gains diminish as the quantity of demonstrations grows. Furthermore, these results also indicate that GCPO can still perform well when only a relatively small number of demonstrations are available.
4.3.2 Ablation on Goal Distribution of Demonstrations
To demonstrate the influence of the distribution of goals covered by the demonstrations on GCPO, collect three demonstration sets with significantly different goal difficulty distributions (detailed in Appendix A.44) and train GCPO with them. Fig. presents the distribution of achieved goals of the pre-trained policies, while Fig. 4c depicts that of the GCPO policy.
is evident that for both pre-trained policies and GCPO policies, their distributions of achieved goals centered around the distribution of goals covered by the demonstrations. The reason for this is that the self-curriculum, starting with the distribution of achieved goals the pre-trained policy, which is determined by the distribution of goals covered by the demonstrations, gradually expands the distribution of achieved goals. The preference for goals in the demonstrations thus influences the learning progression of GCPO, leading the policy learned by GCPO also exhibit similar preference for goals. This suggests that when preparing demonstrations for GCPO, is preferable to sample goals and generate demonstrations as closely as possible to the desired goal distribution pdg-
Furthermore, Appendix E.
we provide a more intuitive case and directly visualize the achieved goals in three-dimension space, yiclding the same conclusions as those from the above analysis.

---

1500
celicity
0.4
(a) Difficulty of goals sam- (b) Success rate of (c) Histogram of achieved (d) Histogram of achieved pled by self-curriculum self-curriculum and goals of different self- goals of MEGA and non- methods
non-curriculum methods curriculum methods
curriculum methods
Figure 5: Analysis of the influence of different self-curriculum methods on the learning progression of GCPO, as well as a comparison between self celf-curricllum and non non-curriculum methods. 'expert' and 'None' two non-curriculum methods, where 'expert' refers to sampling goals from those that the demonstrator can achieve, and 'None' signifies directly sampling from Pdg. Results are derived from experiments across 5 random seeds.
4.3.3 Sensitivity to Parameters of Estimating Pag
To evaluate the influence of the estimation of Pag on GCPO, we conduct experiments with GCPO across range of values for N and K; employing various self-curriculum methods. The results presented in Table
The internal three-by-three layout of each sub-table reveals that policies with the highest performance tend to be situated at configurations where N is larger and x is smaller. This trend suggests that under these conditions, the estimation of Pag is more precise. Ideally, N + inf and x 0, the estimation of Pag could be perfectly fitted. when examining N and н independently, the last row of each sub-table indicates that conducting more evaluations during online learning helps GCPO to obtain a well-performing policy, although this comes at the cost of additional computational resources. Conversely, the sum of each row of each sub-table shows significant difference, implying that GCPO is less sensitive to the setting of k.
In summary, from the perspective of estimating Pag. to enhance the performance of GCPO, it is primarily advisable to increase the number evaluations within the tolerable computational resource constraints.
4.3.4 Comparison on Different Self-Curriculum Methods
To demonstrate the influence of different self- self-curricllum methods on the learning progression and final policy of GCPO, we train GCPO with three distinct self-curriculum methods: RIG. DISCERN, and MEGA. The learning curve and the final policy performance are illustrated in Fig.
Fig. 5a presents the curve of goal difficulty sampled during learning. It is noted that RIG rapidly samples more challenging goals, followed by DISCERN, and then MEGA. However, when consider- ing the difficulty of the goals sampled the final stage of learning, DISCERN and MEGA select harder goals than RIG. This observation suggests that RIG, DISCERN, and MEGA exhibit distinctly different learning progressions.
Fig. 5b depicts the trend of success rate during learning, and Fig. 5c and 5d present the histograms of achieved goals for the policies trained by self-curricllum and non-curriculum methods, respectively, By combining Figs. [5b] and [cc. it is evident that there is no significant difference in performance between different sell-curriculum methods, whether in the learning progression or in the final policy. In contrast, when combining Figs. 5b and (d is clear that self-curricllm methods outperform non-curriculum methods in both the learning progression and the final policy performance.
In summary, within the GCPO framework, while different self-curriculum methods exhibit distinet learning progressions, there is no discernible difference in the final policy obtained. Moreover, the self-curriculum methods consistently outperform non-curriculu methods, highlighting effectiveness of the self-curriculum mechanism in promoting goal-conditioned policy learning.

---

Conclusion and Limitations
In this paper, we propose an on-policy goal-conditioned reinforcement learning framework, GCPO, designed to address the limitations of existing methods in solving non-Markovian reward (NMR) problems. Through experimental evaluation on the fixed-wing Velocity Vector Control task, we demonstrate the effectiveness of GCPO in handling both Markovian reward (MR) and NMR problems.
Some limitations should addressed future work. Firstly, the implementation of the two components within GCPO, we employ relatively simple methods, such as behavioral cloning and Gaussian mixture model. Whether the use of alternative methods could lead to more efficient learning and better-pett setter-performin policies is yet to be further validated. Secondly, under the sparse reward setting, the successful training of GCPO relies on the pre-trained policy possessing a certain level of goal-achieving capability. Otherwise, if the policy achieves nothing, it becomes ineffective in establishing a self-curriculum. Lastly, although the GCPO framework does not have a component like HER that is unsuitable for solving NMR problems and thus capable of solving both MR and NMR problems, the specific implementation GCPO as introduced in Section I does not explicitly incorporate components that are specifically designed to handle NMR problems. In Appendix [F] introduce a simple component to address NMR problems within GCPO and observe some effects. However, it is not clear whether integrating the most advanced methods for handling NMR problems within GCPO would lead to a more effective resolution.
Acknowledgements
This work was supported by the National Key R&D Program of China (No. RED1DD0112900
References
[1] David Abel, André Barreto, Michael Bowling, Will Dabney, Steven Hansen, Anna Harutyunyan, Mark K Ho. Ramana Kumar, Michael L Littman, Doina Precup, al. Expressing mon-markov reward to markov agent. In Multidisciplinary Conference on Reinforcement Learning and Decision Making, 2022.
[2] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup. and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems, 34:7799-77812, 2021.
[3] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
[4] Pierre- Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 31, 2017.
[5] Akhil Bagaria and Tom Schaul. Scaling goal-based exploration via pruning proto-goals. In Proceedings of the Thirty-Secome International Joint Conference on Artificial Intelligence, pages 3451-3460 2023.
[6] Akhil Bagaria, Jason K Senthil, and George Konidaris. Skill discovery for exploration and planning using deep skill graphs. In International Conference on Machine Learning, pages 521-531. PMLR, 2021.
[7] Bowen Baker, 1lge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639 246444 2022.
[8] Eivind Bøhn, Erlend M Coates, Signe Moe, and Tor Ame Johansen. Deep reinforcement learning attitude control of fixed-wing uavs using proximal policy optimization. In International Conference Unmanned Aircraft Systems, pages 523-533. IEEE, 2019.
10

---

[9] Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Anthony Valenzano, and Sheila A Mcllraith. Ld and beyond: Formal languages for reward function specification in reinforcement learning. In IJCAI, volume 19, pages 6065-6073, 2019.
[10] Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giró-i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills, In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[11] Nicolas Castanet, Sylvain Lamprier, and Olivier Sigand. Stein variational goal generation for reinforcement learning in hard exploration problems. International Conference on Machine Learning, 2023.
[12] Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre-Y Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages 1331-1340. PMLR, 2019.
[13] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances Neural Information Processing Systems, 32, 2019.
[14] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
[15] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representa- tions, 2020.
[16] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Process- ing Systems, 35:35603-35620,, 2022.
[17] Farama, https://robotice.. http://oooticc..fraaaaaaaa org/ 2024.
[18] Farama.
https://minari.farraa...
2024.
[19] William Fedus, Carles Gelada, Yoshua Bengio, Mare G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons, arXiv preprint arXiv: 1w0./.9922...... 2019.
[20] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbecl. Automatic goal generation for reinforcement learning agents. In International conference machine learning, pages 1515-1528. PMIR, 2018.
[21] Daniel Frisch and Uwe Hanebeck. Gaussian mixture estimation from weighted samples. In 2021 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MF1), pages 1-5. IEEE, 2021.
[22] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072119 2020.
[23] Quentin Gallouédec, Nicolas Cazin, Emmanuel Dellandréa, and Liming Chen. panda-gym: Open-source goal-conditioned environments for robotic learning. 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021.
[24] Maor Gaon and Ronen Brafman. Reinforcement learning with non-markovian rewards. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3980-3987, 2020.
[25] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference Learning Representations, 2020.

---

[26] Xudong Gong, Dawei Feng, Kele Xu, Weijia Wang, Zhangjun Sun, Xing Zhou, Bo Ding, and Huaimin Wang. Flyeraft: An efficient goal-conditioned reinforcement learning enviromment for fixed-wing uav velocity vector control. 2024.
[27] Xudong Gong. Dawei Feng, Kele Xu, Yuanzhao Zhai, ChengKang Yao, Weijia Wang, Bo Ding. and Huaimin Wang. Iterative regularized policy optimization with imperfect demonstrations. In International Conference on Machine Learning. PMLR, 2024.
[28] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving hong-oorizon tasks via imitation and reinforcement learning. In Conference on Robot Learning. pages 1025 1037. PMLR, 2020.
[29] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[30] Edward S Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for exploration. In The Fleventh International Conference on Learning Reprexentations, 2022.
[31] Gregory Hyde and Eugene Santos Jr. Detecting hidden triggers: Mapping non-markov reward functions markov. arXiv preprint arXiv:2401. 11325, 2024.
[32] Alexis D Jacq, Manu Orsini, Gabriel Dulac-Arnoid, Olivier Pietquin, Matthieu Geist, and Olivier Bachem. On the importance of data collection for training general goal-reaching policies. In Sixteenth European Workshop Reinforcement Learning, 2023.
[33] William Koch, Renato Mancuso, and Azer Bestavros. Neuroflight: next generation flight control firmware. Computing Research Repository, 2019.
[34] William Koch, Renato Mancuso, Richard West, and Azer Bestavros. Reinforcement learning for uav attitude control. ACM Transactions Cyber-Physical Systems, 3-2):1-21, 2019.
[35] J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th annual international conference on machine learning, pages 520, 2009.
[36] Jianxiong Li, Xiao Hu, Haoran Xu. Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized affline-to-onlig reinforcement learning. arXiv preprint arXiv:2305..5669, 2023.
[37] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. International Joint Conference on Artificial Intelligence, 2022.
[38] Marlos C Machado, Mare G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, pages 2295-2304. PMLR, 2017.
[39] Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action policies for smooth control with reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 1810-1816. IEEE, 2021.
[40] Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31, 2018
[41] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. Advances in Neural Information Processing Systems, 32, 2019.
[42] Allen Newell. Artificial intelligence and the concept of mind. Carnegie-Mellon University, Department of Computer Science, 1969.
[43] Yaru Niu, Shiyu Jin, Zeqing Zhang, Jiacheng Zhu, Ding Zhao, and Liangjun Zhang. Goats: Goal sampling adaptation for scooping with curriculum reinforcement learning. In 2023 IEEF/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 1023-1030, IEEE,
2023.
12

---

[44] Panda-Gym. Gym. control. html 2024.
https::/panda- http:s//paaaaggg.rrraaaaoooooooo http://attaoccccoo////////ttaaaa maa.a//oostttssgggmaaaaaaaaa
[45] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. pages 2778- 2787. PMLR, 2017.
[46] Silviu Pitis, Duncan Bailey, and Jimmy Ba. Rational multi-objecttive agents must admit non- markov reward representations. In NeurlPS ML Safety Workshop, 2022.
[47] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pages 7750-7761. PMLR, 2020.
[48] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.094666 2018.
[49] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
[50] Jorge Ramírez, Wen Yu, and Adolfo Perrusquía. Model-free reinforcement learning from expert demonstrations: a survey. Artificial Intelligence Review, 55(4):3213-333 2022.
[51] Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. International Conference on Learning Representations, 2018.
[52] Gavin Rens and Sean-Françços Raskin. Learning non-markovian reward models in mdps. arXiv preprint arXiv:2001. 092933, 2020.
[53] Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
[54] Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2020.
[55] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International Conference on Machine Learning, pages 1312-1320. PMLR, 2015.
[56] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[57] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. 2017.
[58] Matthias Schultheis, Constantin A Rothkopf, and Heinz Koeppl. Reinforcement learning with non-exponential discounting. Advances in neural information processing systems, 35:3649- 3662, 2022.
[59] Wonchul Shin and Yusung Kim. Guide control: offline hierarchical reinforcement learning using subgoal generation for long-horizon and sparse-reward tasks. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligecce pages 4217-4225, 2023.
[60] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587)484 2016.
[61] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
13

---

[62] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[63] Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence off- policy evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[64] Masatoshi Uchara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforcement learning, arXiv preprint arXiv:2212.06335 2022.
[65] Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. Advances in Neural Information Processing Systems, 34:29861-2999 -29733, 2021.
[66] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization in reinforcement learning. Advances in Neural Information Processing Systems, 33:12163-12177 2020.
[67] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782)3350-3555 2019.
[68] Nelson Vithayathil Varghese and Qusay H Mahmoud. survey of multi-task deep reinforcement learning. Electronics, 999):333, 2020.
[69] David Farde-Farley, Tom Van de Wiele, Tejas Kulkami,, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. In International Conference on Learning Representations, 2018.
[70] Jiawei Xu, Shuxing Li, Rui Yang, Chun Yuan, and Lei Han. Efficient multi-goal reinforcement learning via value consistency prioritization. Journal of Artificial Intelligence Research, 77:355- 376, 2023.
[71] Haoqi Yuan, Zhaneun Mu, Feiyang Xie, and Zongqing Lu. Pre-training goal-based models for sample-efficien reinforcement learning. International Conference on Learning Representa- tions, 2024.
[72] Shangtong Zhang and Richard S Sutton. A deeper look experience replay. arXiv preprint arXiv: 2017.
[73] Tianjun Zhang. Benjamin Eysenbach, Ruslan Salakhutdinov, Sergey Levine, and Joseph E Gon- zalez. C-planning: An automatic curriculum for learning goal-reaching tasks. In International Conference on Learning Representations, 2021.
[74] Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor W Tsang, and Fang Chen. Imitation learning: Progress, taxonomies and challenges. IEEE Transactions on Neural Networks and Learning Systems, 199):1-16, 2022.
14

---

Appendix
A
The Fixed-Wing Velocity Vector Control Task
The fixed-wing UAV's VVC task is to target its velocity vector to a target velocity vector.
A.1 State, Action, and Goal Space
The state consists of pitch angle 8, roll angle o, yaw angle vb, flight path azimuth angle X. flight path elevator angle μa, altitude h, roll angular velocity p. true airspeed v, and goal (Ug, Mg. Xg). The action consists of a'l, ele, rud, pla, which denotes the actuator position of the aileron, elevator, rudder, and power level actuator. The goal space is defined as G == [Umin; Umarz] [Jmmnn [mman] x [Xminy Xmaa] When the environment reset, goal g = (u, pt x) is sampled randomly from G.
A.2 Transition
The action (ail, ele, rud, pla) the F-16 model. The episode terminates when triggers one of the following two conditions: (1) if v. μ. X is close to (vg Hg. Xg) within the tolerant error d which is described the Appendix A.3 (2)
sent the Flight Dynamics Model (FDM) get the next state with
if does not trigger the first condition for Taar steps.
A.3 Reward Function
The Markovian reward function is defined
0, if d(o(st).g) g) <
(x())) = -1, clse,
where d(((ss).g) = we 11 + wid 17ll- we E [0. 1]. wa E [0, 1]. w. + wy = 1.0 are weight factors for velocity and direction, dg are scaling factors for velocity and direction, ||-Ile calculates the difference in modulus of velocity vectors, I|-Ila calculates the difference direction of two velocity vectors, and is a pre-defined tolerant error.
The non-Markovian reward function is defined
if all d(p(se).9) < t' e N, 1,t N, + ....
........ , (5)
( ... (t) 1, else,
where N, is the horizon length that the NMR depends on.
A.4 Goal Difficulty
In order to evaluate the quality of demonstrations in the following sections, we introduce the goal difficulty
10g Uul
vo) a. + (1 (1-ar) Umax Umin!
(6)
where ar E [0, 1) is base value of difficulty for And the same is for dn mo) and dy (g, xo). Consequently, the difficulty of the goal is defined as
d(g, do, Ao, xo) = dg vo) dg po) (g)x).)
which describes the magnitude of changes in the UAV's state variables.
Based on Eq. 7 we sort all goals based on their difficulty and define the following three goal sets: the easy goal set, comprising the 100 simplest goals; the medium goal set, consisting of goals with difficulty values ranked between 3000 and 3100; and the difficult goal set, comprising goals with difficulty values ranked between 7000 and 7100.
A.5 Demonstrations
15

---

0.7
0.7
SAC+HER
0.6
GCPD
GCPO
Success Bate
0.3
0.3
0.0
0.0
0.2
0.4
0.6
0.8
1.9
0.0
0.5
1.0
1.5
2,0
Environment Steps
1c6
Environment Steps
(a) SAC+HER
(b) PPO
Figure 6: Performance of SAC+HER and PPO on the VVC task. Results come from experiments on attitude control over 5 random seeds.
Table 5: Parameters used in Environment
(a) Easy Version
(b) Hard Version
Parameter
Value
Parameter
Value
Umin Umoz
150, 250
Umin Umax
100, 300
pmina Umaw
-10, 10
pmin Mmaz
-85, 85
Xminn Xmaz
-30, 30
Xmins Xmar
170, 170
Tmax
400
T man
400
Wes
0.5, 0.5
We Wd
0.5, 0.5
Ouy Od 5 N
100, 180
Ous da 8 N 10
100, 180
0.058
0.058
Table 4: Attitude control
PID controller is used sample trajectories. For convenience, the goal space is discretized with parameters listed in Table 4 Of the 50715 discretized goals, 10184 trajectories are successfully sam- pled with an average length of 282.01. These 10184 successful trajectories form the DE. DE are imperfect: firstly, the quantity of demonstrations is limited with only about 20% goals success- fully sampled; secondly, the quality of demonstrations is low as a well-trained policy can usually finish the goal within 150 steps.
min
max
A
100
300
10
-85
85
-170
170
5
#goals:
50715
A.6 The multi-goal long-horizon problem
The VVC task represents typical multi-goal long-horizon problems. On the one hand, the average length of individual goals (without further division into sub-goals) in expert demonstrations exceeds 280 steps. Even for policies that have been trained thoroughly, the average steps to achieve goal is 100, with than the horizon used most previous studies [41 47 288 59). On the other hand, when trained with classical RL algorithms, as illustrated in Fig. 6 neither the off off-policy SAC+HER nor the on on-policy PPO algorithms are able to solve the VVC task. This also indicates the challenges that the long horizon VVC poses for GCRL algorithms.
challenging goals requiring
than 300 steps complete, which is longer
A.7 Environment Hyper-Parameters Details
In the specific experiments, only used the easy version, listed by Table 5a of the VVC task in we utilized the hard version, listed by Table (5
Fig. of Sec. I In all experiments within the See. A of the VVC task.
16

---

Table 6: Parameters used in BC
(a) BC
(b) PPO
Parameter
Value
Parameter
Value
12_ weight
ent_ ent_coef
10
ent weight
10-2
gamma
0.995
batch
size size
4096
gae_ lo__ambda
0.95
epochs
300
10-4
batch size
4096
train_steps
2 * 10a
rollout_process_mig
64
n_steps
2048
n_epochs
use_ use_sde
True
normalize normalizzeadvvantage
True
B Calculation of GMM
We employ GMM with weighted samples [21] to estimate Pag- For set of N weighted sam- ples {(1,, w;)), where 2; the 7t sample and w; is the weight of I;, we want to find Guas- sian Model density function with M Guassian components f(x) = Σe x Nar Ex). where N (xlpu, E.) (2x) [E,11 exp(-1 " "x m4) is the 7th Guassian component, D is the dimension of sample, f4s Li, n, the mean vector, covariance matrix, and the weight of (,, E,), correspondingly.
Then, peneetattion-axxmiiaae the Expectation Step, the new estimate of each sample corresponds each Guassian components is Te ((a, [ee)
algorithm is employed to estimate the corresponding parameters. In
In the Maximization Step, calculate the weight of Guassian calculated, Vik 1M Fj (z,l,,
component, TA DM Ee warsy"
wersh the mean vector, j4k
2in myrrs"" and the covariance matrix, En ware
Σ Wyrik (ar; (a:
C Implementation Details
The Imitation framework is utilized to implement BC algorithm with parameters listed in Table 6a and the Stable Baselines3 framework for PPO with parameters listed in Table 6b. 128*128 fully connected network and the Tanh activation function are used for VVC.
As BC only learns policy network, we add warm-up for the value network at the beginning of online learning. When online learning begins, we first freeze the parameter of the policy and train the value network with online samples until it converges, then proceed with the normal RL training, For the parameter X, 10 is utilized for all the experiments.
D Experiments on Reach and PointMaze
conduct two sets of experiments to demonstrate the general applicability of GCPO.
D.1 Experimental Setups
For the first set of experiments, we conduct evaluations customized PointMaze environment (PointMaze Large DIVERSE G-v3) from Gymnasium-Ron Robotics [17] within the Mu- joco physics engine. The only modification we made to the environment is to expand the number of desired goals from 7 to 45, making our customized version of PointMaze more challenging than the original version. For the second set of experiments, we employ customized Reach (PandaReach-v3) 23 task on the Franka Emika Panda robot physics engine. The only modification we made to the environment is to change the distance, threshold used determine goal reaching from 0.05 to 0.01.
Environments:
17

---

Table 7: Comparison between GCPO and baselines on Reach and PointMaze. The mean and variance of % success rates are presented over random seeds. Optimal values are highlighted in bold, and sub-optimal values are underlined.
Task
Reward
SAC+ + HER + MEGA
BC
GCPO
MR
100.0#0.0
70.63+2.99
100.0=0.0
Reach
NMR
0.72+1.34
10.52+11.70
80.26:+17.01
MR
100.0=0.0
75.96+5.34
93.33+3.06
PointMaze
NMR
4.17=0.93
22.8+3.71
47.50=8.06
Consequently, customized version of the Reach task has stricter criterion for determining goal arrival, making it more difficult than the original version of Reach.
The original rewards for both the PointMaze and Reach tasks are Markovian. To evaluate the performance of our algorithm under different NMR settings, we design two distinct types of NMRs. For the PointMaze, the NMR we designed is: the task is considered successful only if, after the point reaches the goal, it moves away by at least certain distance and then returns to the goal. For the Reach task, the NMR designed is: the Panda robot must first pass through specific waypoint before reaching the goal to be considered successful, and each goal has different waypoint. Both of these settings strictly adhere the definition of NMR, where the reward is defined by the states and actions over multiple steps.
Reward Settings:
Demonstrations: The demonstrations for PointMaze are sourced from Minari [1] mointmaze-largee vl), while the demonstrations for Reach are generated by us, with reference to the PID controller as described in the official documentation
44
D.2 Results
We evaluate SAC+HER+MEGA, BC, and GCPO on the PointMaze and Reach tasks under both MR and NMR settings. Table 7 presents presents the success rates of these algorithms. It can be observed that under the MR settings, GCPO exhibits similar performance to SAC+HER+MEGA However, under the NMR settings, where HER cannot be effective, the performance of GCPO significantly better than that of SAC. Taking into account the performance of GCPO on the VVC task as illustrated in we showcase the general applicability GCPO across variety of tasks.
Section 4.2
E
The Impact of Demonstrations on GCPO Training
In this section, we conduct a detailed experimental analysis of the impact of the distribution of goals that Demonstrations can cover, demonstration quantity, and demonstration quality on the training of GCPO.
E.1 Goal Distribution Covered by Demonstrations
To more intuitively illustrate the impact of the goal distribution in demonstrations on GCPO training, select three subsets of demonstrations from De g that are spatially distant from each other, compris- ing all x = 10 demonstrations for D Dix=10 all x = 90 demonstrations for Dxx= and all x = 170 demonstrations for D |x=170]
is evident that, for both the pre-trained policy and the GCPO policy, the achieved goals are distributed around the goal distribution of the demonstrations. This suggests that the goal distribution in the demonstrations biases the learning of both pre-training and GCPO, leading to policies that share a similar goal distribution the demonstrations. Therefore, the best practice when preparing demonstrations to make the distribution of goals covered by the demonstrations as closely resemble the desired goal distribution possible.
18

---

achieved
200
achieved by
200
# Tiz-
100
Miy 10)
Dlyn sor
5
Dly sol
1700
Dty
100
200
200
200
200
100
100
200
1.00
100
200
100
-100
4
-100
-200
(a) Goals covered by demonstrations and goals achieved by pre-trained policy
(b) Goals covered by demonstrations and goals achieved by GCPO policy
Figure 7: Visualization of goals covered by demonstrations and achieved by policies. The pre-trained policies with D Pix=10j- D/x=90j. and Dix=170] are denoted as n zÿ and πÿ. respectively. The corresponding GCPO policies are denoted as Ti, π2 and zÿ. respectively.
Table 8: Performance of GCPO Reach, PointMaze, and VVC with 1000 demonstrations, and more results on VVC with various demonstration quantities. The mean and variance of % success rates are presented over 5 random seeds.
Task
Demo Quantity
Transition Quantity
BC
GCPO
Reach
1000
82589
70.63+2.99
100.0#0.0
PointMaze
1000
1000000
75.9615.34
93.33+3.06
VVC +00DE
1000
294702
13.74+0.99
39.18+2.64
+VC+ + Xx=90
144
42177
1.18+0.38
3.34+0.61
VVC DE
10264
3106516
17.08+0.57
45.06+1.3
E.2 Demonstration Quantity
We note that commonly used RI demonstration datasets typically consist of 1000 demonstrations [22] Consequently, we evaluate the performance GCPO on Reach, PointMaze, and VVC with 1000 demonstrations. In addition, conduct ablation study demonstration quantity VVC. The results are presented in Table g
The source of demonstrations for PointMaze and Reach is elaborated in Section D For GCPO, 10%Dg is employed, as detailed in Section 4.3.1 Additionally, for comparison purposes, also present the performance of GCPO the VVC task using D Di== which comprises only 144 demonstrations with a flight path azimuth angle of 90, as described in detail in Appendix E and using Dg- The results simple PointMaze and Reach tasks, GCPO achieved nearly 100% success rate when using 1000 demonstrations. On the more challenging VVC task, the success rate with 1000 demonstrations reached 81.12% of the success rate achieved with 10264 demonstrations, while also being significantly higher than the success rate achieved with 144 demonstrations. These results indicate that across tasks of varying complexity, GCPO can achieve good performance with the use of 1000 demonstrations.
Demonstrations:
presented in Table 8 It can be observed that on the relatively
E.3 Demonstration Quality
To obtain demonstration set for comparison with Dr. we generate trajectories for these goals using all the policies trained in our experiments. For a specific goal, we retain only the shortest trajectory. These generated trajectories are denoted D'.
19

---

Table Demonstration quality of De and D'
Demonstrations
Quantity
4
I.
DE
10264
282.02+149.98
2.11+2.21
D'
10264
101.42432.41
10.1948.74
Table 10: Performance of GCPO VVC with demonstrations of different demonstration qualities. ((T) indicates the success rate of policy x. The mean and variance of % success rates are presented over 5 random seeds.
Demonstrations
<(TDC) f
[((BCC) 4
(x 4
s(FOCPo)
[((GGrr)
(xGaar))
DE
17.0840.57
241.72481.36
1.97
133.86+53.24 39.26+2.02
45.87+3.09
6.8425.60
D'
19. 122.81454.36
8.85#9.60
150.59#63.89
18.11=12.69
We use two metrics to measure demonstration quality: Trajectory length 4. Since we employ (- 1, 0) sparse rewards, this implies that shorter trajectories yield higher cumulative reward. Control smoothness J. In control problems, minimal control gains is expected to reduce wear on actuators. Hence, we refer to [39 to define the control smoothness. Trajectory length and control smoothness each describe certain characteristies of demonstrations from the distinct perspectives of reinforcement learning optimization and optimal control, respectively. Table 9
shows the demonstration quality of these two demonstration sets. It can be observed that: From an RL perspective, 'D' is of higher quality because the trajectories are shorter, leading to a higher expected cumulative reward. From a control perspective, DE is better because the trajectories are smoother.
The performance of the BC policy TBC and the GCPO policy TGCPO trained on these two sets of demonstrations is shown in Table 10. The results reveal that TBC closely aligns with the demon- strations on both quality metrics, indicating that demonstration quality has a direct impact on BC. Additionally, the BC policy trained on D' has slightly higher success rate, which speculate is due to D' being more suitable for RL (the network architecture and training hyperparameters used to generate D' are the same those for the BC policy). However, after the self-curriculum learning, the GCPO policy corresponding to DE performs better and exhibits a shorter trajectory length. This suggests that the influence of demonstration quality GCPO's online learning may not be as direct as pre-training, and further research is required to understand this relationship.
In summary, on one hand, it is challenging to define demonstration quality suitable for RL through a few metrics. On the other hand, demonstration quality does affect GCPO pre-training. How demonstration quality potentially influences the self-curriculum learning of GCPO remains an intriguing question for further exploration.
Furthermore, despite the aforementioned challenges, GCPO is capable of training well- rell-performed policies from non-expert demonstrations. The intrinsic reason is that GCPO employs online learning to fine-tune pre-trained policies. Consequently, even if the demonstrations are non-expert and the pre-trained policies perform poorly, GCPO can still continuously optimize these policies through online learning.
In Section 4.2 although the average trajectory length of D4 reached 281.83, covering only 20.24% of goal space, the GCPO policy trained on it achieves success rate of 45.87%, with an average trajectory length of 134.47. This comparison indicates that DD consists of non-expert demonstrations. On the other hand, in contrast to D which covers 78.55% of the goal space with an average trajectory length of 116.56. D is only a quarter in size and has trajectories that are 2.42 times longer, implying substantial decrease in its quantity and quality. Nonetheless, the GCPO policy trained on Dg achieves 76.58% of the success rate of the policy trained on D
F Extending GCPO with A Method for Solving NMR Problems
To demonstrate the effects of integrating a method for handling NMR problems into GCPO, we extend GCPO with basic method for addressing NMR problems, which involves extending the input
20

---

0.7
SAC-HER+MEGA frameslack
0.7
GCPO wio framestack
0.6
SAC-HER+MEGA w/ framestack
GCPO framestace dk
0.6
BC framestack
0.3
BC w/ framestack
Success Rote
Rate
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.0
1.0
Environment steps
Environment Steps
168
(a) SAC+HER+MEGA
(b) GCPO
Figure 8: Success rate of SAC+HER+MEGA and GCPO extended with consecutive states. 'w/ framestack' indicates that the policy input is extended with consecutive states, whereas 'w/o frames- tack denotes that the input remains the current state.
of the policy with consecutive states to make the reward Markovian 24 We evaluate the extended GCPO on the VVC task and show the result in Fig. 8
is evident that, regardless of whether it is the pre-trained policy or the GCPO policy or the SACHHER+MEGA policy, expanding the input of the policy leads certain degree of performance degradation. We conjecture that this is likely due to the expansion of the policy's 's search space, which makes the original number of demonstrations and online learning resources relatively insufficient, indirectly increasing the difficulty of solving the policy. Therefore, although expanding the input is theoretically beneficial for addressing NMR problems, in practice, also requires balance between the increased difficulty of the problem and the available resources.
G Societal Impacts
This paper introduces a general on-policy goal-conditioned reinforcement learning framework, GCPO, which is capable of addressing both Markovian and non-Markovian reward problems. This flexibility makes GCPO particularly well-suited for a wide range of practical applications that rely on non- Markovian rewards, such robotie arm control with stringent stability requirements and drone control that demands steady goal achievement. However, in practical settings, is imperative to pay close attention to the evaluation of the GCPO policy, carefully analyzing the disparities between the policy's 's achieved goal distribution and the desired goal distribution of the actual task. Allowing the policy attempt goals beyond its capabilities in production environment could result in hardware failures, damage, and other problems.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claim we make in the abstract and introduction is that propose
an on-policy goal-conditioned reinforcement learning framework. We detail the proposed framework in Sec. and evaluate the effectiveness of each component in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are discussed Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: Our main contribution is to propose an on- on-policy goal-conditioned reinforce- ment learning framework, in which we give references to the theories involved.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We detail all experimental details in Appendix A and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Please refer to
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttpcccpppppppppeeeeeeeeeeeeeeee http:////ggeeee/edeSSuuiiiissss /Cc////oddiiisssseeeeeee sww.iissisiiiiiiiico for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We detail experimental details in Appendix A and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: figures.
We average all results over 5 random seeds and show error bars suitably
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We show the training epochs of pre-training in Table 6a and compute workers and environment steps of online learning in Table 6b
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /tttps://neurii http:p/ccccppppccccccccccccteeaa
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our contribution poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Our work does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
26

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "pVKz2Y4FRJ", "reviewer": "Reviewer_U5WJ", "summary": "This paper proposes a new on-policy goal conditioned reinforcement learning framework which targets non-markovian rewards, which HER based approaches are unsuccessful. GCPO is a combination of offline pre-training from expert policies using behavior cloning, and online learning from a curriculum which automatically determines complex goals outside of the agent's current goal conditioned policy. They show this frameworks success in the Velocity Vector Control task, and perform several ablation and sensitivity studies.", "questions": "- How were the baselines and your method's hyperparameters tuned?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This paper provides a novel (as far as I am aware) framework for developing goal conditioned algorithms. This paper excels in clarity, interesting empirical and algorithmic contributions, and clear directions for future improvement. \n\n- **Clarity:** The paper is very clearly written, and provides ample details about the algorithms and experimental design. I also appreciate the work done in the introduction and related works section, clearly grounding this work in the literature and showing the deficiencies of HER approaches.\n- **Algorithmic contributions:** The GCRL framework is flexible enough to inspire a large amount of future work, and the authors do a great job enabling future research in this direction. The combination of pre-training and a goal-driven curriculum based on estimating successfully achieved goals with a GMM is interesting and novel.\n- **Empirical Work:** The empirical work is well rounded, and several interesting ablations going into the specific details of all the parts of the framework is appreciated and helps readers understand further. And the success on an extremely difficult domain is a great achievement in the space.\n- **Clear future directions** The authors are very clear on where their contributions are limited, and open the door to several future algorithmic contributions in this space.", "weaknesses": "While I think this paper would be an excellent contribution to this year's conference, there are some improvements I would like to see. I think the authors should make changes to address I-1, but the others are less consequential in my view.\n\n- I-1. I haven't seen a discussion around how hyperparameters were tuned. While I put this work in more of a demonstration lane, but I think if the paper was clearer around how these hyperparameters were chosen, and how the baseline methods were tuned would immediately strengthen the paper. I think this is critical when discussing the results of Figure 1. Adding how extensively these methods were tuned could strengthen the evidence supporting the claims.\n\n- I-2. While the ablation showing how the quantity of demonstration data on the final performance is appreciated, I think the lack of ablations on the quality of demonstrations is a major limitation. There are some experiments using demonstrations with different distributions of goals, these don't sufficiently capture what would happen if there is a lack of *expert* data. In many domains, there is often a lack of true expert data, but a large amount of sub-standard data. If GCPO is also performant in instances where expert data is sparse, but non-expert data isn't that would be a major achievement for the method (although my guess is issues would arise from using BC as you already state as a limitation).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1 & W1: How were the baselines and your method's hyperparameters tuned?\n\nWe employ Grid Search to search the following hyperparameters of the evaluated algorithms:\n\n|params|search range|\n|:-|:-|\n|SAC: Network Architecture|128\\*2, 128\\*3, 128\\*4, 128\\*5|\n|SAC: ent_coef|$10^{-3},10^{-2},10^{-1}$|\n|SAC: gamma|0.99, 0.995, 0.999|\n|SAC: lr|$10^{-3},10^{-4},10^{-5}$|\n|SAC: use_sde|True, False|\n|HER: buffer_size|$10^4,10^5,10^6$|\n|HER: n_sampled_goal|2,4,8,16|\n|HER: goal_selection_strategy|episode,final,future|\n|Self-curriculum: buffer_size|$10^3,10^4,10^5$|\n|BC: Network Architecture|128\\*2, 128\\*3, 128\\*4, 128\\*5|\n|BC: l2_weight|$0,10^{-4},10^{-3},10^{-2}$|\n|BC: ent_weight|$10^{-3},10^{-2},10^{-1}$|\n|BC: batch_size|512,1024,2048,4096|\n|PPO: Network Architecture|128\\*2, 128\\*3, 128\\*4, 128\\*5|\n|PPO: ent_coef|$10^{-3},10^{-2},10^{-1}$|\n|PPO: gamma|0.99, 0.995, 0.999|\n|PPO: lr|$10^{-3},10^{-4},10^{-5}$|\n|PPO: use_sde|True, False|\n\nThe other parameters are set to StableBaselines3 defaults. Taking the network architecture of BC as an example, once the optimal set of parameters has been searched, we maintain all other parameters constant and demonstrate the policy performance when using different network architectures with the following table:\n\n|Network Architecture|128\\*2|128\\*3|128\\*4|128\\*5|\n|:-:|:-:|:-:|:-:|:-:|\n|Success Rate|17.08±0.57|16.56±0.91|15.31±0.56|14.78±2.45|\n\nAs [128, 128] achieved the best performance, we maintain this architecture for all experiments on BC.\n\n> W2: the lack of ablations on the quality of demonstrations is a major limitation. How does GCPO perform on non-expert demonstrations?\n\n**Firstly, we supplement experiments on demonstration quality.**\n\nIn Section 4.2, $\\mathcal{D}_E$ covers 10264 goals. We generate trajectories for these goals with all the policies trained in our experiments. For a specific goal, we retain only the shortest trajectory. These generated demonstrations are denoted as $\\mathcal{D}'$.\n\nWe use two metrics to measure demonstration quality: **Trajectory length** $I_l$. Since we employ (-1, 0) sparse rewards, this implies that shorter trajectories yield a higher cumulative reward. **Control smoothness** $I_s$. In control problems, minimal control gains is expected to reduce wear on actuators. Hence, we refer to [1] to define the control smoothness. Trajectory length and control smoothness each describe certain characteristics of demonstrations from the distinct perspectives of reinforcement learning optimization and optimal control, respectively.\n\n| Demo | #$\\mathcal{D}$ | $I_l(\\mathcal{D}) \\downarrow$ | $I_s(\\mathcal{D}) \\downarrow$ |\n|:-:|:-:|:-:|:-:|\n|$\\mathcal{D}_E$|10264|282.02±149.98|2.11±2.21|\n|$\\mathcal{D}'$|10264|101.42±32.41|10.19±8.74|\n\nIt can be observed that: From an RL perspective, $\\mathcal{D}'$ is of higher quality because the trajectories are shorter, leading to a higher expected cumulative reward. From a control perspective, $\\mathcal{D}_E$ is better because the trajectories are smoother.\n\nThe performance of the BC policy $\\pi_{BC}$ and the GCPO policy $\\pi_{GCPO}$ trained on these two sets of demonstrations is:\n\n| Demo | $s(\\pi_{BC}) \\uparrow$ | $I_l(\\pi_{BC}) \\downarrow$ | $I_s(\\pi_{BC}) \\downarrow$ | $s(\\pi_{GCPO}) \\uparrow$ | $I_l(\\pi_{GCPO}) \\downarrow$ | $I_s(\\pi_{GCPO}) \\downarrow$ |\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|$\\mathcal{D}_E$|17.08±0.57|241.72±81.36|1.97±1.97|45.87±3.09|133.86±53.24|6.84±5.60|\n|$\\mathcal{D}'$|19.10±0.22|122.81±54.36|8.85±9.60|39.26±2.02|150.59±63.89|18.11±12.69|\n\nThe results reveal that $\\pi_{BC}$ closely aligns with the demonstrations on both quality metrics, indicating that demonstration quality has a direct impact on BC. Additionally, the BC policy trained on $\\mathcal{D}'$ has a slightly higher success rate, which we speculate is due to $\\mathcal{D}'$ being more suitable for RL (the network architecture and training hyperparameters used to generate $\\mathcal{D}'$ are the same as those for the BC policy). However, after the self-curriculum learning, the GCPO policy corresponding to $\\mathcal{D}_E$ performs better and exhibits a shorter trajectory length. This suggests that the influence of demonstration quality on GCPO's online learning may not be as direct as pre-training, and further research is required to understand this relationship.\n\nIn summary, on one hand, it is challenging to define demonstration quality suitable for RL through a few metrics [2]. This is a research direction that deserves further exploration. On the other hand, demonstration quality does affect GCPO pre-training. How demonstration quality potentially influences the online self-curriculum learning of GCPO remains an intriguing question for further exploration.\n\n**Secondly, GCPO is capable of training well-performed policies from non-expert demonstrations.**\nPlease refer to the response to Q2 in the Global Author Response for a detailed analysis.\n\n[1] Mysore S, Mabsout B, Mancuso R, et al. Regularizing action policies for smooth control with reinforcement learning[C]//International Conference on Robotics and Automation. 2021.\n\n[2] Belkhale S, Cui Y, Sadigh D. Data quality in imitation learning[C]//Advances in Neural Information Processing Systems. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vYfF83cbb6", "reviewer": "Reviewer_emMz", "summary": "This paper proposes Goal-Conditioned Policy Optimization (GCPO), an on-policy variant for goal-conditioned RL that can also handle non-markovian reward structures. Common goal-conditioned RL methods are usually related to Hindsight Experience Replay (HER) which however can only solve tasks under the Markovian properties. Notably, GCPO leverages pre-training from demonstrations and online self-curriculum learning that can progressively select challenging goals based on the current learning process of the policy.", "questions": "- The method requires a desired goal distribution which might be task dependent. In which sense is this restricting the algorithm's applicability?\n- How exactly is the GMM estimated? No information was given in the main text \n- The work states that the self-curriculum \"leads to an average 8.2% increase in policy performance\" as evidenced in Table 2. However, Table 2 also shows that GCPO w/o pre-training can only achieve 4% success rate indicating that the performance boost is mainly achieved by the imitation learning policy. On the other hand, Fig.3 shows that GCPO successfully reaches more goals than the expert and BC, especially in the goal difficulty up to 0.7 but is not able to significantly reach more difficult goals. This observation is also discussed in Section 4.3.2.\nWhile I see that the goals that are already covered by the BC can be reached successfully more often by GCPO, the self-curriculum does not seem to cover more difficult goals significantly. Section 4.3.2 suggests to \"sample goals and generate demonstrations as closely as possible to the desired goal distribution\". Isn't this restricting the method in the sense that the demonstration data needs to be collected accordingly?\n\n- Connecting to the question before, is this indicating that exploring new goals that are much more difficult can not covered well by GCPO? Could this be discussed in more detail? Why is this the case? Is the KL constraint restricting the updates to more difficult goals? To my understanding, expanding to more difficult goals should be covered by the bonus (MEGA) as described in Section 3.3.", "limitations": "- While the detailed ablations are highly appreciated, a more thorough evaluation across more tasks is needed to assess the effectiveness of the method. This can be done by using sophisticated existing non-markovian rewarded environments, or, for demonstration purposes, adjusting the reward accordingly in existing benchmark suits such as e.g. (Meta World)\n\nMeta World: https://github.com/Farama-Foundation/Metaworld \n\nPlease also see the weaknesses and questions.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- well-motivated\n- contributions are well outlined\n- detailed ablations\n- An important and interesting problem is considered", "weaknesses": "The paper lacks discussion of relevant works that also consider contextual/goal-conditioned RL with self-curriculum learning with similar motivations, some of which consider non-Markovian rewarded environments. Here are some of those:\n\n- Klink et al. Self-Paced contextual reinforcement learning (CoRL 2019)\n- Celik et al. Specializing Versatile Skill Libraries using Local Mixture of Experts (CoRL 2021)\n- Klink et al. Self-Paced Deep Reinforcement Learning (NeurIPS 2020)\n- Ottot et al. Deep Black-Box Reinforcement Learning with Movement Primitives (CoRL 2022)\n- Celik et al. Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts (ICML 2024)\n\nplease also see the questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Response to Q3 and Q4\n\nWe consolidate your two questions into three sub-questions.\n\n1. _The effectiveness of GCPO seems to primarily come from the pre-training, and the self-curriculum does not appear to significantly cover more difficult goals._\n\nThe objective of GCRL is to achieve more goals under the expectation of $p_{dg}$, which is independent of any form of goal difficulty. Both MEGA and our GCPO do not alter the objective of GCRL, so the appropriate evaluation metric should be the success rate over $p_{dg}$. Therefore, considering the change in success rate, both pre-training and self-curriculum play significant roles. Specifically:\n\n* In GCRL, difficulty describes how easy or hard it is for a policy to achieve a goal while learning. This difficulty is closely linked to the policy's abilities, and we prefer to term it **learning-process-oriented difficulty**. Conversely, **task-oriented difficulty** pertains to the inherent challenge of a goal, unrelated to the policy's form or learning process. It is dictated by the goal itself. A goal that is task-oriented easy might be learning-process-oriented hard. There is no inherent link between these two difficulty types.\n* GCRL aims to maximize expected cumulative rewards under the goal distribution $p_{dg}$. When rewards are binary indicators of goal achievement, GCRL's objective is essentially to achieve more goals under $p_{dg}$. Hence, **GCRL's objective is inherently independent of any goal difficulty**. MEGA, while identifying goals by learning-process-defined difficulty to optimize exploration and learning, does not alter GCRL's objective. Thus, MEGA is not centered on achieving more task-defined difficult goals.\n\n* In VVC, we define $p_{dg}$ as the uniform distribution over the entire goal space. For clarity, we define task-oriented difficulty in Appendix A.1.4 and use it as the x-axis in our figures. Fig.1(a) in the Global Author Response PDF shows all discrete goals in a 3D space. Fig.1(b) depicts the goal distribution over task-oriented difficulty. It is evident that, under our definition, the distribution of desired goals over task-oriented difficulty is not uniform but rather low at both ends and high in the middle. This is exactly why our experimental results exhibits this particular shape.\n\nIn essence, the valid metric for evaluating GCRL is the success rate under $p_{dg}$. Table 2 shows that the success rate of GCPO is 2.69 times that of BC, indicating that online learning with self-curriculum can significantly improve a pre-trained policy. Fig.3(b) and 3(c) are merely analyses on learned policies and learning process from the task-oriented difficulty perspective. We find that self-curriculum does gradually sample task-oriented difficult goals during training, and the resulting policies can achieve some of these goals.\n\n2. _Does the requirement for sampling demonstrations in Section 4.3.2 limit the applicability of the method?_\n\nIn Section 4.3.2, the experiment examines how demonstration goal distribution affects GCPO. Results indicate that demonstrations sampled according to $p_{dg}$ yield effective GCPO policies. We believe that this is a point to focus on for enhancing GCPO's performance, rather than a limitation of its applicability. Comparison between the results in Sections 4.3.2 and 4.2 reveals that GCPO can still train when the demonstration goal distribution differs substantially from $p_{dg}$. For instance, using $\\mathcal{D}_2$ in Section 4.3.2, with 100 demonstrations (cover only 0.20% goal space) focused on difficulties within (0.2328, 0.2336), GCPO can still train a policy with a 21.86% success rate.\n\n3. _What impact does KL have on training?_\n\nIndeed, KL impacts GCPO's training. As discussed in our response to the first sub-question, we believe that this impact is only reflected in the success rate, not in the achievement of the task-oriented difficult goals. We train GCPO with various $\\lambda$:\n\n|$\\lambda$|KL Value|Success Rate|\n|:-:|:-:|:-:|\n|$10^{-1}$|1.79±0.42|25.37±3.68|\n|$10^{-2}$|9.27±0.82|38.75±1.83|\n|$10^{-3}$|33.27±2.34|45.87±3.09|\n|$10^{-4}$|133.32±26.42|28.20±5.31|\n|0|368.19±15.72|25.68±4.05|\n\nAs shown, a higher $\\lambda$ leads to a lower KL, suggesting the GCPO policy is more statistically similar to the pre-trained policy, aligning its success rate with the pre-trained policy's 17.08%. As $\\lambda$ decreases, KL increases, and the GCPO policy's success rate improves. Yet, at $\\lambda < 10^{-3}$, the success rate drops, which we attribute to the weak KL constraint leading to catastrophic forgetting of the pre-trained knowledge, impeding effective policy learning.\n\n> Response to Q1\n\nThe desired goal distribution is a part of the problem formulation of GCRL. We contend that this distribution, which should be a known condition, does not restrict GCPO. Even if the demonstration goal distribution significantly differs from the desired goal distribution, GCPO can still train a reasonably good policy. For a detailed analysis, please refer to our response to the second sub-problem in the first question.\n\n> Response to Q2\n\nGMM estimates a distribution with a weighted sum of multiple Gaussian kernels, $\\sum_{i=1}^M \\pi_i N(x|\\mu_i,\\Sigma_i)$, and estimate parameters with the Expectation-Maximization algorithm. Due to space constraints, we will include detailed calculation in Appendix.\n\n> Response to W1\n\nThank you for the suggestion. We've thoroughly read the papers, with four focusing on curriculum learning and one on NMR. The key contribution of GCPO is its on-policy GCRL training framework, where self-curriculum is vital for efficient online learning. Thus, other curriculum learning methods could replace MEGA within GCPO. We will discuss these works in the related work section. The episode-based RL research offers valuable guidance for enhancing GCPO's suitability for NMR problems. We will explore this possibility in the future work.\n\n> Response to L1\n\nPlease refer to Q1 in Global Author Response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uaNjblL5qb", "reviewer": "Reviewer_aw7R", "summary": "After rebuttal\n\nThe authors have mostly addressed my minor concerns. I recommend acceptance.\n\n------\n\n\nThis paper aims to improve goal-conditioned RL. Two problems with prior methods (e.g. HER) are discussed, namely (a) their inability to cope with non-markovian rewards; and (b) the necessity of using an off-policy algorithm. This work then goes on to introduce a method that is applicable to on-policy algorithms *and* NMR problems. This method first uses demonstrations to pre-train a policy to achieve the goals sometimes. Then, it uses a curriculum-based approach to select harder and harder goals (but still within the agent's capability). They demonstrate on a UAV problem with NMR, they outperform other methods, notably SAC + HER.", "questions": "- Could you run GCPO with only a 1000 demonstrations?", "limitations": "1. Requiring demonstrations is a relatively strong limitations, as e.g. HER does not. Is there any way around this requirement?", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Having a good on-policy GCRL algorithm is a great thing to shoot for, and the problem is useful.\n2. Solving the problem of existing methods not being applicable to NMR domains is beneficial.\n3. The set of baseline algorithms/ablations makes sense, and most of the obvious things to compare against have been included.", "weaknesses": "1. It would be nice to have experiments on another domain, to illustrate the general applicability of your algorithm. \n\t1. In particular, having results of the baseline algorithms (and GCPO) in an MR domain would be very helpful.\n2. Relatedly, having results for e.g. SAC + HER when doing the same thing as in appendix A.4 would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We address the concern about GCPO performance in more RL domains (W1) in Global Author Response, and the other concerns below.\n\n> Q1: Could you run GCPO with only a 1000 demonstrations?\n\nWe evaluate the performance of GCPO with 1000 demonstrations on the Reach, PointMaze, and VVC tasks.\n\n**Demonstrations**: The source of demonstrations for PointMaze and Reach is elaborated in the Global Author Response. For GCPO, 10% $\\mathcal{D}_E$ is employed, as detailed in Section 4.3.1. \n\nAdditionally, for comparison purposes, we also present the performance of GCPO on the VVC task using $\\mathcal{D}_{[\\chi=90]}$, which comprises only 144 demonstrations with a flight path azimuth angle of 90, as described in detail in Appendix A.3, and using $\\mathcal{D}_E$. The results are presented in the following table:\n\n|Task|Demo Quantity|Transition Quantity|BC|GCPO|\n|:---:|:---:|:---:|:---:|:---:|\n|Reach|1000|82589|70.63±2.99|100.0±0.0|\n|PointMaze|1000|1000000|75.96±5.34|93.33±3.06|\n|VVC + 10% $\\mathcal{D}_E$|1000|294702|13.74±0.99|39.18±2.64|\n|VVC + $\\mathcal{D}_{[\\chi=90]}$|144|42177|1.18±0.38|3.34±0.61|\n|VVC + $\\mathcal{D}_E$|10264|3106516|17.08±0.57|45.06±1.3|\n\n**Performance**: It can be observed that on the relatively simple PointMaze and Reach tasks, GCPO achieved nearly 100% success rate when using 1000 demonstrations. On the more challenging VVC task, the success rate with 1000 demonstrations reached 81.12% of the success rate achieved with 10264 demonstrations, while also being significantly higher than the success rate achieved with 144 demonstrations. These results indicate that **across tasks of varying complexity, GCPO can achieve good performance with the use of 1000 demonstrations.**\n\n> L1: Requiring demonstrations is a relatively strong limitations, as e.g. HER does not. Is there any way around this requirement?\n\nWe answer this question from two perspectives: Firstly, while GCPO's training relies on demonstrations, the quantity and quality of these demonstrations do not need to be high. GCPO is capable of learning well-performed policies from non-expert demonstrations. Secondly, for complex tasks, methods like SAC and HER are also unable to learn from scratch; leveraging demonstrations to assist learning is a more mainstream approach. We will elaborate on these points in the following discussion.\n\nOn one hand, although GCPO relies on demonstrations, it is capable of learning well-performed policies from non-expert demonstrations. Please refer to the response to Q2 in the Global Author Response for a detailed analysis.\n\nOn the other hand, for complex tasks, methods such as SAC, HER, etc., also struggle to learn well-performed policies from scratch. For instance, on the VVC task, the performance of policies obtained using SAC+HER+MEGA is shown in the table below, where it can be observed that this method barely learn any capability to complete tasks. In other complex tasks, such as StarCraft [1], Minecraft [2], ObjectGoal Navigation [3], and others, researchers widely rely on demonstrations to train RL policies. Therefore, for the complex tasks mentioned above, we have not yet found a RL method that can bypass the pre-train phase. Hierarchical RL may be a direction worth exploring [4].\n\n||SAC+HER+MEGA|BC|GCPO|\n|:---:|:---:|:---:|:---:|\n|Success Rate|5.75±1.98|17.08±0.57|45.87±3.09|\n\nIn summary, on complex tasks, we have not yet found an RL method that can bypass the use of demonstrations. **Although the training of GCPO depends on demonstrations, its capability to learn policies from imperfect demonstrations somewhat relaxes the conditions for using GCPO**.\n\n[1] Vinyals O, Babuschkin I, Czarnecki W M, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning[J]. nature, 2019, 575(7782): 350-354.\n\n[2] Baker B, Akkaya I, Zhokov P, et al. Video pretraining (vpt): Learning to act by watching unlabeled online videos[J]. Advances in Neural Information Processing Systems, 2022, 35: 24639-24654.\n\n[3] Ramrakhya R, Batra D, Wijmans E, et al. Pirlnav: Pretraining with imitation and rl finetuning for objectnav[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 17896-17906.\n\n[4] Pope A P, Ide J S, Mićović D, et al. Hierarchical reinforcement learning for air-to-air combat[C]//2021 international conference on unmanned aircraft systems (ICUAS). IEEE, 2021: 275-284.\n\n> W2: Relatedly, having results for e.g. SAC + HER when doing the same thing as in appendix A.4 would be helpful.\n\nWe conduct experiments related to SAC+HER with reference to the experimental setup for GCPO described in Appendix A.4. The training process curves are shown in Fig.2 in the Global Author Response PDF. It can be observed that expanding the input of the SAC+HER policy also results in a certain degree of performance degradation. We believe that the reasons are consistent with the analysis provided in Appendix A.4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate your valuable feedback and careful review of our paper. We address the two common concerns of many reviewers in the following:

> Q1: How does GCPO perform on other RL domain tasks?

**We conduct two sets of experiments to demonstrate the general applicability of our method.**

**Environments**: For the first set of experiments, we conduct evaluations on a customized PointMaze environment (PointMaze_Large_DIVERSE_G-v3) from Gymnasium-Robotics [1] within the Mujoco physics engine. The only modification we made to the environment is to expand the number of desired goals from 7 to 45, making our customized version of PointMaze more challenging than the original version. For the second set of experiments, we employ a customized Reach (PandaReach-v3) task on the Franka Emika Panda robot physics engine. The Reach task is akin to those in the Meta-World environment, both involving robotic arm tasks where the objective is to reach a specified goal state. The only modification we made to the environment is to change the distance_threshold used to determine goal reaching from 0.05 to 0.01. Consequently, our customized version of the Reach task has a stricter criterion for determining goal arrival, making it more difficult than the original version of Reach.

**Reward Settings**: The original rewards for both the PointMaze and Reach tasks are Markovian. To evaluate the performance of our algorithm under different NMR settings, we design two distinct types of NMRs. For the PointMaze, the NMR we designed is: the task is considered successful only if, after the point reaches the goal, it moves away by at least a certain distance and then returns to the goal. For the Reach task, the NMR we designed is: the Panda robot must first pass through a specific waypoint before reaching the goal to be considered successful, and each goal has a different waypoint. Both of these settings strictly adhere to the definition of NMR, where the reward is defined by the states and actions over multiple steps. 

**Demonstrations**: The demonstrations for PointMaze are sourced from Minari [3] (pointmaze-large-v1), while the demonstrations for Reach are generated by us, with reference to the PID controller as described in the official documentation [4].

**Performance**: We evaluate SAC+HER+MEGA, BC, and GCPO on the PointMaze and Reach tasks under both MR and NMR settings. The following table presents the success rates of these algorithms. It can be observed that under the MR settings, GCPO exhibits similar performance to SAC+HER+MEGA. However, under the NMR settings, where HER cannot be effective, the performance of GCPO is significantly better than that of SAC. Taking into account the performance of GCPO on the VVC task as illustrated in the main paper, we showcase the general applicability of GCPO across a variety of tasks.

|Task|Reward|SAC+HER+MEGA|BC|GCPO|
|:---:|:---:|:---:|:---:|:---:|
|Reach|MR|100.0±0.0|70.63±2.99|100.0±0.0|
|Reach|NMR|0.72±1.34|10.52±11.70|80.26±17.01|
|PointMaze|MR|100.0±0.0|75.96±5.34|93.33±3.06|
|PointMaze|NMR|4.17±0.93|22.8±3.71|47.50±8.06|

_**Note**: Due to the inability of HER to get rewards in the NMR settings, in the experiments, SAC+HER+MEGA is employed for the MR settings, while SAC is employed for the NMR settings._

[1] https://robotics.farama.org/

[2] Gallouédec Q, Cazin N, Dellandréa E, et al. panda-gym: Open-source goal-conditioned environments for robotic learning[C]//4th Robot Learning Workshop: Self-Supervised and Lifelong Learning@ NeurIPS 2021. 2021.

[3] https://minari.farama.org/

[4] https://panda-gym.readthedocs.io/en/latest/usage/manual_control.html

> Q2: How does GCPO perform on non-expert demonstrations?

**GCPO is capable of training well-performed policies from non-expert demonstrations.** The intrinsic reason is that GCPO employs online learning to fine-tune pre-trained policies. Consequently, even if the demonstrations are non-expert and the pre-trained policies perform poorly, GCPO can still continuously optimize these policies through online learning.

In Section 4.2, although the average trajectory length of $\mathcal{D}_E^0$ reached 281.83, covering only 20.24% of goal space, the GCPO policy trained on it achieves a success rate of 45.87%, with an average trajectory length of 134.47. This comparison indicates that $\mathcal{D}_E^0$ consists of non-expert demonstrations. On the other hand, in contrast to $\mathcal{D}_E^3$, which covers 78.55% of the goal space with an average trajectory length of 116.56, $\mathcal{D}_E^0$ is only a quarter in size and has trajectories that are 2.42 times longer, implying a substantial decrease in its quantity and quality. Nonetheless, the GCPO policy trained on $\mathcal{D}_E^0$ achieves 76.58% of the success rate of the policy trained on $\mathcal{D}_E^3$.

Therefore, we contend that GCPO has the capability to train from non-expert demonstrations.

Meta-review: The paper presents an on-policy Goal-Conditioned Reinforcement Learning (GCRL) framework (GCPO), which addresses the limitations of existing HER-based methods, in handling non-Markovian rewards (NMR). GCPO combines pre-training from demonstrations with an online self-curriculum learning strategy, allowing it to progressively tackle more challenging goals based on the agent's current capabilities. The authors demonstrate the efficacy of GCPO on a challenging task involving fixed-wing UAV velocity vector control, showcasing its applicability to both Markovian and non-Markovian reward problems. The paper shows the strength in handling both Markovian and non Markovian reward scenarios. The rebuttal helped clarify some concerns such as that on general applicability to diverse tasks and demonstration quality. Given the overall contributions and its strong empirical performance, I recommend the paper for acceptance with a revision to address concerns raised through the reviewer discussions.
Decision: Accept (poster)