Goal-Conditioned On-Policy Reinforcement Learning

Authors: xudong  gong
dawei  feng
kele  xu
bo  ding
huaimin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 05:17:53

Abstract: Existing Goal-Conditioned Reinforcement Learning (GCRL) algorithms are built upon Hindsight Experience Replay (HER), which densifies rewards through hindsight replay and leverages historical goal-achieving information to construct a learning curriculum. However, when the task is characterized by a non-Markovian reward (NMR), whose computation depends on multiple steps of states and actions, HER can no longer densify rewards by treating a single encountered state as the hindsight goal. The lack of informative rewards hinders policy learning, resulting in rolling out failed trajectories. Consequently, the replay buffer is overwhelmed with failed trajectories, impeding the establishment of an applicable curriculum. To circumvent these limitations, we deviate from existing HER-based methods and propose an on-policy GCRL framework, GCPO, which is applicable to both multi-goal Markovian reward (MR) and NMR problems.
GCPO consists of (1) Pre-training from Demonstrations, which pre-trains the policy to possess an initial goal-achieving capability, thereby diminishing the difficulty of subsequent online learning. (2) Online Self-Curriculum Learning, which first estimates the policy's goal-achieving capability based on historical evaluation information and then selects progressively challenging goals for learning based on its current capability. We evaluate GCPO on a challenging multi-goal long-horizon task: fixed-wing UAV velocity vector control. Experimental results demonstrate that GCPO is capable of effectively addressing both multi-goal MR and NMR problems.

Main paper: 
 introduction: 
 Multi-goal problems are ubiquitous in real-world applications, such as controlling robotic arms to grasp objects at any location on a table [13, 16] , and operating fixed-wing Unmanned Aerial Vehicles (UAVs) to navigate towards any specified velocity vector [8, 34] , etc. To address the challenge of automatically learning policies capable of achieving and generalizing across a range of diverse goals [47] , Goal-Conditioned Reinforcement Learning (GCRL) [48, 37] has emerged as a prominent area of research. Serving as a generalization of standard Reinforcement Learning (RL) [61] , GCRL learns goal-conditioned policies [55] through interactions within multi-goal environments [48] .In existing GCRL algorithms, Hindsight Experience Replay (HER) [3] plays a pivotal role in facilitating the learning of goal-conditioned policies. First, HER enhances sample efficiency by replacing the desired goals of failed trajectories with achieved states, thereby providing more informative rewards for policy learning [70, 11] . Second, HER contributes to creating a curriculum that enables the policy to progressively master challenging goals [40, 69, 47] . This process involves fitting the current policy's goal-achieving capability with historical goal-achieving information and selecting goals of appropriate difficulty for the current policy learning. Due to the reliance on replay buffers [72] , current research on GCRL predominantly focuses on off-policy RL approaches [51, 25] .: × → ℝ : × * → ℝHindsight works: Assuming is the desired goal, then , = 1, indicating that 0: , 0: is a successful trajectory for achieving . Success Rate NMR [10] w/ HER NMR [20] w/ HER NMR [30] w/ HER NMR [10] w/o HER Despite its widespread success, HER is subject to a limitation: the reward function must be able to determine goal achievement based solely on the current state, which implies that the reward function must be Markovian [2] . In real-world applications, many reward functions depend on multiple steps of states and actions, i.e., rewards are non-Markovian [24, 1] . For instance, determining whether an UAV has stably achieved a goal based on a sequence of states, or calculating a penalty for control oscillation based on a sequence of actions [33, 8] . For non-Markovian reward (NMR) problems, on one hand, it is not feasible to treat a single state from the sampled trajectory as a hindsight goal; on the other hand, it is challenging for the trajectory obtained through exploration to include the special transition sequence that satisfies the specific NMR. In Fig. 1 , we present a specific case to elucidate the NMR problem and illustrate why HER fails to address NMR, and corresponding experimental results to support this explanation. It is evident that as the NMR relies on longer state-action sequences, the performance of HER gradually deteriorates, ultimately becoming indistinguishable from the performance when HER is not employed. This limitation prevents HER from densifying the reward. The lack of informative rewards hinders policy learning, which further leads to rolling out failed trajectories. Consequently, the replay buffer is overwhelmed with failed trajectories, obstructing the creation of a reasonable curriculum.In light of the aforementioned limitations of HER in addressing NMR problems, we propose a novel on-policy [62, 57] GCRL framework, termed the Goal-Conditioned Policy Optimization (GCPO) framework, which deviates from existing HER-based off-policy approaches. GCPO comprises: (1) Pre-training from Demonstrations: This phase involves leveraging demonstrations to pre-train the policy offline, equipping it with an initial capability to achieve goals before online learning begins, thereby diminishing the difficulty of subsequent online learning. (2) Online Self-Curriculum Learning: This phase involves periodically evaluating the policy, estimating its goal-achieving capability, and optimizing it through online learning with progressively challenging goals sampled based on its current capability. It is important to note that GCPO is not specifically designed for NMR problems; instead, it is a general on-policy GCRL framework applicable to both multi-goal Markovian reward (MR) and NMR problems.To evaluate GCPO, we conduct experiments on the Velocity Vector Control (VVC) task of fixed-wing UAVs. The VVC task represents a typical multi-goal problem that can be formulated as both MR and NMR, and it also requires long interaction sequences, categorizing it as a challenging long-horizon problem [28] . The complexity inherent in the VVC task provides a rigorous testbed for evaluating the efficacy of GCPO. Experimental results indicate that GCPO effectively addresses multi-goal problems, both MR and NMR. Our contributions are summarized as follows:• We analyze the reason why existing GCRL algorithms, which rely on HER as a central component, fail in handling NMR problems and validate this finding through experiments.• We propose an on-policy GCRL framework, GCPO, which incorporates pre-training from demonstrations and online self-curriculum learning to address both MR and NMR problems.• We evaluate the performance of GCPO on the challenging VVC task, demonstrating its effectiveness in solving both MR and NMR problems. Additionally, we conduct ablation studies to analyze the influence of the two components and hyper-parameters on learning.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claim we make in the abstract and introduction is that we propose an on-policy goal-conditioned reinforcement learning framework. We detail the proposed framework in Sec. 3 and evaluate the effectiveness of each component in Sec. 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Goal-conditioned reinforcement learning. Many prior works assume access to a goal-conditioned reward function [15, 73] and view GCRL as a reward-driven, multi-task learning problem [68] .Existing GCRL methods are predominantly based on HER, situating them within the off-policy RL domain [61] . Pitis et al. [47] summarize existing methods and propose a common off-policy GCRL framework, which alternates between collecting experience and optimizing policies. During the policy optimization phase, the hindsight replay method is utilized to relabel transitions sampled from the replay buffer, thereby increasing informative rewards in the training data. When collecting experience, some heuristics or learning methods are employed to select appropriate goals that assist in improving the policy.Selecting appropriate goals involves (1) several heuristics for discovery, including reward relevance [4, 65] , diversity [14, 10] , coverage [38, 6] , difficulty [32, 43] , etc. (2) selecting goals of appropriate difficulty based on the capability of the current policy, which essentially craft objectives that try to optimize for learning progress. These methods fit the current policy's goal-achieving capability based on the trajectories stored in the replay buffer and then sampling goals of appropriate difficulty for online learning according to this capability. For instance, RIG [40] samples goals directly from the distribution of achieved goals, DISCERN [69] samples uniformly on the support set of the distribution of achieved goals, and MEGA [47] uses inverse probability weighting sampling [32] on the distribution of achieved goals to samples goals that the current policy can achieve but not well.The difference between our method and the aforementioned methods lies in the learning framework: our method is on-policy, whereas the existing methods, all based on HER, are off-policy. This difference manifests in two key aspects: First, our method does not include a component like HER, which cannot be used for solving NMR problems. To achieve an effect similar to HER's enhancement of informative rewards during learning, we design a component that pre-trains the policy with demonstrations, making it suitable for on-policy RL. Second, our method employs off-policy evaluation [63, 64] to estimate the current policy's goal-achieving capability, rather than estimating this capability based on the trajectories in the replay buffer. Table 1 summarizes the similarities and differences between GCPO and existing GCRL methods. Non-Markovian rewards. Abel et al. [2] underscore the necessity of NMRs by demonstrating the existence of environment-task pairs for which no MR function can realize the task. Moreover, numerous exploration strategies implicitly depend on NMRs [35, 45] , and studies considering non-Markovian discount factors [19, 58] can also be interpreted as special forms of NMRs [46] . The fundamental approach for addressing NMRs involves augmenting the state space to render the reward Markovian [24] . Various techniques have been proposed to achieve this, such as Reward Machines [9] and the Split-MDP [1] . Nevertheless, the expanded reward state space may grow exponentially with the number of acceptable policies and could incorporate an infinite number of simple reward functions [1] . Our work diverges from the aforementioned research by presenting a general GCRL framework, rather than a specialized approach for NMR problems. Unlike existing GCRL methods that depend on HER, which is not applicable to NMR problems, our framework can be applied to Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of the work are discussed in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 GCPO is designed as an on-policy GCRL framework. We draw upon the key insights from existing HER-based GCRL methods that have led to their success and incorporate two critical components into the GCPO framework: pre-training from demonstrations and online self-curriculum learning. The overall GCPO framework is depicted in Fig. 2 and a practical implementation of GCPO is detailed in Algorithm 1.GCRL can be described by goal-augmented MDP [37] M = ⟨S, A, T , r, γ, G, p dg , ϕ⟩, where S, A, γ, G and p dg denote the state space, action space, discount factor, goal space and desired goal distribution of the environment, respectively. T : S × A → P(S) is the transition function, where P(X ) denote the probability distribution over a set X . r is the goal-conditioned reward function. It can be both Markovian r = {r g |r g : S × A → R, g ∈ G} and non-Markovian r = {r g |r g : (S × A) * → R, g ∈ G}. ϕ : S → G is a tractable mapping function that maps the state to a specific goal. The objective of GCRL is to reach goals via a goal-conditioned policy π : S × G → P(A) that maximizes the expectation of the cumulative rewards over the desired goal distribution J(π) = E at∼π(•|st,g),g∼p dg ,st+1∼T (•|st,at) [ t γ t r g (•)] . Additionally, previous works [47, 37] identify two common definitions of goals: Achieved goal, which refers to the goal accomplished by the policy in the current state. The notation p ag denotes the distribution of achieved goals. Behavioral goal, which represents the specific task that is targeted for sampling within a rollout episode [37] .GCRL encounters more substantial exploration challenges compared to standard RL due to the inclusion of an additional goal space [30] . Pre-training from demonstrations is primarily designed to facilitate biased exploration [50] . Specifically, the policy can be pre-trained with Imitation Learning (IL) [74] or goal-conditioned IL [13, 25] on demonstrations. The pre-training provides the policy with a warm start [60, 71] , which refers to an initial ability to achieve some of the desired goals. Pre-training is vital for on-policy RL as it enhances informative rewards during online learning. Without such informative rewards, the policy would struggle to acquire any meaningful knowledge or skills [61] . Through subsequent online learning, the policy can effectively discern when it is more advantageous to adhere to states and actions from the demonstration trajectories or to explore superior alternatives.In our implementation of GCPO, we utilize Behavioral Cloning (BC) [49] for pre-training the policy, as indicated in line 2 of Algorithm 1. For the demonstration D E , the policy is learned by optimizing a supervised loss function to maximize the likelihood of expert actions [54] update π θ by Eq. 3 on D ▷ update policy 21: end whileEQUATIONIn GCRL, selecting goals that match the current policy's capabilities is crucial for effective learning [20, 12, 5, 11] . To address this, we design an online self-curriculum learning mechanism that autonomously constructs a curriculum, generating behavioral goals that are incrementally more difficult than those the policy is currently capable of achieving during training. Specifically, online self-curriculum learning consists of three processes: (1) Estimating the current policy's goal-achieving ability, p ag . This can be done through methods such as online evaluation or off-policy evaluation (OPE) [63, 64] . (2) Setting or learning a probability transform function f : P × P → P, followed by sampling progressively challenging behavioral goals based on f (p ag , p dg ) for online learning. (3) Conducting online RL learning with behavioral goals sampled in the second part, facilitating the agent's progression towards more challenging goals. In our implementation of GCPO:Gaussian Mixture Model (GMM) [53] is employed to estimate p ag , as detailed in lines 4-13 of Algorithm 1. During the online self-curriculum learning, the policy is periodically evaluated. In each evaluation, the policy is evaluated with N goals sampled from p dg . Information about the achieved goals, along with an initial weight of 1.0, is stored in a goal buffer B g . As the online self-curriculum learning proceeds, the weight of historically achieved goals is reduced by a factor κ. Ultimately, a GMM is used to estimate p ag based on the data in B g (The specific calculation can be referred to Appendix B.).Maximum Entropy Gain Exploration (MEGA) [47] is utilized as the probability transform function f , as indicated in lines 14-19 of Algorithm 1. The core idea behind MEGA is to encourage exploration in sparsely explored areas of the achieved goal distribution. In discrete settings, inverse probability weighting [32] can be applied to sample goals from p ag . A goal g is chosen with the probability given byf M EGA (p ag , p dg ) (g) = 1 pag(g) p ′ 1 pag(g ′ ).(EQUATIONIn continuous settings, a generate and test strategy [42, 47] is employed for goal sampling. Specifically, M goals {g i } M i=1 are randomly sampled from supp(p ag ), the support set of p ag , and the goal with the minimum density under f M EGA (p ag , p ag ) is selected: g = arg min gi f M EGA (p ag , p ag ) (g i ) .This approach biases the sampling towards goals that are less likely under the current achieved goal distribution, promoting exploration in under-explored regions.KL-regularized RL [67, 66, 7, 27] is employed as the on-policy RL algorithm to optimize the policy, as indicated in line 20 of Algorithm 1. To prevent catastrophic forgetting of latent skills and to continuously improve exploration during the RL fine-tuning phase [7] , the policy π θ is initially set to the pre-trained policy π θ0 and is then fine-tuned by maximizing the following objective:EQUATIONwhere r can represent both Markovian r(s t , a t ) and non-Markovian r(s 0:t , a 0:t ) rewards, and λ ∈ [0, 1] controls the strength of the KL regularization. Optimizing Eq. 3 is analogous to optimizing the original RL objective within the log-barrier of π θ0 , and can be viewed as a trust-region-style [56] learning objective [36] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 4.1 Experimental Setups RL environment. Experiments are conducted on the Fixed-Wing UAV Velocity Vector Control (VVC) task [26] , which is a representative multi-goal problem. The VVC task is characterized by a long horizon, with the average length of demonstrations exceeding 280 steps (detailed in Appendix A.5). Even for well-trained policies, the average number of steps required to achieve a goal is over 100, and more challenging goals can demand upwards of 300 steps to achieve [27] . This exceeds the horizon typically used in most previous studies [41, 47, 28, 59 ]. Additionally, the rewards for VVC can be designed as either MR or NMR, thereby accommodating a range of real-world requirements and complexities. The specifics of the VVC environment setup are detailed in Appendix A. Thus, VVC presents a challenging multi-goal, long-horizon problem that poses significant difficulties for policy learning. Standard SAC+HER [3] and PPO [57] are unable to solve the VVC task, as demonstrated in Appendix A.6, further highlighting the task's complexity. To our knowledge, previous research on NMR algorithms has primarily been tested in simpler environments such as multi-arm bandits [24] and grid worlds [52, 24, 2, 1, 31] . Our work is the first to evaluate the performance of algorithms on complex, real-world NMR problems. Additionally, to demonstrate the broad applicability of GCPO, we conduct experiments on the commonly used RL environments Reach and PointMaze. The corresponding results and analysis can be found in Appendix D.Demonstrations. We collect a demonstration set D E , also denoted as D 0 E , with a PID controller (detailed in Appendix A.5). Subsequently, we employ the IRPO algorithm [27] , which iteratively optimizes policies and demonstrations, to generate 2 presents the quantity and quality of these four demonstration sets. The '#traj' column represents the number of demonstrations contained within each demonstration set, while 'traj length' indicates the average length of demonstrations in the set. A shorter demonstration length suggests a faster completion of the corresponding goal, indicative of higher demonstration quality. It can be observed that the demonstration quantity and quality ofD 1 E , D 2 E , D 3 E . TableD 0 E , D 1 E , D 2 E , D 3 E increase sequentially.We compare GCPO with several baselines on the VVC task under different demonstration conditions, including (1) SAC [29] + HER + MEGA, which is a strong baseline in GCRL; (2) BC, a fundamental yet effective IL algorithm; (3) GCPO without pre-training, which corresponds to PPO + self-curriculum; (4) GCPO without self-curriculum, which corresponds to BC + KL-regularized RL. GCPO itself is equivalent to BC + KL-regularized RL + self-curriculum. Table 2 reports the performance of GCPO and the baselines on NMR, and Fig. 3 visualizes the learning progression on both NMR and MR, as well as the final learned policy of GCPO.GCPO is applicable to both MR and NMR problems. Table 2 shows that GCPO outperforms all baselines on NMR, with SAC+HER+MEGA achieving only 20% of GCPO's performance. This demonstrates the limitations of HER in addressing NMR problems and highlights the superiority of GCPO in these contexts. Furthermore, Fig. 3a illustrates the learning progression of GCPO for both NMR and MR, showing that GCPO is effective in solving both types of problems. In summary, GCPO exhibits versatility and applicability across both MR and NMR problems.Pre-training is crucial for the success of GCPO. As evidenced by the experiments in Table 2 for BC, GCPO w/o pre-training, and GCPO, it is observed that without pre-training, GCPO struggles to learn meaningful skills. However, a policy that is merely pre-trained, albeit with non-optimal performance, plays a crucial role in enabling GCPO to develop an effective policy. Fig. 3b provides a histogram of the achieved goals for the trained policies. It is evident that, even with a pre-trained policy that initially exhibits inferior performance compared to the demonstrator, GCPO's online self-curriculum learning facilitates significant improvement in the policy's performance, surpassing that of the demonstrator.Online self-curriculum facilitates the mastery of challenging goals. Table 2 demonstrates that the application of self-curriculum within GCPO leads to an average 8.2% increase in policy performance compared to its absence. This enhancement is illustrated in Fig. 3c , which shows that the online selfcurriculum mechanism systematically introduces more difficult goals into the learning progression as the policy gains proficiency. This mechanism effectively explains the advantages of self-curriculum for GCPO in mastering challenging goals.In this section, we conduct ablation studies on the demonstration's quantity and goal distribution, analyze the sensitivity of GCPO to the parameters used for estimating p ag , and compare the effectiveness of different self-curriculum methods.To illustrate the influence of the quantity of demonstrations on GCPO, we train GCPO with 10%, 50%, and 100% of D E and present the performance of pre-trained policies and GCPO policies in Fig. 4a . As shown, a policy pre-trained with 10% of D E achieves 81.12% of the performance of the policy pre-trained with 100% of D E , while a policy pre-trained with 50% of D E achieves 96.53% of the performance. Similarly, a policy trained by GCPO with 10% of D E achieves 86.95% of the performance of the policy trained by GCPO with 100% of D E , whereas a policy trained by GCPO with 50% of D E achieves 95.94% of the performance. These results suggest that an increase in the quantity of demonstrations can enhance the performance of GCPO, yet the marginal gains diminish as the quantity of demonstrations grows. Furthermore, these results also indicate that GCPO can still perform well when only a relatively small number of demonstrations are available.To demonstrate the influence of the distribution of goals covered by the demonstrations on GCPO, we collect three demonstration sets with significantly different goal difficulty distributions (detailed in Appendix A.4) and train GCPO with them. Fig. 4b presents the distribution of achieved goals of the pre-trained policies, while Fig. 4c depicts that of the GCPO policy.It is evident that for both pre-trained policies and GCPO policies, their distributions of achieved goals are centered around the distribution of goals covered by the demonstrations. The reason for this is that the self-curriculum, starting with the distribution of achieved goals of the pre-trained policy, which is determined by the distribution of goals covered by the demonstrations, gradually expands the distribution of achieved goals. The preference for goals in the demonstrations thus influences the learning progression of GCPO, leading the policy learned by GCPO to also exhibit a similar preference for goals. This suggests that when preparing demonstrations for GCPO, it is preferable to sample goals and generate demonstrations as closely as possible to the desired goal distribution p dg .Furthermore, in Appendix E.1, we provide a more intuitive case and directly visualize the achieved goals in three-dimensional space, yielding the same conclusions as those from the above analysis. To evaluate the influence of the estimation of p ag on GCPO, we conduct experiments with GCPO across a range of values for N and κ, employing various self-curriculum methods. The results are presented in Table 3 .The internal three-by-three layout of each sub-table reveals that policies with the highest performance tend to be situated at configurations where N is larger and κ is smaller. This trend suggests that under these conditions, the estimation of p ag is more precise. Ideally, as N → inf and κ → 0, the estimation of p ag could be perfectly fitted. when examining N and κ independently, the last row of each sub-table indicates that conducting more evaluations during online learning helps GCPO to obtain a well-performing policy, although this comes at the cost of additional computational resources. Conversely, the sum of each row of each sub-table shows no significant difference, implying that GCPO is less sensitive to the setting of κ.In summary, from the perspective of estimating p ag , to enhance the performance of GCPO, it is primarily advisable to increase the number of evaluations within the tolerable computational resource constraints.To demonstrate the influence of different self-curriculum methods on the learning progression and final policy of GCPO, we train GCPO with three distinct self-curriculum methods: RIG, DISCERN, and MEGA. The learning curve and the final policy performance are illustrated in Fig. 5 .Fig. 5a presents the curve of goal difficulty sampled during learning. It is noted that RIG rapidly samples more challenging goals, followed by DISCERN, and then MEGA. However, when considering the difficulty of the goals sampled at the final stage of learning, DISCERN and MEGA select harder goals than RIG. This observation suggests that RIG, DISCERN, and MEGA exhibit distinctly different learning progressions.Fig. 5b depicts the trend of success rate during learning, and Fig. 5c and 5d present the histograms of achieved goals for the policies trained by self-curriculum and non-curriculum methods, respectively. By combining Figs. 5b and 5c , it is evident that there is no significant difference in performance between different self-curriculum methods, whether in the learning progression or in the final policy.In contrast, when combining Figs. 5b and 5d , it is clear that self-curriculum methods outperform non-curriculum methods in both the learning progression and the final policy performance.In summary, within the GCPO framework, while different self-curriculum methods exhibit distinct learning progressions, there is no discernible difference in the final policy obtained. Moreover, the self-curriculum methods consistently outperform non-curriculum methods, highlighting the effectiveness of the self-curriculum mechanism in promoting goal-conditioned policy learning. 
conclusion and limitations: 
 In this paper, we propose an on-policy goal-conditioned reinforcement learning framework, GCPO, designed to address the limitations of existing methods in solving non-Markovian reward (NMR) problems. Through experimental evaluation on the fixed-wing Velocity Vector Control task, we demonstrate the effectiveness of GCPO in handling both Markovian reward (MR) and NMR problems.Some limitations should be addressed in future work. Firstly, in the implementation of the two components within GCPO, we employ relatively simple methods, such as behavioral cloning and Gaussian mixture model. Whether the use of alternative methods could lead to more efficient learning and better-performing policies is yet to be further validated. Secondly, under the sparse reward setting, the successful training of GCPO relies on the pre-trained policy possessing a certain level of goal-achieving capability. Otherwise, if the policy achieves nothing, it becomes ineffective in establishing a self-curriculum. Lastly, although the GCPO framework does not have a component like HER that is unsuitable for solving NMR problems and thus capable of solving both MR and NMR problems, the specific implementation of GCPO as introduced in Section 1 does not explicitly incorporate components that are specifically designed to handle NMR problems. In Appendix F, we introduce a simple component to address NMR problems within GCPO and observe some effects. However, it is not clear whether integrating the most advanced methods for handling NMR problems within GCPO would lead to a more effective resolution. A PID controller is used to sample trajectories. For convenience, the goal space is discretized with parameters listed in Table 4 . Of the 50715 discretized goals, 10184 trajectories are successfully sampled with an average length of 282.01. These 10184 successful trajectories form the D E . D E are imperfect: firstly, the quantity of demonstrations is limited with only about 20% goals successfully sampled; secondly, the quality of demonstrations is low as a well-trained policy can usually finish the goal within 150 steps. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We detail experimental details in Appendix A and C.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We average all results over 5 random seeds and show error bars suitably on figures.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We show the training epochs of pre-training in Table 6a and compute workers and environment steps of online learning in Table 6b .Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the societal impacts in Appendix G.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our contribution poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: Our work does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: The VVC task represents typical multi-goal long-horizon problems. On the one hand, the average length of individual goals (without further division into sub-goals) in expert demonstrations exceeds 280 steps. Even for policies that have been trained thoroughly, the average steps to achieve a goal is over 100, with more challenging goals requiring more than 300 steps to complete, which is longer than the horizon used in most previous studies [41, 47, 28, 59 ]. On the other hand, when trained with classical RL algorithms, as illustrated in Fig. 6 , neither the off-policy SAC+HER nor the on-policy PPO algorithms are able to solve the VVC task. This also indicates the challenges that the long horizon of VVC poses for GCRL algorithms.In the specific experiments, we only used the easy version, listed by Table 5a , of the VVC task in Fig. 1 of Sec. 1. In all experiments within the Sec. 4, we utilized the hard version, listed by Table 5b , of the VVC task. We employ GMM with weighted samples [21] to estimate p ag . For a set of N weighted samples {(x i , w i )}, where x i is the i th sample and w i is the weight of x i , we want to find a Guassian Model density function withM Guassian components f (x) = M i=1 π i N (x|µ i , Σ i ), where N (x|µ i , Σ i ) = 1 (2π) D/2 |Σi| 1/2 exp{-1 2 (x -µ i ) T Σ -1 i (x -µ i )}is the i th Guassian component, D is the dimension of sample, µ i , Σ i , π i are the mean vector, covariance matrix, and the weight ofN (•|µ i , Σ i ), correspondingly.Then, Expectation-Maximization algorithm is employed to estimate the corresponding parameters. In the Expectation Step, the new estimate of each sample corresponds to each Guassian components is calculated, r ik = π k N (xi|µ k ,Σ k )The Imitation framework is utilized to implement BC algorithm with parameters listed in Table 6a , and the Stable Baselines3 framework for PPO with parameters listed in Table 6b . 128*128 fully connected network and the Tanh activation function are used for VVC.As BC only learns a policy network, we add a warm-up for the value network at the beginning of online learning. When online learning begins, we first freeze the parameter of the policy and train the value network with online samples until it converges, then proceed with the normal RL training. For the parameter λ, 10 -3 is utilized for all the experiments.We conduct two sets of experiments to demonstrate the general applicability of GCPO.Environments: For the first set of experiments, we conduct evaluations on a customized PointMaze environment (PointMaze_Large_DIVERSE_G-v3) from Gymnasium-Robotics [17] within the Mujoco physics engine. The only modification we made to the environment is to expand the number of desired goals from 7 to 45, making our customized version of PointMaze more challenging than the original version. For the second set of experiments, we employ a customized Reach (PandaReach-v3) [23] task on the Franka Emika Panda robot physics engine. The only modification we made to the environment is to change the distance_threshold used to determine goal reaching from 0.05 to 0.01. Reward Settings: The original rewards for both the PointMaze and Reach tasks are Markovian. To evaluate the performance of our algorithm under different NMR settings, we design two distinct types of NMRs. For the PointMaze, the NMR we designed is: the task is considered successful only if, after the point reaches the goal, it moves away by at least a certain distance and then returns to the goal. For the Reach task, the NMR we designed is: the Panda robot must first pass through a specific waypoint before reaching the goal to be considered successful, and each goal has a different waypoint.Both of these settings strictly adhere to the definition of NMR, where the reward is defined by the states and actions over multiple steps.The demonstrations for PointMaze are sourced from Minari [18] (pointmaze-large-v1), while the demonstrations for Reach are generated by us, with reference to the PID controller as described in the official documentation [44] .We evaluate SAC+HER+MEGA, BC, and GCPO on the PointMaze and Reach tasks under both MR and NMR settings. Table 7 presents the success rates of these algorithms. It can be observed that under the MR settings, GCPO exhibits similar performance to SAC+HER+MEGA. However, under the NMR settings, where HER cannot be effective, the performance of GCPO is significantly better than that of SAC. Taking into account the performance of GCPO on the VVC task as illustrated in Section 4.2, we showcase the general applicability of GCPO across a variety of tasks.In this section, we conduct a detailed experimental analysis of the impact of the distribution of goals that Demonstrations can cover, demonstration quantity, and demonstration quality on the training of GCPO.To more intuitively illustrate the impact of the goal distribution in demonstrations on GCPO training, we select three subsets of demonstrations from D E that are spatially distant from each other, comprising all χ = 10 demonstrations for D It is evident that, for both the pre-trained policy and the GCPO policy, the achieved goals are distributed around the goal distribution of the demonstrations. This suggests that the goal distribution in the demonstrations biases the learning of both pre-training and GCPO, leading to policies that share a similar goal distribution as the demonstrations. Therefore, the best practice when preparing demonstrations is to make the distribution of goals covered by the demonstrations as closely resemble the desired goal distribution as possible. We note that commonly used RL demonstration datasets typically consist of 1000 demonstrations [22] . Consequently, we evaluate the performance of GCPO on Reach, PointMaze, and VVC with 1000 demonstrations. In addition, we conduct an ablation study on demonstration quantity on VVC.The results are presented in Table 8 .Demonstrations: The source of demonstrations for PointMaze and Reach is elaborated in Section D. For GCPO, 10%D E is employed, as detailed in Section 4.3.1. Additionally, for comparison purposes, we also present the performance of GCPO on the VVC task using D [χ=90] , which comprises only 144 demonstrations with a flight path azimuth angle of 90, as described in detail in Appendix E.1, and using D E . The results are presented in Table 8 . It can be observed that on the relatively simple PointMaze and Reach tasks, GCPO achieved nearly 100% success rate when using 1000 demonstrations. On the more challenging VVC task, the success rate with 1000 demonstrations reached 81.12% of the success rate achieved with 10264 demonstrations, while also being significantly higher than the success rate achieved with 144 demonstrations. These results indicate that across tasks of varying complexity, GCPO can achieve good performance with the use of 1000 demonstrations.To obtain a demonstration set for comparison with D E , we generate trajectories for these goals using all the policies trained in our experiments. For a specific goal, we retain only the shortest trajectory. These generated trajectories are denoted as D ′ . We use two metrics to measure demonstration quality: Trajectory length I l . Since we employ (-1, 0) sparse rewards, this implies that shorter trajectories yield a higher cumulative reward. Control smoothness I s . In control problems, minimal control gains is expected to reduce wear on actuators. Hence, we refer to [39] to define the control smoothness. Trajectory length and control smoothness each describe certain characteristics of demonstrations from the distinct perspectives of reinforcement learning optimization and optimal control, respectively. Table 9 shows the demonstration quality of these two demonstration sets. It can be observed that: From an RL perspective, D ′ is of higher quality because the trajectories are shorter, leading to a higher expected cumulative reward. From a control perspective, D E is better because the trajectories are smoother.EQUATIONThe performance of the BC policy π BC and the GCPO policy π GCP O trained on these two sets of demonstrations is shown in Table 10 . The results reveal that π BC closely aligns with the demonstrations on both quality metrics, indicating that demonstration quality has a direct impact on BC. Additionally, the BC policy trained on D ′ has a slightly higher success rate, which we speculate is due to D ′ being more suitable for RL (the network architecture and training hyperparameters used to generate D ′ are the same as those for the BC policy). However, after the self-curriculum learning, the GCPO policy corresponding to D E performs better and exhibits a shorter trajectory length. This suggests that the influence of demonstration quality on GCPO's online learning may not be as direct as pre-training, and further research is required to understand this relationship.In summary, on one hand, it is challenging to define demonstration quality suitable for RL through a few metrics. On the other hand, demonstration quality does affect GCPO pre-training. How demonstration quality potentially influences the self-curriculum learning of GCPO remains an intriguing question for further exploration.Furthermore, despite the aforementioned challenges, GCPO is capable of training well-performed policies from non-expert demonstrations. The intrinsic reason is that GCPO employs online learning to fine-tune pre-trained policies. Consequently, even if the demonstrations are non-expert and the pre-trained policies perform poorly, GCPO can still continuously optimize these policies through online learning.In Section 4.2, although the average trajectory length of D 0 E reached 281.83, covering only 20.24% of goal space, the GCPO policy trained on it achieves a success rate of 45.87%, with an average trajectory length of 134.47. This comparison indicates that D 0 E consists of non-expert demonstrations. On the other hand, in contrast to D 3 E , which covers 78.55% of the goal space with an average trajectory length of 116.56, D 0 E is only a quarter in size and has trajectories that are 2.42 times longer, implying a substantial decrease in its quantity and quality. Nonetheless, the GCPO policy trained on D 0 E achieves 76.58% of the success rate of the policy trained on D 3 E .To demonstrate the effects of integrating a method for handling NMR problems into GCPO, we extend GCPO with a basic method for addressing NMR problems, which involves extending the input of the policy with consecutive states to make the reward Markovian [24] . We evaluate the extended GCPO on the VVC task and show the result in Fig. 8 .It is evident that, regardless of whether it is the pre-trained policy or the GCPO policy or the SAC+HER+MEGA policy, expanding the input of the policy leads to a certain degree of performance degradation. We conjecture that this is likely due to the expansion of the policy's search space, which makes the original number of demonstrations and online learning resources relatively insufficient, indirectly increasing the difficulty of solving the policy. Therefore, although expanding the input is theoretically beneficial for addressing NMR problems, in practice, it also requires a balance between the increased difficulty of the problem and the available resources.This paper introduces a general on-policy goal-conditioned reinforcement learning framework, GCPO, which is capable of addressing both Markovian and non-Markovian reward problems. This flexibility makes GCPO particularly well-suited for a wide range of practical applications that rely on non-Markovian rewards, such as robotic arm control with stringent stability requirements and drone control that demands steady goal achievement. However, in practical settings, it is imperative to pay close attention to the evaluation of the GCPO policy, carefully analyzing the disparities between the policy's achieved goal distribution and the desired goal distribution of the actual task. Allowing the policy to attempt goals beyond its capabilities in a production environment could result in hardware failures, damage, and other problems.NeurIPS Paper ChecklistJustification: We confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper presents an on-policy Goal-Conditioned Reinforcement Learning (GCRL) framework (GCPO), which addresses the limitations of existing HER-based methods, in handling non-Markovian rewards (NMR). GCPO combines pre-training from demonstrations with an online self-curriculum learning strategy, allowing it to progressively tackle more challenging goals based on the agent's current capabilities. The authors demonstrate the efficacy of GCPO on a challenging task involving fixed-wing UAV velocity vector control, showcasing its applicability to both Markovian and non-Markovian reward problems. The paper shows the strength in handling both Markovian and non Markovian reward scenarios. The rebuttal helped clarify some concerns such as that on general applicability to diverse tasks and demonstration quality. Given the overall contributions and its strong empirical performance, I recommend the paper for acceptance with a revision to address concerns raised through the reviewer discussions.

Reviews: review: paper_summary: paper_summary: This paper proposes a new on-policy goal conditioned reinforcement learning framework which targets non-markovian rewards, which HER based approaches are unsuccessful. GCPO is a combination of offline pre-training from expert policies using behavior cloning, and online learning from a curriculum which automatically determines complex goals outside of the agent's current goal conditioned policy. They show this frameworks success in the Velocity Vector Control task, and perform several ablation and sensitivity studies.
strength_weakness: strength_weakness: While I think this paper would be an excellent contribution to this year's conference, there are some improvements I would like to see. I think the authors should make changes to address I-1, but the others are less consequential in my view.

- I-1. I haven't seen a discussion around how hyperparameters were tuned. While I put this work in more of a demonstration lane, but I think if the paper was clearer around how these hyperparameters were chosen, and how the baseline methods were tuned would immediately strengthen the paper. I think this is critical when discussing the results of Figure 1. Adding how extensively these methods were tuned could strengthen the evidence supporting the claims.

- I-2. While the ablation showing how the quantity of demonstration data on the final performance is appreciated, I think the lack of ablations on the quality of demonstrations is a major limitation. There are some experiments using demonstrations with different distributions of goals, these don't sufficiently capture what would happen if there is a lack of *expert* data. In many domains, there is often a lack of true expert data, but a large amount of sub-standard data. If GCPO is also performant in instances where expert data is sparse, but non-expert data isn't that would be a major achievement for the method (although my guess is issues would arise from using BC as you already state as a limitation).
questions: questions: - How were the baselines and your method's hyperparameters tuned?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes Goal-Conditioned Policy Optimization (GCPO), an on-policy variant for goal-conditioned RL that can also handle non-markovian reward structures. Common goal-conditioned RL methods are usually related to Hindsight Experience Replay (HER) which however can only solve tasks under the Markovian properties. Notably, GCPO leverages pre-training from demonstrations and online self-curriculum learning that can progressively select challenging goals based on the current learning process of the policy.
strength_weakness: strength_weakness: The paper lacks discussion of relevant works that also consider contextual/goal-conditioned RL with self-curriculum learning with similar motivations, some of which consider non-Markovian rewarded environments. Here are some of those:

- Klink et al. Self-Paced contextual reinforcement learning (CoRL 2019)
- Celik et al. Specializing Versatile Skill Libraries using Local Mixture of Experts (CoRL 2021)
- Klink et al. Self-Paced Deep Reinforcement Learning (NeurIPS 2020)
- Ottot et al. Deep Black-Box Reinforcement Learning with Movement Primitives (CoRL 2022)
- Celik et al. Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts (ICML 2024)

please also see the questions
questions: questions: - The method requires a desired goal distribution which might be task dependent. In which sense is this restricting the algorithm's applicability?
- How exactly is the GMM estimated? No information was given in the main text 
- The work states that the self-curriculum "leads to an average 8.2% increase in policy performance" as evidenced in Table 2. However, Table 2 also shows that GCPO w/o pre-training can only achieve 4% success rate indicating that the performance boost is mainly achieved by the imitation learning policy. On the other hand, Fig.3 shows that GCPO successfully reaches more goals than the expert and BC, especially in the goal difficulty up to 0.7 but is not able to significantly reach more difficult goals. This observation is also discussed in Section 4.3.2.
While I see that the goals that are already covered by the BC can be reached successfully more often by GCPO, the self-curriculum does not seem to cover more difficult goals significantly. Section 4.3.2 suggests to "sample goals and generate demonstrations as closely as possible to the desired goal distribution". Isn't this restricting the method in the sense that the demonstration data needs to be collected accordingly?

- Connecting to the question before, is this indicating that exploring new goals that are much more difficult can not covered well by GCPO? Could this be discussed in more detail? Why is this the case? Is the KL constraint restricting the updates to more difficult goals? To my understanding, expanding to more difficult goals should be covered by the bonus (MEGA) as described in Section 3.3.
limitations: limitations: - While the detailed ablations are highly appreciated, a more thorough evaluation across more tasks is needed to assess the effectiveness of the method. This can be done by using sophisticated existing non-markovian rewarded environments, or, for demonstration purposes, adjusting the reward accordingly in existing benchmark suits such as e.g. (Meta World)

Meta World: https://github.com/Farama-Foundation/Metaworld 

Please also see the weaknesses and questions.
paper_summary: paper_summary: After rebuttal

The authors have mostly addressed my minor concerns. I recommend acceptance.

------


This paper aims to improve goal-conditioned RL. Two problems with prior methods (e.g. HER) are discussed, namely (a) their inability to cope with non-markovian rewards; and (b) the necessity of using an off-policy algorithm. This work then goes on to introduce a method that is applicable to on-policy algorithms *and* NMR problems. This method first uses demonstrations to pre-train a policy to achieve the goals sometimes. Then, it uses a curriculum-based approach to select harder and harder goals (but still within the agent's capability). They demonstrate on a UAV problem with NMR, they outperform other methods, notably SAC + HER.
strength_weakness: strength_weakness: 1. It would be nice to have experiments on another domain, to illustrate the general applicability of your algorithm. 
	1. In particular, having results of the baseline algorithms (and GCPO) in an MR domain would be very helpful.
2. Relatedly, having results for e.g. SAC + HER when doing the same thing as in appendix A.4 would be helpful.
questions: questions: - Could you run GCPO with only a 1000 demonstrations?
limitations: limitations: 1. Requiring demonstrations is a relatively strong limitations, as e.g. HER does not. Is there any way around this requirement?

Comments: comment: Dear Reviewer U5WJ,

We sincerely appreciate the time and effort you spent on our work. We will address the above discussions in the final version. Thank you once again for your valuable feedback.
Sorry, this paper got missed when I was responding to author comments. you have addressed the small concerns I had for this paper. I especially appreciate the extra results on demonstration quality and performance, I think this is particularly important whenever we are using data collected using unknown policies.
Dear Reviewer U5WJ, Do you have any further concerns? Please let us know, we will try our best to address them quickly. We sincerely anticipate your response.
Dear Reviewer aw7R,

We sincerely appreciate the time and effort you spent on our work. Your insightful comments and concerns have helped greatly in improving our paper. We will address these discussions in the final version. Thank you once again for your valuable feedback.
Dear Reviewer emMz,

We express our heartfelt gratitude for the time and dedication you've invested in reviewing our manuscript. Your constructive feedback and perceptive comments have been immensely valuable in refining our paper. We are committed to addressing these insights in our revised work. Thank you once again for your valuable feedback.
Thank you. I will update my score to 7 on the condition that please add these experiments and discussions to the updated manuscript.
I highly appreciate the authors' responses and additional experiments. My questions were clarified in the responses. I am adjusting the score on the condition that the authors discuss the relevant works in the final version and accordingly adjust their claims in lines 66-68.
Dear Reviewers: We are approaching the conclusion of the author-reviewer discussion period, which is set to end in one-or-two days. We kindly request that you review the rebuttal. Should you have any further questions or require clarification on any points, please feel free to reach out. We are committed to addressing your queries promptly. We greatly value your feedback and look forward to your insights.