Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization for Goal-Conditioned Reinforcement Learning
Authors: Xudong Gong
Sen Yang
Feng Dawei
Kele Xu
Bo Ding
Huaimin Wang
Yong Dou
Venue: ICML 2025 poster (2025)
Keywords: Goal-Conditioned Reinforcement Learning
Policy Regularization
Continuity of Goal-Achievement Ability
Primary area: reinforcement_learning->policy_search
TL;DR: We propose a margin-based policy self-regularization approach to improve the continuity of goal-achievement ability for goal-conditioned reinforcement learning.

Abstract: This paper addresses the challenge of discontinuity in goal-achievement capabilities observed in Goal-conditioned Reinforcement Learning (GCRL) algorithms. Through a theoretical analysis, we identify that the reuse of successful trajectories or policies during training can aid in achieving adjacent goals of achievable goals. However, the policy discrepancy between achievable and adjacent goals must be carefully managed to avoid both overly trivial and excessively large differences, which can respectively hinder policy performance. To tackle this issue, we propose a margin-based policy self-regularization approach that optimizes the policy discrepancies between adjacent desired goals to a minimal acceptable threshold. This method can be integrated into popular GCRL algorithms, such as GC-SAC, HER, and GC-PPO. Systematic evaluations across two robotic arm control tasks and a complex fixed-wing aircraft control task demonstrate that our approach significantly improves the continuity of goal-achievement abilities of GCRL algorithms, thereby enhancing their overall performance.

Improving the Continuity of Goal-Achievement Ability via Policy Seff-Regularization for Goal-Conditioned Reinforcement Learning
Xudong Gong" 12
Sen Yang'
Dawei Feng 12 Kele Xu
Bo Ding 12 Huaimin Wang' 12
Yong Dou
Abstract
This paper addresses the challenge of disconti- nuity in goal-achievement capabilities observed in Goal-conditioned Reinforcement Learning (GCRL) algorithms.. Through theoretical analy- sis, we identify that the reuse of successful trajec- tories or policies during training can aid in achiev- ing adjacent goals achievable goals. How- ever, the policy discrepancy between achievable and adjacent goals must be carefully managed to avoid both overly trivial and excessively large differences, which can respectively hinder pol- icy performance. To tackle this issue, we pro- pose a margin- based policy regflaagulii ap- proach that optimizes the policy discrepancies between adjacent desired goals to a minimal ac- ceptable threshold. This method can be inte- grated into popular GCRL algorithms, such GC-SAC, HER, and GC-PPO. Systematic eval- uations across two robotic arm control tasks and complex fixed-wing aircraft control task demonstrate that our approach significantly im- proves the continuity of goae-achiev abil- ities of GCRL algorithms, thereby enhancing their overall performance. Our code is avail- able at https//ggtttbbccmmm/onnnuuuoo/// examples.
0.6
4
(a) GC-PPO (haseline)
(b) MSR-GC-PPO (ours)
Figure 1. Success rate of adjacent goals around an achievable goal on the fixed-wing aircraft control task. In this task, X and j represent dimensions of the goal space. The central point in the figure denotes an achievable goal. To enhance the clarity of the two coordinate axes, introduce the baseline noise parameters Ax and A,.- The horizontal axis, fx- signifies that the distance between an adjacent goal and the achievable goal along the x dimension is Ax. A similar interpretation applies to the vertical axis, Eu- We train 5 policies, each initialized with distinct random seed. For each policy, we conduct evaluations with 100 achievable goals. The results presented herein reflect average performance across the total of 500 achievable goals.
2018). This method demonstrates superiority over conven- tional approaches of training policy for each individual goal, as it facilitates the transfer of knowledge across dif- ferent goals (Andrychowicz et al., 2017). 7). GCRL has been extensively applied in control domains, such as robotic arm manipulation (Pitis et al., 2020) and fxed-wing aireraft con- trol (Gong et al., 2024b; 2025b), well in games like MineCraft (Yuan et al., 2024) and Atari (Warde-Farley et al., 2018; Schaul et al., 2015).
1. Introduction
Goal-conditioned Reinforcement Learning (GCRL) (Liu et al., 2022) is an approach that acquires goal-conditioned behaviors by maximizing cumulative rewards over a desired goal distribution through trial-and-cren (Sutton & Barto,
However, in practice, we observe that GCRL algorithms commonly encounter discontinuity in their capability to achieve goals. For instance, when the error threshold for determining achievement, 8 = 1, an aircraft capable of ex- ecuting a right turn of 30 degrees, yet it fails to accomplish right turn of 30.1 degrees. This is an issue that should not arise, given that the change in the goal, 0.1, is significantly less than d. Consequently, reusing the trajectory for the 30- degree right turn should suffice to achieve the 30. -deeree turn, as demonstrated in Fig. 2 (a) and (b). To provide a more intuitive illustration, we conduct systematic evalua-
Coolege of Computer Science and Technology, National Uni- versity of Defense Technology, Changsha, Hunan, China "State Key Laboratory of Complex & Critical Software Environment, Changsha, Hunan, China. Correspondence to: Dawei Feng <davyfeng @q..con..eoo..oo...... cq..om>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
30
30.1
(a) Research Question: Suppose the aircraft can turn right 30". Then, can turn right 30.1" (error threshold 8 1)?
(b) Observation 1:
(c) Observation 2: It is necessary to change the reused policy in order to obtain higher returns.
Directly reusing the policy for turning right 30 achieve 30.1" and obtain guaranteed return.
(d) Observation 3:
If the policy differs too much, it may lead to failure.
Figure 2. Research question and three corresponding observational findings.
tions on the fixed-wing aircraft control task, and the results are depicted in Fig. 1. It is evident that the success rate for goals in the vicinity of an achievable goal typically falls below 60%. In contrast, our method significantly elevates the success rate well above 60%.
We conduct a theoretical analysis to elucidate the rea- sons for the discontinuity in goal-achievement capabil- ities of GCRL algorithms. Based on the insights from theoretical analysis, we design a margin-basee policy self reffreegularzzation method. This method, serving an auxiliary optimization for policies, can be integrated as plug-in in module into nearly all GCRL algorithms.
To address the aforementioned issue, we conduct a theo- retical analysis which reveals that, during training, reusing successful trajectories can facilitate the policy in learning to achieve adjacent goals around the achieved ones, while also guaranteeing baseline level of cumulative rewards. The difference between policies for achieved goals and their adjacent goals should be small but not too trivial; an exces- sively small discrepancy may hinder the policy from further enhancing cumulative rewards beyond the baseline (Fig. 2 (c)), whereas a large discrepancy may result in the policy's 's failure to obtain the baseline cumulative rewards (Fig. 2
We conduct systematic evaluations on two robotic arm control tasks and a fixed-wing aircraft control task. Results indicate that the integration of MSR into GCRL algorithms substantially improves the continuity of their goal-achievement abilities.
2. Preliminaries
(d)).
GCRL can be described by goal-augmented MDP (Liu et al., 2022) M = (S,
T,,, Φ), where S, A, y, G and Pdg denote the state space, action space, discount factor, goal space and desired goal distribution of the environment, respectively. T S A + P(S) is the transition func- tion, where P(X) denote the probability distribution over a set X. r S x A x g + R is the goal-conditioned reward function. Φb S - G is a tractable mapping func- tion that maps the state to a specific goal. The objective of GCRL is to achieve goals via a goal-conditioned policy क : Sxg + P(A) that maximizes the expectation of the cu- mulative rewards over the desired goal distribution J(r)
Building upon these insights, we develop a Margin-Based Policy Self- Regularguaiiittionn (MSR) approach, which opti- mizes the difference between the polices for a desired goal and its adjacent goals to a threshold that represents the minimal acceptable discrepancy. We incorporate MSR into the widely used GCRL algorithms, including the goal- conditioned off- off-policy algorithms GC-SAC (Pitis et al., 2020) and HER (Andrychowicz al., 2017), as well as the goal-conditioned on-policy algorithm, GC-PPO (Gong et al., 2024a). We conduct experiments on two robotic arm control tasks (Gallouédec et al., 2021) and one com- plex, real-world fixed-wing aircraft control task (Gong et al., 2025a). Experiment results indicate that MSR effectively improves the continuity of gohe-actievement abilities for GCRL algorithms, and furthermore, improves the ability of the policies to obtain cumulative rewards.
(a.. ( s1,0x) E, 2, "st, azs (t.g)]]
Without loss of generality, we assume that: (1) In the tran- sition function, a parameter is incorporated to assess the achievement of a goal. The criterion for determining that a state & has achieved the goal y is given by the condition |Ø(s) - gll < 6, where -ll represents distance metric. If this condition is satisfied, it is said that the state has achieved the goal g. (2) The reward function is bounded by Rma. i.e.,
To summarize:
|r a)) < Rumxs V(s, a) E S x A.
We assess multiple prevalent GCRL algorithms across a range of tasks, substantiating the prevalent challenge of discontinuity in the goal-achievemee capabilities of GCRL algorithms.
To facilitate later analysis, we introduce the discounted sta- tionary state distribution (s) = (1 - 7) E > Pr((s, = 8; 1), and the discounted stationary state-action distribution
2

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Pa (8, a) (1- E. 7Pr(s 8, a, a; T). Intu-
goals. Theorem 3.3, meanwhile, attempts to quantify the cu- mulative rewards that can be obtained by reusing successful trajectories and policies.
itively, discounted stationary state (state-action) distribution measures the overall frequency of visiting state (state- action) (Xu et al., 2020). Their relationship Pa (s.sa) = T (als)dd (s) holds for any policy и. Additionally, we intro- duce an alternative method for computing the cumulative rewards, given by J(T)) = - E(s,a)>p." r(s, a), where I- analogous to the effective planning horizon (Puterman, 2014).
Theorem 3.3. Consider goal- coll-augmented MDP M (S, A,T,Tr, 9, Pddg 4). For any goal g E G and perturbu- tion E G, the following two inequalities hold:
[Tg+et 2
Σ (e
Poy (r.(5,aar(ss..a90
To maintain the clarity of the formulas, we employ the
notation Rg as an abbreviation for g)...g). We denote the cumulative rewards that policy π can obtain with re- spect to a desired goal as J(Tg), which is defined as (mg)) = (o [Ex 2 (1(,, at, at,g)]
(I)
2//2Rmax
DRI. (T)g)(((()))
(1 - -)
1 E (a as)
1-y
[Tigte) S 1-7 (s.a)
Σ Emm((s.a)rss...81
3. Margin-Based Policy Self-Regularization
(2)
2m
3.1. Theoretical Analysis
+ (1 y1- 7)2 V DKL (Tg+c(11))
We commence by presenting Theorem 3.1 and Corollary 3.2, which underpin the insight depicted in Fig. 2(b).
The proof of Theorem 3.3 can be found in Appendix A.4. Theorem 3.3 establishes the upper and lower bounds for [TT+ee) Here, I- E(s.a) Pra (s. a))(s, 9 + <) repre- sents the first part of return determined by the state-action distribution induced by Tg and the reward function defined by g + E, while 2v2RRmu
Theorem 3.1. Consider a goal-augmented MDP M
(8, A, T.r, 7, G. Pdg: 6), where the error threshold for de- termining goal achievement is 5. Suppose we have trajec- tory (8o, do, 81, $1,01,''.. ......., ,8T, satisfying ((ST) = 9.
For an arbitrary perturbation E E G, [[e|| < 6, then T may also be regarded as achieving +
(1 (1->)* v DRL. (Tg+e(-)s)) Tg (-])) denotes
the second part of return that is determined by the discrep-
ancy between Ty and Tgte.
The proof of Theorem 3.1 can be found in Appendix A.2. Theorem 3.1 shows that if a trajectory successfully achieves goal g. then it is also able to serve as trajectory for achiev- ing goals in the vicinity of g. Theorem 3.1 analyzes the properties of achieved goals from the perspective of tra- jectories. The following Corollary 3.2, on the other hand, analyzes the properties of the achieved goal from the per- spective of the policy.
As indicated by Equation 1, when Tg and Tgte identical, [Tgt+e) obtains the maximum lower bound, ((..)) Pmg (s, (s),))(s, a a, + We refer to this value as the guaranteed return for Tg+e- The guaranteed re- turn corresponds to the cumulative rewards obtainable from reusing trajectories or policies as discussed in Theorem 3.1 and Corollary 3.2.
Corollary 3.2. Consider a goal-augmenter MDP M = (S, A, T,T,T, N,,,,9 G, pag: 0), where the error threshold for de- termining goal achievement is 8. For desired goal g', suppose we have policy that can achieve this goal (with the actually achieved goal being denoted as g). For any per- turbation e E G such that [|e]| < 8, the policy is guaranteed to achieve the goal g + E on the condition that it satisfies Exdds DKL. (T(-),, g)), (-1s,, +c) =0
According to Equation 2, to enhance the return of Ty+e beyond the guaranteed return, there must be certain de- gree of discrepancy between Tg+e and Tg (as depicted by Fig. 2(c)). In the following section, we present Theorem 3.4 to elucidate the relationship between the policy discrepancy between I Tg+e and xg and the corresponding return gap.
Theorem 3.4. Consider a goal-augmented MDP M (S, A, T, T. G. Pág. Φ). For policy and any pair of goals gi G and g2 E G. the following inequality holds:
The proof of Corollary 3.2 can be found in Appendix A.3. Corollary 3.2 indicates that if the policy remains unchanged, the same policy can also achieve goals in the vicinity of achieved goals.
- (TTz2)
Σ
Panz (s,0)11(4,,,991) - -rsisann--rrraa.pp 02922l
Theorem 3.1 and Corollary 3.2 demonstrate that the tra- jectories or policies associated with achieved goals can be reused to achieve goals in the vicinity of these achieved
(s.a)
(3)
2V2Rmax 13 Wondra / Voxe(T (-15), T (-1s)
(1 7]
3

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
The proof of Theorem 3.4 can be found in Appendix A.5. Theorem 3.4 demonstrates that the gap in cumulative re- wards between two goals, 91 and 92. under policy is bounded by two components: the first part is the return gap determined by the state-action distribution induced by T and the reward gap determined by I1 and 92; the second part the return gap determined by the discrepancy between T and T9z. In the context of multi-goal problems, com- monly employed reward function is -l]}(s) g||- For this type of reward, we further simplify Equation 3 through Corollary 3.5.
3.2. The Proposed Method
We commence by summarizing the conclusions presented in Section 3.1:
Corollary 3.2 indicates that a policy corresponding to an achieved goal can be reused to achieve goals in the vicinity of this achieved goal.
Theorem 3.3 suggests that reusing policy ensures a guaranteed return, yet higher returns can only be attained through policy modification.
MDP M (S. 7,, Pdg: Q), where the reward function is de- fined as r(s, a, g) = - (l)(() gll. For a policy a and any pair of goals 91 E G and y2 E G. the following inequality holds:
Corollary 3.5. Consider a
Corollary 3.6 demonstrates that when modifying the reused policy, the extent of modification should not be excessive, as this may result in the policy's return being unbounded.
Inspired by the aforementioned three conclusions, we are led to consider the reduction of the discrepancy between Ty and Tyte while simultaneously preventing their convergence to excessive similarity. Consequently, we propose the MSR method, as delineated in Equation 6. The core component of this regularization involves the divergence between g and g quantified by DKL. (Tgg(1s), --g+((--s)] Hence, the primary function of MSR is to mitigate the divergence between Fg and Tg+e- The role of the parameter 3 and the maximization operator is to ensure that the MSR ceases to exert influence when DKL. thereby preventing excessively small discrepancies between Tg and Tg+e-
(T - J (
1
92
2 2//2Rmax E Eandro V d (+|s), ga (-1s)
+ (1 - 7)2
1 1-9 Y
(4)
The proof of Corollary 3.5 can be found in Appendix A.6. It can be observed that the first part of the return gap is reduced to -لا1= a value that is solely determined by the difference between and 92- When applied to our research problem, considering goals and g + c, we further simplify Equation 4 through Corollary 3.6.
1-y
[ (+1s), [-1=(- is less than B,
Corollary 3.6. Consider a goal-augmented MDP M (8. A. T,r,Y,9, Pdg: Φ) where the reward function de- fined as r(s, a. 9) - (l0(s) alو and the error threshold for determining goal achievement is 8. For policy n, a specific goal and any perturbation E G. ||e|] < 6, the following inequality holds:
LMSR(T) max (E endy DKI. [Tg(116), T Tg (1+)(3)))
(6)
Given that MSR serves as a regularization for policies, it can be integrated with any policy-based GCRL method, whether it is off-policy, such as GC-SAC, or on-policy, such as GC-PPO. The specifics of this integration are detailed in Equation 7.
(J(g)) -[(gttc)
1 + 2v 2/2.mmat E Benning V (Tg())))))) (-1ss)
(5)
(1 7)
The proof of Corollary 3.6 can be found in Appendix A.7. It is evident that for goal g and a goal within its 8-ball, + c, the return gap under policy æ is governed by two parts: a constant part determined by d and a variable part de- termined by the discrepancy between Ty and Tg+e- Current GCRL algorithms generally lack constraints on the discrep- ancy between policies for adjacent goals, which leads to the observed discontinuity in the policy's 's capability to achieve goals (as depicted by Fig. 2(d)). Consequently, Corollary 3.6 indicates that to reduce the return gap of a policy on two adjacent goals, it is necessary to diminish the policy discrep- ancy between these two goals.
L(T) = (TT( + ALMss (T),
where (x) represents the optimization objective on æ in reinforcement learning and À is the strength of MSR. Algo- rithm delineates the specific procedure for computing the MSR loss. It is important to note that for each transition, we sample N perturbations, and based on Equation 6, we calculate N MSR values. The average of these N MSR values is then taken as the MSR for that particular transition. Increasing the value of N can make the estimation of Equa- tion more accurate, however, it also leads to an increase in computational expense.

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Algorithm 1 Margin-Based Policy Self- Relf-aegularization
Table 1. Policy discrepancy, Dx (Tgtee Tg). return gap, J (Tg+te) - J(Tg), and expect return, ((r), for different algorithms, Italics indicates that these results should be interpreted with caution duc to the bascline algorithm's performance limitations. The mean and variance are shown over 10 random seeds.
Input: training batch [(5t, ge]] 1. M, policy n, perturbation number for each transition N, the maximum perturbation e', .SR threshold 3, MSR strength X
MSR losses []]
for transition (st, (i) ) in training batch do this trissstrnnitionn.t LMS__MSssssS.S.S =
(a) Reach
for j = to N do
Algorithums
(r((T
((Tge)) ((Ty)
J(T)
sample a perturbation ~ (-)
GC-SAC
0.5
-0.89anss
-12.03a063
tmp_MSR_lon max DDxx(Di[TTT (-1 (t), (-11%
MSR-GC-SAC
0.25
0.00aa00
-0.85a0.0
HER
a.7l
-0.01
-1.1620.10
- 3,
MSR-HER
0.31
-0.01
-0.96x0.0s
GC-PPO
0.01
-5.29~2.1
-24.89-..2
end for MSR wow.sss..ppeen(ttsssssomm irtppppenndtiis...rstttiooon MSR lo..lossss..man)))
MSR-GC-PPO
2.85ga72
-0.01
-1,03
(b) Push
end for return X- MSR losses.mean()
Algorithms
Das (x
(( - J(ag)
((r)
GC-SAC
9.692s81
-7,58230s
20
MSR-GC-SAC
4,27+142
16.48-414
26
HER
5,62e057
-7.05x2.28
-18.00+224
4. Experiments
MSR-HER
3.80aa31
-7.02+1.33
-16,31a192
GC-PPO
0.80aaa2
-8.21
-24.762299
MSR-GC-PPO
0.61
-7.0821.78
-23.0825.06
4.1. Settings
(C) VVC
To evaluate the effectiveness of the proposed method, we conduct experiments three tasks across two physics sim- ulation engines. The tasks included: (1) Reach and (2) Push tasks on the Panda robotic arm (Gallouédec et al., 2021), which are two multi-goal tasks frequently used in academic research. We modify the Reach and Push tasks to increase their difficulty. For Reach, we expand the target range, re- duce the error threshold for determining achievement, and employ the joints control mode which means larger action space; for Push, we expand the target range, employ the joints control mode and sparse rewards. (3) The Velocity Vector Control (VVC) task on a fixed-wing aircraft (Gong et al., 2025a), which is a typical multi-goal long-horizon task. The long-horizon nature of the VVC task implies that the algorithm must overcome more challenging explo- ration difficulties. The specific configurations for the three tasks are detailed in Appendix B. Overall, in terms of task difficulty, the order is Reach < Push < VVC.
Algorithms
(Mgteee mg)
[(ggt) - ((ag)
J(r)
GC-SAC
0.37aa12
-22.00
-138 20
MSR-GC-SAC
0.33a0.09
-20.9427.2
-132.09aa06
HER
0.65
-7.91a322
-69.21a12.87
MSR-HER
0.58
-7.52a3.15
-64.73a13.55
GC-PPO
0,08
-44.50a1209
-669.4423337
MSR-GC-PPO
0. 0.19
-28.8921475
-146.6422u05
cies for adjacent desired goals, DKL (Tyte (Tggte)Tg)) Tg). As ob- served in the second column of all tables, except for the comparison with GC-PPO on the Reach and VVC tasks, MSR consistently lowers DKL (TTte Tg) across all tasks and algorithms. By considering the policy return J(r) and the success rate presented in Appendix D.2 and D.4, it is ev- ident that GC-PPO barely acquires any capability to achieve goals on these two tasks. The failure to learn the distinc- tions between adjacent goals results in a small value of (T Rg)) Conversely, MSR aids the policy in learn- ing the distinctions between adjacent goals, thereby slightly increasing Dxi (T Rg).
We evaluate the following three baseline algorithms: (1)
GC-SAC, representing off-policy GCRL, algorithms; (2) HER, another off-policy GCRL algorithm that incorporates the goal-relabeling technique; and (3) GC-PPO, represent- ing on-policy GCRL algorithms. Furthermore, extend these baselines with MSR: (4) SCC-G (5) MSR- HER; and (6) MSR-GC-PPO. Detailed descriptions of these algorithms can be found in Appendix C.
MSR also effectively diminishes the return gap between policies for adjacent desired goals, [J(T++)) - J(Ty)) The third column of all tables demonstrates that MSR re- duces (J( J (zg)] across all tasks and algorithms. In conjunction with Theorem 3.4 and the aforementioned analysis of D Tg), it can be concluded that MSR can effectively reduce the discrepancy between policies for adjacent desired goals, thereby lowering the return gap. Ad- ditionally, we present the success rate for adjacent desired goals surrounding achievable goals in Appendix D, which exhibits similar trend to [J(g++c) J(g))l
4.2. Main Results
Table presents the performance of our method and the baselines across three tasks.
MSR effectively reduces the discrepancy between poli-
5

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Reeularization
13
10
(a) HER, DRa (Tg Tg)
(b) MSR-HER, DxI. (Tg mg
(c) HER, (Tg+++) J (mg)
(d) IMSR-HER J ( J (
Figure 3. Policy discrepancy and return gap between policies for adjacent desired goals. Results come from experiments Push over 10 ransom seeds. The interpretation the coordinate axes and the data collection methods are analogous to those descrihed Fig.
40 SC-EER
crepancy and return gap between achievable goals and their adjacent goals. can be observed that the color in Fig. 3b is darker than that in Fig. 3a, indicating that MSR-HER has smaller DRL (Ta+e: πyy) value, which suggests that our method effectively reduces the discrepancy between poli- cies for adjacent desired goals. Furthermore, the color in Fig. 3d is lighter than that in Fig. 3c, indicating that the value of (TTg++) - (TT,)) for MSR-HER is closer to zero, demonstrating that our method effectively reduces the return gap between policies for adjacent desired goals. For results on other tasks and algorithms, please refer Appendix D.
25
22
DC HR
OCHER
Dird crement Stope
Erstrrmmnt Stage
HER, DEL. (Tg Tg)
(b) MSR-HER, ((T)
Figure 4. Trends of policy discrepancy between policies for ad- jacent desired goals and cumulative rewards of policy during training. It is important to note that the data presented this figure are derived from the training process. In the computation of (T (.). may represent both achievable and unachiev- able goals. In contrast, the data shown Fig. are obtained from testing conducted after the completion of training. where y is only comprised of achievable goals. Results come from HER and MSR- HER on Push over 10 random seeds.
In addition, Fig. 4 illustrates the trends of DKL ( (T Tg) and JTT) during training. As observed in Fig. 4a, our ap- proach successfully confines the values of DKL (Tg Tg) within a relatively low range, demonstrating its effectiveness in limiting the discrepancy between policies for adjacent desired goals without overly minimizing it. Furthermore, Fig. 4b reveals that the policy trained by our method ex- hibits a faster increase in cumulative rewards, indicating that our approach not only enhances the policy's capability to acquire cumulative rewards but also accelerates the rate at which these rewards are obtained.
Furthermore, MSR enhances the cumulative rewards of the policy, (TT) The fourth column of all tables shows that MSR consistently improves (T). We posit that the enhancement of J(r) by MSR is an indirect result of its reduction in (T(++t)) [Tg)].. Specifically, MSR's en- hancement of the continuity in the policy's ability to achieve goals subsequently boosts the policy's capability to accumu- late rewards across all goals.
4.4. Ablation Studies
In this section, we take the example on the Reach task to analyze the impact of the four hyperparameters in MSR. e', X, 3, N, on the training.
algorithm as an
Additionally, we analyze the impact of MSR on training efficiency. For detailed experiments and analysis, please refer Appendix D.I.
Setting c' 8 facilitates the reduction of discrepancy between policies for adjacent desired goals.. The value of e' determines the definition of adjacent desired goals, making it a parameter tightly coupled with the task. Typi- cally, setting é' = 8 is sufficient. To demonstrate the impact of e' on training, we set X = 3, = 0.0 and to 0818, 0.56, 8, and respectively evaluate the trained policies within the ranges of 0.16, 0.58, 5 around achievable goals. Figure 5 presents the corresponding average DKL (T kg) of achievable goals and their adjacent goals with sampled randomly from the range of evaluation e'. It can be ob- served that regardless of the value of é' used in evaluation,
In summary, the core capability of MSR is to reduce the policy discrepancy between adjacent desired goals, This reduction in policy discrepancy facilitates decrease in the return gap between adjacent desired goals, thereby enhane- ing the overall ability of the policy to accumulate rewards.
4.3. A Micro-level Analysis the Results
To provide a more intuitive demonstration of the effective- ness of our method, we present Fig. 3 the policy dis-
6

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
0.4620
0.3980
0.2498
13
100
92 Entkaattin
0.3
05
00
0.1
55 Tastinngg
(a) B
(b) 0.01
(c) 3 0.1
Figure 5. Policy discrepancy be- tween adjacent desired goals, (iu Ry), under different e' settings. Results derived from MSR-GC-SAC on Reach across 5 random seeds.
Figure 6. Policy discrepancy between adjacent desired goals, (T Tg), under different / settings. The meanings of the coordinate axes and the evaluation methods are consistent with those in Fig.1. Results are derived from MSR-GC-SAC on Reach across 5 random seeds.
3.6
Table 2. Policy discrepancy between adjacent desired goals, DRL (ggte Tg)) return gap, J(my+e) - [(,,), and expected re- turn, [Tr,, under different N settings. The mean and variance are shown over random seeds.
15-
1.0
233
N
DKL. (Tgte: Tg)
[ (mmg)
J(x)
31
A-10
0.23a0.18
0.00aa.01
-0.85.a01
Srarommet Steps
1c6
Erxiorment Steps
100
0.24+0.19
0.00a0.01
-0.83+0.02
(a) DKL (Tgte,
(b) ((x
16
0.25+0.20
0.00aa.01
-0.8440.02
64
0.22+0.17
0.00#0.01
-0.82+0.01
Figure 7. Trends of policy discrepancy between policies for ad- jacent desired goals, (KI(TTTe (g)) and expected return, JTr), during training under different À settings. Results are derived from MSR-GC-SAC on Reach across 5 random seeds.
excessively large À can compromise the optimization of the original RL objectives.
A moderate increase in 3 is beneficial for preventing MSR from over-optimizing the discrepancy between poli- cies for adjacent desired goals. The initial motivation for introducing the 33 parameter into MSR to prevent the over vvrroppimiiztiion of the discrepancy between policies for adjacent desired goals during the policy optimization pro- cess. To validate the role of (3, we conduct evaluations under the conditions e' 0.15. X 0.1, and present the DKL (T (Ty+e: ng) between policies for adjacent desired goals in Fig. 6. It can be observed that as increases, the policy discrepancy also increases, effectively demonstrating 93's role in preventing the over-optimization of the discrepancy between policies for adjacent desired goals. However, it is important note that excessively large 3 can diminish MSR's 's self- reuuaaaulatinttion effects on the policy during training, an overly large / implies that MSR is unable to function.
increasing used in training towards aids in reducing the
discrepancy between policies for adjacent desired goals.
The selection of > necessitates balance between the effects of RL optimization and MSR optimization. À is utilized to balance the relative importance between op- timization objectives of RL and MSR. A larger value of indicates greater proportion the MSR optimization objective in the overall optimization objective. We conduct ablation experiments with 8, 3 0.1 and À set at 0.001, 0.01, 0.1, and 1, and the trends of (T Wg) and J (T) during training
with 3 set to 0.0, 0.01, and
shown in Fig. 7. As observed in Fig. 7a, an increase in À corresponds to decrease in the value of DKL(T DKL(TTte: ) suggesting that larger X enhances the self-regularization effect on the policy. However, Fig. 7b reveals that a larger results reduced capability of the policy to acquire cumulative rewards. This is attributed to the extension of the RL optimization objective by MSR; greater emphasis MSR leads to diminished focus the original RL optimization objective, adversely affecting the policy's 's ability to accumulate rewards. Therefore, A a parameter that requires careful consideration when ap- plying MSR. A too small À may not adequately leverage MSR's s regulaegglaiizatioon effects on the policy, while an
MSR exhibits robustness to the value of N. The role of N to average Equation over multiple noises; as N increases, the estimation of Equation becomes more accu- rate, albeit with corresponding increase in computational cost. To analyze the varying effects of different N values on training, we conduct experiments with N set to 1, 4, 16, and 64, under the conditions e' = 6, A = 0.001, 3 = 0.1.

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Figure 8 illustrates the trend of DEL ( (Tg+e mg)) during training, while Table 2 04
larly, (Korkmaz, 2024) identified overfitting as primary factor limiting the generalization capabilities of policies. Consequenty, research on RL generalization has largely fo- cused on overcoming overfitting. Examples include studies like (Laskin et al., 2020; Yarats et al., 2021), which con- centrate on data regularization through data augmentation methods, and (Igl et al., 2019; Lee et al., 2020; Liu et al., 2021). which focus on policy regularization via neural net- work regularization techniques. In the context of GCRL, (Yang et al., 2023) combined both data and policy regu- larization methods to enhance the generalization ability of offline GCRL approaches. Despite these similarities, our research differs significantly from generalization studies. We discuss the continuity of the policy's 's goal-achievement ability, without emphasizing whether pertains to training or unseen environments. As our experiments demonstrate. for tasks with continuous state spaces, the policy's 's continu- ity of goal-achievement ability is suboptimal even within the training tasks.
presents the evaluation re-
sults after training. can be observed that, both during training and in the final eval- uation, there is negligible differential impact of vary- ing N values on the training process. This indicates that MSR is robust to the choice of N.
02
01
Rols
Rodi
0.
Eraforment Steps
Figure 8. Policy discrepancy between adjacent desired goals, (Tg(T Ty) under different N settings. Results come from MSR-GC-SA on Reach over 5 random seeds.
5. Related Work
Goal-Conditioned Reinforcement Learning. Research in GCRL primarily focuses on two directions. 'hee first direction investigates sampling methods for behavior goals (Liu et al., 2022), which leverage the concept of curriculum learning (Narvekar et al., 2020) to select behavior goals from easy to difficult, thereby accelerating the convergence of the policy. This line of research encompasses two main questions: how to estimate the policy's ability to achieve goals, and how to select appropriate behavior goals based on this ability. In the general off- offlco GCRL framework pro- posed by (Pitis et al., 2020), the policy's goal-achievement ability is estimated using data from the experience replay buffer. Conversely, in the general on-policy GCRL frame- work proposed by (Gong et al., 2024a). the policy's 's ability is assessed periodically during training, and the Off-Police Evaluation (OPE) (Uchara et al., 2022) method is used to fit this ability based on historical evaluation data. Regarding methods for sampling behavior goals, approaches such as RIG (Nair et al., 2018), DISCERN (Warde-Farley et al., 2018), and MEGA (Pitis et al., 2020) sample behavior goals based on variations in the distribution of achieved goals. The second research direction explores sub-goal generation methods, which primarily aimed at addressing the long- horizon challenge. These methods decompose the desired goal into several sub-goals (Park et al., 2024; Chane-Sane et al., 2021), and then use GCRL to sequentially solve these sub-goals, thereby achieving the original desired goal. It evident that the primary research areas in GCRL have not yet taken note of the continuity of the policy's goal- achievement ability. To the best of our knowledge, our work is the first in the field of GCRL to focus on the continuity of the policy's 's goal-achiie acal-eccievement ability. Generalization in Reinforcement Learning. Our work shares similarities with research on generalization in RL (Korkmaz, 2024). For instance, (Lee al., 2020) observed that RL agents frequently overfit to training environments and struggle to generalize to unseen environments. Simi-
Additionally, KL divergence is commonly used as a measure of distance to constrain policies in policy optimization. In the context of multi-goal settings, we compare MSR with two KL-based policy constraint approaches and provide a detailed analysis in Appendix E.
6. Conclusion and Limitations
In this paper, we delve into the challenge of discontinuity in goal-achievement capabilities inherent in Goal-conditioned Reinforcement Learning (GCRL) algorithms. Through comprehensive evaluation of various prevalent GCRL algo- rithms across diverse tasks, we empirically demonstrate the prevalence and significance of this issue. Our theoretical analysis provide deeper insights into the underlying causes of this discontinuity, highlighting the importance of regular- izing policy discrepancies between achieved goals and their adjacent goals.
In response to these findings, we propose a novel margin- based policy seff-regularization (MSR) approach. MSR, designed to optimize the minimal acceptable discrepancy between policies for a desired goal and its adjacent goals, offers a robust solution to the discontinuity problem. By integrating MSR as a plug-in module into existing GCRL algorithms, we show its effectiveness through systematic evaluations two robotic arm control tasks and complex, real- world fixed-wing aircraft control task.
Some limitations should be addressed in future work. Firstly, exploring the scalability of our method to even more complex tasks and environments, as well as investigating its applicability to more GCRL algorithms, remain im- portant directions. Secondly, further theoretical analysis could provide more nuanced understandings of the relation-
8

---

Improving the Continuity of Goal-Achievement Ability via Policy Seff-Regularization
ship between policy differences and cumulative rewards. Thirdly, our approach mitigates the discontinuity in goal- achievement capabilities under expected scenarios, but it does not address the discontinuity in achieeenhieem ca- pabilities in the worst-case scenarios.
Gong, X., Feng, D., Xu, K., Zhai, Y., Yao, C., Wang, W., Ding, B., and Wang, H. Iterative regularized policy opti-
mization with imperfect demonstrations. In International Conference on Machine Learning. PMLR, 2024b.
Gong, X., Feng, D., Xu, K., Wang, W., Sun, Z., Zhou, X., Zheng, S., Ding, B., and Wang, H. Vve-gym: A fixed- wing uav reinforcement learning environment for multi- goal long- mag-horizon problems. International Conference on Learning Representaionss, 2025a.
In conclusion, our research contributes to the advancement of GCRL by providing novel and effective solution to the discontinuity problem, thereby enhancing the perfor- mance and applicability of GCRL algorithms in real-world scenarios.
Gong, X., Feng, D., Xu, K., Zhou, X., Zheng, S., Ding, B., and Wang, H. V-pilot: velocity vector control agent for fixed-wing uavs from imperfect demonstrations. In 2025 IEEE International Conference Roboties and Automation (ICRA). IEEE, 2025b.
Acknowledgements
This work was supported by the Open Fund National Key Laboratory of Parallel and Distributed Computing (PDL) (NO.2024-KJWPDL-O)) and the Science and Technology Innovation Program of Human Province (No.2023RC1005).
Igl. M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C., Devlin, S., and Hofmann, K. Generalization in rein- forcement learning with selective noise injection and in- formation bottleneck. Advances in neural information processing systems, 32, 2019.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Korkmaz, E. A survey analyzing generalization in deep reinforcement learning. arXiv preprint urXiv:2401.02344 2024.
References
Laskin, M., Lee, K., Stooke, A., Pinto, L.,, Abbeel, P., and Srinivas, A. Reinforcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020.
Andrychowiez, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
Lee, K., Lee, K., Shin, J., and Lee, H. Network randomiza- tion: A simple technique for generalization in deep rein- forcement learning. In Eighth International Conference on Learning Representations, ICLR 2020. International Conference on Learning Representations, 2020.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2446 -26644, 2022.
Liu, M., Zhu, M., and Zhang, W. Goal-conditioned rein- forcement learning: Problems and solutions. Interna- tional Joint Conference on Artificial Intelligence, 2022.
Chane-Sane, E., Schmid, C., and Laptev, conditioned reinforcement learning with imagined sub- goals. In International conference machine learning, pp. 1430-1440) PMLR, 2021.
Goal-
Liu, Z., Li, X., Kang, B., and Darrell, T. Regularization matters in policy optimization-an empirical study on con- tinuous control. In International Conference Learning Representations, 2021.
Csiszár, I. and Körner, J. Information theory: coding theo- rems for discrete memoryless systems. Cambridge Uni- versity Press, 2011.
Gallouédec, Q.. Cazin, N., Dellandréa, E., and Chen, L. panda-gym: Open-source goal-conditioned environments for robotic learning. 4th Robot Learning Workshop: Self- Supervised and Lifelong Learning at NeurlPS, 2021.
Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, and Levine, S. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31, 2018.
Gong, X., Feng, D., Xu, K., Ding, B., and Wang, H. Goal- conditioned on- on-policy reinforcement learning. In The eight Annual Conference on Neural Information Processing Systems, 2024a.
Narvekar, S., Peng. B., Leonetti, M., Sinapov, J., Taylor, M. E.,, and Stone, P. Curriculum learning for reinforce- ment learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1 2020.

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Panda-Gym. https:// gymm. gym. readthedocs. haan//ates/uuaeee maaeee/mmaaalla. http//mmaaaal__orlll..o control. html, 2024.
Yuan, H., Mu, Z., Xie, F,, and Lu, Z. Pre-training goal-based
models for sample-efficient reinforcement learning. In International Conference on Learning Representations, 2024.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024.
Pitis, S., Chan, H., Zhao, S., Stadie, B., and Ba, J. Maximum entropy gain exploration for long horizon multi-goal re- inforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. :asel-bbaselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: html.
//jmlr. org/ http:/ppppss//22///// //
Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International Confer- ence on Machine Learning, pp. 1312-1323 1320. PMLR, 2015.
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
Sutton, R. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018.
Uehara, M., Shi, C., and Kallus, N. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.063355 2022.
Warde-Farley, D., Van de Wiele, T., Kulkarni, T,, Ionescu, C., Hansen, S., and Mnih, V. Unsupervised control through non-parametric discriminative rewards. In Inter- national Conference on Learning Representations, 2018.
Xu. T., Li, Z., and Yu, Y. Error bounds of imitating poli- cies and environments. Advances in Neural Information Processing Systems, 33:15737-15749 2020.
Yang, R., Yong, L.., Ma, X., Hu, H., Zhang, C., and Zhang, T. What essential for unseen goal generalization of offline goal-conditioned r!? International Conference on Machine Learning, pp. 39543-3957 PMLR, 2023.
Yarats, D., Kostrikov, I., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2021.
10

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Appendix
A. Theoretical Proofs
A.1. Useful Lemmas
Lemma A.1. (Pinsker's inequality (Csiszár & Körner, 2011)) For two arbitrary distributions and v, Drv(p,
(2DxL
Lemma A.2. (Xu et al., 2020) For any two policies γi and n2, we have that
Drv(pmy Paz) S 1-7 Bandin (Dv (1l)
(8)
A.2. Proof of Theorem 3.1
Proof. Based on that ((ST) and ||e]| < 6, we can derive that ([Ø(s)) (g + +e]l = -l---- (g + e)ll = |e|l < 6.
A.3. Proof of Corollary 3.2
Proof. We give the proof under the conditions of deterministic environment transitions and deterministic policies.
Suppose that the trajectory r achieved by policy ı for goal g' is r = (so. 40. 81, 01 ".. 8r, 4r), where a; = x =s;,,'),, 1 i S T, and it satisfies ((ST) = g with llg' - gll < 6. For the goal g + e, let the corresponding trajectory be denoted as (s)) db. s ... ... s ) where af (s% g + e), <
We proceed prove that r' by induction:
Firstly, since we assume fixed initial state, it follows that Banda DKL (T(s, g'), x(s,++e)) = 0. we can infer that dj = ( + E) [ + x'so.g') = which implies To.1 = Tó:1
%- Furthermore, from the condition
Secondly, assuming To:k Tes we have 8p (s T(s' db-1) S/g, and a =(k. T + ( 9 <) a thus To:4+1 To:k+1
The above proof establishes that πT.99 + <) can yield the same trajectory т as (', By combining this with Theorem 3.1, we conclude the proof.
A.4. Proof of Theorem 3.3 Proof.
J (Tg+e)
(9)
Prote (3,)))(,,))
(10)
(e.0)
Pr, (s, - (..
Σ
Px, (s.))
(11)
1-7 (s,0)
(1,0)
Proto (s, Pr ((..)
Pr, (n,(ss))((ssaagtt))
(12)
1
(8.0)
(e.a)
Baax
Pryte (8, a) Pr, (s)
Σ
Pr, (s.0
(13)
7
(8,0)
(s.a)

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
mmmxx
DTv(Prg-e: Pro) + 1 Σ
(14)
1 1-
(a.a)
MRmax Bandan (DVV (Tatec-10)) [g((1))))
+
1 7
Σ
Pag (r.(3,0))((s)0))))))
(15)
(1 -7)2
(x,a)
2 2/2Raax
(1 -1)) V DKL (1T
Σ
Prz (r,(5,0))(3,a.99t))
(16)
(s:0)
[TT+te
(17)
P (8,))((
(18)
(e,a)
1
1
Σ (r,(,,))(ss...tt))
+ 1-7 (v.a)
(19)
(x,a)
1
(9.00 +
Pa (1.(3,)))((s.a)tt))
(20)
(8,0)
(a,0)
Ryuax
PA, (s,
+ 1-7
1
Σ
Png (n)(8.)))(8..g)))))
(21)
(8.0)
(s.a)
2Pmux
DTv Pry(Ptotc:Pty 1 7
Pr, (m)(0,)))(ssa000))))
(22)
1-7
(s,a)
2Rmax Handrr (Drv
1 Emnpss,))(s...0t)
5 (1 1--1 7)2
+ 1-7 (a.0)
(23)
2/2Rmmax
2 (1 - 7/2 V DKL (T) To (-1)) +
Σ
Pr. (3)))
(24)
1 (s,a)
A.5. Proof of Theorem 3.4
Proof.
(F(gg)
[(ggz)
(25)
1
B(saa)ppan r(s,a,91) - B(ra) r(s, -E(.a)-,,, 92))
(26)
1->
1
Σ
P (s,0))(s,a,9)) Pr,, (3,
(27)
(s,a)
1
Σ
|An, (s,a)r(s, - PF,, (s, (2) + PFoi (s, (s),))(),,,92)) - Page (s,
(28)
1
(s,a)
<
1
Σ (s,a)
Ir (s, 122) |RT, (s, Pappo (s,
(29)
(s,a)
1
(s, + Ramax
<
PM,,
Σ
|[ (8, - Pmoz (s,a)]
(30)
(s,0)
(s,a)
1
Σ
Papy (s, (s, (s, (s)) (s) 1 1-y
2muxx
Drv (Priy a Proz))
(31)
(x,a)
12

---

Improving the Continuity of Goal-Achievement Ability via Policy Seff-Regularization
Table 3. Environment Haper-Parammetes
(a) Reach
(b) Push
(c) VVC
Parameter
Value
Parameter
Value
Parameter
Value
control.tpps
joints
control.type
joints
v_min
150
reward_type
dense
reward_type
sparse
V_max
250
goal_range
0.5
goal xy_range
0.5
mu min
-30
distance threshold
0.01
obj xy range
0.0
mu max
30
max.episode_steps 50
distance_threshigg _threeholdd
0.05
chi_min
-60
episode steps
.tteps
50
chi chi_max
60
max_episode_stepp
400
2V2Rmox
Σ
Pags (1, ge) (1 - (1- Bander V ddx (-1s), Tga (-1=)
(32)
(5,a)
A.6. Proof of Corollary 3.5
Proof.
( - (TTz2l
(33)
2 E Bonday v d (f T (-1s)
$
Σ
Pro:: (s.0))))))))). 92)1 (s,,a)))))
(34)
7
(1 - 7)2
(s.a)
2
Σ (s,a)
Pr., (11())) g1ll a1l+ + (lo(s) g2lll + (1 7) Bondroi ddk((TT (-1s), TTz (+1s)
(35)
Pro. (s,a)ll9 -2ell + 2v2Rmar E Bonden v ddL (-1s), (g2(1))))
<
Σ (s.a)
(36)
(1 7)
||91
92ll
2v2R E Eandan, V dde((Tg (T (-1s), (g2 (-15)
+
(37)
7
(1 7)2
A.7. Proof of Corollary 3.6
Proof.
J(āg) J(TT+)))
(38)
|1g +g)l)ll 2
Bende, V dek (-g()))), (gt((1))))
(39)
1 1-7 (1 7)2
8 2V2RRmux E Eandr, / /ddx(TTT T-+e())s))
(40)
1 1- 7)
B. Environments Details
The configurations for the Reach and Push tasks are presented in Tables 3a and 3b, respectively, with all other parameters adhering to the default settings of Panda-Gym (Panda-Gym, 2024). For the VVC task, the settings are detailed in Table 3c, while the remaining parameters are maintained at the default settings of FlyCraft (Gong et al., 2025a).
13

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Table Algorithm Hyper-Parammetrs Used on Reach and Push
(a) SAC
(b) HER
(c) PPO
Parameter
Value
Parameter
Value
Parameter
Value
net.arch
[256,256]
net.arch
[256,256]
net_arch
[256,256]
gamma
0.95
gamma
0.95
gamma
0.95
train_stepy
106
train steps (Reach/Push)
5 x 10*/1 x 10
train train_steps
107
batch_size
256
batch_sizi
256
batch_size
512
rollout palloes-rrcess.cce puwmeesssemmmm .um
1
prlocessprocess.oo num
n_steps
512
learning rate
3 10-4
learning rate
x 10-4
n.epochs
5
gradient_steps
1
gradient.stepp buffer size
prrooutpprcesssnmm
16
buffer size
2 10
2 x 10a
learning rate
3 x 10
Table Algorithm Parameters Used VVC
(a) SAC
(b) HER
(c) PPO
Parameter
Value
Parameter
Value
Parameter
Value
net arch
[128,128]
net arch
(1281128)
net arch
1288,188]
gamma
0.995
gamma
0.995
gamma
0.95
train_steps
106
train_steps
5 10
train_steps
2 10
batch size
1024
batch size
1024
batch size
4096
rollout_prov process. _um
1
rollout ppoccess _um _um
1
n_steps
512
learning rate gradient_steps
3 10-4
learning rate gradient_steps
3 10-4
a.epochs
5
1
1
cillout_process_uum
64
buffer.size
2 x 105
buffer.sizz
2 10
learning rate rate
3 x 10
Table Comparison of training wall clock time between baselines and MSR+baselines. Results come from PPO (10? environment steps), SAC (10 environment steps), and HER (10 environment steps) on VVC.
GC-PPO MSR-GC-PPO GC-SAC MSR-GC-SAC
HER
MSR-HEL
Wall Clock Time (s)
3498
3573
2621
3175
5096
5912
C. Implementation Details
All our GC-SAC, GC-HER, and GC-PPO are implemented with the Stable-Baselines3 (Raffin et al., 2021) framework. For our MSR method, we employ = 5, = 10-3 3 3 = 0.1, 16 for Reach and Push and c' = 0..1 5, À = 10-3 0.1, 16 for VVC.
D. Addiiional Results
In this section, we provide supplementary presentations of the experimental results discussed in the main text.
D.1. Impact on Training Efficiency
MSR adds training overhead without influencing inference, due to added regularization only in policy optimization. Table 6 shows the wall clock time for training different baselines and Maselines+MSR For on-policy RL, where the main cost is environment sampling, MSR's effect is negligible, with only a 2.14% increase. In off-policy RL, which leverages replay and 16.02% for MSR-HER, Figs. 13, 17,
buffer data, the impact is more pronounced: a 21.14% 14% increase for and 22 reveal MSR's 's convergence rate is on par with baselines. Overall, despite the slight added training overhead, MSR markedly improves policy continuity in achieving goals.
14

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
000
-11
-11
53
-4
-24
(a) GC-SAC, (Tgt: my)
(b)
MSR-GC-SAN (c) GAC, (mg++)) (mg) (d) ASR-GC--AC. ((Tg++) (Tt(
J(Tg))
Figure 9. Policy discrepancy and return gap between policies for adjacent desired goals. Results from experiments Reach with GC-SAC and MSR-GC-SAC. The interpretation of the coordinate axes and the data collection methods are analogous to those described Fig. 1.
9.22
0.01
0.53
5.00
(a) HER, (Tgte Tg)
(b) MSR-HER, DEL (TgtT Tg)
(c) HER, g(g++e) - (TT)) (d) MSRR J(F+te) (
Figure 10. Policy discrepancy and return gap between policies for adjacent desired goals. Results come from experiments Reach with HER and MSR-HER. The interpretation of the coordinate axes and the data collection methods are analogous to those described in Fig. 1.
D.2. On Reach
Figure 9 illustrates the evaluation results of GC-SAC and SSS-GGC-ACC. It is evident that ASR-GC-SAC significantly reduces the policy discrepancy between adjacent desired goals, DxL ( (Totes Ty): while effectively diminishing the return gap between policies for adjacent desired goals, J(Tg+e) - J(Tg). Figure 10 presents the evaluation results of HER and MSR-HER, yielding conclusions similar to those of MSR-GC-SAC. Figure depicts the evaluation results of GC-PPO and MSR-GC-PPO, showing that MSR-GC-PPO slightly inereases J(Tg+e) J(Tg). In conjunction with the success rates presented in Table 7, it is observed that the success rate of GC-PPO is nearly zero, suggesting that GC-PPO lacks the capability to achieve goals. This is reflected in DRL (Tyter Rg), indicating that the policy cannot distinguish between different goals, resulting in particularly small value of DRL. D (T ng)-
Figure 12 shows the trend of policy discrepancy between adjacent desired goals, ( xy), during training, indicating that MSR maintains DKL (T Tg) within relatively small but not excessively small range. Figure 13 demonstrates the trend of cumulative rewards during training, revealing that MSR not only enhances the upper limit of cumulative rewards acquisition but also accelerates the rate of cumulative rewards acquisition.
Table 7 presents the success rates of adjacent goals around achievable goal and the overall success rate. Given that method diminishes the return gap between policies for adjacent desired goals and enhances the overall capability of the policy to acquire cumulative rewards, the policy exhibits a notable improvement in both the success rate of adjacent goals around an achievable goal and the overall success rate.
D.3. On Push
Figure 14 illustrates the evaluation results of GC-SAC and MSR-GC-SAC. It is evident that MSR-GC-SAC significantly reduces the policy discrepancy between adjacent desired goals, DxL. (Te ) while effectively diminishing the return gap between policies for adjacent desired goals, J((m++)) - J(Tg). Figure 15 presents the evaluation results of GC-PPO and MSR-GC-PPO, yielding conclusions similar to those of MSR-GC-SAC.
15

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
0.10
0.88
0.0
0.2
GC-PPO, DxL. (Tggtes Rg)
MSR GC-PPO, (c) GC-PPO, (Fg+.)) (mg) (d) MSR ASR-GC-PPO.. ((Tg+e) (
Dxi. (
Figure 11. Policy discrepancy and return gap between policies for adjacent desired goals. Results from experiments Reach with GC-PPO and MSR-GC-PPO. The interpretation of the coordinate axes and the data collection methods are analogous to those described Fig. 1.
CC-SAC
10
HES
GC-PPD
1.0
MST-GC
5f 0.6
0.4
5.0
2,4
0.6
0.8
1.0
Enviromment Steps
0.0
0.2
0.4
05
0.8
1.0
(a) GC-SAC, MSR-GC-SAC GC-SAC
(b) HER and HER
(c) GC-PPO and MSR-GC GC-PPO
Figure Trends policy discrepancy between policies adjacent desired goals during training on Reach. Results come algorithms over 5 random seeds.
Cumulattre Neward
-20
cmmubttrreeee Roward
-10
-35
-20
-20
300
-00
40
-30
-40
-50
Ensfronment Steps
106
Envtronment Steps
Envfronment Steps
(a) GC-SAC, MSR-GC-SAC
(b) HER GC-HER
(c) GC-PPO and MSR-GC-PPO
Figure 13. Cumulative rewards of policy during training on Reach. Results come from algorithms over random seeds.
Table Success rate adjacent goals around an achievable goal and overall success rate policy for different algorithms on Reach. The mean and variance are shown over 5 random seeds. Italics indicates that these results should be interpreted with caution due to the baseline algorithm's 's performance limitations.
Algorithms
Adjacent Success Rate (%)
Success Rate (%
GC-SAC MSR-GC-SAC
65.58+9.10
0,20+0.99
99.98+0.44
99.95=0.12
HER MSR-HER
98.982942
94.60s4443
99.3440.27
90.69=6.85
GC-PPO
44.63+13.8
0.04+0.09
MSR-GC-PPO
98.79+0.36
98.05=3.19
Figure 16 shows the trend of policy discrepancy between adjacent desired goals, DxL DxI(Tgte, Tg), during training. Figure 17
16

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
20
10 0.0
10
(a) GC-SAC, DxL. (KI.(ggtte (g)
(b)
MSR-GC-SAC. (c) GC-S J(T (Tg) (d) MSR-GC-SAC ((Tgee) )(m,)
(ga(T Tg)
Figure 14. Policy discrepancy and return gap between policies for adjacent desired goals. Results from experiments Push with
GC-SAC and MSR-GC-SAC. The interpretation of the coordinate axes and the data collection methods are analogous to those described Fig. 1.
-10
(a) GC-PPO, ( (Tytes [y)
(b) DRL. (Tg Rg)
MSR-GCI GC-PPO, (c) GC-PPO, ( (T₂)) (m,
(d)
J(Tgts)
Figure Policy discrepancy return gap between policies for adjacent desired goals. Results come from experiments on Push with GC-PPO and MSR-GC-PPO. The interpretation of the coordinate axes and the data collection methods are analogous to those described Fig. 1.
GC-SAC
40
GCMEA
GC MSP-GC-F90
MSB-SC-AAC
125
10.0
7.5
10
0.0
0.0
0.2
0.4
1-2
0.2
0.2
3.5
1.0
0.0
0.2
5.6
0.8
1.2
(a) GC SAC, SCCSC
(b) HER and GC- HER
(c) GC-PPO and MSR-GC ASR-GC-PPO
Figure Trends policy discrepancy between policies adjacent desired goals during training on Push. Results come from algorithms over 5 random seeds.
demonstrates the trend of cumulative rewards during training. Table 8 presents the success rates of adjacent goals around an achievable goal and the overall success rate. All of the above results demonstrated conclusions similar those observed on the Push task.
D.4. On VVC
Figure 18 illustrates the evaluation results of GC-SAC and MSR-GC-SAC. Figure 19 illustrates the evaluation results of HER and MSR-HER. Figure 20 illustrates the evaluation results of GC-PPO and MSR-GC-PPO.
Figure 21 shows the trend of policy discrepancy between adjacent desired goals, (T 7,), during training. Figure 22 demonstrates the trend of cumulative rewards during training. Table 9 presents the success rates of adjacent goals around an achievable goal and the overall success rate. All of the above results demonstrated conclusions similar those observed on
17

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
-15
Cammilaooe Nowaard
Camuuaaio Dewved
-20
-4
-30
Enwinement Steps
1e5
Enstronment Steps
Enviromment Steps
303
(a) GC-SAC, MSR-GC-SAC
(b) HER and GC-HER
(c) GC-PPO and MSR-GC-PPO
Figure 17. Cumulative rewards of policy during training Push. Results come from algorithms over 5 random seeds.
Table Success rate adjacent goals around an achievable goal and overall success rate policy for different algorithms on Push. The mean and variance are shown over 5 random seeds.
Algorithms
Adjacent Success Rate (%)
Success Rate (%)
GC-SAC
81.26x3.28
58.37219.37
MSR-GC-SAC HER MSR-HER HER
84.37+2,34
62.1 17.1.236.23
82.27+1.34
79.49=5.34
82.29x1.51
84.27=3.92
GC-PPO
77.07a3.36
66.9026.84
MSR-GC-PPR
80. 173.79
68.82x 18.882
05
-30
(a) GC-SAC. DxI. (Tgtte
(b) DRI. (Tgtes Rg))
MSR-GC-SAC, (c) GC-SAC, ((T++)) (TT)) (d) MSR-GC-SAC, ((T+te) (
Figure 18. Policy discrepancy and return gap between policies for adjacent desired goals. Results come from experiments on VVC with GC-SAC and MSR-GC-SAC. The interpretation of the coordinate and the data collection methods analogous to those described in Fig. 1.
20
10
10
20
20
-10
-30
(a) HER, (Ia(Totee m.)
(b) MSR-HSEREE DEL (Togee m
(c) HER, ( J (mgg)
(d) MSR-HER, ( J (
Figure 19. Policy discrepancy and return gap between policies for adjacent desired goals. Results come from experiments on VVC with HER MSR-HER. The interpretation the coordinate axes the data collection methods are analogous to those described Fig. 1.
the Reach task.
18

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
34
-0
-10
20
2
31
40
(a) GC-PPO, (Tg
(b) Drt. (Tg mg)
MSR-GC-PPOO (c) GC-PPO, (T (Fg) (d) MSR-GC-PPO ((Tg+e) (
Figure 20. Policy discrepancy and return gap between policies for adjacent desired goals. Results from experiments VVC with GC-PPO and MSR-GC-PPO. The interpretation of the coordinate axes and the data collection methods are analogous to those described Fig. 1.
0.0200
GC-SAC
VS3-0CHP9
0.0250
0.0325
15
030000
0.6
0.0222
0.2
0. 0.0000
Enstronmment
Environemntt
Enviiomment Stapss
(a) GC SAC, WSR-MSSRSSS... GC-SAC
(b) HER and GC-HEE
(c) GC-PPO and MSR GC-PPO
Figure 21. Trends of policy discrepancy between policies for adjacent desired goals during training VVC. Results come from algorithms over 5 random seeds. Note that the policy discrepancy is measured by noise ~ [-0.1 8, 0.1 6].
-120
-140
-00
-140
Cumulottve Revord
Cunuiiiive Roward
2000
-150
-1.00
-150
-200
220
-180
200
CCSAC
-200
-260
GC-FFO
VSh SCC
-220
HER-1ER
-200
M5h 188--9--99
0.4
5.6
0.5
1.0
2.5
2.0
Environment Steps
Environment Steps
Enstronment Steps
(a) GC-SAC, MSR-GC-SAC
(b) HER and GC-HER
(c) GC-PPO and MSR-GC-PPO
Figure 22, Cumulative rewards of policy during training on VVC. Results come from algorithms over random seeds.
Table 9, Success rate of adjacent goals around an achievable goal and overall success rate of policy for different algorithms on VVC. The
mean variance are shown over random seeds.
Algorithms
Adjacent Success Rate (%)
Success Rate (%)
GC-SAC
71.7228.7
20.8627.75
MSR-GC-SAC
74.32+7.46
22.2826.84
HER
89.6943.03
62.84412.12
MSR-HER
90.97+3.59
67.94+13.08
GC-PPO
54.22+8.02
7.62+8.06
MSR-GC-PPO
69. 1
26.5243364
19

---

Improving the Continuity of Goal-Achievement Ability via Policy Self-Regularization
Table 10. Comparison of TRPO, Offline-to-Online and MSR.
Algorithm
Optimization Objective
TRPO
Eg-vas K _ (+119)] (11,g)] Egrpnn KL (- g) To. (11.9) (- K To, (11 (1 +e]
Offline-to-Online
MSR
E. Comparison of MSR with Other KL-Based Policy Optimization Methods
Table 10 delineates the optimization objectives of TRPO (Schulman al., 2015), Offlne-to-Online (Baker al., 2022: Gong al., 2024b), and MSR under multi-goal settings. It can be observed that:
TRPO aims to prevent the policy from changing too much during optimization, thus it achieves this objective by constraining the KI. divergence between policies in consecutive training iterations.
The Offline-to-Onlinn method aims prevent the policy from forgetting the knowledge learned offline during online learning, by constraining the KL divergence between the offline-leared policy, To. and the policy currently being learned online, Te.-
MSR, the other hand, seeks enhance the policy's 's continuity of goal-achievement ability by limiting the divergence between policies corresponding to adjacent goals.
In summary, although all these methods utilize KL divergence, from the perspective of optimization objectives, MSR is fundamentally distinct from other KL-based methods.
20


Reviews:
[{"review_id": "HZmLusncVq", "reviewer": "Reviewer_MjdV", "summary": "This paper presents a regularization technique to improve the capabilities of Goal-Conditioned Reinforcement Learning (GCRL) algorithms. The authors start by motivating the need for their approach and presented prelimaries about GCRL. Next, the authors present a cohesive theoretical analysis displaying that modifying a policy toward a goal variant may achieve higher returns, and that the extent of the modification should not be excessive. Following, the authors present the main regularization technique: Margin-Based Policy Self-Regularization. The experiments in Section 4 are conducted across three different settings, two in a robotic control domain and one in aircraft control. Key takeaways include that MSR regularization 1) can enhance cumulative rewards for the policy while reducing the discrepancy between policies, and 2) The importance weight for the MSR objective must be chosen carefully.\n\n\n\n#### Post-Rebuttal\nThe author's rebuttal includes an example of an improved caption, which will benefit the paper, and a statistical analysis of the results, which will improve the validity of the experimental findings. Both of these improvements should be included in the final manuscript.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and valuable feedback on our work. We address your concerns in the following: \n\n`W1: Many of the figure captions could be improved.`\n\nWe greatly appreciate your suggestions. In accordance with the principles of accuracy and conciseness, we have re-formulated the titles of all figures and tables. Taking Figure 6 as an example, which illustrates the policy discrepancy between adjacent desired goals, $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$, under various $\\beta$ settings, supporting our ablation analysis on $\\beta$, we have revised its title from:\n\n*\"Ablation study on $\\beta$. $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$ between policies for adjacent desired goals. The interpretation of the coordinate axes and the data collection methods are analogous to those described in Fig.1. Results come from MSR-GC-SAC on Reach over 5 random seeds.\"*\n\nto:\n\n*\"Policy discrepancy between adjacent desired goals, $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$, under different $\\beta$ settings. The meanings of the coordinate axes and the evaluation methods are consistent with those in Fig.1. Results are derived from MSR-GC-SAC on Reach across 5 random seeds.\"*\n\n`W2: It would be beneficial to verify that the claims made in Section 4 (that are based on Table 1) are valid with respect to statistical significance.`\n\nWe appreciate your suggestion. Taking $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$ on VVC as an example, we have conducted a statistical significance analysis between HER and MSR-HER. The table below presents the T-test's $\\alpha$ values for the evaluation results of policies trained with 5 different random seeds. All the $\\alpha$ values are below 0.05, indicating that MSR-HER can train policies with a smaller $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$ compared to HER.\n\n||MSR-HER(1)|MSR-HER(2)|MSR-HER(3)|MSR-HER(4)|MSR-HER(5)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|HER(1)|$2.4*10^{-5}$|$1.2*10^{-4}$|$3.3*10^{-11}$|$5.2*10^{-33}$|$8.5*10^{-13}$|\n|HER(2)||$7.1*10^{-16}$|$2.6*10^{-14}$|$5.4*10^{-4}$|$2.6*10^{-8}$|\n|HER(3)|||$2.9*10^{-4}$|$4.0*10^{-2}$|$2.3*10^{-4}$|\n|HER(4)||||$1.5*10^{-5}$|$2.5*10^{-6}$|\n|HER(5)|||||$7.6*10^{-7}$|\n\n_**Note**: We trained multiple HER and MSR-HER policies with distinct random seeds (numbers in parentheses indicate the seed index), each evaluated over 12100 episodes. As the T-test necessitates i.i.d. data for both groups being compared, and the test outcomes from different seeds do not belong to the same distribution, we can only conduct pairwise comparisons between the policies trained by HER and MSR-HER with varying random seeds._\n\nAdditionally, we noted the large variance in the results of Table 1. We have increased the number of random seeds from 5 to 10 and re-conducted the training and testing, updating Table 1. The updated table for the VVC shows a reduction in variance for the results.\n\n|Algorithms|$D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$|$J(\\pi_{g+\\epsilon}) - J(\\pi_g)$|$J(\\pi)$|\n|:-:|:-:|:-:|:-:|\n|GC-SAC|0.37±0.12|-22.00±5.06|-138.20±14.16|\n|MSR-GC-SAC|0.33±0.09|-20.94±7.62|-132.09±8.06|\n|HER|0.65±0.22|-7.91±3.22|-69.21±12.87|\n|MSR-HER|0.58±0.16|-7.52±3.15|-64.73±13.55|\n|GC-PPO|0.08±0.08|-44.50±12.09|-169.04±25.57|\n|MSR-GC-PPO|0.17±0.21|-30.65±20.19|-146.64±28.05|\n\nOnce again, we appreciate your suggestion and will include the discussion on statistical significance in our manuscript.", "claims_and_evidence": "Yes, the claims are supported by sufficient evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation make sense for the problem at hand.", "theoretical_claims": "Section 3 was read carefully. Proofs in the appendix were not verified in-depth.", "experimental_designs_or_analyses": "The experimental design within the main paper was read carefully.", "relation_to_broader_scientific_literature": "This paper presents an important theoretical analysis that can benefit GCRL algorithms and inspire future research in this area.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n+ The proposed approach is backed by theoretical analysis.\n+ There is an abundance of results supporting the method. The authors do a good job of displaying the policy discrepancy and cumulative rewards side-by-side.\n\n\nWeaknesses:\n- Many of the figure captions could be improved. \n- It would be beneficial to verify that the claims made in Section 4 (that are based on Table 1) are valid with respect to statistical significance.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please respond to my weaknesses noted above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sV8lOY18II", "reviewer": "Reviewer_xiNi", "summary": "In their present paper, the authors address an evident issue appearing in goal conditioned RL: discontinuity between control policies even in cases of adjacent goals, i.e. where their respective goals are only marginally separated by some distance \\eps. The insights generated by a comprehensive analysis of the mentioned effect, which takes into account the KL divergence as a central measure to define policy discrepancy, are consequently turned into a new policy regularization method, MSR (Margin-Based Policy Self-Regularization), which can be attached to common off- and on-policy RL algorithms, such as SAC, HER and PPO, in order to improve not only the continuity properties between adjacent goal policies but also the overall policy performance (as measured by the total return value over all achievable goals). Based on three individual benchmark environments, the authors provide sound experimental support for their analytical claims and demonstrate the practical applicability of their method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and careful review of our paper. We address your concerns in the following:\n\n`My only remark is that a higher number of random seed varying trials would improve their statistical evaluation of the experimental results, as the current number (5 trials) yields relatively large statistical error bars on the individual KPIs of Table 1.`\n\nThank you for your suggestion. We have increased the number of random seeds from 5 to 10, re-conducted the training and testing, and updated Table 1. The table below presents the results on VVC, showing that the variance of the obtained results has decreased to some extent.\n\n|Algorithms|$D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$|$J(\\pi_{g+\\epsilon}) - J(\\pi_g)$|$J(\\pi)$|\n|:-:|:-:|:-:|:-:|\n|GC-SAC|0.37±0.12|-22.00±5.06|-138.20±14.16|\n|MSR-GC-SAC|0.33±0.09|-20.94±7.62|-132.09±8.06|\n|HER|0.65±0.22|-7.91±3.22|-69.21±12.87|\n|MSR-HER|0.58±0.16|-7.52±3.15|-64.73±13.55|\n|GC-PPO|0.08±0.08|-44.50±12.09|-169.04±25.57|\n|MSR-GC-PPO|0.17±0.21|-30.65±20.19|-146.64±28.05|\n\nAdditionally, we have expanded our discussion to include the statistical significance of the results. Taking $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$ on VVC as an example, the table below presents the T-test's $\\alpha$ values for the evaluation results of policies trained with 5 different random seeds between HER and MSR-HER. All the $\\alpha$ values are below 0.05, indicating that MSR-HER can train policies with a smaller $D_\\text{KL}(\\pi_{g+\\epsilon}, \\pi_g)$ compared to HER.\n\n||MSR-HER(1)|MSR-HER(2)|MSR-HER(3)|MSR-HER(4)|MSR-HER(5)|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|HER(1)|$2.4*10^{-5}$|$1.2*10^{-4}$|$3.3*10^{-11}$|$5.2*10^{-33}$|$8.5*10^{-13}$|\n|HER(2)||$7.1*10^{-16}$|$2.6*10^{-14}$|$5.4*10^{-4}$|$2.6*10^{-8}$|\n|HER(3)|||$2.9*10^{-4}$|$4.0*10^{-2}$|$2.3*10^{-4}$|\n|HER(4)||||$1.5*10^{-5}$|$2.5*10^{-6}$|\n|HER(5)|||||$7.6*10^{-7}$|\n\n_**Note**: We trained multiple HER and MSR-HER policies with distinct random seeds (numbers in parentheses indicate the seed index), each evaluated over 12100 episodes. As the T-test necessitates i.i.d. data for both groups being compared, and the test outcomes from different seeds do not belong to the same distribution, we can only conduct pairwise comparisons between the policies trained by HER and MSR-HER with varying random seeds._\n\nWe are continuing to increase the number of random seeds to obtain more reliable experimental results. Once again, we thank you for your valuable suggestion.\n\n`However, the authors must improve the readability of axis labels and (color) legends used in their figure, as they are currently far too small or sometimes missing/unreadable at all. This is a mandatory issue for any final version provided if accepted for publication. Some of the plots in the main text are borderline small, so I strongly recommend finding a solution to make central results more visible.`\n\nWe sincerely thank the reviewer for highlighting the issue regarding the readability of our figure axis labels and legends. To enhance the readability of all figures, we have taken the following steps:\n\n1. We have increased the font size of all axis labels and legends to ensure they are easily readable.\n2. For figures utilizing color legends, we have not only enlarged the text but also ensured that the color distinctions are more pronounced and clearly labeled.\n\n` If indeed GC-PPO fails entirely at solving the general RL tasks in the mentioned benchmarks, it is questionable to consider any of those results to draw conclusions about the (dis-)continuity properties and total return accumulation (as partly admitted by the authors). Maybe those results should be marked or put into brackets in Table 1, to prevent false conclusions.`\n\nAs suggested, we have marked the results of GC-PPO in Table 1 with italics to indicate that these results should be interpreted with caution due to the baseline algorithm's performance limitations. Additionally, we have included a description in the title to explain this notation. Thank you once again for your valuable insights.", "claims_and_evidence": "All claims in the present work are supported by both theoretical arguments, mathematical proofs and experimental evidence. All terms and pre-requisite are sufficiently introduced and explained, as there also is a good balance of showing essential parts of their analysis in the main text and providing further details in the appendix. The authors present a thorough experimental analysis of their central hypotheses. My only remark is that a higher number of random seed varying trials would improve their statistical evaluation of the experimental results, as the current number (5 trials) yields relatively large statistical error bars on the individual KPIs of Table 1.", "methods_and_evaluation_criteria": "I don’t find any major flaw or concern in the continuity analysis for adjacent goal conditioned policies, as provided in Section 3. The proposed method, MSR, is reasonably introduced as a direct consequence of the analytical results and, as the authors claim, can be used to augment essentially any GCRL policy algorithm as an addition regularization term added to the reward definition. It authors thereby introduce a reasonable and relatively low number of (hyper-)parameters to enable fine-tuning of their regularization term. A sufficiently detailed ablation study for all mentioned hyper-parameters is provided as part of the experimental evaluation. \n\nThe evaluation criteria chosen (essentially three metrics shown in columns of Table 1) are reasonably chosen to highlight and support the claims and conclusions of the authors. As mentioned in the previous section, it appears that all results reported in Table 1 still come with a comparably large statistical error. In a few cases, the error become large enough to weaken the substance of conclusion taken by the authors, as they may also be result of random fluctuations. Reducing this error by increasing the number of random seed trials would help to add significance to the results and should be considered in the final version of the paper.", "theoretical_claims": "I checked soundness and plausibility of all equations as part of the main text, and some (but not all) of the more detailed derivations provided in the appendix.", "experimental_designs_or_analyses": "To my opinion, the authors have chosen a reasonable selection of benchmark comparisons, including representatives of both off-policy and on-policy GCRL SOTA algorithms. The simplicity of extending each candidate with the proposed regularization term helps to have a clear structured comparison between methods with and without the MSR extension. The visual illustrations are generally well-chosen and supportive to understand the most relevant effect.\n\nHowever, the authors must improve the readability of axis labels and (color) legends used in their figure, as they are currently far too small or sometimes missing/unreadable at all. This is a mandatory issue for any final version provided if accepted for publication. Some of the plots in the main text are borderline small, so I strongly recommend finding a solution to make central results more visible.", "relation_to_broader_scientific_literature": "The authors were generally short on introducing related works in Section 5, however, do provide a sound embedding of their work into the context of GCRL. They underline that, according to the best of their knowledge, the paper presents the first to focus on the continuity issue in context of GCRL. I personally cannot judge whether this is true but I also have no counter-example to object their claim.", "essential_references_not_discussed": "The authors mainly discuss the relation to GCRL as applied to model-free policy-gradient and/or Q-value („critic“ function) related research. What is somehow missing is the relation to model-based RL methods, which are also strongly used in connection with multi-objective and goal-conditioned RL tasks, such as\n\n- Marc P. Deisenroth and Dieter Fox. Multiple-target reinforcement learning with a single policy. ICML 2011 Workshop on Planning and Acting with Uncertain Models, 2011\n- M. Weber, P. Swazinna, D. Hein, S. Udluft and V. Sterzing, \"Learning Control Policies for Variable Objectives from Offline Data,\" 2023 IEEE Symposium Series on Computational Intelligence (SSCI)", "other_strengths_and_weaknesses": "Strengths:\n———— \nThe authors find a clear language and easy-to-follow structure to motivate the underlying issue with discontinuity in relation to adjacent goal conditioned control policies. The paper appears well written and succeeds in providing a clear relation of claims and evidence.\n\nComments/Remarks:\n————————\n\nl. 255 (right column): If indeed GC-PPO fails entirely at solving the general RL tasks in the mentioned benchmarks, it is questionable to consider any of those results to draw conclusions about the (dis-)continuity properties and total return accumulation (as partly admitted by the authors). Maybe those results should be marked or put into brackets in Table 1, to prevent false conclusions.", "other_comments_or_suggestions": "I repeat from previous section:\nThe authors must improve the readability of axis labels and (color) legends used in their figure, as they are currently far too small or sometimes missing/unreadable at all. This is a mandatory issue for any final version provided if accepted for publication. Some of the plots in the main text are borderline small, so I strongly recommend finding a solution to make central results more visible.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1w7wxLcK1i", "reviewer": "Reviewer_Lfqn", "summary": "Reaching adjacent goals utilizing the same policy is non-trivial due to the limited robustness of policy improvement. The paper studies discontinuity in goal-achievement observed in Goal-Conditioned Reinforcement Learning (GCRL). Theoretically, the paper identifies constraints between goal reaching policies of the current goal and adjacent goal in order strike a balance between similarity and policy diversity which can hinder policy performance. This leads to the creation of Margin-Based Policy Self-Regularization (MSR) which regularizes a given RL policy with the KL constraint between the current and adjacent goal reaching policies. MSR can be combined with any RL policy and on the evaluated benchmarks, the method shows performant goal reaching ability while minimizing the KL constraint.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`W1: Theory To Practice`\n\nIn our work, we aim to address the issue of ensuring that if a policy can achieve a goal $g$, it should also be capable of achieving goals in the vicinity of $g$, which we denote as $g+\\epsilon$. From the perspective of cumulative rewards, our objective is to minimize $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}|J(\\pi(\\cdot|\\cdot,g))-J(\\pi(\\cdot|\\cdot,g+\\epsilon))|$.\n\nOur method is informed by the following insights:\n\n1. We show analytically that $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')} |J(\\pi(\\cdot|\\cdot,g))-J(\\pi(\\cdot|\\cdot,g+\\epsilon))|$ can be bounded by $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$, as concluded in Theorem 3.4, Corollary 3.5 and 3.6. This led us to consider constraining $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$ during policy optimization.\n2. Theorem 3.3 suggests that the difference between $\\pi(\\cdot|\\cdot,g)$ and $\\pi(\\cdot|\\cdot,g+\\epsilon)$ should not be entirely eliminated, implying that $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$ should not be optimized to be too small.\n\nBased on these findings, we develop the margin-based policy self-regularization method, as outlined in Eq.(6). The $KL$ term in the equation corresponds to the first point, and the max operation corresponds to the second point.\n\n`W2: Empirical Evaluation`\n\nWe address your concerns from the following three aspects:\n\n**Firstly**, we believe that our experimental design aligns well with our theoretical derivations.\n\nIn conjunction with our response to W1, our method aims to reduce $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}|J(\\pi(\\cdot|\\cdot,g))-J(\\pi(\\cdot|\\cdot,g+\\epsilon))|$ by constraining $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$, thereby achieving the objective of enabling the policy to achieve not only $g$ but also goals in the vicinity of $g$. When designing our experiments, we sought to answer the following questions:\n\n1. Can MSR reduce $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$?\n2. whether a reduction in $KL[\\pi(\\cdot|\\cdot,g)||\\pi(\\cdot|\\cdot,g+\\epsilon)]$ leads to a decrease in $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}|J(\\pi(\\cdot|\\cdot,g))-J(\\pi(\\cdot|\\cdot,g+\\epsilon))|$?\n3. can MSR enhance the policy's cumulative rewards, $J(\\pi)$?\n\nThe first two questions directly correspond to our theoretical derivations, while the third question assesses the practicality of our method, indicating that improving the continuity of the policy's goal-achievement can also enhance its ability to obtain cumulative rewards. Therefore, we consider our experimental design to be well-aligned with our theoretical development.\n\n**Secondly**, our focus is on improving policy continuity for goal achievement by reducing $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}|J(\\pi(\\cdot|\\cdot,g))-J(\\pi(\\cdot|\\cdot,g+\\epsilon))|$, not on reaching farther goals, long-horizon reasoning, or robustness to environmental perturbations.\n\n**Thirdly**, our Section 4.4 ablation study with $\\lambda=1$ shows $KL(\\pi_g||\\pi_{g+\\epsilon})$ nearing 0 (Fig.7(a)), but $J(\\pi)$ is lowest (Fig.7(b)). This observation aligns with Theorem 3.3, emphasizing the need to balance the reduction of $KL(\\pi_g||\\pi_{g+\\epsilon})$ to maintain goal-achievement continuity without overly compromising performance.\n\n`W3: Contribution and Utility`\n\nThe KL divergence is commonly used as a measure of distance to constrain policies in policy optimization. In the context of multi-goal settings, we compare MSR method with two KL-based policy constraint approaches:\n\n||Optimization Objective|\n|:-:|:-:|\n|TRPO[1]|$E_{g\\sim p_{dg}}KL[\\pi_{\\theta_{t-1}}(\\cdot\\vert\\cdot,g)\\Vert\\pi_{\\theta_t}(\\cdot\\vert\\cdot,g)]$|\n|Offline-to-Online[2]|$E_{g\\sim p_{dg}}KL[\\pi_0(\\cdot\\vert\\cdot,g)\\Vert\\pi_{\\theta_t}(\\cdot\\vert\\cdot,g)]$|\n|MSR|$E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}KL[\\pi_{\\theta_t}(\\cdot\\vert\\cdot,g)\\Vert\\pi_{\\theta_t}(\\cdot\\vert\\cdot,g+\\epsilon)]$|\n\n1. TRPO aims to prevent the policy from changing too much during optimization, thus it achieves this objective by constraining the KL divergence between policies in consecutive training iterations.\n2. The Offline-to-Online method aims to prevent the policy from forgetting the knowledge learned offline during online learning, by constraining the KL divergence between the offline-learned policy, $\\pi_0$, and the policy currently being learned online, $\\pi_{\\theta_t}$.\n3. MSR, on the other hand, seeks to enhance the policy's continuity of goal-achievement ability by limiting the divergence between policies corresponding to adjacent goals.\n\n**In summary, although all these methods utilize KL divergence, from the perspective of optimization objectives, MSR is fundamentally distinct from other KL-based methods.**\n\n[1] Schulman, John, et al. Trust region policy optimization. ICML,2015.\n\n[2] Baker, Bowen, et al. Video pretraining (vpt): Learning to act by watching unlabeled online videos. NeurIPS,2022.", "claims_and_evidence": "Please refer to strengths and weaknesses.", "methods_and_evaluation_criteria": "Please refer to strengths and weaknesses.", "theoretical_claims": "Please refer to strengths and weaknesses.", "experimental_designs_or_analyses": "Please refer to strengths and weaknesses.", "relation_to_broader_scientific_literature": "Please refer to strengths and weaknesses.", "essential_references_not_discussed": "Please refer to strengths and weaknesses.", "other_strengths_and_weaknesses": "### Strengths\n\n* The paper is well written and easy to follow.\n* Authors have presented the theoretical results in an intuitive manner.\n\n### Weaknesses\n\n* **Theory To Practice**: My main concern is the connection of theoretical results to the practical algorithm. Theoretical claims present that previously trained policies can reach similar and adjacent goals while obeying similarity constraints, ie- the goal reaching policies are similar but not the same. However, I am struggling to understand how this connects to the empirical method of regularizing the policy with the divergence between policies. Intuitively, we want to balance between the similarity between the two policies but the method regularizes the RL objective using this as a constraint. It would be more suitable to automatically adjust the policy using a distillation scheme or perhaps learn a temperature parameter. Could the authors explain on how clipping the KL term connects to theoretical results?\n* **Empirical Evaluation**: The current empirical evaluation is limited to only a few tasks and evaluation of $J(\\pi)$ and $\\text{KL}(\\pi_{g}||\\pi_{g+\\epsilon})$ as optimization metrics. However, the paper does not answer the central question of how does MSR benefit the goal reaching abilities of RL algorithms? Does MSR help the policy reach farther goals? Does MSR benefit long-horizon reasoning of RL agents? How does MSR provide robustness to environmental perturbations? Another important aspect to validate is the empirical value of theoretical claims. Do the bounds obtained by authors hold for KL constraints? When $\\text{KL}(\\pi_{g}||\\pi_{g+\\epsilon}) = 0$, does the policy actually reach more adjacent goals? In its current form, the work only explores performance of goal reaching policy using prefixed metrics.\n* **Contribution and Utility**: I am struggling to understand the novel contribution of the work and its utility for the RL community. How does the algorithm benefit RL algorithms since KL regularization is already a well known technique for policy improvement as well as GCRL. Besides, KL minimization is akin to likelihood maximization which raises the question of the new findings the paper contributes. Perhaps the main contribution lies in reaching adjacent goals via KL constraint as a distribution matching technique. In that case, it would be helpful for authors to study unseen environments or few-shot goal reaching improvement of the policy.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please refer to strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "JGnMY0UWbX", "reviewer": "Reviewer_F6UM", "summary": "This paper addresses the issue of discontinuity in goal-achievement capabilities in Goal-Conditioned Reinforcement Learning (GCRL) algorithms. First, this paper theoretically proof that reusing successful trajectories can help achieve adjacent goals, but policy discrepancies must be controlled to avoid performance degradation. To tackle this, this paper proposes a Margin-Based Policy Self-Regularization (MSR) approach, which constrains policy differences between adjacent goals within a minimal threshold. The approach is evaluated on two robotic arm control tasks and an aircraft control task, demonstrating improved continuity in goal-achievement ability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`... discuss the relationship between MSR and TRPO, PPO methods in designing the MSR loss.`\n\nWe denote the policy at the t-th iteration as $\\pi_{\\theta_t}$, with $\\theta$ denoting policy parameters. $\\pi_{\\theta_t}(\\cdot|s,g)$ maps state $s$ and goal $g$ to an action distribution.\n\nTRPO and PPO's key principle is to limit policy updates by constraining the KL divergence between policies in consecutive training iterations, $E_{g\\sim p_{dg}}D_{KL}(\\pi_{\\theta_{t-1}} (\\cdot|\\cdot,g),\\pi_{\\theta_t}(\\cdot|\\cdot,g))$. In contrast, MSR restricts the policy's action distribution variation across goals within the same iteration, $E_{g\\sim p_{dg},\\epsilon \\sim(-\\epsilon',\\epsilon')}D_{KL}(\\pi_{\\theta_t}(\\cdot|\\cdot,g),\\pi_{\\theta_t}(\\cdot|\\cdot,g+\\epsilon))$.\n\nIn essence, TRPO, PPO, and MSR use KL divergence for policy constraints but focus on distinct optimization objectives. This clarification will be added to our paper.\n\n`Would MSR still work for other reward types?`\n\nMSR is effective for all reward types:\n\n1. Theorem 3.4 makes no assumptions about the reward function. The return gap remains bounded, albeit with a complex expression. For clarity, we derived a simpler bound under the distance-based reward, facilitating understanding of the relationship between $|J(\\pi_{g_1})-J(\\pi_{g_2})|$ and $d_{KL}(\\pi_{g_1}(\\cdot|s),\\pi_{g_2}(\\cdot|s))$.\n2. In our experiments, we used both distance-based rewards (in Reach and VVC) and sparse rewards (in Push). Results demonstrate MSR's effectiveness across these different reward settings.\n\n`Does MSR introduce significant training overhead? How does MSR impact convergence speed?`\n\nMSR adds training overhead without influencing inference, due to added regularization only in policy optimization. For on-policy RL, where the main cost is environment sampling, MSR's effect is negligible, with only a 2.14% increase. In off-policy RL, which leverages replay buffer data, the impact is more pronounced: a 21.14% increase for MSR-GC-SAC and 16.02% for MSR-HER. Figs. 13, 17, and 21 reveal MSR's convergence rate is on par with baselines. Overall, despite the slight added training overhead, MSR markedly improves policy continuity in achieving goals.\n\n||GC-PPO|MSR-GC-PPO|GC-SAC|MSR-GC-SAC|HER|MSR-HER|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Wall Clock Time (s)|3498|3573|2621|3175|5096|5912|\n\n_**Note**: Results come from PPO (1e7 steps), SAC (1e5 steps), and HER (1e5 steps) on VVC._\n\n`Why the optimal value of λ is quite small?`\n\nWe address your concerns from three aspects:\n\n1. MSR is designed to lower $D_{KL} (\\pi_{g_1}(\\cdot|s), \\pi_{g_2}(\\cdot|s))$ without being excessively small. While 0.001 may seem trivial, our ablation studies confirm it sufficiently meets our objectives.\n2. We've included a [plot](https://github.com/anonymouslinks/anonymous/blob/main/her_losses_lambda_1e-3_in_training.png) showing training loss trends with $\\lambda = 0.001$. MSR loss is comparable to entropy and critic losses and an order of magnitude less than actor loss, demonstrating its impact on policy optimization despite the small $\\lambda$.\n3. Small coefficients for auxiliary losses are standard in RL. The primary goal is to maintain the RL objective's dominance, with other losses in a supporting role. For instance, entropy loss in PPO typically falls within $[10^{-5}, 10^{-2}]$, as seen in the [Stable Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ppo.yml).\n\n`Are the bounds of Eq.(1) and (2) tight?`\n\nEq.(1) and (2) can be unified as $|J(\\pi_{g+\\epsilon})-\\frac{1}{1-\\gamma}\\sum_{(s,a)}\\rho_{\\pi_g}(s,a)r(s,a,g+\\epsilon)| \\le |\\frac{2\\sqrt{2}R_{\\text{max}}}{(1-\\gamma)^2}\\sqrt{D_{KL}(\\pi_{g+\\epsilon}(\\cdot|s),\\pi_g(\\cdot|s))}|$. We examine the tightness of this bound up to a constant with [a particular MDP instance](https://github.com/anonymouslinks/anonymous/blob/main/MDP.png). In the case of $g_1$ with policy $\\pi_{g_1}(\\cdot|s_0)=(0.6,0.4)$:\n\n1. For $\\pi_{g_2}$ identical to $\\pi_{g_1}$, the left-hand side equals $\\frac{0.2\\gamma}{1-\\gamma}$, while the right-hand side is $\\frac{0.4\\gamma}{1-\\gamma}$, showing a constant factor difference.\n2. When $\\pi_{g_2}$ is slightly different, with $\\pi_{g_2}(\\cdot|s_0)=(0.55,0.45)$, the left-hand side becomes $\\frac{0.15\\gamma}{1-\\gamma}$, and the right-hand side is $\\frac{0.4\\gamma + 0.2}{1-\\gamma}$, again differing by a constant factor.\n\nWe will include the detailed proof in the Appendix of manuscript.\n\n`(1) Is Figure 4(b) converged?. (2) The lack of explanation for Figure 7's results ...`\n\nThank you for your suggestion. We have increased the training steps for HER and MSR-HER on Push and updated Fig.4 and added a convergence analysis in the manuscript. Fig.7 illustrates that a larger MSR strength $\\lambda$ results in a stronger constraint on $D_{KL}(\\pi_{g+\\epsilon}(\\cdot|s),\\pi_g(\\cdot|s))$ (values approach 0). We will provide a detailed analysis of the experimental data and corresponding conclusions in the manuscript.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes. Evaluation metrics include policy discrepancy (KL divergence), return gap, and cumulative rewards, which are appropriate for the problem.", "theoretical_claims": "Yes. The theoretical results are well-structured and logically sound. Theorem 3.3 correctly bounds cumulative rewards in terms of KL divergence, providing rigorous justification for the method.", "experimental_designs_or_analyses": "Yes. This paper conducts performance analysis  on different GCRL algorithms.", "relation_to_broader_scientific_literature": "The paper positions itself within GCRL literature.", "essential_references_not_discussed": "Yes. It could be good if this paper can discuss the relationship between MSR and TRPO, PPO methods in designing the MSR loss.", "other_strengths_and_weaknesses": "Strengths: \n1. The paper effectively identifies and addresses the challenge of discontinuity in goal-achievement capabilities within GCRL. This clear motivation underpins the design of MSR.\n2. The authors present a robust theoretical foundation for their work, enhancing the credibility and depth of the proposed MSR method.\n3.  The inclusion of a thorough ablation study strengthens the paper by demonstrating the impact of various components of the MSR approach.\n\nWeaknesses:\n1. The MSR method introduces at least three key hyperparameters ($\\lambda$, $\\beta$, $\\epsilon$) that need careful tuning. This requirement may increase the complexity of applying the approach across different environments and tasks.\n2. The ablation study indicates that the optimal value for the hyperparameter λ is 0.001, which seems quite small. This raises questions about the significance of the MSR loss contribution. Additionally, the lack of explanation for Figure 7's results makes it challenging to understand the limited impact observed. A comparative analysis would be beneficial to elucidate this point.\n3. Theoretical Equations (1) and (2) present symmetric upper and lower bounds. There is a concern that these bounds might be too loose in practical settings, potentially affecting the precision of the theoretical guarantees.", "other_comments_or_suggestions": "1. It would be beneficial for the paper to analyze whether the policy regularization introduced by MSR adds significant computational overhead during inference. \n2. The paper should consider incorporating a convergence analysis of the proposed MSR method. As shown in Figure 4(b), the cumulative reward appears not to have converged, raising concerns about the stability and reliability of the learning process. A detailed convergence analysis would provide insights into the long-term performance and robustness of the MSR approach.", "questions_for_authors": "1. The theoretical results assume a distance-based reward. Would MSR still work for other reward types?\n2. Does policy self-regularization introduce significant training overhead? How does MSR impact convergence speed?\n3. Why the optimal value of λ is quite small? Could the authors add some analysis?\n4. Is Figure 4(b) converged?\n5. Equations (1) and (2) have symmetric upper and lower bounds, are the bounds tight?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a theoretically motivated policy self-regularization method to improve continuity in goal-conditioned RL. Reviewers praised the clear theoretical analysis and empirical validation. Minor concerns about statistical significance and figure clarity were adequately addressed in rebuttal. The proposed approach is original and broadly applicable.
Decision: Accept (poster)