f-Policy Gradients: A General Framework for Goal-Conditioned RL using f-Divergences
Authors: Siddhant Agarwal
Ishan Durugkar
Peter Stone
Amy Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Goal Conditioned Reinforcement Learning
Shaping Rewards
Reward Design

Abstract: Goal-Conditioned Reinforcement Learning (RL) problems often have access to sparse rewards where the agent receives a reward signal only when it has achieved the goal, making policy optimization a difficult problem.
  Several works augment this sparse reward with a learned dense reward function, but this can lead to sub-optimal policies if the reward is misaligned. 
  Moreover, recent works have demonstrated that effective shaping rewards for a particular problem can depend on the underlying learning algorithm. 
  This paper introduces a novel way to encourage exploration called
  $f$-Policy Gradients, or $f$-PG. $f$-PG minimizes the f-divergence between the agent's state visitation distribution and the goal, which we show can lead to an optimal policy. We derive gradients for various f-divergences to optimize this objective. Our learning paradigm provides dense learning signals for exploration in sparse reward settings. We further introduce an entropy-regularized policy optimization objective, that we call $state$-MaxEnt RL (or $s$-MaxEnt RL) as a special case of our objective. We show that several metric-based shaping rewards like L2 can be used with $s$-MaxEnt RL, providing a common ground to study such metric-based shaping rewards with efficient exploration. We find that $f$-PG has better performance compared to standard policy gradient methods on a challenging gridworld as well as the Point Maze and FetchReach environments. More information on our website https://agarwalsiddhant10.github.io/projects/fpg.html.

f-Policy Gradients: A General Framework for Goal Conditioned RL using f fDiivergences
Siddhant Agarwal
Ishan Durugkar
The University of Texas at Austin siddhant&cs. utexas.ed. edu
Sony Al
ishan. dwwugkarrsony. com
Peter Stone
Amy Zhang
The University of Texas at Austin Sony AI pstoneâcs www.us.eessc.... edu
The University of Texas Austin
a may.zhann@austin utexas edu
Abstract
Roal-Conditionaa Reinforcement Learning (RL) problems often have access to sparse rewards where the agent receives reward signal only when it has achieved the goal, making policy optimization difficult problem. Several works augment this sparse reward with learned dense reward function, but this can lead sub-optimal policies if the reward is misaligned. Moreover, recent works have demonstrated that effective shaping rewards for a particular problem can depend on the underlying learning algorithm. This paper introduces novel way to encourage exploration called f-Policy Gradients, or f-PG. f-PG minimizes the f-divergence between the agent's 's state visitation distribution and the goal, which we show can lead to an optimal policy. We derive gradients for various f-divergences to opti- mize this objective. Our learning paradigm provides dense learning signals for exploration in sparse reward settings. We further introduce an entropy-regullaized policy optimization objective, that call state-MaxEnt RL (or s-MaxEnt RI.) a special case of our objective. We show that several metric-based shaping rewards like 1.2 can be used with s-MaxEnt RL, providing common ground study such metric- based shaping rewards with efficient exploration. We find that f-PG has better performance compared standard policy gradient methods challeng- ing gridworld as well as the Point Maze and FetchReach environments. More information our website https:/agaawwalsiddaan 10, 10.iiffubb.o/orrjeet////ssg
Introduction
Reinforcement Learning (RL) algorithms aim to identify the optimal behavior (policy) for solving a task by interacting with the environment. The field of RL has made large strides recent years (Mnih al., 20133; Silver et al.] 2017 HHarnoja al. 2018 Ouyang al.] 2022: Wurman al.) 2022) and has been applied to complex tasks ranging from robotics (Gupta et al. 2019), protein synthesis (Jumper al. 202T). computer architecture (Fawzi al. 2022 and finance (Liu al.
202T). Goal--Cndditooed RI. (GCRL) is generalized form of the standard RL paradigm for learning policy that can solve many tasks, as long each task can be defined by single rewarding goal state. Common examples of goal-conditioned tasks arise in robotics where the goal states can be target object configuration for manipulation-base based tasks (Kim et al. 2022 Gupta et al. 2019 OpenAl
et al. 2021) or a target location for navigation- based tasks (Shah et al. 2020 Gervet et al. 2023).
In any reinforcement learning setup, the task is conveyed to the agent using rewards (Silver et al.) 2021 In goal-conditioned RL settings, a common reward function used is 1 when the goal is
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

achieved and 0 everywhere else. This reward function is sparse and poses a huge learning challenge to obtain the optimal policy without any intermediate learning signal. Prior works (Ng et al. 1999] Ni
el 2020 Durugkar et a 2021] Arjona- Medina et al. 2019 Goyal et al. 2019) have augmented the
reward function to provide some dense signal for policy optimization. A major issue with augmenting
reward functions is that the optimal policy for the new reward function may no longer be optimal under the original, true reward function (Ng et al., 1999). Moreover, it has been shown (Booth et al. 2023) that shaping rewards that improve learning for one learning algorithm may not be optimal for another learning algorithm. Algorithms that learn reward functionn (Ni et al. 2020; Durugkar et al. 2021 Zheng et al. 2018) are inefficient because the reward function must first be learned before it can be used for policy optimization. These challenges lead to the following research question: Is there another way provide dense learning signals for policy optimization other than through dense shaping rewards?
In this work, we look at using divergence minimization between the agent's s state visitation and the goal distribution (we assume that each goal can be represented as a distribution, Dirac distribution being the simplest) as an objective to provide additional learning signals. Similar perspectives to policy learning has been explored by prior works (Ziebart et al., 2008 Haarnoja et al. 2017 2018
Ho & Ermon 2016 Ni et al. 2020 Ghasemipour et al.] 2019 Fu et al. 2017). but they reduce their methods into reward-centre view. MaxEnt RL methods (Ziebart et al. 2008 Haarnoja et al.; 2017
2018) use the distribution over trajectories rather than state visitations and still suffer from sparsity
the task rewards are sparse. Imitation learning works like those Ho & Ermon (2016); Fu et al.
(2017); Ghasemipour et al. a (2019) use variational lower bound to obtain min-max objectives that require discriminators. These objectives suffer from mathematical instabilities and often require coverage assumptions i.e., abundant overlap between the agent's state visitation distribution and goal distribution. Our method does not rely on discriminators nor does it assume state coverage. It provides dense signals to update the policy even when the agent has not seen the goal. These signals push the policy towards higher entropy state visitations until the goal is discovered.
Our method, f-PG or f-Policy Gradient, introduces a novel GCRL framework that aims to minimize a general measure of mismatch (the f-divergence) between the agent's 's state visitation distribution and the goal distribution. We prove that minimizing the f-divergence (for some divergences) recovers the optimal policy. The analytical gradient for the objective looks very similar to a policy gradient which allows to leverage established methods from the policy gradient literature to come up with an efficient algorithm for goal- coal-conditioned RL. We show the connection of our method to the commonly used metric-based shaping rewards for GCRL like L.2 rewards. We show that special case of f-PG jointly optimizes for maximization of reward and the entropy of the state-visitation distribution thus introducing -Late-MaEEn RL (or s-MaxEnt RL). Using sparse gridworld, we establish the benefits of using /-G as a dense signal to explore when the agent has not seen the goal. We also demonstrate that our framework can be extended to continuous state spaces and scale to larger and debber-dmensional state spaces in maze navigation and manipulation tasks.
Our key contributions are 1) developing novel algorithm for coal-conditioned RL that provably produces the optimal policy, 2) connecting our framework to commonly known metric-based shaping rewards, 3) providing new perspective to RL (s-MaxEnt RL) that focuses maximizing the entropy of the state-visitation distribution and 4) empirical evidence demonstrating its ability to provide dense learning signals and scale larger domains.
2 Background
This section goes over the standard goal-conditioned reinforcement learning formulation and the f-divergences that will be used in the rest of the paper.
Goal-conditioned reinforcement learning. This paper considers an agent in goal-conditioned
MDP (Puterman 1990; Kaelbling, 1993). A goal-conditioned MDP is defined as a tuple (S, A,A, r,, Y, Ma pg) where S is the state space, A is the action space, P= S x + A(S) is the transition probability (A(-) denotes probability distribution over set), y E 00. 1) is the discount factor, 14 is the distribution over initial states, g C is the set of goals, and Pg A() is the distribution over goals. At the beginning of an episode, the initial state su and the goal g are sampled from the distríbutions μl and Pg. The rewards r S g - R are based on the state the agent visits and conditioned on the goal specified during that episode. This work focuses on sparse rewards, where r(s', g) when s' = g, and is r(s', g) = O otherwise. In continuous domains, the equality is relaxed to s' B(g.r) r) where B(g,r) r) represents a ball around the goal g with radius r.
2

---

A trajectory is defined as the sequence (so, ao s1,..,,,..1111........ 11, The return (s(s) is defined as the cumulative undiscounted rewards (s := > [r(5++1, g)]s0 = s], where I the length of a trajectory. We will assume the trajectory ends when maximum number of policy steps (T) have been executed. The agent aims to learn a policy π : S g - ((A) that maximises the expected return ET,,, g The optimal policy π arg max Hen E Ex,,, (a,))HH))))),, where the space of policies II is defined by a set of parameters 0 E O.
Distribution matching approach to goal-conditioned RL. The distribution over goal-conditioned trajectories is defined as po(T: g) pl 80) (T 8t-1; at- (ro1))r((aa (): The trajectory- dependent state visitation distribution is defined as 7) (s). It is the number of times the state s visited in the trajectory т. The agent's goal-conditioned state visitation can then be defined as:
p)()g)) po (s)
Z
I mp(a 11,, as (o
dr. J Upp(s4+ st, To (aclse: 9) (s)drds
(2)
The goal defines an idealized target distribution Py (SS), considered here as Dirac distribution which places all the probability mass the goal state Ps = ((g). Such formulation has been used previously in approaches to learn goal-conditioned policies (Durugkar et al. (021). This work
focuses on minimizing the mismatch of an agent's goal-conditioned state visitation distribution po(s; g) to this target distribution Py. In this paper, we will be using Po and pa interchangeably i.e., po corresponds to the visitation distribution induced by policy π that parameterized by
To do so, this paper considers a family of methods that compare the state-visitation distribution induced by a goal-conditioned policy and the ideal target distribution for that goal g. called f- divergences. f-divergences are defined as (Polyanskiy & Wu 2022).
Dy(())Q) = - P(x) Q(x) P(x) dr (()x))((((() = O)),
where f is convex function with f(1) f'( is not defined (is oo) for several f f-divergences and so it is common assumption that Q 0 wherever P = 0. Table I shows list of commonly
used fddivergences with corresponding. f and ('())).
f-divergence
(P(())
f(u)
f'(u)
f(())
FKL
S P(x) log Q(x)
P(x) dr
u log u
1+ log
Undefined
RKL
(((() log Q(x) dr
- llog
-
0
P(x)
I P(x) log PinTQ
2P(x)
u log u-
JS
2t.. log
log
Q(x) log (())+Q0)) dr
2Q26))
(1 + u) log
14 2
1+u
x2
{((()) 3
Q(3)
P(x) d)
1 uu - 1)2
u
Undefined
Table Selected list of f-divergences (Pl()) where f is convex, lower-semicontinuon and f(1) O.
Q) with generator functions f and their derivatives f',
3 Related Work
Shaping Rewards. Our work is related separate class of techniques that augment the sparse reward function with dense signals. Ng et al. (1999) proposes a way to augment reward functions without changing the optimal behavior. Intrinsic Motivation (Durugkar et al. |022 Bellemare et al., 2016, Singh et al. [2010] Barto 2013) has been an active research area for providing shaping rewards. Some work (Nickum) 2010 Zheng al. 2018) learn intrinsic or alternate reward functions for the underlying task that aim to improve agent learning performance while others (Durugkar et al.) 20 et al. 2020; Goyal et al. 2019) learn augmented rewards based on distribution matching. AIM (Durugaar et al. 2021) learns a potential based shaping reward to capture the time-step distance but requires a restrictive assumption about state coverage, especially around the goal while we do not make any such assumption. Recursive classification methods (Eysenbach et al. |021 2020) use future state densities as rewards. However, these methods will fail when the agent has never seen the goal. Moreover, in most of these works, the reward is not stationary (is dependent the policy)
3

---

which can lead to instabilities during policy optimization. GoFAR (Ma et al. 2022) is an offline goal-conditioned RL algorithm that minimizes a lower bound to the KL divergence between Po(s) and the (s)). It computes rewards using discriminator and uses the dual formulation utilized by the DICE family (Nachum et al. 2019), but reduces to GAIL. (Ho & Ermon 2016) in the online setting, requiring coverage assumptions. Our work also minimizes the divergence between the agent's s visitation distribution and the goal distribution, but we provide a new formulation for on-policy goal-conditioned RL that does not require a discriminator or the same coverage assumptions.
Policy Learning through State Matching. We first focus on imitation learning where the expert distribution (s(s a) is directly inferred from the expert data. GAIL (Ho & Ermon 2016) showed that the inverse RL objective is the dual of maathnn f-MAX (Ghasemipour et al. 2019) uses f-divergence as a metric to match the agent's 's state-action visitation distribution P- (3, a) and Pr a). Ke et al. p:19); Ghasemipour et al. (2019) shows how several commonly used imitation learning methods can be reduced to divergence minimization. But all of these methods optimize lower bound of the divergence which is essentially min-max bilevel optimization objective. They break the min-max into two parts, fitting the density model to obtain a reward that can be used for policy optimization. But these rewards depend on the policy, and should not be used by RL algorithms that assume stationary rewards. f-IRL (Ni et al. 2020) escapes the min-max objective but learns reward function that can be used for policy optimization. We do not aim to learn a reward function but rather directly optimize for a policy using dense signals from f-divergence objective.
In reinforcement learning. the connections between entropy regularized MaxEnt RL and the mini- mization of reverse KI. between agent's trajectory distribution, (=) and the "optimal" trajectory distribution, p (r) x e*(r) has been extensively studied Ziebart (2100); Ziebart et al. 20008 ; Kappen et al. (0112);
(Haarnoja et al. (2018). MaxEnt RL optimizes for a policy with maximum entropy but such a policy does not guarantee maximum coverage of the state space. Hazzan et al. (2018) discusses an objective for maximum exploration that focuses on maximizing the entropy of the state-visitation distribution or KI. divergence between the state-visitation distribution and uniform distribution. A few works like Durugkar et al. (2023 202T); Ma al. (2022), that have explored state-matching for reinforcement learning, have been discussed above. Several works like (Belousov) & Peters 2018 2019 Touati et al. (220)) have used divergence to constraint the policy improvement steps making the updates more stable.
Levine (011))
Limitations of Markov Rewards. Our work looks beyond the maximization of a Markov reward for policy optimization. The learning signals that we use are non- non-stationary. We thus discuss the limitations of using Markov rewards for obtaining the optimal policy. There have been works (Abel et al. 2021: Clark& & Amodei 2016: Icarte et al.) 2018 2021) that express the difficulty in using Markov rewards. Abel et al. (2021 proves that there always exist environment-tass pairs that cannot be described using Markov rewards. Reward Machines (Icarte et al.] (018)) create finite automata to specify reward functions and can specify -or-Markov rewards as well but these are hand-crafted.
f-Policy Gradient
In this paper, we derive an algorithm where the agents learn by minimizing the following f-divergence:
J(0) = Dy (po
In this section, we shall derive an algorithm to minimize (() and analyze the objective more closely in the subsequent section. Unlike f-max (Ghasemipour et al. 2001)). we directly optimize (0). We differentiate (0) with respect
get this gradient. We follow similar technique as Ni al.
(2020) to obtain the analytical gradient of J(B).
Theorem 4.1. The gradient of J() as defined in Equation
is given by, T (P((s) (g(
T
(0) = (T)
v log To (arls)))
(5)
The gradient looks exactly like policy gradient with rewards - -f' po(ss) (+))
However, this does not
mean that we are maximizing RRL = Er-pp(t) f (R) is not the same (0. For Dirac goal distributions, the gradient in Equation 5 cannot be
(po(es) ((((()) (o.)
This is because the gradient of

---

used (as f in Equation to derive gradient for such distributions.
po(ee) (e( will be defined when (5()) = 0). We can use the definition of f-divergence
The gradient is obtained in terms of the state visitation frequencies 1/, (8). Further examination of the gradient leads to the following theorem,
Theorem 4.2. Updating the policy using the gradient (Equation) 5 maximizes pp [m- (g)l.
Theorem 4.2 provides another perspective for f-Policy Gradient - 7/- (1) is equivalent to the expected return for goal-based sparse reward, hence optimizing the true condiiiodd RL objective. We shall prove the optimality the policy obtained from minimizing // (() in the next section.
In practice, a Dirac goal distribution can be approximated by clipping off the zero probabilities at c, similar to Laplace correction. Doing so, we will be able to use dense signals from the gradient in Equation 5 while still producing the optimal policy. This approximation is different from simply
adding an e reward at every state. This is because the gradients are still weighed by f' (o((e) which depends on (sx)-
Simply optimizing (() is difficult because it faces similar issues to REINFORCE (Williams & Peng 1991). A major shortcoming of the above gradient computation is that it requires completely on-policy updates. This requirement will make learning sample inefficient, especially when dealing with any complex environments. However, there have been number of improvements to naive policy gradients that can be used. One approach is to use importance sampling (Precup 2000). allowing samples collected from previous policy ner to be used for learning. To reap the benefits (P(())
of importance sampling, need the previous state vittevisitatiin distributions to compute
(o(+)
Hence, we need to ensure that the current policy does not diverge much from the previous policy. This condition ensured by constraining the KL divergence between the current policy and the previous policy. We use the clipped objective similar to Proximal Policy Optimization (Schulman et al. (217)) which has been shown work well with policy gradients. PPO has shown that the clipped loss works well even without an explicit KL constraint in the objective. The gradient used in practice is,
VoJ(O) EE E. (p
min(ro(se) (st), clip(ro(s)) (l)) -,, c)
(6)
where rl)
no malarles) Ter (e) and (s)(
Algorithm 1 f-PG
Etzm x² fl (porls)) The derivation for this objec- tive is provided Appendix B 7 is added to improve
Let, be the policy, G be the set of goals, be a buffer
the stability of gradients and to prevent the sum of
for = 1 to num iter do
p'(
(se) from exploding.
+
for j 1 to num_ cou__taj_per_itee do
Sample g, set (,(())
For the purpose of this paper, estimators to estimate the goal distribution and the agent's state visitation distribution. We may also use discriminators to estimate the ratio of these den- sities like Ho & Ermon (2016); Fu et al.) (2017); Ghasemipour et al. (2019). But unlike these methods, we will not be incorrectly breaking a minmax objec- tive. In our case, the estimate of the gradient requires the value of the ratio of the two distributions and does not make any assumptions about the stationarity of these values. While the adversarial methods break the minmax objective and assume the discriminator to be fixed (and rewards stationary) during policy optimization.
kernel density
Collect goal conditioned trajectories,
T:9
Fit po(s) using KDE
Store f' Po(*) for each in
(.(
B - { :g
end for
for j = 1 to num pum__polccy_uppoeeppee do
0 0 0 (e) (Equation
end for end for
Theoretical analysis of f-G
In this section, we will first show that minimizing the f-divergence between the agent's state visitation distribution and goal distribution yields the optimal policy. We will further analyze to metric based shaping rewards and implicit exploration boost from the learning signals. For the rest
the connections
5

---

of the paper, we will refer to f-PG using FKL divergence as fkl-PG, f-PG using RKL divergence as rkl-PG and on.
5.1 Analysis of J())
This section shows that the policy obtained by minimizing an f-divergence between the agent's state visitation distribution and the goal distribution is the optimal policy,
Theorem 5.1. The policy that minimizes D (P(Pllp))) for a convex function with f(1) and (o)) being defined, is the optimal policy.
The proof for Theorem 5.1 provided in Appendix A The Theorem states that the policy obtained by minimizing the fddivergence between the agent's 's state-visitation distribution and the goal distribution the optimal policy for class of convex functions defining the f-divergence with f (o(o) defined. thus makes sense minimize the f-divergence between the agent's 's visitation and the goal distribution. must be noted that the objective does not involve maximizing a reward function. Note that the
condition that (oo) is defined is not true for all f-divergences. The common f f-divergences like RKL, TV, and IS have f (oo) defined rkl-PG, tv-PG, and js-PG will produce the optimal policy.
Forward KI. divergence (FKL.) has = log and so does not have ('(o)) defined. Does this mean that the policy obtained by minimizing the FKI. divergence is not optimal? Lemma [5. (proof in Appendix (A) shows that the policy obtained maximizes the entropy of the agent's state-visitation distribution along with maximizing reward of log Pa (s).
Lemma 5.1. fki-PG produces policy that maximizes the reward log Pg()) along with the entropy of the state-visitation distribution.
A similar result can be shown for x2 --dvvergence well. It must be understood that Lemmal 5.1 does not mean that fkl-PG the same as the commonly studied MaxEnt RL.
Differences from MaxEnt RL: MaxEnt RL, as studied in Haarnoja et al. (2017 2018) maximizes the entropy of the policy along with the task reward achieve better exploration. However, maximizing the entropy of the policy does not imply maximum exploration. Hazan et al. (2018) shows that maximizing the entropy of the state- visttviiitation distribution provably provides maximum exploration. Lemma 5.] shows that fkl-PG maximizes the entropy of the state-visitation distribution along with the reward making better suited for exploration. To distinguish our work, we call the MaxEnt RI, as discussed in works like Haarnoja et al. [217 2018), as m-MaxEnt RI because it only focuses on the entropy the policy. On the other hand, fk(-PG maximizes the entropy of the state-visitation distribution we call it state-MaxEnt RL or s-MaxEnt RL. Similarly, sa-MaxEnt RL can be defined to maximize the entropy of the state-action visitation distribution.
(a) a-MaxEnt RI.
(b) R-MaxEnt RL
Figure 1: Comparison of the evolution state-visitation distributions with training for M-MaxEnt RL and MaxEnt RL. The darker regions imply lower visitation while the bright regions imply higher visitations.
Since the agent's 's state visitation distribution depends on both the policy and the dynamics, simply increasing the entropy of the policy (without considering the dynamies) will not ensure that the agent will visit most of the states or will have state-visitation distribution with high entropy. In Figure I we compare the efficiencies of E-M RI and s-MaxEnt RI. to explore around a wall discrete gridworld. The initial and the goal distributions highlighted in green and red respectively)

---

Figure 2:
Evolution of - ( for f = u log u through policy learning. Top: I'( (p(a) ). darker blue are (g(5))
relatively lower values (higher values for the learning signal) while red corresponds to high values (lower values for the learning signal). Bottom: Corresponding state-visitation of the policy.
are separated by a wall. This environment is further discussed in Section [.n and Appendix C Figure [ shows the evolution of the agent's state-visitation distribution with training for s-MaxEnt RL (kl-PG) and M-MaxEnt RL (Soft Q Learning (Haarnoja et al. 2017))
Metric-based Shaping Reward: A deeper look into Lemma [5] shows that an appropriate choice of (s) can lead to entropy maximizing policy optimization with metric-basee shaping rewards. Define the goal distribution as (y(s) (s) = (#19) where f(s; g) captures the metric of the underlying space.
Then the fll--G objective becomes,
min DF (Pe. Pa) = max Ep (s: g)] Ey [og.
(7)
The above objective maximizes the reward [ss; g) along with the entropy of the agent's state visitation distribution. For 1.2 Euclidean metrie, [s; g) will be -||s g|13 which is the L2 shaping reward, and the goal distribution will be Gaussian. If the goal distribution is Laplacian, the corresponding shaping reward will be the LI norm.
AIM (Durugkar et al., 202T) used a potential -based shaping reward based on a time step quasimetric. If we define [(s; g) as Lipschitz function for the time step metric maximizing at s g. we end up optimizing for the AIM reward along with maximizing the entropy of the state- visitation distribution.
5.2 Analysis of the learning signals
f-PG involves a learning signal I'( (((aa) to weigh the gradients of log probabilities of the policy. Since we are minimizing the objective (in contrast to policy gradients) the visitation will be pushed towards states with lower values of the learning signal. It is thus important to understand how (()())) behaves for coal-conditioned RL settings. During the initial stages of training, the agent (p]
visits regions with very low Py. For such states, the signal has higher value compared to the states that have lower pe, i.e., the unexplored states. This is because for any convex function f. f f'(x)) is an increasing function, so minimizing f'( (Pa(s) will imply minimizing Po(s) for the states with low (p(
(8). The only way to do this is to increase the entropy of the state-visitation distribution, directly making the agent explore new states. As long as there is no significant overlap between the two distributions, it will push Pe down to a flatter distribution until there is enough overlap with the goal distribution when it will pull back the agent's visitation again to be closer to the goal distribution.
This learning signal should not be confused with reward in reinforcement learning. It is non-stationary and non- -or-aaaooiann as depends on the policy. More importantly, we are not maximizing this signal, just using it to weigh the gradients of the policy.
In the following example, shall use the Reacher environment (Todorov al. 2012) to illustrate how our learning signal (f'(Pu(e)) (e) )) varies as the agent learns. We will also show how this signal can push for exploration when the agent has not seen the goal yet. Consider the We fix the goal at (-0.21,
0) and show how the learning signal evolves with the policy. While Figure 2 shows the evolution of
-I'( po(s)) (note the negation) for fkl-PG, the rest can be found in Appendix P
The value of (((P(()) is highest where the agent's 's visitation is high and lower where the agent is not (s)
visiting. f(( (((Po(e) has the lowest value at the goal. As the policy converges to the optimal policy, the
7

---

regions where the state-visitation distribution is considerably high (towards the bottom-right in the figure), the value for f' (p) decreases for those states (to still push for exploration) but its value at the goal low enough for the policy converge.
6 Experiments
Our experiments evaluate our new framework (f-PG) as an alternative to conventional reward maximization for goal-conditional RL. We pose the following questions:
1. Does f-PG provide sufficient signals to explore in otherwise challenging sparse reward settings?
2. How well does our framework perform compared to discriminator-base approaches?
3. Can framework scale to larger domains with continuous state spaces and randomly generated goals?
4. How do different f-divergences affect learning?
The first two questions are answered using toy gridworld environment. The gridworld has goal contained in a room which poses a significant exploration challenge. We also show how the dense signal to the gradients of the policy evolves during training a continuous domain like Reacher. To answer the third question, our framework is compared with several baselines on a 2D Maze solving task (Point Maze). Additionally, we scale to more complex tasks such as FetchReach [Plappert et al. (2018) and an exploration-heary PointMaze.
6.1 Gridworld
We use gridworld environment to compare and visualize the effects of using different shaping rewards for exploration. We discussed this environment briefly in Section 5. The task is for the agent reach the goal contained in room. The only way reach the goal is go around the wall. The task reward is when the agent reaches the room otherwise it is O. The state space is simply the (ar, y) coordinates of the grid and the goal is fixed. A detailed description of the task is provided in Appendix [C] Although the environment seems simple, exploration here is very difficult as there is no incentive for the agent to go around the wall.
IIgH
Iaga
(a) fki-PG
(b) rkd-PG
(c) AIM
(d) GAIL
Figure 3: Gridworld: The agent needs to move from the green circle the red circle. The state visitations of the policies (after 500 policy updates) are shown when using our framework for training (fkl, rkl) compared with AIM and GAIL trained on top of soft Q learning.
Our framework is compared against AIM (Durugkar et al., 2021). which initially introduced this environment and uses a shaping reward obtained from state-mant mattheinthhig to solve it, and GAIL (Ho & Ermon 2016). which uses discriminator to Iearn the probability of state being the goal state. We provide a comparison to other recent methods in Appendix C All the baselines are implemented on top of Soft Q Learning (Haarnoja et al. 2017) which along with maximizing the augmented rewards, also maximizes the entropy the policy while f-PG is implemented as an on-policy algorithm without any extrinsic entropy maximization objective. It can be seen from Figure 3 that, f-PG can explore enough to find the way around the room which is difficult for methods like GAIL even after the entropy boost. AIM learns potential function and can also find its way across the wall. As expected, fkl-PG converges to the policy maximizing the entropy of the state visitation while rk/-PG produces the optimal state visitation as expected from Theorem [51]] This simple experiment clearly illustrates two things: (1) f-GG can generate dense signals to explore the state space and search for the goal and (2) although discriminator- based methods like GAIL try to perform state- matching. they fail to explore the space well.
8

---

6.2 Point Maze
While the gridworld poses an exploration challenge. the environment is simple and has only one goal. This experiment shows that f-PG scales larger domains with continuous state space and a large set of goals. We use the Point Maze environments (Fu et al. (020) which are set of offline RL environments, and modify it support our online algorithms. The state space is continuous and consists of the position and velocity of the agent and the goal. The action is the force applied in each direction. There are three variations of the environment namely PointMazeU, PointMazeMedium, PointMazeLarge. For the details of the three environments, please refer to Appendix E
We compare f-PG with several goal-based shaping reward, (used alongside the task reward as The rewards tried (along with their
described in Ng et al. (199)) to optimize a PPO policy !
abbreviations in the plots) are AIM (Durugkar al., 2021 (aim), GAIL (Ho & Ermon (0T6)gaii), AIRL (Fu et al., (air)) and F-AIRL (Ghasemipour et al., 2019 (fair)). All these methods employ state-man matching objective. AIM uses Wasserstein's distance while the rest use some form of f-divergence. But, all of them rely on discriminators. Along with these baselines, we experiment using our learning signal as shaping reward (fkl-rew). Additionally, we also compare with PPO being optimized by only the task reward (none). For our method, we have only shown results for fki-PG. For the rest of the possible f --divergences, refer to Section 6.4
PointMazeU
PointMazaMedlum
PointMazeLarge
1.0
0.5
fkl-pg
gail
0.8
0.
0.4
0.6
Sarsans
fairl
fkllew
0.2
0.8
1.0
1.5
stenostops
Timestags
timesnops
Figure 4: Success rates (averaged over 100 episodes and 3 sceds) of fkl-PG and all the baselines. fkl- PG performs well in all three environments and better than the baseline shaping rewards in the two tougher environments.
Figure (plotting mean and std-dev for 3 seeds) clearly illustrates that fk-PG is able to perform well in all three environments. In fact, it performs better than the baselines in the more difficult environments. It can also be seen that shaping rewards can often lead to suboptimal performance as none is higher than few of the shaping rewards. As expected, the curve fkl-new performs poorly. In the simpler PointMazeU environment, the performance for most of the shaping rewards are similar (along with none) but in more complex PointMazeMedium and PointMazeLarge, lot of these shaping rewards fail.
6.3 Scaling to Complex Tasks
We scale our method to more complex tasks such as FetchReach (Plappert et al. 2018) and a difficult version of PointMaze. the PointMaze environments used in the previous section, distributions from which the initial state and the goal are sampled, have significant overlap easing the exploration. We modify these environments to ensure a significant distance between the sampled goal distributions and the agent's 's state-visitation distribution shown in Figure 5 (top), making exploration highly challenging. Figure (bottom) shows the comparison of f fki-PG with GAIL (Ho & Ermon 2016)
and AIM (Durukkar al. 2021
The following can be concluded from these experiments: (1) The discriminative- based methods heavily depend on coverage assumptions and fail in situations where there is no significant overlap between the goal distribution and the agent's state visitation distribution. fki-PG does not depend on any such assumptions. (2) f-PG is considerably more stable than these baselines (as indicated by the variance of these methods).
6.4 Comparing different f f-divergences
We perform an ablation compare different f-divergences on their performances on the three Point Maze environments. Figure 6 (plotting mean and std-dev for 3 seeds) show that, empirically, fkl-PG performs the best followed by 2-PPG. Interestingly, both of these do not guarantee optimal policies
UUsing spinning implementation: http:://ppaaiiiggp..opppaoooao//
.tml

---

+
PointMazeLargeTough
PointMazeMedluumToughh
FetchReach-vO
0.8
1.0
Success Rate
0.6
Success Rate
0.8
Success
0.6
0.4
0.4
GAIL
0.2
0.2
0.2
FPG
0.0
1.0
1.5
2.0
1.0 Timesteps
1.5
100000
200000 300000
Timesteps
1e7
1e7
Timesteps
Figure 5: (top): Description of the environments. In the PointMaze environments, the green and red shades represent the distributions from which the initial state and goal states are sampled. (bottom): Success rates (averaged over 100 episodes and 3 secds) of fkl-PG, GAIL and AIM. fkl-PG outperforms these baselines with considerably lower variance.
PointMazeMedium
Large
1.0
0.8
rid po
Success Tulte
Success Sate
ja-p9
0.6
Pg
0.4
0.1
0.2
0.0
0.0
Timesteps
Ther srrestees
Timesteps
Figure 6: Success rates (averaged over 100 episodes and 3 seeds) of f-PG for differentt f. fkI-PG performs the best followed by x PPG
but it can be shown from Lemma 5.] that fk/-PG converges the policy that along with maximizing for a "reward", maximizes the entropy of the state-visitation. A similar result can be shown for x² as well (proof the Appendix A).. This result can explained by the need for exploration in the larger mazes, hence learning policies to keep the entropy of the state visitation high.
7 Discussion
This paper derives novel framework for goal-conditioned RL in the form of an on-policy algorithm f-policy gradients which minimizes the /'diverrgence between the agent's state visitation and the goal distribution. It proves that for certain /-divergences, we can recover the optimal policy while for some, we obtain a policy maximizing the entropy of the state-visitation Entropy-regularize policy optimization (s-MaxEnt RL) for metric-based shaping rewards be shown special case of f-PG where is fkl. f-GG can provide an exploration bonus when the agent has yet not seen the goal. We demonstrated that f-PG can scale up complex domains.
Through this work, we introduce new perspective for coal-conditioned RL. By circumventing rewards, f-G can avoid issues that arise with reward misspecification (Knox et al., 2021). There are several avenues to focus on for future work. First, the current framework on-policy and poses an exploration challenge. An avenue for future work could be develop an off-policy way to solve the objective. Second, this paper does not tackle goal distributions with several modes. Such target distribution would be interesting tackle in future work.
10

---

8 Acknowledgements
This work was in part supported by Cisco Research. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research.
This work has partially taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (FAIN-2019844 2019944, NRT-2125858), ONR (N00014-18. 2243), ARO (E2061621), Bosch, Lockheed Martin, and UT Austin's 's Good Systems grand challenge. Peter Stone serves the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy objectivity in research.
References
David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael L. Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. CoRR, abs/211 0t///2111.000 11.00766, 2021. URL.
http://ab/22111..0000 00876
Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards, 2019.
Andrew G. Barto. Intrinsic Motivation and Reinforcement Learning, pp. 17-47. Springer Berlin Hei- delberg, Berlin, Heidelberg, 2013. ISBN 978-3-644-333375-1, doi: 10. 10.100779978-3-622-3337---- URL http://////....000000000//.1///0
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi
Munos. Unifying count-hased exploration and intrinsic motivation, 2016.
Boris Belousov and Jan Peters. f-divergence constrained policy improvement. CoRR, has//880100056. 2018.
0bss/197.00214,, 2019. URL org/
00056
Boris Belousov and Jan Peters. Entropic regularization of markov decision processes. CoRR, 04214
Serena Booth, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and- error reward design: misdesign through overfitting and invalid task specifications. 2023.
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL httpht:sp//:nnn aarresearcc/fuutty--errr -ww.----rtoaddddoioiooo..
Ishan Durugkar, Mauricio Tec, Scott Nickum, and Peter Stone. Adversarial intrinsic motivation for
reinforcement learning. CoRR, 0bs/22005.13345 2021. URL https::/aaxxiv.rgg//// /ttgg/////////aa///
13345
Ishan Durugkar et al. Estimation and control of visitation distributions for reinforcement learning. PhD) thesis, 2023.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. CoRR, abs/201 I. 1.8990,, 2020. URL https://arxiv org/ abs/2011.088
Benjamin
EennaaminEyeebbachh Eysenbach, Sergey Levine, and Ruslan Salakhutdinov, Replacing rewards with examples: Example-based policy search via recursive classification. CoRR, 1ass22003.12666 2021. URL
https://arxiv. www.aaiii..orrggbbbaaaa/a
2203 12656
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mo- hammadamin Barckatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grze- gorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022. doi: 10.1038/s41558 011238/441188-022-5117771
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. CoRR, abs/1710.11248, 2017. URL http: //tp////.vv////////////a////aav/ org/
11

---

Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL httops///aaxx org/ abs/2004.0720
Theophile Gervet, Soumith Chintala, Dhruv Batra, Jitendra Malik, and Devendra Singh Chaplot. Nav- igating to objects in the real world. Science Robotics, 8(79)xeadf699, 2023. doi: 10.1
scirobotics.ae adf6991. URL httpp://wwit http://wwwwsseenee...o science..cc.ccccccc.. httpiice..rg/o///ab//o///ss
addf6991
Seyed Kamyar Seyed Ghasemipour, Richard S. Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. CoRR, 0os//9111.02551 2019.
org/abs/ 1911. 1911.02256)
Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. CoRR, abs/1903.02020, 2019. htttp:////a/ org/ aas////s///11 1903
02020
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learn- ing: Solving long-horizon tasks via imitation and reinforcement learning. CoRR, aatpaab..././////19 2019. URLpttp:://rxixi
org/abs//11// 11956
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. CoRR, abs/ abs// 702.000655 2017. http wttp.o//g/////////////// 08165
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. CoRR, abs/1801 100..00011 201.01190, 2018.
01290
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum 2018. URL httt::/ttrxii....///// org/ abs///////8/ /81/..11....02 008../22991000000000
entropy exploration. CoRR,
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. CoRR, abs/1 606.03476. 2016. URL.ptt:://arxit htatp:://arrd..oorr/a/////// 03476
Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila Mcllraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Muchine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2107-2116. PMLR, 10-15 Jul 2018. mlr. press/ v80 aares/v80/// http//icarteea.... html
URL
Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, Richard Anthony Valenzano, Margarita P. Castro, and Sheila A. Mcllraith. Learning reward machines: A study in partially observable reinforcement learning. CoRR, abs/2 12.0947, 2021. URL httpp://aaaatt http//aavvvvvv/ wwwxiv.ogg/aaee.....oooo.oo 09977
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera-Pare Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, and Demis Hassabis. Highly accurate protein struc- ture prediction with alphafold. Nature, 596: 1111, 08 2021. doi: 10. 12.1038/44158--22-0003
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pp. 1094-1099. Citeseer, 1993.
Hilbert J. Kappen, Vicenç Gómez, and Manfred Opper. Optimal control as graphical model inference problem. Machine Learming, 87(2):159-182, feb 2012. doi: 10. 10.11007/1109944---5557777 URL htttps://doi. org/
Liyiming Ke, Matt Barnes, Wen Sun, Gilwoo Lee, Sanjihan Choudhury, and Siddhartha S. Srinivasa. Imitation learning as f-divergence minimization. CoRR, 1tss/1955.12883 2019. URL http
//arxiv. www.oogg//aaoo abs/a9/5. 12888
Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Robot peels banana with goal-conditioned dual-action deep imitation learning, 2022.
12

---

W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. CoRR, abs/2104.13906, 2021. URL https://axxi org/ abs/2104, 13906.
Sergey Levine. Reinforcement learning and control as probabilissiic inference: Tutorial and review. CoRR, abs//800..a// 880500999, 2018. URR.hhttp://arxiv.ooo http:/iwww..rr/a/////5/555/ http:/aaa///11.0.0001 00909
Xiao- Yang Liu, Hongyang Yang. Jiechao Gao, and Christina Dan Wang. FinRL. In Proceedings of the Second ACM International Conference on AI in Finance. ACM, nov 2021. doi: 10.1145/
04900544.444366 URL. httptttts://oo// org/ 10. 1145/233990554 3494366
Yecheng Jason Ma. Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal-conditioned reinforcement learning via f-advantage regression, 2022. URL httph/tttss//aiiiii org/abs/ 2206. 03023
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. BRt.http://arxit wtgg/aa////////// 5602
Ofrr Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-annostic estimation of discounted stationary distribution corrections. ArXiv,
2019.
A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application reward shaping. International Conference on Machine Learning, 1999.
Tianwei Ni, Harshit S. Sikchi, Yufci Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. CoRR, abs/201 1. 1.04709, 2020. URL 04709
https://arxiv.orgg
Scott Nickum. Evolved intrinsic reward functions for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, (9511)199555))111 Jul. 2010. doi: 10. (.009/aai.v2441..777. URL https://ojs aaai. org wwrgginndx..eg http////aaaallleeeeee////e/aaa//
OpenAl, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' D'Sa, Arthur Petron, Henrique Pondé de Oliveira Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng, Qiming Yuan, Casey Chu, and Wojciech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation. CoRR, abs/2101.04882. 2021. URL https
//arxiv wragivvorrgaa htsp////.///00888
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Matthias Plappert, Marcin Andrychowiez, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Yury Polyanskiy and Yihong Wu. "Information Theory From Coding to Learning". Cam- bridge University Press, 2022. URL httth::t//s:pppleeee lids. webu/ypphhmepage/daatee. itbook eecookk-xpprt.ppo. pdf.
Doina Precup.
Eligibility traces for afff-poliy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Martin I. Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URI, http::/ttt//arx.. wwweexivvoorgaaaae oww.oos////ooooooooooooooooe /107 06347
13

---

Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Ving: Learning open-world world navigation with visual goals. CoRR. abs/2012.09812, 2020. URL https
//arxiv. org./ abs/2012. 09812
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. 1712.01815
URL http: /attp://arxt// //axxv/...ivv org/ abs/
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299: 103535, 2021. ISSN 0004-3702. doi:
103533.
URL https: www. wwwencedirect......m com htt.a//cc..................oaooa
Satinder Singh, Richard Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010. doi: 10. 10/100MM..000000
1031.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.11
Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. Stable policy optimization via off off-po divergence regularization. CoRR, abs/2003.04108, 2020. URL https://axxiv..orrr abs//203. 04108
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Peter Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
Zeyu Zheng. Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. CoRR, abs/ 1e0s1004,,659, 2018. URL http::///aa/ wwwps/iiiiorr.eooo orgoorg//aoo harg//bb//ee 1804. aa////////.....// 06459
Brian D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian Ziebart, Andrew L. Maas, Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes (eds.), AAAI, pp. 1433 1438. AAAI Press, 2008. ISBN 978-1-57735-368-3 URL http:://tllp: uni- trier de/db/ conf /aaai/ a0ai htm #Zi tMDDb
14

---

Appendix
Analysis of (
16
A.1 Proof for Theorem 5.1
16
A.2 Proof for Lemma 5.1
17
Gradient based optimization
18
B.1 Derivation of gradients
18
B.2 Practical Algorithm
20
B. Discounted State- Visitations
Gridworld Experiments
21
Description of the task
21
Performance of f
D Visualizing the learning signals
21
D.I Description of the task
21
D.2 Comparing different f PGG
22
PointMaze experiments
23
15

---

A Analysis of (()
In this section, we will present the proofs for the Lemmas and Theorems stated in Section 5.1.
A.1 Proof for Theorem 5.1
To prove Theorem 5.1, we need the following Lemmas. Lemma the optimal policy has the highest state visitation at the goal.
A.1 states that among all policies,
Lemma A.1. Let D be the set of all possible state visitations for the agent agent following some policy E II. Let n¹ the optimal goal-conditioeed policy. This optimal policy's 's state-visitation distribution will have the most measure the goal for all pn E D i.e., It" = Px (g) 2 Pa (g), V Px E D.
Proof. Let x* be the optimal policy and Ps- be the corresponding state visitation distribution. The reward for the sparse setting designed as.
r(s) {1
8 8 g. otherwise.
(o
Hence the expected return for a policy is R, is
R₂ = E2 [r()]
(g)
The return for the optimal policy is maximum among all policies R., 2 R., Vn E II. This implies Pr- (g) 2 (g), vp= E D.
Lemma A.2 states that the f-divergence between pa (s) and (y(s) is decreasing function with respect to P- (s). This means that the objective J(0) obtains its minimum value when Pr(g) is highest.
Lemma A.2. (->(()))p is decreasing function with respect Pa (g)lf if f'(o) is defined.
Proof. The goal distribution is assumed to be Dirac distribution i.e. ps(s) = if s = g and everywhere else. The f f-divergence between the agent state-visitation distribution, pr and the goal distribution, Py can be defined as,
Dy(p~1lpg) Σ pa()) (=(3)) + ('00))=> 0] pp(s)
Po30
- = f + (10 pg(())
DD Differentiating w.r.t (g) we get F' = (p(((()) - f'(o). Since f is convex function (by the definition of f-divergence), f'(x) S f' S
Hence, f((o) is defined, F' S 0. Hence F D (y(p=llp)))) IPg) ) decreasing function with respect Pr(g).
Additionally, we need Lemma A.3 and Corollary to complete the proof Theorem 5.1
Lemma A.3. If any two policies πi and T2 have the same state visitation at a given goal, they have the returns for that goal.
Proof. Follows directly from the definition of returns. R2 = Ep, [r[s]] = (g). Hence two policies Ti and π2 with the same state visitation the goal will have the same returns.
Corollary Any policy that can lead the state-visitation distribution of the optimal policy pa is optimal.
Proof. Directly follows from Lemmal A.3
Theorem 5.1. The policy that minimizes Dy(p*llps) for a convex function with f(1) and (o)) being defined, is the optimal policy.
16

---

proves that the optimal policy has the maximum state-visitation probability. Lemma A.2 proves that the f-divergence objective decreases with increasing the -tate-visitation probability at the goal. In other words, to minimize the f-divergence, we need to maximize the state visitation goal. Corollary further indicates that any policy that can lead to the state-visitation distribution of the optimal policy i.e., any policy that maximizes the ttate-viiitatin distribution at the goal state an optimal policy.
Proof. Lemma A.1
A.2 Proof for Lemma 5.1
Lemma 5.1. fkl-PG produces policy that maximizes the reward lo along with the entropy of the state-visitation distribution.
Proof. For fkl-PG, f = u log u. Hence, J(0) = Dy (P* (p1)) can be written as,
Dy(P~1log) Ep log Pr
Pg-
p. [log Pg]
E.p [log p+)]
By, [log Pa] + H(Pp))
where H(Pr) the entropy of the agent's state visitation distribution. Minimizing Dy(Pal1pp) will correspond to maximizing the reward r(s) = log and the entropy of px.
A similar result could be proved for x² divergence:
Lemma A.4. If (u) (u-1)2 (x2 divergence), Dy(Pallpg) is the upper bound of (PK((P=lP (Pr))
1. Hence minimizing D2² will also minimize DrKL recovering the entropy regularized policy.
Proof. With f = (u 1)), 1),,1)D2ee DD(Pallp))) can be written as,
/ Pg(s)
PT(s) 2
Dy(pallps)
- 1) ds
Pg(s)
(s( (Px(s) 2
-2 Pr(s)
Ps(s) + 1 ds
(3())
/ Pa
Px(8)
(s)
2px(s) + Py (s)
pg(s)
/ Pa
(s(s)
(s)
ds
(s)
Pr (8)
Epa(s)
Since, > log I,
= E (e) [x]]
> E (F) a]
(s)
Pa (s) (s(
F (e))
F. (p)
log
(g(s)
Px (x(s))
log (s() Pg (8)
(p)
> E (p)
(g)))
Minimizing LHS will also minimize RHS. RHS essentially DrI(P=lly, 1. The -1 will not have any effect on the minimization of DKL (px llp)).
17

---

B Gradient based optimization
B.1 Derivation of gradients
Theorem 4.1. The gradient of ((0) as defined in Equation 4 is given by.
VoJ(0)- + Erspo(r)
T
Vo log Ta (ie)
T
(o((5)) Pg(
(8)
Proof. We follow the proof from Ni et al. (2020). Lets start with the state-visitation distribution. In Section 2, it was shown that the state- visitation distribution can be written as,
po(s) x
/ p(()) log (o(e))
po(6) x
1/ (b)dr
S ((T)eet to 1/ (8)
po(s)
J p(F)e2in log (.)) 17 (s)
J p((T)e log (s
17 (s)
ps(
J p( log log)) nr (rr(s))ss dsdd
(s()) I ((( log mo(()) ( (s)dT
TJ I ((T)e log (o
((s)
po(s) Z
where f(s)
log 1og (s)dT and Z = T p(p)(( p(T)c log mo( (o()) dr. Differentiating w.r.t To (s(),,
p log log botreeeein )dr
(s)
dro
To(s))
and,
d7
T [ log log )dr
d
(s()
Tf(s*)
Te (8*)
Computing dpa(s) ma(s")
using df(s) and dz dmals') diTo("">
Z df(x)
f(s) dz dra(s*)
ds)(s)
dmals*)
(x*)
Z2
p log (x())
(s)
[(0) S(8))
ZTTOls))
2 ZT(
( log (o(o)) ( (s) )dr
T
T po (s(3)po(s))
(To(s*)
ds)(s)
Now we can compute
do
dso(s)
ds)(s) diTo(s").
ds*
d0
To(s*) d0
= ((T)eet= log mo(s)) (s)n (s") dr
To(s Po Po())))( po())po(0)))
T
dro(l0) ds"
(x²)
d0
18

---

1 / / p(r)eEiz loR ma(e)) mr ( (8)) (8")
VoTTa(s))
ds'dr dr Tpo(s) / po(s") VoTTo(s)) ds"
To T8(
TT(
Ve To(s) (ompp)))))))) To(o) / ((T) log m (8) = 7
1
The objective L(Ø) = ((0)==((o(()))bb(>> - S (s)s (polo) ds
(pol))
The gradient for L(0) will be given by,
VoL(0) = / Po(s) Po(s) Vopo(s)
(p(s) Pg(s) ds
= / Vpo(s)) Po(s)
ds
(s)
f r(po(e))
pp(s))
2 p(T)c lox no
Ve log No (s())
(po(s)EErrpon Vo log To (6)] (5))
z / ((T)CEZ- log (o(
1 7
Vo Fologmo(si) / nn ( (Po(s) (Po(s) (s)
dsdr
/ TPo(s)) (Po(S)
E Earmo (0)
Vo log No (s))ds
Ps(s)
1 2 / P(T)oIi- log mo(m))
T
Ve log (s(e)
Σ
T
lr (P((s)) dr
=
Pg (s.)
t=1
T f po((s) (Po(s)
E Banpo(s)
Vo log (s)
(p)(s)
7 / (T( V log (8))
1
T
(o(()) dr Pg (g(()))
{ (Po(())
]
TEE Tix)po(()
E (a)) Vo log [s)]
pg(s))
(s()
Brapo(t)
Σ Vo (
Ply
(
(Po(s) (s)
E Benpo(s) Vo log (s)
(po(5))
T Bympo(r)
Vo log TO (8t)
(g(
(o(se)
T
Esnpo(t)
E Eropa(t)
Vo log [T()))]
(o()))
1
T
po($t)
15 Bpp()))
Ve log ( st))
fr
T
Pg (s(st)
1=1
Theorem 4.2. Updating the policy using the gradient maximizes Ep In (g)].
19

---

Proof. In goal- setting, pd is sparse, so we need to use the full definition of f-divergence, Dy(polleg) = Ep (P())) (oc Ep (p) + po(g)). Difterentiating with respect to gives,
VaL(O) - (f'(p(()) Vopo())
T
(f'(Po())) (((0))
& po(T)n+())
Vo loggoo()))dT- (po(s)) (oooll)).o
(f'(po(9)) f'(o)) (E (nr (n((9) 2
V log TT
T
- -Po(o)Err-po(77
Vo log To(s]]
T
1 -((0)))))
Σ Vo log (0()))))))
The gradient has two terms, the first term (f f(o)) weighs the gradient based on the value of (g) and is always negative. It acts as an adaptive learning schedule, reducing its magnitude as the po increases. The second term is the gradient of E,g [n- (g)]. Hence using (gL( we minimize L(0) which would imply maximizing Ep, (g)].
B.2 Practical Algorithm
As mentioned in Section 4, the derived gradient is highly sample inefficient. We employ established methods improve the performance of policy gradients like importance sampling.
The first modification is to use importance sampling weights to allow sampling from previous policy 0'. The gradient now looks like,
V (T-
(T(
V log (a s.)
[
f
po (st)
(9)
(T)
Py (st)
To reduce the variance in the gradients, the objective can be modified use the causal connections in the MDP and ensure that the action taken at step only affects rewards at times t' + it. T7. Moreover,
a discount factor is used to prevent the sum Σt=
pa(ss) from exploding.
(..)
Additionally, the expectation is modified to be over states rather than trajectories.
To (a(alss)) Tig (e)
(Po(52) Pg
V (() E perpa (sc.aa))
Ve & log To (8))
Σ 7d
(10)
This gradient computation is still inefficient, because even though the samples are from previous policy Tg', it still needs to compute Et I'
pals.)
requiring iteration through full trajectories.
(o(
We can add a bias to the gradient by modifying f' pofez) to f' (Por(ee) in the objective. To
Pu(es)
ensure the bias is small, an additional constraint needs to be added to keep 0' close to 0. Following the literature from natural gradients, the constraint Optimization (Schulman et al. 2017) showed that in practical scenarios, clipped objective can be enough to do away with the KL regularization term. The final objective that
add is Dr (pr(per (Pp)). Proximal Policy
is,
y (0) Es
min(re(s:) (st), clip(ro(st).
(): (11)
where ro(s)) Vono(aylss) and (x)) = Et' 7f' (x.()))))))
Par(se)
Pyla.)
20

---

B.3 Discounted State-Vise Visitations
The sistetviiiatiin distribution defined so far has not considered a discount factor. To include
discounting, the state-visitation frequency gets modified to 7// (s) = Ei=1 7 7 Throughout
the derivation of the gradient, we used J (s)) = Ei= f(x)) but this will be modified to (s) E (st). the corresponding gradient will be,
1 (0) = T E (r)
T
Vo log log s.))
T |
7 f
P
(12)
Py
This gradient can be modified as before,
VoJ(O) Fe a.rpo(eee)))))) ( Vo log no (0r 8x))
T rf Σ
po(st) (g((t)
(13)
Adding importance sampling the gradient in Equation 13 will give gradient very similar to Equation I0] In fact, Equation i0 is a biased estimate for the gradient of the f-divergence between the discounted state-visitation distribution and the goal distribution. We can either of the two gradients but Equation I0 will be preferred for long horizon tasks.
C Gridworld Experiments
C.1 Description of the task
The task involves navigating gridworld to reach the goal state which is enclosed room. The agent can move in any the four directions and has idea where the goal is. It needs to explore the gridworld to find the path around the room to reach the goal. The task is further elaborated in Figure 7
The green square represents the agent position while the red square represents the goal.
State:
The state visible to the policy is simply the normalized æ and coordinates.
Action:
The action is discrete categorical distribution with four categories one for each left, top, right and bottom.
Figure 7: Description of the gridworld: The bold lines show the walls, green square the start position and the red square the goal.
Reward: The task reward at the goal and everywhere else. f-G does not require rewards but the baselines use task rewards.
C.2 Performance of f-G
In Section 5.1, we had compared fkI-PG and rk--P with AIM (Durugkar et al. 2021) and GAIL (Ho & Ermon| 2016). In Figure 8 we present additional baselines AIRL (Fu et al. 2017) and f-AIRL Ghhseemipour et al. (019).
D Visualizing the learning signals
D.1 Description of the task
To visualize the learning signals, we use the Reacher envi- ronment (Figure (Todorov al.] 2012). The task involves rotating a reacher arm (two joints with one end fixed). The ap- plied actions (torques) would rotate the arm so that the free end reaches the goal. The goal is fixed to be at (-0.21,0) 0) and the goal distribution is normal centred at the goal with a standard deviation of 0.02.
21
Figure The Reacher environment with fixed

---

target agent
largei agent
Sarget again
7-
(a) FKL
(b) RKL
(c) AIM
0
(d) GAIL
AIRL
FAIRL
Figure 8: Gridworld: The agent needs to move from the green circle to the red circle. The state visitations of the final policies shown when using our framework for training (fkl, rkl) compared with AIM and GAIL. trained on top soft learning.
The state of the original environment contains several things but here we simplify the state space to simply be the position of the free end and the target or the goal position.
State:
Actions: The actions are two dimensional real numbers in [-1, 1] which correspond to the torques applied the two joint respectively.
Reward: The reward is sparse i.e., when the goal is reached by the tip of the arm. But f-PG does not use rewards for training policies.
D.2 Comparing different f-PG
Figure 10 shows the evolution of (f'( for the environment. The red regions correspond to signals having a lower value while the darker blue regions correspond to signals with high value. For fkl, the scale of these signals generally varies from 10 to -5 while for x², the scale varies from 600 50. Also, for the same objective, the policy trains, these scales generally get smaller in magnitude.
Pa(s]
The following be observed from these plots:
1. In all the cases, the signals are maximum at the the goal pulling the state-visitations towards the goal.
2. All of these also push for exploration. This is pronounced in f'l and x These provide significant push towards the unexplored regions which show their inclination towards entropy-mane mooompymaiiiiiiion, confirming the theory (Lemma
22

---

E PointMaze experiments
PointMaze (Fu al. 2020) are continuous state-space domains where the agent needs navigate to the goal in the 2D maze. The agent and the goal are spawned at a random location in the maze for every episode. There are three levels based on the difficulty of the maze as shown Figure
State: The state consists of the agent's 2D position and the velocity in the maze. The goal position is appended to the state.
Action: The actions are 2D real numbers in the range [-1, 1] correspond to the force applied the agent in each of the two directions.
Reward: Although f-PG does not use rewards, the baselines use the task reward which is sparse (1 when the goal is reached and 0 everywhere else).
+
Figure 11: Description of PointMaze environments: PointMaze-U (left), PointMaze- Mediuum (mid- dle). PointMaze-Large(ighh)
For the experiments in Section 6.2 the initial and goal states are sampled uniformly over all the "VALID" states i.c., states that can be reached by the agent. Such initialization allows discriminator- based methods to fulfill their coverage assumptions. In Section 6.3 the initialization procedure is modified so that the initial state and the goal state are considerably far. This is done restricting the sampling of the initial and goal states from disjoint (considerably far away) distributions as shown in Figure
23

---

O
(a) FKI.
(b) RKI.
(S)
(d) x
Figure 10: Evolution of - ('(a(a)) along with the corresponding state-visitations for different /-divergences
fki, rkl, js and 2 The scales for the value of these signals are not shown they vary as the policy converges. f-PG provides dense signals for pushing towards exploration.
24


Reviews:
[{"review_id": "Yi1UezNKJP", "reviewer": "Reviewer_cLon", "summary": "This paper proposes a novel framework for goal-conditioned RL called f-Policy Gradients (f-PG), which minimizes the f-divergence between the agent's state visitation and the goal distribution to provide dense learning signals for exploration in sparse reward settings. The paper derives gradients for various f-divergences to optimize this objective and shows that entropy maximizing policy optimization for commonly used metric-based shaping rewards can be reduced to special cases of f-divergences. The paper compares f-PG with standard policy gradient methods on grid world environment as well as the Point Maze environments. It shows that f-PG performs well in all three environments and better than the baseline shaping rewards. The paper also illustrates how the learning signal evolves with the policy and performs an ablation to compare different f-divergences on their performances on the three Point Maze environments.", "questions": "Q1: What will be like if replacing f-divergence with some other mathematical complete metrics (i.e. L2-norm or wasserstein distance)? It is suggested to explain the motivation of utilizing f-divergence more.\n\nQ2: How does f-PG compare to some famous baselines such as curiosity-driven[1] method and DIAYN[2]? It is suggested to add some Comparative experiment。\n\nQ3: Ablation studies about different components of the methodology are suggested to added.\n\nQ4: When the state of environment is in the form of image, is there any difference about the computation of f-divergence?\n\nQ5: Some notations in the paper are not explained clearly ( e.g. “trajectory dependent state visitation” in section 3, ).\n\n[1] Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., & Efros, A. A. (2018). Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355.\n\n[2] Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018). Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070.\n\n", "limitations": "It would be better if the paper conducts more experiments from the aspects of ablation and comparison with other baselines. Whether effective or not in MARL setting may also be a good extension for the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Firstly, a new framework for goal-conditioned RL which provides dense learning signals for exploration in sparse reward setting is proposed. Secondly, it is proved that minimizing the f -divergence (for some divergences) recovers the optimal policy. Furthermore, the analytical gradient for the objective looks very similar to a policy gradient which allow authors to use established methods from the policy gradient literature to come up with an efficient algorithm for goal-conditioned RL. Finally, empirical evidence in experiment part demonstrate that f-Policy Gradients outperform standard policy gradient methods and baseline shaping rewards in challenging environments. Overall, the paper presents a promising approach for addressing the exploration challenge in goal-conditioned RL.", "weaknesses": "In this paper, utilizing f-divergence to minimize the mismatch of an agent’s goal-conditioned state visitation to this target distribution is the main contribution. Besides, theoretical as well as empirical analysis has indicated the effectiveness of the methodology. However, some weaknesses still exist and remain to be polish up.\n\n(1) Firstly, f-divergence is not a complete metric in math. Some shortcomings or limitations of it are not been discussed. \n\n(2) Secondly, some notations in the paper are not explained clearly ( e.g. “trajectory dependent state visitation” in section 3, ).\n\n(3) Furthermore, Experiment environments which are adopted in the paper seem to be too simple. Effectiveness of the methodology may not illuminated. Furthermore, some well-known methods in sparse reward setting are not compared.\n\n(4) Novelty of the paper seems to be relatively weak.\n\nWhile the main body of the paper is well-written, there is space for improvement. I defer some of my issues in the appendix to \"Questions\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for lending their expertise to reviewing our paper, as well as for their constructive critique and suggestions for improvement.\nWe address each of the concerns raised by reviewer as follows:\n\nWeakness:\n\n(1) F-divergence is not symmetric but we are not sure how that will affect our learning process. In fact, value functions themselves are quasi-metrics [1].\n\n(2) Trajectory dependent state visitation is the frequency of states visited for the particular trajectory. Say for a 5 step trajectory, the states visited were s0, s1, s2, s2, s1. The state-visitation would be s0-> 1, s1-> 2 and s2->2.\n\n(3) We have discussed the issue of simple experiments in the section above on common questions (point 6). We have provided some additional experiments as well. We have compared our work with recent works that use a learnt shaping reward or use a distribution matching approach in reinforcement learning and imitation learning.\n\n(4) Our novelty is providing an alternative perspective to goal-conditioned reinforcement learning in terms of distribution matching. We do not need to learn a dense signal for augmenting the task reward rather the dense signals are provided by the f-divergence objective. We provide comprehensive proofs on why this objective makes sense. To the best of our knowledge, we are the first to propose this novel view for goal conditioned reinforcement learning.\n\nQuestions\n\n(1) AIM uses Wasserstein distance. The benefit of f-divergence is inspired by the fact that minimizing f-divergence between distributions without overlap will push the distributions to have an overlap which means pushing for exploration. We have discussed the use of L2 in our common rebuttal.\n\n(2) Curiosity-driven methods mentioned in the cited paper use an encoder network to embed image observations and a dynamics model. We have not used visual inputs or model-based learning in our work so the comparison with curiosity-driven methods is inapplicable. DIAYN is an unsupervised reinforcement learning approach that discovers skills in an environment. It does not find the optimal policy for a particular task or goal so comparison with DIAYN is inappropriate.\n\n(3) We have compared f-PG for different f-divergences quantitatively in Section 5.4 and qualitatively in the supplementary material. We do not believe there are any other major ablations to perform on our method.\n\n(4) We have not dealt with vision based observations but using images as states can lead to several changes in the algorithm based on the environment. If the image states are assumed to be Markov, the KDE can be computed on encoded observations or use generative models to predict the probability of a sample in the distribution. Further modifications might be required, but it would be hard to predict these without empirical evaluation. \n\n(5) We explained the meaning of “Trajectory dependent state visitation” (point 2 of weaknesses).\n\nLimitations\n\nWe have provided ablations and comparisons with the relevant baselines to the best of our knowledge. We have restricted our scope to single agent environments but it will be interesting to see if our method can be extended to a MARL setting.\n\n[1]: T Wang, et. al., Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning, ICML 2023\n"}, {"review_id": "D3D7uiwe8E", "reviewer": "Reviewer_soXp", "summary": "The paper proposes to minimise the f-divergence $J(\\theta) = D_f(p_\\theta(s) \\| p_g(s))$ between the desired state-distribution $p_g(s)$ and the current one $p_\\theta(s)$. Clipped likelihood-ratio policy gradient is used to estimate the gradient. Kernel density estimation used to estimate the state visitations. Experiments on point reaching in grid world and in continuous-state maze are provided.", "questions": "See comments above", "limitations": "The paper does not provide a limitations section. Only future work is mentioned in the conclusion, but this is different from limitations. The authors are encouraged to add the limitations section.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is straightforward to follow. The idea and motivation are clear.\nOriginality: this niche has been explored in a number of works. The paper proposes a slightly novel approach, but its merit is not clear as it is only tested on simple point grid world and maze environments. In contrast, the baselines have been shown to work on robotics environments and scale quite well.\nQuality: the paper is OK as a first step in validating the idea, but to really make claims about this algorithm, comparisons on harder environments are necessary. Important details are missing, e.g., it is not really clear from the paper how exactly the gradient is estimated and how KDE was used to estimate visitation frequencies, etc.\nClarity: the paper is mostly clear but it can be improved. E.g., the lemmas and theorems in Sec. 4 are obvious. It would be better to show Algorithm box there and provide more experiments.\nSignificance: currently quite low. Basically, experiments use f-KL, which is the well-known MaxEnt RL. So, the benefit of introducing the whole f-divergence story is unclear.", "weaknesses": "It is unclear how the authors deal with having $p_g(s)$ in the denominator of the ratio, since by definition it is a delta function. Some epsilon-modification is proposed for grid world, but it doesn't seem like it would scale to any reasonably high-dimensional task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for lending their expertise to reviewing our paper, as well as for their constructive critique and suggestions for improvement.\nWe have addressed several of the concerns in the main rebuttal as well.\n\nStrengths:\n\n(1) Comparison to harder environments: We have provided additional experiments in the rebuttal. We have discussed why we have restricted ourselves to simple domains in the main rebuttal (point 6).\n\n(2) We have explained the computation of KDE in the main rebuttal (point 5). We shall include an algorithm box in the camera ready version.\n\n(3) Some Lemmas in Section 4 might seem obvious but for the sake of completeness, we felt it pertinent to include them as well.\n\n(4) f-KL is not the same as MaxEnt RL. MaxEnt RL fails with sparse rewards even in the simplest of gridworlds (Section 5.1). f-KL maximizes the entropy of state-visitation while MaxEnt RL maximizes the entropy of policy which are different. We have discussed this further in the main rebuttal (point 3).\n\nWeakness:\n\n(1) The approximation should work in most cases. It is the same mathematical approximation to prevent divide by 0 that is commonly applied in many domains i.e. cut off the low values at an epsilon. We have discussed this approximation further in the main rebuttal.\n\nLimitations:\n\nSection 6 provides a combined discussion of the limitations and future work indicated by this approach. Taking the reviewer’s suggestion into account, we will flesh this section out in the camera ready version.\n"}, {"review_id": "c6c8nVlXEu", "reviewer": "Reviewer_PJFh", "summary": "The paper focuses on goal-conditioned reinforcement learning (RL), where the goal-conditioned reward function is often sparse. Consequently, standard goal-conditioned RL algorithms suffer from the exploration problem, thus requiring the need to shape rewards. The paper proposes a a new formulation that minimizes the $f$-divergence between the policy's state-visitation distribution and the goal distribution. The paper theoretically shows that minimizing such objective yields the optimal policy and provided empirical evaluations on two maze tasks.", "questions": "- Based on Figure 4, the paper suggests that the maximizing entropy of state-visitation matters.\n\t- When computing the success rate, are we sampling from the policy, or taking the mean of the policy?\n\t- How would PPO with entropy-regularized objective (or SAC) perform?\n- The paper proposes to learn the distributions separately using kernel density estimators (KDE).\n\t- It seems like this is not really the consequence of the theory, but just a choice, is that right?\n\t- If it is indeed a choice, I am not completely sure if the instability of the discriminator plays a role.\n\t- In the related works section, there were few sentences indicating instability due to non-stationary rewards (in particular, under the subsection shaping rewards). Isn't this approach also non-stationary? Shouldn't there be similar instability as other approaches then?\n\n**Possible typos**\n- On page 2, line 52: \"insufficiency\" instead of \"insufficient\"\n- On page 2, line 90: \"learning\" instead of \"Learning\"\n- On page 3, line 94: \"expert's\" instead of \"expert\"\n- On page 3, line 102: \"learning\" instead of \"Learning\"\n- On page 3, line 133: The expectation should also be over the goal $g$\n- On page 3, line 137: The distribution should include $\\mu_0(s_0)$\n- On page 3, equation (1): $p_\\theta(\\tau ; g)$ instead of $p_\\theta(\\tau)$\n- On page 3, equation (2): $(s) d\\tau$ should be $d\\tau$ on the numerator\n- On page 4, line 160: $\\pi$ instead of $p_\\pi$.\n- Figure 2, legend: $fkl-rew$ should be $fkl-new$\n- Appendix, on page 6, line 86: Should be \"Theorem 4.3\" instead of \"Theorem 4.2\"", "limitations": "- The paper requires an assumption of knowing the goal function. That is, reward $r(s, a, s';g) = \\mathbb{I}(s' = g)$. Designing such a goal function can be challenging due to hardware constraints in real-life applications, notably in robotics tasks.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- As far as I know, the $f$-divergence between state-visitation distribution and goal distribution is a novel contribution.\n- The paper theoretically shows that optimizing their objective yields the optimal policy.\n- Good visualizations on demonstrating the policies' behaviours.", "weaknesses": "- I think $\\gamma$ should not be introduced as it does not really contribute to the idea. Also, since we wish to maximize $\\mathbb{E}_\\theta\\left[\\eta\\right]$, the aim to achieve the task as quickly as possible is there.\n- How is $\\int \\eta_\\tau(s) ds = T$ in theorem 4.3 when we are not assuming fixed horizon $T$---this is based on page 3, on line 131, where the paper assumes that the trajectory ends when the agent reaches the goal or if maximum horizon $T$ is reached.\n\t- It seems like if the paper is using the fixed-horizon setting, then theorem 4.3 will make sense, in terms of being similar to maximizing the expected return.\n\t- Furthermore, I believe we can actually relate the policy error even if the trained policy is sub-optimal. It will be nice if there is some results regarding suboptimality.\n- On page 5, second last paragraph, why do we not just assign equiprobability to the correct goal states, but $\\varepsilon$ to all other states? It seems like we already have such access anyway. Otherwise, is this not a \"coverage\" assumption?\n- If I am not mistaken, under Gaussian policies, $f$-PG with FKL essentially recovers some sort of L2 distance. I think for reaching tasks this is naturally a good and dense reward, which may explain the better performance than other approaches.\n\t- It will be interesting if the paper provides plot describing the correlation of the \"constructed\" reward and the \"natural\" dense reward (i.e. L2 distance between state and goal).\n\t- It will also be nice to show a task aside from maze and analyze whether FKL still performs better in general.\n- The paper should further describe how the state-visitation distribution $p_\\pi$ and the goal distribution $p_g$ are trained, maybe in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for lending their expertise to reviewing our paper, as well as for their constructive critique and suggestions for improvement.\nWe have addressed several of the concerns in the main rebuttal as well.\n\nWeaknesses:\n\n(1) $\\gamma$ is used later in 4.2 when deriving the practical gradient.\n\n(2) Thank you for pointing this out. We consider the fixed horizon setting. Only during evaluations, the episodes end when the goal is reached. During training, we terminate after a fixed episode length. We will correct this in our camera ready version.\n\n(3) We have not looked into policy errors for suboptimal policies in this work. It is an interesting direction for future research, and we thank the reviewer for suggesting it.. In this work, we focus on proving that our algorithm gives us the optimal policy.\n\n(4) We have discussed the approximation in detail in our main rebuttal. We are providing equiprobability to all the goal states and $\\epsilon$ to all the other states. This distribution does not require any coverage assumptions on the part of the agent’s visitation.\n\n(5) A Gaussian policy with f-PG does not lead to an L2, rather assuming a gaussian distribution over the goal with FKL can lead to an L2 like shaping reward but with an additional term for maximizing the entropy of the state-visitation distribution. In the gridworld and maze environments, using only L2 performs very poorly as the agent gets stuck by the walls.\n\n(6) We have discussed the use of KDE in the main rebuttal. We shall provide an algorithm box in our camera ready version.\n\nQuestions:\n\n(1) While evaluating we always use the mean of the Gaussian distribution generated by the policy.\n\n(2) As explained in our main rebuttal, FKL is different from MaxEnt RL. Still, in Section 5.1, all the baselines used a policy entropy-maximizing objective (based on MaxEnt RL) which failed to outperform f-PG (detailed results in supplementary material (D.2)).\n\n(3) KDE is a choice and independent of the theory.\n\n(4) In previous works like GAIL, AIRL, f-AIRL, the objective itself is defined as a min-max objective so they cannot do away with the discriminator. Since we do not have a min-max objective, we can do away with the discriminator and its instabilities. \n\n(5) The previous methods perceive their non-stationary rewards as stationary and use conventional RL to maximize the reward. They assume reward is independent of the policy throughout their optimization. We do not  make such an assumption. We obtain a gradient weighted by a signal that depends on the policy. Thus the signal we obtain cannot be considered equivalent to the reward used by previous approaches.\n\nLimitation:\n\nIn any continuous domain, the goal distribution can be defined using a normal distribution with very small variance. This is a standard assumption[1, 2] in most of the robotic domains as well. In point maze, we have used the same assumption.\n\n[1] I. Durugkar, et. al., Adversarial Intrinsic Motivation for Reinforcement Learning, NeurIPS 2021 \n\n[2] B. Eysenbach, et. al., C-Learning: Learning to Achieve Goals via Recursive Classification, ICLR 2021\n"}, {"review_id": "uMma1U3qfJ", "reviewer": "Reviewer_trTZ", "summary": "This paper studies goal-conditioned reinforcement learning and proposes a framework based on f-divergence. The authors show that minimizing the f-divergence between the agent’s visitation and the proposed goal distribution leads to the optimal policy and discuss the practical implementations, connections with previous GCRL methods.", "questions": "See the weakness section above.", "limitations": "The authors only discussed some future avenues of their work. I suggest the authors to discuss more about the benefit over the shaping rewards GCRL algorithms, and discuss the previous theoretical results established for shaping rewards methods.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The motivation to study the divergence between the state visitation and goal distribution is straightforward, which is good.\n3. The analysis is thorough. The discussion between the proposed framework and previous methods is also interesting.", "weaknesses": "1. The authors state that they are focusing on sparse-reward (1 if the agent reaches the goal and 0 otherwise). However, $p_g(s)$ will become a Dirac distribution and the f-policy gradient objective makes no sense in this case. \n2. As the authors admit, the algorithm and analysis are meaningless in a full Dirac distribution setting. The authors choose to \"add an ϵ probability to all other states\". Will this uniform ϵ addition + f-PG performs better than uniform reward shaping + standard PG with GC-reward? Intuitively, such uniform addition will still lead to sparse training signal.\n3. It is not very clear about the advantage of the proposed framework. In the strict sparse-reward setting, the framework degrades. Even in dense-reward settings, the analysis only states that optimizing the f-PG objective moves to the optimal policy, which is also true for standard PG methods in GCRL that are based on goal-conditioned (dense) rewards only. Are there any efficiency advantage of f-PG?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for lending their expertise to reviewing our paper, as well as for their constructive critique and suggestions for improvement.\n\nWe have provided a detailed analysis of the use of the Dirac distribution in our common rebuttal. We request the reviewer to go refer to it if clarifications are required.\n\nWeaknesses:\n\n(1) The f-divergence objective (Equation 4) works for both situations when the goal distribution is sparse or dense. Section 4.1 provides the analysis assuming that the goal distribution is sparse. The gradient in Theorem 4.2 is not defined when the p_g(s) is sparse but the gradient derived for such cases is provided in Theorem 4.3. Subsequent paragraphs explain how to use the gradient from Theorem 4.2 in sparse reward settings.\n\n(2) In all the experiments, the goal distribution is sparse with an $\\epsilon$ approximation. In all cases, it performs better than uniform shaping rewards.\n\n(3) All our experiments are in sparse settings where our method outperforms the standard PG methods. We show that in sparse reward schemes, we are able to provide dense signals for training. These dense signals are different from dense rewards. For each f-divergence, we get a different type of dense learning signal. In the Reacher environment, we show how these learning signals evolve with training in the supplementary section for different f-divergences. While in Section 5.3, we do show these dense signals for one particular f-divergence.\n\nLimitations:\n\nWe have discussed connections to metric based shaping rewards in Section 4.4 and how it is better as it encourages exploration. "}]

Author rebuttal:
We thank the reviewers for their reviews. We observe that the reviewers have unanimously agreed to the novelty of the f-divergence perspective of goal conditioned RL. They agree that this method is a promising approach for solving the exploration challenge in sparse reward RL.
We respectfully submit that, compared to reviews on other papers we have seen both this year and in past years,  the numerical scores given by the reviewers are significantly lower than is indicated by the text of the reviews. As detailed below, most of the questions and comments were clarification questions that we can easily address, or are requests to compare with baselines that are not directly comparable to our work. 

Please recall that the main contribution of our paper is providing a new perspective on goal conditioned RL by posing it as a distribution matching problem. To our knowledge, we are the first to provide a formal proof that minimizing the f-divergence between the agent’s visitation distribution and the goal distribution produces the optimal policy. The divergence minimization objective provides dense signals for policy optimization, unlike the traditional reward maximization perspective which struggles to learn in sparse reward settings.

We now reiterate a short comparison with related work to emphasize why this approach is novel and valuable. Some prior works have looked at imitation learning as a distribution matching problem but model the problem using minmax objectives which necessarily require discriminators. The commonly used MaxEnt RL can be thought of as minimizing the KL divergence between the agent’s trajectory distribution and a hypothetical optimal trajectory distribution represented using the exponent of the return. MaxEnt RL can be shown to maximize the reward as well as policy entropy, a tradeoff that is difficult to tune in sparse reward settings. Several methods exist that learn an additional reward (referred to as shaping reward) that can be augmented to the task reward. Our work differs from MaxEnt RL as we do not hypothesize a target trajectory distribution – rather we assume a goal distribution and we minimize the f-divergence between the agent’s state visitation distribution and the goal distribution. We also do not learn an additional reward signal to optimize the policy, rather use the dense learning signals from the f-divergence to directly optimize the policy.

There were several common clarification questions which we answer here.

(1) How did we use the Dirac distribution for the goal distribution as the f-divergence will not make sense?
In the proofs of Lemma 4.2 and Theorem 4.1, we assume the goal distribution to be a dirac distribution pointing out that this might not be true for all f-divergences. In fact, f-divergences can be defined between two distributions that do not overlap if $f’(\infty)$ is defined. For the computation of the f-policy gradient, we show in Theorem 4.3, that a dirac distribution at the goal will lead to the true policy gradient while a more diffuse target distribution will lead to the gradient in Theorem 4.2. We make an approximation for the Dirac distribution by cutting off the zero probability at \epsilon only in our experiments, so that we can use the gradient from Theorem 4.2. This is a standard approximation used to prevent divide by zero instabilities in several implementations. Our approximation is further explained in Figure 1 of the rebuttal.

(2) Does using an $\epsilon$ approximation for the Dirac distribution make it same as uniform reward shaping?
No. A uniform reward is not the same as the $\epsilon$ approximation. The details are further explained in Figure 1 of the rebuttal. A uniform reward will not push the agent to explore unvisited states while f-PG pushes the agent to increase the entropy of the state-visitation distribution. It will thus act as an impetus for the agent to explore more effectively.

(3) Is FKL the same as MaxEnt RL?
FKL is not the same as MaxEnt RL. MaxEnt RL aims to maximize the entropy of the policy while FKL will increase the entropy of the agent’s state-visitation distribution. These are not the same as shown in Figure 2.

(4) How does L2 shaping reward compare with f-PG?
L2 shaping reward incentivises the agent to move along the line segment connecting the agent’s state and the goal state. In maze situations, L2 shaping reward fails terribly as the optimal path often does not involve following the line segment from the current state to the goal state. We have explained this further in Figure 2.

(5) How is the agent’s state-visitation distribution computed?
As mentioned in Section 4.3, we use kernel density estimation to compute the agent’s state visitation. For a given goal distribution, we collect several trajectories $\tau_i ; g$. We fit a kernel density estimator on the states in these trajectories $s \in {\tau_i ; g}$. This gives $p_\theta(s)$ for the goal distribution $p_g(s)$ which is computed by fitting a kernel density estimator on the samples from $g$.

(6) Why do the experiments look simple?
One limitation of our algorithm (as discussed in the paper) is that it is on-policy. To the best of our knowledge, all the traditional robotics benchmarks like Fetch or ShadowHand use an off-policy algorithm like DDPG or TD3. Off-policy algorithms like these are considerably more sample efficient and can make use of samples from exploratory policies, while an on-policy algorithm can only use samples from policies that are very close to the current policy. We have provided some additional experiments on a complex PointMaze and FetchReach in the rebuttal in Figure 3. Extending this approach to more difficult domains is an exciting direction for future research. However, it will involve augmenting f-PG to allow for off-policy updates, which is out of the scope of this paper.


Meta-review: This paper proposes f-divergence as a new theoretical framework for goal-conditioned RL problems. The main theoretical result is that minimising f-divergence between the agent's state visitation distribution and the goal yields an optimal policy. The gradient of this f-divergence naturally gives a dense learning signal for the policy. The results on PointMaze and FetchReach (added during the rebuttal period) show competitive performance to some of the existing methods such as AIM, GAIL, and F-AIRL. 

Most of the reviewers acknowledged that the theoretical result is quite novel and interesting. In the meantime, some of the reviewers remained not convinced by the practicality of the proposed method as it is on-policy, which tends to be fundamentally less data-efficient than off-policy methods, as acknowledged by the authors. Nevertheless, I found that the new theoretical framework and its connection to max-entropy RL with reward shaping based on L2 distance are promising and novel enough to be presented at the NeurIPS community. Thus, I recommend accepting this paper. I also suggest the authors point out the distinction from the previous work on f-divergence in RL such as [1] in the revision. 

[1] Ke et al., Imitation Learning as f-Divergence Minimization
Decision: Accept (poster)