SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The proof for Theorem 5.1 is provided in Appendix A. The Theorem states that the policy obtained by minimizing the  $f$ -divergence between the agent's state-visitation distribution and the goal distribution is the optimal policy for a class of convex functions defining the  $f$ -divergence with  $f'(\infty)$  defined. It thus makes sense to minimize the  $f$ -divergence between the agent's visitation and the goal distribution. It must be noted that the objective does not involve maximizing a reward function. Note that the condition that  $f'(\infty)$  is defined is not true for all  $f$ -divergences. The common  $f$ -divergences like RKL, TV, and JS have  $f'(\infty)$  defined  $rkl$ -PG,  $tv$ -PG, and  $js$ -PG will produce the optimal policy.
Forward KL divergence (FKL) has  $f = u\log u$  and so does not have  $f^{\prime}(\infty)$  defined. Does this mean that the policy obtained by minimizing the FKL divergence is not optimal? Lemma 5.1 (proof in Appendix A) shows that the policy obtained maximizes the entropy of the agent's state-visitation distribution along with maximizing a reward of  $\log p_{g}(s)$ .
Lemma 5.1.  $fkl - PG$  produces a policy that maximizes the reward  $\log p_{g}(s)$  along with the entropy of the state-visitation distribution.
A similar result can be shown for  $\chi^2$ -divergence as well. It must be understood that Lemma 5.1 does not mean that  $fkl$ -PG is the same as the commonly studied MaxEnt RL.
Differences from MaxEnt RL: MaxEnt RL, as studied in Haarnoja et al. (2017, 2018), maximizes the entropy of the policy along with the task reward to achieve better exploration. However, maximizing the entropy of the policy does not imply maximum exploration. Hazan et al. (2018) shows that maximizing the entropy of the state-visitation distribution provably provides maximum exploration. Lemma 5.1 shows that  $fkl$ -PG maximizes the entropy of the state-visitation distribution along with the reward making it better suited for exploration. To distinguish our work, we call the MaxEnt RL, as discussed in works like Haarnoja et al. (2017, 2018), as  $\pi$ -MaxEnt RL because it only focuses on the entropy of the policy. On the other hand,  $fkl$ -PG maximizes the entropy of the state-visitation distribution so we call it state-MaxEnt RL or s-MaxEnt RL. Similarly, sa-MaxEnt RL can be defined to maximize the entropy of the state-action visitation distribution.
(a)  $s$ -MaxEnt RL
(b)  $\pi$ -MaxEnt RL
Figure 1: Comparison of the evolution state-visitation distributions with training for  $\pi$ -MaxEnt RL and  $s$ -MaxEnt RL. The darker regions imply lower visitation while the bright regions imply higher visitations.
Since the agent's state visitation distribution depends on both the policy and the dynamics, simply increasing the entropy of the policy (without considering the dynamics) will not ensure that the agent will visit most of the states or will have a state-visitation distribution with high entropy. In Figure 1, we compare the efficiencies of  $\pi$ -MaxEnt RL and  $s$ -MaxEnt RL to explore around a wall in a discrete gridworld. The initial and the goal distributions (highlighted in green and red respectively)
Figure 2: Evolution of  $-f^{\prime}(\frac{p_{\theta}(s)}{p_g(s)})$  for  $f = u\log u$  through policy learning. Top:  $f^{\prime}(\frac{p_{\theta}(s)}{p_g(s)})$ , darker blue are relatively lower values (higher values for the learning signal) while red corresponds to high values (lower values for the learning signal). Bottom: Corresponding state-visitation of the policy.
are separated by a wall. This environment is further discussed in Section 6.1 and Appendix C. Figure 1 shows the evolution of the agent's state-visitation distribution with training for  $s$ -MaxEnt RL (fkl-PG) and  $\pi$ -MaxEnt RL (Soft Q Learning (Haarnoja et al., 2017))
Metric-based Shaping Reward: A deeper look into Lemma 5.1 shows that an appropriate choice of  $p_g(s)$  can lead to entropy maximizing policy optimization with metric-based shaping rewards. Define the goal distribution as  $p_g(s) = e^{f(s;g)}$  where  $f(s;g)$  captures the metric of the underlying space. Then the  $fkl$ -PG objective becomes,
$$
\min  D _ {F K L} \left(p _ {\theta}, p _ {g}\right) = \max  \mathbb {E} _ {p _ {\theta}} [ f (s; g) ] - \mathbb {E} _ {p _ {\theta}} [ \log p _ {\theta} ]. \tag {7}
$$
The above objective maximizes the reward  $f(s; g)$  along with the entropy of the agent's state visitation distribution. For an L2 Euclidean metric,  $f(s; g)$  will be  $-||s - g||_2^2$  which is the L2 shaping reward, and the goal distribution will be Gaussian. If the goal distribution is Laplacian, the corresponding shaping reward will be the L1 norm.
AIM (Durugkar et al., 2021) used a potential-based shaping reward based on a time step quasimetric. If we define  $f(s;g)$  as a Lipschitz function for the time step metric maximizing at  $s = g$ , we end up optimizing for the AIM reward along with maximizing the entropy of the state-visitation distribution.
5.2 Analysis of the learning signals
$f$ -PG involves a learning signal  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_g(s)}\right)$  to weigh the gradients of log probabilities of the policy. Since we are minimizing the objective (in contrast to policy gradients) the visitation will be pushed towards states with lower values of the learning signal. It is thus important to understand how  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_g(s)}\right)$  behaves for goal-conditioned RL settings. During the initial stages of training, the agent visits regions with very low  $p_{g}$ . For such states, the signal has a higher value compared to the states that have lower  $p_{\theta}$ , i.e., the unexplored states. This is because for any convex function  $f$ ,  $f^{\prime}(x)$  is an increasing function, so minimizing  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_g(s)}\right)$  will imply minimizing  $p_{\theta}(s)$  for the states with low  $p_{g}(s)$ . The only way to do this is to increase the entropy of the state-visitation distribution, directly making the agent explore new states. As long as there is no significant overlap between the two distributions, it will push  $p_{\theta}$  down to a flatter distribution until there is enough overlap with the goal distribution when it will pull back the agent's visitation again to be closer to the goal distribution.
This learning signal should not be confused with reward in reinforcement learning. It is non-stationary and non-Markovian as it depends on the policy. More importantly, we are not maximizing this signal, just using it to weigh the gradients of the policy.
In the following example, we shall use the Reacher environment (Todorov et al., 2012) to illustrate how our learning signal  $(f'(\frac{p_{\theta}(s)}{p_g(s)}))$  varies as the agent learns. We will also show how this signal can push for exploration when the agent has not seen the goal yet. Consider the We fix the goal at  $(-0.21, 0)$  and show how the learning signal evolves with the policy. While Figure 2 shows the evolution of  $-f'(\frac{p_{\theta}(s)}{p_g(s)})$  (note the negation) for  $fkl$ -PG, the rest can be found in Appendix D.
The value of  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_{g}(s)}\right)$  is highest where the agent's visitation is high and lower where the agent is not visiting.  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_{g}(s)}\right)$  has the lowest value at the goal. As the policy converges to the optimal policy, the
regions where the state-visitation distribution is considerably high (towards the bottom-right in the figure), the value for  $f^{\prime}\left(\frac{p_{\theta}(s)}{p_{g}(s)}\right)$  decreases for those states (to still push for exploration) but its value at the goal is low enough for the policy to converge.
Our experiments evaluate our new framework ( $f$ -PG) as an alternative to conventional reward maximization for goal-conditional RL. We pose the following questions:
1. Does  $f$ -PG provide sufficient signals to explore in otherwise challenging sparse reward settings?
2. How well does our framework perform compared to discriminator-based approaches?
3. Can our framework scale to larger domains with continuous state spaces and randomly generated goals?
4. How do different  $f$ -divergences affect learning?
The first two questions are answered using a toy gridworld environment. The gridworld has a goal contained in a room which poses a significant exploration challenge. We also show how the dense signal to the gradients of the policy evolves during training on a continuous domain like Reacher. To answer the third question, our framework is compared with several baselines on a 2D Maze solving task (Point Maze). Additionally, we scale to more complex tasks such as FetchReach Plappert et al. (2018) and an exploration-heavy PointMaze.
6.1 Gridworld
We use a gridworld environment to compare and visualize the effects of using different shaping rewards for exploration. We discussed this environment briefly in Section 5.1. The task is for the agent to reach the goal contained in a room. The only way to reach the goal is to go around the wall. The task reward is 1 when the agent reaches the room otherwise it is 0. The state space is simply the  $(x, y)$  coordinates of the grid and the goal is fixed. A detailed description of the task is provided in Appendix C. Although the environment seems simple, exploration here is very difficult as there is no incentive for the agent to go around the wall.
(a)  $fkl$ -PG
(b)  $rkl$  -PG
(c) AIM
Figure 3: Gridworld: The agent needs to move from the green circle to the red circle. The state visitations of the policies (after 500 policy updates) are shown when using our framework for training  $(fkl, rkl)$  compared with AIM and GAIL trained on top of soft Q learning.
(d) GAIL
Our framework is compared against AIM (Durugkar et al., 2021), which initially introduced this environment and uses a shaping reward obtained from state-matching to solve it, and GAIL (Ho & Ermon, 2016), which uses a discriminator to learn the probability of a state being the goal state. We provide a comparison to other recent methods in Appendix C. All the baselines are implemented on top of Soft Q Learning (Haarnoja et al., 2017) which along with maximizing the augmented rewards, also maximizes the entropy of the policy while  $f$ -PG is implemented as an on-policy algorithm without any extrinsic entropy maximization objective. It can be seen from Figure 3 that,  $f$ -PG can explore enough to find the way around the room which is difficult for methods like GAIL even after the entropy boost. AIM learns a potential function and can also find its way across the wall. As expected,  $fkl$ -PG converges to the policy maximizing the entropy of the state visitation while  $rkl$ -PG produces the optimal state visitation as expected from Theorem 5.1. This simple experiment clearly illustrates two things: (1)  $f$ -PG can generate dense signals to explore the state space and search for the goal and (2) although discriminator-based methods like GAIL try to perform state-matching, they fail to explore the space well.
6.2 Point Maze
While the gridworld poses an exploration challenge, the environment is simple and has only one goal. This experiment shows that  $f$ -PG scales to larger domains with continuous state space and a large set of goals. We use the Point Maze environments (Fu et al., 2020) which are a set of offline RL environments, and modify it to support our online algorithms. The state space is continuous and consists of the position and velocity of the agent and the goal. The action is the force applied in each direction. There are three variations of the environment namely PointMazeU, PointMazeMedium, PointMazeLarge. For the details of the three environments, please refer to Appendix E.
We compare  $f$ -PG with several goal-based shaping reward, (used alongside the task reward as described in Ng et al. (1999)) to optimize a PPO policy<sup>1</sup>. The rewards tried (along with their abbreviations in the plots) are AIM (Durugkar et al., 2021)(aim), GAIL (Ho & Ermon, 2016)(gail), AIRL (Fu et al., 2017)(airl) and F-AIRL (Ghasemipour et al., 2019)(fairl). All these methods employ a state-matching objective. AIM uses Wasserstein's distance while the rest use some form of  $f$ -divergence. But, all of them rely on discriminators. Along with these baselines, we experiment using our learning signal as a shaping reward (fkl-rew). Additionally, we also compare with PPO being optimized by only the task reward (none). For our method, we have only shown results for fkl-PG. For the rest of the possible  $f$ -divergences, refer to Section 6.4.
Figure 4: Success rates (averaged over 100 episodes and 3 seeds) of  $fkl$ -PG and all the baselines.  $fkl$ -PG performs well in all three environments and better than the baseline shaping rewards in the two tougher environments.
Figure 4 (plotting mean and std-dev for 3 seeds) clearly illustrates that  $fkl$ -PG is able to perform well in all three environments. In fact, it performs better than the baselines in the more difficult environments. It can also be seen that shaping rewards can often lead to suboptimal performance as none is higher than a few of the shaping rewards. As expected, the curve  $fkl$ -new performs poorly. In the simpler PointMazeU environment, the performance for most of the shaping rewards are similar (along with none) but in more complex PointMazeMedium and PointMazeLarge, a lot of these shaping rewards fail.
6.3 Scaling to Complex Tasks
We scale our method to more complex tasks such as FetchReach (Plappert et al., 2018) and a difficult version of PointMaze. In the PointMaze environments used in the previous section, distributions from which the initial state and the goal are sampled, have a significant overlap easing the exploration. We modify these environments to ensure a significant distance between the sampled goal distributions and the agent's state-visitation distribution as shown in Figure 5 (top), making exploration highly challenging. Figure 5 (bottom) shows the comparison of  $fkl$ -PG with GAIL (Ho & Ermon, 2016) and AIM (Durugkar et al., 2021).
The following can be concluded from these experiments: (1) The discriminative-based methods heavily depend on coverage assumptions and fail in situations where there is no significant overlap between the goal distribution and the agent's state visitation distribution.  $fkl$ -PG does not depend on any such assumptions. (2)  $f$ -PG is considerably more stable than these baselines (as indicated by the variance of these methods).
6.4 Comparing different  $f$ -divergences
We perform an ablation to compare different  $f$ -divergences on their performances on the three Point Maze environments. Figure 6 (plotting mean and std-dev for 3 seeds) show that, empirically,  $fkl$ -PG performs the best followed by  $\chi^2$ -PG. Interestingly, both of these do not guarantee optimal policies
<sup>1</sup>Using spinning up implementation: https://spinningup.opennai.com/en/latest/_modules/spinup/algos/pytorch/ppo/ppo.html
Figure 5: (top): Description of the environments. In the PointMaze environments, the green and red shades represent the distributions from which the initial state and goal states are sampled. (bottom): Success rates (averaged over 100 episodes and 3 seeds) of  $fkl$ -PG, GAIL and AIM.  $fkl$ -PG outperforms these baselines with considerably lower variance.
Figure 6: Success rates (averaged over 100 episodes and 3 seeds) of  $f$ -PG for different  $f$ .  $fkl$ -PG performs the best followed by  $\chi^2$ -PG.
but it can be shown from Lemma 5.1 that  $fkl$ -PG converges to the policy that along with maximizing for a "reward", maximizes the entropy of the state-visitation. A similar result can be shown for  $\chi^2$  as well (proof in the Appendix A). This result can be explained by the need for exploration in the larger mazes, hence learning policies to keep the entropy of the state visitation high.
This paper derives a novel framework for goal-conditioned RL in the form of an on-policy algorithm  $f$ -policy gradients which minimizes the  $f$ -divergence between the agent's state visitation and the goal distribution. It proves that for certain  $f$ -divergences, we can recover the optimal policy while for some, we obtain a policy maximizing the entropy of the state-visitation. Entropy-regularized policy optimization (s-MaxEnt RL) for metric-based shaping rewards can be shown as a special case of  $f$ -PG where  $f$  is  $fkl$ .  $f$ -PG can provide an exploration bonus when the agent has yet not seen the goal. We demonstrated that  $f$ -PG can scale up to complex domains.
Through this work, we introduce a new perspective for goal-conditioned RL. By circumventing rewards,  $f$ -PG can avoid issues that arise with reward misspecification (Knox et al., 2021). There are several avenues to focus on for future work. First, the current framework is on-policy and poses an exploration challenge. An avenue for future work could be to develop an off-policy way to solve the objective. Second, this paper does not tackle goal distributions with several modes. Such a target distribution would be interesting to tackle in future work.
This work was in part supported by Cisco Research. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Cisco Research.
This work has partially taken place in the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (FAIN-2019844, NRT-2125858), ONR (N00014-18-2243), ARO (E2061621), Bosch, Lockheed Martin, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael L. Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. CoRR, abs/2111.00876, 2021. URL https://arxiv.org/abs/2111.00876.
Jose A. Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards, 2019.
Andrew G. Barto. Intrinsic Motivation and Reinforcement Learning, pp. 17-47. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013. ISBN 978-3-642-32375-1. doi: 10.1007/978-3-642-32375-1_2. URL https://doi.org/10.1007/978-3-642-32375-1_2.
Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation, 2016.
Boris Belousov and Jan Peters. f-divergence constrained policy improvement. CoRR, abs/1801.00056, 2018. URL http://arxiv.org/abs/1801.00056.
Boris Belousov and Jan Peters. Entropic regularization of markov decision processes. CoRR, abs/1907.04214, 2019. URL http://arxiv.org/abs/1907.04214.
Serena Booth, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. 2023.
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https://openai.com/research/faulty-reward-functions.
Ishan Durugkar, Maurizio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. CoRR, abs/2105.13345, 2021. URL https://arxiv.org/abs/2105.13345.
Ishan Durugkar et al. Estimation and control of visitation distributions for reinforcement learning. PhD thesis, 2023.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. CoRR, abs/2011.08909, 2020. URL https://arxiv.org/abs/2011.08909.
Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. CoRR, abs/2103.12656, 2021. URL https://arxiv.org/abs/2103.12656.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-53, 2022. doi: 10.1038/s41586-022-05172-4.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. CoRR, abs/1710.11248, 2017. URL http://arxiv.org/abs/1710.11248.
Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, and Sergey Levine. Ving: Learning open-world navigation with visual goals. CoRR, abs/2012.09812, 2020. URL https://arxiv.org/abs/2012.09812.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. URL http://arxiv.org/abs/1712.01815.
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. ISSN 0004-3702. doi: https://doi.org/10.1016/j.artint.2021.103535. URL https://www.sciencedirect.com/science/article/pii/S0004370221000862.
Satinder Singh, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010. doi: 10.1109/TAMD.2010.2051031.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.
Ahmed Touati, Amy Zhang, Joelle Pineau, and Pascal Vincent. Stable policy optimization via off-policy divergence regularization. CoRR, abs/2003.04108, 2020. URL https://arxiv.org/abs/2003.04108.
Ronald J Williams and Jing Peng. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241-268, 1991.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896): 223-228, 2022.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. CoRR, abs/1804.06459, 2018. URL http://arxiv.org/abs/1804.06459.
Brian D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Dieter Fox and Carla P. Gomes (eds.), AAAI, pp. 1433-1438. AAAI Press, 2008. ISBN 978-1-57735-368-3. URL http://dblp.uni-trier.de/db/conf/aaai/aaai2008.html#ZiebartMBD08.
A Analysis of  $J(\theta)$  16
A.1 Proof for Theorem 5.1 16
A.2 Proof for Lemma 5.1 17
B Gradient based optimization 18
B.1 Derivation of gradients 18
B.2 Practical Algorithm 20
B.3 Discounted State-Visitations 21
C Gridworld Experiments 21
C.1 Description of the task 21
C.2 Performance of  $f$  -PG 21
D Visualizing the learning signals 21
D.1 Description of the task 21
D.2 Comparing different  $f$  -PG 22
E PointMaze experiments 23
A Analysis of  $J(\theta)$
In this section, we will present the proofs for all the Lemmas and Theorems stated in Section 5.1.
A.1 Proof for Theorem 5.1
To prove Theorem 5.1, we need the following Lemmas. Lemma A.1 states that among all policies, the optimal policy has the highest state visitation at the goal.
Lemma A.1. Let  $\mathcal{D}$  be the set of all possible state visitations for the agent following some policy  $\pi \in \Pi$ . Let  $\pi^{*}$  be the optimal goal-conditioned policy. This optimal policy's state-visitation distribution will have the most measure at the goal for all  $p_{\pi} \in \mathcal{D}$  i.e.,  $\pi^{*} \Longrightarrow p_{\pi^{*}}(g) \geq p_{\pi}(g), \forall p_{\pi} \in \mathcal{D}$ .