Adversarial Intrinsic Motivation for Reinforcement Learning
Authors: Ishan Durugkar
Mauricio Tec
Scott Niekum
Peter Stone
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Intrinsic Motivation
Distribution Mismatch
Wasserstein distance

Abstract: Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein-1 distance between a policy's state visitation distribution and a target distribution, can be utilized effectively for reinforcement learning (RL) tasks. Specifically, this paper focuses on goal-conditioned reinforcement learning where the idealized (unachievable) target distribution has full measure at the goal. This paper introduces a quasimetric specific to Markov Decision Processes (MDPs) and uses this quasimetric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in as few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute a supplemental reward function. Our experiments show that this reward function changes smoothly with respect to transitions in the MDP and directs the agent's exploration to find the goal efficiently. Additionally, we combine AIM with Hindsight Experience Replay (HER) and show that the resulting algorithm accelerates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.

Adversarial Intrinsic Motivation for Reinforcement Learning
Ishan Durugkar
Mauricio Tec
Department of Computer Science The University of Texas Austin
Department of Statistics and Data Sciences
The University of Texas at Austin Austin, TX, USA 78703
Austin, TX, USA 78703
ishandecs www.us.............. edu
mauricicggter@ttexes edu
Scott Niekum
Peter Stone
Department of Computer Science
Department of Computer Science
The University of Texas at Austin
The University of Texas at Austin
Austin, TX, USA 78703
Austin, TX, USA 78703 and Sony AI
sniekum@cs wta. edu
pstonelcs www.tuteee.......... edu
Abstract
Learning with an objective to minimize the mismatch with a reference distribution has been shown to be useful for generative modeling and imitation learning. In this paper, we investigate whether one such objective, the Wasserstein- distance between policy's state visitation distribution and target distribution, utilized effectively for reinforcement learning (RI.) tasks. Specifically, this paper focuses coal-conditioned reinforcement learning where the idealized (unachiev- able) target distribution has full measure at the goal. This paper introduces a quasimetric specific Markov Decision Processes (MDPs) and uses this quasimet- ric to estimate the above Wasserstein-1 distance. It further shows that the policy that minimizes this Wasserstein-1 distance is the policy that reaches the goal in few steps as possible. Our approach, termed Adversarial Intrinsic Motivation (AIM), estimates this Wasserstein-1 distance through its dual objective and uses it to compute supplemental reward function. Our experiments show that this reward function changes smoothly with respect transitions in the MDP and directs the agent's 's exploration to find the goal efficiently. Additionally, we combine ATM with Hindsight Experience Replay (HER) and show that the resulting algorithm acceler- ates learning significantly on several simulated robotics tasks when compared to other rewards that encourage exploration or accelerate learning.
be
Introduction
Reinforcement Learning (RL) 774 deals with the problem of learning policy to accomplish a given task in an optimal manner. This task is typically communicated to the agent by means of a reward function. If the reward function is sparse [4] (e.g.. most transitions yield reward of 0), much random exploration might be needed before the agent experiences any signal relevant to learning [II 2]
Some of the different ways to speed up reinforeement learning by modifying augmenting the reward function are shaped rewards [52]. redistributed rewards 22, intrinsic motivations [8.69
72 57), and learned rewards [8] 54 Unfortunately, the optimal policy under such modified rewards might
71
sometimes be different than the optimal policy under the task reward 52
18 The problem of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

learning reward signal that speeds up learning by communicating what to do but does not interfere by specifying how to do it is thus a useful and complex one [82].
This work considers whether a task-dependent reward function learned based on the distribution mismatch between the agent's state visitation distribution and target task (expressed as distribution) can guide the agent towards accomplishing this task. Adversarial methods to minimize distribution
mismatch have been used with great success in generative modeling (29] and imitation learning
39
25
79 76 28]. In both these scenarios, the task is generally to minimize the mismatch with a
target distribution induced by the data or expert demonstrations. Instead, we consider the task of goal-conditioned RL. where the ideal target distribution assigns full measure to goal state. While the agent can never match this idealized target distribution perfectly unless starting at the goal, intuitively, minimizing the mismatch with this distribution should lead to trajectories that maximize the proportion of the time spent at the goal, thereby prioritizing transitions essential to doing so.
The theory of optimal transport 778] gives us a way to measure the distance between two distributions (called the Wasserstein distance) even if they have disjoint support. Previous work [3] 32 has shown how a neural network approximating a potential function may be used to estimate the Wasserstein-1 distance using its dual formulation, but assumes that the metric space this distance is calculated on is Euclidean. A Euclidean metric might not be the appropriate metric to use in more general RL tasks however, such as navigating in a maze or environments where the state features change sharply with transitions in the environment.
This paper introduces quasimetric tailored to Markov Decision Processes (MDPs), the time-step metric, to measure the Wasserstein distance between the agent's state visitation distribution and the idealized target distribution. While this time-step metric could be an informative reward on its own, estimating it is a problem as hard as policy evaluation [31]. Instead, we show that the dual objective, which maximizes difference in potentials while utilizing the structure of this quasimetric for the necessary regularization, can be optimized through sampled transitions.
We use this dual objective estimate the Wasserstein-1 1 distance and propose reward function based on this estimated distance. An agent that maximizes returns under this reward minimizes this Wasserstein-1 distance. The competing objectives of maximizing the difference in potentials for estimating the Wasserstein distance and minimizing it through reinforcement learning on the subsequent reward function leads
algorithm, Adversarial Intrinsic Motivation (AIM).
Our analysis shows that if the above Wasserstein-1 distance is computed using the time-step metric, then minimizing it leads to policy that reaches the goal in the minimum expected number of steps. It also shows that if the environment dynamics are deterministic, then this policy is the optimal policy,
In practice, minimizing the Wasserstein distance works well even when the environment dynamics are stochastic. Our experiments show that AIM learns reward function that changes smoothly with transitions in the environment. We further conduct experiments on the family of goal-conditions reinforcement learning problems [3] and show that AIM when used along with hindsight experience replay (HER) greatly accelerates learning of an effective coal-condtiioeed policy compared to learning with HER and the sparse task reward. Further, our experiments show that this acceleration similar to the acceleration observed by using the actual distance to the goal as dense reward.
2 Related Work
We highlight the related work based on the various aspects of learning that this work touches, namely intrinsic motivation, goal-conditioned reinforcement learning, and adversarial imitation learning.
2.1 Intrinsic Motivation
Intrinsic motivations R, 57 56 are rewards presented by an agent to itself in addition to the external task specifcci reward. Researchers have pointed out that such intrinsic motivations are a characteristic of naturally intelligent and curious agents [30] 5
6
Intrinsic motivation has been proposed as a
way to encourage RL agents to learn skills [10] 68
62)
that might be useful across a variety of
tasks, or as way to encourage exploration [11 67 24
23 The techniques that encourage general
exploration [6 58 have also been studied as inducing an artifical curiosity [66 65). The optimal
reward framework 69 72 and shaped rewards [52] (if generated by the agent itself) are way to

---

assist an RL agent in learning the optimal policy for given task. Such an intrinsically motivated reward signal has previously been learned through various methods such evolutionary techniques (54 (44, meta- meta-gradient approaches [71 81 82 and others. The Wasserstein distance has been used
to present valid reward for imitation learning 779 [9]
as well as program synthesis 27
2.2 Eoal-Conditioned Reinforcement Learning
43 can be considered form of multi-task reinforcement learning [17] where the agent is given the goal state needs to reach at the beginning of every episode. and the reward function is sparse with a non-zero reward only on reaching the goal state. UVFA 63 HER I. and others 80 20 consider this problem
condiiondi reinforcement learning
reaching certain states in the environment, Relevant to our work, Venkattaramanujam et al. 77 learns distance between states using a random walk that is then used to shape rewards and speed up learning, but requires goals to be visited before the distance estimate is useful. DisCo RL [51] extends the idea of goal-conditioned RL to distribution-conddtionee RL.
Contemporaneously, Eysenbach et al. 22 has proposed a method which considers goals and examples of success and tries to predict and maximize the likelihood of seeing those examples under the current policy and trajectory. For successful training, this approach needs the agent to actually experience the goals or successes. Their solution minimizes the Hellinger distance the goal. a form of f-divergence. ATM instead uses the Wasserstein distance which is theoretically more informative when considering distributions that are disjoint, and does not require the assumption that the agent has already reached the goal through random exploration. Our experiments in fact verify the hypothesis that AIM induces a form of directed exploration order to reach the goal.
2.3 Adversarial Imitation Learning and Minimizing Distribution Mismatch
28 has been shown to be an effective method to learn agent policies that minimize distribution mismatch between an agent's state-action visitation distribution and the -tate-action visitation distribution induced by expert's trajectories. In most cases this distribution that the expert induces is achievable by the agent and hence these techniques aim to match the expert distribution exactly. In the context of goal-conditioned reinforcement learning, GollGAIL 20 uses adversarial imitation learning with a few expert demonstrations to accelerate the learning of a goal-conditioned policy. In this work, we focus on unrealizable target distributions that cannot be completely matched by the agent, and indeed, are not induced by any trajectory distribution.
Adversarial imitation learning 39
25
79
76
FAIRL 28 is an adversarial imitation learning technique which minimizes the Forward KL diver- gence and has been shown experimentally to cover some hand-speeified state distributions, given smoothness regularization as used by WGAN (2) f-IRI. [53 learns a reward function where the optimal policy matches the expert distribution under the more general family of f-divergences. Further, techniques beyond imitation learning [46] 37 have looked at matching uniform distribution over states to guarantee efficient exploration.
Background
In this section first set up the goal-conditioned reinforcement learning problem, and then give brief overview of optimal transport.
3.1 Goal-Conditioned Reinforcement Learning
Consider goal-conditioned MDP as the tuple
with discrete state space S, discrete action space A, a subset of states which is the goal set 9 C S, and transition dynamics P:8 x A x 9 - A(S) (A(-) is distribution over set) which might vary based on the goal (see below). Po : A(S) is the starting state distribution, and a (9)) is the distribution a goal is drawn from. 7 j0, 1) is the discount factor. We use discrete states and actions for ease of exposition, but our idea extends to continuous states and actions, as seen in the experiments.
At the beginning of an episode, the starting state is drawn from po and the goal for that episode drawn from o. The reward function r : x
x 9 -_ IR. deterministic, and r(s,, ae st+1|s,): 1[s sg]. That is, there is a positive reward when agent reaches the
3

---

goal ($t+1 sy), and O everywhere else. Since the goal is given to the agent at the beginning of the episode, in goal-conditione RL the agent knows what this task reward function is (unlike the more general RI. problem). The transition dynamics are goal-conditioned as well, with automatic transition to an absorbing state on reaching the goal 8y and then staying in that state with no rewards thereafter (P (P(5)sp: 0, 8g) = 1 V E A and P(0)s, a, 8g) 1 A a A). In short, the episode terminates on reaching the goal state.
The agent takes actions in this environment based on a policy т E II - ((A). The return Hg for an episode with goal 8g is the discounted cumulative reward over that episode Hg 2t=0 ("r(st, Qe. S1+1 Sgy) where So ~ Po- as x[-|st, Sg), and st+1 ~ P(-|st, Qt, sy). The agent aims to find the policy π arg max bo [Hg] that maximizes the expected returns in this conddiioit MDP. For a policy т. the agent's 's coal-conditioned state distribution P- (s(8)) = Expr-po [11 - 7) Ei= P(s, ? = s|m 8g)]- Overloading the terminology a bit, we also define the goal-conditioned target distribution (g(8)))) 0(8g), Dirac measure at the goal state 8y.
While learning using traditional RL paradigms is possible in goal-conditioned RL, there has also been previous work (Section 2.2 on leveraging the structure of the problem across goals. Hindsight Experience Replay (IER) attempts speed up learning in this sparse reward setting by taking episodes of agent interactions, where they might not have reached the goal specified for that episode, and relabeling the transitions with the goals that were achieved during the episode. Off- off-policy learning algorithms are then used to learn from this relabeled experience.
3.2 Optimal Transport and Wasserstein-1 Distance
The theory of optimal transport 778 [4] considers the question of how much work must be done to transport one distribution to another optimally, where this notion of work is defined by the use of a ground metric d. More concretely, consider metric space (M, d) where M is set and d is metric on M (Definitions in Appendix (A). For two distributions f and v with finite moments on the set M, the Wasserstein-p distance is denoted by:
W,(,, inf Exx,))~cl(( [d( ()pp// (EZ(U.)
(1)
where Z is the space of all possible couplings, i.e. joint distributions ( E (MxM) whose marginals are μ and y respectively. Finding this optimal coupling tells us what is the least amount of work, as measured by d, that needs to be done to convert to v. This Wasserstein-P distance can then be used
as a cost function (negative reward) by an RL agent to match a given target distribution 77 19 27
Finding the ideal coupling above is generally considered in- tractable. However, if what we need is an accurate estimate of the Wasserstein distance and not the optimal transport plan we can turn our attention the dual form of the Wasserstein-1 distance. The
duality [78 (5) for the Wasserstein-1 distance (which we refer to simply the Wasserstein distance hereafter) on ground metric d gives us:
Wi(H,v) = sup Ey [f(()] - -Eey [f() Lip(f)S1
(2)
where the supremum is over all 1-Lipschitz functions f M R. in the metric space. Importantly, Jevtić [41] has recently shown that this dual formulation extends to quasimetric spaces as well. More details, such as the definition of the Lipschitz constant of function and special cases used in WGAN [3 32 28 are elaborated in Appendix B Note that the Lipschitz constant of the potential function is computed based the ground metric d.
Figure 1:
Grid world example
4 Time-Step Metric
The choice of the ground metric d is important when computing the Wasserstein distance between two distributions. That is, if we want the Wasserstein distance to give estimate of the work needed to transport the agent's i state visitation distribution to the goal state, the ground metric should incorporate notion of this work.

---

Consider the grid world shown in Figure II where wall (bold line) marks an impassable barrier in part of the state space. If the states are specified by their Cartesian coordinates on the grid, the Manhattan distance between the states specified by the blue and red circles is not representative of the optimal cost to go from one to the other. This mismatch would lead to an underestimation of the work involved if the two distributions compared were concentrated at those two circles. Similarly, there will be errors in estimating the Wasserstein distance if the grid world is toroidal (where an agent is transported to the opposite side of the grid if it walks off one side) or if the transitions are asymmetric (windy grid world (74)).
To estimate the work needed to transport measure in an MDP when executing policy म, we consider a quasimetric metric that does not need be symmetric dependent on the number of transitions experienced before reaching the goal when executing that policy.
Definition 1.
The time-step metric di in an MDP with state space 8, action space A, transition function P, and agent policy π is a quasimetric where the distance from state E S to state 8g E 8 is bused on the expected number of transitions under policy n.
di sy): [T()
where T( 5) is the random variable for the first time-step that state 8y encountered by the agent after starting in state s and following policy x.
This quasimetric has the property that the per step cost is uniformly 1 for all transitions except ones from the goal to the absorbing state (and the absorbing state to itself), which are 0. Thus, it can be written recursively as:
00 {1 +1+Ea<T (( Esr- P((s.,a))))) (s', sy)]
if = 89
(F 8y)
otherwise
Recall that in order to estimate the Wasserstein distance using the dual (Equation 2) in a metric space where the ground metric d is this time-step metric, the potential function f -ー IR needs to be 1- 1-Lipschitz with respect to d7.. In In[Appendix C we prove that L-Lipschitz continuity can be ensured by enforeing that the difference in values of on expected transitions from every state are bounded by L, implying
Lip(J) S sup sEB (~xx(-18,,,) s s'~ P((e,00)))))
E
EE
[I[(s) - s('))
(4)
Note that finding proper way to enforce the Lipschitz constraint in adversarial methods remains an open problem [49]. However, for the time-step metric considered here, equation 4 is one elegant way of doing so. By ensuring that the Kantorovich potentials do not drift too far from each other on expected transitions under agent policy π in the MDP, the conditions necessary for the potential function to estimate the Wasserstein distance can be maintained 778 3.. Finally, the minimum distance d from state to given goal state 8g (corresponding to policy π π*) is defined by the Bellman optimality condition (Equation 16 in Appendix D).
Consider how the time-step distance to the goal and the value function for goal- coal-conditioned RL relate to each other. When the reward is O everywhere except for transitions to the goal state, the value becomes V (T(ss))) = EE [~TT(sa m..)]. dFf sg) and V (solsg) are related as follows.
Proposition 1. A lower bound on the value of any state under a policy n can be expressed in terms of the time-step distance from that state the goal: V(8olsg) 2 1777
2 (e0:8)))
The proofs for all theoretical results are in Appendix D y (s.sg) describes the sharpness of the lower bound in the proposition above and is zero if and [7]. From this line of reasoning, we deduce the following proposition: Proposition 2. If the transition dynamics are deterministic, the policy that maximizes expected return is the policy that minimizes the time-step metric (x* )
The Jensen gap (eessn()) = (s(sg)
only if Var(T (s)
5 Wasserstein-1 Distance for
Reinforcement Learning
In this section we consider the problem of goal-conditioned reinforcement learning. In Section 5.1 we analyze the Wasserstein distance computed under the time-step metric di. Section 5.2 proposes an
5

---

algorithm, Adversarial Intrinsic Motivation (AIM), to learn the potential function for the Kantorovich- Rubinstein duality used estimate the Wasserstein distance, and giving an intrinsic reward function used to update the agent policy in tandem.
5.1
Wasserstein-1 Distance under the Time-Step Metric
From Sections 3.2 and the Wasserstein distance under the time-step metric di of an agent policy with visitation measure Pn to particular goal $y and its distribution Py can be expressed as:
W (Pm, Pg) = E (s]88) (d7-(s,, 9g)
(5)
where WT refers to the Wasserstein distance with the ground metric dj.
The following proposition shows that the Wasserstein distance decreases as (8, 8g) decreases, while also revealing a surprising connection with the Jensen gap.
Proposition 3. For given policy the Wasserstein distance of the state visitation measure of that policy from the goal state distribution Pg under the ground metric di. can be written as
WT (P::
2 (dF 1- ~ - Tensen (80) - 1)
(6)
som-po
where h is increasing function of dž-
The first component the above analytical expression shows that the Wasserstein distance depends on the expected number of steps, decreasing if the expected distance decreases. The second component shows the risk-averse nature of the Wasserstein distance. Concretely, the bounds for the Jensen inequality given by Liao and Berg [47] imply that there are non-negative constants C₁ Ci (d) 7) and C2- = (277 7) depending only on the expected distance and discount factor such that
Ci Var(T s)) S Tonsen (s) S C2 Var(T 8)).
From the above, we can deduce that a policy with lower variance will have lower Wasserstein distance when compared to policy with same expected distance from the start but higher variance. The relation between the optimal policy in goal gonl-conditioned RL and the Wasserstein distance can be made concrete if we consider deterministic dynamics.
Theorem 1. If the transition dynamics are deterministic, the policy that minimizes the Wasserstein distance over the time-step metrics in goal-conditioned MDP (see equation 3 is the optimal policy.
5.2 Adversarial Intrinsic Motivation to minimize Wasserstein-1 Distance
The above section makes clear that minimizing the Wasserstein distance the goal will lead to a policy that reaches the goal in as few steps as possible in expectation. If the dynamics of the MDP are deterministic, this policy will also be optimal. Note that the dual form (Equation 2) can be used to estimate the distance, even if the ground metric di is not known. The smoothness requirement on the potential function f can be ensured with the constraint Equation 4 on all states and subsequent transitions expected under the agent policy.
Now consider the full problem. The reinforcement learning algorithm aims to learn goal-conditioned policy with parameters 0 E θ whose state visitation distribution Pe minimizes the Wasserstein distance goal-conditioned target distribution Pg for given goal Sy a. AIM leverages the presence of the set of goals that the agent should be capable of reaching with goal-conditioned potential function fo
b R with parameters á E &. These objectives of the potential function and the agent can be expressed together using the following adversarial objective:
min max leee ΦED ~a
fo(sg.
E
[8)(s)
orup
where the potential function fo is 1-Lipschitz over the state space. Combining the objectives in Equations 7 andd 4 the loss for the potential function for then becomes:
Sal 87) Ly: ~y
[o sw)] smpp
A
E 19,)))
[(max([fo(s, (g) - (o(s:,)))l-,,0)
(8)

---

Where the distribution 'D should ideally contain all states in 8, expected goals in G, and the transitions according to the agent policy To and transition function P. Such distribution is difficult to obtain directly. ATM approximates it with a small replay buffer of transitions from recent episodes experienced by the agent, and relabels these episodes with achieved goals (similar to HER (I). Such an approximation does not respect the discounted measure of states later on in an episode, but is consistent with how other approaches in deep reinforcement learning tend to approximate the state visitation distribution, especially for policy gradient approaches [55]. While it does not include all states and all goals, we see empirically that the above approximation works well.
Now we turn to the reward function that should be presented to the agent to minimize the Wasserstein distance. The Wasserstein discriminator a potential function [52] (its value depends on the state). can thus be used to create shaped reward F(s, a, s' 8g) = r(s, a, s'|ss)) + "ff((s' sg)) (s, sg) without risk of changing the optimal policy. Alternatively, we can explicitly minimize samples of the Wasserstein distance: F(s, a, 8', 8g) fa(s', f (a(gg 8y); we can just use constant bias term. practice, all these choices work well, and the experiments use the latter (with max.es fa(s,5,) 8g)) reduce variance in P.
f0((g 8g). Finally, instead of the second term
i(s, a, kg) = ss .8)) b
The basic procedure to learn and use adversarial intrinsic motivation (AIM) is laid out in Algorithm D and also includes how use this algorithm in conjunction with HER. If not using HER, Line 23 where hindsight goals are added to the replay buffer can be skipped.
6 Experiments
Our experiments evaluate the extent to which the reward learned through AIM is useful as a proxy for the environment reward signal, or tandem with the environment reward signal. In particular, we ask the following questions:
Does AIM speed up learning of a policy get a single goal compared to learning with a sparse reward?
Does the learned reward function qualitatively guide the agent to the goal?
Does AIM work well with stochastic transition dynamics or sharp changes in the state features?
Does AIM generalize a large set of goals and continuous state and action spaces?
Our experiments suggest that the answer to all 4 questions is "yes", with the first three questions tested in the grid world presented in Figure where the goal is within room, and the agent has to go around the room from its start state to reach the goal. coal-conditiooed tasks in the established Fetch robot domain show that AIM also accelerates learning across multiple goals in continuous state and action spaces.
This section compares agent learning with reward learned through AIM with other intrinsic motivation signals that induce general exploration or shaped rewards that try to guide the agent to the goal. The experiments show that AIM guides the agent's exploration more efficiently and effectively than a general exploration bonus, and adapts to the dynamics of the environment better than other techniques we compare to. As overview, the baselines compare are:
RND: with random network distillation (RND) [66] used to provide a general exploration bonus. MC: with the distance between states learned through regression of Monte Carlo rollouts of the agent policy, similar to Hartikainen et al. (35].
SMiRL: SMiRL
13 is used to provide a bonus intrinsic motivation reward that minimizes the
overall surprise in an episode.
DiscoRL The DiscoRL [51] approach presents a reward to maximize the likelihood of target distribution (normal distribution at the goal). In practice this approach is equivalent to a negative
1.2 distance the goal, which we compare to in the grid world domain.
GAII: additional GAII. 39 rewards using trajectories relabeled with achieved goals considered
having come from the expert in hindsight. This baseline is compared to in the Fetch robot domain, since that is the domain where we utilize hindsight relabeling.
Grid World In this task, the goal is inside room and the agent's 's starting position is such that needs to navigate around the room to find the doorway and be able to reach the goal. The agent can
7

---

move in the 4 cardinal directions unless blocked by wall or the edge of the grid. The agent policy is learned using soft Q-learning [33 with no hindsight goals used for this experiment.
The agent's s state visitation distribution after just 100 Q-function updates when using ATM-learned rewards is shown in Figure 2a and the learned rewards for each state are plotted in Figure 2b The state visitation when learning with the true task reward shows that the agent is unable to learn a policy the goal (Figure 2c). These figures show that AIM enables the agent reach the goal and learn the required policy quickly, while learning with the sparse task reward fails to do so.
In Appendix F we also compare the baselines described above and show that AIM learns reward that is more efficient at directing the agent's 's exploration and more flexible to variations of the environment dynamics, such as stochastic dynamics transitions that cause sharp change in the state features. None of the baselines compared to were able to direct the agent to the goal in this grid world even after given up to 5x more interactions with the environment to train. AIM'S of the time-step metric also enabled to adapt to variations of the environment dynamics better than the gradient penalty based regularization used in Wasserstein GANs [32] and adversarial imitation learning [28] approaches.
Fetch Robot The generalization capability of AIM across multiple goals in goal-conditioned RL tasks with continuous states and actions is tested in the MuJoCo simulator [5], on the Fetch robot tasks from OpenAl gym [[5] which have been used to evaluate learning of goal-conditioned policies previously 80) Descriptions of these tasks and their goal space is in Appendix [H] We soften the Dirac target distribution for continuous states to instead be Gaussian with variance of 0.01 of the range of each feature.
The goals in this setting are not the full state, but rather the dimensions of factored states relevant to the given goal. The task wrapper additionally returns the features of the agent's state in this reduced goal space, and so AIM can use it to learn our reward function, rather than the full state space. It unelear how this smaller goal space might affect AIM. While the smaller goal space might make learning easier for potential function fa. the partially observable nature of the goals might lead to less informative reward.
We combine AIM with HER (refer (subsectio 3.1) and refer to as [AIM + HER]. We compare this agent the baselines referred above, as well the sparse environment reward (R HER) and the dense reward derived from the negative Euclidean (L2) distance to the goal (-22 + HER). The L2 distance proportional to the number of steps should take the agent to reach the goal in this environment, and so the reward based on it can act as an oracle reward that we can use to test how efficiently AIM can learn reward function that helps the agent learn its policy. We used the HER implementation using Twin Delayed DDPG (TD3) 26 as the underlying RL. algorithm from the stable baselines repository [38] We did an extensive sweep of the hyperparameters for the baseline HER + (laid out in Appendix with a coarser search on relevant hyperparameters for AIM
target
Sager:
agert
-44
-06
(a) Agent state visitation learning with AIM rewards (100 iterations)
(b) Learned Reward (100 training itcrations)
(c) Agent state visitation learning with task reward (500 iterations)
Figure 2: Grid world experiments. Agent's 's undiscounted state visitation 2a
2ch:
Blue circle
indicates agent's 's start state. Red circle is the goal. Blue bubbles indicate relative time agent's 's policy causes to spend in respective states. Learned reward function (2b): AIM reward at each state of the grid world. Bold black (or white) lines indicate walls the agent cannot transition through.
8

---

-10
-20
0.6
-30
AIM + HER
+ + HER
AND
-40
GAIL R. HER
R HER
0.0
10000
25000
0.0
0.2
0.6
0.8
1.0
1.2
1.4
tmestops
le6
(a) Reach
(b) Pick and Place
0.8
0.8
Episode Return
0.6
0.6
0.4
0.4
0.0
0.2
0.4
0.8
1.0
1.2
1.4
0.0
0.6
0.8
1.4
timesteps
(c) Push
(d) Slide
Figure 3: Evaluating AIM with HER on some goal-conditioned RL tasks in the Fetch domain. AIM learns the reward function in tandem with the policy updates. The L--2" reward the negative distance to goal, acting as oracle reward in this domain. The other baselines are detailed above.
Figure 3 shows that using the AIM-learned reward speeds up learning in three of the four Fetch domains, even without the environment reward. This improvement is very close what would see if we used the dense reward (based on the actual distance function in this domain). An additional comparison with an agent learning with both the AIM-learned reward as well as the task reward (AIM + R HER) can be seen in Figure 7in in the Appendix, showing that using both signals accelerates learning even more. These results also highlight that continues work in continuous state and action spaces, even though our analysis focuses on discrete states and actions. Results are averaged across the different seeds, with shaded regions showing standard error across runs. Statistical analysis using mixed effects ANOVA and a Tukey test at significance level of 95% (more detail Appendixx G) show that three of the four environments AIM and AIM+ R have similar odds of reaching the goal as the dense shaped reward, and in all four environments AIM and AIM+ have higher odds of reaching the goal compared the sparse reward.
The other baselines compare well to AIM in the Fetch Reach domain (Figure 3a but do not do as well on the other problems. fact, none of the other baselines outperform the vanilla baseline [R + HER] in all the domains. The RND rewards help the agent to start learning faster in the Push domain (Figure 3c). but lead worse performance in Pick and Place (Figure 3b). On the other hand, learning the distance function through MC regression helps in the Pick and Place domain, but slows down learning when dealing with Push. Most notably, both these approaches cause learning to fail in the Slide domain (Figure (3]), where the credit assignment problem is especially difficult. GAIL works as well as ATM and the vanilla baseline in Slide, but underperforms in the other domains. We hypothesize that the additional rewards in these baselines conffict with the task reward. Additionally, none of the three new baselines work well if we do not provide the task reward in addition to the specific bonus for that algorithm.

---

We did not find any configuration in the Fetch Reach domain where [SMiRL + R + HER] was able to accomplish the task the given training budget. Since SMiRL did not work on the grid world or Fetch Reach, we did not try it out on any of the other domains.
FAIRI. 28 (which has been shown to learn policies that cover hand-specified state distributions) was also applied on these 4 domains but it failed to learn at all. Interestingly, scaling the reward such that it is always negative led to similar performance to (but not better than) AIM. We hypothesize that FAIRL, as defined and presented, fails in these domains because the environments are episodic, and the episode ends earlier the goal is reached. Since the FAIRL reward is positive closer to the target distribution, the agent can get close to the target, but refrain from reaching it (and ending the episode) to collect additional positive reward.
The domain where AIM does not seem to have a large advantage (Slide) is one where the agent strikes an object initially and that object has come to rest near the goal. In fact, ATM-learned rewards, the vanilla environment reward R, and the oracle - L.2 rewards all lead to similar learning behavior, indicating that this particular task does not benefit much from shaped rewards. The reason for this invariance might be that credit assignment has to propagate back to the single point when the agent strikes the object regardless of how dense the subsequent reward is.
7 Discussion and Future Work
Approaches for estimating the Wasserstein distance to target distribution by considering the dual of the Kantorovich relaxation have been previously proposed 3 32 79., but assume that the ground metric is the L2 distance. We improve upon them by choosing metric space more suited to the MDP and notions of optimality in the MDP. This choice allows us to leverage the structure introduced by the dynamics of the MDP to regularize the Kantorovich potential using novel objective.
Previous work 12] has pointed out that the gradients from sample estimates of the Wasserstein distance might be biased. This issue is mitigated in our implementation through multiple updates of the discriminator, which they found be empirically useful in reducing the bias. Additionally, recent work has pointed out that the discriminator in WGAN might be bad at estimating the Wasserstein dis- tance 73]. While our experiments indicate that the potential function in AIM learned appropriately. future work could look more deeply to verify possible inefficiencies in this estimation.
The process of learning the Wasserstein distance through samples of the environment while simultane- ously estimating the cost of the full path is reminiscent of the A" algorithm (34), where the optimistic heuristic encourages the agent explore in directed manner, and adjusts its estimates based on these explorations.
The discriminator objective (Equation 8) also bears some resemblance to linear program formulation of the RL problem [60]. The difference is that this formulation minimizes the value function on states visited by the agent, while AIM additionally maximizes the potential at the goal state. This crucial difference has two main consequences. First, the potential function during learning is not equivalent to the value of the agent's 's policy (verified by using this potential critic). Second, increasing the potential of the goal state in AIM directs the agent exploration in a particular direction (namely, the direction of sharpest increase in potential).
In the goal-conditioned RL setting, AIM seems to be an effective intrinsic reward that balances exploration and exploitation for the task at hand. The next step is to consider whether the Wasserstein distance can be estimated similarly for more general tasks, and whether minimizing this distance in those tasks leads to the optimal policy. A different potential avenue for future work is the problem of more general exploration (37 46) by specifying a uniform distribution as the target, or using this directed exploration an intermediate step for efficient exploration 42
Finally, reward design an important aspect of practical reinforcement learning. Not only do properly shaped reward speed up learning [52), but reward design can also subtly influence the kinds of behaviors deemed acceptable for the RL agent (55 and could be a potential safety issue keeping reinforcement learning from being deployed on real world problems. Learning-basedd approaches that can assist in specifying reward functions safely given alternative approaches for communicating the task could be of value in such a process of reward design, and an avenue for future research.
10

---

Acknowledgements and Funding Information
We thank Caroline Wang, Garrett Warnell, and Elad Liehman for discussion and feedback on this work. We also thank the reviewers for their thoughtful comments and suggestions that have helped to improve this paper.
This work has taken place in part in the Learning Agents Research Group (LARG) at the Artificial Intelligence Laboratory, and in part in the Personal Autonomous Robotics Lab (PeARL) at The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (CPS-1739964, IIS- 172-172,,,, Anmy Research Office (W91 INF- 191-F--992--00 19-2-0333), DARPA, Lockheed Martin, General Motors, Bosch, and Good Systems, research grand challenge at the University of Texas Austin. PeARL research is supported in part by the NSF UIS-1724157, IIS-1638107 (6638107,10 IIS-1749204, (IS-1925082) ONR AFOSR (FA9550-20-1-0077) and ARO (78372-CS). This research also sponsored by the Army Research Office under Cooperative Agreement Number W911NF--1-22 19-2-0333. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.
the Office of Naval Research (N00014-18-2244
100014-18-22430
References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. arXiv: 1707.01495 [cs], February 2018. URL httpp://arx...oor//b/////// arXiv: 1707.01495.
[2] Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In
Advances in Neural Information Processing Systems, pages 13566-13577, 2019.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks, In International Conference on Machine Learning. pages 214-223, July 2017. URL http://proceeding
ISSN: 2640-3498
Section: Machine Learning.
[4] Dilip Arumugam, P. Henderson, and P. Bacon. An information-thhorrtiicc perspective credit assignment in reinforcement learning. ArXiv, abs/2103.06224, 2021.
[5] Gianluca Baldassarre. What intrinsic motivations? a biological perspective. In 2011 IEEE international conference on development and learning (ICDL), volume 2, pages 1-8. IEEE, 2011.
[6] Gianluca Baldassarre, Tom Stafford, Marco Mirolli, Peter Redgrave, Richard M Ryan, and Andrew Barto. Intrinsic motivations and open-ended development animals, humans, and robots: an overview. Frontiers in psychology. 5:985, 2014.
[7] Adrien Baranes and Pierre-Yves Oudeyer. R-iae: Robust intrinsically motivated exploration and active learning. IEEE Transactions on Autonomous Mental Development, 1(3): 153-1155169, 2009.
[8] Andrew G. Barto. Intrinsic Motivation and Reinforcement Learning. In Gianluca Baldassarre and Marco Mirolli, editors, Intrinsically Motivated Learning in Natural and Artificial Systems, pages 17-47. Springer Berlin Heidelberg, Berlin, Heidelberg. 2013. ISBN 978-3-648------ 978-3-642-32375. doi:
http:ttp:///in/ httpt:::///ikkkkiiiiie.. springer
comm 22
[9] Andrew G Barto and Ozgür Simsek. Intrinsic motivation for reinforcement learning systems. In Proceedings of the Thirteenth Yale Workshop on Adaptive and Learning Systems, pages 113-118, 2005.
[10] Andrew G Barto, Satinder Singh, and Nuttapong Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of the 3rd International Conference on Development and Learning, pages 1 112-19. Cambridge, MA, 2004.
11

---

[11] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances neural information processing systems, pages 1471-1479, 2016.
[12] Marc G. Bellemare, Ivo Danihelka, Will Dabney, S. Mohamed, Balaji Lakshminarayanan,
Stephan Hoyer, and R. Munos. The cramer distance a solution to biased wasserstein gradients. URL BtRLhttts:://// http:tt:t........rorpp//
ArXiv, abs/1705. 10743, 2017.
/ttpa//www..//b//////aa///// 10743
[13] Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing lenaronn in unstable environments. arXiv preprint arXiv: 1912.055 2019.
[14] Olivier Bousquet. Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Gaboil, and Bern- hard Schoelkopf. From optimal transport to generative modeling: the VEGAN cookbook. 07642
arXiv: 07642 [stat], May 2017. URL
arXiv: 1705.07642.
[15] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint (606://606.1111 2016.
[16] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
[17] Rich Caruana. Multitask learning. Machine learning, 28(1):41-77) 1997.
[18] Jack Clark and Dario Amodei. Faulty reward functions in the wild, Dec 2016. URL https
/ //openai. www/blcccooooooooo faw.eaaaaaaeeeaeaeeeeeetiooo.aa.
[19] Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal Wasserstein
Imitation Learning. arXiv:2006. 04678 [cs, stat], June 2020.
abs/200s 04678 arXiv: 2006.04678
[20] Y. Ding, Carlos Florensa, Mariano Phiclipp, and P. Abbeel. Goal-conditioned imitation learning.
In NeurlPS, 2019.
[21] Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhutdinov. Replacing rewards with exam- ples: Example-based policy search via recursive classification. arXiv preprint MirXxi::10.... 12656, 2021.
[22] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to
achieve goals via recursive classification. In International Conference on Learning Representa- tions, 2021.
fest//forumiinn cerrummiiitttiiiiiii. cot.sssiissBB-B
[23] Yannis Flet- Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist. Adver- sarially guided actor-critic. In ICLR 2021-Internationn Conference on Learning Representa- tions, 2021.
[24) Sébastien Forestier, Rémy Portelas, Yoan Mollard, and Pierre- Yves Oudeyer. Intrinsically motivated goal exploration processes with automatic curriculum learning. arXiv preprint 1708 /900002990000 190, 2017.
[25] Justin Fu, Katie Luo, and Sergey Levine. Learning Robust Rewards with Adversarial Inverse Reinforcement Learning. arXiv:17100 11248 August 2018. arXiv: 1710. 12248.
abs/1710. 11248
[26] Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018
[27] Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthe- sizing programs for images using reinforeed adversarial learning. In International Conference on Machine Learning, pages 1666-1675. PMLR, 2018
[28] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence mini- mization perspective on imitation learning methods. In Conference on Robol Learning, pages
1259-1277. PMLR, 2020.
[29] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural In- formation Processing Systems 27, pages 2672 2680. Curran Associates, Inc., 2014. URL http: nets pdf
7/papers ottp::/cccppppee/aaaaaaaaaaeeaee
12

---

[30] Jacqueline Gottlieb, Pieree Yves Oudeyer, Manuel Lopes, and Adrien Baranes. Information- seeking, curiosity, and attention: computational and neural mechanisms. Trends in cognitive sciences, 17( (((11):585-93) 2013.
[31]
Matthieu Guillot and Gautier Stauffer. The stochastic shortest path problem: a polyhedral combinatorics perspective. European Journal of Operational Research, 285(1):: 148-1188 2020.
[32]
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved Training of Wasserstein GANs. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5767-5777. Curran Associates, Inc., 2017. URL http::ttttpppp.teeee
nips otw.aainnnno---wassseeeeeeeeeeee www.wrsteeeeeeeeeeeeeeeeeeeeeeee gans pdf
CC oc/paper/7159
[33] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1352-1361, 2017.
[34] Peter Hart, Nils J Nilsson, and Bertram Raphael. formal basis for the heuristic determination of minimum cost paths. IEEE transactions Systems Science and Cybernetics, 4(2):100-107, 1968.
[35] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. In International Conference on Learning Representations, 2020.
[36] Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action
space. In International Conference on Learning Representations, 2016. URL https://
arxiv. orw. abs/ www.a////............. 04143)
[37] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691.
PMLR, 2019.
[38] Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene
Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https? 2018.
//github. www.ccccmll...e...ee..e..eeeeeea http/hilllaaaaa lit-aa/stallebbaeeeee
[39]
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29:4565 4573, 2016.
[40] Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal Regret Bounds for Reinforcement Learning. Journal of Machine Learning Research, 1 11511)15633-1600 2010. ISSN 1533-7928. URL http::/ttmm: org/ pwwppss/vl/spaeeeepppppppp
html
[41]
Filip Jevtić. Combinatorial Structure of Finite Metric Spaces. PhD thesis, The University of Texas at Dallas, August 2018
[42] Yuu Jinnai, Jee Won Park, Marlos C. Machado, and George Konidaris. Exploration in re- inforcement learning with deep covering options. In International Conference on Learning Representations, 2020.
wttt:/forumiidddd0 IyattWe Citeseer, 1993. [44] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209-22 2220, 2002.
[43] Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, pages
[45] W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. arXiv preprint arXiv:2 13906, 2021.
[46] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Rus- lan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint
19X6v21906.05244, 2019.
[47] JG Liao and Arthur Berg. Sharpening 's inequality. The American Statistician, 2018.
[48] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[49] Kanglin Liu and Guoping Qiu. Lipschitz constrained gans via boundedness and continuity. Neural Computing and Applications, pages 1-13, 2020.
13

---

[50] Douglas C Montgomery. Design and analysis of experiments. John wiley & sons, 2017.
[51] Soroush Nasiriany. Disco rl: Distribution-conditionne reinforcement learning for general- purpose policies. Master's thesis, EECS Department, University of California, Berke- ley. Aug 2020. UtLhttp://wwae eecs. .er.sbebrkee wew.eeee.....uu........... http://uuub//ecc//s////2200000
html
[52] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In ICMI,, volume 99, pages 278-287, 1999.
[53] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Benjamin Eysenbach. F-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, 2020.
[54) Scott Niekum. Evolved intrinsic reward functions for reinforcement learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, pages 1955-1956, 2010.
[55] Chris Nota and Philip Thomas. the policy gradient gradient? In International Conference on Autonomous Agents and Multi-Agent Systems, pages 939-947, 2020.
[56] Pierre-Yve Yves Oudeyer and Frederic Kaplan. How can we define intrinsic motivation? In the 8th International Conference on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems. Lund University Cognitive Studies, Lund: LUCS, Brighton, 2008.
[57] Pierre- Oudeyer and Frederic Kaplan. What intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:6, 2009.
(58) Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity exploration by self-supervised prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[59] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6)3555 607, 2019. URL http: /wtpaaarvvv..............eea.a.s http//aaiii./aa/....aaaaa
1803 orq. a803
00567
[60] Martin Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
[61] Antonin Raffin.
RI baselines ZOO.
https:///gttub.com araffin/
r1 sl-baselines--an
2018
[62] Vieri Giuliano Santucci, Gianluca Baldassarre, and Marco Mirolli. Which is the best intrinsic motivation signal for learning multiple skills? Frontiers in neurorobotics, 7:22, 2013.
[63] T. Schaul, Daniel Horgan, K. Gregor, and D. Silver. Universal value function approximators. In ICML, 2015.
[64] Massimiliano Schembri, Marco Mirolli, and Gianluca Baldassarre. Evolving internal reinforcers for an intrinsically motivated reinforeementleamming robot. In 2007 IEEE 6th International Conference on Development and Learning, pages 282-287. IEEE, 2007.
[65] Jürgen Schmidhuber. Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458-1463, 1991.
[66] Jürgen Schmidhuber. possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behuvior: From animals to animats, pages 222-227, 1991.
[67] Özgür Şimşek and Andrew G Barto. An intrinsic reward mechanism for efficient exploration. In Proceedings of the 23rd international conference on Machine learning, pages 333-840, 2006.
[68] Satinder Singh, Andrew G Barto, and Nuttapong Chentanez. Intrinsically motivated rein- forcement learning. In Advances in neural information processing systems, pages 1281-1288, 2005.
[69] Satinder Singh, Richard L. Lewis, Andrew G. Barto, and Jonathan Sorg. Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective. IEEE Transactions Autonomous Mental Development 2(2))70-82, June 2010. ISSN 1943-0612. doi: 10.1 19//TAMDD...00... 2051031. Conference Name: IEEE Transactions Autonomous Mental Development.
[70] M. Smyth. Quasi uniformities: Reconciling domains with metric spaces. In MFPS, 1987.
14

---

[71] Jonathan Sorg, Richard L Lewis, and Satinder P Singh. Reward design via online gradient ascent. In Advances in Neural Information Processing Systems, pages 2190-2198, 2010.
[72] Jonathan Sorg, Satinder P Singh, and Richard Lewis. Internal rewards mitigate agent boundedness. In Proceedings of the 27th international conference on machine learning (ICML- 10), pages 1007-1014, 2010.
[73] Jan Stanczuk. Christian Etmann, L. Kreusser, and C. Schönlieb. Wasserstein gans work because they fail (to approximate the wasserstein distance). ArXiv, abs/2103.01677 2021.
[74] Richard Sutton and Andrew G Barto, Reinforcement learning: An introduction. MIT press, 2018.
[75] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 TEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[76] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative Adversarial Imitation from Obser- vation. 18Xiv:1807 00000006688888 [cs, stat], June 2019. URL http://arxiv......///// org. aas//b////// 1807 06158
Xrv:
[77] Srinivas Yenkattaramanujam, Crawford, T. Doan, and Doina Precup. Self-supervised learning of distance functions for goal-conditioned reinforcement learning. ArXiv, 0hss/190.009999 2019,
[78] Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008.
[79] Huang Xiao, Michael Herman, Joerg Wagner, Sebastian Ziesche, Jalal Etesami, and Thai Hong Linh. Wasserstein Adversarial Imitation Learning. WwXiv:/906.000 ///1 13 [cs, stat], June 2019. UtL.tttp://arxxx org/ abs 9066 08113 arXiv: 1906.081 013..
[80] Yunzhi Zhang, Pieter Abbeel, and I Lerrel Pinto. Automatic Curriculum Learning through Value Disagreement. arXiv:2006. 096441 [cs, stat), June 2020. URLtttp:://arza www.xiiivoorg.aa..
2006. 09641
arXiv: 2006.09644
[81] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On Learning Intrinsic Rewards for Policy Gradient Methods. arXiv:/804 06459 [cs, stat), June 2018. URL http:://a//ax// 06459 arXiv: 1804.06459.
[82] Zeyu Zheng, Junhyuk Oh, Matteo Hessel, Zhongwen Xu, Manuel Kroiss, Hado van Has- selt, David Silver, and Satinder Singh. What Can Leamnd Intrinsic Rewards Capture? arXiv:1912.055: 05500 [cs/. July 2020. URLhttp: htttht/tp//xiw 1912.05500.
05500 arXiv:
15


Reviews:
[{"review_id": "xnYyiPLAMlE", "reviewer": "Reviewer_wjsZ", "summary": "This paper proposes a method that uses the Wasserstein-1 distance between policy State distributions in order to provide additional reward and intrinsic motivation for learning policies. The method comes up with an approximation for being able to compute and use the Wasserstein one distance that can work in some practical settings. The method in the paper is used to improve the learning performance on a collection of 2d navitaion tasks and fetching tasks.\n\nPros\nThe paper does propose an interesting way to be able to efficiently estimate the Wasserstein-1 distance that can be used to compute policies state visitation distributions distance.\nThe method is shown to accelerate learning on some example environments.\n\nCons\nIt's not entirely clear how to place this paper. The paper is proposing an intrinsic motivation method but the actual objective or method in the paper is a better way to estimate the Wasserstein-1 distance and how it can be used to match Target distributions as in a type of imitation learning. So then either the paper should have better comparisons and experiments that compare to other imitation learning methods that are also used to learn distance based metrics. Or if the paper is thoroughly proposing an intrinsic motivation method then the paper should discuss and compare to recent state-of-the-art intrinsic motivation methods like ICM or RND or even SMiRL.\n\n\n\n\nReference:\n[1] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.   Curiosity-driven Exploration by Self-498supervised Prediction. 2017\n[2] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random networkdistillation.ICLR, 2018b\n[3] G. Berseth,  D. Geng,  C. M. Devin,  N. Rhinehart,  C. Finn,  D. Jayaraman,  and S. Levine.377{SM}irl: Surprise minimizing reinforcement learning in unstable environments. InInternational378Conference on Learning Representations, 2021.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nAdditional comments on the paper:\n- The work in this paper sounds very familiar to the state space covering paper [4] that also trained an objective function in order to have an agent minimize the distance between a type of distribution but in that case, the distribution was covering the entire State space.\n- Section 2.1 is outlining previous methods and intrinsic motivation and ends with a sentence saying that the Wasserstein distance has been used before as a valid reward for imitation learning. This is making it sound like the use of this metric that's proposed in the paper is already been done in two previous works. It would be helpful if the authors could clarify these points by noting the differences between the new proposed method and those previous works.\n- As the authors note, the Wasserstein distance may be more informative when distributions are disjoint, theoretically. It will be very important to be able to show in the paper that in practice the method proposed can perform much better than previous works [4] in this area that's solving a very similar problem. It would also be helpful to back up the claim online 97 stating that AIM might induce a form of directed exploration.\n- In terms of the focus on the type of problems in the paper, it would be very helpful to understand the reasoning why problems on unrealizable target distributions that cannot be completely matched. Is this a common challenge in reinforcement learning that needs to be addressed and if it is addressed will assist in solving a number of problems that the community is interested in?\n- The experiments in the paper describe how aim May speed up learning a policy towards a single goal. This makes one wonder whether or not the method in the paper can also speed up learning towards multiple goals and whether or not the single goal chosen has any particular biased towards the way AIM works. Were multiple goals evaluated in each of the environments in the experiments?\n- It's not completely clear how figure 2 is showing that AIM enables the agent to reach the goal and learn the required policy quickly. To understand if it's learning a policy quickly it will need to be compared to some other method that is also plotting these types of details and state visitation distribution. It is hard to tell if the visualization for figure two really indicates how or what the agent has learned and what it should follow in order to help it get to the Target better. It almost seems like figure two a to a has learned a better State visitation distribution that will lead it more quickly to the goal?\n- The results in figure 3 appear to be rather similar. Most of the methods seem to have very similar performance with the exception of reward plus HER. It would be beneficial to have results on additional environments to help readers understand what type of environments AIM will work best on. There are also comparisons to other aiAIM like methods that include RND [2] that would be helpful to compare to to make sure AIM is an improvement over the current state of the art.\n\nIf these issues area addressed I would consider updating my score.\n\n\n[4] Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.\n\n--------- Post author response ----- \nI have updated my score to reflect clarifications from the authors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "xH-HUeS-667", "reviewer": "Reviewer_vqmu", "summary": "This paper proposes a method for goal-conditioned RL. The main idea is to learn a function that looks like a value-function and then use that value function as a reward function for RL. This method is motivated as minimizing a certain Wasserstein distance. Empirically, the proposed method outperforms goal-conditioned RL applied to sparse reward functions, and performs roughly on par with goal-conditioned RL with dense rewards. In effect, it seems like this learned value function can serve the role of the dense reward function.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: The method proposed in this paper can be broken down into two main ideas: (1) a new method for learning a value-function-like term, and (2) a method for using that value function.\n\nThe first idea is (to the best of my knowledge) novel and fairly clever. The idea is to formulate goal-conditioned RL as minimizing a Wasserstein-1 distance between the state marginal distribution and some \"target\" distribution centered at the goal (sometimes a Dirac, sometimes a Gaussian). The Wasserstein-1 distance requires defining an underlying metric; the paper proposes to use the shortest path distance as a (quasi-)metric. Cleverly, the paper shows how the dual form of the Wasserstein allows one to estimate the Wasserstein distance without being about the compute the shortest path distance (which, in general, is as difficulty as solving the goal-conditioned RL problem).\n\nThe second idea (using a value-function like term) is not novel. While the paper does not claim this to be a contribution, I'd recommend citing some prior work that employs the same idea (e.g., see [Hartikainen '19, Tian '20] or other prior work on using terminal value functions).\n\n**Quality**:\n\nWhile I'm not an expert on the Wasserstein distance, the derivation of the proposed method seems correct to me. However, the resulting method for learning the value function (Eq. 8) is almost exactly the same as standard temporal difference learning, which is *already* used by standard goal-conditioned RL methods. Precisely, temporal difference learning with the -1/0 reward can equivalently be viewed as solving a constrained optimization problem where (1) the goal state has value 0, (2) the difference between the values of consecutive states is less than 1, and (3) the value of all states is minimized. I suspect that Eq. 8 is equivalent to some linear-program formulation of the RL problem (see the Puterman MDP book). In summary, the resulting method bears a very close resemblance to (1) learning a value function and (2) using that value function as a reward function. This connection is useful because it not only explains what the \"adversarial intrinsic motivation function\" is (a value function, up to an additive constant), but also offers a potentially better way to learn it (using Monte Carlo regression instead of TD learning). I would highly encourage the authors to compare against this baseline (both with a value function learned via MC regression as in Eq. 3 of [Hartikainen '19]) and via TD learning (similar to Eq. 8 of this paper)).\n\nNotwithstanding the results of this experiment, the idea that TD learning is equivalent to minimizing a Wasserstein-1 distance seems pretty cool! I'm not sure how this can be made into a practically useful insight.\n\nSome additional comments about the analysis:\n* It could be worth noting that the proposed objective is actually quite similar to C-learning [Eysenbach '20]. While C-learning minimizes a KL divergence with the goal distribution, the proposed method minimizes a Wasserstein-1 distance. After mentioning this connection, the authors could bring up the point about KL divergences not being informative for distributions with disjoint suppoer (perhaps adding another sentence (e.g., similar to the W-GAN paper) to build intuition.\n* While the proposed method is derived for discrete-state MDPs, it is later applied to continuous state MDPs. I'd recommend including a sentence to explain that these experiments do not match theory (e.g., \"Our experiments on simulated robotics tasks highlight that the method continues to work, even in settings with continuous states and actions, where our theory does not apply.\").\n* The discussion of variance in Sec 5.1 is reminiscent of risk-sensitive control (e.g., [Mihatsch '02]). This raises the question: if risk-sensitivity (i.e., minimizing variance) is important, why don't we just use a standard risk-sensitive objective (e.g., $\\min \\log E[T]$)?\n\nA few questions for the authors:\n1. In the robotic control tasks, it seems like the dense reward (a negative L2 distance) is equivalent to using the log probability of the target density (a Gaussian) as the reward. How is this different from what DiscoRL [Nasiriany '20] does. (If so, I'd recommend highlighting this connection, as it makes the comparison with dense rewards seems less like an \"oracle\" experiment and more like a prior method.)\n2. Can the Wasserstein distance (Eq. 2) be expressed in terms of the expected time to reach the goal, or the probability of reaching the goal (i.e., without the bounds discussed in Sec 5.1)? Such a connection would help me gain intuition for exactly what the proposed method is optimizing.\n3. If the optimal transport plan/coupling (L155) where known, are the clear ways that one could use it? Would it correspond to the optimal policy?\n\n\n**Clarity**:\n\nGenerally, the paper was clearly written. For me, the most confusing part was the discussion of the Wasserstein metric. I think the confusion was in introducing the notion that the Wasserstein metric requires defining another metric function. I was also confused about whether the choice of this metric function was a design decision or was motivated by theory. Please see the minor comments at the end of the review.\n\n**Significance**:\n\nThe idea that temporal difference learning is (approximately?) minimizing a Wasserstein metric seems potentially impactful, even if it doesn't immediately provide better algorithms for goal-conditioned RL. I don't think the actual method proposed will be impactful because (1) it seems nearly the same as prior methods that use a value function as a reward function, (2) the connection between the function $f(s)$ and standard quantities in RL (e.g., value functions) is unclear (but this could be easily corrected), and (3) the proposed method barely outperforms a simple baseline that uses a negative L2 distance as a reward function.\n\nMinor comments:\n* L59 - L60 -- If the policy minimizing the expected number of steps, then wouldn't such a policy also be optimal for stochastic dynamics?\n* L69 - L70 -- This is a great introduction sentence!\n* L87 -- Perhaps also discuss [Hartikainen '19].\n* L120, L121 -- I was initially a bit confused what the notation \"$\\rho: \\Delta (S)$\" and \"$\\sigma: \\Delta (G)$\" meant.\n* L129 -- I'd recommend stating $r(\\bar{s}) = 1$ in math, too.\n* L135 -- The three expectations could be combined into a single expectation.\n* L184 \"Definition 1\" -- I found this definition pretty confusing, and would recommend cutting it and directly defining $d(s, s_g)$ using the equation after L193.\n* L193 -- This equation raises an obvious question: Why not just use $T$ as a reward function? This question is only answered much later. I would recommend answering it immediately after defining $T$.\n* L164 - L197 -- None of this material is novel. I'd recommend moving it to a new paragraph in Section 3.1 and just citing [Kaelbling '93].\n* Eq 3 -- This idea is really clever! I'd highly recommend adding a few more sentences (and perhaps an illustration) to explain it.\n* L207 - L208 -- I found this sentence confusing.\n* L212 -- The importance of Proposition 1 is a bit unclear. I'd recommend moving it to Sec 5 (which focuses on analysis) and providing examples of settings where the inequalty is and is not tight.\n* Sec. 5.1 -- The importance of this section is a bit unclear to me. The result I was expecting was that minimizing the proposed Wasserstein metric corresponds to minimizing expected hitting time. Unless the analysis shows this, I'd recommend moving all of Sec. 5.1 to the appendix.\n* L240 \"Adversarial Intrinsic Motivation\" -- I found the term \"intrinsic motivation\" a bit confusing because it's often used in settings without clear objectives (in contrast, this paper is optimizing a very well defined objective). I'd recommend renaming it something like \"Learning a Value Function to Minimize the Wasserstein-1 Distance\".\n* L244 \"even if the ground metric\" -- This point seems really important! I'd recommend bringing it up (or at least foreshadowing it) when \"T\" is first defined in L193.\n* Fig 3: Please make this figure colorblind friendly.\n\n-----------------------------\n**Review decision**: Overall, I really like the idea of viewing goal-conditioned RL as a distribution matching problem and the idea of using Wasserstein duality to learn a value function. The paper clearly shows that this perspective yields good results on goal-conditioned RL problems. My main concern with the paper is its relationship with prior work, both empirically and theoretically. Empirically, it seems like the proposed method does not perform as well as \"HER + dense\", which I understand to be an instance of prior work [Disco RL applied to a Gaussian target distribution]. I would also recommend comparing against a baseline that learns a value function (either via TD learning or MC regression, as explained above) and uses that value function as a reward function. Theoretically, I'd highly recommend examining and explaining the connection between the adversarial function $f$ and temporal difference learning. Given these concerns, I vote to reject the paper.\n\n**Update after discussion**: The additional experiments and clarifications provided during the rebuttal period have addressed by concerns, and I have changed my vote to \"accept\".\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses limitations and potential broader impacts in the concluding section. I found this section a bit vague, and think it could be shortened and made more concise.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3"}, "extra_text": {}}, {"review_id": "WAXpaGXrpRE", "reviewer": "Reviewer_Qdoa", "summary": "This paper proposes Adversarial Intrinsic Motivation (AIM), a state distribution matching algorithm for goal-conditioned reinforcement learning. State distribution matching is an effective approach to imitation learning where the target distribution is yielded by the expert behavior. However, it is unclear how to apply it to goal-conditioned RL because the target distribution is unachievable - it puts all mass on the goal states and none on the trajectories leading to them. This paper proposes to use Wasserstein-1 distance as the distance measure between the agent's state visiting distribution and the goal distribution. To measure the distance between two states, the authors propose a quasimetric specific to MDPs - the time-step metric. The authors then prove that the policy that minimizes the Wasserstein-1 distance between the agent's state visiting distribution and the goal distribution under the time-step metric also minimizes the steps needed to achieve the goal. Empirical results show that AIM can provide dense and effective reward signals to the agent so that it learns better than using only the sparse reward of achieving the goal. When combined with Hindsight Experience Replay (HER), AIM outperforms the sparse goal-achieving rewards in a goal-conditioned RL setup and approaches the performance of a near-oracle hand-designed reward signal.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well written and easy to follow. The idea of using Wasserstein-1 distance to measure the distance between the state visiting distribution and the unachievable goal distribution makes sense. The theoretical analysis provides insight into the solution found by minimizing the Wasserstein-1 distance. The experiments successfully demonstrate the merits of AIM. I especially like the thorough analysis in the experiment section where the authors provide detailed explanations to both the successful cases and the failed cases.\n\nI do have one suggestion regarding the experiment. One natural baseline I can think of is to combine GAIL with HER in the goal-conditioned setup where the trajectories leading to the hindsight goals are treated as expert trajectories. So far I do not see an obvious reason why AIM would be better than this baseline. Thus I think this can be a strong baseline to compare against.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main contribution of this paper is theoretical and the experiments are mostly for illustration purposes. I do not see any potential negative societal impacts of this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an elegant idea and a fine investigation -- an important contribution to the community. After clarifications made by the authors, all reviewers agreed that this paper should be accepted. 
When preparing the final version, please make sure to address the reviewers' feedback.
Decision: Accept (Poster)