Efficient Exploration via State Marginal Matching
Authors: Lisa Lee
Benjain Eysenbach
Emilio Parisotto
Erix Xing
Sergey Levine
Ruslan Salakhutdinov
Venue:  (2020)
Keywords: reinforcement learning
exploration
distribution matching
robotics

Abstract: Reinforcement learning agents need to explore their unknown environments to solve the tasks given to them. The Bayes optimal solution to exploration is intractable for complex environments, and while several exploration methods have been proposed as approximations, it remains unclear what underlying objective is being optimized by existing exploration methods, or how they can be altered to incorporate prior knowledge about the task. Moreover, it is unclear how to acquire a single exploration strategy that will be useful for solving multiple downstream tasks. We address these shortcomings by learning a single exploration policy that can quickly solve a suite of downstream tasks in a multi-task setting, amortizing the cost of learning to explore. We recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution, which can incorporate prior knowledge about the task. We optimize the objective by reducing it to a two-player, zero-sum game between a state density model and a parametric policy. Our theoretical analysis of this approach suggests that prior exploration methods do not learn a policy that does distribution matching, but acquire a replay buffer that performs distribution matching, an observation that potentially explains these prior methods' success in single-task settings. On both simulated and real-world tasks, we demonstrate that our algorithm explores faster and adapts more quickly than prior methods.

Under review as a conference paper at ICLR 2020
EFFICIENT EXPLORATION VIA STATE MARGINAL MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning agents need to explore their unknown environments to solve the tasks given to them. The Bayes optimal solution to exploration in- tractable for complex environments, and while several exploration methods have been proposed as approximations, it remains unclear what underlying objective is being optimized by existing exploration methods, or how they can be altered to incorporate prior knowledge about the task. Moreover, it is unclear how to acquire a single exploration strategy that will be useful for solving multiple down- stream tasks. We address these shortcomings by learning a single exploration policy that can quickly solve suite of downstream tasks in a multi-task setting, amortizing the cost of learning to explore. We recast exploration as a problem of State Marginal Matching (SMM), where we aim learn a policy for which the state marginal distribution matches a given target state distribution, which can incorporate prior knowledge about the task. We optimize the objective by reducing it to two-player, zero-sum game between state density model and parametric policy. Our theoretical analysis of this approach suggests that prior exploration methods do not learn policy that does distribution matching, but acquire replay buffer that performs distribution matching, an observation that potentially explains prior methods' success in single-task settings. On both simulated and real-world tasks, we demonstrate that our algorithm explores faster and adapts more quickly than prior methods
INTRODUCTION
Reinforcement learning (RL) algorithms must be equipped with exploration mechanisms to effectively solve tasks with limited reward signals. These tasks arise in many real-world applications where providing human supervision is expensive. The inability of current RL algorithms to adequately explore limits their applicability to horghooizon control tasks.
wealth of prior work has studied exploration for RL, While, in theory, the Bayes-optimal exploration strategy is optimal, it is intractable to compute exactly, motivating work on tractable heuristics for exploration. Exploration methods based on random actions have limited ability to cover a wide range of states. More sophisticated techniques, such intrinsic motivation, accelerate learning in the single-task setting. However, these methods have two limitations. First, they do not explicitly define an objective quantify "good exploration." but rather argue that exploration arises implicitly through some iterative procedure. Lacking a well-defined optimization objective, it remains challenging to understand what these methods are doing and why they work. Similarly, the lack of metric to quantify exploration, even if only for evaluation, makes it challenging to compare exploration methods and assess progress in this area. The second limitation is that these methods target the single-task setting. Because these methods aim to converge to the optimal policy for particular task, it is challenging to repurpose these methods to solve multiple tasks.
We address these shortcomings by considering a multi- task setting, where many different reward functions can provided for the same set states and dynamics. Rather than exploring from scratch for each task, we aim to learn a single, lask-agnostic explorution policy that can be adapted to many possible downstream reward functions, amortizing the cost of learning to explore. This exploration
Videos and code: https::/ //si .ocgsggoolle.. com www.c////vvv.eeco. smm- anonymous

---

Under review as a conference paper at ICLR 2020
policy can be viewed prior the policy for solving downstream tasks. Learning will consist of two phases: during training, we acquire this task-agnostic exploration policy; during testing, we use this exploration policy to quickly explore and maximize the task reward.
Learning a single exploration policy is considerably more difficult than doing exploration throughout the course of learning single task. The latter is done by intrinsic motivation (Pathak et al. 2017
Tang et al. 2017 Oudeyer et al. 2007) and busnt-based exploration methods (Bellemare et al. 2016 which can effectively explore find states with high reward, at which point the agent can decrease exploration and increase exploitation of those high-reward states. While these methods perform efficient exploration for learning a single task, the policy at any particular iteration is not a good exploration policy. For example, the final policy at convergence would only visit the high-reward states discovered for the current task.
What objective should be optimized to obtain a good exploration policy? We recast exploration as a problem of State Marginal Matching: given a desired state distribution, we learn a mixture of policies for which the state marginal distribution matches this desired distribution. Without any prior information, this objective reduces to maximizing the marginal state entropy H[s]. which encourages the policy to visit as many states as possible. The distribution matching objective also provides convenient mechanism to incorporate prior knowledge about the task, whether in the form of safety constraints that the agent should obey; preferences for some states over other states; reward shaping; or the relative importance of each state dimension for a particular task.
We also propose an algorithm optimize the State Marginal Matching (SMM) objective. First, we reduce the problem of SMM to a two-player, zero-sum game between a policy player and a density player. We find Nash Equilibrium for this game using fictitious play (Brown, 1951). classic procedure from game theory. Our resulting algorithm iteratively fits a state density model and then updates the policy to visit states with low density under this model. Our analysis of this approach sheds light on prior work on exploration. In particular, while the policy learned by existing exploration algorithms does not perform distribution matching, the replay buffer does, an observation that potentially explains the success of prior methods. On both simulated and real-world tasks, we demonstrate that our algorithm explores more effectively and adapts more quickly to new tasks than state-of-the-art baselines.
2
RELATED WORK
Most prior work on exploration has looked at exploration bonuses and intrinsic motivation. One class of exploration methods uses prediction error of some auxiliary task as an exploration bonus, which provides high (intrinsic) reward in states where the predictive model performs poorly (Pathak et al. 2017) Oudeyer et al., 2007; [Schmidhuber] 1991; Houthooft et al., 2016)] Burda et al. 2018). Another set of approaches (Tang et al. 2017 [ellemare et al. 2016 Schmidhuberr (010)) directly encourage the agent to visit novel states. While all methods effectively explore during the course of solving single task (Taiga et al. 2019), the policy obtained at convergence is often not good exploration policy (see Section 4). In contrast, our method converges to a highly-explorators policy by maximizing state entropy in the training objective (Eq. 2
Many exploration algorithms can be classified by whether they explore in the space of actions, policy parameters, goals, states. Common exploration strategies including c-greedy and Omn- stein-Uhlenbeck noise (Lillicrap et al.) al., 2015). as well as standard MaxEnt RL algorithms (Ziebart) 2010; Haarnoja al. 2018), explore
the action space. Recent work (Fortunato et al. 2017 Plappert et al. 2017) shows that adding noise to the parameters of the policy can result in good exploration. Most closely related our work are methods that perform exploration in the space of states or goals (Colas et al. 2018 [eldd et al., 2017; Nair et al., 2018 Pong et al. 2019 Hazan et al. 2018). In fact, Hazan et al. (2018) consider the same State Marginal Matching objective that we examine and propose similar algorithm. In relation to tozHa et al. (2018), our main contributions are (1) empirically showing that exploration based on state-entropy is competitive with existing state-of-the-art exploration methods, and (2) explaining how existing exploration methods based on prediction error are implicitly maximizing this state-entropy objective. In Appendix C.T we also discuss how goal-com conditiiond RL (Kaelbling) 1993] [Schaul et al. (015) can be viewed as a special case of State Marginal Matching when the goal-samplin distribution is learned jointly with the policy.
2

---

Under review as a conference paper at ICLR 2020
The problems of exploration and meta-reinffrreemen learning are tightly coupled. Meta-reinforcemennt learning algorithms (Duan et al. [016]] Finn et al. 2017 Rakelly et al. al. [2019] Mishra et al. 2017)) must perform effective exploration if they hope to solve a downstream task. Some proor work has explicitly looked at the problem of learning to explore (Gupta et al. 2018 Xu et al. 2018). Our problem statement is similar meta-learning, in that also aim to learn policy downstream tasks. However, whereas meta-RL requires a distribution of task reward functions, our method will require only single target state marginal distribution. Due to the simpler problem assumptions and training procedure, our method may be easier to apply in real-world domains.
prior for solving
Related to our approach are standard maximum action entropy algorithms (Haarnoja et al. 2018 Kappen et al. 2012; Rawlik et al. 2013; Ziebart et al., 2008 Theodorou & Todorov (0122) While these algorithms are referred to as MaxEnt RI,, they are maximizing entropy over actions, not states.
These algorithms can be viewed as performing inference on a graphical model where the likelihood
of a trajectory is given by its exponentiated reward (Toussaint & Storkey] 2006) Levine| [018]]
Abdolmaleki et al. 2018). While distributions over trajectories induce distributions over states, computing the exact relationship requires integrating over all possible trajectories, an intractable problem for most MDPs. A related but distinct class of relative entropy methods entropy-based objective to limit the size of policy updates (Peters et al. 2010 [Schulman et al." 2015).
similar
Finally, the idea of distribution matching has been employed successfully in imitation learning (Ziebart et al. 2008 Ho & Ermon| [0166] Finn et al. 2016;; Fu et al., (017). Similar to some inverse RL algorithms (Ho& & Ermon| 2016; Fu et al. 2018), our method iterates between learning policy and learning a reward function, though our reward function is obtained via a density model instead of a discriminator. While inverse RL algorithms assume access to expert trajectories, we instead assume access to the density of the target state marginal distribution. In many realistic settings, such as robotic control with many degrees of freedom, providing fully-specified trajectories may be much more challenging than defining a target state marginal distribution. The latter only requires some aggregate statistics about expert behavior, and does not even need to be realizable by any policy.
In summary, our work unifies prior exploration methods as performing approximate distribution matching, and explains how state distribution matching can be performed properly. This perspective provides clearer picture of exploration, and this observation is useful particularly because many of the underlying ingredients, such as adversarial games and density estimation, have seen recent progress and therefore might be adopted to improve exploration methods.
STATE MARGINAL MATCHING
In this section, we propose the State Marginal Matching problem as a principled objective for learning to explore, and offer an algorithm for optimizing it. We consider parametric policy To II A { 8 } that chooses actions E A in Markov Decision Process (MDP) M with fixed episode lengths T, dynamics distribution p( 8t, (t), and initial state distribution po (s). The MDP M together with the policy mo form an implicit generative model over states. We define the state marginal distribution Px (s) as the probability that the policy visits state s:
T
P
s1 (S), oeristo (A(ee)
T
1(s = s)
Se41 (s..
We emphasize that (s) is not distribution over trajectories, and is not the stationary distribution of the policy after infinitely many steps, but rather the distribution over states visited in finite-length episode P We also note that any trajectory distribution matching problem can be reduced to a state marginal matching problem by augmenting the current state to include all previous states.
We assume that we are given target distribution ps(s) over states E S that encodes our belief about the tasks we may be given at test-time. For example, a roboticist might assign small values of (s(s) states that are dangerous, regardless of the desired task. Alternatively, we might also learn ps()) from data about human preferences (Christiano et al. 2017). For geal-reaching tasks, we can analytically derive the optimal target distribution (Appendix C Given p" (s), our goal to find a
(s)(s) approaches the policy's stationary distribution in the limit as the episodic horizon T oo.
3

---

Under review as a conference paper at ICLR 2020
training
single policy
mixture policy
('(s)
Pny
Pa(s)
Pny(s)
State
state
state
Figure 1: State Marginal Matching: (Leftt) Our goal is to learn policy whose distribution over states (blue histogram) matches some target density (black line). Our algorithm iteratively increases the reward states visited too infrequently (green arrow) and decreases the reward states visited too frequently (red arrow). (Center) At convergence, these two distributions are equal. (Right) For complex target distributions, we use mixture of policies p- (a) = prs (s)p(z)dz. (See Appendix
parametric policy that is "closest" to this target distribution, where we measure discrepancy using the Kullback-Leible (KI.) divergence:
max ((px()),2) (+)) A max - DxI. (px 11 (s)) rEI
menl
max (s)p=(s) [logp" (())"((s))))))))""))))){{)) (8) - Po (s)]
(1)
nell
max (s) llgg p (s)] + H..ss
This is the same objective in Hazan et al. which is mode-seeking (i.e.. exploratory). We show in Appendix [ that the policies obtained via State Marginal Matching provide an optimal exploration strategy for a particular distribution over reward functions. To gain intuition for the State Marginal Matching objective, we decomposed it in two ways. Equation we see that State Marginal Matching is equivalent to maximizing the reward function r(s) & log p" (s) while simultaneously maximizing the entropy of states. Note that, unlike traditional MaxEnt RL algorithms (Ziebart et al., 2008 Haarnoja et al. 2018), we regularize entropy the state distribution, not the conditional distribution of actions given states, which results in exploration in the space of states rather than in actions. Moreover, Equation suggests that State Marginal Matching maximizes pseudo-reward r(s) log log (=(s), which assigns positive utility to states that the agent visits too infrequently and negative utility to states visited too frequently (see Figure D. We emphasize that maximizing this pseudo-reward is not RL problem because the pseudo-reward reward depends on the policy.
(2018). Note that
the reverse-KL. (Bishop 2006),
3.1 OPTIMIZING THE STATE MARGINAL MATCHING OBJECTIVE
Optimizing Equation I to obtain single exploration policy is more challenging than standard RL because the reward function itself depends on the policy. To break this cyclic dependency, we
introduce a parametric state density model (x) E Q a {4p v E V to approximate the policy's
state marginal distribution, Pa (s). We assume that the class of density models Q is sufficiently expressive to represent every policy:
Assumption 1. For every policy x E II, there exists q E Q such that Dxu(px (s) g(s)) =
Under this assumption, optimizing the policy w.r.t. this approximate distribution g(s) will yield the same solution as Equation| I (see Appendix for the proof):
Proposition 3.1. Let policies II and density models Q satisfying Assumption be given. For any target distribution p", the following optimization problems equivalent:
max E p (s) log (s)] max min E (p.(ss]logg) (oooopp((>> (s) log {(s)]
(3)
Solving the new max-min optimization problem is equivalent to finding the Nash equilibrium of two-player, zero-sum game: a policy player chooses the policy π while the density player chooses the density model g. To avoid confusion, policy π the traditional RL problem and strategies to refer to the decisions л E 11 of the policy player and decisions Q of the density player. The Nash existence theorem (Nash 1951 proves that such a stationary point always exists for such a two-player, zero-sum game.
actions to refer to controls
A output by the
One common approach saddle point games is to alternate between updating player A w.r.t. player B. and updating player B w.r.t. player A. However, games such as Sock-Paper-Seissoor illustrate that such greedy approach is not guaranteed converge stationary point. slight variant,
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Learning to Explore via Fictitious Play
Input:
Target distribution (s)
Initialize policy xa [s), density model g(s), and replay buffer B.
while not converged do
(m)) <- arg m B. 1) [log {s)]]
T (m) <- arg max, E, Ea-pe(a) [r(s)] where r(s) è logp"(s) (a) logg!m/ (a)
B(m) <- g(m-1) U {(S,, #1+1]] with new transitions {(se, dee 1441))= sampled from ((m)
return historical policies { ,,.,,, (m)
An algorithm for optimizing the State Marginal Matching objective (Equation D. The algorithm iterates between (1) fitting density model ((m) and (2) training the policy n ((m) with a RL objective to optimize the expected return w.r.t. the updated reward function r(s). The algorithm returns the collection of policies from each iteration, which do distribution matching in aggregate.
Algorithm| 1
fictitioous play (Brown 1951) does converge to Nash equilibrium in finite time (Robinson, 1951
Daskalakis & Pan_ (014)). 4). At each iteration, each player chooses their best strategy in response to the historical average of the opponent's strategies. In our setting, fictitious play alternates between fitting
the density model to the historical average of policies (Fquation D. and updating the policy with RI.
minimize the log-density of the state, using a historical average of the density models (Equation 5):
1
9m++1 < arg max E, (t) [lo [s)]
where
p
Pm.
(8)
(4)
Tm+1 < arg max Banxx (s) (x) loopp" (s) log 9m (s)]
where
gm gm((s)
9:(()
(5)
m
Crucially, the exploration policy not the last policy, Tm+11 but rather the historical average policy:
Definition 3.1. A historical average policy F(a s), parametrized by a collection of policies T1,".", Tm. is a policy that randomly samples one of the policy iterates ~ Unif[T1, Tm. at the start of each episode and takes actions according that policy for each step in the episode. A new policy is sampled for the next episode.
We summarize the resulting algorithm in Algorithm I In practice, we can efficiently implement Equation 4 and avoid storing the policy parameters from every iteration by instead storing sampled states from each iteration 3 We cannot perform the same trick for Equation 5, and instead resort to approximating the historical average of density models with the most recent iterate. Algorithm [ looks similar to prior exploration methods based on prediction- error, suggesting that we might use SMM to understand how these prior methods work.
4 WHY DOES PREDICTION-ERROT EXPLORATION WORK?
Exploration methods based on prediction error BBurda al. 2018 Stadie al. 2015 Pathak et al. 2017; [Schiidhuber] 1991 Chentanez et al. 2005) do not converge to an exploratory policy, even in the absence of extrinsic reward. For example, consider the asymptotic behavior of ICM (Pathak ct al. (017) in a deterministic MDP, such as the Atari games where it was evaluated. At convergence, the predictive model will have zero error in all states, so the exploration bonus is zero the ICM objective has no effect on the policy at convergence. Similarly, consider the exploration bonus in Pseudocounts (Bellemare al. 2016): 1/ñ(s), where A(()) is the (estimated) number of times that state has been visited. In the infinite limit, each state has been visited infinitely many times, so the Pseudocount exploration bonus also goes to zero - Pseudocounts has no effect at convergence. Similar reasoning can be applied to other methods based prediction error (Burda et al. 2018 Stadie et al.] 2015). More broadly, we can extend this analysis to stochastic MDPs, where we consider abstract exploration algorithm that alternates between computing some intrinsic reward and performing RL (to convergence) on that intrinsic reward. Existing prediction-erse exploration
'One approach is to maintain an infinite-sizer replay buffer, and fit the density to the replay buffer at each iteration. Alternatively, we can replace older samples in fixed-size replay buffer less frequently such that sampling from B is uniform over iterations.
5

---

Under review as a conference paper at ICLR 2020
Initial State
SAC
SMM
% Goals Reathed]
Goal
SMM
SMK,
SMG, n=5
20
SIM, n-10
Hall Length
Halls longth 10
Holls
(a) Navigation environment
(b) Statc visitation
(c) % Goals reached during training
Figure 2: Exploration in State Space (SMM) vs. Action Space (SAC) for Navigation: (a): A point-mass
agent is spawned at the center of m long hallways that extend radially outward, and the target state distribution places uniform probability mass number of hallways to control the task difficulty. (b) A heatmap showing states visited by SAC and SMM during training illustrates that SMM explores a wider range of states. (c) SMM reaches more goals than the MaxEnt baseline. SM4 is an extension of SMM that incorporates mixture modelling with n > 1 skills (see Appendix B). and further improves exploration of SMM.
the end of each hallway. We can vary the length of the hallway and the
methods are all special cases. At each iteration, the RI. step solves a fully-observed MDP, which always admits deterministic policy solution (Puterman 2014). Thus, any exploration algorithm in this class cannot converge to a single, exploratory policy.
Despite these observations, prior methods do excel solving hard exploration tasks. We draw an analogy to fictitious play to explain their success. While these methods never acquire an exploratory policy, over the course of training they will eventually visit all states. In other words, the historical average over policies will visit wide range of states. Since the replay buffer exactly corresponds to this historical average over states, these methods will obtain a replay buffer with a diverse range of experience, possibly explaining why they succeed at solving hard exploration tasks. Moreover, this analysis suggests surprisingly simple method for obtaining an exploration from these prior methods: use mixture of the policy iterates throughout training. The following section will not only compare SMM against prior exploration methods, but also show that this historical averaging trick can be used to improve existing exploration methods.
5 SIMULATED EXPERIMENTS
We used simulated control tasks to determine if SMM learns exploratory policy, to compare SMM to prior exploration methods, and to study the effect of historical averaging. More details can be found in Appendix [] and code will be released upon publication.
Baselines and Implementation Details: We compare to a ofate-of-the-art off ooo-cy MaxEnt RL algorithm, Soft Actor-Critic (SAC) (Haarnoja et al. 2018); an inverse RL algorithm, Generative
Adversarial Imitation Learning (GAIL) (Ho & Ermon 2016); and three exploration methods:
Count-based Exploration (C), which discretizes states and uses - log (s) an exploration bonus. Pseudo-counts (PC) (Bellemare al. 20166, which uses the recoding probability Intrinsic Curiosity Module (ICM) (Pathak et al. 2017). which uses prediction error as bonus.
bonus.
We used SAC as the base RL algorithm for all exploration methods (SMM, C, PC, ICM). To implement SMM, we define the target distribution in terms of the extrinsic environment reward: p²(s) exp(rem (s)). We use variational autoencoder (VAE) to model the density q(s) for both SMM and Pseudocounts (PC), For the GAIL baseline, we generated synthetic expert data by sampling expert states from the target distribution p0(s) (see Appendix D.2 for details). Results for all experiments are averaged over 4-5 random seeds.
We start with a sanity check: Is exploration in state space (as done by SMM) better than exploration in action space (as done by MaxEnt RL, e.g., SAC)? To study this question, we implemented a 2D Navigation environment, shown in Figure 2a To evaluate each method, we counted the number of hallways that the agent fully explored (i.e., reached the end) during training. Figure 26 shows the state visitations for the three hallway environment, illustrating that SAC only explores one hallway, whereas SMM explores all three. Figure 2c also shows that SMM consistently explores 60% of hallways, whereas SAC rarely visits more than 20% of hallways.
The remaining simulated experiments used the Manipulation environment PPlappert et al. 2018). shown in Figure Bal Our first experiment evaluates whether the exploration policy acquired by SMM allows us to solve downstream tasks more quickly. We defined the target distribution to be uniform over the entire state space (joint block configuration), with the constraints that put

---

Under review as a conference paper at ICLR 2020
SM4 SMM
SMM WO
SAC
ICM
Object Position
Gripper Position
Divergence
MelSopeel
9/6
SMH
0.5
0.4
0.3
Distance
0.2
0.1
0
50
120
SACICM
SMM RL
SAC KCM SMM
SMH
Episodo
INDO HA
W/ HA
(a)
(b)
(d)
The robot agent controls single gripper arm to move block object to goal location on the table surface. The goal is not observed by the robot, thus requiring the rohot to explore by moving the block to different locations on the table. (b) Test-Time Exploration: At test-time, we sample goal locations uniformly on the table. and plot the percentage of goals found within N episodes. SMM and its mixture-mode" variant SM4 (Algorithm 2 both explore faster than the baselines, allowing to successfully find the goal in fewer episodes. (c) State Entropy: After training, we rollout the policy for le3 epochs, and record the entropy of the object and gripper positions. SMM achieves higher state entropy than the other methods. Historical avoraging also improves the exploration of prior methods. (d) Non-Uniform Exploration: We measure the discrepancy between the state marginal distribution, p- p=(s), and non-uniform target distribution. SMM matches the target distribution better than SAC and is on par with Count. Error bars show std. dev. across 4 random seeds.
Figure 3: Exploration for Manipulation. (a) Task:
low probability mass on states where the block has fallen off the table; that actions should be small; and that the arm should be close to the object. As shown Figure 33b SMM adapts substantially more quickly than other exploration methods, achieving success rate 20% higher than the next best method, and reaching the same level of performance of the next baseline (ICM) in 4x fewer episodes. SMM without historical averaging attains similar performance as the next best baseline (ICM), suggesting that historical averaging is the key ingredient, while the particular choice of prediction error or VAE is less important. We provide further ablation studies of SMM in Appendix B.2
While historical averaging is necessary to guarantee convergence ($ (.T). most prior exploration methods do not employ historical averaging, raising the question of whether is necessary in practice. To answer this question, we compare SMM to three exploration methods. In Figure [c] we compare the policy obtained at convergence with the historical average of policy iterates over training for each method. We measure how well each method explores by computing the marginal state entropy, which we compute by discretizing the state space The results show that SMM maximizes state entropy at least as effectively as prior methods, if not better. While this comparison is somewhat unfair, as we measure exploration using the objective that SMM maximizes, none of the methods we compare against propose metrics for exploration that we could use instead. Furthermore, we see that historical averaging is not only beneficial to SMM, but also improves the exploration of prior methods.
In our final simulated experiment, we check whether prior knowledge injected via the target distribu- tion is reflected in the policy obtained from State Marginal Matching. Using the same Manipulation modified the target distribution to assign larger probability to states where
environment above, the block was on the left half of the table than on the right half. In Figure 33d we measure whether SMM is able to achieve the target distribution by measuring the discrepancy between the block's 's horizontal coordinate and the target distribution. Compared to the SAC baseline, SMM and the Count baseline are half the distance to the target distribution. No method achieves zero discrepancy, suggesting that future methods could be better at matching state marginals.
6 REAL-WORLD EXPERIMENTS
While almost all research on exploration focus on simulated domains, attributes of the real world such as partial observability, nonstationarity, and stochasticity may make the exploration more challenging. The aim of this section is to see if SMM explores effectively on real-world robotic control task. We used the D 'CCaw (Ahn et al. 2019) robotic manipulator, which is a 3-fingered hand positioned vertically above handle that it can turn. For all experiments on the D 'Claw, used target distribution that places uniform mass over all object angles ,
DDscrretization used only for evaluation, no policy has access to it (except for Count).
7

---

Under review as a conference paper at ICLR 2020
Number of Rotations
Min Angle
Max Angle
Win Anole Mox Andle
IRL
SMM
SMM
PC
PC
SAC
SAC
ICM
ICM
Count
00
Count
Random
Random
-180
-90
90
180
trainin knv Steps
(a) D'Claw
(b) Sim2Real (le6 training env. steps)
(c) Training Real Robot
Figure 4: Real-Worid Exploration: (a) D'Claw is 9-DoF robotic hand (hh et al. 2019) that is trained
to turn a valve object. (b) Sim2Real: We trained each algorithm in simulation, and then measured how far the trained policy rotated the knob on the hardware robot. We also measured the maximum angle that the agent turned the knoh in the clockwise and counter-clockwise directions within one episode. (c) Training on Hardware: We trained SAC and SMM on the real robot for 1e5 environment steps (about 9 hours in real time), and measured the maximum angle turned throughout training. We see that SMM moves the knob more and visits a wider range of states than SAC. All results are averaged over 4-5 seeds.
In a first experiment, we trained SMM and other baselines in simulation, and then evaluated the acquired exploration policy on the real robot using two metrics: the total number of rotations (in either direction), and the maximum radians turned (in both directions). For each method, we computed the average metric across 100 evaluation episodes. We repeated this process for 5 independent training runs. Figure 4bb shows that SMM turns the knob more than the baselines, and turns the knob to a wider range of angles. To test for statistical significance, hypothesis that SMM turned the knob more and to a wider range of angles than SAC. The p-values were all less than 0.05: p 0.046 for number of rotations, p 0.019 for maximum clockwise angle, and 0.001 for maximum counter-clockwise angle.
used 1-sided Student's t-test to test the
In our second experiment, we investigated whether it was possible to learn an exploration policy directly in the real world, without the need for a simulator. Learning to explore in the real world quite important, as building faithful simulators of complex systems is challenging. The physical constraints of the real robot make data efficiency paramount, suggesting that learning to explore will require an effective exploration strategy. In Figure 4c we plot the range of angles that the policy explores throughout training. Not only does SMM explore a wider range of angles than SAC, but its ability explore increases throughout training, suggesting that the SMM objective correlated with real-world metrics of exploration.
In summary, the results in this section suggests that exploration techniques may actually be useful in the real world, which may encourage future work to study exploration methods on real-world tasks.
7 DISCUSSION
In this paper, introduced a formal objective for exploration. While it is often unclear what existing exploration algorithms will converge to, our State Marginal Matching objective has a clear solution: convergence, the policy should visit states in proportion their density under a target distribution. Not only does this objective encourage exploration, it also provides human users with a flexible mechanism bias exploration towards states they prefer and away from dangerous states. Upon convergence, the resulting policy can thereafter be used as prior in a multi-task setting, amortizing exploration and enabling faster adaptation new, potentially sparse, reward functions. The algorithm we proposed looks quite similar to previous exploration methods based on prediction error, suggesting that those methods are also performing some form of distribution matching. However, by deriving our method from first principles, we note that these prior methods omit crucial historical averaging step, without which the algorithm is not guaranteed to converge. Experiments on both simulated and real-world tasks demonstrated how SMM learns to explore, enabling an agent to efficiently explore in new tasks provided at test time.
In future work, we aim to study connections between inverse RL, MaxEnt RL and state marginal matching, all of which perform some form of distribution matching. Empirically, we aim to scale to more complex tasks by parallelizing the training of all mixture components simultancously. Broadly, we expect the state distribution matching problem formulation to enable the development of more effective and principled RL methods that reason about distributions rather than individual states.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. arXiv preprint arXiv:/806.066: 2018.
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint 1vXiv1887.112990 2018.
David Barber Felix Agakov. The im algorithm: variational approach information maximization. Advances in Neural Information Processing Systems, 16:201, 2004.
Michael Ahn, Henry 7hu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and VIKASH KUMAR. ROBEL: RObotics BEnchmarks for Learning with low-cost robots. In Conference on Robot Learning (CoRL), 2019.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count- based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471 1479, 2016.
Christopher M Bishop. Puttern recognition and machine learning. springer, 2006.
G. Brown. Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation, 1951.
Yuri Burda, Harrison Edwards. Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: /810. /2894, 2018
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances neural information processing systems, pp. 1281-1288, 2005.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodci. Deep reinforeement learning from human preferences. Advances in Neural Information Processing Systems, pp. 4299-43011 43007 2017.
John D Co-Reyes, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach. Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. arXiv preprint
2018.
Cédric Colas, Olivier Sigaud, and Pierre Yves Oudeyer. Curious: Intrinsically motivated multi-task, multi-goal reinforcement learning. arXiv preprint arXiv: 1810,06284 2018.
Constantinos Daskalakis and Qinxuan Pan. A counter-example to karlin's 's strong conjecture for fictitious play. In 2014 IEEE 55th Annual Symposium Foundations of Computer Science, pp. 11-20. IEEE, 2014.
Maria Dimakopoulou and Benjamin Van Roy. Coordinated exploration in concurrent reinforcement learning. arXiv preprint arXiv: / 802. 022.0/2222, 2018
Yan Duan, John Schulman. Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arX1v:/611.02779, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint 1rXiv:1802.06070, 2018.
Chelsea Finn, Sergey Levine, and Pieter Abbecl. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58 2016
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1126- 1135. JMLR. org, 2017.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv: 102.9 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: 1248, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement
learning. In International Conference on Learning Representations, 2018. htt.::ttttt//ttttttt.e net/ forummi d-mmmiirr---wl----

---

Under review as a conference paper at ICLR 2020
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforeement learning of structured exploration strategies. In Advances Neural Information Processing Systems, pp. 5302-5311, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, arXiv preprint arXiv: 180100/90 2018.
Elad Hazan, Sham M Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. arXiv preprint arXiv: 122000,,, 2018.
David Held, Xinyang Geng, Carlos Florensa, and Picter Abbeel. Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv:1705.06336 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565 4573, 2016.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Leslie Pack Kaclbling. Learning to achieve goals. In ICAI,, pp. 1094-1099. Citeseer, 1993.
Hilbert Kappen, Vicenç Gómez, and Manfred Opper. Optimal contral as graphical model inference problem. Machine learning, 87(2):159-182, 2012.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.000099 2018
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint wrXiv:150900999 2015.
Nancy A Lynch. Distributed algorithms. Elsevier, 1996.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. arXiv preprint arXiv:/707.0314) 2017.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
John Nash. Non-cooperative games. Annals of mathematics, pp. 286-295, 1951.
Pierre- Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental development. IEEE transactions evolutionary compulation, 11(2):655-266 2007.
Emilio Parisotto, Soham Ghosh, Sai Bhargav Yalamanchi, Varsha Chinnaobireddy, Yuhuai Wu, and Ruslan Salakhutdinov. Concurrent meta reinforvement learning. arXiv preprint arXiv:1903.02710, 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17,22 2017.
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal. Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. arXiv preprint 1rXiv:1706.0/900 2017.
Matthias Plappert, Marein Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider. Josh Tobin, Maciek Chociej. Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: S.kiv:882..09444 2018.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State- covering sclf-supervised reinforcement learning. arXiv preprint 1903v 2019.
Martin I. Puterman. Markov Decision Processes.: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
10

---

Under review as a conference paper at ICLR 2020
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta- reinforcement learning via probabilistic context variables. arXiv preprint arXiv: 090.008444, 2019.
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Twenty-Third International Joint Conference Artificial Intelligence, 2013.
Julia Robinson. An iterative method of solving game. Annals of mathematics, pp. 296-301, 1951.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning. pp. 1312 1320, 2015.
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals animats, pp. 222-227, 1991.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2011) IEEE Transactions Autonomous Mental Development, 2(3):230 247, 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Bradly C Stadic, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforcement learning with deep predictive models. arXiv preprint 157.0 2015.
Adrien Ali Taīga, William Fedus. Marlos Machado, Aaron Courville, and Marc G Bellemare. Benchmarking bonus-based exploration methods on the arcade learning environment. arXiv preprint 1xX.:1908.002388, 2019.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Picter Abbeel. # exploration: A study of count-basee exploration for deep reinforcement learning. In Advances in neural information processing systems. pp. 2753-2762 2017.
Evangelos A Theodorou and Emanuel Todorov. Relative entropy and free energy dualities: Connections to path integral and kl control. In 2012 IEEE 51st IEEE Conference Decision and Control (CDC), pp. 1466-1473. IEEE, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujeco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-500 5033, IEEE, 2012.
Mare Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state markov decision processes. In Proceedings the 23rd international conference Machine learning, pp. 945-952.
ACM, 2006.
Dan Xie, Sinisa Todorovic, and Song- Chun Zhu. Inferring "dark matter" and "dark energy" from videos. In The IEEE International Conference on Computer Vision (ICCV), December 2013.
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning explore with meta-policy gradient. arXiv
preprint arXiv: 1aoe:8803..004 2018.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cor cost.st. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24. 2019, pp. 16577, 2019, doi: 10.1109/ .org/ 10. 109/ CRRA. 2.0 9.8794102
ICRAA2011.879411
[htt/tts:::/ http://oooooooooooo
Brian D Ziehart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian D Ziebart, Andrew I. Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Aaai, volume 8. pp. 1433-1433 1438. Chicago. IL. USA, 2008
Brian D Ziebart, Nathan Ratifff Garratt Gallagher, Christoph Mertz, Kevin Peterson. Andrew Bagnell. Martial Hebert, Anind K Dey, and Siddhartha Srinivasa. Planning- based prediction for pedestrians. In 2009 JEFF/RS) International Conference on Intelligent Robots and Systems, pp. 3931-3936. IEEE, 2009.

---

Under review as a conference paper at ICLR 2020
A PROOFS
Proof of Proposition 3.7 Note that
pp (e) llgp" log](s)] = Ppp (4) [logp" (orgp"((]] (s) - -log
(s)] -logp=(8)
+ DKL. (p= (8) |l (())).
(6)
By Assumption 1
(pL(P (px || q(s) for some q E Q, so we obtain the desired result:
max min E (p)()) (8) log a(+)) = max [ (=) (s) - log Pa (s)] + min (xx.(P((s) |l ((())
max (s) - log Pm (s)].
B BETTER MARGINAL MATCHING WITH MIXTURES OF MIXTURES
This section introduces an extension of SMM, SM4, that incorporates mixure modelling. Given the challenging problem of exploration in large spaces, it is natural wonder whether we can accelerate exploration by automatically decomposing the potentially-muliiodda multimodal target distribution into mixture of "casier-to-leam" distributions and learn corresponding set of policies do distribution matching for each component. Note that the mixture model we introduce here is orthogonal to the historical averaging step discussed before. Using Pr, (s) denote the state distribution of the policy conditioned on the latent variable E 2, the state marginal distribution of mixture policies
(=(8) = / Pr. Pt. (s)((2)d Exmp(x) (s),
where p(z) is latent prior. As before, we will minimize the KL divergence between this mixture distribution and the target distribution. Using Bayes' rule re- re-write Pr (s) in terms of conditional probabilities, we obtain the following optimization problem:
(s) log
max E zmp(2) (x2)2e2 (M)p)(()
= E =><(=>)
log (s)
log Pa, (s) + p(z /
|s)
log p(2)
Pr, (s)p(z) p(=(s)
am-Pa
(a)
(b)
Intuitively, this says that the agent should go to states (a) with high density under the target state distribution, (b) where this agent has been before, and (c) where this agent clearly distinguishable from the other agents. The last term (d) says to explore in the space of mixture components 2. This decomposition bears a resemblance
objectives in recent work (Achiam et al. 2018 Eysenbach et al. 2018 Co-Reyes al. 2018). Thus, one interpretation of our work is as explaining that mutual information objectives almost perform distribution matching. The caveat that prior work omits the state entropy term log Ps. (s) which provides high reward for visiting novel states, possibly explaining why these previous works have failed to scale to complex tasks.
the mutual-infore
B.1 ALGORITHMIC SUMMARY
We summarize the resulting procedure in Algorithm 2 which we refer to as SM4 (State Marginal
Matching with Mixtures of Mixtures). The algorithm (1) fits density model q: (mm) (s) approximate the state marginal distribution for each policy и=i (2) learns discriminator d(rm) s) to predict which policy πı will visit state s; and (3) uses RL to update each policy π. to maximize the expected
return of its corresponding reward function (s() è log p" (s) log Pa., (s) + log | log p()
derived in Equation
The only difference from Algorithm is that we learn a discriminator d(z s). in addition to updating the density models (s)s) and the policies (a s). Jensen's 's inequality tells that maximizing the lon-density of the learned discriminator will maximize a lower bound on the true density (see Agakoy (2004)):
(e (p).a))(s)) [log d(= s)] (Egr (s),spp(e) Dog
12

---

Under review as a conference paper at ICLR 2020
Algorithm 2 State Marginal Matching with Mixtures of Mixtures (SM4)
Input:
Target distribution p*(s)
Initialize policy T. (a s), density model (x(s), discriminator d(> s), and replay buffer B. while not converged do
for 2 ==
=,,.. 1. n2 do
> (1) Update density model for each policy zz-
92 (m)
f- arg max, Ri B0)-B( (e)(("))))-Br 1 ",===))
log [(s)
d(m)) f arg max, Bim))) [log d(> |)]
>> (2) Update discriminator.
for
.. n do
r2 (m)
(8) logp" (s) log (m) (8) + log dlm) 8) - (0gp(=)
(m) arg max pp) 12= (s) (s) (m)
m2
P. (3) Update each policy w..
Sample latent skill (mm)) p(2)
Sample transitions {(5). ae; 1 with (m) (a n2
B(m) + B (amm11)) U (22 (m(m), St, as, $(+1)
(1) return [11 1.. (1) ), 1 (m)
nm }}
[m]]
,...
Algorithm 2 An algorithm for learning mixture of policies Tı, ".....,......................... Th that do state marginal matching
in aggregate. The algorithm (1) fits a density model q! (mm) (s) to approximate the state marginal distribution for each policy #2: (2) learns discriminator d'm) s) to predict which policy will visit state s; and (3) uses RI. to update each policy n. to maximize the expected return of its corresponding reward function derived in Equation 8 (s) A log p" (s) log Pr. (s) logp(= | s) log p(z). In our implementation, the density model 9z (s) is a VAE that inputs the concatenated vector (s, 2} of the state s and the latent skill z used to
obtain this sample s; and the discriminator a foedforward MLP. The algorithm returns the historical average of mixtures of policies (a total of n m policies).
In our implementation, the density model g= is VAE that inputs the concatenated vector {s, 2} of the state and the latent skill z used to obtain this sample s; and the discriminator is feedforward MLP. The algorithm returns the historical average of mixtures of policies (a total of n m policies). Our implementation uses uniform categorical distribution for the prior p(2), and does not implement the update for p(z).
Note that the updates for each z can be conducted in parallel. A distributed implementation would emulate broadcast-colled algorithms 4Lynch] 1996). with each worker updating the policy indepen- dently, and periodically aggregating results to update the discriminator d(2 s). Such a distributed implementation has the appealing property that each compute node would explore a different part of the state space. While there has been some work on multi-agent coordinated exploration (Parisotto) jet al., 2019) and concurrent exploration (Dimakopoulou & Van Roy 2018). remains a fairly un- explored area (pun intended) and we believe that SMM with Mixtures of Mixtures offers simple approach to this problem.
B.2 ABLATION STUDY
To understand the relative contribution of each component in the SM4 objective (Equation B
we compare SM4 to baselines that lack conditional state entropy Hz [s] = - log pm, (s), latent conditional action entropy logp(2 s), or both (i.e, SAC). In Figure 5a we plot the training time performance on the Navigation task with 3 halls of length 50. We see that SM4 relies heavily on both key differences from SAC.
In Figure 5b] we study the effect of mixture modelling on test-time exploration in the Manipulation environment. After running SMM/SM4 with a uniform distribution, we count the number episodes required to find an (unknown) goal state. We run each method for the same number of environment transitions; a mixture of three policies does not get to take three times more transitions. We find that increasing the number of mixture components increases the agents success. However, the effect was smaller when using historical averaging. Taken together, this result suggests that efficient exploration requires either historical averaging mixture modelling, but might not need both.
13

---

Under review as a conference paper at ICLR 2020
Longth Hal May
SMM
n=
n=2
n=4
w/ HA
w/o HA
0.8
Robum
Success Rate
0.7
0.6
0.5
10
0.4
0.3
20
40
60
80
100
Episode
(a) Train-time Performance on Navigation
(b) Test-time Performance on Manipulation
Figure 5: Ablation Analysis of State Marginal Matching with Mixtures of Mixtures (SM4). (a): On the Navigation task, we compare SM4 (with three mixture components) against ablation baselines that lack conditional state entropy, latent conditional action entropy. or both (i.e., SAC) in the SM4 objective (Equation S). We see that both terms contribute heavily to the exploration ability SM4, but the state entropy term is especially critical. (b): We compare SMM/SM4 with different numbers of mixtures, and with vs. without historical averaging. We found that increasing the number of latent mixture components E (1,2, 4) accelerates exploration, does historical averaging. Error bars show std. dev. across 4 random seeds.
C CHOOSING p" (s) FOR
TASKS
In general, the choice of the target distribution p*(s) will depend on the distribution of test-time tasks. In this section, consider the special case where the test-time tasks correspond to goal-reaching derive the optimal target distribution p" (s). We consider the setting where goals g ~ Pg(g) are sampled from
known distribution. Our goal is to minimize the number of episodes required to reach that goal state. We define reaching the goal state as visiting a state that lies within an ball of the goal, where both > 0 and the distance metric are known.
We start with a simple lemma that shows that the probability that we reach the goal at any state in trajectory is at least the probability that we reach the goal at randomly chosen state in that same trajectory. Defining the binary random variable A 1 [[|s; all < e) the event that the state time reaches the goal state, we can formally state the claim as follows: Lemma C.1.
T
P
Σ 3 >
2 p(2)
where t~ Uniff1,
Proof. We start by noting the following implication:
2 >>>0
(10)
Thus, the probability of the event on the RHS must be at least as large as the probability of the event on the LHS:
p(z)) S
Σ 21 22>0
(11)
L
Next, we look at the expected number of episodes to reach the goal state. Since each episode is independent, the expected hitting time simply
HITTINGTIME(S)
(12)
p( some state reaches s)
-i=1
0)
p(z))
Note that we have upper-bounded the hitting time using Lemma C.I variable, we take expectation over g:
Since the goal is a random
(spp,()) (e) [HITTINGTIME(S) & Espp(()
(13)
p(2))
14

---

Under review as a conference paper at ICLR 2020
We rewrite the RHS using p" (s) to denote the target state marginal distribution:
E Bapp' (a) [IITTINGTIME(S)] I (1) [i
(3) $ll - c)ds
F(p")
(14)
We will minimize F, an upper bound the expected hitting time.
Lemma C.2. The state marginal distribution (s)) &
V (()) minimizes F(p"), where
p(s) _ I pp) a|| < e)da is a smoothed version of the target density.
Before presenting the proof. we provide a bit intuition. In the case where - 0, the optimal target distribution is p" (s) x v (s). For non-zero e, the policy Lemma C.2 is equivalent to convolving (s(s) with box filter before taking the square root. In both cases, we see that the optimal policy does distribution matching to some function of the goal distribution. Note that p(-) may not sum to one and therefore is not proper probability distribution.
Proof. We start by forming the Lagrangian:
(s(s) ds (f p" (3(()d ( $1 s s)dd
C(p")
(15)
The first derivative is
dc
- -Po 311 S ds =
(16)
dp*(6)
25)
Note that the second derivative is positive, indicating that this Lagrangian is convex, all stationary points must be global minima:
dC
/ 2p
(s) 51) ds 0
(17)
dp) (3)2
3
Setting the first derivative equal to zero and rearranging terms, we obtain
x(3) / (o(())((ls $ c)ds
(18)
Renaming + s, we obtain the desired result.
C.I CONNECTIONS TO GOAL-CONDITIONED RL
Coal-Conditioned RL (Kaelbling [993 Nair et al. 2018 Held et al. 2017) can be viewed as a special case of State Marginal Matching when the goal-sampling distribution is learned jointly with the policy, In particular, consider the State Marginal Matching with mixture policy (Algorithm 2 where the mixture component maps bijectively to goal states g. In this case, learn coal-conditioned policies the form (a s. s.g). We start by swapping g for 2 in the SMM objective with Mixtures of Policies (Fquation 8):
(px((((s) (s)) E gor(g) [logg"((s) ()) log p(g 8) log (s g) log [g)] s>>((1g)
(19)
The second term plg 8) is an estimate of which goal the agent is trying to reach, similar to objectives intent inference (Ziebart al. 22099; Xie et al., 2013).. The third term i(s g) is the distribution over states visited by the policy when attempting to reach goal g. For an optimal goal-conditioned policy an infinite-horizon MDP, both of these terms are Dirac functions:
Tl s) Pa (s g) 1
(20)
In this setting, the State Marginal Matching objective simply says to sample goals g <(g) with probability equal to the density of that goal under the target distribution.
D (8) p² (x)) = E. g~)((g) s
[logp' (s) - log (g))
(21)
15

---

Under review as a conference paper at ICLR 2020
Whether goad-cond RL is the preferable way do distribution matching depends (1) the difficulty of sampling goals and (2) the supervision that will be provided at test time. is natural to conditonnd RL in settings where it is easy to sample goals, such as when the space of goals small and finite or otherwise low-dimensional. a large collection of goals is available apriori, we could use importance sampling generate goals to train the coal-conditioned policy (Pong al.] 2019). However, many real-world settings have high-dimensional goals, which can be challenging to sample. While goal-conditioned RL likely the right approach when we will be given test-time task, a latent- cotent-conditioned policy may explore better in settings where the goal-state is not provided at test-time.
D ADDITIONAL EXPERIMENTS & EXPERIMENTAL DETAILS
D.1 ENVIRONMENT DETAILS
We summarize the environment parameters for Navigation (Figures 2 5a and D'Claw (Figure 4 in Table
Manipulation (Fig-
ures
5b
Navigation: Episodes have maximum time horizon of 100 steps. The environment reward is
otherwise {00
[Ps
if |lobor 9 l1 for any E [ni]
Teav (s)
where Say is the xy xy-position of the agent. We used a uniform target distribution over the end all m halls, so the environment reward at training time is (ene(s) if the robot is close enough to the end of any of the halls.
We used a fixed hall length of in Figures 2b and 2c and length 50 Figure 5a All experiments used = 3 halls, except in Figure 2c where varied the number of halls {3, 5,
Manipulation. We used the simulated Fetch Robotics arm unnSiimlemented by Plappert et al. (2018) using the MuJoCo simulator Todorov et al. 2012). The state vector coordinates Subj. Stobat E R of the block and the robot gripper respectively, as well as their velocities, orientations, and relative position Subj Sroba At the beginning of each episode, spawn the object at the center of the table, and the robot gripper above the initial block position. We terminate each episode after 50 environment steps, if the block falls off the table.
25 includes the xyz-
We considered two target state marginal distributions. In Manipulation- the target density is given by
p" (s) x exp [paagoa(()) + (igTrobor + Cr3Taction (s))
where 01, 02, 0g > are fixed weights, and the rewards
Tgoal (s) = 1 (Sahj is on the table surface)
(robor(s) == 1 Soty Spobol < 0.1)
Taction (s) = -|al12
correspond (1) uniform distribution of the block position over the table surface (the agent receives +0 reward while the block is the table), (2) an indicator reward for moving the robot gripper close to the block, and (3) action penalty, respectively. The environment reward is a weighted sum of the three reward terms: (s) A goat (>) + (sobo((s) + 1 (action (s). At test-time, we sample a goal block location g E R3 uniformly on the table surface, and the goal is not observed by the agent.
In Manipulation-Hacl the target state density places higher probability mass to states where the block on the left-side of the table. This is implemented by replacing I (s))((s) with a reward function that gives slightly higher reward +0.1 for states where the block is on the left-side of the table.
D'Claw. The D'Claw robot (Ahn al. 2019 Zhu al. 2019 9 controls three claws to rotate a valve object. The environment consists of 9-dimensional action space (three joints per claw) and 12-dimensional observation space that encodes the joint angles and object orientation. We fixed each
httos://fetccrobootii www.:roboitisaco. www.robbtiicsbeechmark...
16

---

Under review as a conference paper at ICLR 2020
Table 1: Environment parameters specifyyng the observation space dimension |S]; action space dimension |Al; max episode length T; the environment reward, related to the target distribution by expfrenv($) (s), and other environment parameters.
Environment
[8]
[
T
Env Reward (log (s))
Other Parameters
Figure
# Halls: 5, 7
Uniform halls
all
2
2
2
100
Hall length: 10
Navigation
# Halls: 3
Uniform over all m halls
Hall length: 50
Uniform block pos. table surface
36
3c
5b
Manipulation
More block pos, density on left-half of table
D'Claw
12
9 50
Uniform object angle
4
over [ 180", 180%)
episode at 50 timesteps, which is about 5 seconds on the real robot. In the hardware experiments, each algorithm was trained on the same four D'Claw robots ensure consistency.
We defined the target state distribution to place uniform probability mass over all object angles in [-180%, 180%]. also incorporates reward shaping terms that place lower probability mass on states with high joint velocity and on states with joint positions that deviate far from the initial position (see (Zhu al. 2019)
D.2 GAIL
GAIL assumes access to expert demonstrations, which SMM and the other exploration methods do not require. To compare GAIL with the exploration methods on a level footing, we sampled synthetic states from p" (s) to train GAIL,, and restricted the GAIL. discriminator input to states only (no actions).
For D'Claw (Fig. we sampled the valve object angle uniformly in [-180%, 180%] For Manipulation- Cnnipullaiio-Unfformm (Fig. [c)) we sampled object positions Sohject uniformly the table sur- face, and tried two different sampling distributions for the gripper position Syoher (see Fig. (6. For both environments, all other state dimensions were sampled uniformly in [-10, 10). and used le4 synthetic state samples to train GAIL.
Since the state samples from p" (s) may not be reachable from the initial state, the policy may not be able to fool the discriminator. To get around this problem, we also tried training GAIL with the discriminator input restricted to only the state dimensions corresponding the object position or gripper position (Manipulation)) or the object angle (D Claw). We summarize these GAIL ablation experiments in Fig. (6] In experiments, used the best GAIL. ablation model to compare against the exploration baselines Figures 3c and
D.3 VAE DENSITY MODEL
In our SMM implementation, we estimated the density of data x as p(z) se decoder(ā = ø|2 encoder(z)). That we encoded to 2, reconstruction f from and then took the likelihood of the true data x under a unit-variance Gaussian distribution centered at the reconstructed á. The log-likelihood is therefore given by the mean-squared error between the data and the reconstruction f, plus a constant that is independent of æ: log (2x) = 1| + C.
D.4 COMPUTATIONAL COMPLEXITY
We compare the wall-clock time of each exploration method in Table 2 The computational cost of method is comparable with prior work.
17

---

Under review as a conference paper at ICLR 2020
SodSaeeel tobject Pos.)
Maloobeel foripper Pos
Classification Loss
Classification Loss
NdSSupel (Angle)
Nax Angle Anglo (Radiansi (Radians)
Input
Discrimnator
1.0
Lodpett Sodee)
Uniform
Narmal
env steps
Gripper "Shusae
1e5 onv stops
(a) GAIL ablations on Manipulation
(b) GAIL ablations on D'Claw
Figure 6: GAIL Ablation Study: We studied the effect of restricting the GAIL discriminator input to fewer state dimensions. (a) Manipulation: We trained the GAIL discriminator on the entire state vector S; on the object and gripper positions {Sobject: Stobot only; or on the object position Sobject only, We also varied the sampling distribution for the gripper position, p" (Srobor): we compare using a normal distribution, AV (Sabject, Iz), to sample gripper positions closer to the object, versus uniform distribution, Uniform - 10, 10). for greater entropy of the sampled gripper positions. We observe that sampling gripper positions closer to the object position improves the entropy of the object position Ha [Sabjec]] but hurts the entropy of the gripper position H. [. (b) D'Claw: We restricted the discriminator to the entire state vector 8, or to the object angle and position Sebject Analysis: both domains, we observe that restricting the discriminator input to fewer state dimensions (e.g.. Subject) makes the discriminator less capable of distinguishing between expert and policy states (orange and green curves). On the other hand, training the entire state vector causes the discriminator loss to approach 0 (i.e., perfect classification), partly because some of the "expert" states sampled from p" (s) are not reachable from the initial state, and the policy is thus unable to fool the discriminator.
Table 2: Per-epoch wall-clock time on the Manipulation environment. One epoch is le3 env. steps.
SAC
ICM
Count
SMM (ours)
PseudoCounts
17.95s (+0%)
22.74s (+27%)
25.24s (+41%)
25.82s (+44%)
33.87s (+89%)
D.5 ALGORITHM HYPERPARAMETERS
We summarize hyperparameter settings in Table 3 Navigation, 1e6 steps on Manipulation, le6 steps on D' D'Claw Sim2Real, and le5 steps on C'Clww hardware.
All algorithms were trained for le5 steps on
Loss Hyperparameters. For each exploration method, we tuned the weights of the different loss components. SAC reward scale controls the weight of the action entropy reward relative to the extrinsic reward. Count coeff controls the intrinsic count-based exploration reward w.r.t. the extrinsic reward and SAC action entropy reward. Similarly, Pseudocount coeff controls the intrinsic pseudocount exploration reward. SMM coeff for H[s z] and H|= s] control the weight of the different loss components (state entropy and latent conditional entropy) of the SMM objective in Equation g
Historical Averaging. In the Manipulation experiments, we tried the following sampling strategies for historical averaging: (1) Uniform: Sample policies uniformly across training iterations. (2) Exponential: Sample policies, with recent policies sampled exponentially more than earlier ones. (3) Last: Sample the N latest policies uniformly at random. We found that Uniform worked less well, possibly due to the policies at early iterations not being trained enough. We found negligible difference in the state entropy metric between Exponential vs. Last, and between sampling 5 vs. 10 historical policies, and we also note that it is unnecessary to keep checkpoints from every iteration.
Network Hyperparameters. For all algorithms, we use a Gaussian policy with two hidden layers with Tanh activation and a final fully-connected layer. The Value function and Q-function each are a feedforward MLP with two hidden layers with ReLU activation and a final fully-connected layer. Each hidden layer is of size 300 (SMM, SAC, ICM, C, PC) or 256 (GAIL). The same network configuration is used for the SMM discriminator, d(z s), and the GAIL discriminator, but with different input and output sizes. The SMM density model, g(s), is modeled by VAE with encoder and decoder networks each consisting of two hidden layers of size (150, 150) with ReLU activation. The same VAE network configuration is used for Pseudocount.
18

---

Under review as a conference paper at ICLR 2020
GAIL Hyperparameters: The replay buffer is filled with le4 random actions before training, for training stability. We perform one discriminator update per SAC update. For both Manipulation and D Claw, used 1e4 states sampled from p" (s). Other hyperparameter settings, such as batch size for both discriminator and policy updates, are summarized in Table [ We observed that GAIL training is more unstable compared the exploration baselines. Thus, for GAIL,, we did not take the final iterate (e.g., policy at convergence) but instead used early termination (e.g., take the best iterate according the state entropy metric).
D.6 VISUALIZING THE MANIPULATION ENVIRONMENT
We visualize where different methods push the block in the Manipulation environment. More precisely, we visualize the log state marginal log Px_ (s) over block XY-coordinates a = (x, y) in Figures 7 and In Figure 9 we plot goals sampled at test-time, colored by the number of episodes each method required to push the block to that goal location. Blue dots indicate that the agent found the goal quickly. We observe that SMM has the most blue dots, indicating that it succeeds in exploring a wide range of states at test-time.
Visualization Block Position Density Count
SAC
SMM
Figure 7: The log state marginal log Pr (s) over block XY-coordinates,, averaged over 1e3 epochs.
Visualization Density (SMM, n=8)
z=3
2=6
2=7
All
Figure 8: SM4 with Eight Mixture Components. The log state marginal log Pa. (s) over block XY-coordinates for each latent skill (O...., 7), averaged over 1000 epochs.
ty
Episodes Till
Figure 9; Goals sampled uniformly on the table surface, colored by the number of episodes until the policy finds the goal. Red (100 episodes) indicates failure. The block always starts at the center.
19

---

Under review as a conference paper at ICLR 2020
Count
SMM-2
SMM-4
SHM-8
SMM--16
Return
Ret Tablet
Avg (Action Penatty
H:2(5)
2.5
0.0
-0.
2.0
-5.0
60
-7.5
155
-100
-12.5
120
055
2000
Epoch
Epoch
Epoch
Fpoch
Epoch
Epoch
Average Return
Epoch
(b) Latent action & state entropies for SM4 with E (2,4,8, 16) skills
Figure 10: Train curves on Manipulation. One epoch is 1e3 steps. (a) The environment reward weighted sum of three terms: (s) (+0 object on table, -1 otherwise), Frobe(()) (+1 robot gripper is close block), and Faction (action penalty term). with weights -20, 1, 0.1 respectively (see Appendix (.I)). The three exploration methods (ICM, Count, SMM) also optimize an auxilliary exploration loss, which makes the agent more likely move around the block. Compared to SAC, this causes the exploration methods to get worse returns for Fgua(() (8) and Faction()) (due to the agent moving the block around), but also quickly learns to maximize the sparse reward Trobor (s) (indicator reward for moving gripper within a threshold distance to the block). (b) The latent action entropy H[z (discriminator) and latent state entropy H[s (density model) per epoch.
20

---

Under review as a conference paper at ICLR 2020
Table 3: Hyperparameter Settings. Hyperparametees were chosen according to the following eval metrics: State entropy of the discretized gripper and block positions (bin size 0.05), after rolling out the trained policy for 50K steps. Manip.-Holi (x (s) |l (a)) and TV (s), (r))) of the discretized gripper and block positions (bin size 0.01), after rolling out trained policy for 50K env steps. 2D Navigation: State entropy discretized XY-positions the trained policy. D Claw:: State entropy of the object angle.
Environment
Algorithm
Hyperparameters Used
Hyperparameters Considered
Batch size: 128
lс6 env training steps
SMM, SAC.
RL discount: 0.99
ICM, Count,
Network size: 300
N/A (Default SAC hyperparameters)
All
Pseudacount
Policy Ir: 3e-4
Q-function Ir: 3e-4
Value function Ir: 3c-4
le6 env training steps
Policy Ir: le-S
Critic le-3
GAIL
N/A (Default GAIL hyperparameters)
Random actions
before training: le4
Network size: 256
Navigation (Fig. 5a
SMM. SAC
SAC reward scale: 25
SAC reward scale: le-2, 0.1, 1, 10, 25, SMM H[s X] coeff: le-3, le-2, 1, SMM H[ cooff: le-3, le-2, 1,
SMM H 2] coeff:
SMM
SMM 1
coeff: 1
Num skills: 4
Num skills: 1, 2, 4, 8, 16
VAE le-2
VAE Ir: le-4, 1e-3, le-2
SMM 7 2] coeff: 1
SMM
SMM 7| coeff: 1
HA sampling: Exponential
HA sampling: Exponential, Uniform, Last
Manip, Uniform (Fig.
HA policies: 10
HA policies: 10
3b
3c 5bp
SMM Latent Prior Coeff:
SMM Latent Prior Coeff: 1,
SAC
SAC reward scale: 0.1
SAC reward scale: 0.1, 1. 10, 100
Count coeff: 10
Count
Count coeff: 0.1, 1,
Histogram bin width: 0.05
Pscudocount coeff:
0.1, 1, 10 (Use same VAE Ir as SMM)
Pseudocount
Pscudocount
VAE lr: lc-2
ICM
Learning rate: 1c-3
Learning rate: lc-4, 1c-3, 1c-2
Batch size: 512
Batch size: 128, 512, 1024
# SAC updates per step:
SAC updates per step: 1,4
GAIL
Discriminator input: s
Discriminator input: 8, Subjecs [Sobjest Brobut
Training iterate: le6
Training iterate: le5, 2e5, 3e5. 9e5, le6
# State Samples: le4
State Samples: le4
SMM, SAC,
SAC reward scale: 0.1
(Best reward scale for -ninfor)
ICM, Count
Manip.-Halff
Num skills: 4
(Fig. 3d
Num skills: 1, 2,
SMM
SMM H(s
z] coeff: 1
SMM H| coeff: 1
Count coeff: 10
Count
Count coeff: 0.1, 1, 10
Histogram bin width: 0.05
ICM
Learning rate: Ic-3
Learning rate: Ic-4, Ic-3, 1c-2
SMM, SAC
SAC reward scale: 5
SAC reward scale: le-2, 0.1, 1, 5, 10, 100
SMM
SMM H[s coeff:
SMM H[s 8 coeff: 1, 10, 100, 250, 500, Ic3
C'Caw
Count coeff:
Count coell: 1, 10
Count
(Fig.
Histogram bin width: 0.05
Histogram bin width: 0.05, 0.1
Pseudocount coeff:
Pseudocount coeff: 1, 10
Pseudocount
VAE le-3
VAE Ir: le-1, le-2, le-3
Learning rate: 1e-3
Leaminn rate: le-2, le-3, le-4
ICM
VAE Ir. le-1
VAE Ir: le- 1, 1e-2, le-3
Batch size: 512
Batch size: 128, 512, 1024
# SAC updates per step:
SAC updates per step: 1,4
GAIL.
Discriminator input: Sugerr
Discriminator input: 8, Subjer
Training iterate: le5
Training iterate: le5, 2e5. 3e5. 9e5, le6
# State Samples: le4
State Samples: le4
21


Reviews:
[{"review_id": "Ske-6qohFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes to frame exploration in reinforcement learning as a distribution matching problem. More specifically, the proposed method (SMM) aims to minimize the reverse KL between the state distribution induced by the policy and a desired state distribution. The desired state distribution can be used to guide exploration (e.g. by penalizing bad states) or can be chosen uniformly (resulting in a policy that maximizes state entropy).\nSMM iteratively optimizes the policy and a model for its induced state-distribution. The latter is used for approximating the policies state entropy. The algorithm is framed as a two-player zero-sum game between a \"policy player\" and a \"density player\". In order to justify that one of the players is assumed fixed while optimizing the other player, SMM optimizes the players against a historical average of the opponent. Such approach is known as fictitious play in game theory and ensures convergence to a Nash equilibrium.\nThe density player maximizes the likelihood of the states encountered during all previous roll-outs using a VAE. The policy update is framed as a standard reinforcement learning problem where the log-pdf of the target distribution serves as reward and the log-pdf of the model acts as cost. For practical reasons, the policy update does not consider an historical average but only uses the most recent density model. As the learned exploratory policy should correspond to an historical average exploration for the downstream task is achieved by sampling one of the learned policy at the beginning of each roll-out. The historical averaging also benefits prior exploration methods.\nNotably, the appendix also describes a modification that learns a mixture of policies in each iteration where a discriminator approximates the responsibilities for each component which is used as additional reward to divide the search space among the components. SMM is compared to prior exploration methods, ICM and Pseudo-Counts and standard SAC, on a simulated robotic box-pushing task and against SAC on simulated point-mass problem and a real-robot valve turning task.\n\nSignificance:\nEfficient exploration is arguably the main challenge of reinforcement learning as providing shaped reward functions is difficult even for experts and may lead to undesired behavior. I think that maximizing the state-entropy (or matching target state-distributions if available) is a sound objective for learning exploratory policies and the paper could, thus, be of relatively broad interest. \n\nNovelty:\nMaximizing the state entropy for exploration has already been used by (Hazan et al., 2018, reference from manuscript). However, in contrast to this prior work, SSM does not need to learn oracles that predict optimal state-distributions/policies for any given policies/reward functions. While distribution matching is a common approach to imitation learning, it has been little employed for manually specified distributions. Still, a similar objective has been used to replace reward functions by desired distributions in a RL-like setting [1] (not for exploration). However, their approach is quite restricted by assuming Gaussian target distributions.\n\nSoundness:\nIf I understand correctly, fictitious play assumes optimal responses to understand that the state distribution would often be more important than provably converge to a Nash equilibrium. The paper fails to provide stopping criteria for the optimization steps of the individual players; however, I assume that only few gradient steps are used for practical reasons. Hence, I am not sure whether the actual algorithm can be justified by the theory.\n\nThe paper mentions that VAEs are used to model the state distribution. Given that VAEs are not likelihood-based, I do not understand how the reward term log(q(s)) can be computed. \n\nClarity:\nThe paper is well-written and the structure seems fine. \nI think that the density estimation should be better discussed. Models of the state distribution of the policy are often highly desirable, not only for optimizing its entropy, but also, for example, for importance sampling. However, modeling the state distribution is also inherently difficult--especially for large state spaces.\n\nExperiments:\nI like that the paper uses a real robot experiment and the ablations with respect to the historical averaging are interesting. Unfortunately, the paper only compares to standard SAC on the hallway task and on the real robot task. I would not consider the entropy regularization of SAC a proper baseline for a proposed exploration technique.\n\nQuestions:\nIs the same extrinsic reward, i.e. log(p*), also provided to the other exploration methods?\n\nSome imitation learning methods such as GAIL are also able to match desired state-distributions. I think that these methods could be in principle also applied to the setting considered in the manuscript by using samples from the desired distribution as demonstrations. The paper briefly mentions that IRL methods are not applicable because they require access to trajectories, however, the discriminator of GAIL is only trained on individual state samples. I also do not see a problem of providing unachievable demonstrations to such imitation learning methods because, such like SSḾM, they would try to minimize the divergence. I think that GAIL would actually be an important baseline for the proposed method.\n\nHow does the method scale with the dimensions of the state? SSM has only be evaluated on relatively low-dimensional problems (compared to some rllab/mujoco tasks with >100 states). I would assume that obtaining meaningful density estimates in such settings might be problematic. May imitation learning methods based on discriminators actually be more promising?\n\nSMM only considers matching state distributions. If I understand correctly, the approach could be easily generalized to state-action distributions, correct? Wouldn't it make sense for some tasks to also explore in the action space?\n\nDecision:\nI like the paper in general. The optimization problem seems well-motivated and the algorithm seems reasonable fine. I also like that the paper includes a real robot experiment. However, I am not convinced by the derivation of SMM based on fictitous play and I think that it should be better evaluated with respect to existing exploration methods and also with respect to imitation learning methods. I am therefore slightly leaning towards rejection, currently. \n\nTypo:\n\"Note[sic] only does SMM explore a wider range of angles than SAC, but its ability to explore increasing[sic] throughout training, suggesting that the SMM objective is correlated with real-world metrics of exploration.\"\n\n[1] Arenz, Oleg, Hany Abdulsamad, and Gerhard Neumann. \"Optimal control and inverse optimal control by distribution matching.\" 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgrEg0sKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I thank the authors for their response and I think the added baselines and more in depth discussion of prior work have improved the paper. However, given the limited novelty of the technical contribution, I believe the experimental section should be further extended (i.e. add a wider variety of domains and make thorough comparisons to relevant methods) in order to draw more general and robust conclusions. \n\nSummary:\n\nThis paper proposes tackles the problem of exploration in RL, with a focus on learning an exploration policy that can be used for a variety of different tasks. They introduce a formal exploration objective that promotes efficient exploration and provides a mechanism for injecting prior knowledge about the task. They also design a practical algorithm to maximize the exploration objective. Finally, they empirically show that  their method can outperform other SOTA exploration methods on challenging exploration tasks for locomotion and manipulation,  both in simulation and in the real world.\n\nMain Comments:\n\nI’ve found the mathematical formulation to be sound and the empirical evaluation convincing. Overall, the paper is clearly written and the authors are quite transparent about the assumptions made. In addition, the problem of learning exploration strategies that are task-agnostic and force the agent to effectively explore within each episode (since the goal or task is not observed) is an important problem for RL and perhaps a more realistic setting than the single fixed-task one. However, I believe some important methodological details are missing from the paper and the empirical evaluation can be improved. In particular, the paper would be more convincing if it contained comparisons against SOTA exploration (e.g. curiosity-driven, pseudo-counts, noisy-networks etc.) and inverse reinforcement learning (e.g. GAIL) methods for all the environments. Such baselines are completely missing in the Navigation and Real-World tasks. \n\nHowever, as the authors note, most baselines used for comparison have been designed specifically to learn from sparse rewards in single task settings and do not have any direct mechanisms for including priors or learning to explore well for any task from some distribution. So I wonder if it’d make sense to include baselines that do make use of prior knowledge such as IRL (i.e. GAIL) or some other state-matching approach. Those could be more appropriate and powerful baselines. \n\nAnother potential disadvantage of this method seems to be the need for a prior, which may be difficult to design or even lead to suboptimal policies if it is not well designed. However, as the authors note, it is still a weaker requirement than having access to demonstrations for example and the  prior could potentially be learned from human preferences / restricted feedback. \n\nOther Comments  /  Questions:\n\n1. SMM uses prior information about the task in the form of the target distribution. Given this, I am worried that the baselines have a clear disadvantage. Did you do anything to provide the baselines with the same type of prior knowledge about the task? It would be useful to see how they would compare if they had access to the task prior (in some way) as well. \n\n2. Can you provide more insights into how this differs from variants of MaxEntRL and InvRL? Both analytically and in practice. I believe a more extended discussion of this would be valuable for readers and would alleviate some of the concerns regarding the novelty of this contribution and how its place in the broader literature.\n\n3. In the Navigation environment, how would the  results change if the goal were visible  (i.e. part of the agent’s observation)? I believe that most baselines would consider that scenario and it would  be interesting to see whether the qualitative conclusions hold or not in that case. I would expect other exploration methods to  be faster in that case.\n\n4. I also wonder if perhaps the reward is actually not that sparse  in some of these tasks but  because it is not visible, it makes the problem much harder for the baselines, which were designed to deal with very sparse reward. Can you comment on the reward sparsity in these tasks?\n\n5. At the top of page 2, you mention that there is a training phase in which the agents learn to optimize the exploration objective and at test time, it is trained with extrinsic reward. Can you please clarify on how these stages reflect in the results and what is  the regime used for the other baselines? Are they also pretrained on a variety of tasks with only their exploration bonus / intrinsic reward and then fine-tuned with extrinsic reward?\n\n6. In Figure 2 (c), why is it that the gap between SMM and SAC decreases as the  number  of halls increases? This seems counterintuitive and I would’ve expected to increase since I do not see why SAC would get better and SMM would get worse. \n\n7. How do you learn the density model? You mention the use of a VAE but the details of how this is trained are not specified.\n\n8. On page 5 before section 4,  you mention that you approximate the historical average  of the density model with the most recent iterate. Can you include ablations on how good this approximation is and how the results change  if you were using the historical average instead?\n\n9. In Figure 4 (b), SMM’s variance of the positive value of the angle differs significantly from the negative one. This strikes me as counterintuitive. Do you have any  intuition on why that is?\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BklTj6E6tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes to optimize the state marginal distribution to match a target distribution for the purposes of exploration. This target distribution could be uniform or could encode prior knowledge about downstream tasks. This matching can be done by iteratively fitting a density model on the historical data from the replay buffer, and training a policy to maximize the log density ratio between the target distribution and the learned density model. Experiments are performed on two domains: a simulated manipulation task and a real robotic control task. Overall, the paper is well-written.\n\n\n### Review\n\nRecommendation: weak reject for the reasons below. The main reason is that this paper ignores very similar prior work which is not properly credited. \n \n\nThe algorithm proposed here is very similar to the algorithm proposed in [1]:\n- the objective proposed in  equations (1) and (2) is the same as the second objective in Section 3.1 of [1]. \n- Algorithm 1 here is almost identical to Algorithm 1 in [1]\n\nThe work of [1] is only briefly mentioned in the related work section, and from the description there seems to be a fundamental misunderstanding of it. \nIt says \"their proposed algorithm requires an oracle planner and an oracle density model, assumptions that our method will not require\".\n\nMaking oracle assumptions is a tool for proving theoretical results, not a feature of an algorithm. An oracle can be any subroutine that one has reason to believe works reasonably well, and how well it works or not is captured in its accuracy parameter (usually \\epsilon).\nThey are used to break down a  more complex algorithm into simpler subroutines (called oracles), and deriving a guarantee on the complex algorithm in terms of the quality of the oracles. \nFor example, [1] assumes a density estimation oracle, which could be instantiated as a kernel density estimator, a VAE, count-based density estimation in the tabular case, etc. \nIt also assumes a planning oracle, which could be instantiated using any method for learning a policy (PPO, SAC, policy iteration, etc), or some search method if the environment is deterministic. \nThe accuracy of the oracles are reflected in the \\epsilon_0 and \\epsilon_1 parameters, which then show up the guarantee for theorem 1. \n\nTheorem 1 of [1] also shows that the entropy of the policy mixture (i.e. replay buffer) matches the maximum entropy over the policies in the policy class, which is one of the main theoretical claims of the work here. \n\nGiven this, I don't see this paper as making any new algorithmic or theoretical contributions. On the other hand, [1] had a very limited empirical evaluation and it would be valuable to have a more thorough empirical investigation of this type of method in the literature. This paper partially does that in the sense that they apply more modern methods (VAEs rather than counts/kernel density estimators) on more complex tasks (a simulated manipulation task and a real robot), and their experiments seem well-executed with proper comparisons. However, since the primary contribution of this paper seems to be empirical, I don't think the current experiments on two domains are enough. \n\nI think this paper could be fine for publication with a fairly significant rewrite placing it in the context of prior work, and expanding the experimental section. \nMy suggestions are to add experiments on several other continuous control tasks (Mujoco/Roboschool) as well as hard exploration Atari games (Montezuma, Freeway, Pitfall etc), to see how well the density estimation works in pixel domains (and the effect of historical averaging). I would be willing to raise my score if these changes can be made within the rebuttal period.   \n\n\n[1] \"Provably Efficient Maximum Entropy Exploration\", Hazan et. al. ICML 2019. \nhttp://proceedings.mlr.press/v97/hazan19a/hazan19a.pdf\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a nice approach to optimizing marginals to improve exploration for RL agents.  The reviewers agree that its improvements w.r.t. the state of the art do not merit a publication at ICLR.  Furthermore, additional experimentation is needed for the paper to be complete.
Decision: Reject