Learning What To Do by Simulating the Past
Authors: David Lindner
Rohin Shah
Pieter Abbeel
Anca Dragan
Venue: ICLR 2021 Poster (2021)
Keywords: imitation learning
reward learning
reinforcement learning

Abstract: Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state. Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale it to complex tasks? In this work, we show that by combining a learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce a specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.

Published as a conference paper at ICLR 2021
LEARNING WHAT To Do BY SIMULATING THE PAST
David Lindner*
Rohin Shah, Pieter Abbeel & Anca Dragan
Department of Computer Science ETH Zurich
Center for Hlman-Compation Al UC Berkeley (rohi imshahn wwwps/ssshaahaaeeeeaaa pabbeel, anca wwwereeleyy.c.oo edu
david. www.llinnddiinin etwze ch
ABSTRACT
Since reward functions are hard to specify, recent work has focused learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state (Shah eet al., 2019). Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale to complex tasks? In this work, we show that by combining learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.
INTRODUCTION
As deep learning has become popular, many parts of Al systems that were previously designed by hand have been replaced with learned components. Neural architecture search has automated architecture design (Zoph & Le 2017: Elsken et al. 2019), popplation- -bsse training has automated hyperparameter tuning (Jaderberg et al., 20170. and self-supervised learning has led to impressive
results in language modeling (Devlin et al. 2019) Radford et al. 2019 Clark et al. 2020) and
reduced the need for labels in image classification (Oord et al. 2018 He al. 20200] Chen et al.)
2020). However, in reinforcement learning, one component continues to be designed by humans: the task specification. Handcoded reward functions are notoriously difficult to specify (Clark & Amodei 2016; Krakovnal 2018), and learning from demonstrations (Ng al. 2000 Fu 20118) or preferences (Wirth et al. 2017 Christiano et al.) 2017) requires a lot of human input. there the specification of what must be done?
way that we can automate
turns out that we can learn part of what the user wants simply by looking at the state of the environment: after all, the user will already have optimized the state towards their own preferences (Shah et al.) 2019). For example, when robot is deployed in a room containing an intact vase, can reason that if its user wanted the vase be broken, it would already have been broken; thus she probably wants the vase to remain intact.
However, we must ensure that the agent distinguishes between aspects of the state that the user couldn't I control from aspects that the user deliberately designed. This requires us to simulate what the user must have done lead to the observed state: anything that the user put effort into in the past is probably something the agent should do as well. As illustrated in Figure T if we observe Cheetah balancing on its front leg, we can infer how must have launched itself into that position. Unfortunately. it is unclear how to simulate these past trajectories that lead to the observed state. So far, this has only been done in gridworlds, where all possible trajectories can be considered using dynamic programming (Shhh et al. 2019).
Our key insight is that we can sample such trajectories by starting at the observed state and simulating backwards in time. To enable this, derive gradient that is amenable to estimation through backwards simulation, and learn an inverse policy and inverse dynamics model using supervised
Work done the Center for Human-Compatible AT, UC Berkeley.

---

Published as a conference paper at ICLR 2021
Figure I: Suppose we observe Cheetah balancing on its front leg (left). Given a simulator for the environment, Deep RLSP able infer how the cheetah must have acted to end up in this position. can then imitate these actions in order to recreate this skill. Note that the state contains joint velocities addition to positions, which makes the task more tractable than this picture might suggest.
learning to perform the backwards simulation. Then, the only remaining challenge is finding reward representation that can be meaningfully updated from a single state observation. To that end, rather than defnning the reward directly on the raw input space, we represent it as a linear combination of features learned through self-supervised representation learning. Putting these components together, we propose the Deep Reward Learning by Simulating the Past (Deep RLSP) algorithm.
We evaluate Deep RLSP MuJoCo environments and show that it can recover fairly good perfor- mance on the task reward given access to a small number of states sampled from a policy optimized for that reward. We also use Deep RLSP to imitate skills generated using skill discovery algorithm (Sharma et al. 2020), in some cases given just a single state sampled from the policy for that skill.
Information from the environment state cannot completely replace reward supervision. For example, would be hard to infer how clean Bob would ideally want his room to be, the room is currently messy because Bob is too busy clean it. Nonetheless, environment state can be used to significantly reduce the burden of human supervision required to train useful, capable agents.
optimistic that information from the
METHOD
In this section, we describe how Deep RLSP can learn reward function for high dimensional environments given access only to simulator and the observed state s0.
Notation. A finite-horizon Markov Decision Process (MDP) M (S, contains a set of states S and a set of actions A. The transition function T : S A 10 determines distribution over next states given a state and an action, and P a prior distribution over initial states. The reward function
S / R determines the agent's objective. T E Z_ is finite planning horizon. A policy π x A + [0, 1] specifies how to choose actions given a state. Given an initial state distribution, policy and the transition function, first state from PP every subsequent action from π, and every subsequent state from T. We denote the probability distribution over trajectories as (P, म, T) and write T ~ (P, ग, T) for the sampling step. We will sometimes write single state instead of a distribution P if the initial state is deterministic. The goal of reinforcement learning (RL) is to find a policy x that maximizes the expected cumulative reward (P,,7, (.-rr(ss))
sample trajectory T by sampling the
We use Φ S > Rd to denote a feature function (whether handcoded or learned) that produces a feature vector of length for every state. The reward function is linear & if it can be expressed in the form r(s) 0 ((s) for some 0 E R".
We assume that some past trajectory T- 8 T-- ... 188 produced the observed state 80-
2.1 IDEALIZED ALGORITHM
We first explain what would ideally do, if we had handcoded feature function & and an enumerable (small) state space S that affords dynamic programming. This is a recap of Reward Learning by Simulating the Past (RLSP: Shah al. 2019).
2

---

Published as a conference paper at ICLR 2021
We assume the human follows selizmann-rrationia policy (a s, exp(Qi(s, a; a)) where the Q values are computed using soft value iteration. Marginalizing over past trajectories, yields a distribution over the observed state p(so 0) -E_ p(T --ra-7.. 15 0). We com- pute the maximum likelihood estimate, argmax, In p(so 8), via gradient ascent, by expressing the gradient of the observed state as weighted combination of gradients of consistent trajectories (Shah) ct al. 2019 Appendix B):
Vo lnso 0)
E
Vo In p(r [))
Ve p(T 8) is a gradient for inverse reinforcement learning. Since we assume Boltzmann-rational human, this is the gradient for Maximum Causal Entropy Inverse Reinforcement Learning (MCEIRL; Ziebart et al. 2010). However, we still need to compute an expectation over all trajectories that end in So. which is in general intractable. (Shah al. (2019) use dynamic programming to compute this gradient in tabular settings.
2.2 GRADIENT AS BACKWARD BACKWARDS-TORWARDD CONSISTENCY
Approximating the expectation. For higher-dimensional environments, we must approximate the expectation over past trajectories p(r-T:-1 so, 0). We would like to sample from the distribution, but it is not clear how to sample the past conditioned on the present. Our key idea is that just as we sample the future by rolling out forwards in time, should be able sample the past by rolling out backwards in time. Note that by the Markov property we have:
-1
PTT-T:-1 80:0) = II p(st ae, st-1...... s0, 0)p(pc 5c+1. 0............30..
t-
II p(st a 01-111)pp((a s)+1,
Thus, given the inverse policy Ti (a: s1+1.0), 0), the inverse dynamics T, (st ag St-1. 0). and the observed state 8p, we can sample a past trajectory T-T:-1 ~ P(T-T:-1 8p, 0) by iter- atively applying -- and T-1, starting from 8p- Analogous to forward trajectories, we express
the sampling as T-T:-1 (8p. π T-1). Thus, we can write the gradient in Equation as
Er
1) Ve Im 0)].
Learning -- and T-1. In order to learn -_ must first know We assumed that the
human was Boltzmann-rational which corresponds to the maximum entropy reinforcement learning objective (Levine) 2018). We use the Soft Actor-Critic algorithm (SAC; Haarnoja al., (2018) to estimate the policy xa s, f), since it explicitly optimizes the maximum entropy RL objective.
Given the forward policy x(a 8; and simulator T, can construct dataset of sampled forward trajectories, and learn the inverse policy ग and the inverse dynamics T-1 using supervised learning. Given these, we can then sample T-T:-1- allowing In general, both x and T-1 could be stochastic and -ime-dependent.
approximate the expectation in the gradient.
Estimating the gradient for trajectory. We now turn the term within the expectation, which is the inverse reinforcement learning gradient given a demonstration trajectory 7= S_T__-T-...... ....0 80-
Assuming that the user is Boltzmann-rational this is the MCEIRL gradient (Ziebart et al. 2010). which can be written as (Shah et al. 2019 Appendix A):
a Σ ((s))) - -F-r(s-T)+
Volup(T(0) =
R 1=-T Thiaaa))
Σ
[Fer(s4+1)] - FF41(8641)
(2)
F is the expected feature count under n, that is. F-o(s-e) & Er ~ (1-17,7)) Ei_
The first term computes the feature counts of the demonstrated trajectory T, while the second term computes the feature counts obtained by the policy for the current reward function 0 (starting from
3

---

Published as a conference paper at ICLR 2021
the initial state s-r). Since rss) 6 ((s). these terms increase the reward of features present in the demonstration T and decrease the reward of features under the current policy. Thus, the gradient incentivizes consistency between the demonstration and rollouts from the learned policy.
The last term is essentially a correction for the observed dynamics: if we see that st, ag led to st+1, corrects for the fact that "could have" seen some other state 5/,41- Since this correction is zero in expectation (and expensive to compute), we drop it for our estimator.
Gradient estimator. After dropping the last term in Equation 2 expanding the definition of F, and our final gradient estimator is:
substituting in to Equation [
( DO Σ ((s)) p((e))
-
]
V8 In p(50 =
E T- (
E
Σ (
(3)
2 24
~' (o-
Thus, given so, 0, n, T, 11, and T-1, computing the gradient consists of three steps:
1. Simulate backwards from so. and compute the feature counts of the resulting trajectories.
2. Simulate forwards from 8-7 of these trajectories, and compute their feature counts.
3. Take the difference between these two quantities.
This again incentivizes consistency, this time between the backwards and forwards trajectories: the gradient leads to movement towards "what the human must have done" and away from "what the human would do if they had this reward". The gradient becomes zero when they identical.
may seem like the backwards and forwards trajectories should always be consistent with each other, since x- and T-1 are inverses of and T. The key difference is that imposes constraints on the backwards trajectories, but not on the forward trajectories. For example, suppose we observe 8p in which vase is unbroken, and our current hypothesis is that the wants break the vase. When we simulate backwards, our trajectory will contain an unbroken vase, but when we simulate forwards from T. π will break the vase. The gradient would then reduce the reward for broken vase and increase the reward for an unbroken vase.
2.3 LEARNING A LATENT MDP
Our gradient still relies on feature function é, with the reward parameterized as r(s) 0 (' ((s). natural way to remove this assumption would be to instead allow 0 to parameterize neural network, which can then learn whatever features are relevant to the reward from the RLSP gradient.
However, this approach will not work because the information contained in the RLSP gradient is insufficient to identify the appropriate features to construct: after all, derived from a single state. If we were to learn a single unified reward using the same gradient, the resulting reward would likely be degenerate: for example, it may simply identify the observed state, that is R(s) = 1[s = su]-
Thus, we continue assume that the reward is linear in features, and instead learn the feature function using self-supervised learning (Oord et al.) 2018 He et al.) 20200. In our experiments, we use a variational autoencoder (VAB; Kingma & Welling (014) learn the feature function. The VAE encodes the states into a latent feature representation, which we can use to learn a reward function the environment is fully observable, i.c., the states contain all relevant information.
For partially observable environments recurrent state space models (RSSMs; Karl et al. 2017 Doerr al. 2018 Buesing et al. 2018 Kurutach al. 2018 Hafner et al. 2019 2020) could be used instead. These methods aim to learn latent MDP, by computing the states using recurrent model over the observations, thus allowing the states encode the history. For such a model, we can imagine that the underlying POMDP has been converted into latent MDP whose feature function ő the identity. We can then compute gradients directly this latent MDP.
2.4
DEEP RLSP
Putting these components together gives the Deep RLSP algorithm (Algorithm I.. We first learn feature function ø using supervuee learning, and then train an inverse dynamics model T-! all using dataset of environment interactions (such random rollouts). Then, we update 0 using
4

---

Published as a conference paper at ICLR 2021
The DEEP RLSP algorithm. The initial dataset of environment interactions D can be constructed in many different ways: random rollouts, human play data, curiosity-driven exploration, etc. The specific method will determine the quality of the learned features.
Algorithm 1
procedure DEEP RLSP((su). 7)
D & dataset of environment interactions Initialize o.. 0d. म, ग TT-, randomly.
dc, Ra SelfSupervisedl Lernnnng ())
D. Train encoder and decoder for latent MDP
Initialize experience exprrieee replay E with data D.
T-1 Supervised
Train inverse dynamics >> Start horizon at 1
T++1
for i [1..num _aam_e_ do
π t SAC(#)
> Train policy
Supervised &e. E)
Train inverse policy D Update 0
0 4 0 + x COMPUTEGRADI
n. 1.=-,,1-,, de)
if gradient magnitudes are sufficiently low then
T ++
Advance horizon
return 6, de
procedure COMPUTEGRAD(( T, T
Tbackward } + Rollout( {5o) T----111 )
b Simulate backwards from so Compute backward feature counts
bbccwaadd
AvvrragFFatturCCumni @e (Tackward))
8-T)
FinalStates( Tauckward)
Tlowwrdd
Rollout( (s-r)
Simulate forwards from 8-r D Compute forward feature counts
Fforward -
AverageFeatuurCounns/ {Tfowaardd
Relabel [backward] ] and add them E.
return backward
Ptorward
Equation and continually train æ, and - alongside 0 to keep them up to date, The full algorithm also adds a few bells and whistles that we describe next.
Initial state distribution P. The attentive reader may wonder why our gradient appears to be independent of P. This is actually not the case: while and T are independent of P, π - and T-1 do depend on it. For example, we observe Alice exiting the San Francisco airport, the corresponding should hypothesize different flights if she started from New York than if she started from Tokyo.
However, in order to actually produce such explanations, we must train π 1 and T solely on trajectories of length T starting from s-r ~ P. We instead train n- and T-1 trajectory data, which loses the useful information in P, but leads to several benefits. First, we can train the models on exactly the distributions that they will be used on, allowing us avoid failures due to distribution shift. Second, the horizon T is no longer critical: previously, T encoded the separation in time between s-T and sg, and result misspecification of T could cause bad results. Since we now only have information about so, it doesn't 't matter much what we set T to, and as result we can use
variety of
curriculum (discussed next). Finally, this allows Deep RLSP be used in domains where an initial state distribution is not available.
Note that we are no longer able use information about through - and T-1, However, having information about P might be crucial in some applications to prevent Deep RLSP from converging to a degenerate solution with s-7 = 80 and a policy æ that does nothing. While we did not find this to be a problem in our experiments, we discuss a heuristic to incorporate information about s into Deep RLSP in Appendix
Curriculum. Since the horizon T is longer crucial, we can it to provide curriculum. We initially calculate gradients with low values of T', to prevent compounding errors in our learned models, and making it easier to enforce backwards-forwrdd consistency, and then slowly grow T, making the problem harder. In practice, we found this crucial for performance: intuitively, it is much easier to make short backwards and forwards trajectories consistent than with longer trajectories; the latter would likely have much higher variance.
Multiple input states. If get multiple independent 8o as input, average their gradients.
5

---

Published as a conference paper at ICLR 2021
Experience replay. We maintain experience replay buffer E that persists across policy training steps. We initialize E with the same set of environment interactions that the feature function and inverse dynamics model trained on. When computing the gradient. collect all backward and forward trajectories and add them to F. To avoid compounding errors from the inverse dynamics model, relabel all transitions using simulator of the environment. Whenever we'd add transition (s, a, s') to E, we initialize the simulator at and execute to obtain and add transition (31, a, E instead.
3 EXPERIMENTS
3.1 SETUP
To demonstrate that Deep RLSP can scaled to complex, continuous, high- nigh-dimenssonal environ- ments, we use the MuloCo physics simulator (Todorov et al. 2012). We consider the Inverted Pendu- lum, Half-Cheetabl and Hopper environments implemented in Open Gym (Brockman et al.] 2016). The hyperparameters of our experiments described in detail in Appendix We provide code to replicate our experiments at https:: htt///gittuuu com/ HumanCompat ibleAI
Baselines. To our knowledge, this is the first work to train policies using single state input. Due to lack of alternatives, we compare against GAIL (Ho & Ermon 2016) using the implementation from the imitation library (Wang et al.] 2020). For each state provide to Deep RLSP, provide a transition (s, a. to GAIL.
Ablations. In Section 2.2 we derived gradient for Deep RLSP that enforces consistency between the backwards and forwards trajectories. However, we could also ignore the temporal information altogether. If optimal policy led the observed state 8g, then is probably good bet that so is high reward, and that the agent should try to keep the state similar to Thus, we can simply set 0
o(sa) and not deal with -- and T-1 all. [[[(
How should we handle multiple states sb...., s Given that these are all sampled i.i.d. from rollouts of optimal policy, a natural choice is to simply average the feature vectors of all of the states, which we call Average
Alternatively, we could view each the observed states as a potential waypoint of the optimal policy, and reward an agent for being near any one of them. We o(s)) implement this Waypoints method as R(s) = max, T[[ (s)). Note that both of these ablations still require us to learn the feature function db
Feature learning dataset. By default, we use random rollouts to generate the initial dataset that is used train the features á and the inverse model T-1, (This is 1) Algorithm I However, the inverted pendulum environment, the pendulum falls very quickly in random rollouts, and T never learns what a balanced pendulum looks like. So, for this environment only, we combine random rollouts with rollouts from expert policy that balances the pendulum.
3.2 GRIDWORLD ENVIRONMENTS
As a first check, we consider the gridworld environments in Shah et al. (2019). In these stylized gridworlds, self-supervised learning should not be expected learn the necessary features. For example, in the room with vase environment, the two door features are just particular locations, with no distinguishing features that would allow self-supervised learning to identify these locations as important. So, we run Algorithm [ without the feature learning and instead use the pre-defined feature function of the environments. With this setup we are able to use Deep RLSP to recover the desired behavior from a single state in all environments in which the exact RLSP algorithm is able to recover it. However, AverageFeatures fails on several of the environments. Since only one state is provided, Waypoints is equivalent to AverageFeatures. It is not clear how to apply GAIL these environments, and so we do not compare to it. Further details on all of the environments and results can be found in Appendix A

---

Published as a conference paper at ICLR 2021
Environment
SAC
states
Deep RLSP
AverageFeatures
Waypoints
GAIL.
303 (299)
(2)
N/A
1000 (0)
Inverted Pendulum
1000
10
335 (333)
(1)
4 (1)
1000 (0)
50
339 (331)
6 (4)
3.7 (0.3)
1000 (0)
4591 (2073)
6466 (3343)
N/A
-288 (55)
Cheetah
13236
10
6917 (421)
6245 (2352)
-10 (23)
-296 (172)
(forward)
50
6078 (589))
4504 (2970)
126 (38)
-54 (295)
5730 (2733)
12443 (645)
N/A
-335 (46)
Cheetah (backward)
13361
10
7917 (249)
12829 (651)
-80 (388)
-283 (45)
50
7588 (171)
11616 (178)
-509 (87)
2113 (1015)
68 (8)
99 (45)
N/A
991 (9)
Hopper
3274
10
47 (21)
1599 (126)
58 (7)
813 (200)
(terminate)
50
72 (1)
65 (36)
14 (4)
501 (227)
1850 (634)
2537 (363)
N/A
990 (9)
Hopper (penalty)
3363
10
2998 (62)
3103 (64)
709 (133)
784 (229)
50
1667 (737)
2078 (581)
1612 (785)
508 (259)
Table 1: Average returns achieved by the policies learned through various methods, for different numbers of input states. The states are sampled from a policy trained using SAC on the true reward function; the return of that policy given as comparison. Besides the SAC policy return, all values are averaged over 3 seeds and the standard error is given in parentheses. We don't 't report Waypoints on state as it is identical to AverageFeatures on 1 state.
3.3 SOLVING THE ENVIRONMENTS WITHOUT ACCESS TO THE REWARD FUNCTION
First we look at the typical target behavior in each environment: balancing the inverted pendulum, and making the half-cheetah and the hopper move forwards. Additionally we consider the goal of making the cheetah run backwards (that is, the negative of its usual reward function). We aim to use Deep RLSP to learn these behaviors without having access to the reward function.
We train a policy using soft actor critic (SAC: Haarnoja et al. 2018 optimize for the true reward function, and sample either 1, 10 or 50 states from rollouts of this policy to use as input. We then use Deep RISPP to infer a reward and policy. Ideally we would evaluate this learned policy rather than reoptimizing the learned reward, since learned reward models can often be gamed (Stiennon et al.) 2020), but it would be too computationally expensive to run the required number of SAC steps during each policy learning step. As result, we run SAC for many more iterations on the inferred reward function, and evaluate the resulting policy on the true reward function (which Deep RLSP does not have access to).
Results are shown in Table In Hopper, we noticed that videos of the policies learned by Deep RLSP looked okay, but the quantitative evaluation said otherwise. It turns that the policies learned by Deep RI.SP do jump, as we might want, but they often fall down, terminating the episode; in contrast GAIL policies stand still fall over slowly, leading to later termination and explaining their better quantitative performance. We wanted to also evaluate the policies without this termination bias, and so we evaluate the same policies in environment that does not terminate the episode, but provides negative reward instead; in this evaluation both Deep RLSP and AverageFeatures perform much better. We also provide videos of the learned policies at ptstts::////......tt sites..sss......ee...eeee.e.eets . google. com/ view/ sp which show that the policies learned by Deep RL.PP do exhibit hopping behavior (though with strong tendency to fall forward).
deep-rl
GAIL is only able to learn a truly good policy for the (very simple) inverted pendulum, even though it gets states and actions as input. Deep RLSP on the other hand achieves reasonable behavior (though clearly not expert behavior) in all of the environments, using only states as input. Surprisingly, the AverageFeatures method also performs quite well, even beating the full algorithm on some tasks, though failing quite badly on Pendulum. It seems that the task of rumning forward backward is very well specified by a single state, since it can be inferred even without any information about the dynamics (except that which is encoded in the features learned from the initial dataset).
7

---

Published as a conference paper at ICLR 2021
Existing skill
Single state
Sample
Policy learned by Deep RLSP
Deep RLSP
Figure 2: We sample few states from policy performing specific skill to provide as input. Here, Deep RLSP learns to balance the cheetah on the front leg from a single state. We provide videos the original skills and learned policies at: https: //sits//ttssss.. google. com/v v ew http//iiwwddeeeeeeeeee deep- seeep-lssp
3.4 LEARNING SKILLS FROM A SINGLE STATE
We investigate to what extent Deep RLSP can learn other skills where the reward is not clear. Evaluation on these tasks is much harder, because there is ground truth reward. Therefore evaluate qualitatively how similar the policies learned by Deep RLSP are to the original skill. We also attempted to quantify similarity by checking how quickly discriminator could learn to distinguish between the learned policy and the original skill, but unfortunately this metric was not conclusive (results reported in Appendix[D.I] Unlike the previous case, do not reoptimize the learned reward and only look at the policies learned by Deep RLSP.
We consider skills learned by running Dyneeics-Aware Unsupervised Discovery of Skills (DADS; Sharma et al., (020).. Since we are not interested in navigation, we remove the "x-y prior" used to get directional skills in DADS. We run DADS on the half-cheetah environment and select all skills that are not some form of running. This resulted in two skills: one in which the cheetah is moving forward making big leaps ("jumping" ") and one in which is slowly moving forward on one leg ("balancing" "). As before we roll out these policies and sample individual states from the trajectories provide as an input for Deep RLSP. We then evaluate the policy learned by Deep RLSP. Since the best evaluation here is to simply watch what the leamed policy does, we provide videos of the learned policies at /tttts://sitt visualizations in Appendix D.2
google. le.olll..o com/ vivvewww deep- rlsp. We also provide
The first thing to notice is that relative to the ablations, only Deep RLSP close to imitating the skill. None of the other policies resemble the original skills at all. While AverageFeatures could perform well on simple tasks such as running, the full algorithm is crucial to imitate more complex behavior.
Between Deep RLSP and GAIL the comparison is less clear. Deep RLSP can learn the balancing skill fairly well from a single state, which we visualize in Figure 2 (though we emphasize that the videos much clearer). Like the original skill, the learned policy balances moves forward by jumping, though with slightly more erratic behavior. However, the learned policy sometimes drops back its feet falls over on its back. We suspect this is an artifact of the short horizon (T" < 10) used for simulating the past in our algorithm. A small horizon is necessary to avoid compounding errors in the learned inverse dynamics model, but can cause the resulting behavior to be more unstable on timescales greater than T. We see similar behavior when given 10 or 50 states. GAIL leads a good policy given a single transition, where the cheetah balances on its front leg and head (rather than just the front leg), but does not move forward very much. However, with 10 or 50 transition, the policies learned by GAIL. do not look at all like balancing.
leg and slowly
However, the jumping behavior is harder to learn, especially from single state. We speculate that here a single state is less informative than the balancing state. In the balancing state, the low joint velocities tell us that the cheetah is not performing flip, suggesting that we had optimized for this specific balancing state. On the other hand, with the jumping behavior, we only get a single state of the cheetah in the air with high velocity, which is likely not sufficient to determine what the jump looked like exactly. In line with this hypothesis, at state Deep RLSP learns to erratically hop. at 10 states it executes slightly bigger jumps, and 50 states it matches the original skill relatively closely.
8

---

Published as a conference paper at ICLR 2021
The GAIL policies for jumping are also reasonable, though in different way that makes it hard to compare. Using 1 or 10 transitions, the policy doesn't move very much, staying in contact with the ground most of the time. However, at 50 transitions, it performs noticeably forward hops slightly smoother than the policy learned by Deep RI.P..
4 RELATED WORK
Learning from human feedback. Many algorithms aim to learn good policies from human demon-
strations, including ones in imitation learning (Ho & Ermon 2016) and inverse reinforcement learning
(IRL:Ng et al. 20000 Abbeel & Ng. 22044] Fu al. (2018). Useful policies can also be learned from
other types of feedback, such as preferences (Christiano al. 2017). corrections (Bajesy et al. 2017))
instructions (Bahdanau et al.) 2019), or combinations of feedback modalities (Ibarz et al.) 2018))
While these methods require expensive human feedback, Deep RLSP instead simulates the tra- jectories that must have happened. This is reflected in the algorithm: in Equation I the inner gradient corresponds to an inverse reinforcement learning problem. While we used the MCEIRL formulation (Ziebart et al. (2010), other IRL algorithms could be used instead (Fu et al.) 2018).
Learning from observations. For many tasks, we have demonstrations without action labels, e-g.. YouTube videos. Learning from Observations (LfO; Torabi et al., 2019; Gandhi et al.) (2019) aims to recover policy from such demonstrations. Similarly to LfO, we do not have access to action labels, but our setting is further restricted to observing only a small number of states.
LIMITATIONS AND FUTURE WORK
Summary. Learning useful policies with neural networks requires significant human effort, whether it is done by writing down a reward function by hand, or by learning from explicit human feedback such as preferences demonstrations. We showed that it is possible to reduce this burden by extracting "free" information present in the current state of the environment. This enables us to imitate policies in MuJoCo environments with access just few states sampled from those policies. We hope that Deep RLSP will help us train agents that are better aligned with human preferences.
Learned models. The Deep RLSP gradient depends having access good model of m, T, x=1 and T-1 In practice, it was quite hard to train sufficiently good versions of the inverse models. This could be significant barrier to practical implementations of Deep RLSP. It also be taken sign for optimism: self-supervised representation learning through deep learning is fairly recent and advancing rapidly; such advances will likely translate directly into improvements in Deep RLSP.
Computational cost. Imitation learning with full demonstrations can already be quite computation- ally expensive. Deep RLSP learns several distinct neural network models, and then simulates potential demonstrations, and finally imitates them. Unsurprisingly, this leads to increased computational cost.
Safe RL. Shah et al. (2019) discuss how the exact RLSP algorithm can be used avoid negative side-effects in RL by combining preferences learned from the initial state with reward function. While we focused on learning hard specify behavior, Deep RLSP can also be used to learn avoid negative side-effects, which is crucial for safely deploying RL systems in the real world (Amodeil ct al. 2016).
Multiagent settings. In any realistic environment, there is not just single "user" who is influencing the environment: many people act simultancously, and the state a result of joint optimization by all of them. However, our model assumes that the environment state resulted from optimization by single agent, which will not take into account the fact that each agent will have constraints imposed upon them by other agents. We will likely require new algorithms for such setting.
ACKNOWLEDGMENTS
This work was partially supported by Open Philanthropy, AFOSR, ONR YIP. NSF CAREER, NSF NRI, and Microsoft Swiss JRC. We thank researchers at the Center for Human- Comma--Coe Al and the InterACT lab for helpful discussion and feedback.

---

Published as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of International Conference on Machine Learning (ICML), 2004.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint 3.XIv:6666006500 2016.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Ed- ward Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations (ICLR), 2019.
Andrea Bajesy, Dylan P Losey, Marcia K O' O'Malley, and Anca D) Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning (CoRL), 2017.
Christopher M Bishop. Mixture density networks. Neural Computing Research Group Report, Aston University, 1994.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl gym. arXiv preprint orXiv:/606.01355 01540,, 2016
Lars Buesing, Theophane Weber, Sébastien Racaniere, SM Eslami, Danilo Rezende, David P Reichert, Fabio Viola, Frederic Besse. Karol Gregor, Demis Hassabis, et al. Learning and querying fast generative models for reinforcement learning. In FAIM workshop "Prediction and Generative Modeling in Reinforcement Learning". 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of International Conference Machine Learning (ICML), 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https::/blog openai. com/
www.:---wwwaaddddddaaeaaea functions
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning. Electra: Pre-training text encoders as discrininators rather than generators. In International Conference on Learning Representations (ICLR), 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Andreas DOeM,, Christian Daniel, Martin Schiegg, Duy Nguyen-Tuong, and Sebastian Trimpe. Probabilistic recurrent state-space models. In Proceedings of International Conference on Machine Learning (ICML), 2018.
Tuong, Stefan Schaal, Marc Toussaint,
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2018.
Sunil Gandhi, Tim Oates, Tinoosh Mohsenin, and Nicholas Waytowich. Learning from observations using a single video demonstration and human feedback. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, 2019.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy max- imum entropy deep reinforcement learning with a stochastic actor. In Proceedings of International Conference Machine Learning (ICML), 2018
10

---

Published as a conference paper at ICLR 2021
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of International Conference Machine Learning (ICML), 2019.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations (ICLR), 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: //tt::/gggtbb...// com/
hill-a/stee htt/stalle-Beeeeee baseee-aaliieee 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations Atari. In Advances Neural Information Processing Systems, 2018.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dumning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv: 1.09846, 2017.
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick Van der Smagt. Deep variational Bayes filters: Unsupervised learning of state space models from raw data. In International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representutions (ICLR), 2014.
Victoria Krakovna. Specification gaming examples in AI, 2018. nttps::s:///kkaaaoo eeamplleeeelless Ln-ai/]
wordpress . http:////////////////////cc.c...
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. In Advances in Neural Information Processing Systems, 2018.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint 100,009,,,,0099, 2018
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Proceedings of International Conference on Machine Leurning (ICML), 2000.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 107.v:180,,03748, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences implicit the state of the world. In International Conference on Learning Representations (ICLR), 2019.
Archit Sharma, Shane Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised skill discovery. In International Conference on Learning Representations (ICLR), 2020.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, 2020.
11

---

Published as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for control. In International Conference on Intelligent Robots and Systems (IROS), 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. In Imitation, Intent, and Interaction (13) Workshop at ICML, 2019.
Steven Wang, Sam Toyer, Adam Gleave, and Scott Emmons. The imitation library https://github.com
for imitation learning and inverse reinforcement learning. HumancComp.tibllll/iinttt.......
2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-44 2017.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of International Conference on Machine Learning (ICML), 2010.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.
12

---

Published as a conference paper at ICLR 2021
A GRIDWORLD ENVIRONMENTS
Here we go into more detail on the experiments in Section 3.2 in which we ran Deep RLSP on the environment suite constructed in Shah al. (2019).
In this test suite, each environment comes equipped with an observed state sp. an initial state s-T, a specified reward Ropee and true reward Ruve. A given algorithm should be also 8-7 and produce an inferred reward Rinterrdd This is then added to the specified reward to produce Rimal Rspec + Xinferred where À is a hyperparameter that determines the weighting between the two. An optimal policy for Ronal is then found using value iteration, and the resulting policy is evaluated according to Bene-
So and optionally
There is no clear way to set X: it depends on the scales of the rewards. We leverage the fact that Repee is deliberately chosen to incentivize bad behavior, such that we know À will always give incorrect behavior. So, we normalize Rintered, and then increase À from 0 until the behavior displayed by the final policy changes.
Since GAIL does not produce reward function as output, we do not run it here. We do however report results with AverageFeatures (which is equivalent Waypoints here, because there only a single observed state).
3XE
3CE
Figure 3: Reproduction of part of Figure 2 in Shah et al. (001 illustrating the gridworld environments that we test on.
From left to right, the environments are:
1. Room with vase: Rypse has weight for the purple door feature, and 0 for all other weights. Rave additionally has weight -1 for the broken vases feature. Since we observe state in which the vase is unbroken, we can infer that the human avoided breaking the vase, and so that there should be negative weight on broken vases. Deep RLSP indeed does this and so avoids breaking the vase. AverageFeaaturs fails to do so, though this due to quirk in the feature encoding. In particular, the feature counts the number of broken vases, and so the inferred has a value of zero for this feature, effectively ignoring it. If we change the fcaturization instead count the number of unbroken vases, then AverageFeatures would likely get the right behavior.
2. Toy train: In this environment, we observe a state which an operational train moving around a track. Once again, Rspee just has weight I on the purple door feature. Rime additionally has weight -1 on broken vases and trains. Deep RLSPP appropriately avoids breaking objects, but AverageFeatures does not.
3. Batteries: We observe a state which the human has put a battery in the train to keep operational (s-T has two batteries while so only has one). Ropec still has weight purple door feature. Ryme additionally has weight - on allowing the train to run out of power. Algorithms should infer that is good to put batteries in the train to keep operational, even though this irreversibly uses up the battery. Deep RLSP correctly does this, while AverageFeatures does not. In fact, AverageFeatures incorrectly infers that batteries should not be used up.
the
4. Apples: We observe a state in which the human has collected some apples and placed them in a basket. Rypee is always zero, while Rave has weight on the number of apples in the basket. The environment tests whether algorithms can infer that good for there to be apples in the basket. Deep RLSP does this, learning policy that continues to collect apples
13

---

Published as a conference paper at ICLR 2021
and place them in the basket. AverageFeatures also learns to place apples in the basket, but does not do so as effectively as Deep RLSP, because AverageFeatures also rewards the agent for staying in the original location, leading it to avoid picking apples from the tree that is furthest away.
5. Room with far away vase: This is environment that aims show what can't be learned: in this case, the breakable vase is so far away, that it is not much evidence that the human has not broken it so far. As result, algorithms should not learn anything significant about whether or not to break vases. This is indeed the case for Deep RLSP, as well as AverageFeatures (though once again, the latter case, this is dependent on the specific form of the feature).
Overall, Deep RLSP has the same behavior on these environments as RLSP, while AverageFeatures does not.
B ARCHITECTURE AND HYPERPARAMETER CHOICES
In this section we describe the architecture choices for the models used in our algorithm and the hyperparameter choices in our experiments. All models are implemented using the TensorFlow framework.
B. FEATURE FUNCTION
We use a variational autoencoder (VAE; Kingma & Welling, 2014) to learn the feature function. The encoder and decoder consist of feed-for forwforwrrd layers of size 512. The latent space has dimension 30. The model is trained for 100 epochs training we use a batch size of 500 and a learning rate of 10 We use the standard VAE loss function, but weight the KL-divergence term with a factor empirically improved the reconstruction of the model significantly. We hypothesize that the standard VAE regularizes much in setting, because the latent space has higher dimension than the input space, which is not the case in typical dimensionality reduction settings.
100 rollouts of random policy in the environment. During
0.001, which reduces the regularization and
B.2 INVERSE DYNAMICS MODEL
Our inverse dynamics model is feed-forward neural network with layers of size 1024 with ReLU activations. We train it on 1000 rollouts of a random policy in the environment for 100 epochs, with a batch size of 500 and learning rate of 10-5
Note that the model predicts the previous observation given the current observation and action; it does not use the feature representation. We found the model perform better it predicts the residual 9--1 given o₁ and a; instead of directly predicting Ot-1-
We normalize all inputs the model to have zero mean and unit variance. To increase robustness, we also add zero-meat Gaussian noise with standard deviation 0.001 to the inputs and labels during training and clip the outputs of the model the range of values observed during training.
B.3 POLICY
For learning the policy we use the stable-baselines implementation of Soft Actor (SAC) with its default parameters for the MuJoCo environments (Haarnoja et al. 2018 Hill et al. 2018). Each policy update during Deep RLSP uses 10º total timesteps for the cheetah, 2 x 10 for the hopper. We pendulum
perform the policy updates usually starting from the last iteration's policy, except environment, where randomly initialize the policy in each iteration and train it using 5 x 1 iterations of SAC. We evaluate the final reward function generally using 2 x 10 timesteps, except for the pendulum, where use x 104
B.4 INVERSE POLICY
Because the inverse policy is not deterministic, we represent with a mixture density network, a Reed-forwart neural network that outputs mixture of Gaussian distributions (Bishop) 1994).
14

---

Published as a conference paper at ICLR 2021
Deep RL.PP (no gradient weights)
Deep RL.SP
Environment
SAC
# states
(with gradient weights)
1
303 (299)
6 (3) 667 (333)
Inverted Pendulum
1000
10
335 (333)
50
339 (331)
5 (3)
1
4591 (2073)
4833 (2975)
Cheetah
13236
10
6917 (421)
6299 (559)
(forward)
50
6078 (589)
7657 (177)
1
5730 (2733)
5694 (2513)
Cheetah (backward)
13361
10
7917 (249)
8102 (624)
50
7588 (171)
7795 (551)
1
68 (8)
70 (33)
Hopper (terminate)
3274
10
47 (21)
(9)
50
72 (1)
81 (15)
1
1850 (634)
1152 (583)
Hopper (penalty)
3363
10
2998 (62)
1544 (608)
50
1667 (737)
2020 (571)
Table 2: Ablation of the gradient weighting heuristic described in Section 2.4
We report average returns (over 3 random seeds) achieved by the policies learned with and without the heuristic, for different numbers of input states. Experiment setup is the same as in Table [
The network has 3 layers size 512 with ReLU activations and outputs mixture Gaussians with fixed variance of 0.05.
To update the inverse policy we sample batches with batch size 500 from the experience replay, apply the forward policy and the forward transition model on the states label the data. We then train the model with learning rate of 10-4.
B.5 DEEP RLSP HYPERPARAMETERS
We run Deep RLSP with learning rate 0.01, and use 200 forward and backward trajectories to estimate the gradients. Starting with T = 1 we increment the horizon when the gradient norm drops below 2.0 or after 10 steps, whichever comes first. We run the algorithm until T 10.
C HEURISTIC FOR INCORPORATING INFORMATION ABOUT THE INITIAL STATE
we discussed that it might necessary for Deep RLSP to have information about the distribution P of the initial state s-r. Since in our setup Deep RLSP can not obtain any information about through -1 and T-1, here we present a heuristic incorporate the information elsewhere.
In Section
2.4
Specifically, we weight every backwards trajectory by the cosine similarity between the final state 8-7, and sample $-T ~ P. This weights gradient terms higher that correspond trajectories that more likely given our knowledge about P and weights trajectories lower that end in state s_T that has low probability under P.
To test whether this modification improves the performance of Deep RLSP, we compared Deep RLSP with this gradient weighting heuristic to Deep RLSP without as was presented in the main paper.
First, Deep RLSP with the gradient weighting the gridworld environments from Shah et al.) (2019), described in Section 3.2 and Appendix The results are identical to the case when using the heuristics.
We report the results in Table 2 alongside the previously reported results without the gradient weighting. The results are quite similar, suggesting that the gradient weighting does not make much of a difference in these environments.
Next, we tested on the tasks in the MuJoCo environments described in Section 3.3
15

---

Published as a conference paper at ICLR 2021
D ANALYSIS OF THE LEARNED SKILLS
D.1 TRAINING A DISCRIMINATOR
In the main text, we focused on visual evaluation of the learned skills, because it is difficult to define a metric that properly measures the similarity between an original skill and one learned by Deep RLSP. In this section, we attempt to quantify the similarity between policies by training discriminator to distinguish trajectories from the policies. Conceptually, the easier it is to train this discriminator, the more different the two policies are. We could thus use this to check how similar our learned policies are to the original skills.
We train a neural network with a single hidden layer of size 10 with ReLU activation functions. We sample trajectories from both policies and randomly sample trajectory pieces consisting of 5 observations to train the model on. We label the trajectory pieces with a binary label depending on which policy they come from, and then use cross-entropy loss to train the model. To ensure comparable results, we keep this setup the same for all policies and average the resulting learning curves over 10 different random seeds.
The resulting learning curves are shown in fig. The differences between the learning curves are relatively small overall, suggesting that we cannot draw strong conclusions from this experiment. In addition, while the AverageFeatures and Waypoints ablations can be seen to be extremely bad visually relative to GAIL and Deep RLSP, this is not apparent from the learning curves. As result, we conclude that this is not actually a good metric to judge performance. (Note that if we were to use the metrie, would suggest that Deep RLSP is best for the balancing learning skill, while for the jumping skill GAIL is better for and 50 states and Deep RLSP is better for 10 states.)
D.2 VISUALIZATION OF LEARNED SKILLS
Here we provide larger visualizations of the skills learned in the experiments discussed in Section 3.4 of the main paper. For each experiment show the original policy, the states sampled from this policy and given as an input to Deep RLSP, the policy learned by the AverageFeetures ablation, and the policy learned by Deep RLSP in figs. figs.F)to [o future pages). Again, we emphasize that the visual comparison is easier with videos of the policies which we provide at tttttps:: //sitesss// google. com/view rls (including Waypoints and AverageFeatures ablations).
E THINGS WE TRIED THAT DID NOT WORK
Here we list a few variations of the Deep RLSP algorithm that we tested on the MuloCo environments that failed to provide good results.
We tried to learn a latent state-space jointly with latent dynamics model using recurrent state-space model (RSSM). However, we found existing models too brittle to reliably learn good dynamics model. The reward function and policy learned by Deep RLSP worked in the RSSM but did not generalize to the actual environment.
We also tried learning a forward dynamics model from the initial set of rollouts, similarly to how we learn inverse dynamics model, rather than relying the simulator T. However, we found this to cause a similar issue as the RSSM: the reward function and policy learned by Deep RLSP did not generalize to the actual environment. However, we hope that progress in model-based RL will allow us to implement Deep RLSP using only learned dynamics models in the future.
Using an mixture density network instead of MLP model the inverse dynamics did not improve the performance of the algorithm. We suspect this to be because in the MuJoCo simulator the dynamics and the inverse dynamics are "almost deterministic".
Updating the inverse dynamics model and the feature function during Deep RLSP by training it on data from the experience replay did not improve performance and in some cases significantly decreased performance. The decrease in performance seems to have been caused by the feature function changing too much and the training of the other models suffering from catastrophic forgetting result.
16

---

Published as a conference paper at ICLR 2021
Balancing
Jumping
cross-entropy loss
cross-entropy loss
4
1
2
1 state
0
500
1000
500 iterations
1000
iterations
cross-entropy loss
cross-nntropy loss
W
2
2
states
0
500
1000
500 iterations
1000
iterations
cross-entropy loss
cross-entrops loss
4
2
2
50 states
0
0
500
1000
0
500
1000
iterations
iterations
GAIL
Deep RL.SP
AverageFeatures
Waypoints
Figure 4: Learning curves for training discriminator to distinguish the learned skill from the original skill averaged over 10 random seeds. slower learning curve indicates that the learned skill is more similar to the original skill, that is, higher is better.
17

---

Published as a conference paper at ICLR 2021
In the main paper evaluated the policies learned by Deep RLSP from jumping and balancing skills. However, we also looked at policies obtained by optimizing for the learned reward. These also showed similarities worse then the policies directly learned by Deep RLSP. For the jumping skill the optimized policies jump very erratically, and for the balancing skill they tend to fall over perform forward flips. This discrepency is a result of the policy updates during Deep RLSP only using limited number of iterations. It seems like in these experiments the learned reward functions lead to good policies when optimized for weakly but do not produce good policies when optimized for strongly. We saw in preliminary experiments that increasing the number of iterations for updating the policies during Deep RLSP reduces this discrepency. However, the resulting algorithm was computationally too expensive to evaluate with our resources.
the original skills but they were significantly
We tried running Deep RLSP for longer horizons up T 30, but found the results be worse than for T 10 which we reported in the main paper. We hypothesize that this is caused by compounding errors in the inverse transition model. This hypothesis is supported by manually looking at trajectories generated by the inverse transition model. While they look reasonable for short horizons T S 10, compounding errors become significantly bigger for horizons 10 < T 30.
18

---

Published as a conference paper at ICLR 2021
Figure 5: Deep RLSP learning the balancing skill from a single state. The first row shows the original policy from DADS. the second row shows the sampled state from this policy, the third row is the GAIL algorithm, and the last row shows the policy learned by Deep RL.P.
Original Policy
Sampled States
Deep RLSP
GAIL
19

---

Published as a conference paper at ICLR 2021
Figure 6: Deep RLSP learning the balancing skill from 10 states. The first row shows the original policy from DADS, the second row shows the sampled states from this policy, the third row is the GAIL algorithm, and the final row shows the policy learned by Deep RLSP.
Original Policy
Sampled States
Deep RLSP
GAIL
20

---

Published as a conference paper at ICLR 2021
Figure 7: Deep RLSP learning the balancing skill from 50 states. The first row shows the original policy from DADS. the next five rows show the sampled states from this policy, the second to last row the GAIL algorithm, and the last row shows the policy learned by Deep RLSP
Original Policy
Sampled States
Deep RLSP
GAIL
21

---

Published as a conference paper at ICLR 2021
Figure 8: Deep RLSP learning the jumping skill from a single state. The first row shows the original policy from DADS, the second row shows the sampled state from this policy, the third row is the GAIL algorithm, and the last row shows the policy learned by Deep RLSP.
Original Policy
Sampled States
Deep RLSP
GAIL
22

---

Published as a conference paper at ICLR 2021
Figure 9: Deep RLSP learning the jumping skill from 10 states. The first row shows the original policy from DADS, the second row shows the sampled states from this policy, the third row is the GAIL algorithm, and the final row shows the policy learned by Deep RL.P.
Original Policy
Sampled States
Deep RLSP
GAIL
23

---

Published as a conference paper at ICLR 2021
Figure 10: Deep RLS) KLSP learning the jumping skill from 50 states. The first row shows the original policy from DADS. the next five rows show the sampled states from this policy, the second to last row is the GAIL algorithm, and the last row shows the policy learned by Deep RLSP.
Original Policy
Sampled States
Deep RLSP
GAIL
24


Reviews:
[{"review_id": "uE-1y4i16bU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an imitation learning algorithm that can take a single state from an expert's trajectory in order to infer the goal of the task. The idea is to train a reward function that explains both the past trajectory and the futur trajectory from that state, assuming that the very goal was that state (hence, the assumption of larger rewards in the past than in the future induced by the gradient). \n\nThe idea makes sens and is well explained. The results are encouraging and support the method.\n\nI also believe this work could be impactful in the explanation domain, as it provides an answer to the question \"why this state?\". Therefore, outside simple imitation, it could even apply to self-imitation or credit-assigment in direct RL settings. The only weakness would be the required heaviness of the implementation: one need to implement and train all the models (direct and inverse dynamics and policies, auto-encoders for features etc).\n\nHowever, I would ask for some clarifications:\n- What idea that could help the reader to understanding equation (2) (without going through the appendix of the cited paper)?\n- In equation (3), is \\tau' independent of \\tau_-T:-1, so the right-hand term (in red) can be moved outside the blue expectation? (so the simple difference in the last line of the algorithm would nicely appear?) \n- What is encoded and decoded by VAE? (I guess the states, but this is not explicitly told)\n\nAlso, I would have expected a discussion regarding the environmental limitation of this approach, for example, when a state can be completely misleading, or at least not providing any information (for example, the initial state of an environment).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "qYadRr6LzR0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an algorithm, called deep reward learning by simulating the past (deep RLSP), that seeks to infer a reward function by looking at states in demonstration data. An example of this described in the paper is an environment with a vase: if demonstration data shows an intact vase in the presence of an embodied agent then breaking the vase is unlikely to be the intended behavior. Otherwise the vase would already be broken in the demo.\n\nTo achieve this, the paper assumes a Boltzmann distribution on the demonstration policy and a reward function that is linear in some pre-trained state features. The paper then derives a gradient of the log probability of a demonstration state. The gradient estimator involves simulating a possible past from a demonstration state (using a learned inverse policy and inverse transition function) and then simulating forward from the possible past (using the policy and a simulator) The gradient is then the difference between features counts from the backward and forward simulations. \n\nThe paper is generally clearly written and works on a crucial problem in reinforcement learning, namely how to specify a human preference without resorting to tedious reward engineering. Novel, scalable approaches to this problem would certainly be of interest to the ICLR community. The primary technical contribution of the paper is the derivation of the gradient estimator which is correct. \n\nI find the idea of the paper very interesting and the results showing meaningful behavior emerge from a single demonstration are quite nice. However I think the paper is limited in a number of ways:\n- It requires access to a pretrained state representation\n- It requires access to a simulator of the environment which requires being able to reset the environment to arbitrary states. This seems quite limiting for real world applications. Worryingly, appendix D states that learning a dynamics model was attempted by the authors but failed to yield good results.\n- I think the choice of evaluation environments is a little odd and simplistic. I think environments more aligned with the eventual application areas for a method such as Deep RLSP would make the paper much more compelling. Given the motivation of the paper, I think perhaps manipulation environments where a robot arm interacts with multiple objects could be an interesting choice.\n- From the empirical results, it is not clear that Deep RLSP works substantially better than the simple average features baseline.\n\nOverall I think the paper has the potential to be a good paper but could still be substantially improved and I'm leaning towards rejection. \n\nMinor comments and questions for the authors:\n- I'm curious how you choose the gradient magnitude threshold? Does Deep RLSP fail without the curriculum? Could you provide an ablation that shows the effect of using a curriculum? \n- I would also be interested in an ablation of the cosine-similarity weighting heuristic.\n- I think the phrase recent work in the abstract could use a reference.\n- I'm a bit confused by the description of the environment suite by Shah et al. in appendix A, in particular the different rewards. Could you clarify and expand the description a bit?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "J8KYY0K9pQC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers an approach for reward learning from a single frame which was developed for tabular environments and explicit dynamic programming, and extends it to more complex environments through deep RL. \n\nMethodologically, the paper stays close to the ideas of RLSP, but instead of computing relevant quantities (optimal policy, forward and inverse dynamics) through explicit derivations, it suggests most exact steps can be replaced by leveraging deep learning, reinforcement learning and self-supervised learning. There are some minor degeneracy issues stemming from the extension (in particular in gridworld environment) which the authors can mostly solve.\n\nWhile the paper does not have strong methodological novelty, it is well written, the approach is sensible and combines well with state of the art deep RL, and the results are certainly interesting. The authors do a good job providing ablations (some ablations working surprisingly well maybe suggests the methods is working for slightly different reasons than we may assume). \n\n\n\nQuestions: \n- How is SAC(theta) computed? Is it a policy with its own parameters?\n- The inverse dynamics model is a function of the current policy, which is changing over time. This usually causes 'tracking/lagging' issues, and here especially so because the inverse dynamics is presumably a very nonlinear function of the policy. Did you observe any such issues?\n\n\n\n\n\n[1] Reward Learning by Simulating the Past (RLSP)", "extra_scores": {}, "extra_text": {}}, {"review_id": "HjHnmLXyqg3", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the question of learning rewards given only certain preferred or terminal states. I found the setting of the problem to be interesting but not clearly motivated or explained. For this reason, I am unable to recommend acceptance at this stage, due to major clarity issues. I am open to revisiting the recommendation based on author feedback.\n\n(1) Can you explain the assumptions and setting a bit better? Do we only have access to terminal states from execution traces of an expert/human policy? I assume additional interactions with the environment are allowed?\n\n(2) The derivation for equation (1) is unclear to me. I understand that $P(s_0 | \\theta) = \\sum_{\\tau \\in \\Omega} P(\\tau | \\theta)$ where $\\Omega$ are the subset of trajectories that terminate in state $s_0$. Since the probability is a sum of probabilities (instead of product), why does the gradient of log probability decompose into a linear combination of the trajectory gradients?\n\n(3) The writing and notations are quite confusing and hard to follow. In particular, the negative time indexing makes a number of expressions counter intuitive. For example, let us take the expression in section 2.2:\n- What are the contents in $\\ldots$? Can you expand the equation further?\n- Why is $P(a_t | s_{t+1}, a_{t+1}, \\ldots s_0, \\theta) = P(a_t | s_{t+1}, \\theta)$? From a causality viewpoint, this seems counter intuitive since a future state should not influence current action. Is it actually a typo? If this instead has a filtering style interpretation, then shouldn't the decomposition be $P(a_t | s_t, s_{t+1}, \\theta)$ and $P(s_t | s_{t+1}, \\theta)$?\n- It might be easier for parsing and understanding to have the product be $\\prod_{t=-T}^{t=-1}$ instead of $\\prod_{t=-1}^{t=-T}$.\n\n(4) Based on an educated guess of the problem setting (per my understanding), an intuitive and perhaps simpler algorithm would be to learn a goal classifier using the provided {s_0} data, i.e. $P(s=\\text{goal})$, as well as a forward dynamics model $P(s_{t+1}|s_t,a_t)$. Then, one can use the goal classifier as a reward for planning. Would this be an applicable algorithm for this setting? Is this related to the GAIL baseline?\n\nOverall, I was unable to understand the problem formulation and setup, which makes it hard to appreciate the experiments. If the authors can better motivate the setting and improve writing clarity and notations during the rebuttal revision, I am happy to revise my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: First as a procedural point, the paper got 7, 7, 5, 5. AnonReviewer3 gave it a 5, but seemed satisfied by the discussion and promised to raise their score. They did not do so, but I must interpret their last messages as indicating they now support the paper. AnonReviewer2, the other 5, had some concerns that other reviewers seem to have helped address during rebuttal. They did not update their score, but were happy to leave their certainty low and defer to other reviewers' recommendation. As such, although the average score looks low in the system, the paper is of an acceptable standard according to reviews.

The paper adapts a method from tabular RL to Deep RL, allowing (as the title aptly says), agents to learn What to do by simulating the past. Reviewers speaking in support of the paper found that the paper was clear and sound in its evaluation, providing interesting results and a useful and reusable method. It is my feeling that after discussion, the case for the paper has been clearly made, and in the absence of any strong objections from the reviewers, I am happy to go with the consensus and recommend acceptance.
Decision: Accept (Poster)