Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
Authors: Yannick Schroecker
Charles Lee Isbell
Venue:  (2021)
Keywords: Imitation Learning
Reinforcement Learning
Universal Value Functions

Abstract: This work considers two distinct settings: imitation learning and goal-conditioned reinforcement learning. In either case, effective solutions require the agent to reliably reach a specified state (a goal), or set of states (a demonstration). Drawing a connection between probabilistic long-term dynamics and the desired value function, this work introduces an approach that utilizes recent advances in density estimation to effectively learn to reach a given state. We develop a unified view on the two settings and show that the approach can be applied to both. In goal-conditioned reinforcement learning, we show it to circumvent the problem of sparse rewards while addressing hindsight bias in stochastic domains. In imitation learning, we show that the approach can learn from extremely sparse amounts of expert data and achieves state-of-the-art results on a common benchmark.

Under review as a conference paper at ICLR 2021
UNIVERSAL VALUE DENSITY ESTIMATION FOR IMI- TATION LEARNING AND GOAL-CONDITIONED REIN- FORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
This work considers two distinct settings: reinforcement learning. either case, effective solutions require the agent to reliably reach a specified state (a goal), or set of states (a demonstration). Drawing a connection between probabilistic long-term dynamics and the desired value function, this work introduces approach that utilizes recent advances in density estimation to effectively learn to reach a given state. We develop a unified view the settings and show that the approach conditioned reinforcement learning, we show it to circumvent the problem sparse rewards while addressing hindsight bias in stochastic domains. In imitation learning, we show that the approach can learn from extremely sparse amounts of expert data and achieves state-of-the-ar of-tth-art results on a common benchmark.
imitation learning and goal-conditioned
be applied both. In goal-
INTRODUCTION
Effective imitation learning relies on information encoded in the demonstration states. In the past, successful and sample-efficient approaches have attempted to match the distribution of demonstrated states (Ziebart et al., 2008 Ho and Ermon 2016 Schroecker et al.) 2019), reach any state that is part of the demonstrations (Wang al. 2019 Reddy et al. 2019). , track reference trajectory to reproduce a specific sequence of states (Peng et al. 2018 Aytar er al. 2018 Pathak et al. 2018). Attempting to reproduce demonstrated states directly allows the agent to exploit structure induced by enviromment dynamics and to accurately reproduce expert behavior with only a very small number of demonstrations. Commonly, this framed as a measure to avoid covariate shift (Ross et al. 201 T) but the efficacy of such methods on even sub-sampled trajectories (e.g. Ho and Ermon [20 6) and their ability learn from observation alone (e.g. [Toabi et al.) 2018 Schroecker al.] 2019) show benefits beyond the avoidance of accumulating errors. Despite significant progress in the field, reproducing set of demonstrated states efficiently, accurately and robustly remains an open field of research. To address the problem, may first ask the question of how reach arbitrary states, question that has thus far been considered separately in the field of goal-conditioned reinforcement learning. In this paper, we introduce unified view goal-conditioned reinforcement learning and imitation learning. We will first address the question of how to reach single states in the former setting and then extend this notion to an imitation learning approach and reproduce distributions of state-action pairs.
Despite significant achievements in the field (Schaul et al. 2015: Andrychowicz et al. 2017 Nair al. 2018 Sahni al. 2219)), learning achieve arbitrary goals remains extremely difficult challenge. In the absence of a suitably shaped reward function, the signal given to the agent can be as sparse as a constant reward if the goal achieved and otherwise. Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) constitutes an effective and popular heuristic to address this problem; however, the formulation ad-hoo and does not lend itself readily the probabilistic reasoning required in a distribution distribution-matching imitation learning approach. Furthermore, HER can be shown to suffer from bias stochastic domains or when applied to some actor-critic algorithms as we will discuss in Section 3.1 To address this challenge, we introduce Universal Value Density Estimation (UVD). By considering an important subset of soal-conditioned value functions similar to similar to Andrychowicz et al. (2017). namely those corresponding reward functions that have an optimal agent reach a specific state, we can observe that the value of a state conditioned on a goal is equivalent to the likelihood of the agent reaching the goal from that state. We normalizing flows

---

Under review as a conference paper at ICLR 2021
(see Sec. 2.4) to estimate this likelihood from roll-outs and thereby obtain an estimate of the value function. As we will show in Section density estimation does not sample goals independently at random and therefore provides dense learning signal to the agent where temporal-differene learning alone fails due to sparse rewards. This allows us to match the performance of HER in deterministic domains while avoiding bias in stochastic domains and while providing the foundation for our imitation learning approach. We will introduce this approach in Section (.4 and show that it performs as well as the state-of-the-art on the most common benchmark tasks while significantly outperforming this baseline on simple stochastic variation of the same domain (Sec. 5.1
Returning to the imitation learning setting, we propose to extend UVD to match a distribution of expert states and introduce Value Density Imitation Learning ( (VDI). Our goal is to design an imitation learning algorithm that is able to learn from minimal amounts of expert data using self-super environment interactions only. Like prior methods (see Sec. 2.3 VDI's objective is to match the expert's state-action distribution. We achieve this by sampling expert states that the agent is currently not likely to visit and using goal-conditioned value-function to guide the agent towards those states. We will show in Sec. that this minimizes the KL divergence between the expert's s and the agent's 's state-action distributions and therefore provides an intuitive and principled imitation learning approach. Note that unlike most prior method, expert demonstrations are used in VDI to generate goals rather than to train an intermediate network such as discriminator or reward function. The value function and density estimate are trained using self-supervised roll-outs alone. This makes intermediate networks much less prone to overfitting and we show in Sec. 5.2 that VDI uses demonstrations significantly more efficiently than the current statc-of-the-a in common benchmarks.
2 BACKGROUND
2.1 MARKOV DECISION PROCESSES
Here, we briefly lay out notation while referring the reader to Puterman (0144) for detailed review. Markov Decision Processes (MDPs) define set of states S, set of actions A. a distribution of initial states ds(s). Markovian transition dynamics defining the probability (density) of transitioning from state 8 to s' when taking action as p(s (s, a), and a reward function r(s, a). In reinforcement learning, we usually try to find a parametric stationary policy μt0 : S + A An optimal policy maxi- mizes the Long-term discounted reward (0) = E 7 a))sss factor y or, sometimes, the average reward (+) state distribution d'(s) and the stationary state-action distribution p"(s, a) are uniquely induced by y under mild ergodicity assumptions. A useful concept this end is the value function (*) E (E y'r(st, (s) = s, 14] or Q function Q"(s,a) = -E 1==0 7'r(s,, at) |s0 = 5. a.
do [o] given a discount
d (s)s a, where the stationary
[
which can be used to estimate the policy gradient V.J,, (c.g. Sutton et al. 1999; Silver al.
2014). Finally, we define (s a 4 s') the probability of transitioning from state to s' after t steps when taking action a in state 8 and following policy μ afterwards.
2.2 GOAL-CONDITIONED REINFORCEMENT LEARNING
Goal-conditioned Reinforcement Learning aims to teach an agent to solve multiple variations of task, identified by goal vector y and the corresponding reward function r (8, a). Solving each variation requires the agent to learn a goal-conditioned policy, which we write as [g Here, we condition the reward and policy explicitly to emphasize that they can be conditioned on different goals. The goal-conditioned value function (GCVF) in this setting is then given by
Qud
T (8, a) = E 2 az) [80 = 8,00 = a, J.
To solve such tasks, [Schaul et al. (2055) introduce the concept of a Universal Value Function Approximator (UVFA), learned model (s, a: g) approximating Q (s, a), i.e. all value functions where the policy and reward are conditioned on the same goal. We extend this notion to models of value-functions which use goal-conditioned reward with single fixed policy. To distinguish such models visually, we write Qu (s, a;gg) to refer to models which approximate Q" (s, a) for a given μ..
We write the policy as a deterministic function, but all findings hold for stochastic policies as well.
2

---

Under review as a conference paper at ICLR 2021
Where Qw represents how good the agent is at achieving any goal if it tries to achieve it, Qu models how good a specific policy is, if the task were to achieve the given goal.
Schaul et al. (2015) show that UVFAs can be trained via temporal difference learning with randomly sampled goals; however, if the reward function is sparse (a common case is an indicator function that tells the agent whether a goal has been achieved), the agent rarely observes non-zero reward. Hindsight updates (HER) (Andrychowicz et al., 2017) are a straight-forward solution, changing goals recorded in replay memory based on what the agent has actually achieved in hindsight. This approach is intuitive and can also be extended to work with visual observations (Nair et al. 1018/Sahm et al. 2019); however, it does not yield correct results in stochastic domains. This has previously been noted by Lanka and Wu (2018), who propose a heuristic method to address this issue, but no principled method to use hindsight samples in a completely unbiased way has been proposed to date. Temporal Difference Models (TDMs) (Pong et al. 2018: Nasiriany et al. 2019). a method related to ours, use a squared error function as a terminal value and bootstrapping to back up this value over finite horizon. Our methods learns to model arbitrary terminal values and allows for bootstrapping over an infinite horizon. TDMs are orthogonal to the concept of hindsight samples.
2.3 IMITATION LEARNING
Imitation learning (IL) teaches agents to act given demonstrated example behavior. While the MDP formalism can still be used in this scenario, we no longer have a pre-defined reward function specifying the objective. Instead, we are given sequence of expert state-action pairs. The goal of imitation learning is to learn a policy (4g that is equivalent to the expert's 's policy μr' which generated the demonstrated states and actions. This problem is underspecified and different formalizations have been proposed to achieve this goal. The simplest solution to IL is known as Behavioral Cloning (BC) (Pomerleau 1989) and treats the problem supervised learning problem. Using demonstrated states as sample inputs and demonstrated actions as target outputs, a policy can be trained easily without requiring further knowledge of, interaction with, the environment. While this approach can work remarkably well, it is known to be suboptimal as standard assumption of supervised learning violated: predictions made by the agents affect future inputs to the policy (Ross et al. 2011). By learning about the environment dynamics and explicitly trying to reproduce future demonstrated states, the agent is able to learn more robust policies from small amounts of demonstration data. A common and effective formalism is to reason explicitly about matching the distribution of state-action pairs that the agent will see to that of the expert. By learning from interaction with the environment and employing sequential reasoning, approaches such as maximum entropy IRL (Ziebart et al. 2008), adversarial IRL (Fu et al. 20188) GPRIL (Schroecker et al. 2019) or Generative Adversarial Imitation Learning (GAIL) (Ho and Ermon 2016) lead the agent to reproduce the expert's observations as well as the expert's 's actions. Especially adversarial approaches (Ho and Ermon, 2016: Fuu et al., 2018 Kostrikov et al., 2019) have gained popularity in recent years and train discriminator to serve as a distance function between the agent's 's's and the expert's state-action distribution. Here, we instead propose to train a generative model to predict multiple time-steps ahead in order to reproduce demonstration states. GPRIL (Schroecker et al., 2019) similarly utilizes a long-term generative model for imitation learning; however, in contrast to VDI, it cannot easily be combined with temporal-difference -iffereece learning techniques and struggles with larger time-horizons and performs badly on locomotion benchmark tasks. In Section 5.2 we show that VDI cannot only be applied in such domains, but that it outperforms the current ofth--of-the-rt common benchmark tasks.
2.4 NORMALIZING FLOWS
Normalizing flows (van den Oord et al. 2017 Dinh al., 2016) are capable of learning explicit representations of complex, highly non- version of RealNVP (Dinh et al. (016) to represent value functions as density functions. Re model consist of a chain of learned, invertible transformations which transform samples = E T from one distribution P> to samples æ E N from another distribution pa. The transformations are affine where half of the input features are scaled and shifted by parameters predicted based on the other half which allows for tractable computation of the gradient. The model can therefore be trained using a traight-forwann maximum likelihood approach. While the original work defines a specific autoregressive order to model images effectively, we use a simplified version of RealNVP in this paper model non-image data and pick the autoregressive order at random.
non-linear linear density functions. In this work, we utilize a simplified
3

---

Under review as a conference paper at ICLR 2021
3 UNIVERSAL VALUE DENSITY ESTIMATION
3.1 ADDRESSING HINDSIGHT BIAS
Before we introduce our approach to estimating GCVFs corresponding to fixed or goal-conditioned policies, written Qw or Qu respectively, we will first point out the issues that arise when using common temporal-differenc learning approaches. To effi ciently approximate a GCVF, we have to address the challenge of learning from sparse rewards. Each TD update requires us to sample a goal and if this is done independently from the trajectory is unlikely to be a goal that has been reached. Andrychowicz et al. (2017) address this with heuristic that oversamples achieved goals. This allows the agent to learn from failures and provides way speed up the training of a UVFA; however, it is a heuristic approach and can lead the agent converge to non-optimal poli- cies in stochastic domains. We illustrate this with a Figure here, the agent has to walk around a cliff to reach the goal in the bottom right but may fall down the cliff due to environment noise affecting its movements. The optimal policy is to take safe path to avoid falling down the cliff by accident; however, using HER we only learn about the true goal based on successful roll-outs and disregard episodes where the environment noise led to failure, i.c. the agent never learns from transitions leading down the cliff, underestimates their probability and therefore incorrectly chooses the shorter path.
f
(a)
(h)
Figure 1: Simple cliff-walk domain. The agent starts in the bottom left, has to avoid the cliff depicted in black and reach the state in the bottom right. a) depicts the transition probabilities of an optimal policy while b) de- picts a sub-optimal policy learned using HER.
gridworld-examge
Mathematically, this hindsight bias manifests itself as being equivalent to changing the environment dynamics. We examine the effect of changing the goal-sampling distribution in a hindsight temporal difference update rule. Using to refer the parameters of the previous iteration, the temporal difference update rule takes the form w + w + aA, with
A = / p0 (s. (o((s(ss,))(()))) (... a, 8 = r + - 13)) as)s.
(1)
Altering P(g) independently would be akin to using out-of- -istribbbttt samples in regression, can be addressed with additional representational capacity and the effect is little different from using using replay buffer (Munos et al. 2016 Fujimoto et al. 2018); however, HER alters the distri- bution of goals p(g) to be dependent on the observed transitions, i.e. transitions are sampled from p(s'ls, (ss a)p(gi s, a, 8'). This factorization
alternatively be written as p(g]s ss a)p(s'|s, (s, a, g) which makes it plain how HER is modifying the transition dynamics and therefore the MDP that the agent
trying to solve. This is the source of hindsight bias in stochastic domains as observed in Figure
3.2 VALUE DENSITY ESTIMATION
Knowing hindsight samples to be good solution to recover dense learning signal but to lead to an inconsistent estimator when combined with temporal-diffent estimation can be used to estimate GCVTs using sampling procedure similar to that of HER. We consider the special case where the task is for the agent to reach a given goal state. This scenario is common in condicond RL and also the one being addressed by Andrychowicz et al. (017). In discrete environments, we can define such tasks by a reward that is positive if the goal is achieved and O otherwise. We define: rd (s. a) = (1 7) 1 (h(s, a) = g), where is function that defines the achieved goal for any given vate-action pair. In slight abuse of notation we extend this definition to continuous environments and write (5, a) (1 -0))00(s,)) We can now show that the Q-function of such tasks forms a valid density function. Specifically, we notice that the Q function is equivalent
learning, we now show that density
Formally, the reward would have to he defined to be non-zero only in e-ball around l(s,, a) and inversely proportional the volume of this ball. All results hold in the limit O.
4

---

Under review as a conference paper at ICLR 2021
to the discounted probability density over goals that the agent is likely to achieve when following its policy, starting from the given state-action pair:
Q =E
y r'rs s00 =5000 = a
-=
=(1-0 Σ x² p is d') =; FF(gls,a)
follows that we can learn an approximation of the goal-conditioned Q-function 2 (s, a) by approximating the value density F [g]ss s a). This can be done using modern density estimators such as RealNVPs (see Section (.4). To train the model, we gather training samples from a roll-out 9t, a de at random time-steps as well future achieved goals h( At+s)i.i j ~ Geomn( - 7). Notice that the above derivation assumes fixed, i.e not conl-conditioned policy. We relax this assumption in Section 3.4
50, ag, $1, di...... collecting state-action pairs
3.3 COMBINING ESTIMATORS
Learning model of 7 already provides a valid estimator of the Q-function; however, relying on density estimation alone is insufficient in practice. As the discount factor approaches 1, estimating Q purely based on density estimation would require state-action-gon triplets that are hundreds of time- steps apart, leading to updates which suffer from extremely high variance. In practice, we therefore wish to limit the time-horizon of the density estimator while relying temporal difference (TD) learning to backup values over long time horizons with low variance. Note that cemporal-differoet learning alone is insufficient in this scenario as it may never observe reward Here, propose approximate the value by combining the value density estimator with temporal difference learning, using the density estimate lower bound on the target value. If the goal within the time-horizon of the density estimator, the (scaled) density estimate approaches the true value and the bounded temporal difference loss simply matches Q to this density estimate. If the goal is outside its time horizon, the lower bound approaches 0 and we are using plain temporal difference learning to back temporal difference loss is given by
up the value. The - (u(ssa;)))
L(W)= = (s, +
:= max g).
(3)
where (glss, a) is a learned estimate of the density defined in Eq. 2
3.4 LEARNING GOAL-CONDITIONED POLICIES
we introduced notation to differentiate between learning model Q, which uses goal-conditioned reward and fixed policy and a model Q, where both quantities are conditioned on the goal. While the previous section gives us the means to efficiently train Q, goal-conditioned RL requires us learn Q. Using a goal-conditioned policy in E we can write down a corresponding
In Section 2.2
equivalence to predictive long-term generative model: Fy* (g]s, a) Q (s, a). Here, F4 (gls, a) the distribution of goals y that the agent will reach when it starts in state s, takes action a and then tries to reach the goal J. As before, we can model this with normalizing flow; however, in this the model has to be conditioned on the intended goal ī and takes the form (g s, a, g). Using this density estimate,
alter the temporal difference target in Eq. 3 to efficiently train UVFA:
max (s, a: g), (a a. (.)))
This yields an iterative algorithm for goal-conditioned RL (Algorithm that efficient and avoids hindsight bias. Implementation details are discussed in Appendix
4 VALUE DENSITY IMITATION LEARNING
We now turn our attention to the problem of sample-efficient imitation learning. We wish to train the agent to imitate an expert's 's policy using only few demonstration samples from the expert.
If the environment is continuous and stochastic and the reward is as defined above, the agent always observes reward of 0 on random samples.
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Universal Value Density Estimation Algorithm 2 Value Density Imitation
function UVD
function VDI
for i <- O..#Iterations do
for i 4- t.#tterations do
Fill replay buffer with experience
Fill replay buffer with experience
for s, a, sampled from short replay buffer do
for 8, a sampled from short replay buffer do
Sample f ~ Geom (1 7)
Sample t ~ Geom (1 --)
Sample achieved g L steps ahead of
Sample target states t steps ahead of
Update Fe with V log F& (gls, a,g)
Update Fg. with V log Fe (s|s,a)
for s, s'.g sampled from long replay buffer do F - F 100'.110 (s').9)
Update dg with - Va log dg (s)
for 8, a, s' saapledd from long replay buffer do
Sample uniformly from expert data
+ max Fr (6))(())
Q <- max Few (5)s, a)) 7Qw (s', (s
Update Qw with v. (v (m,a)))
Update Qw with V (Q Qu (8. az
Update 1 with V.Q. (0 mmgg(
for a from long replay butfer do Sample from expert data with
(x)
Update M4 with V. (8, 0;
Vorse (s)
Different formulations exist to solve this problem (see Section 2.3). but the formulation which has arguably been the most promising is to train the agent to explicitly match the expert's 's state-action distribution or occupancy measure (Ziebart et al., 2008 Ho and Ermon 2016). Our next step is therefore to extend our findings from the previous section to state-action or state distribution matching. In the literature, this is commonly framed as divergence minimization problem with recent work investigating different measures (Ghasemipour et al. 2020 Ke et al. 2019); however, this view is incomplete as the divergence measure has to be approximated and the quality of the approximation can be significantly more impactful than the choice of measure. Similar to Schroecker and Isbell 20 Schroecker et al. 2019). we propose to estimate the gradient Vo log d (x) from self-supervised data (where we write refer to demonstrated states). With estimate of this gradient, we can maximize the likelihood of expert state-action pairs which gives a straight-forward approach to minimizing the KL-divergence between the two occupancy distributions. Note that while the gradient is evaluated at expert states, the derivative itself depends only on the agent's 's behavior. Unlike the discriminator in an adversarial approach, the quality of the approximation therefore does not depend on the number available expert samples. Using the state- ciitte-dstibbttion gradient alone in maximun-likelihodd algorithm disregards expert actions, but is optimal the expert's behavior can be uniquely described using reward-function that depends only on the current state. We will evaluate state-distribution matching as imitation from observation in Section 5.2) To extend the approach to unambiguous state-action distribution matching, we can combine it with the behavioral cloning gradient as V log pTa (s, a) = Vo log To (als) V log d= (3), where we assume a stochastic policy Ty in place of deterministic one. We found, however, that the behavioral cloning gradient can dominate a noisy estimate of the state-distribution gradient and lead to overfitting. Instead, we propose augment the state include the previous action that lead to the state. This approach attempts to match the joint distribution of action and next state which implies the state-actionddistributioo based on environment dynamics.
If applied to a single state, following the state-distribution gradient teaches the agent to go to that state. It thus no surprise that it can be shown to be equivalent to the policy gradient for the right goal-conditioned reward function (also see (Schroecker et al. 2019). Specifically, the state- distribution gradient is equivalent the weighted policy gradient the average-reward setting (using (s, a) 6,,z). We have:
Vedi (3)
d!d (s)
VoJ (0)
Ve log (
V
(S)
dise
die
diae (3)
Intuitively, the policy gradient leads the agent toward a demonstration state 3, while the weight ensures that all demonstration states are visited with equal probability. This gives rise Value Density Imitation Learning: Using self-supervised roll-outs, the algorithm learns a goal-conditioned Q-function well as unconditional density estimator du (s) of the agent's state-distribution. Next,

---

Under review as a conference paper at ICLR 2021
FeemSttew)
Opparse)
IBr TD
TDC+i
100+101
10
terscon
tuston
tersto)
(a)
(b)
(C)
(d)
Figure 2: Average and range of success rate across 3 seeds of TD3, UVD and HER on variations of the Fetch manipulation domains. FetchPush and FetchSlide correspond the original domains. Fetch- Push (Sparse) shows the advantage of hindsight updates when the required precision is significantly higher while FetchSlide + Noise shows the effect of hindsight bias in stochastic domains.
demonstration states are sampled with probability proportional حل(د and used as goals when estimating the policy-gradient (see Alg. for details).
5
EXPERIMENTS
5.1
GOAL-CONDITIONS REINFORCEMEN LEARNING
We first evaluate UVD on a suite of simulated manipulation tasks involving a Fetch robot arm. This suite has been developed by] Andrychowicz et al. (2017) to show the strengths of hindsight experience replay. Our goal here is twofold: 1. We show that UVD matches the performance of HER on the unmodified deterministic tasks where HER does not suffer from hindsight bias. 2. We show that HER converges to sub-optimal policy on stochastic variation of the task while UVD is still able to solve the task efficiently. We compare all methods using the same papper-rrrmeterss (see Appendix
The first domain in the suite, FetchPush, requires the robot arm to learn to push an object to any given target location. The reward signal is sparse as the agent is given a non-zero reward only if the object reaches the desired location. In Figure 2a we can see that TD3 with HER and TD3 with UVD perform similarly. Contrary to the original findings by by/Anrychowiiz et al. (2017). we also find unmodified TD3 to be able to solve the task accurately using roughly twice the amount of training samples. This indicates that the area around the goal in which the object is considered to be at the desired location is relatively large. If we apply a stricter criterion for the goal being reached by reducing the size of the goal area by a factor of 100, we can see that hindsight samples are necessary to learn from sparse rewards. In this variant, TD3 fails learn useful policy (see Figure 2c).
The second domain in the suite, FetchSlide, requires the robot to slide the object toward a desired location that is out of reach of the robot arm. In Figure 2b we can see UVD and HER learning to solve the task quickly while TD3 without hindsight samples requires significantly more training samples. Unlike in the case FetchPush, TD3+UVD learns slightly faster in this domain than TD3+HER but both are able to solve the task eventually. Both, FetchSlide and FetchPush, are deterministic domains that play to the strengths of hindsight experience replay. In practice, however, manipulation with a real robot arm is always noisy. In some cases, HER can overcome this noise despite suffering from hindsight bias; however, this is not always the case. Here, we introduce a variation of the FetchSlide domain which adds Gaussian noise the actions of the agent. We scale this noise based on the Ze max(0, a 0.5 1)1.3 This scaling allows the agent to adapt the noise; however, doing so requires the agent to accurately assess the risks of its actions. In Figure 2d we can see that while HER initially learns quickly as in the deterministic domain, it converges to a policy that noticeably worse than the policy found by TD3+UVD. This shows the effect of hindsight bias: in the presence of noise, the agent assumes the noise to be benign. UVD, on the other hand, estimates the risk accurately and achieves higher success rate than HER.
squared norm of the chosen actions L
5.2 IMITATION LEARNING
We compare the demonstration-effiiiine of Value Density Imitation with GAIL, on common benchmark locomotion tasks (Brockman et al. 2016). We compare against GAIL method has
7

---

Under review as a conference paper at ICLR 2021
0200
0200
4200
Vatee Seacity Iniition
Vatu Dencity Intisiion
GAL
GA
2300
Expoit Random
2300
Export
Sandom
1 sumples
15 tamp en
50 tample
1150 samp
10 203 samp in 20 200 camples 52 200 samples
(a)
(b)
Figure 3:
Comparison of GAIL and VDI on variations of standard benchmark tasks. The x-axis shows the number of demonstrated trajectories times state-action pairs per trajectory. VDI requires fewer demonstrations compared to GAIL.
reported better demonstration-effiiie on these domains to date. Recently, the suite of locomotion tasks has become the go-to benchmark for imitation learning (c.g. Ho and Ermon, (016)); however, we find that the unmodified locomotion tasks, despite their popularity, are easily exploited in an imitation learning context and are a bad measure of whether an algorithm imitates the expert or not. While it is impressive that GAIL is able to solve Humanoid-v2 with less than a dozen of disjointed demonstrated states, it is also clear that the agent is not learning to imitate the motion itself. The dominating source of reward in these tasks comes from the velocity in a particular direction. This is problematic as the velocity is fully observable as part of the state and, in the case of humanoid, may be encoded in more than one of the features found in the vector. Even the simple average of the state-features may be equivalent to a noisy version of the original reward signal. Moreover, as the reward is linear combination of featuree, we know that accurate distribution matching is not necessary and matching feature expectations is sufficient (Abbeel and Ng. [0044 Ho and Ermon, 2016). To alleviate this, we remove task task-space velocities in z. y directions from the state-space. A second source of bias can be found in the termination condition of the locomotion domains. Kostrikov et al. (2019) point out that GAIL is biased toward longer trajectories and thus tries to avoid termination, which in the case of locomotion means to avoid falling. While Kostrikov et al. adjust the algorithm itself to avoid such bias, we instead propose to remove the termination condition and use an evaluation which cannot be exploited by biased method. We find that other methods such as DAC (Kostrikov let al. 2019) may require re-tuning to solve these tasks but find that GAIL still performs well on the modified tasks. This is unsurprising as GAIL has been applied to real-world applications (e.g. Wang iet al. 2017).
We focus on two locomotion tasks in particular: HalfCheetah-r2 and Humanoid-v2. HalfCheetah-v2 is comparatively easy to solve while providing a high skill ceiling. With the original threshold for solving the task being set at a score of 4500, recent advances in reinforcement learning train policies that achieve 3-4 times as high score (Fujimoto et al. 2018 Haarnoja et al. 2018) (removing velocity from the state reduces the top-speed achieved by the TD3-trained expert). Our second domain of choice is Humanoid-v2, which is generally considered to be the most complex locomation task. Unlike in the HalfCheetah-v2 domain, learning to move without falling can be a significant challenge for a learning agent. We furthermore find it sufficient to match state-distributions to solve HalfCheetah-v2 and thus teach the agent from observation only when using VDI. In the case Humanoid-v2, find that demonstrated actions significantly aid exploration and thus include them. The results can be seen in Figures Bal and (bbas well as in Appendix C While both methods are able to achieve near-expert performance on trajectory sub-sampled at the same rate as used by Ho and Ermon, we find VDI to be able to imitate the expert if the trajectory is sub- sub-sampled even further. On Humanoid-v2 -aw.aanoidd-2--nnnpplooloo ExnnEiiiii,alle, we find the difference to be more drastic: while GAIL is able to learn locomotion behavior from a similar number of trajectories as used in the original paper (but using more state-action pairs), the performance drops off quickly if we reduce the number of trajectories further. VDI is able to learn from only a single demonstration. Both methods are able to achieve great demoontration--efiiiieee the expert's 'sssi state-action distribution; however, GAIL does so by learning distance function using demonstrations as training samples. As the number of demonstrations shrinks, learning a good discriminator to serve reward becomes more difficult. VDI side-steps this issue by learning goal-conditioned Q-function based on supervuee roll-outs alone. The demonstrations are used to condition the Q-function and don't t serve directly training data for any network.
using a single demonstrated
by matching
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In R. Greiner and Schuurmans, editors, International Conference on Machine Learning, pages 1-8, 2004.
M. Andrychowicz, F. Wolski, A. Ray. J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba. Hindsight experience replay, Advances in Neural Information 2017.
Processing Systems, pages
Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang. and N. de Freitas. Playing hard exploration games by watching youtube. In Advances Neural Information Processing Systems, pages 2930-2941, 2018.
G. Brockman, V. Cheung. L. Pettersson, Schneider, Schulman, Tang, and W. Zaremba. OpenAl Gym. 2016.
L. Dinh, Sohl-Dickstein, and Bengio. Density estimation using Real NVP. arXiv preprint 1oXi::6655.8803, 2016.
J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement leannnnn In International Conference Learning Representations, 2018. URL URLAttpps://ppeeeeviee net /eot/foumms id=rkHyw A
S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods.
In International Conference on Machine Learning, pages 1582-1591, 2018.
S. S. Ghasemipour, R. Zemel, and Gu. A divergence minimization perspective on imitation learning methods. In Conference Robot Learning, pages 1259-1277. PMLR, 2020.
Haarnoja, Zhou, Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1856-1865, 2018.
J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Information 2016.
Processing Systems, pages
I. Ke, M. Bames,, W. Sun, G. Lee, Choudhury, and S. Srinivasa. Imitation learning f-divergence minimization. arXiv preprint arXiv: 128888, 2019.
I. Kostrikov, K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriininatoractoorrriii Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019. URL htt.:ttps://// http::::///peeevvee... net/ wwt/ffruutt id=Hk4
Lanka and T. Wu. ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replay. arXiv preprint arXiv: 1009.000900000 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare. Safe and efficient off- policy rein- forcement learning. In Advances in Neural Information Processing Systems, pages 1054-1062, 2016.
A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pages 9191-9200, 2018.
S. Nasiriany, V. Pong, S. Lin, and S. Levine. Planning with cond-coodd policies. In Advances in Neural Information Processing Systems, pages 14814-14825, 2019.
C. Nota and P. S. Thomas. Is the policy gradient gradient? arXiv preprint arXiv:1906.07073, 2019.

---

Under review as a conference paper at ICLR 2021
D. Pathak, P. Mahmoudich, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. International Conference on Learning Representations, 2018.
X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep reinforcement learning physics- character skills. ACM Transactions on Graphics, 37(4): 143, 2018.
D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305-313, 1989.
V. Pong, S. Gu, Dalal, and S. Levine. Temporal Difference Models: Model-Free Deep RL for Model-Based Control. In International Conference on Learning Representations, 2018.
M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via regularized behavioral cloning. arXiv preprint arXiv:1905.11 111000,,1..,,,.. 2019.
S. Ross, G. Gordon, and D. Bagnell. A reduction imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 627-635, 2011.
H. Sahni, T. Buckley, Abbeel, and Kuzovkin. Visual Hindsight Experience Replay. Advances in Neural Information Processing Systems, 2019.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal Value Function Approximators. In International Conference on Machine Learning, pages 1312-1320, 2015.
Y. Schroecker and C. Ishell. State Aware Imitation Learning. Advances Neural Information Processing Systems, pages 2915-2924, 2017.
Y. Schroecker, M. Vecerik, and J. Scholz. Generative predecessor models for sample-efficient imitation learning. International Conference on Learning Representations, 2019. URL https //openreview.oo net A
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395, 2014.
R. Sutton, D. Mcallester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems, pages 1057-1063, 1999. ISSN 0047-2875. doi: 10.1.1.3..9144.
F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158, 2018.
A. van den Oord, Y. Li, Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. van den Driessche, E. Lockhart, L. C. Cobo. F. Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprini arXiv: 171.v:1711.10433 2017.
R. Wang, C. Ciliberto, P. V. Amadori, and Y. Demiris. Random expert distillation: Imitation learning via expert policy support estimation. In International Conference Machine Learning, pages 6536-6544, 2019.
Z. Wang, J. S. Merel, S. E. Reed, N. de Freitas, G. Wayne, and N. Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pages 5320-5329, 2017.
B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
10

---

Under review as a conference paper at ICLR 2021
A HYPERPARAMETERS
In Table we list the hyper-parameters used for TD3, TD3+HER and TD3+UVD the Fetch experiments. When applicable, we largely use the same hyper- parameersssssrss for each algorithm. Two important exceptions are the learning which is tuned individually for each algorithm (TD3+UVD benefits reliably from higher learning rates whereas TD3 and TD3+HER does not always converge to good solution at higher learning rates) as well as the output activation the Q-network. tanh activation is used to scale the value to the range of -50 to 50 in the case of TD3 and TD3+HER as bounded from above, the
we found this to drastically improve performance. Since the density same activation cannot be used in the case of TD3+UVD.
In Table we list the syper-parameters used for VDI the locomotion experiments. Parameters are largely identical between environments; however, in some cases we trade off higher learning speed for reduced accuracy on HalfCheetah. the case of GAIL, we use the implementation found in OpenAl baseline using 16 parallel processes (collecting 16 trajectories per iteration) and do not modify the parameters.
Table I: Common parameters in Fetch environments
General parameters
Environment steps per iteration
0.98
Batch size
5T2
Replay memory size
1500000
Short replay memory size
50000
Sequence Truncation (Density estimation training)
4
Optimizer
Adam
Policy
Hidden layers
400, 400
Hidden activation
leaky relu
Output activation
tanh
Exploration noise
0.1
Target action noise o
0.0
Learning rate
10
Q-network
Hidden layers
400, 400
Hidden activation
leaky relu
Output activation
50 tanh, linear (TD3+UVD)
Learning rate
2 10 10-1.11000 8. 10 (TD3+UVD)
Re
Bijector hidden layers
300, 300
Hidden activation
leaky relu
Output activation, scale
tanh(log()
Output activation, translate
linear
num bijectors
5 (slide), 6 (push)
Learning rate
2 10-4
11

---

Under review as a conference paper at ICLR 2021
Table 2: VDI parameters in locomotion environments
Parameter
HalfCheetah
Humanoid
General parameters
Environment steps per iteration
10
Y
0.995
0.995
Batch size
256
256
Replay memory size
1500000
1500000
Short replay memory size
500000
500000
Sequence Truncation (Density estimation training)
4
Optimizer
Adam
Adam
Policy
Hidden layers
400, 300
400, 300
Hidden activation
leaky relu tanh
leaky relu tanh
Output activation
Exploration noise
0.3 (until iteration 25000), 0.1 (after)
0.1
Target action noise a
Learning rate
3 10
3 10
Q-network
Hidden layers
400, 400
400, 400
Hidden activation
Icaky relu
leaky relu linear
Output activation
linear
Learning rate
3. 10
1 10 4
RealNVP
Bijector hidden layers
400, 400
400, 400
Hidden activation
Icaky relu
leaky relu
Output activation, scale
tanh(log())
tanh(log()
Output activation, translate
linear
linear
num bijectors
Learning rate
1 10
2 10
L2-regularization
1 10
I 10
Spatial smoothing
Temporal smoothing
O.
0.98
B PRACTICAL CONSIDERATIONS
There are a number of implementations decisions that were made to improve the sample-efficiency and stability of Value Density Estimation and its application imitation learning. Here, review these decisions in more detail.
B.1 UNIVERSAL VALUE DENSITY ESTIMATION
Using an exploration policy: In most cases self-supervised roll-outs will require the agent to explore. In method, combine temporal difference update rule is usually found in deterministic policy gradients with density estimation. While the temporal-difference update rule can handle off-policy data from an exploration policy, density-estimation is on-policy. In practice, however, Fujimoto et al. (2018) add Gaussian noise to the target-Q function and report better results by learning smoothed Q-function that is akin the on-policy Q-function with Gaussian exploration noise. It thus stands to reason that we can omit off-policy correction in the Sennsiy--sttmaate
The training data for learning a long-term model can be fairly noisy. While we can expect density estimation to be efficient over horizon of just a few time-steps, the variance increases dramatically as increases. This the primary motivation for utilizing temporal- difference learning in conjunction with universal value density estimation. To better facilitate stable training, we truncate the time-horizon of the density eeststty to a fixed number of time-steps T. The lemporal-differenee learning component is thus solely responsible for propagating the value beyond this fixed horizon. The effect this has on the optimal policy is small: the Q-value will be underestimated by ignoring visitations with time-to-recurrece greater than T. A greater time-horizon
Truncating the time horizon:
12

---

Under review as a conference paper at ICLR 2021
boosts the effect of hindsight samples and leads to learning signal that is less sparse while smaller time-horizon reduces the variance of the estimator.
Using a replay buffer: Using a replay buffer is essential for sample-efficiett training with model- free reinforcement- and imitation-leaminng methods and improves the stability of deterministic policy gradients. We find that this is true for training long-term generative models well. While importance sampling based off-policy correction for density estimation is possible, we find that it introduces instabilities and is thus undesirable. Instead, we propose to use a separate, shorter replay-buffer for density estimation to mitigate the undesirable effects of off- aff-pooicy learning while retaining some of the benefits.
Delayed density updates: Temporal-differenn learning with non- non-linear function approximation is notoriously unstable. To help stabilize it, a common (Mnih et al.) 2015 Fujimoto et al.] (018) trick is delay the update of the target network and allow the Q-function to perform multiple steps of regression toward fixed target. Since we use the long-term predictive model F' to calculate the temporal-difference regression target, we apply the same trick here. We maintain a target network Pa which we set to be equal to the online density estimator F. after fixed number of iterations. In Value Density Imitation, we use the same procedure to maintain a frozen target network of the unconditional state density estimator d
As our method depends on density estimation, the resulting values are heavily affected by the scale of the features. We therefore normalize our data based on the range observed in random roll-outs as well as, in the case of imitation learning, the range seen in the given demonstra- tions.
Normalizing states:
B.2 VALUE DENSITY IMITATION
Averaging logits: While the dimensionality of the goal in goal-conditioned reinforcement learning typically small, Value Density Imitation requires us to use the entire state goal. This, however, can be difficult if the number of features is large. If the state-features are independent, the density suffers from the curse of dimensionality as it is multiplicative and the Q-values will be either extremely large or extremely small. Even if the true density function does not exhibit this property, Normalizing Flows predict the density as a product N predicted logits and prediction errors are therefore multiplicative. To combat this, take the average of the predicted logits rather than the sum, effectively taking the N - th square root of the O-function. We find that this approximation works well in practice and justify it further in appendix D
Bounding weights on demonstrated states: In Value Density Imitation, we sub-sample demon- strations states proportional to (3) to ensure demonstration states to be visited with equal probability. In this formulation, demonstration states that are especially difficult to reach may be over-sampled by large factor and destabilize the learning process. To counteract this, we put bound the weight of each demonstration state: for each batch, the weights are normalized and an upper bound is applied.
Spatial and temporal smoothing: We apply two kinds of smoothing to the learned UVFA to improve the stability of the learning algorithm. Spatial smoothing ensures that similar state-action pairs have similar value and is achieved by adding Gaussian noise to training samples of the target state when training the density estimator. Temporal smoothing ensures that the learned value does not spike too suddenly. Instead of using F,, (3) as a temporal difference regression target, we use lower bound. Using a temporal
mixture of the density estimation and the temporal-differene smoothing factor X, the full temporal difference loss is then given by:
L(w): = (s, a) 7Q (u(3));))))
Q -=
(1 )) max (Qu(s, Pa(ols,a)
13

---

Under review as a conference paper at ICLR 2021
C ADDITIONAL EXPERIMENT DETAILS
GAL -HalfC StaalChheetahi wh- NonExplotabeee
VDI hhallchheeet-----nennhhhhiaee
7000
1 12 samples
1 samples
5900
4000
4000
2000
2000
12 camples
25
samples
0.0
0.4 Environment Stops
Environment Steps
(a)
(b)
VDI Humanold- Nonpapllae
6000
8000
5000
1 50 samples
Return
10 200 samplex
Return
samples samples Expert
4000
4000
1 50 simples
Bandom
2000 samp
2000
50 200 somp les Expert
Random
1.0
3.5
2.0
1.0
1.5 Environment
2.5 le)
Environment Steps
1eB
(c)
(d)
Figure 4: Individual learning curves for each imitation learning experiment.
we describe imitation learning experiments on non-exploitable versions of the commonly used HalfCheetah and Humanoid benchmark tasks. Figure the individual learning curves for both GAIL and VDI on each domain. Note, that the number of environment steps for GAIL is an order of magnitude larger than VDI and several times higher than what is necessary to solve the unmodified, exploitable variant of these domains. Because takes several days to collect this amount of experience for each run, we terminate some runs early if it is clear they have converged and at least 75 or 150 million environment steps have been taken in each respective domain. While more sample-efficient versions of GAIL. have been proposed (Kostrikov al. 2019) and work well on the original formulation of the domains, we were unable to achieve good results on the corrected version of the domains. It is likely that the large amount of regularization imposed on the discriminator in Discriminator-Acton wwteiior--ttto--CCe Actor-Critic makes this formulation more likely to exploit the velocity given as part of the observation space and that significant effort would have to be put in to find the right to apply this method on the con-exploit domains.
In Section 5.2
D ESCAPING THE CURSE OF DIMENSIONALITY
In section 3 we introduced a method which uses the probability density predicted by a normalizing flow as Q function. We showed that this density Q function a valid Q function with a corresponding reward function that is sensible for many practical task. In this appendix, we consider the numerical properties of the universal value density estimator and propose a slight variation that is easier to handle numerically. To see the numerical challenge in using density estimators as value functions, we take a look another look at single bijector of a RealNVP: here, the bijector
(z) = (sw ( (2)) is predicting affine transformation of = ~ (-) (0, 1) to E Pa (.), i.e. = (z) + (z),
(m) +log Var It is readily The log- deg-density of x is then given Pa (x) Σm (x) *
(z),
apparent that this value can easily explode, especially when used as a target Q value in the mean- squared loss of temporal difference update. To combat this, propose to scale the logits with the
14

---

Under review as a conference paper at ICLR 2021
(1 (0))=~)=)__0))) [0) * (z)
t Ei 45 (x)
+log vेअ As this corresponds
dimensionality N, i.c. we use Pæ (x) = e
to only a constant factor on log Fy, the gradient-based baaee density estimation is not affected.
While the change is simple, average logits instead of taking the sum, we need to justify the approxi- mation anywhere the value density and the Q-function are used: first, we show that ((O) can be used in place of J(0) in both, goal-conditioned reinforcement learning and imitation learning without changing the optimal policy; second, we justify using Q (8,a) a) & in place of Q (s, a) when computing the policy gradient; and, finally, we show that can justify the of the N-th root in the temporal difference learning update.
Using the scaled objective J(0):: Here, we have to consider two cases. In the case of goal- conditioned RL, have to show that maxę J(0) maxę J(Ø) x. To this end, it is sufficient to note that the reward function is strictly non-negative and thus (-) d is a monotonous function. In the case of imitation learning, we can immediately see that the change corresponds constant factor on the state- ditt-diistibbtton gradient:
1 Vo log duo (5) N
(0)
duu (3)
Estimating the policy gradient (0) The deterministic policy gradient theorem (Silver
ct al. 2014) shows that maximizing the Q-value in states sampled from the agent's 's discounted on-policy state-distribution is equivalent to maximizing the reinforcement-learmnng objective. This is immediately apparent if the representation of the policy is sufficiently expressive and agent is able to
take the action with maximum value in every state. Due monotonicity of (-)*² R this is true when using Qx a: g) as well. the policy not able to maximize the Q-function everywhere, the deterministic policy gradient theorem shows that sampling from the discounted state-distribution leads the agent making the right trade-offs. Using ((s, a: g) i however, leads different trade-off. In practice, this is typically ignored: deterninistic policy gradients used off-policy with a replay-buffer are not guaranteed to make the right trade-off and even if they are used on-policy, the discount-factor is typically ignored and the resulting estimate of the policy gradient is biased (Nota) and Thomas 2019).
(H, a) & as TD target: Finally, we need to show that we can use the N-th root in the computation of the temporal difference learning target. To this end, we make use of the fact that the reward in continuous environments can be assumed be when computing the temporal difference target. In
this case (s, a) + +HQ (s', p(s')))* becomes à Q (s', p(s')) and increasing 7 is sufficient compute valid regression target for Qt (s, a)
15


Reviews:
[{"review_id": "jEjm71bVrFC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors offer the perspective that the goal-conditioned Q-value for state-action $(s,a)$ can be viewed as the density of that goal ($g$) under a predictive model $F(g|s,a)$. The density estimator is modeled with normalizing flows (RealNVPs) and trained with MLE using policy rollouts. The estimator is integrated with TD-learning and actor-critic algorithms for goal-conditioned-RL and imitation-learning are derived. Experiments for the former include a comparison with HER on fetch-push/slide tasks, while for IL, the proposed approach is compared with GAIL on two locomotion tasks and shown to exhibit superior demonstration-efficiency.\n\nI would like to appreciate the authors for an excellent presentation of the material. The notation used is rigorous and this makes understanding the hairy parts of the approach easier. Unifying goal-conditioned-RL and IL under a common view of training an agent to reach desired states and leveraging density estimators to achieve that is an interesting approach. I do have some concerns which (currently) hold me from rating this paper higher:\n\n1.\tUVD in stochastic domains: It is argued that HER suffers from hindsight bias in stochastic domains; Figure 2d empirically shows this. However, the dip in the performance of UVD from deterministic to stochastic Fetch-slide is also noticeable. I am wondering if this is because the density estimation becomes challenging in highly stochastic domains, since the support of the modeled distribution (over $g$) gets wider with increasing stochasticity, assuming the time-horizon of the density estimator is the same. Could the authors please comment on this observation from Figure 2?\n\n2.\tAdding Fetch-push-noise results in Figure 2, using the same noise-model as that used for Fetch-slide-noise, would make for a better claim.\n\n3.\tLocomotion tasks modification: These tasks are modified by removing the x/y velocities from the state given to the agent at each timestep. However, these components are used in the internal transition-dynamics function and also the reward function. Therefore, I reckon that this modification changes the MDP into a POMDP. Does that sound correct? If yes, it’s a bit strange that the RL machinery developed for MDPs (GAIL and VDI) is applied to a POMDP task. For reference, it would be good to include the results w/o the velocity removal in the Appendix.\n\n4.\tAlgorithm 1 vs. Algorithm 2: Is there a particular reason why Algorithm 2 does not use the immediate rewards and the density-estimator from the \"next\"-state-action in the TD-error computation, as it does in Algorithm 1?\n\nTypo: $F_\\omega$ should be $F_\\phi$ in Algorithm 2?\n\n\n===== POST REBUTTAL UPDATE =====\n\nI maintain my score of 6.  While the proposed approach is interesting, the experimental section could be strengthen to firmly stake a claim that it's broadly applicable.  (c.f. point 2. in the review)", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZCYkQfqOjIe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for solving (state-distribution matching) imitation learning and goal-conditioned reinforcement learning (GCRL) by training a value function both with standard TD learning and Monte Carlo (MC) learning. They point out that for the case of indicator rewards, the Q function corresponds to a density model over the discounted state occupancy measure, and so they propose to train the MC-Q function by training a normalizing flow via MLE. The authors propose to combine the estimates of the two during TD backups by taking the max of the two estimates, and demonstrate that the overall algorithm results in better performance on the Fetch GCRL tasks (relative to HER) and on “non-exploitable” versions of locomotion tasks which remove the velocity and termination information that past work on imitation learning has included.\n\nOverall, the paper studies an interesting connection between GCRL and imitation learning, and the idea of training a goal-conditioned value function via MLE seems novel and intriguing. I have concerns that the contribution of the paper is a bit unclear, and the overall performance gains are rather small.\n\nIn more detail:\n\nIt’s unclear if the benefits are from the “Q function is a density model” perspective, or if the benefits are just coming from mixing TD and Monte-Carlo estimates together. For example, how would the performance change if the normalizing model F was simply replaced with a Q function trained on Monte-Carlo estimates?\n\nThere were a number of design decisions and claims that I did not understand. For one, I don’t understand why VDI is immune to hindsight bias. The authors have experiments that shows that there is some small performance gain when stochasticity is introduced, but it seems that the distribution of goals that F is trained on will similarly be biased. Another question is why the authors propose to approximate the two values by taking their max. Is there any theoretical reason why this would be desired? Lastly, what does it mean for a goal to be “within the time-horizon of the density estimator”? Is there a mathematical definition of this condition, and why taking the max would address this problem?\n\nThe related works section is generally good, though I recommend the authors also discuss the relationship to [1] which also uses the gradient of the state distribution to train the policy.\n\nLastly, perhaps the main weakness is that the performance gains seem relatively small, which makes the impact of the work relatively small given that the main contribution seems to be empirical.\n\n[1] Hazan, Elad, et al. \"Provably efficient maximum entropy exploration.\" International Conference on Machine Learning. 2019.\n\n\n--- Post Rebuttal ---\n\nI had some confusion and concerns about the paper. Most of the confusions were addressed and made me view the paper slightly more favorably. However, my main concern wasn't addressed. In particular, it's unclear if the benefits are because of the \"Value Density Estimation\" benefits or if it's because of the mix of non-bootstrap updates. Comparing to a method that mixes in more Monte Carlo estimates (e.g. n-step bootstrap) would address this concern. Theoretically, the authors suggest that the reward should be 1/epsilon with epsilon -> 0. However, it seems like this was not done in the experiments, and it's unclear how epsilon should be decayed in practice. Given these concerns, I will maintain my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MfbTuUGak5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method to compensate for \"hindsight bias\". This bias can occur in stochastic environments, during the hindsight relabeling process, when the value of actions are overestimated due to the probability of the success of that action in that state being assumed deterministic. The paper proposes Universal Value Density Estimation to account for this overestimation. The method learns approximates the proper goal conditioned Q function using a value density estimate.\n\nWell the discussion at the end of the related work section will most related papers is interesting it's a bit difficult to start to see the novelty and addition this paper has to this problem when previous methods have solved this task for finite Horizons.\n\nThe example in figure 1 is very odd it shows an action choice that is basically impossible to cross so you would never end up constructing or relabeling a trajectory that would reach another state beyond the black hole where the agent would fall off the cliff.\n\nAlgorithm 2 does not appear to be referenced in the paper, It would be far more helpful if the authors could highlight the differences between these algorithms carefully in the text. \n\nThe results in this paper we only shown for a single environment. This does not elicit confidence in the paper's findings. Do either of the domains chosen for the goal conditioned rl tasks have stochastic transitions that would lead to a need for UVD? Artificially adding noise does not seem like a motivating situation for the use of the method...", "extra_scores": {}, "extra_text": {}}, {"review_id": "1QdLwNct6g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper focuses on goal-conditioned policy learning in the environment with stochastic dynamics and tries to address the bias of HER. Given a policy and a sparse reward function according to the goal,  the universal value function is converted to the discounted probability density over goals. And the authors use the recent work RealNVP about density estimation to learn the universal value function. The proposed value density estimation is also extended to imitation learning. Experiments show that the proposed method outperforms baselines in both goal-conditioned policy learning and imitation learning.\n\nClarity:\nThe clarity in writing can be improved in some points. First, it will be good if more details about RealNVP are introduced. How is it applied to learn the probability density? Second, in section 3.3, \"if the goal is within the time-horizon of the density estimator....\". Does it mean another hyper-parameter, e.g. time horizon of the density estimator, is introduced here? Is the proposed method sensitive to the value of this hyper-parameter? Third, in figure 1, do the arrows represent the learned policy with HER and UVD. It will be more convincing if the learning curves on this toy sample is also presented (in the appendix). \n\nOriginality:\n\nSignificance:\nThe proposed method is technically sound and experiments demonstrate that it performs well in comparison to the baselines in some continuous control tasks. My concern lies in the complexity of implementing the proposed method compared with the baselines, and whether the method can be easily used in other domains with rich observation space and discrete action space. \n\nPros:\n*The proposed method is well-motivated to solve a problem of HER for goal-conditioned policy.\n*The proposed method works well in some continuous control tasks and the authors carefully discuss the advantage of the proposed method in different environment settings (Figure 2) and with the different datasets (Figure 3).\n\nCons:\n*In table 1 and table 2, I notice the use of RealNVP introduces some hyper-parameters. Is it easy to tune these hyper-parameters? \n*Is the proposed method applicable to domains with discrete action space?\n*In Appendix D, there is a discussion about dimensionality. The data samples on Mujoco has low-dimensional state features.  If the observation space is more complicated and rich (e.g. RGB images), could the proposed method be easily used for imitation learning as GAIL?\n*What's the complexity of implementing the proposed method compared with the baselines? Can we easily use it as a better alternative for HER in goal-conditioned policy learning and GAIL in imitation learning without much additional cost of tuning or running the algorithm?\n\n***Post Rebuttal***\nThank the authors for the response. My main concern about the generality of the proposed method is not fully addressed. The baseline HER and GAIL can easily be applied in the domains with discrete action space. Also, the base RL approach for discrete action space exists. Experiments will be more convincing if the proposed method can outperform the baselines on various domains.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes an interesting connection between the density matching in imitation learning and reaching the goal state in goal-oriented reinforcement learning. Reviewers generally expressed that the paper proposes an interesting approach, but some aspects of the paper need room for improvement. By reinforcing the experiments that address the reviewers’ various concerns, this paper will make a good contribution towards reinforcement learning research.
Decision: Reject