Outcome-Driven Reinforcement Learning via Variational Inference
Authors: Tim G. J. Rudner
Vitchyr H. Pong
Rowan Thomas McAllister
Yarin Gal
Sergey Levine
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Variational Inference
Goal Reaching

Abstract: While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design decisions, such as manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view reinforcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us to derive a well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand-craft reward functions for a suite of diverse manipulation and locomotion tasks and leads to effective goal-directed behaviors.

Outeome-Driven Reinforcement Learning via Variational Inference
Tim G. J. Rudner* University of Oxford
Vitchyr H. Pong*t University of California, Berkeley
Rowan McAllister University California, Berkeley
Yarin Gal
Sergey Levine University of California, Berkeley
University Oxford
Abstract
While reinforcement learning algorithms provide automated acquisition of optimal policies, practical application of such methods requires a number of design deci- sions, such manually designing reward functions that not only define the task, but also provide sufficient shaping to accomplish it. In this paper, we view rein- forcement learning as inferring policies that achieve desired outcomes, rather than as a problem of maximizing rewards. To solve this inference problem, we establish a novel variational inference formulation that allows us derive well-shaped reward function which can be learned directly from environment interactions. From the corresponding variational objective, we also derive a new probabilistic Bellman backup operator and use it to develop an off-policy algorithm to solve goal-directed tasks. We empirically demonstrate that this method eliminates the need to hand- craft reward functions for suite of diverse manipulation and locomation tasks and leads to effective goal-directed behaviors.
1 Introduction
Reinforcement learning (RL) provides an appealing formalism for automated learning of behavioral skills, but requires considerable care and manual design to use in practice. One particularly delicate decision is the design of the reward function, which has a significant impact on the resulting policy but is largely heuristic in practice, often lacks theoretical grounding, can make effective learning difficult, and may lead to reward mis-specificction.
To avoid these shortcomings, we propose to circumvent the process of manually specifying a reward function altogether: Instead of framing the reinforcement learning problem as finding a policy that maximizes heuristically-define reward function, we express it probabilistically. as inferring state-action trajectory distribution conditioned on desired future outcome. By building off of prior work on probabilistic perspectives on RL [10, 23, 35, 46. 47, 58] and goal- goal-directed RL in partieular [3, 6, 19, 48), we derive tractable variational objective, an temporal lemporal-difference algorithm that provides shaping-like effect for effective learning, as well as reward function that captures the semantics of the underlying decision problem and facilitates effective learning.
We demonstrate that unlike prior works that proposed inference methods for finding policies that achieve desired outcomes [3, 11, 19, 48], the resulting algorithm, Outcome-Driven Actor-Critic (ODAC), is amenable to off-policy learning and applicable to complex, high-dimenssonal continuous control tasks over finite and infinite horizons. The resulting variational algorithm can be interpreted as an automatic shaping method, where each iteration learns a reward function that automatically pro- vides dense rewards, as we visualize in Figure 1. tabular settings, ODAC is guaranteed to converge
Equal contribution. Corresponniigg authors: tim. rii.erunerccs. ox. ac. uk vätchyr@berkeley.co
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

sparse
iteration
iteration
Figure 1: Illustration of the shaping effect of the reward function derived from the goal-directed variational inference objective. Left: 2-dimensional grid world with a desired outcome marked by a star. Center-left The corresponding sparse reward function provides little shaping. Center-right: The reward function derived from our variational inference formulation at initialization. Right: The derived reward function after training. We see that the derived reward learns to provide dense reward signal everywhere in the state space.
to an optimal policy, and in non-tabular settings with linear Gaussian transition dynamics, the derived optimization objective is convex the policy, facilitating easier learning. In nigh-dimensional and non-linear domains, our method can be combined with deep neural network function approximators to yield a deep reinforcement learning method that does not require manual specification of rewards, and leads good performance range of benchmark tasks.
Contributions. The core contributions of this paper are the probabilistic formulation of general framework for inferring policies that lead to desired outcomes and the derivation of a variational objective from which obtain novel nutcome-driven Bellman backup operator. We show that this Bellman backup operator induces a shaping- effect which ensures a clear and dense learning signal in the early stages of training. Crucially, unlike heuristic approaches for incorporating shaping often used in standard RI., this "shaping" emerges automatically from variational inference. We demonstrate that the resulting variational objective is lower bound the tog-marginal likelihood of achieving the outcome given an initial state and that it leads to an off-policy temporal-differenee algorithm. We evaluate this algorithm-Outcome Daiieme Variational Inference (ODAC)-on range of reinforcement learning tasks without having to manually specify task-specific reward functions. In our experiments, we find that our method results in significantly faster learning across variety of robot manipulation and locomotion tasks than alternative approaches.
2 Preliminaries
Standard reinforcement learning (RL) addresses reward maximization in Markov decision pro- cess (MDP) defined by the tuple A, PS..Pd: r. ?) [43, 44), where S and A denote the state and action space, respectively, Po denotes the initial state distribution, Pa is state transition dis- tribution, r is an immediate reward function, and is a discount factor. To sample trajectories, an initial state is sampled according to PS, and successive states are sampled from the state tran- sition distribution S, ~ pp(- St, ar) and actions from a policy A, ~ nT- TT- s.). We will write T 0: (So, Ao, SI, .I,...5 Se, A.) } to represent a finite-horizon and To - (A,, S 1 repre- sent an totinite-horizon stochastic state- state-action trajectory, and write To: = {so. ag, $1.... Sp. a and To = {a,, st+1] - for the respective trajectory realizations. Given a reward function S X R and discount factor (0. 1). the objective in reinforcement learning is to find policy π that max- imizes the returns, defined as Ey, EE 7'r(s, at)] where pr. denotes the distribution of states induced by policy π.
Coal-Conditioned Reinforcement Learning. In goal-conditioned reinforcement learning [22], which can considered a special case of the broader class of stochastic boundary value problems [L. 14], the objective is for an agent to reach some pre-specified goal state, E S, so that the policy and reward function introduced above become dependent on the goal and are expressed 7(a s, s. and r(s, a, g). respectively. Typically, such a reward function needs to be defined manually. with common choice being to use a sparse indicator reward r(s, g) I(s g). However, this approach presents a number of challenges both in theory and in practice. From a theoretical perspective, the indicator reward will almost surely equal zero for environments with continuous goal spaces and non-trivial stochastic dynamics. From practical perspective, such sparse rewards can be slow to learn from, as most transitions provide no reward supervision, while manually designing dense reward functions that provide better learning signal is time-consuming and often based on heuristics.

---

In Section 3, we will present framework that addresses these practical and theoretical considerations by casting goal-conditioned RL as probabilistic inference.
Q-Learning. Off-policy Q-learning algorithms [52] allow training policies from data collected under alternate decision rules by estimating the expected return Q for given state-action pair:
Q" (s. a) = Ep, > 20 r
a.) So s, Ao
Crucially, the expected return given state-actin pair can be expressed recursively as Qa (s. a) = r(s, a) + Ep. [Q" a1) S. = s, Ao a),
(1)
which makes possible estimate the expectation on the right-hand side from single-step transitions. The resulting estimates can then be used to find policy that results in actions which maximize the expected return Q" (s, a) for all available state- sate-action pairs.
3 Outvome-Driven Reinforcement Learning
In this section, we derive a variational inference objective to infer an approximate posterior policy for achieving desired outcomes. Instead of using the heuristic goal-reaching rewards discussed in Section 2, we will derive a general framework for inferring actions that lead to desired outcomes by formulating a probabilistic objective, using the tools of variational inference. As we will show in the following sections, we use this formulation to translate the problem of inferring policy that leads desired outcome into tractable variational optimization problem, which we show corresponds to an RL problem with a well-shaped, dense reward signal from which the agent can learn more easily.
We start with warm-up up problem that demonstrates how frame the task of achieving desired outcome as an inference problem in simplified setting. We then describe how to extend this approach more general settings. Finally, show that the resulting variational objective can be expressed as a recurrence relation, which allows us to derive an outcome-driven variational Bellman operator and prove an outcome-driven probabilistic policy iteration theorem.
3.1 Warm-up: Achieving Desired Outcome at a Fixed Time Step
We first consider a simplified problem, where the desired outcome is to reach specific state g E S at specific time step 1* when starting from initial state Sp. We can think the starting state So and the desired outcome g as boundary condi- tions, and the goal is to learn a stochastic policy that induces a trajectory from So to g. To derive a control law that solves this stochastic boundary value problem, we frame the problem probabilistically, as inferring a state action trajectory posterior distribution conditioned on the dexired oulcome and the initial stale. We will show that, by framing the learning problem this way, we obtain an algorithm for learning outcome-driven policies without needing to manually specify reward function. We consider model of the state- date-action trajectory up and including the desired outcome
So
S
S
S.i
Ao
A
Ar-
Figure 2: trajectory with observed random variables = Su and S.-
A Probabilistic graphical model of state-action
PTo... S.. (To g so) = pa(g st a p(a ss) II t==0
pa(sy | Sy ar )p(ay sy)),
where 1" - 1, T 0:t is the state-action trajectory up to an including & but excluding Sg. p(a, (s a conditional action prior, and pa(S Sc. a,) is the environment's state transition distribution. If the dynamics are simple (e.g., tabular or Gaussian), the posterior over actions can be computed in closed form [3], but we would like to be able to infer outcome-driven posterior policies in any environments, including those where exact inference may intractable. To do so, we start by expressing posterior inference the variational minimization problem
min
DKL (14 (-18u) Prr..S.. (-|Su, g)),
(2)
07.1185 EQ
3

---

where (-1-) is the KL divergence, and Q denotes the variational family over which to optimize. We consider family of distributions parameterized by policy x and defined by
f-1
or (Fost 8o) maa se) II pa(se41 |se, ar ( (se);
(3)
e==
where π II, a family of policy distributions, and where I Pa(s:+1 St. a²) is the true action- conditional state transition distribution up to but excluding the state transition at t* - 1. since S,- observed. Under this variational family, the inference problem in Equation (2) can be equivalently stated as the problem of maximizing the following objective with respect to the policy æ::
Given I (Toie (a) from Equation (3), any state So, termination time t*, and outcome solving Equation (2) is equivalent to maximizing this objective with respect to II:
Proposition 1.
F(T,, g) = =Buro. 80 (For (s)) log logpa(g st, ar)
Σ DK (se) ||p( (o((-ss)))
(4)
Proof. See Appendix A.I.
variational problem of this form-which corresponds to finding a posterior distribution over state-action trajectories-con can equivalently be viewed as a reinforcement learning problem:
Corollary 1. The objective in Equation (4) corresponds to KL-regularized reinforcement learning with a time-varying reward function given by r(se, ay g, t') = ll(t' = t} log pa(g Sy ar).
Corollary 1 illustrates how reward function emerges automatically from a probabilistic framing of outcome-driven reinforcement learning problems where the sole specification is which variable (S,)) should attain which value (g). In particular, Corollary 1 suggests that we ought to learn the environment's i state-transition distribution, and view the log-likelihood of achieving the desired outcome given state-action pair "reward" that can be used for off-policy learning as described in Section 2. Importantly- and unlike in model- RL RL-such- transition model would not have be accurate beyond single-step predictions,
would not be used for planning (see Appendix B). Instead, log Pa(g st, ax) only needs to be well shaped, which we expect to happen for commonly used model classes. For example, when the dynamics are linear- Gaussian, using a conditional Gaussian model [29] yields a reward function that is quadratic in S making the objective convex and thus more amenable to optimization.
3.2 Ourvo Reinforcement Learning as Variational Inference
Thus far, we assumed that the time at which the outcome is achieved is given. In many problem settings, we do not know (or care) when an outcome is achieved. In this section, we present a variational inference perspective and no termination time are given, but only desired outcome provided. As in the previous section, we derive variational objective and show that principled algorithm and reward function emerge automatically when framing the problem as variational inference.
achieving desired outcomes in settings where no reward function
To derive such an objective, we modify the probabilistic model used in the previous section to model that the time at which the outcome is achieved is not known. As before, we define an "outcome" as point in the state space, but instead of defining the event of "achieving desired outcome" as realization S,, = g for a known 1*, we define it as realization ST, = g for an unknown termination time 7 at which the outcome is achieved. Specifically, we model the distribution over the trajectory and the unknown termination time with
Pro So (Fo.. t so) = (t)ppa( st a )(a (s))
II pa Syr, ay )p(av Sy)). (5) ===0
where (t( is the probability of reaching the outcome at t + 1. Since the trajectory length is itself
random variable, the joint distribution Equation (5) is a transdimensional distribution defined on (1) x st+1 ++11 [19].
Unlike in the warm-up, the problem of finding an outcome-driven policy that eventually achieves the desired outcome corresponds to finding the posterior distribution over state-action trajectories and

---

the termination time 7 conditioned on the desired outcome Sy- and starting state. Analogously Section 3.1, we can express this inference problem variationally as
min (1t. T
(0)) 7 (-150, g)),
(6)
TTo. 90 EQ
where / denotes the time immediately before the outcome is achieved, @ denotes the variational family. In general, solving this variational problem in closed form is challenging. but by choosing a variational family of. T.cr.T/S. (fo.s I (1))) = 4 (T. S. (Foze (, (,))))(()) where yr is a distribution
over T in some variational faailly Qr parameterized by
@r (t) = 9.. (A++1 = 1) II (A. = =), e'=1
with Bernoulli random variables A denoting the event of "reaching g time given that has not yet been reached by time - 1," we can equivalently express the variational problem in Equation (6) in way that is tractable and amenable to off- off-policy optimization:
Theorem 1. Let (t) and I.. (Fuze It, so) be as defined before, and define
(T g: (T) Enja. 1xt) 1Q" (St; ar g: QT)) (xx.(T(-1 (-((1ss))
II pl 8:))
(8)
Q" as, g: (r) = r(s,, ac. g: 9 + 9 (Ai+1 = 0) E E 11 s. [VT (S1+1, g: n, x,
(9)
r(se, as, gi qa) = q (Ae+1 1) log (d St,, ar) Wxs PA.:.). Then given any initial state So and outcome g.
(10)
DRL. (kk.(9to.r:T5
So) PT.rr..SSsss (-1 (-180,8) g)) -V* 8: qT) + logp(g so),
where pig (8)) is independent of π and qr and hence maximizing V*(So= g; n. (r) is equivalent
to minimizing Equation (6).
Proof. See Appendix A.2.
This theorem tells us that the maximizer of V (*.ss, gi QT) is equal the minimizer of Equation (6). In other words, Theorem presents a variational objective with dense reward functions defined solely in terms of the desired outcome and the environment dynamics, which we can learn directly from environment interactions. It further makes precise that the variational objective, V (So g; 2r), is lower bound on the log marginall likelihood, that is, log so) 2 V* g: qr), where
t
- =0) -0) (r(ss, g: 9 Bx ( (s) Il I 1s)1ss))
(= g: QT) E
Σ
(s. ('=l
t=0
with the expectation taken with respect to the infinite-horizon trajectory distribution 97, So. Ao 8o). Thanks to the recursive expression of the variational objective, we can find the optimal variational over 7 a function of the current policy and Q-function analytically:
Proposition The optimal distribution 4fr with respect to Equation (8)
(Aut1 0; n. Q*) (A(s), n. or, Q²) +o- - (A = O)))
(ps
(11)
where
A(st, n. QT, Q²) = Erraaess (pa1)) ( 1x.) (x)) 1Q" (5141, a1+1: QT) log palg s,,
and (() is the sigmoid function, that is, (xx) I=1 and o (x) log 1-xx
Proof. See Appendix A.3
Alternatielly, if instead of learning ou variationally, we fix qr to the prior Pr, we recover the more conventional fxed-discoum factor objective [13, 16, 39]:
Corollary 2. Let 4r = Pr, assume that pr is Geometric distribution with parameter E (0, 1).
Then the inference problem Equation (6) of finding a goal-directed variational trajectory distribu- tion simplifies to maximizing the following recursively defined variational objective with respect to n:
(So, = TEx ( (Q ag. g: 7)] - ( ( So) p(- (o)));
(12)
where
Q' (s au. au.g: 7: = (1 (1-7) log so. an) + Ep (o)) [(si. 7)]-
(13)
In the next section, we derive temporal-difference algorithm and discuss how we can learn the Q-function in Theorem I using off-policy transitions.
5

---

Outcome-Driven Reinforcement Learning
In this section, we show that the variational objective in Theorem amenable to off-policy learning and that it can be estimated efficiently from single-step transitions. We then describe how instantiate the resulting outcome-driven algorithm in large environments where function approximation is necessary.
4.1 Salcome-Driven Policy Iteration
To develop outcome-directed off off-policy algorithm, we define the following Bellman operator: Definition 1. Given function Q S x A x S + R, define the operator T as
T (s,, a ag qT) r(s,, a a q&) + qd (Au+1 = =0) Bpa(se+1 (s,) ((((411.:: (14)
where r(s,, a g: q1) is from Theorem I and
V(s,, gi (T) = Enta, s.) (Q a, gi qT)l ExL (TT- (s)) ||
(15)
Unlike the standard Bellman operator, the above operator has a varying weight factor 9 (A++1 =). with the optimal weight factor given by Equation (11). From Equation (11), we see that as the outcome likelihood pa(g s, a) becomes large relative to the Q-function, the weight factor automatically adjusts the target rely more the rewards.
Below, we show that repeatedly applying the operator T (policy evaluation) and optimizing и with respect to Q" (policy improvement) converges policy that maximizes the objective in Theorem 1. Theorem 2. Assume MDP is ergodic and [A| < co.
Outcome-Driven Policy Evaluation (ODPE): Given policy and function Qo 8 x AxS TR, define Qi+1 = =T=Q'. Then the sequence Q' converges to the lower bound in Theorem 1.
2. Outcome-Driven Policy Improvement (ODPI): The policy
arg max (ss) [Q" ( (st, a g: qr)] - DKL( ((p())))) x'ETI
π
(16)
and the variational distribution over T defined in Equation (11) improve the variational objective, that is, F( F( s0) 2 F(T, qr, So) and F(T,, 97 .o) ( 2 F(x, qr. so) so) for all So. n, qr-
3. Alternating between ODPE and ODPI converges to a policy ग* and a variational distribution over T. Qr, such that Q (s, a, g: 4T) 2 (s, a.s.a,g:90 gi (r) for all (T, qr) E x Qr and any
(s. a) E S A.
Proof. See Appendix A.4.
This result tells us that alternating between applying the outcome-drin driven Bellman operator in Defini- tion 1 and optimizing the bound in Theorem 1 using the resulting Q Qufunction, which can equivalently be viewed as expectation maximization, will lead to policy that induces an outcome-driven trajectory and solves the inference problem in Equation (6). As discuss in Appendix A.4, this implies that Variational Outcome-Drivee Policy Iteration is theoretically at least as good as or better than standard policy iteration for KL-regularized objectives.
4.2 Outcome-Driven Actor-Critic (ODAC)
We now build on previous sections to develop a practical algorithm that handles large and continuous domains. In such domains, the expectation in the Bellman operator in Definition I is intractable. and so we approximate the policy Ty and Q-function Qo with neural networks parameterized by parameters 0 and ø, respectively. We train the Q-function minimize
Fo()) E [(aass...) (r(s, (A, = =0) V(s ()(()))))) {)
(17)
where the expectation is taken with respect to as,aggss') sampled from replay buffer, D, of data collected by policy. We approximate the V V-function using target Q-function Q3:

---

Algorithm 1 ODAC: Outcome-Driven Actor-Critic
1: Initialize policy Te. replay buffer R, Q-function Qs and dynamics model Pú-
2: for iteration i = 1, ... do
3: Collect on-policy samples to add to R. by sampling g from environment and executing
4: Sample batch (s, a, s', g) from R.
5: Compute approximate reward and optimal weights with Equation (20) and Equation (11).
6: Update Qs with Equation (17), सy with Equation (18), and py with Equation (19).
7:
end for
(s' g) x Qa(s', a', - logo (a' g), where a' are samples from the amortized variational
policy Ne(- s': We further assume a uniform prior policy pl: (s)) in all of our experiments. The parameters slowly track the parameters of at each time step via the standard update T + (1 -) [27]. We then train the policy to maximize the approximate Q-function by performing gradient descent on
(0) = -By-Daanaa (-1 (Q a. g) - log (a s: g:g]
(18)
We estimate (A.+1 (At+1 0) with a Monte Carlo estimate of Equation (11) obtained via single Monte Carlo sample (s, a, s,," a)) from the replay buffer. practice, value of - (A 0) = can lead to numerical instabilities with bootstrapping, and so we also up- per bound the estimated ga... (Ar+1 =0) 0) by the prior distribution PA.., (Ar+1 =).
To compute the rewards, we need to compute the likelihood of achieving the desired outcome. If the transition dynamics are unknown, we learn a dynamics model from environment interactions by training a neural network Pe that parameterizes the mean and scale of a factorized Laplace distribution. We train this model by maximizing the log-likelihood of the data collected by the policy,
FV) (=(...s")=- |s,
(19)
and use it to compute the rewards
f(St, a,, gi (A) ب (A++1 = 1) log Pog St, a.) - DKL(4A)) || PA.).
(20)
The complete algorithm is presented in Algorithm and consists of alternating between collecting data via policy and minimizing Equations 17, 18, and 19 via gradient descent. This algorithm alternates between approximating the lower bound in Equation (8) by repeatedly applying the outcome-driven Bellman operator to an approximate Q-function, and maximizing this lower bound by performing approximate policy optimization on Equation (18).
Related Work
Our problem definition is related coal-conditioned RL, where desired outcomes are often defined in terms of an exact goal-equality indicator [22, 40]. Unlike in our work, goal-conditioned RL typically requires specifying reward function reflecting the desired outcome. The most natural choice for a reward function in this setting is an exact goal-equality indicator, which gives non-zero reward whenever outcome is reached. However, this type of reward function makes learning difficult, since the associated reward signal is sparse at best and impossible to attain at worst (in continuous state spaces).
To overcome this limitation, prior work has proposed heuristics for creating dense reward functions, such as the Euclidean distance or thresholded distance goal [2, 26, 28, 32, 33, 41], estimating auxiliary metrics to encourage learning, such as the mutual information or time between states reward function results
and goal [9, 18, 34, 50, 51]. In contrast, a dense, generally automatically from our variational objective. In Section 6, we demonstrate that this reward function is substantially easier to optimize than sparse rewards, and that it removes the need to choose arbitrary thresholds or distance metrics needed in alternative approaches.
Several prior works cast RL and control as probabilistic inference [10, 11, 19, 25, 35, 42, 45, 58] or KI. divergence minimization [24, 31], but with the aim of reformulating standard reward-based RL, assuming that the reward function is given. In other words, these prior works similarly study
7

---

(a) Box 2D
(b) Ant
(c) Sawyer Push (d) Fetch Push
(e) Faucet
Window
Figure From left to right, we evaluate on: a 2D environment in which an agent must move around a box, a locomotion task in which quadruped robot must match a location and pose (yellow), and four manipulation tasks in which the robot must push objects, rotate faucet valve, or open window.
how optimizing T mmmmal a) + HT) corresponds to solving probabilistic inference problem where Tmanual used to define the outcome's log-likelihood function. However, these prior methods assume that Tmmnual is provided ex ante. In contrast, our work removes the need to manually specify task-specific reward or likelihood function, and instead derives both an objective for learning an environment--pecifie likelihood function and learning algorithm from the same inference problem.
Past work has also studied control probabilistic inference in the context of reaching goal or desired outcome [3, 11, 19, 47]. Toussaint et al. [48] and Hoffman et al. [19] focus on exact inference methods that require time- varying tabular or time-narying Gaussian value functions. In contrast, we propose a variational inference method that eliminates the need to train time-varying value function, and enables to use expressive neural networks represent an approximate value function, making our method applicable to high-dimensional, continuous, and non-linear domains. Unlike the approach in Attias [3], our formulation is not constrained fixed-horizon settings, obtains a closed-looy rather than open-loop policy, and is applicable to non-tabular dynamic models. More recently, Fu et al. [11] proposed a probabilistic inference method for solving the unknown time-step formulation, but required on-policy trajectory samples. In contrast, we derive off-policy method by introducing variational distribution (r over the time when the outcome is reached.
Lastly, the closely related problem of finding control laws that allow agents to move from an initial state to some desired goal state while incurring minimal cost has been studied in the stochastic control literature [5, 15, 20, 54]. Rawlik et al. [36] consider continuous-time setting and propose an expectation maximization algorithm that assumes linear Gaussian dynamics, while more recent work has explored finding control laws for non-linear stochastic system dynamics [37, 55). In contrast this strand of research, our framework considers a discrete-time setting and does not make assumptions assumes knowledge of the system dynamics but only requires the ability to interact with the environment to learn an outcome-driven policy.
6 Empirical Evaluation
Our experiments compare ODAC to prior methods for learning goal-conditioned policies and evaluate how the components of our variational objective impact the performance of the method. Specifically, we compare ODAC to prior methods on wide range of manipulation and locomotion tasks that require achieving a desired outcome. To answer the second question, we conduct several ablation studies and visualize the behavior of (a.., (Ar+1 0). In our experiments, we use a uniform action prior p(a) and the time prior pr is geometric with parameter 0.01, that is, Pa.+1 (Ar+1 = 0) = 0.99. We begin by describing prior methods and environments used for the experiments.
6.1 Learning Achieve Desired Outcomes
To avoid over-fitting to any one setting, we compare to these methods across several different robot morphologies and tasks, all illustrated Figure the appendix.
Environments. We compare ODAC to prior work on simple 2D navigation task, in which an agent must take non-greedy actions to move around box, well as the Ant, Sawyer Push, and Fetch Push simulated robot domains, which have each been studied in prior work on reinforcement learning for reaching goals [2, 28, 30. 34, 41]. For the Ant and Sawyer tasks, desired outcomes correspond to full states (that is, desired positions and joints). For the Fetch task, we use the same goal representation in prior work [2] and only represent g with the position of the object. Lastly, we demonstrate the

---

Normalized Distance
Box 2D
Normalized Distance
Ant
Normalized Distance
Sawyer Push
1.0
0.5
0.3
0.5
0.0
0.0
0.0
0K
20K Environment Steps
40K
0K
200K
400K
0K
200K
400K
Environment Steps
Environment Steps
Sawyer Faucet
Sawyer Window
Normalized Distance
Fetch Push
Normalized Distance
1.0
Normalizes alized Distance
1.0
1.0
0.5
0.5
0.0
0.0
0.0
0K
200K
400K
0K
200K
400K
0K
200K
400K
Environment Steps
Environment Steps
Environment
Steps
ODAC (ours)
HER
SAC (hand trafted reward)
DISCERN
Figure learning curves showing final distance vs. environment steps all six environments. Only ODAC consistently performs well on six tasks. Prior methods struggle to learn. especially in the absence uniform goal sampling, See text for details.
feasibility of replacing manually designed rewards with outcome-driven paradigm by evaluating the methods on the Sawyer Window and Sawyer Faucel tasks from the Meta World benchmark [56]. These tasks come with manually designed reward functions, which we replace by simply specify desired outcome g. We plot the mean and standard deviation of the final Euclidean distance to the desired outcome across four random seeds. We normalize the distance to be 1 at the start of training. For further details, see Appendix C.I.
Goal Sampling. In all tasks, rather than assuming that we are able to perform oracle goal sampling (which can be difficult or even impossible in complex real-world environments), fixed desired outcome is commanded as the exploration goal in each episode. During training, the goals are relabeled using the future-style relabeling scheme from Andrychowicz et al. [2]. Unlike oracle goal sampling, this approach does not assume knowledge of the set admissible states in the environment, but is more realistic and presents we choose the desired goal to be far from the starting state.
challenging exploration problem. To challenge the methods,
We compare our method to hindsight experience replay (HER) [2], goal-conditioned method, where the learner receives reward of -1 if it is within an distance from the goal, and 0 otherwise, universal value density estimation (UVD) [41], which also uses sparse rewards as well as generative model of the future occupancy measure to estimate the Q-values, and DISCERN [51], which learns reward function by training a discriminator and using a clipped log-likel lilkiiiiiood the reward. Lastly, we include oracle Soft Actor-Critic (SAC) baseline that uses a manually designed reward. For the Meta World tasks, this baseline uses the benchmark reward for each task. For the remaining environments, this baseline uses the Euclidean distance between the agent's 's current and the desired outcome for the reward.
Baselines and Prior Work.
In Figure 4, we see that ODAC outperforms virtually every method on all tasks, consistently learning faster and often reaching final distance that is orders of magnitude closer to the desired outcome. The only exception is that the hand-crafter reward learns slightly faster on the 2D task, but this gap is closed within a few ten thousand steps.
Results.
6.2 Ablation Study on the Effecc of a Variational Discount Factor
Next, we study the importance of the dynamic discount factor qaa (A,+1 0) and the sensitivity of our method the dynamics model. On all tasks, we evaluate the performance when the posterior exactly matches the prior, that is. 94.+: (A.1 0) 0.99 (labeled "fixed qr" in Table 1). Our

---

analysis in Appendix A.3 suggests that this setting is sub-optimal, and this ablation empirically evaluate its benefits. We also measure how the algorithm's 's performance depends on the accuracy of the learned dynamics model used for the reward in ODAC. To do this, we evaluate ODAC with the dynamies model fixed to multivariate Laplace distribution with fixed variance, centered at the previous state (labeled "fixed Pd in Table 1). This ablation represents an extremely crude model, and good performance with such a model would indicate that our method does not depend on obtaining an particularly accurate model.
In Table 1, we see that fixing the distribution qr to the prior Pr as described in Corollary 3 deterio- rates performance, and that using a learned or fixed model both perform relatively well. These results suggest that the derived optimal variational distribution 2 (Ac+1 0) given in Proposition 2 is better not only in theory but also in practice, and that ODAC is not sensitive to the accuracy of the dynamics model. Moreover, we note that more expressive dynamics mooeel-whice could lead to tighter variational bound- may not necessarily lead to a better variational policy if the functional form of the log-density under the dynamics model does not also provide favorable shaping.
Table I: Ablation results, showing mean final normalized distance
(x 100) at the end of training across seeds. Best mean is in bold and standard error in parentheses. ODAC is not sensitive to the dynamics models Pa but benefits from the dynamic ør variant.
Env
ODAC
fixed P
fixed @r
fixed @r, Pd
2D Ant
1.7 (1.20)
1.2 (0.14)
1.0 (0.24)
1.3 (0.29)
9 (0.48)
11 (0.57)
12 (0.41)
13 (0.20)
Push
35 (2.7)
34 (1.5)
37 (1.5)
38 (3.1)
Fetch
(6)
15
53 (13)
66 (15)
Window Faucet
5.4 (0.62)
5.0 (0.62)
7.9 (0.71)
6.0 (0.12)
13 (4.2)
15 (3.3)
37 (8.3)
(7.2)
In Appendix B, we provide the full learning curves for this ablation study and present further experiments. For example, we compare ODAC to variant in which we use the learned dynamics model for model-based planning. We find that using the dynamics model only to compute rewards significantly outperforms the variant where it is used for both computing rewards and model-based based planning. This result suggests that ODAC does not require learning a dynamics model that is accurate enough for planning, and that the derived Bellman updates sufficient for obtaining policies that can achieve desired outcomes. In Figure 6, we also visualize qr and find that as the policy reaches an irrecoverable state, 9 (A++1 = 0) drops in value, suggesting that ODAC automatically learns dynamic discount factor that terminates an episode when an irrecoverable state is reached.
7 Conclusion
We proposed a probabilistic approach for achieving desired outcomes in settings where no reward function and no termination condition are given. We showed that by framing the problem of achieving desired outcomes as variational inference, we can derive an off-policy lemporal-differnnee algorithm, a reward function learnable from environment interactions, and novel Bellman backup that contains a state- aattee-ccionn dependent dynamic discount factor for the reward and bootstrap term.
Our experimental results demonstrated that the resulting algorithm, ODAC, leads to efficient outcome- driven approaches to RL. While ODAC requires choosing dynamics model, we found that it works well even for simple dynamics models and believe that the use of more sophisticated dynamics models that incorporate epistemic uncertainty [7] or domain-specific structure [8, 38, 49, 57] is promising avenue for future research.
Acknowledgments and Disclosure of Funding
We thank Marvin Zhang, Michael Janner, Abhishek Gupta, and various RAIL and OATML students for their discussions and feedback on early drafts of this paper. Tim G. Rudner is funded by the Rhodes Trust, by a Qualcomm Innovation Fellowship, and by the Engineering and Physical Sciences Research Council (EPSRC). This research was further supported by the Alan Turing Institute, the National Science Foundation, the DARPA Assured Autonomy Program, and ARL DCIST CRA W911NF-11-2-0181.
10

---

References
[1] G.M Aly and W.C Chan. Numerical computation of optimal control problems with unknown final time. Journal of Mathematical Analysis and Applications, 45(2):274-284, 1974. ISSN 0022-247X.
[2] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob Megrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), 2017..
[3] H. Attias, Planning by probabilistic inference. Proceedings of the 9th International Workshop on Arlificial Intelligence and Statistics, 2003.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: /606. 540, 2016.
[5] Yongxin Chen, Tryphon T. Georgiou, and Michele Pavon. Optimal steering of linear stochastic system to a final probability distribution- part iii. IEEE Transactions on Automatic Control, 63 (9)31122-1188, 2018.
[6] Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Vari- ational empowerment as representation learning for coal-based reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
[7] Kurtland Chua, Roberto Calandra, Rowan Mcallister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, 2018.
[8] Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learming (CoRL), 2017.
[9] Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. arXiv preprint 1906:v::906.05200 2019.
[10] Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, and Shimon Whiteson. VIREL: A variational inference framework for reinforcement learning. In Advances in Neural Information Processing Systems 32, 2019.
[11] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Neural Information Processing Systems (NeurIPS), pages 8538-8547. 2018.
[12] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International Conference on Machine Learning (ICML), 2018.
[13] Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech M. Czarnecki, Yee Whye Tch, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[14] Manfred Goebel and Uldis Raitums. Optimal control of two point boundary value problems. In
H. J. Sebastian and K. Tammer, editors, System Modelling and Optimization, pages 281-290. Berlin, Heidelberg, 1990. Springer Berlin Heidelberg.
[15] Karolos M. Grigoriadis and Robert Skelton. Minimum-energy covariance controllers. Auto- matica, 33(4):569-578, 1997. ISSN 0005-1098.
[16) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, 2018.
[17] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications, 2018.

---

[18] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for unsupervised and semi cemi-supervised skill discovery. In International Conference on Learning Representations (ICLR), 2020.
[19] Matthew Hoffman, Nando Freitas, Arnaud Doucet, and Jan Peters. An expectation maximiza- tion algorithm for continuous markov decision processes with arbitrary reward. In Artificial intelligence and statistics, pages 232-239, 2009.
[20] Anthony F. Hotz and Robert E. Skelton. A covariance control theory. In System Identification and Adaptive Control, Part of 3, volume 26 of Control and Dynamic Systems, pages 225-276. Academic Press, 1987.
[21] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
[22] Leslie P Kaelbling, Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAT), volume vol.2, pages 1094 8, 1993.
[23] J. Kappen, V. Gómez, and M. Opper. Optimal control as graphical model inference problem. Machine Learning, 87(2):159-182, 2012.
[24] Miroslav Kárný. Towards fully probabilistic control design. Automatica, 32(12):1719-1722. 1996.
[25] Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. 2018
[26] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierar- chies with hindsight. arXiv preprint 171.0027,, 2017.
[27] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.
[28] Ofir Nachum, Google Brain, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Neural Information Processing Systems (NeurIPS). 2018.
[29] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In IEEE International Conference on Robotics and Automation (ICRA), 2018.
[30] Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. In Neural Information Processing Systems (NeurIPS), 2018
[31] Jan Peters, Katharina Mülling, and Yasemin Altün Relative entropy policy search. In AAAI Conference on Artificial Intelligence, pages 1607-1612, 2010.
[32] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob Megrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv: 088.0090444 2018.
[33] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference mod- els: Model-free deep RL for mosel-based control. In International Conference on Learning Representations (ICLR), 2018
[34] Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-Fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.00 03698, abs/1903.03698, 2019.
[35] K. Rawlik, M. Toussaint, and S. Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Roboties: Science and Systems (RSS), 2013.
12

---

[36] Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. An approximate inference approach to temporal optimization in optimal control. Jn Lafferty, C. Williams, Shawe-Taylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
[37] Jack Ridderhof, Kazuhide Okamoto, and Panagiotis Tsiotras. Nonlinear uncertainty control with iterative covariance steering, 2019.
[38] Tim G. Rudner, Zonghao Chen, and Yarin Gal. Rethinking function-space variational inference in Bayesian neural networks. In Third Symposium on Advances in Approximate Bayesian Inference. 2021.
[39] Tim G. J. Rudner, Cong Lu, Michael A. Osborne, Yarin Gal, and Yee Whye Teh. On pathologies in KL-regularized reinforcement learning from expert demonstrations, In Advances in Neural Information Processing Systems 34. 2021.
[40] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function ap- proximators. In International Conference on Machine Learning (ICML), pages 1312-1320, 2015.
[41] Yannick Schroecker and Charles Isbell. Universal value density estimation for imitation learning and goal-conditioned reinforcement learning. arXiv preprint arXiv:2002.06-773 2020.
[42] Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, and Sergey Levine. End-to-end robotic reinforcement learning without reward engineering. arXiv preprint arXiv: 1904.:9900.000 07854, 2019.
[43] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. 1998.
[44] Csaba Szepesvári. Algorithms for Reinforcement Learning, volume 4. 2010.
[45] E. Todorov. Linearly-solvable Markov decision problems. In Neural Information Processing Systems (NeurIPS), 2006.
[46] M. Toussaint. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML), 2009.
[47] Mare Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state Markov decision processes. In Proceedings of the 23rd international conference Machine learning, pages 945-952, 2006.
[48] Marc Toussaint, Stefan Harmeling, and Amos Storkey. Probabilistic inference for solving (PS)MDPs. Technical report, 2006.
[49] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Conference on Robot Learning, pages 1439-1456. PMLR, 2020.
[50] Srinivas Venkattaramanujan, Eric Crawford, Thang Doan, and Doina Precup. Self-supervised learning of distance functions for goal-conditioned reinforcement learning. 1rXiv:1907.02990 2019.
[51] David Warde-Farley, Tom Van De Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, Volodymyr, and Mnih Deepmind. Unsupervised control through non -arammratttiic discrimina- tive rewards. In International Conference on Learning Representations (ICLR), 2019.
[52] Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3:4)279-299 1992.
[53] Martha White. Unifying task specification in reinforcement learning. In International Confer- ence on Machine Learning, pages 3742-3750. PMLR, 2017.
[54] J. Xu and R.E. Skelton, An improved covariance assignment theory for discrete systems, 1992.
13

---

[55] Z. Yi, Z. Cao, E. Theodorou, and Y. Chen. Nonlinear covariance control via differential dynamic programming. In 2020 American Control Conference (ACC). pages 3571-3576, 2020.
[56] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094-1100. PMLR, 2020.
[57] Marvin Zhang. Sharad Vikram, Laura Smith, Pieter Abbeel, Matthew J. Johnson, and Sergey Levine. SOLAR: Deep structured representations for model-based based reinforcement learning. In International Conference on Machine Learning (ICML), aug 2019.
[58] Brian D Ziebart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, pages 1433-1438, 2008.
14


Reviews:
[{"review_id": "ZStBxLCaeZa", "reviewer": "Reviewer_DJiQ", "summary": "The paper proposes ODAC, a new RL algorithm for goal-directed tasks. Using a variational inference approach, ODAC automatically infer a dense environment reward from the goal and derives a probabilistic Bellman backup operator. The latter is used to develop an off-policy RL algorithm. The paper presents detailed theoretical derivation and empirical results on a suite of environments.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well presented. The theoretical derivation is detailed and easy to follow. In addition, the derived practical reward appears intuitive, including a  log-likelihood term for reaching the goal in the next step and a KL term for long-term reachability. The empirical results shows improved performance over existing methods.\n\n**Presentation**: There appears to be some inconsistencies in the presentation of the method. For the reward function, Eq. 10 is different from Eq. 18 (the subscript $t$ in the KL term is off by 1), and again different from Alg. 2 in the appendix (the coefficient $q(\\Delta=1)$ for the log-likelihood $\\log p(g|s, a)$ is missing). Could the authors please clarify which reward function is used in practice. If approximation is used, please provide some intuition/motivation for the approximation. For the value function, Eq. 8 and Eq. 13 have a different sign for the KL term. It appears that Eq. 13 has the incorrect sign.\n\n**Method**: Alg 1 and Alg 2 in the appendix are different. It appears that the general idea of hindsight experience replay (HER) is part of Alg 2, but not part of Alg 1. It is important to highlight if HER plays a significant role in the practical algorithm, and study it in the ablation (e.g. ODAC without HER vs ODAC with HER).\n\n**Empirical Evaluation**: The experiment setting from the paper appears to restrict the goal to a single state (i.e. no oracle goal sampling) during training. It is difficult to understand why this setting is chosen, given that oracle goal sampling appears easily implementable, and was used by several previous methods originally. Could the authors provide the comparison results for more environments (in addition to Fetch Push in the appendix), allowing oracle goal sampling? This would allow a more comprehensive comparison with the baseline methods and a better understanding of the ODAC’s advantages over existing methods.\n\nTable 1 appears to also suggest that the learned dynamic model $p_d $  performs noticeably worse than a fixed one, wrt to the policy learning. Specifically, the learned dynamic model obtains worse mean performance and noticeably larger standard deviation in 4 out of 6 tasks. Could the authors provide some intuition as to why this is the case, since this is rather unexpected. I cannot support the current interpretation that the proposed method is insensitive to the dynamic model.\n\nMinor:\nIn A.15, t should be t’ in the summation sign. Line 808 has a missing latex reference.\n\n------------- After responses ---------------------\nMy concerns are mostly addressed. I have revised my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As the proposed method is quite complex, a general limitation is the difficulty in understanding the actual working of the method, and its advantages over the baseline methods. Several related issues are discussed above, such as why the learned dynamic model performs worse than a fixed one, whether HER/goal relabeling is a critical component for ODAC as well, and how much of the observed performance improvements comes from restricting goal states during training. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "RV-hlSQhIbb", "reviewer": "Reviewer_4Y6i", "summary": "This paper derives goal based reinforcement learning as a task of variational inference.\nThey propose a novel bellman operator and learning scheme and provide extensive evaluations across various complex tasks.\nEmpirical evidence backs the theoretical results showing that this method works in practice.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find this line of work to be very interesting and important.\nThe problem formulation and paper are well written and the experiments show the ability of this approach to solve problems in practice.\n\nTwo minor comments:\n\n1) I'd like to see a clean analysis in a tabular setting, while deep learning experiments are impressive they usually require a lot of \"hacks\" and it isn't always clear why the method works better/worse than prior work. This is not to say the method does not achieve impressive results nor that the inability/ability to break SOTA results in deep RL are sufficient/relevant for acceptance/rejection.\n\n2) The numerical table is a bit misleading. I know that the common practice recently is to bold methods with the better mean performance. I'd urge the authors to re-think this broken approach and highlight all results that *might* be optimal (overlapping confidence intervals at the top).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LSR7JFOaF2", "reviewer": "Reviewer_SSfs", "summary": "This paper combines the recent work on viewing RL as Probabilistic Inference and the goal-conditioned RL setting. The proposed approach assigns rewards based on the likelihood of reaching the desired state-action pair and propagates this via learning a dynamic discount function based on time elapsed in the episode. The authors propose solving the objective via variational inference and demonstrate this leads to a probabilistic policy evaluation operator. An actor-critic approach is then taken to perform approximate policy iteration.  The authors show the advantage of their methods on high-dimensional continuous control tasks. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n\nThe combination of control as inference take on the off-policy goal-driven RL is novel to the best of my knowledge and is also motivated sufficiently. I have some questions regarding the approach, in particular:\n\n- The connection between viewing the log-likelihood of reaching particular state-action pair as the reward has been explored also in other works that also take a probabilistic inference view on different RL settings and propose solving it via variational inference such as [1].  Also, the authors mention that the difference between [3] and this work is that [3] requires on-policy samples whereas the proposed approach is off-policy. Can the authors elaborate on why extending [3] to off-policy is particularly tough or challenging? Even though the proposed approach is based on off-policy updates it is still online, so a comparison could be made. What advantage does the proposed approach provides compared to an off-policy version of [3] or the general approach in [1]?\n\n- The mechanism through which the rewards/values are propagated is via learning a state-dependent discounting function. I think the authors should also mention some of the other works that have look at a similar approach for learning the discount factor such as [2,5]. \n\n### Quality: \n\nThe work is technically sound and empirically supported with sufficient ablation studies. However, I have a few questions regarding the choice of baselines. The authors motivate in Sec 5 that the main difference between the prior work on control as probabilistic inference assumes that the manual reward is required ($r_{\\text{manual}}$) whereas the main advantage of the proposed approach is that instead of requiring manual reward function as input it requires manual goals as input. Based on this, I would have expected some of the goal-based version of the existing control as probabilistic inference as some of the baselines. For instance, why was VIREL [4] excluded from the baselines? The VIREL methodology should be extendable to this setting? If not, some discussion of why the other work prior work can’t be extended to this setting should be provided. \n\n### Clarity:\nThe paper is mostly clear. The paper references outcomes throughout the text without formally defining them. It wasn’t clear to me what is the difference between goal-directed and outcome-driven? Does outcome-driven signify the probabilistic nature of the original goal-directed setting? \n \n\n### Significance:\n\nThe paper looks at an important problem setting and the results seem to demonstrate that the proposed approach is able to achieve its goal. I have some reservations about the originality and the need for this approach as mentioned in the points above. I think if the authors can shed some light on the questions raised above, then that will be helpful in raising the usefulness of this work to the community.\n\n### References:\n\n- [1] Weber, Théophane, et al. \"Credit assignment techniques in stochastic computation graphs.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.\n- [2] Pitis, Silviu. \"Rethinking the discount factor in reinforcement learning: A decision theoretic approach.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n- [3] Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control 387 with events: A general framework for data-driven reward definition. In S. Bengio, H. Wallach, 388 H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Neural Information 389 Processing Systems (NeurIPS), pages 8538–8547. 2018.\n- [4] Matthew Fellows, Anuj Mahajan, Tim G. J. Rudner, and Shimon Whiteson. VIREL: A Varia384 tional Inference Framework for Reinforcement Learning. In Advances in Neural Information 385 Processing Systems 32, 2019.\n- [5] Xu, Z., van Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pp. 2396–2407, 2018.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6Fwpyd175_", "reviewer": "Reviewer_3mzu", "summary": "This paper proposes to formulate goal-conditional reinforcement learning as a variational inference problem, termed as outcome-driven RL, and shows that the variational objective naturally generates shaped reward signals for learning. The variational objective results in an off-policy temporal-difference algorithm. Experimental results in robotic control tasks show the advantages of the novel algorithm. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of goal-conditional RL is an active area of research and many recent progresses have been made. This work proposes a novel formulation of the problem and demonstrates that such a formulation leads to an algorithm that outperforms prior methods. The paper is well-written with clarity. The related work section does a good job of overviewing relevant literature in both RL and control. The experiments cover six task domains with different characteristics and demonstrate ODAC performs comparably or outperforms prior methods in most of the tasks. An ablation study is also performed to provide insights into the limitations of ODAC and show that ODAC is not sensitive to the accuracy of the dynamics model.\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned the limitations of the proposed work and potential future work to improve it. The authors did not comment on potential societal impacts.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new variational inference approach for goal-based reinforcement learning. The author's goal is to more directly tackle reinforcement learning problems where the goal is to reach a certain outcome, without resorting to manual reward shaping. Similarly to some previous work, they define an inference objective which is maximize using techniques from variational optimization. Different from such works, in their inference formulation they condition on the goal being reached, rather than using the reward function as pseudo-likelihood. 

The reviewers consider the considered problem (goal-conditioned RL) important and very relevant for the machine learning community. 
The resulting approach was considered novel by the reviewers, especially the extension of a framework similar to VICE [3] to an off-policy setting. 
The reviewers also considered the paper technically sound and sufficiently well supported by experiments and ablations (that show the method outperforms e.g. SAC, another method in this space). 
Also, the reviewers consider the paper well-written. 

Decision: Accept (Poster)