Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Authors: Benjamin Eysenbach
Sergey Levine
Ruslan Salakhutdinov
Venue: NeurIPS 2021 Oral (2021)
Keywords: reinforcement learning
example-based control

Abstract: Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another reinforcement learning algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.

Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Benjamin Eysenbach" 12
Sergey Levine
Ruslan Salakhutdinoy! UCC Berkeley
Carnegie Mellon University, Google Brain,
beysenbaucs cmu. edu
Abstract
Reinforcement learning (RL) algorithms assume that users specify tasks by man- ually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, derive control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer byperparameters tune and lines of code debug. We show that our method satisfies new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions."
Introduction
In supervised learning settings, tasks are defined by data: what causes car detector to detect cars is not the choice of loss function (which might be the same as for airplane detector), but the choice of training data. Defining tasks in terms of data, rather than specialized loss functions, arguably makes easier to apply machine learning algorithms to new domains. In contrast, reinforcement learning (RL) problems are typically posed in terms of reward functions, which are typically manually designed. Arguably, the challenge of designing reward functions has limited RL to applications with simple reward functions, and has been restricted to users who speak this language of mathematically- reward functions. Can we make task specification in RI. similarly data-driven?
Whereas the standard MDP formalism centers around predicting and maximizing the future reward, we will instead focus on the problem classifying whether a task will be solved in the future. The user will provide a collection of example success states, not reward function. We call this problem setting example- caaslle-based control. In effect, these examples tell the agent "What would the world look like if the task were solved?" For example, for the task of opening door, success examples correspond to different observations of the world when the door is open. The user can find examples of success not know how to solve. For example, the user could solve the
even for tasks that they themselves task using actions unavailable to the agent (e.g., the user may have two arms, but robotic agent may have only one) or the user could find success examples by searching the internet. As we will discuss in Sec. 3.1, this problem setting different from imitation learning: maximize a different objective function and only assume access to success examples, not entire expert trajectories.
Project site with videos and code: https://ben--eysebbacch wit.ttttttttbioooeooe
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

r(s,d) = -1.0] Spain Saail2
Bsoa
10 Aunoe Aowalla 0.0i]]saaootvllle
observations
observations
actions
actions
success examples
Markov decision process
Example-based Control
Figure 1:
Example-based control: Whereas the standard MDP framework requires a user-defined reward function, example-based based control specifies tasks handful of user-provided success examples.
Learning from examples is challenging because must automatically identify when the agent has solved the task and reward for doing so. Prior methods (either imitation learning from demon- strations or learning from success examples) take indirect approach that resembles inverse RL: first learn a separate model to represent the reward function, and then optimize this reward function with standard RL algorithms. Our method is different from these prior methods because it learns to predict future success directly from transitions and success examples, without learning a separate reward function. This key difference has important algorithmic, theoretical, and empirical benefits. Algorithmically. our end-to-end approach removes potential biases learning a separate reward function, reduces the number of hyperparameters, and simplifies the resulting implementation. Theo- retically, we propose a method for classifying future events using a variant of temporal difference learning that call recursive classification. This method satisfies a new Bellman equation, where success examples are used in place of the standard reward function term. We use this result to provide convergence guarantees, Empirically, we demonstrate that our method solves many complex manipulation tasks that prior methods fail to solve.
Our paper also addresses subtle but important ambiguity in formulating example-based control. Some states might always solve the task while other states might rarely solve the task. But, without knowing how often the visited each state, we cannot determine the likelihood that each state solves the task. Thus, an agent can only estimate the probability of success by making an additional assumption about how the success examples were generated. We will discuss two choices of assumptions. The first choice of assumption is convenient from an algorithmic perspective, but is sometimes violated in practice. A second choice is worst-case approach, resluting in a problem setting that we call robust example- -aaee control. Our analysis shows that the robust example-based control objective is equivalent to minimizing the squared Hellinger distance (an f-divergence).
In summary, this paper studies a data-driven framing of control, where reward functions are replaced by examples of successful outcomes. Our main contribution is an algorithm for off-polic example- based control. The key idea of the algorithm to directly learn to predict whether the task will be solved in the future via recursive classification, without using separate reward learning and policy search procedures. Our analysis shows that our method satisfies a new Bellman equation where rewards replaced by data (examples of success). Empirically, our method significantly outperforms state-of-the-ar imitation learning methods (AIRL [7), DAC [17], and SQIL (28)) and recent methods that learn reward functions (ORIL [41], PURL [38]. and VICE [8]). Our method completes complex tasks, such as picking up a hammer to knock a nail into a board, tasks that none of these baselines can solve. Using tasks with image observations, demonstrate agents learned with our method acquire notion of success that generalizes to new environments with varying shapes and goal locations.
2 Related Work
Learning reward functions. Prior works have studied RL in settings where the task is specified either with examples of successful outcomes or complete demonstrations. These prior methods typically learn reward function from data and then apply RL to this reward function (e.g.. Fu et al. [8], Ziebart et al. [40]). Most inverse RL algorithms adopt this approach [1, 25, 27, 29, 36, 40], as do more recent methods that learn a success classifier to distinguishing successful outcomes from random states [8, 16, 31, 41]. Prior adversarial imitation learning methods [7, 12] can be

---

viewed as iteratively learning a success classifier. Recent work in this area focuses on extending
these methods the offline setting [16, 41], incorporating additional sources of supervision [42], and learning the classifier via positive-unlabeled classification [13, 38, 41]. Many prior methods for robot learning have likewise used classifier to distinguish success examples [2, 22, 34, 37]. Unlike these prior methods, our approach only requires examples of successful outcomes (not expert trajectories) and does not learn a separate reward function. Instead, our method learns a value function directly from examples, effectively "cutting out the middleman." This difference from prior work removes hyperparameters and potential bugs associated with learning a success classifier. Empirically, we demonstrate that our end-to-ens to-dto-dd approach outperforms these prior two-stage approaches. See Appendix D for more discussion of the relationship between our method and prior work.
Imitation learning without auxiliary classifiers. While stample-based control is different from imitation learning, our method similar two prior imitation learning methods that likewise avoid learning a separate reward function [18, 28]. ValueDICE [18], a method based on convex duality, uses full expert demonstrations for imitation learning. In contrast, our method learns from success examples, which are typically easier to provide than full expert demonstrations. SQIL [28] is modification of SAC [11] that labels success examples with reward of +1. The mechanics of our method are similar to SQIL [28], but key algorithmic differences (backed by stronger theoretical guarantees) result better empirical performance. Our analysis Sec. 4.2 highlights connections and differences between imitation learning and example-based control.
Gonl-conditioned RL. Goal-conditioned RL provides one way specify tasks in terms of data, and prior work has shown how goal-conditioned policies can be learned directly from data, without a reward function [4, 15, 21, 30]. However, the problem that we study in this paper, example-based control, is different from goal-conditioned RL because it allows users to indicate that tasks can be solved in many ways, enabling the agent to learn more general notion of success. Perhaps the most similar prior work in this area is C-learning [4], which uses lemporal-difference difference update to learn the probability that goal state will be reached in the future. Our method will use a similar temporal-differesn update to predict whether any success example will be reached in the future. Despite the high-level similarity with C-learning, our algorithm will be markedly different; for example, our method does not require hindsight relabeling, and learns a single policy rather than a goal-conditioned policy.
3 Exxaple-Basse Control via Recursive Classification
We aim to learn policy that reaches states that are likely to solve the task (see Fig. 1), without relying on a reward function. We start by formally describing this problem, which we will call example-based control. We then propose method for solving this problem and provide convergence guarantees.
3.1 Problem Statement
Example- control is defined by a controlled Markov process (i.e., an MDP without reward function) with dynamics p($t+1 St, ax) and an initial state distribution pr (s1). where St E S and a denote the time-indexed states and actions. The variable St+ A denotes a state A steps in the future.
The agent given a set of success examples, S* = {s*} S. The random variable e E {0, 13 indicates whether the task is solved at time i, and p(et (t))) denotes the probability that the current state s, solves the task. Given a policy Ta(au st), we define the discounted future state distribution:
pd (st Se, a,) $ (1 71 Σ (S
= St+ Se. at).
A=0
Using this definition, we can write the probability of solving the task at future step as
pd (ce+ St, ax) - Epp (ore4,s..00 ( (st)]]
(2)
Example-basee based control maximizes the probability of solving the task in the (discounted) future:
Definition 1 (Example-based control). Given controlled Markov process and distribution over success examples p(st et = 1), the example-base based control problem is to find the policy that optimizes the likelihood of solving the task:
arg maxp" (er++ = 1) = Ep (p1())) (x) (et+ = |1,1,1]) (1,a1]]-
x
3

---

Although this objective is equivalent to the RL objective with rewards r(se: a = p(e = st), we assume the probabilities p(e, st)) are unknown. Instead, we assume that we have samples of successful states, s" (st because imitation learning requires full expert demonstrations. In the special case where the user provides a single success state, example-hase hased control is equivalent to goal-conditioned RL.
1). Example-based control differs from imitation learning
Since interacting with the environment to collect experience is expensive in many settings, we define off-policy example-based control as the version of this problem where the agent learns from environment interactions collected from other policies. In this setting, the agent learns from two distinct datasets: (1) transitions, {Se, ag, St+1) ~ (st, ag $t-1)}, which contain information about the environment dynamics; and (2) success examples, S {s* (s, e = 1)}, which specify the task that the agent should attempt to solve. Our analysis will assume that these two datasets are fixed. The main contribution of this paper is an algorithm for off- off-policy example- control.
An assumption on success examples. The probability of solving the task at state St. p(et = St), cannot be uniquely determined from success examples and transitions alone. To explain this ambiguity, we define Pu (st) as the state distribution visited by the user, note that the user may be quite bad at solving the task themselves. Then, the probability of solving the task at state St depends on how often a user visits state versus how often the task is solved when visiting state sp:
p(et= st) (st e 1) (ut = 1)).
(3)
(St(s:)
For example, the user may complete a task using two strategies, but we cannot determine which of these strategies is more likely to succeed unless know how often the user attempted each strategy. Thus, any method that learns from success examples must make an additional assumption on po (st). We will discuss two choices of assumptions. The first choice is to assume that the user visited states with the same frequency that they occur in the dataset of transitions. That is,
pU = /f (St, at, St+
(4)
Intuitively, this assumption implies that the user has the same capabilities as the agent. Prior work makes this same assumption without stating it explicitly [8. 24, 31]. Experimentally, we find that our method succeeds even in cases where this assumption is violated.
However, many common settings violate this assumption, especially when the user has different dynamics constraints than the agent. For example, a human user collecting success examples for a cleaning task might usually put away objects on a shelf at eye-level, whereas transitions collected by robot interact with the ground-level shelves more frequently. Under our previous assumption, the robot would assume that putting objects away on higher shelves is more satisfactory than putting them away on lower shelves, even though doing so might be much more challenging for the robot. To handle these difference in capabilities, the second choice is to use a worst-case formulation, which optimizes the policy to be robust to any choice of (s.).. Surprisingly, this setting admits a tractable solution, as we discuss in Sec. 4.2.
3.2 Predicting Future Success by Recursive Classification
We now describe our method for example- based control. We start with the more standard first choice for the assumption on pu (st) (Eq. 4); we discuss the second choice in Sec. 4.2. Our approach estimates the probability in Eq. 2 indirectly via future success classifier. This classifier, C3 (St. ar), discriminates between "positive" state-action pairs which lead to successful outcomes (i.e., sampled from p(St, at et+ = 1)) and random "negatives" (i.e., sampled from the marginal distribution p(s,, We will use different class-specifie weights, using a weight of p(ec+ 1) for the "positives" and a weight of for the "negatives." " Bayes-optimal classifier is
(St, at ee+ = 1 1)p(ec+ = 1)
(st, at) p= (= a | ett 1)p(ec+ 1) + p(se. at)"
(5)
These class specific weights let us prediet the probability of future success using the optimal classifier:
C% (s,,
= p" (et+ = 1 St. at).
(6)
1- CJ (St, an))
4

---

Algorithm 1 Recursive Classification of Examples
success examples S* Initialize policy a 8t), classifier CJ (Se, at), replay huffer D while not converged do
Input:
Collect new trajectory: DD +- D fr not
Sample success examples: {sL (1) S" at. ~ To(ar | se
Sample transitions: {(Se (2) ar (2) 8t41) D, a To(a sc41)]
Eq.
1- (
(()) < () (2) at 3y 1) (1 (WE) ac ()) )
Update classifier: l + 0 (oL(
Eq. 8
Update policy: +- e V V &.En (an Ise (Co(st:
return
Importantly, the resulting method will not actually require estimating the weight p(ec+ = 1). We would like to optimize the classifier parameters using maximum likelihood estimation:
L (0) _ p(e++ = 1)
llog C/ (st, at)] + (orttr.a,)logg
C. (st. at))]. (7)
However, we cannot directly optimize this objective because we cannot sample from p(s,, ag et+ 1). We convert Eq. 7 into an equivalent loss function that we can optimize us- ing three steps; see Appendix A for a detailed derivation. The first step is to factor the distribution p(St, a et+ 1). The second step to decompose (et+ sponding to the probabilities of solving the task at time t' = t + 1 and time t' > + We can estimate the probability of solving the task at the next time step using the set of success examples. The third step classifier at the next time step. Combining these three steps, we can equivalently express the objective function in Eq. 7 using off-policy data:
1 St. at) into two terms, corre-
estimate the probability of solving the task at time
++ by evaluating the
(0) =(1 - 7) (ts 1) [log CT (St, ac)] + Epire (ru log (se, at) + log(1 Cā
(St. a
p(a)(e))
(e)
(b)
(c)
(8)
where
C7 (St+1, a++1)
Ep(ae (1
(9)
1- C- (Se+1, act1).
the classifier's prediction (ratio) at the next time step. Our resulting method can be viewed as temporal difference [33] approach to classifying future events. We will refer to our method as recursive classification of examples (RCE). This equation has an intuitive interpretation. The first term (a) trains the classifier to predict for the success examples themselves, and the third term (c) trains the classifier to predict for random transitions. The important term the second term (b), which is analogous to the "bootstrapping" term in temporal difference learning [32]. Term (b) indicates that the probability of future success depends on the probability of success at the next time step, as inferred using the classifier's ' own predictions.
Our resulting method is similar to existing actor-critic RI. algorithmss. existing actor-critic methods, we can combine the (b) and (c) terms in the classifier objective function (Eq. 8) to express the loss function in terms two cross entropy losses:
To highlight the similarity to
min (1 - 7) T. pley 11) (CE (St, at); 1)] + (1 + yw)) (ep (CE(CT (Se. a))y 0
me (at (Rt)
YPu +
(10)
These cross entropy losses update the classifier to predict y 1 for the success examples and to predict y
7w0 for other states.
1470
Algorithm summary. Alg. summarizes method, which alternates between updating the classifier, updating the policy, and (optionally) collecting new experience. We update the policy to choose actions that maximize the classifier's confidence that the task will be solved in the future: max ( le.) IC (St, ).)). Following prior work [5, 35]), we regularized the policy updates by adding an entropy term with coefficient a = 10-4. We also found that using N-step returns significantly improved the results of RCE (see Appendix F for details and ablation experiments.). Implementing our method on top of existing methods such as SAC [11] or TD3 [9] requires only changing the standard Bellman loss with the loss Eq. 10. See Appendix E for implementation details; code is available on the project website.
5

---

4 Analysis
In this section, we prove that RCE satisfies many of the same convergence and optimality guarantees (for example-based control) that standard RI algorithms satisfy (for reward-based MDPs). These results are important as they demonstrate that formulating control in terms of data, rather than rewards, does preclude algorithms from enjoying strong theoretical guarantees. Proofs of all results given in Appendix B, and we include further discussion of how RCE relates to prior work in Appendix D.
4.1 Bellman Equations and Convergence Guarantees
To prove that RCE converges to the optimal policy, will first show that RCE satisfies new Bellman equation:
Lemma 4.1. The Bayes- classifier C* for policy x satisfies the following identity:
C (st, -(1 >))((; St) + p(s+ 1 [sc
C (1, C C5 (8 at+1)
(II)
1 C* (st, at)
(P (8111)
The proof combines the definition of the Bayes-optimal classifier with the assumption from Eq. 4. This Bellman equation is analogous to the standard Bellman equation for Q-learning, where the reward function is replaced by (1 7)p(e
st) and the Q function is parametrized as Q7 at) = (s),aa)) While we do not know how to compute this reward function, the update
rule for RCE is equivalent to doing value iteration using that reward function and that parametrization of the Q-function:
Lemma 4.2. In the labular setting, the expected updates for RCE are equivalent doing value iteration with the reward function r(8e) = -7)ppe = 1 8t) and a Q-function parametrized as Q7 ax) C7 (e)
(St
This result tells us that RCE is equivalent to maximizing the reward function (1-7)p(et = st); however, RCE does not require knowing p(e, = 1 St), the probability that each state solves the task. Since value iteration converges in the tabular setting, an immediate consequence of Lemma 4.2 that tabular RCE also converges:
Corollary 4.2.1. RCE converges in the tabular setting.
So far we have analyzed the training process for the classifier for a fixed policy. We conclude this section by showing that optimizing the policy w.r.t the classifier improves the policy's performance.
Lemma 4.3. Let policy Ta Se) and success examples S be given, and let C (st. at) denote the corresponding Bayes-optimal classifier. Define the improved policy as acting greedily w.r.t. C n' r'(a, Se) a(a = arg max, C (²((s,))) a)). Then the improved policy is at least as good as the old policy at solving the task: p" (er+ = 2 nr (e++ 1).
4.2 Robust Example-based Control
In this section, we derive principled solution for the case where po (St) ) is not known, which will correspond to modifying the objective function for example-based control. However, we will argue that, in some conditions, the method proposed in Sec. 3.2 is already robust unknown po Pu (s,), that method is used with online data collection. The goal of this discussion is to provide theoretical relationship between our method and a robust version of example-base based control that makes fewer assumptions about pe (St). This discussion will also clarify how changing assumptions on the user's s capabilities can change the optimal policy.
When introducing example-based control in Sec. 3.1, we emphasized that we must make an assump- tion make the example-based control problem well defined. The exact probability that a success example solves the task depends on how often the user visited that state, which the agent does not know. Therefore, there are many valid hypotheses for how likely each state solve the task. We can express the set of valid hypotheses using Bayes' Rule:
st) = Pu (st et 1)p(et = =1)
Payls $ |lle
||s
(st(S)

---

Previously (Sec. 3.2), we resolved this ambiguity by assuming that (st) was equal to the distribution over states in our dataset of transitions. As discussed in Sec. 3.1, many problem settings violate this assumption, prompting us to consider the more stringent setting with no prior information about pr((st) (e.g., no prior knowledge about the user's capabilities). To address this settting, we will assume the wors! possible choice of pr (St). ). This approach will make the agent robust to imperfect knowledge of the user's abilities and to mislabeled success examples. Formally, we define the robust example-based control problem
pu (st et = ple: = 1) (s.)
max
min P(or)ae)EPeel ( Ip(ee+++ 11| [|++)]]] = max min Bpriol)) (pc)
(12)
This objective can be understood as having the adversary assign a weight of 1 1/pu (8t) ) to each success example. The optimal adversary will assign lower weights success examples that the policy frequently visits and higher weights to less-visited success examples. Intuitively, the optimal policy should try to reach many of the success examples, not just the ones that are easiest to reach. Thus, such a policy will continue to succeed even if certain success examples are removed, or are later discovered to have been mislabeled. Surprisingly, solving this two-player game corresponds to minimizing an f-divergence:
/ dx as the squared Hellinger distance, an f-divergence. Robust example-based control (Eq. 12) equivalent to minimizing the squared Hellinger distance between policy's 's discounted state occupancy measure and the conditional distri- bution p(s: 1e==1). 1):
Lemma 4.4. Define H²(p(x), ([p(x)){(xi]==== =[x))]][[Pp
min p p(ex )
- (-.( [ppst = 1).,"(sst = =[]]]-
The main idea of the proof (found in Appendix C) is compute the worst-case distribution (st))) using the calculus of variations. Preliminary experiments (Fig. 5 in Appendix C) show that a version of RCE with online data collection finds policies that perform well on the robust example-based control objective (Eq, 12). In fact, under somewhat stronger assumptions, we can show that the solution of robust xxamplebbased control is fixed point of iterated RCE (see Appendix C.2). Therefore, in our experiments, we use RCE with online data collection.
5 Experiments
Our experiments study how effectively RCE solves example-based control tasks, especially in comparison to prior methods that learn an explicit reward function. Both RCE and the prior methods receive only the success examples as supervision; reward functions. Additional experiments in Sec. 5.2 study whether RCE can solve tasks using image observations. These experiments test whether RCE solve tasks in new environments that are different from those where the success examples were collected, and test whether RCE learns policies that learn a general notion of success rather than just memorizing the success examples. We include videos of learned policies online? and include implementation details, hyperparameters, ablation experiments, and a list of failed experiments in the Appendix.
method has access to expert trajectories of
We compare RCE against prior methods that infer a reward function from the success examples and then apply an off-the-shelff RL algorithm; some baselines iterate between these two steps. AIRL [7] a popular adversarial imitation learning method. VICE [8] is the same algorithm as AIRL, but intended to be applied success examples rather than full demonstrations. We will label this method as "VICE" in figures, noting that it is the same algorithm as AIRL. DAC [17] is a more recent, off-policy variant of AIRL. We also compared against two recent methods that learn rewards from demonstrations: ORII. [41] and PURL (38). Following prior work [16], we also compare against "frozen" variants of some baselines that first train the parameters of the reward function and then apply RI. to that reward function without updating the parameters of the reward function again. Our method differs from these baselines in that we do not learn a reward function from the success examples and then apply RL, but rather learn a policy directly from the success examples. Lastly, we compare against SQIL [28], an imitation learning method that assigns a reward of +1 to states from demonstrations and 0 to all other states. SQIL does not learn a separate reward function and structurally resembles our method, but is derived from different principles (see Sec. 2.).
hhttps://ben: http://bbb..yeeeab....ec.a github. tattthu..o/rreee..
7

---

sawyer drwwwr.wwoooo peeen
sawyer_ push
sawyer lift
doar- hurma-vv
2.06
Return
2.00
1:00
2.00
00
25
10
15
20
25
50
25
10
2.3
0.0
25
10
1.
2.
25
30
10
85
15
15
20
25
sawyer box close
sawyer bin picking
Saamee-humanee
Training Steps
RCE
VIC
SQL
ORL-AL
05-
UAL
1.00
DAC--roaes
PURL
VICE
PURL frozen
0.0 2.5
1.5
2-4
2.5
3.0
00
05
15
1.0
0.0
10
24
255
Training stops
Training staps
Training staps
Figure 2: Recursive Classification of Examples for learning manipulation tasks: We apply RCF to a range of manipulation tasks, each accompanied with a dataset of success examples. For example, the sawyer_. __ _,fft task, we provide success examples where the object has heen lifted above the table. We use the cumulative task return (1 is better) solely for evaluation. Our method (blue line) outperforms prior methods across all tasks.
5.1 Evaluating RCE for Example-Based Control.
We evaluate each method on five Sawyer ma- nipulation tasks from Meta-World [39] and two manipulation tasks from Rajeswaran et al. [26]. Fig. 3 illustrates these tasks. On each task, we provide the agent with 200 successful out- comes define the task. For example,
the open_ deeewrd task, these success examples show an opened drawer. As another example, on the sawyer_ -areer_pushhhe task, success examples not only have the end effector touching the puck. but (more importantly) the puck position is different. We emphasize that these success examples only reflect the final state where the task is solved and are not full expert trajectories. This setting is important in practical use-cases: it is often easier for humans to arrange the workspace into a successful configuration than it is to collect an entire demonstration. See Appendix E.3 for details on how success examples were generated for each task. While these tasks come with existing user-defined reward functions, these rewards are not provided to any of the methods in our experiments and are used solely for evaluation (f is better). We emphasize that this problem setting is exceedingly challenging: the agent provided only with examples of success states (e.g., an observation where an object has been placed in the correct location). Most prior methods that tackle similar tasks employ hand-designed reward functions or distance functions, full demonstrations,
Figure 3: Manipulation Environments
surcfullly-cossruuee initial state distributions.
The results in Fig. 2 show that RCE significantly outperforms prior methods across all tasks. The transparent lines indicate one random seed, and the darker lines are the average across random seeds. RCE solves many tasks, such as bin picking and hammering, that none of the baselines make any progress towards solving. The most competitive baseline, SQIL, only makes progress on the easiest two tasks; even on those tasks, SQIL learns more slowly than RCE and achieves lower asymptotic return. To check that all baselines are implemented correctly, we confirm that all can solve a very simple reaching task described in the next section.
5.2 Example-Based Control from Images
Our second set of experiments studies whether RCE can learn image-based tasks and assesses the generalization capabilities of our method. We designed three inage-based manipulation tasks. The reach_rrndae
on task entails reaching a red puck, whose position is randomized in each episode. The reaac_eraaneeeeeee psiieesssseeeee task entails reaching red object, but the actual shape of that object varies from one episode to the next. Since the agent cannot change the size of the object and the size is randomized from one episode to the next, it is impossible to reach any of the previously- observed success examples. To solve this task, the agent must leam a notion of success that is more general than reaching a fixed goal state. The third task, sawyer_ weryerrclear__r clear diderrimaee,,___ll entails clearing an object off the table, and is mechanically more challenging than the reaching tasks.
Fig. 4 shows results from these image-based experiments, comparing RCE to the same baselines. We observe that RCE has learned to solve both reaching tasks, reaching for the object regardless of the location and size of the object. This task is mechanically easier than the state- based tasks in Fig, 2,
8

---

sereerrree__..e randon..ppaa petaddnn_oeeiiiooo poeition, inage
eavyer rww.crreccc__aee randon, sime_ image
aakyer clear inage
6.00
SC4
VCC
RRL
6:00
0.0
8.5
Figure 4: Example-hased control from images: We evaluate RCE on three manipulation tasks using image- observations. (Top) We show examples the mmitaal state and success examples for each task. (Bottom) RCE (blue line) outperforms prior methods, especially on the more challenging clearing task. For the random_size task (center), this entails reaching for new objects that have different sizes from any seen in the success examples.
and all the baselines make some progress on this task, but learn more slowly than our method. The good performance of RCE on the a new environment, where the object size is different from that seen in the success examples. We hypothesize that RCE learns faster than these baselines because it "cuts the middleman," " learning value function directly from examples rather than indirectly via separate reward function. To support this hypothesis, we note SQIL, which also avoids learning a reward function, learns faster than other baselines on these tasks. On the more challenging clearing task, only our method makes progress, suggesting that RCE more effective algorithm for learning these image- control tasks. In summary, these results show that RCE outperforms prior methods at solving example-based control tasks from image observations, and highlights that RCE learns policy that solves tasks in new environments that look different from any of the success examples.
tit__random_ssii_ size task illustrates that RCE can solve tasks in
5.3 Ablation Experiments
We ran seven additional experiments to study the importance hyperparameters and design decisions. Appendix F provides full details and figures. These experiments highlight that RCE is not imitation learning method: RCE fails when applied to full expert trajectories, which are typically harder to provide than success examples. Other ablation experiments underscore the importance of using n-step returns and validate the approximation made in Sec. 3.2.
Conclusion
In this paper, we proposed a data-driven approach to control, where examples of success states are used in place of reward function. Our method estimates the probability of reaching success example the future and optimizes a policy to maximize this probability of success. Unlike prior imitation learning methods, our approach is end-to-end and does require learning reward function. Our method is therefore simpler, with fewer hyperparameters and fewer lines of code to debug. Our analysis rests on new data-driven Bellman equation, where example success states replace the typical reward function term. We use this Bellman equation to prove convergence of our classifier and policy. We believe that formulating control problems in terms of data, rather than the reward-centric MDP, better captures the essence of many real-world control problems and suggests a new set of attractive learning algorithms.
Limitations and future work. One limitation with RCE is that, despite producing effective policies, the classifier's predictions are not well calibrated. This issue resembles the miscalibration in Q- functions observed in prior work [9, 20]. Second, both RCE and the baselines we compared against all struggled learn more challenging amage-based tasks, such image-based versions of the tasks shown in Fig. 3. Techniques such as explicit representation learning [19, 23] may be important for scaling example-based control algorithms to high-dimensional tasks.
Acknowledgements. This work is supported by the Fannie and John Hertz Foundation. NSF (DGE1745016.. IISI I1S1763562), ONR (N000141812861) and US Army. We thank Ilya Kostrikov for providing the implementation of DAC. We thank Ksenia Konyushkova, Konrad Zoina, and Justin Fu for discussions about basclines and prior work. We thank Oscar Ramirez for help setting up the image-based experiments, Ryan Julian and anonymous reviewers for feedback on drafts of the paper, and Alex Irpan for help releasing code.

---

References
[1] Abbecl, P. and Ng. A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page
[2] Calandra, R., Owens. A.. Upadhyaya, M., Yuan. W., Lin, J., Adelson, E. H., and Levine. S. (2017). The feeling of success: Does touch sensing help predict grasp outcomes? In Conference on Rohot Learning. pages 314-323. PMIR.
[3] Elkan, C. and Noto, K. (2008). Learning classitiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference Knowledge discovery and data mining, pages 213-220.
[4] Eysenbach, B., Salakhutdinov, R., and Levine, S. (2021). C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representations.
[5] Fox, R., Pakman, A., and Tishby, N. (2016). Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Secone Conference on Uncertainty in Artificial Intelligence, pages 202-211.
[6] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07211
[7] Fu, J., Luo, K., and Levine, S. (2018a). Learning robust rewards with adverserial inverse reinforcement learning. International Conference on Learning Representations.
[8] Fu, J., Singh, A.. Ghosh, D., Yang. L., and Levine. S. (2018b). Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31:8538- 8547.
[9] Fujimoto, S.. Hoof. H., and Meger, D. (2018). Addressing function approximation enror in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR.
[10] Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N.. Kokiopoulou. E., Sbaiz, L... Smith, J., Bartók, G.. Berent, J., Harris, C., Vanhoucke, V.. and Brevdo, E. (2018). Ag-Att:::: A library for reinforcement learning in tensorflow. [Online; accessed 25 25-June-2019],
[11] Haarnoja, T., Zhou, A., Abheel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR.
[12] Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. In NIPS, pages 3565-4573.
[13] Irpan, A., Rao, K., Bousmalis, K., Harris, C., Ibarz, and Levine, S. (2019). Off-policy evaluation via pof-po classification. In Advances in Neural Information Processing Systems, pages 5437-5448.
[14] Jaakkola, T., Jordan, M. L., and Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural computation, 6(6): 165::185-1201..
[15] Kaelbling, L. P. (1993). Learning achieve goals. In IJCAI, pages 1094-1099, Citeseer.
[16] Konyushkova, K., Zolna, K., Aytar, Y., Novikov, A., Reed, S., Cabi, S., and de Freitas, N. (2020). Semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899
[17] Kostrikov, Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, (2018). Discriminator actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation leariing In International Conference on Learning Representations.
[18] Kostrikov, Nachum. O., and Tompson. (2019). Imitation learning off-policy distribution matching. In International Conference on Learning Representations.
[19] Laskin, M., Srinivas, A., and Abbeel, P. (2020). Curl: Contrastive unsupervised representations for
reinforcement learning. International Conference on Machine Learning, pages 5639-5650. PMLR.
[20] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D, (2016). Continuous control with deep reinforcement learning. In ICLR (Poster).
[21] Lin, X., Baweja, H. S., and Held, D. (2019). Reinforcement learning without ground-truth state. arXiv preprint arXiv: 190.00786..
(22] Lu, Q.. Van der Merwe, M., Sundaralingam, B., and Hermans, T. (2020). Multifingered grasp planning via inference in deep neural networks: Outperforming sampling by learning differentiable models. IEEE Robotics Automation Magazine, 27(2):55-66 65.
10

---

[23] Nachum, O., Gu, S., Lee. H., and Levine. S. (2018). Near-optimal representation learning for hierarchical reinforcement learning. In International Conference Learning Representations.
[24] Nasiriany, S. (2020). DisCo RL: Distribution-Cnnlltoone Reinforcement Learning for General- Purpose Policies. PhD thesis.
[25] Pomerleau, D. A. (1989). Alvinm: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305-313.
[26] Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2018). Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS).
[27] Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. A. (2006). Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pages 729-736.
[28] Reddy, S., Dragan, A. D., and Levine, S. (2020). (SQIL): Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations.
[29] Ross, S., Gordon, G., and Bagnell, D. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635.
[30] Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015). Universal value function approximators. In International conference machine learning, pages 1312-1320.
[31] Singh, A., Yang. I.,, Finn, C., and Levine, (2019). End-to-end robotic reinforcement learning without reward engincering. In Robotics: Science and Systems.
[32] Sutton, S. (1988). Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44.
[33] Sutton, (1995). Td models: Modeling world a mixture time scales. Machine Learning Proceedings 1995, pages 531-539. Elsevier.
[34] Vecerik, M., Sushkov, O., Barker, Rothörl, T., Hester, T., and Scholz, J. (2019). A practical approach insertion with variable socket position using deep reinforeement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 754-760. IEEE.
[35] Williams, algorithms. Connection Science, 3(3):241-2))
Peng. (1991). Function optimization using connectionist reinforvement learning
[36] Wulfmeier, M., Ondruska, and Posner, I. (2015), Maximum entropy deep inverse reinforeement lcarning. arXiv preprint arXiv:1507.04885 04888.
[37] Xie, A., Singh, A., Levine, S., and Finn, (2018). Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pages 40-52. PMLR.
[38] Xu, D. and Denil, M. (2019). Positive-unlabeled reward learning. arXiv preprint arXiv:1911.00459
[39] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020). Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Rohot Learning, pages 1094-1100 1100. PMLR.
[40] Ziebart, B. D., Maas, A. L.. Bagnell, J. A., and Dey, A. K. (2008). Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433- 1438. Chicago. IL, USA.
[41] Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, 7., Aytar, Y., Denil, M., de Freitas, N., and Reed, S. (2020). Offline learning from demonstrations and unlabeled experience. arXiv preprint 1rXiv:2011.1380 1.100.11188..
[42] Zolna, K. Reed, S., Novikov, A., Colmenarej. S. G., Budden, D., Cabi. S., Denil, M. de Freitas, N., and Wang. Z. (2019). Tesk-relevant adversarial imitation learning, arXiv preprint 1.1001797.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do foresee plausible negative impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] Code is available at:
httpa://github. cwmcccm///eee/eceeeeeeeeoeeeeeee hete/ree-aanoon anonymous/rce-annntooo anonymooss github heo/trree/min/cooe
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A and provided code.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Each experiment took approximately one day on a standard CPU server. The exact compute resources are proprietary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "cOtSxtI8cl", "reviewer": "Reviewer_3uQ8", "summary": "This paper asks an important and timely question: Can we make task-specification in RL “data-driven”? Rather than prescribing reward functions to incentive behavior in robotic agents,  RCE works by providing the agent examples of the solved task. The agent learns to classify whether or not the task will be solved in the future on a state-by-state basis in what they call \"example-based\" control. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-motivated and is backed by sufficient theory. The approach can be dropped into existing algorithms, making it immediately useful for a large variety of practitioners. \n\nThe algorithm is (perhaps boldly) positioned as a generalization of Goal-conditioned RL, but doesn't back this statement up strongly enough in the experimental section. In line 157, the authors claim that the algorithm performs well even when the examples come from an agent with different capabilities from the agent. This is quite relevant to the community and it seems worth linking to these experiments in this paragraph.  \n\nI appreciate the related-work section, though I think It may be worth discussing work outside of traditional RL/imitation learning. How does this paper compare to research which re-distributes sparse reward, such as \n\"RUDDER: Return Decomposition for Delayed Rewards\" or planning paradigms that attempt to map values onto states from a terminal reward?\n\nIt is not clear to me how some of the tasks are not underspecified. For example, in \"Sawyer-Push\", the successful example is defined by the end-effector touching the puck. This seems like it would incentivize the agent to simply touch the puck, rather than push the puck towards the goal. Can the authors elucidate this example?\n\nThe experiments on \"Example-Based Control from Images\" experiments involve relatively simple tasks, making the results somewhat underwhelming. Overall, I would prefer a stronger experimental section in the body of the paper rather than the appendix. How were the experimental tasks chosen? If this narrow subset of tasks were chosen specifically for RCE's performance, can you describe which tasks the algorithm failed on and speculate as to why? \n\n\nSuggestions: \n\n- Results in this paper are evaluated via reward, however, simple task success/failure is a better-calibrated measurement for this paper. \n\n- Figure 2:  This figure takes up valuable paper space and would be better positioned in the appendix. \n\n- Figure 4: This plot is too crowded to be useful. The information may be better represented in a table or a combination of table/plot. Please indicate what these runs entail (avg of 5 seeds /  most performant agent / etc). \n\nMinor Comment: \n\n- Line 163: \"The handle\" should be \"To handle\"\n\nOverall, this is an important and immediately useful paper that demonstrates a new way of thinking about learning agents. A stronger and more coherent experimental section will further improve a good paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "YvzYOGGq4AL", "reviewer": "Reviewer_yauQ", "summary": "**Summary:** This paper offered a new perspective for continuous control tasks, e.g robotic, that root in one question, can agent learn the task by successful examples rather than rewards. By learning a classifier that predicts whether the task could be solved in the future to estimates the probability of solving the task successfully, this method could solve some control tasks that the prior method failed to solve.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I am really enjoying this inspiring paper. The proposed method offered a new perspective for continuous control tasks where agents could learn to solve tasks by successful examples rather than rewards. However, few concerns flooded my mind. \n\n**Originality**:\n* I think most of the theoretical result has been done in this work[1]. Line 106-114 describes your work and C-Learning[1] 's relation, but I don't think that applied to the theoretical part.\n\n**Clarity**:\n* Line 135, $p_{U}(s_{t} \\mid {e}_{t}=1)$ occur before definition. Didn't align with Line 142.\n\n* The loss function in Algorithm 1 doesn't make sense to me. $s_{t+1}$ should be $s_{t}$?\n\n* You acclaim that goal-conditional RL is the special case of the example-based control. Why is such baseline (e.g., C-learning) absent in the experiment setting?\n\n**Significance**:\n* While the author would like to deliver the point that agents could learn from successful examples rather than reward in offline settings.  I think the choice of baseline didn't make much sense to me: only one imitation learning(SQIL) baseline is included. \n\n** Post rebuttal **\n\nAfter reading the response and other reviewers' feedback, I'd like to improve my score. thank you for your nice work.\n\n\n\n[1] Eysenbach, Benjamin, Ruslan Salakhutdinov, and Sergey Levine. \"C-learning: Learning to achieve goals via recursive classification.\" arXiv preprint arXiv:2011.08909 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "I don't have ethical concerns against this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Im0ya3HMZy", "reviewer": "Reviewer_FCFz", "summary": "In this paper, the authors explore reinforcement learning solutions that train via examples for success states rather than an explicit reward function. The papers proposes an approach that is distinct from standard imitation learning - in that only success states are required rather than entire trajectories. The primary contribution of the paper is an off-policy RL algorithm that relies on predicting the success of future states. The paper addresses limitations of expert-given success states - and explores two formulations of estimating the relative effectiveness of two different states that solve the same task. The authors provide an analysis of convergence guarantees. Finally, experiments demonstrate that this approach beats SOTA methods on several enviroments.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well motivated - the problem of reward construction is a fundamental impediment in learning effective policies in RL settings. The approach is novel - the authors do a good job of differentiating this from similarly motivated solutions like inverse reinforcement learning and imitation learning. The recursive classification approach is a clever way to estimate future success states - and effectively eliminates the need for an explicit success label for each visited state. The experiments clearly demonstrate superior performance on a number of control tasks. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer's initially agreed upon the strong novelty of this paper's "data driven" task specification, which shares ideas with existing thoughts in imitation and goal-conditioning, but is both intuitively useful in practice and well supported by the paper's elements. Good points were raised by reviewers 3uQ8 and yauQ, in particular, about the relationship of the theory to existing proofs and of the most fair discussion of baseline tasks and comparisons. After a healthy exchange, the picture was quite clear that the paper's elements are nicely distinct and add important new ideas. 

I'm certain this paper will draw a healthy audience from offline-RL and imitation practitioners. In some domains where the author's task specification interface is indeed a much better fit, this work may have rather significant impact. For the overall RL/imitation community, it is another nicely done example of using a mix of existing and new tools to develop coherent theory and algorithms for a new spin on the problem. This is likely to be cited and to support others innovating in different ways in the space.
Decision: Accept (Oral)