Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Authors: Benjamin Eysenbach
Sergey Levine
Ruslan Salakhutdinov
Venue: NeurIPS 2021 Oral (2021)
Keywords: reinforcement learning
example-based control

Abstract: Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another reinforcement learning algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.

Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification
Benjamin Eysenbach" 12
Sergey Levine
Ruslan Salakhutdinoy! UCC Berkeley
Carnegie Mellon University, Google Brain,
beysenbaucs cmu. edu
Abstract
Reinforcement learning (RL) algorithms assume that users specify tasks by man- ually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, derive control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer byperparameters tune and lines of code debug. We show that our method satisfies new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions."
Introduction
In supervised learning settings, tasks are defined by data: what causes car detector to detect cars is not the choice of loss function (which might be the same as for airplane detector), but the choice of training data. Defining tasks in terms of data, rather than specialized loss functions, arguably makes easier to apply machine learning algorithms to new domains. In contrast, reinforcement learning (RL) problems are typically posed in terms of reward functions, which are typically manually designed. Arguably, the challenge of designing reward functions has limited RL to applications with simple reward functions, and has been restricted to users who speak this language of mathematically- reward functions. Can we make task specification in RI. similarly data-driven?
Whereas the standard MDP formalism centers around predicting and maximizing the future reward, we will instead focus on the problem classifying whether a task will be solved in the future. The user will provide a collection of example success states, not reward function. We call this problem setting example- caaslle-based control. In effect, these examples tell the agent "What would the world look like if the task were solved?" For example, for the task of opening door, success examples correspond to different observations of the world when the door is open. The user can find examples of success not know how to solve. For example, the user could solve the
even for tasks that they themselves task using actions unavailable to the agent (e.g., the user may have two arms, but robotic agent may have only one) or the user could find success examples by searching the internet. As we will discuss in Sec. 3.1, this problem setting different from imitation learning: maximize a different objective function and only assume access to success examples, not entire expert trajectories.
Project site with videos and code: https://ben--eysebbacch wit.tttttttbbioooe.oe
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

r(s,d) = -1.0] Spain Sabil2
Bsoa
10 Aunoe Aowalla 0.0i]]saaootvllle
observations
observations
actions
actions
success examples
Markov decision process
Example-based Control
Figure 1:
Example-based control: Whereas the standard MDP framework requires a user-defined reward function, example-based based control specifies tasks handful of user-provided success examples.
Learning from examples is challenging because must automatically identify when the agent has solved the task and reward for doing so. Prior methods (either imitation learning from demon- strations or learning from success examples) take indirect approach that resembles inverse RL: first learn a separate model to represent the reward function, and then optimize this reward function with standard RL algorithms. Our method is different from these prior methods because it learns to predict future success directly from transitions and success examples, without learning a separate reward function. This key difference has important algorithmic, theoretical, and empirical benefits. Algorithmically. our end-to-end approach removes potential biases learning a separate reward function, reduces the number of hyperparameters, and simplifies the resulting implementation. Theo- retically, we propose a method for classifying future events using a variant of temporal difference learning that call recursive classification. This method satisfies a new Bellman equation, where success examples are used in place of the standard reward function term. We use this result to provide convergence guarantees, Empirically, we demonstrate that our method solves many complex manipulation tasks that prior methods fail to solve.
Our paper also addresses subtle but important ambiguity in formulating example-based control. Some states might always solve the task while other states might rarely solve the task. But, without knowing how often the visited each state, we cannot determine the likelihood that each state solves the task. Thus, an agent can only estimate the probability of success by making an additional assumption about how the success examples were generated. We will discuss two choices of assumptions. The first choice of assumption is convenient from an algorithmic perspective, but is sometimes violated in practice. A second choice is worst-case approach, resluting in a problem setting that we call robust example- -aaee control. Our analysis shows that the robust example-based control objective is equivalent to minimizing the squared Hellinger distance (an f-divergence).
In summary, this paper studies a data-driven framing of control, where reward functions are replaced by examples of successful outcomes. Our main contribution is an algorithm for off-polic example- based control. The key idea of the algorithm to directly learn to predict whether the task will be solved in the future via recursive classification, without using separate reward learning and policy search procedures. Our analysis shows that our method satisfies a new Bellman equation where rewards replaced by data (examples of success). Empirically, our method significantly outperforms state-of-the-ar imitation learning methods (AIRL [7), DAC [17], and SQIL (28)) and recent methods that learn reward functions (ORIL [41], PURL [38]. and VICE [8]). Our method completes complex tasks, such as picking up a hammer to knock a nail into a board, tasks that none of these baselines can solve. Using tasks with image observations, demonstrate agents learned with our method acquire notion of success that generalizes to new environments with varying shapes and goal locations.
2 Related Work
Learning reward functions. Prior works have studied RL in settings where the task is specified either with examples of successful outcomes or complete demonstrations. These prior methods typically learn reward function from data and then apply RL to this reward function (e.g.. Fu et al. [8], Ziebart et al. [40]). Most inverse RL algorithms adopt this approach [1, 25, 27, 29, 36, 40], as do more recent methods that learn a success classifier to distinguishing successful outcomes from random states [8, 16, 31, 41]. Prior adversarial imitation learning methods [7, 12] can be

---

viewed as iteratively learning a success classifier. Recent work in this area focuses on extending
these methods the offline setting [16, 41], incorporating additional sources of supervision [42], and learning the classifier via positive-unlabeled classification [13, 38, 41]. Many prior methods for robot learning have likewise used classifier to distinguish success examples [2, 22, 34, 37]. Unlike these prior methods, our approach only requires examples of successful outcomes (not expert trajectories) and does not learn a separate reward function. Instead, our method learns a value function directly from examples, effectively "cutting out the middleman." This difference from prior work removes hyperparameters and potential bugs associated with learning a success classifier. Empirically, we demonstrate that our end-to-ens toddto--d approach outperforms these prior two-stage approaches. See Appendix D for more discussion of the relationship between our method and prior work.
Imitation learning without auxiliary classifiers. While stample-based control is different from imitation learning, our method similar two prior imitation learning methods that likewise avoid learning a separate reward function [18, 28]. ValueDICE [18], a method based on convex duality, uses full expert demonstrations for imitation learning. In contrast, our method learns from success examples, which are typically easier to provide than full expert demonstrations. SQIL [28] is modification of SAC [11] that labels success examples with reward of +1. The mechanics of our method are similar to SQIL [28], but key algorithmic differences (backed by stronger theoretical guarantees) result better empirical performance. Our analysis Sec. 4.2 highlights connections and differences between imitation learning and example-based control.
Gonl-conditioned RL. Goal-conditioned RL provides one way specify tasks in terms of data, and prior work has shown how goal-conditioned policies can be learned directly from data, without a reward function [4, 15, 21, 30]. However, the problem that we study in this paper, example-based control, is different from goal-conditioned RL because it allows users to indicate that tasks can be solved in many ways, enabling the agent to learn more general notion of success. Perhaps the most similar prior work in this area is C-learning [4], which uses lemporal-difference difference update to learn the probability that goal state will be reached in the future. Our method will use a similar temporal-differesn update to predict whether any success example will be reached in the future. Despite the high-level similarity with C-learning, our algorithm will be markedly different; for example, our method does not require hindsight relabeling, and learns a single policy rather than a goal-conditioned policy.
3 Exxaple-Basse Control via Recursive Classification
We aim to learn policy that reaches states that are likely to solve the task (see Fig. 1), without relying on a reward function. We start by formally describing this problem, which we will call example-based control. We then propose method for solving this problem and provide convergence guarantees.
3.1 Problem Statement
Example- control is defined by a controlled Markov process (i.e., an MDP without reward function) with dynamics p($t+1 St, ax) and an initial state distribution pr (s1). where St E S and a denote the time-indexed states and actions. The variable St+ A denotes a state A steps in the future.
The agent given a set of success examples, S* = {s*} S. The random variable e E {0, 1} indicates whether the task is solved at time i, and p(et (t))) denotes the probability that the current state s, solves the task. Given a policy Ta(au st), we define the discounted future state distribution:
pd (st Se, a,) $ (1 71 Σ (S
= St+ Se. at).
A=0
Using this definition, we can write the probability of solving the task at future step as
pd (ce+ St, ax) - Epp (ore4,ss.00 ( (st)]]
(2)
Example-basee based control maximizes the probability of solving the task in the (discounted) future:
Definition 1 (Example-based control). Given controlled Markov process and distribution over success examples p(st et = 1), the example-base based control problem is to find the policy that optimizes the likelihood of solving the task:
arg maxp" (er++ = 1) = Ep (p1())) (x) (et+ = |1,1,1]) (1,a1]]-
x
3

---

Although this objective is equivalent to the RL objective with rewards r(se: a = p(e = st), we assume the probabilities p(e, st)) are unknown. Instead, we assume that we have samples of successful states, s" (st because imitation learning requires full expert demonstrations. In the special case where the user provides a single success state, example-hase hased control is equivalent to goal-conditioned RL.
1). Example-based control differs from imitation learning
Since interacting with the environment to collect experience is expensive in many settings, we define off-policy example-based control as the version of this problem where the agent learns from environment interactions collected from other policies. In this setting, the agent learns from two distinct datasets: (1) transitions, {Se, ag, St+1) ~ (st, ag $t-1)}, which contain information about the environment dynamics; and (2) success examples, S {s* (s, e = 1)}, which specify the task that the agent should attempt to solve. Our analysis will assume that these two datasets are fixed. The main contribution of this paper is an algorithm for off- off-policy example- control.
An assumption on success examples. The probability of solving the task at state St. p(et = St), cannot be uniquely determined from success examples and transitions alone. To explain this ambiguity, we define Pu (st) as the state distribution visited by the user, note that the user may be quite bad at solving the task themselves. Then, the probability of solving the task at state St depends on how often a user visits state versus how often the task is solved when visiting state sp:
p(et= st) (st e 1) (ut = 1)).
(3)
(St(s:)
For example, the user may complete a task using two strategies, but we cannot determine which of these strategies is more likely to succeed unless know how often the user attempted each strategy. Thus, any method that learns from success examples must make an additional assumption on po (st). We will discuss two choices of assumptions. The first choice is to assume that the user visited states with the same frequency that they occur in the dataset of transitions. That is,
pU = /f (St, at, St+
(4)
Intuitively, this assumption implies that the user has the same capabilities as the agent. Prior work makes this same assumption without stating it explicitly [8. 24, 31]. Experimentally, we find that our method succeeds even in cases where this assumption is violated.
However, many common settings violate this assumption, especially when the user has different dynamics constraints than the agent. For example, a human user collecting success examples for a cleaning task might usually put away objects on a shelf at eye-level, whereas transitions collected by robot interact with the ground-level shelves more frequently. Under our previous assumption, the robot would assume that putting objects away on higher shelves is more satisfactory than putting them away on lower shelves, even though doing so might be much more challenging for the robot. To handle these difference in capabilities, the second choice is to use a worst-case formulation, which optimizes the policy to be robust to any choice of (s.).. Surprisingly, this setting admits a tractable solution, as we discuss in Sec. 4.2.
3.2 Predicting Future Success by Recursive Classification
We now describe our method for example- based control. We start with the more standard first choice for the assumption on pu (st) (Eq. 4); we discuss the second choice in Sec. 4.2. Our approach estimates the probability in Eq. 2 indirectly via future success classifier. This classifier, C3 (St. ar), discriminates between "positive" state-action pairs which lead to successful outcomes (i.e., sampled from p(St, at et+ = 1)) and random "negatives" (i.e., sampled from the marginal distribution p(s,, We will use different class-specifie weights, using a weight of p(ec+ 1) for the "positives" and a weight of for the "negatives." " Bayes-optimal classifier is
(St, at ee+ = 1 1)p(ec+ = 1)
(st, at) p= (= a | ett 1)p(ec+ 1) + p(se. at)"
(5)
These class specific weights let us prediet the probability of future success using the optimal classifier:
C% (s,,
= p" (et+ = 1 St. at).
(6)
1- CJ (St, an))
4

---

Algorithm 1 Recursive Classification of Examples
success examples S* Initialize policy a 8t), classifier CJ (Se, at), replay huffer D while not converged do
Input:
Collect new trajectory: DD +- D fr not
Sample success examples: {sL (1) S" at. ~ To(ar | se
Sample transitions: {(Se (2) ar (2) 8t41) D, a To(a sc41)]
Eq.
1- (
(()) < () (2) at 3y 1) (1 (WE ac ()) )
Update classifier: l + 0 (oL(
Eq. 8
Update policy: +- e V V &.En (an Ise (Co(st:
return
Importantly, the resulting method will not actually require estimating the weight p(ec+ = 1). We would like to optimize the classifier parameters using maximum likelihood estimation:
L (0) _ p(e++ = 1)
llog C/ (st, at)] + (opttr.a,)logg
C. (st. at))]. (7)
However, we cannot directly optimize this objective because we cannot sample from p(s,, ag et+ 1). We convert Eq. 7 into an equivalent loss function that we can optimize us- ing three steps; see Appendix A for a detailed derivation. The first step is to factor the distribution p(St, a et+ 1). The second step to decompose (et+ sponding to the probabilities of solving the task at time t' = t + 1 and time t' > + We can estimate the probability of solving the task at the next time step using the set of success examples. The third step classifier at the next time step. Combining these three steps, we can equivalently express the objective function in Eq. 7 using off-policy data:
1 St. at) into two terms, corre-
estimate the probability of solving the task at time
++ by evaluating the
(0) =(1 - 7) (ts 1) [log CT (St, ac)] + Epire (ru log (se, at) + log(1 Cā
(St. a
p(ac)e))
(e)
(b)
(c)
(8)
where
C7 (St+1, a++1)
Ep(ae (1
(9)
1- C- (Se+1, act1).
the classifier's prediction (ratio) at the next time step. Our resulting method can be viewed as temporal difference [33] approach to classifying future events. We will refer to our method as recursive classification of examples (RCE). This equation has an intuitive interpretation. The first term (a) trains the classifier to predict for the success examples themselves, and the third term (c) trains the classifier to predict for random transitions. The important term the second term (b), which is analogous to the "bootstrapping" term in temporal difference learning [32]. Term (b) indicates that the probability of future success depends on the probability of success at the next time step, as inferred using the classifier's ' own predictions.
Our resulting method is similar to existing actor-critic RI. algorithmss. existing actor-critic methods, we can combine the (b) and (c) terms in the classifier objective function (Eq. 8) to express the loss function in terms two cross entropy losses:
To highlight the similarity to
min (1 - 7) T. pley 11) (CE (St, at); 1)] + (1 + yw)) (e (CE(CT (Se. a))y 0
m (at (Rt)
YPu +
(10)
These cross entropy losses update the classifier to predict y 1 for the success examples and to predict y
7w0 for other states.
1470
Algorithm summary. Alg. summarizes method, which alternates between updating the classifier, updating the policy, and (optionally) collecting new experience. We update the policy to choose actions that maximize the classifier's confidence that the task will be solved in the future: max ( le.) IC (St, ).)). Following prior work [5, 35]), we regularized the policy updates by adding an entropy term with coefficient a = 10-4. We also found that using N-step returns significantly improved the results of RCE (see Appendix F for details and ablation experiments.). Implementing our method on top of existing methods such as SAC [11] or TD3 [9] requires only changing the standard Bellman loss with the loss Eq. 10. See Appendix E for implementation details; code is available on the project website.
5

---

4 Analysis
In this section, we prove that RCE satisfies many of the same convergence and optimality guarantees (for example-based control) that standard RI algorithms satisfy (for reward-based MDPs). These results are important as they demonstrate that formulating control in terms of data, rather than rewards, does preclude algorithms from enjoying strong theoretical guarantees. Proofs of all results given in Appendix B, and we include further discussion of how RCE relates to prior work in Appendix D.
4.1 Bellman Equations and Convergence Guarantees
To prove that RCE converges to the optimal policy, will first show that RCE satisfies new Bellman equation:
Lemma 4.1. The Bayes- classifier C* for policy x satisfies the following identity:
C (st, -(1 >))((; St) + p(s+ 1 [sc
Ca (1, C C5 (8 at+1)
(II)
1 C* (st, at)
(P (8111)
The proof combines the definition of the Bayes-optimal classifier with the assumption from Eq. 4. This Bellman equation is analogous to the standard Bellman equation for Q-learning, where the reward function is replaced by (1 7)p(e
st) and the Q function is parametrized as Q7 at) = (s) While we do not know how to compute this reward function, the update
rule for RCE is equivalent to doing value iteration using that reward function and that parametrization of the Q-function:
Lemma 4.2. In the labular setting, the expected updates for RCE are equivalent doing value iteration with the reward function r(8e) = -7)ppe = 1 8t) and a Q-function parametrized as Q7 ax) C7 (e)
(St
This result tells us that RCE is equivalent to maximizing the reward function (1-7)p(et = st); however, RCE does not require knowing p(e, = 1 St), the probability that each state solves the task. Since value iteration converges in the tabular setting, an immediate consequence of Lemma 4.2 that tabular RCE also converges:
Corollary 4.2.1. RCE converges in the tabular setting.
So far we have analyzed the training process for the classifier for a fixed policy. We conclude this section by showing that optimizing the policy w.r.t the classifier improves the policy's performance.
Lemma 4.3. Let policy Ta Se) and success examples S be given, and let C (st. at) denote the corresponding Bayes-optimal classifier. Define the improved policy as acting greedily w.r.t. C n' r'(a, Se) a(a = arg max, C (²((s,))) a)). Then the improved policy is at least as good as the old policy at solving the task: p" (er+ = 2 nr (e++ 1).
4.2 Robust Example-based Control
In this section, we derive principled solution for the case where po (St) ) is not known, which will correspond to modifying the objective function for example-based control. However, we will argue that, in some conditions, the method proposed in Sec. 3.2 is already robust unknown po Pu (s,), that method is used with online data collection. The goal of this discussion is to provide theoretical relationship between our method and a robust version of example-base based control that makes fewer assumptions about pe (St). This discussion will also clarify how changing assumptions on the user's s capabilities can change the optimal policy.
When introducing example-based control in Sec. 3.1, we emphasized that we must make an assump- tion make the example-based control problem well defined. The exact probability that a success example solves the task depends on how often the user visited that state, which the agent does not know. Therefore, there are many valid hypotheses for how likely each state solve the task. We can express the set of valid hypotheses using Bayes' Rule:
st) = Pu (st et 1)p(et = =1)
Payls $ |lle
||s
(st(S)

---

Previously (Sec. 3.2), we resolved this ambiguity by assuming that (st) was equal to the distribution over states in our dataset of transitions. As discussed in Sec. 3.1, many problem settings violate this assumption, prompting us to consider the more stringent setting with no prior information about pr((st) (e.g., no prior knowledge about the user's capabilities). To address this settting, we will assume the wors! possible choice of pr (St). ). This approach will make the agent robust to imperfect knowledge of the user's abilities and to mislabeled success examples. Formally, we define the robust example-based control problem
pu (st et = ple: = 1) (s.)
max
min P(or)ae)EPeel ( Ip(ee++ 11| [|++)]]] = max min Bpriol)) (pc)
(12)
This objective can be understood as having the adversary assign a weight of 1 1/pu (8t) ) to each success example. The optimal adversary will assign lower weights success examples that the policy frequently visits and higher weights to less-visited success examples. Intuitively, the optimal policy should try to reach many of the success examples, not just the ones that are easiest to reach. Thus, such a policy will continue to succeed even if certain success examples are removed, or are later discovered to have been mislabeled. Surprisingly, solving this two-player game corresponds to minimizing an f-divergence:
/ dx as the squared Hellinger distance, an f-divergence. Robust example-based control (Eq. 12) equivalent to minimizing the squared Hellinger distance between policy's 's discounted state occupancy measure and the conditional distri- bution p(s: 1e==1). 1):
Lemma 4.4. Define H²(p(x), ([p(x)){(xi]==== =[x))]][[Pp
min p p(ex )
- (-.( [ppst = 1).,"(sst = =[]]]-
The main idea of the proof (found in Appendix C) is compute the worst-case distribution (st))) using the calculus of variations. Preliminary experiments (Fig. 5 in Appendix C) show that a version of RCE with online data collection finds policies that perform well on the robust example-based control objective (Eq, 12). In fact, under somewhat stronger assumptions, we can show that the solution of robust xxamplebbased control is fixed point of iterated RCE (see Appendix C.2). Therefore, in our experiments, we use RCE with online data collection.
5 Experiments
Our experiments study how effectively RCE solves example-based control tasks, especially in comparison to prior methods that learn an explicit reward function. Both RCE and the prior methods receive only the success examples as supervision; reward functions. Additional experiments in Sec. 5.2 study whether RCE can solve tasks using image observations. These experiments test whether RCE solve tasks in new environments that are different from those where the success examples were collected, and test whether RCE learns policies that learn a general notion of success rather than just memorizing the success examples. We include videos of learned policies online? and include implementation details, hyperparameters, ablation experiments, and a list of failed experiments in the Appendix.
method has access to expert trajectories of
We compare RCE against prior methods that infer a reward function from the success examples and then apply an off-the-shelff RL algorithm; some baselines iterate between these two steps. AIRL [7] a popular adversarial imitation learning method. VICE [8] is the same algorithm as AIRL, but intended to be applied success examples rather than full demonstrations. We will label this method as "VICE" in figures, noting that it is the same algorithm as AIRL. DAC [17] is a more recent, off-policy variant of AIRL. We also compared against two recent methods that learn rewards from demonstrations: ORII. [41] and PURL (38). Following prior work [16], we also compare against "frozen" variants of some baselines that first train the parameters of the reward function and then apply RI. to that reward function without updating the parameters of the reward function again. Our method differs from these baselines in that we do not learn a reward function from the success examples and then apply RL, but rather learn a policy directly from the success examples. Lastly, we compare against SQIL [28], an imitation learning method that assigns a reward of +1 to states from demonstrations and 0 to all other states. SQIL does not learn a separate reward function and structurally resembles our method, but is derived from different principles (see Sec. 2.).
hhttps://ben: http://bbb..yeeeab....ec.a github. tattthu..o/rreee..
7

---

sawyer drwwwr.wwoooo peeen
sawyer_ push
sawyer lift
doar-human hurma-vv
2.06
Return
2.00
1:00
2.00
000
25
10
15
20
25
50
25
10
2.3
0.0
25
10
1.
2.
25
30
10
85
15
15
20
25
sawyer box close
sawyer bin picking
Saamee-humanee
Training Steps
RCE
VIC
SQL
ORL-AL
05-
UAL
1.00
DAC--roaes
PURL
VICE
PURL frozen
0.0 2.5
1.5
2-4
2.5
00
05
15
1.0
0.0
10
24
255
Training stops
Training staps
Training staps
Figure 2: Recursive Classification of Examples for learning manipulation tasks: We apply RCF to a range of manipulation tasks, each accompanied with a dataset of success examples. For example, the sawyer_. __ _,fft task, we provide success examples where the object has heen lifted above the table. We use the cumulative task return (1 is better) solely for evaluation. Our method (blue line) outperforms prior methods across all tasks.
5.1 Evaluating RCE for Example-Based Control.
We evaluate each method on five Sawyer ma- nipulation tasks from Meta-World [39] and two manipulation tasks from Rajeswaran et al. [26]. Fig. 3 illustrates these tasks. On each task, we provide the agent with 200 successful out- comes define the task. For example,
the open_ deeewr task, these success examples show an opened drawer. As another example, on the sawyer_ -areer_pushhhe task, success examples not only have the end effector touching the puck. but (more importantly) the puck position is different. We emphasize that these success examples only reflect the final state where the task is solved and are not full expert trajectories. This setting is important in practical use-cases: it is often easier for humans to arrange the workspace into a successful configuration than it is to collect an entire demonstration. See Appendix E.3 for details on how success examples were generated for each task. While these tasks come with existing user-defined reward functions, these rewards are not provided to any of the methods in our experiments and are used solely for evaluation (f is better). We emphasize that this problem setting is exceedingly challenging: the agent provided only with examples of success states (e.g., an observation where an object has been placed in the correct location). Most prior methods that tackle similar tasks employ hand-designed reward functions or distance functions, full demonstrations,
Figure 3: Manipulation Environments
surcfullly-cossruuee initial state distributions.
The results in Fig. 2 show that RCE significantly outperforms prior methods across all tasks. The transparent lines indicate one random seed, and the darker lines are the average across random seeds. RCE solves many tasks, such as bin picking and hammering, that none of the baselines make any progress towards solving. The most competitive baseline, SQIL, only makes progress on the easiest two tasks; even on those tasks, SQIL learns more slowly than RCE and achieves lower asymptotic return. To check that all baselines are implemented correctly, we confirm that all can solve a very simple reaching task described in the next section.
5.2 Example-Bare Based Control from Images
Our second set of experiments studies whether RCE can learn image-based tasks and assesses the generalization capabilities of our method. We designed three inage-based manipulation tasks. The reach_rrndee
on task entails reaching a red puck, whose position is randomized in each episode. The reaac_eraaneeeeeee psiieesssseeeee task entails reaching red object, but the actual shape of that object varies from one episode to the next. Since the agent cannot change the size of the object and the size is randomized from one episode to the next, it is impossible to reach any of the previously- observed success examples. To solve this task, the agent must leam a notion of success that is more general than reaching a fixed goal state. The third task, sawyer_ weryerrclear__r clear diderrimaee,,___ll entails clearing an object off the table, and is mechanically more challenging than the reaching tasks.
Fig. 4 shows results from these image-based experiments, comparing RCE to the same baselines. We observe that RCE has learned to solve both reaching tasks, reaching for the object regardless of the location and size of the object. This task is mechanically easier than the state- based tasks in Fig, 2,
8

---

sereerrree___.e randon..ppaa wetaddnn_oeeiiiooo poeition, inage
eavyer rww.crreccc__aee randon, sime_ image
aakyer clear inage
6.00
SC4
VCC
RRL
6:00
0.0
8.5
Figure 4: Example-hased control from images: We evaluate RCE on three manipulation tasks using image- observations. (Top) We show examples the mmitaal state and success examples for each task. (Bottom) RCE (blue line) outperforms prior methods, especially on the more challenging clearing task. For the random_size task (center), this entails reaching for new objects that have different sizes from any seen in the success examples.
and all the baselines make some progress on this task, but learn more slowly than our method. The good performance of RCE on the a new environment, where the object size is different from that seen in the success examples. We hypothesize that RCE learns faster than these baselines because it "cuts the middleman," " learning value function directly from examples rather than indirectly via separate reward function. To support this hypothesis, we note SQIL, which also avoids learning a reward function, learns faster than other baselines on these tasks. On the more challenging clearing task, only our method makes progress, suggesting that RCE more effective algorithm for learning these image- control tasks. In summary, these results show that RCE outperforms prior methods at solving example-based control tasks from image observations, and highlights that RCE learns policy that solves tasks in new environments that look different from any of the success examples.
tit__random_ssii_ size task illustrates that RCE can solve tasks in
5.3 Ablation Experiments
We ran seven additional experiments to study the importance hyperparameters and design decisions. Appendix F provides full details and figures. These experiments highlight that RCE is not imitation learning method: RCE fails when applied to full expert trajectories, which are typically harder to provide than success examples. Other ablation experiments underscore the importance of using n-step returns and validate the approximation made in Sec. 3.2.
Conclusion
In this paper, we proposed a data-driven approach to control, where examples of success states are used in place of reward function. Our method estimates the probability of reaching success example the future and optimizes a policy to maximize this probability of success. Unlike prior imitation learning methods, our approach is end-to-end and does require learning reward function. Our method is therefore simpler, with fewer hyperparameters and fewer lines of code to debug. Our analysis rests on new data-driven Bellman equation, where example success states replace the typical reward function term. We use this Bellman equation to prove convergence of our classifier and policy. We believe that formulating control problems in terms of data, rather than the reward-centric MDP, better captures the essence of many real-world control problems and suggests a new set of attractive learning algorithms.
Limitations and future work. One limitation with RCE is that, despite producing effective policies, the classifier's predictions are not well calibrated. This issue resembles the miscalibration in Q- functions observed in prior work [9, 20]. Second, both RCE and the baselines we compared against all struggled learn more challenging amage-based tasks, such image-based versions of the tasks shown in Fig. 3. Techniques such as explicit representation learning [19, 23] may be important for scaling example-based control algorithms to high-dimensional tasks.
Acknowledgements. This work is supported by the Fannie and John Hertz Foundation. NSF (DGE1745016.. IIS I1S1763562), ONR (N000141812861) and US Army. We thank Ilya Kostrikov for providing the implementation of DAC. We thank Ksenia Konyushkova, Konrad Zoina, and Justin Fu for discussions about basclines and prior work. We thank Oscar Ramirez for help setting up the image-based experiments, Ryan Julian and anonymous reviewers for feedback on drafts of the paper, and Alex Irpan for help releasing code.

---

References
[1] Abbecl, P. and Ng. A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, page
[2] Calandra, R., Owens. A.. Upadhyaya, M., Yuan. W., Lin, J., Adelson, E. H., and Levine. S. (2017). The feeling of success: Does touch sensing help predict grasp outcomes? In Conference on Rohot Learning. pages 314-323. PMIR.
[3] Elkan, C. and Noto, K. (2008). Learning classitiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference Knowledge discovery and data mining, pages 213-220.
[4] Eysenbach, B., Salakhutdinov, R., and Levine, S. (2021). C-learning: Learning to achieve goals via recursive classification. In International Conference on Learning Representations.
[5] Fox, R., Pakman, A., and Tishby, N. (2016). Taming the noise in reinforcement learning via soft updates. In Proceedings of the Thirty-Secone Conference on Uncertainty in Artificial Intelligence, pages 202-211.
[6] Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07211
[7] Fu, J., Luo, K., and Levine, S. (2018a). Learning robust rewards with adverserial inverse reinforcement learning. International Conference on Learning Representations.
[8] Fu, J., Singh, A.. Ghosh, D., Yang. L., and Levine. S. (2018b). Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31:8538- 8547.
[9] Fujimoto, S.. Hoof. H., and Meger, D. (2018). Addressing function approximation enror in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR.
[10] Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N.. Kokiopoulou. E., Sbaiz, L... Smith, J., Bartók, G.. Berent, J., Harris, C., Vanhoucke, V.. and Brevdo, E. (2018). Ag-Att:::: A library for reinforcement learning in tensorflow. [Online; accessed 25 25-June-2019],
[11] Haarnoja, T., Zhou, A., Abheel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR.
[12] Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. In NIPS, pages 3565-4573.
[13] Irpan, A., Rao, K., Bousmalis, K., Harris, C., Ibarz, and Levine, S. (2019). Off-policy evaluation via pof-po classification. In Advances in Neural Information Processing Systems, pages 5437-5448.
[14] Jaakkola, T., Jordan, M. L., and Singh, S. P. (1994). On the convergence of stochastic iterative dynamic programming algorithms. Neural computation, 6(6): 165::185-1201..
[15] Kaelbling, L. P. (1993). Learning achieve goals. In IJCAI, pages 1094-1099, Citeseer.
[16] Konyushkova, K., Zolna, K., Aytar, Y., Novikov, A., Reed, S., Cabi, S., and de Freitas, N. (2020). Semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899,
[17] Kostrikov, Agrawal, K. K., Dwibedi, D., Levine, S., and Tompson, (2018). Discriminator actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation leariing In International Conference on Learning Representations.
[18] Kostrikov, Nachum. O., and Tompson. (2019). Imitation learning off-policy distribution matching. In International Conference on Learning Representations.
[19] Laskin, M., Srinivas, A., and Abbeel, P. (2020). Curl: Contrastive unsupervised representations for
reinforcement learning. International Conference on Machine Learning, pages 5639-5650. PMLR.
[20] Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D, (2016). Continuous control with deep reinforcement learning. In ICLR (Poster).
[21] Lin, X., Baweja, H. S., and Held, D. (2019). Reinforcement learning without ground-truth state. arXiv preprint arXiv: 190.00786..
(22] Lu, Q.. Van der Merwe, M., Sundaralingam, B., and Hermans, T. (2020). Multifingered grasp planning via inference in deep neural networks: Outperforming sampling by learning differentiable models. IEEE Robotics Automation Magazine, 27(2):55-66 65.
10

---

[23] Nachum, O., Gu, S., Lee. H., and Levine. S. (2018). Near-optimal representation learning for hierarchical reinforcement learning. In International Conference Learning Representations.
[24] Nasiriany, S. (2020). DisCo RL: Distribution-Cnnlltoone Reinforcement Learning for General- Purpose Policies. PhD thesis.
[25] Pomerleau, D. A. (1989). Alvinm: An autonomous land vehicle in a neural network. In Advances in neural information processing systems, pages 305-313.
[26] Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2018). Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS).
[27] Ratliff, N. D., Bagnell, J. A., and Zinkevich, M. A. (2006). Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pages 729-736.
[28] Reddy, S., Dragan, A. D., and Levine, S. (2020). (SQIL): Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations.
[29] Ross, S., Gordon, G., and Bagnell, D. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635.
[30] Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015). Universal value function approximators. In International conference machine learning, pages 1312-1320.
[31] Singh, A., Yang. I.,, Finn, C., and Levine, (2019). End-to-end robotic reinforcement learning without reward engincering. In Robotics: Science and Systems.
[32] Sutton, S. (1988). Learning to predict by the methods of temporal differences. Machine learning, 3(1):9-44.
[33] Sutton, (1995). Td models: Modeling world a mixture time scales. Machine Learning Proceedings 1995, pages 531-539. Elsevier.
[34] Vecerik, M., Sushkov, O., Barker, Rothörl, T., Hester, T., and Scholz, J. (2019). A practical approach insertion with variable socket position using deep reinforeement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 754-760. IEEE.
[35] Williams, algorithms. Connection Science, 3(3):241
Peng. (1991). Function optimization using connectionist reinforvement learning
[36] Wulfmeier, M., Ondruska, and Posner, I. (2015), Maximum entropy deep inverse reinforeement lcarning. arXiv preprint arXiv:1507.04885 04888.
[37] Xie, A., Singh, A., Levine, S., and Finn, (2018). Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pages 40-52. PMLR.
[38] Xu, D. and Denil, M. (2019). Positive-unlabeled reward learning. arXiv preprint arXiv:1911.00459
[39] Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020). Meta-world: benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Rohot Learning, pages 1094-1100 1100. PMLR.
[40] Ziebart, B. D., Maas, A. L.. Bagnell, J. A., and Dey, A. K. (2008). Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433- 1438. Chicago. IL, USA.
[41] Zolna, K., Novikov, A., Konyushkova, K., Gulcehre, C., Wang, 7., Aytar, Y., Denil, M., de Freitas, N., and Reed, S. (2020). Offline learning from demonstrations and unlabeled experience. arXiv preprint 1rXiv:2011.1380 1.100.11188..
[42] Zolna, K. Reed, S., Novikov, A., Colmenarej. S. G., Budden, D., Cabi. S., Denil, M. de Freitas, N., and Wang. Z. (2019). Tesk-relevant adversarial imitation learning, arXiv preprint 1.10.1797.
11

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 6.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do foresee plausible negative impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] Code is available at:
httpa://github. cwmcccm///eee/eceeeeeeeeoeeeeeee htte/ree-aanoon anonymous/rce-annntooo anonymooss github heo/trree/min/cooe
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A and provided code.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] Each experiment took approximately one day on a standard CPU server. The exact compute resources are proprietary.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "uvrAAKXtjoM", "reviewer": "Reviewer_6QWr", "summary": "The paper proposes an approach for reinforcement learning in which the reward function is replaced by a recursive classification operation. More specifically, the method identifies if a success state will be reached. The set of success states is provided by an expert as a replacement for the reward function. The underlying rationale is that such information is much easier and more intuitive to provide that complex reward functions. The paper introduces the algorithm but also provides a rigorous formal discussion of its properties. In particular, proofs are provided that ensure that learned classifiers converge to the Bayes-optimal case and that the system converges to the optimal policy. Beyond that, the paper also discusses a variant of the method under stricter assumptions, called robust example-based control. Experiments in simulated manipulation tasks show that the method substantially outperforms previous methods.  ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main rationale behind this paper is not necessarily novel, other papers have already indicated that examples and outcomes can be used for improved RL. However, this paper consequently and rigorously follows this line of thought and discusses a number of exciting algorithmic and theoretical tricks to completely do away with the reward function. The derivation of the algorithm is carefully explained and detailed (even if not necessarily trivial or easy to understand). I also seems to me like the introduced framework can be reused by others to create a family of example-based policy search algorithms. For example, the Bayes classifier may be replace with a more powerful method down the road, as long as it satisfies similar theoretical conditions. The presented work is how very high quality and includes clear improvements along both the theoretical and practical dimension. The empirical results show a clear advantage over a number of SotA methods from very recent papers. The chosen tasks and domains are among the trickier 3D and physics-based benchmarks for RL. The paper is very well written and meticulous in it's presentation. The authors also do a great job in highlighting interesting insights they discovered along the way, i.e., the equivalence to Hellinger distance. Overall, the presentation is very clear.\n\nOverall, I think that the introduced method is high significance to the NeurIPS community and that the authors did a great job in deriving the method and presenting it. In particular, the paper addresses different modifications of the method - generally a sign that the authors carefully analyzed their results from different vantage points and under different assumptions. The empirical results are pretty convincing, too. The only slightly confusing aspect is why the authors chose to formulate it in an off-policy fashion. Does this come with any drawbacks or is the decision purely to ensure that information from other policies can be re-used?       ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors convincingly discussed limitations to their approach, e.g., calibration of the classifier. No societal impact expected.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "rJTHwxDPfr", "reviewer": "Reviewer_LgSM", "summary": "This paper proposes a novel RL algorithm that operates without an explicit reward function. Instead, the algorithm directly learns a value function from transitions and successful outcomes without ever learning an intermediate reward function. This approach also satisfies a new data-driven Bellman equation, where examples replace the reward function. The authors prove that this new Bellman equation converges to a Bayesian optimal classifier and optimal policy. By proving these results, this method retains the theoretical guarantees of the standard reward-based RL  algorithms, while simplying implementation and reducing the number of hyperparameters to be learned. The authors illustrate that their proposed approach performs significantly better than the reward-based RL algorithms on the 'Sawyer manipulation tasks'.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n* The paper is well written and easy to follow.\n* Presenting a data-driven approach to RL is very useful, and this work opens avenues for future work in this area, mitigating the issue of carefully hand-crafting reward functions.\n* By presenting strong theoretical convergence results with the new Bellman equation, the proposed algorithm successfully retains the properties of the existing RL algorithms while still bypassing the reward function. And this result makes the algorithm useful in a practical setting.\n* The experimental evaluations are also comprehensive and illustrate the key ideas of the paper, on complex environments.\n\n**Cons**\n\n* Evaluation: This is not really a negative of the paper, but it would be interesting to see the performance of this approach in a setting where the definition of success of a task is non-binary.  \n\n\n**General Comments**\nThe paper is well written and the ideas presented are very clear. The experimental evaluations are comprehensive. This work opens avenues for future research in data-driven algorithms in RL.\n\n\n*Originality*: Novel\n\n*Clarity*:  Well written.\n\n*Quality*: Good\n\n*Significance*: High\n\nUPDATE (Post Rebuttal): Thank You authors for your responses to the specific queries. After reading the other reviews and the authors' responses to them, I am updating my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of the their work and it poses no apparent negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "mZlF28MKg7W", "reviewer": "Reviewer_xycR", "summary": "This is an interesting work in which the authors propose a new RL method to learn from examples instead of a pre-defined reward function. Existing inverse RL methods commonly introduce additional bias in estimating the reward function, which can be potentially addressed using the approach proposed in this paper.  In the experiments, the authors have demonstrated the effectiveness of this method in complex control tasks. ", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall, I think this paper is very well presented and the work is solid. The authors have done extensive analysis and comparisons to show the effectiveness of the proposed method. I just have one minor comment. I agree that the estimation of reward function in IRL may degrade the performance, but it may also provide some interpretability (if the method is designed properly) on the expectation of the successful actions. Indeed, the proposed method has achieved better accuracy, but it may lose such interpretability. It would be great that the authors can comment on this.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This method can be potentially used in a wide range of real-world applications (e.g., automation control) where the reward function cannot be easily obtained.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "fywO_bbSBDc", "reviewer": "Reviewer_9r4w", "summary": "The paper describes a novel algorithm (RCE) to solve control tasks without an explicit reward function or expert trajectories. Rather, the method relies on a relatively small set of success (goal) states provided by a user. The paper is well written and offers intuitively clear algorithmic ideas which are novel, to my knowledge. The results on several manipulation tasks show strong performance, outperforming state-of-the-art IL baselines. Overall, the paper seems like a good contribution to the literature on learning control from expert data with the main difference that the proposed algorithm only requires a small sample of the distribution over goal states. My score primarily reflects my open questions and I'm open to increasing the score based on the authors response.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The key algorithmic idea is to learn a classifier whose prediction prob ratio is the expected prob that the current (s_t, a_t) will lead to a success state at some point in the future. The main challenge is sampling from successful trajectories, which are not available here. Practically, this means training the classifier to discriminate between state-action pairs on successful trajectories vs the rest, although the \"negative\" examples seem to potentially include successful (s, a) pairs. The loss function combines both types of data examples, which is used to update the classifier's parameters. The behavior policy is trained to maximize the classifier's output. Overall, the approach seems to be intuitively clear and the final algorithm is relatively straightforward. As far as I can tell, this is a novel approach to the problem of learning a policy from a sample distribution of goal states and the environment.\n\n- While the main ideas of the paper are reasonably well described, I think the paper might become a bit easier to understand if a single running example was used throughout the problem setup. This would potentially have the benefits of making the core concepts of task, success state, etc. concrete up front, motivate the need to learn a generalized notion of success (vs simply memorizing the provided examples) as well as the need for assumptions on the user's state distribution in Section 3.1. reach_random_size seems like a candidate but there are probably better domains / tasks.\n\n- Is learning a stopping criteria relevant here? It wasn't clear to me if and how well the agent learns to stop in success states. What's the action $a_t$ in the dataset $(s_t, a_t | e_t = 1)$? During evaluation (test), does the evaluation automatically end when the agent first reaches a success state? It seems to be a mixed bag in the videos (which are a very useful visualization of the learned behavior), where the agent seems to halt in some tasks but not in others (e.g., put the green object in the blue bin). Is the failure case (Pick up the ball) an instance of this issue or something else?\n\n- The experiments are performed on a number of manipulation tasks. The user provides 200 success states for each of the manipulation tasks. The main novelty here is that these are individual states, not trajectories (which IL methods typically require). This is an impressive result and the supporting materials (videos, code, appendix) are quite detailed. However, the overall data requirements of RCE vs IL methods are not clear to me. RCE presumably uses much less expert data than IL but this could be quantified more clearly. A somewhat different set of questions involve analyzing the relatively poor performance of IL methods. Is the reason insufficient expert data? Something else? A more careful comparison and analysis of RCE vs IL methods would significantly improve the paper, in my opinion.\n\n- Overall, I think the paper is well written, tackles important problems, and introduces new algorithmic ideas supported with analysis. The experiments are sufficiently detailed and demonstrate strong performance and outperform existing IL methods. While I think the empirical discussion and analysis could be improved, overall, I think this might be a good addition to the literature on learning control from (expert) data. My score reflects my open questions and I'm open to increasing the score based on the authors response.\n\nUPDATE: I thank the authors for their detailed response to the reviewers. I think that most concerns about this paper have been well addressed at this point. I'm now more inclined to accept the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper includes a brief discussion on its limitations and the authors state that they do not foresee plausible negative impacts of this work. That seems reasonable to me. Perhaps a future consideration might be how to ensure the data collection of success states is conducted well as end users will likely be involved.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This was a well-received paper by all the reviewers.  The discussion strengthened the acceptance score.
Decision: Accept (Oral)