Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

Abstract: Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another reinforcement learning algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.


Reviews and author responses:

**Summary:** This paper offered a new perspective for continuous control tasks, e.g robotic, that root in one question, can agent learn the task by successful examples rather than rewards. By learning a classifier that predicts whether the task could be solved in the future to estimates the probability of solving the task successfully, this method could solve some control tasks that the prior method failed to solve.
  Overall, I am really enjoying this inspiring paper. The proposed method offered a new perspective for continuous control tasks where agents could learn to solve tasks by successful examples rather than rewards. However, few concerns flooded my mind. 

**Originality**:
* I think most of the theoretical result has been done in this work[1]. Line 106-114 describes your work and C-Learning[1] 's relation, but I don't think that applied to the theoretical part.

**Clarity**:
* Line 135, $p_{U}(s_{t} \mid {e}_{t}=1)$ occur before definition. Didn't align with Line 142.

* The loss function in Algorithm 1 doesn't make sense to me. $s_{t+1}$ should be $s_{t}$?

* You acclaim that goal-conditional RL is the special case of the example-based control. Why is such baseline (e.g., C-learning) absent in the experiment setting?

**Significance**:
* While the author would like to deliver the point that agents could learn from successful examples rather than reward in offline settings.  I think the choice of baseline didn't make much sense to me: only one imitation learning(SQIL) baseline is included. 

** Post rebuttal **

After reading the response and other reviewers' feedback, I'd like to improve my score. thank you for your nice work.



[1] Eysenbach, Benjamin, Ruslan Salakhutdinov, and Sergey Levine. "C-learning: Learning to achieve goals via recursive classification." arXiv preprint arXiv:2011.08909 (2020). N/A

 Thank you for the detailed and thorough review.

> example-based control is a generalization of goal-conditioned RL

Our aim was to compare the *objectives* of these two problems. The problem of reaching a success state (example-based control) is equivalent to the problem of reaching one particular goal state (goal-conditioned RL) in the setting where just a single success example is provided. We will clarify the text to emphasize that we are not claiming that RCE is competitive with goal-conditioned RL algorithms on goal-reaching tasks.

> relationship with prior work that re-distributes sparse reward

The key difference between our problem setting and these prior methods is that these prior methods operate on trajectories. That is, these prior methods assume that success examples are embedded in trajectories. Our setting is different because the success examples do not come aligned with any transition data. We will clarify this in the related work section, and include these citations.

> success examples in the sawyer-push task

In the sawyer-push task, success examples not only have the end effector touching the puck, but (more importantly) the puck position is different. As shown in the GIF of success examples on the project website, all success examples for the sawyer-push task have the puck located at the far right side of the table (the puck is always initialized at the left side of the table).

> How were the experimental tasks chosen?

For both the state-based experiments and the image-based experiments, we choose tasks using two design criteria: (1) we wanted to include a breadth of tasks, and (2) we wanted to include the most difficult tasks that could be learned by either our method or any of the baselines. For the image-based tasks, we were unable to get the prior methods to learn tasks more complex than those included in Fig. 4

> tasks the algorithm failed

We found that RCE (and all the baselines) failed on image-based versions of the tasks in Fig. 3. Our hypothesis is that this setting is challenging for RCE for the same reason that image-based tasks are challenging for off-policy RL algorithms: it is challenging to learn Q functions via temporal different learning on high-dimensional inputs. In future work, we aim to study the use of explicit representation learning (e.g., [1, 2]) to assist RCE in learning complex image-based tasks.

> suggestions for figures

Thank you, we will incorporate this feedback for the final version of the paper.

> random seeds

The transparent lines in Fig 3 and Fig 4 each indicate one random seed. The dark lines are the average across random seeds. We'll update the paper to note this.

> Line 163: "The handle" should be "To handle"

We'll fix this.

[1] Srinivas, Aravind, Michael Laskin, and Pieter Abbeel. "Curl: Contrastive unsupervised representations for reinforcement learning." arXiv preprint arXiv:2004.04136 (2020).

[2] Kostrikov, Ilya, Denis Yarats, and Rob Fergus. "Image augmentation is all you need: Regularizing deep reinforcement learning from pixels." arXiv preprint arXiv:2004.13649 (2020).


 Thank you for the detailed and insightful review.

> relationship with C-learning

While the *objective* for RCE is similar to C-learning, the actual *algorithm* is quite different. For example, RCE makes use of an additional dataset of success examples, and must reason about how these success examples are aligned with environment transitions. These algorithm differences reflect large differences in the theoretical derivations. The analysis of convergence and improvement guarantees are similar to C-learning.

> Line 135, $p_U(s_t∣e_t=1)$ occur before definition. Didn't align with Line 142.

> Algorithm 1, ... $s_{t+1}$ should be $s_t$?

Thank you for pointing out these two typo. We will fix these issues.

> Why is such baseline (e.g., C-learning) absent in the experiment setting?

C-learning solves a different problem than RCE and thus cannot be directly applied to this setting. Specifically, C-learning learns a goal-conditioned policy from a dataset of transitions, whereas RCE learns a single policy (not conditioned on goals) from a dataset of transitions and another dataset of success examples. Nonetheless, we initially tried to solve example-based control problems using a version of C-learning modified to use success examples. However, we found that this method performed quite poorly. For example, on the sawyer-push task, RCE achieves an average return of 0.2, whereas this variant of C-learning achieved a return of about 0.07 (65% worse). The poor performance of this variant of C-learning motivated us to design a new method from the ground-up, specifically designed to solve the example-based control problem. We will include a plot of these initial experiments in the final paper.

> only one imitation learning(SQIL) baseline is included.

We'd like to clarify that we also compared against *three* recent and competitive imitation learning methods: AIRL and DAC, in addition to SQIL. Our method outperforms all three imitation learning methods by a wide margin (please see Fig. 3).


 Thank you for the detailed and thorough review.

In this paper, the authors explore reinforcement learning solutions that train via examples for success states rather than an explicit reward function. The papers proposes an approach that is distinct from standard imitation learning - in that only success states are required rather than entire trajectories. The primary contribution of the paper is an off-policy RL algorithm that relies on predicting the success of future states. The paper addresses limitations of expert-given success states - and explores two formulations of estimating the relative effectiveness of two different states that solve the same task. The authors provide an analysis of convergence guarantees. Finally, experiments demonstrate that this approach beats SOTA methods on several enviroments.  This paper is well motivated - the problem of reward construction is a fundamental impediment in learning effective policies in RL settings. The approach is novel - the authors do a good job of differentiating this from similarly motivated solutions like inverse reinforcement learning and imitation learning. The recursive classification approach is a clever way to estimate future success states - and effectively eliminates the need for an explicit success label for each visited state. The experiments clearly demonstrate superior performance on a number of control tasks.  Yes

This paper asks an important and timely question: Can we make task-specification in RL “data-driven”? Rather than prescribing reward functions to incentive behavior in robotic agents,  RCE works by providing the agent examples of the solved task. The agent learns to classify whether or not the task will be solved in the future on a state-by-state basis in what they call "example-based" control.   The paper is well-motivated and is backed by sufficient theory. The approach can be dropped into existing algorithms, making it immediately useful for a large variety of practitioners. 

The algorithm is (perhaps boldly) positioned as a generalization of Goal-conditioned RL, but doesn't back this statement up strongly enough in the experimental section. In line 157, the authors claim that the algorithm performs well even when the examples come from an agent with different capabilities from the agent. This is quite relevant to the community and it seems worth linking to these experiments in this paragraph.  

I appreciate the related-work section, though I think It may be worth discussing work outside of traditional RL/imitation learning. How does this paper compare to research which re-distributes sparse reward, such as 
"RUDDER: Return Decomposition for Delayed Rewards" or planning paradigms that attempt to map values onto states from a terminal reward?

It is not clear to me how some of the tasks are not underspecified. For example, in "Sawyer-Push", the successful example is defined by the end-effector touching the puck. This seems like it would incentivize the agent to simply touch the puck, rather than push the puck towards the goal. Can the authors elucidate this example?

The experiments on "Example-Based Control from Images" experiments involve relatively simple tasks, making the results somewhat underwhelming. Overall, I would prefer a stronger experimental section in the body of the paper rather than the appendix. How were the experimental tasks chosen? If this narrow subset of tasks were chosen specifically for RCE's performance, can you describe which tasks the algorithm failed on and speculate as to why? 


Suggestions: 

- Results in this paper are evaluated via reward, however, simple task success/failure is a better-calibrated measurement for this paper. 

- Figure 2:  This figure takes up valuable paper space and would be better positioned in the appendix. 

- Figure 4: This plot is too crowded to be useful. The information may be better represented in a table or a combination of table/plot. Please indicate what these runs entail (avg of 5 seeds /  most performant agent / etc). 

Minor Comment: 

- Line 163: "The handle" should be "To handle"

Overall, this is an important and immediately useful paper that demonstrates a new way of thinking about learning agents. A stronger and more coherent experimental section will further improve a good paper.  yes

Meta-review: The reviewer's initially agreed upon the strong novelty of this paper's "data driven" task specification, which shares ideas with existing thoughts in imitation and goal-conditioning, but is both intuitively useful in practice and well supported by the paper's elements. Good points were raised by reviewers 3uQ8 and yauQ, in particular, about the relationship of the theory to existing proofs and of the most fair discussion of baseline tasks and comparisons. After a healthy exchange, the picture was quite clear that the paper's elements are nicely distinct and add important new ideas. 

I'm certain this paper will draw a healthy audience from offline-RL and imitation practitioners. In some domains where the author's task specification interface is indeed a much better fit, this work may have rather significant impact. For the overall RL/imitation community, it is another nicely done example of using a mix of existing and new tools to develop coherent theory and algorithms for a new spin on the problem. This is likely to be cited and to support others innovating in different ways in the space.

Decision: accept