Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning
Authors: Kun Huang
Edward S. Hu
Dinesh Jayaraman
Venue: CoRL 2022 Oral (2022)
Keywords: Reinforcement Learning
Interactive Perception
Task Specification

Abstract: Physical interactions can often help reveal information that is not readily apparent. For example, we may tug at a table leg to evaluate whether it is built well, or turn a water bottle upside down to check that it is watertight. We propose to train robots to acquire such interactive behaviors automatically, for the purpose of evaluating the result of an attempted robotic skill execution. These evaluations in turn serve as "interactive reward functions" (IRFs) for training reinforcement learning policies to perform the target skill, such as screwing the table leg tightly. In addition, even after task policies are fully trained, IRFs can serve as verification mechanisms that improve online task execution. For any given task, our IRFs can be conveniently trained using only examples of successful outcomes, and no further specification is needed to train the task policy thereafter. In our evaluations on door locking and weighted block stacking in simulation, and screw tightening on a real robot, IRFs enable large performance improvements, even outperforming baselines with access to demonstrations or carefully engineered rewards.

Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learning
Kun Huang Edward S. Hu
Dinesh Jayaraman
GRASP Lab, University of Pennsylvania {huangkun, hued, dineshj] upenn.
Abstract: Physical interactions can often help reveal information that is not readily apparent. For example, we may tug at a table leg to evaluate whether it is built well, turn water bottle upside down to check that is watertight. We propose to train robots to acquire such interactive behaviors automatically, for the purpose of evaluating the result of attempted robotic skill execution. These evaluations in turn serve as "interactive reward functions" (IRFs) for training reinforcement learning policies perform the target skill, such as screwing the table leg tightly. In addition, even after task policies are fully trained, IRFs can serve as verification mechanisms that improve online task execution. For any given task, our IRFs can be conveniently trained using only examples of successful outcomes, and no further specification is needed to train the task policy thereafter. In our evaluations on door locking and weighted block stacking in simulation, and screw tightening on real robot, IRFs enable large performance improvements, even outperforming baselines with access to demonstrations or carefully engineered rewards. Project website: https: /t///sitttss/ssee google. www/ooiiieeeeeeeeoooooe lirf-
1 Introduction
Consider a kitchen robot that must perform large number of tasks, such opening a refrigerator, cutting vegetables, tightening a water bottle lid, or flipping a pancake. How might this robot acquire these skills? Common skill acquisition approaches involve heavy engineering and expertise for each skill, directed either towards developing model-based control policies, or towards specifying dense reward functions for reinforcement learning (RL). These approaches do not scale well to the goal of acquiring large numbers of skills. Consequently, there growing interest scalable approaches for RL-based skill acquisition that permit easy task specification by non-experts.
particularly promising task specification framework that we call exemplar rewards, involves specifying tasks merely by showing the robot learner what the environment should look like after a well-executed skill [1, 2, 3, 4, 5, 6, 7]. For example, to specify the task of opening a refrigerator door, it would suffice to show the learner some images of open refrigerators. Then, once the skill is specified, the robot performs RL with the aim of altering the environment to generate observations similar to those examples. When this framework is successful, human user teaching the robot does not need technical expertise and does not need to even demonstrate full behaviors for task execution. Instead, they need only capture images of the outcome.
Although this framework is appealing, many tasks cannot be specified by image observations of task success, or any other fixed sensor setup. For example, consider the task of closing water bottle, as in Figure 1. What image might specify a successful execution of this skill? An image of water bottle with slightly loose cap would look identical to one with tight cap, but bottle closing policy that does not fully tighten the cap would be useless. Thus, in this setting, an observation of the task success state from a camera constitutes only partial observation and does not suffice to specify the task. Specifically, when non-success states might look visually similar to success states, image-based task specification and assessment fails. Figure 1 shows few examples of this state aliasing problem: the bottle cap tight? Is the door locked properly? Is the tower of objects with unknown masses stable? There are many more: Is screw tight? Is the leg of the table secured well? Does the smoothic have the right consistency? None these questions can be answered correctly from passive image observations.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Task: Close bottle cap
Task: Lock door
Task: Build stable tower
FUp
Pull
poke
Loose
Unlocked
Unstable
Flip
Pull
Pake
Door Locked?
Bottle cap Loose? Tight?
Block tower Stable?
Tight
Locked
Stable
Figure I: In each of these three settings, task success failure cannot be determined from passive image observations, but is easily revealed through appropriate interactions.
The failures of passive image-based assessment in these settings preclude robotic skill learning from image examples of task success. To fix this while retaining the versatility and user-friendliness of the exemplar rewards framework, we propose an interactive approach for evaluating task success. As motivation, consider how human might turn water bottle upside down confirm that it does not leak, and is therefore closed well. Here, the interaction reveals the key determinant of task success, namely, whether or not the cap is watertight.
We design an approach to automatically learn such interaction behaviors to evaluate task success. Our examples are no longer mere images, but instead real actionable physical instances. In the water bottle setting, rather than image snapshots of closed water bottles, our approach receives actual physical bottles with tightly closed caps. The robot can interact with these objects, such as by turning them upside down, to discover what about them constitutes successful execution of the desired "bottle closing" skill (i.e., the bottle does not leak) and how they behave differently from the outcomes of unsuccessful execution (i.e., it leaks). the other examples in Figure I, a robot might try to pull the door open nudge the tower to evaluate whether the task is complete.
We make two key contributions. First, we propose an approach to train interactive reward functions, which are policies that specify what it means to correctly perform a skill and thus drive the improve- ment of the task policy learners same reward policies can be reused to enable in-the-loop introspective verification behaviors during online skill execution, making robots more scrupulous and reliable. We evaluate these contributions on three simulated and real robotic control settings, and show substantial improvements in task policy learning and execution.
reinforcement learning framework. Next, we show how those
2
Background and Problem Setup
2.1 Classifier-Based Exemplar Rewards in fully observed MDPs
Reinforcement learning (RL) methods offer the promise of scalable data-driven synthesis of robot controllers for arbitrary new tasks. Consider robotic task settings formalized as Markov decision processes (MDPs), defined by the tuple A, R. p). At each time t, an agent selects an action a E in state 8 E S, transitions the next state s1+1 with probability T( St, at), and receives reward = R(st, ae, $t+1). Thus, the agent emits actions into the environment and receives two things return: new state observations and rewards. Ignoring discounting, good RL task policy Tr (aelas (8.) selects actions to maximize the sum of rewards over time E, It, starting state drawn from an initial state distribution μ.
Thus, it is the reward function R that specifies what task to perform in a given environment, and RL. approaches practice require expertise and effort for reward engineering [8] to specify each new task, avoiding misspecification and guiding elficient learning. This often takes the form of tuned weighted combinations many carefully constructed heuristic terms the RL reward objective [9, 10, 11, 12], or even privileged task-specific sensors added to the environment during training [13, 14]. To circumvent laborious manual reward design, many methods aim learn rewards from data. Inverse reinforcement learning methods [15, 16, 17, 18] learn task reward functions from optimal demonstrations, but such demonstrations are typically expensive and may even be impossible to obtain. Other methods train RI. agents by learning rewards explicitly or implicitly from interactive human feedhack [19, 20, 21, 22, 23), but these have the drawback of requiring in in-the-loop queryable human teachers.

---

We build upon the popular "exemplar rewards" framework, which explores task specifications purely through examples [1, 3, 4, 6. 7]. In classifier- exemplar rewards approaches [1, 3], the human teacher trains a classifier R(s) to label task outcomes as successes or failures, by gathering some success examples manually, and then generating failure examples from the task policy as it trains. Armed with this learned reward function R(s). the agent trains its task policy Ty (a(a)s) to generate success outcomes and thereby maximize the estimated rewards. An appealing property of this framework is that training a policy for each new task only requires specification of the final goal state that should be achieved, without any need to specify how to achieve it.
Our work extends this exemplar rewards framework: as motivated in Section 1, rather than relying on passive observations, we will train interactive evaluation policies to evaluate task success, and these evaluations will in turn serve as "interactive reward functions" for training task policies. This permits applying exemplar rewards to "partially observed" settings that do not permit accurate task success evaluation from images alone. We expand on the partial observability problem below.
2.2 Partial Observability, State Aliasing, and History-Based POMDP Policies
Why are classifier- exemplar rewards approaches not suited to settings such the door locking task in Figure (middle)? The root cause is "partial observability." To solve this task, the robot must push the door into the closed position and then rotate the latch into the locking position. It only sees images from a fixed camera in front of the door. The latch, however, is occluded, and the robot can only observe and act upon the four handles on the front of the door, which rotate the latch through an axle mechanism. The handles are visually identical, so any visual configuration of the handles corresponds to any one of four configurations of the latch behind the door. While this setting is constructed as a pedagogical example to expose and study the state aliasing problem, it is representative of several real-world settings, as discussed in Section and Figure 1.
Such settings are modeled as partially observable MDPs, or POMDPs [24]. At each time, the agent only partially observes the underlying latent task state s (i.e. the latch configuration), through an observation or ~ (()) st). where the observation function O generates samples from the observation space Ω (i.e, images of the handle configuration). The original task state 8, is not directly observable. Thus, the task POMDP is defined by A, T, 1,).
Several prior works deal with the question of how the task policy in a POMDP might map to good actions despite only seeing partial observations at each time. The key to this is the assumption, very often reasonable, that while individual observations are indeed incomplete, observation histories contain much more information about the state. Then, a POMDP policy can perform well by relying on a history of observations and actions, either direetly [25, 26, 27,28] Ty Dt-Hit Qt- -1---1 or by first estimating& a belief distribution (s, (o- Ot-H-1:t-----
over the state and then mapping to actions [24, 29, 30, 31, 32], as TT (a, (6)) In the door locking setting, the robot might observe the latch position before closing the door. Therefore, even if the latch is occluded after the door is closed, history- based policy could still track its position. In other words, if trained well with the right reward function, such history-basee based POMDP policies can still learn good task behaviors.
3 Exemplar Interactive Reward Functions
This brings us to the key question: how to specify reward functions R for training POMDP policies? In particular, could we extend the appealing exemplar rewards framework to POMDPs?
First, note that partial observability has to do with what policies can see, not where rewards come from. Therefore, it is feasible in theory to provide state-based exemplar rewards R(s) that are not limited by partial observability. This implies relying on additional sensors for reward estimation during training that are not available to the policy गт that is being trained. In other words, while the states 8 are no longer observable to nT, may still be possible to learn exemplar rewards R((+;) through privileged instrumentation of the environment during training to allow the reward function to observe the full state st. For example, privileged thermal cameras could measure fluid levels to train an RGB image-basee fluid pouring policy [13]. However, such additional instrumentation is labor-intensive and specifc,,,,,, and sacrifices the appealing scalability and ease of use of the exemplar rewards framework.
How might we do this? Learning exemplar rewards R((x)) based on the agent's instantaneous observations would suffer from state aliasing: in the door locking setting, such a reward function would not be able to distinguish correctly latched door from one in which the latch is off by factor
3

---

Initial RL Training
IRF Policy Training
LIRE Task Policy Training
Pask-POMDP
Task-POMDPP Gral: close door & larch
Goal:: close door & latch
IRF POMDP
Env
Actionable Examples
Negative
Actionable Examples
Posilive
IRF-POMDP
IRF Policy
Low
High
Inirial (Aliased)
Classification
IRF Reward
Reward
0,
Initial Task Policy
o P
dles
IRF
LIRE Task Policy
9
Policy
Figure 2: Method Overview. In partially observable environment, (left) we first learn an initial task policy ni using passive classifier-based rewards. (middle) Then we train an IRF policy an to distinguish between provided "actionable positive examples" and πリ- -rgnerred negative examples. (right) Finally, we use RR to provide the correct rewards for training LIRF task policy ят.
of 90 Nor does the exemplar reward learning framework permit relying observation histories, like the solution we described above for POMDP policies. To see this, recall that exemplar rewards are trained from samples of successful task outcomes rather than full demonstrations, so there is no observation-action history information available for training R(o Hit, Ot- -=1=-1)-
3.1 Actionable Examples and Interactive Reward Functions (IRFs)
Our primary contribution is a solution to this conundrum: rather than presenting successful outcome examples as singular pre-recorded observations, we propose to present them as "actionable examples". " with which the robot can interact and generate new observations. In other words, to learn a reward function for locking door, the robot gets access to several physical instances of successfully locked doors. Then, instead of training passive reward functions from image observations, we propose learning "interactive reward dowardffuncion functions" " (IRF), which consist of robot action policies TR that reveal the task rewards. See the schematic in Figure 2. We expand upon this idea below.
fot ..... xx1 } and "actionable failure examples" N for ok]. P and N contain object or environment configurations: in Figure 1, each o might be a tight bottle cap, well-locked door, a stable tower, and so on. Now, the IRF policy may be seen as the solution to a new "IRF POMDP". This identical to the task POMDP, except for two key differences. First, it is initialized to actionable example state reward function based on discovering the label of sp- was it drawn from or from N? All other elements AS S2, O) of the task POMDP are retained."
First, we gather "actionable success examples"
~ P U N. Next, its
Algorithm shows pseudocode for our "Learning from IRFs" (LIRF) approach. We point out three key features here. (1) First, we bootstrap the task policy by training against Lingle-observation based reward elassifier D(o,). This corresponds to running prior exemplar reward approaches; we use VICE [1]. (2) Next, in keeping with the exemplar rewards framework, we provide only the positive examples P. Negative examples N are instead generated by the above-initialized task policy. Thus, the IRF policy nn serves as a GAN-like adversarial critic [33, 18) for training the task policy my- (3) Finally, how could we discover the true task completion reward, i.e., the label (corresponding to positive reward) or the label N (negative reward) at the end of IRF execution? For this, we could, in theory, train classifier from the full IRF trajectory history. Instead, we find that it suffices to reuse the single-observation classifier (oo.) from (1) above, and apply it only to the final state after executing πR. Intuitively, RR learns to modify the environment state o TR, s" such that o* O(s*) reveals whether was drawn from P or N. For example, in door locking, गæ might learn to tug at the door: if the door correctly locked (P), it stays closed, and if not locked (N), it opens. Thus, single post-IRF observation suffices to classify a. Figure shows schematic.
In some experiments, we different actions A for the IRF policy an than for the task policy nr.

---

Algorithm 1 Learning from Interactive Reward Functions (LIRF) Framework
Require: set P of positive actionable examples that specify successful task exccution.
1: Following image-based exemplar rewards approaches such as VICE [1], train a single-observation reward
D(o) + [0. 1] and initial task policy x in the task POMDP.
2: Rollout тy for times, collecting negative actionable outcomes in N.
Train IRF policy aR in the IRF POMDP, where the environment initialized from state P, maximize the classification reward:
E P0EP
Σ log(D(o.))
+
E SpEN
Σ Loog(1-
4: Train the final LIRF task policy in the task POMDP, modificd to include the additional terminal state reward Ii(or) = D(or)= + AD(o"), where or is the observation for the terminal state sr, and o* is generated by the IRF policy as ST ER, s", o² -(s*). The final reward function uses the dense single-observation reward for intermediate states, and the terminal state reward for the final state.
D(or) + AD(o") ift =ft=1 T ((o.)
R((o.)
(2)
otherwise
3.2 IRFs as Verification Mechanisms for Introspective, Fault-Tolerant Behaviors
Next, observe that IRFs я rely on privileged information beyond the task policy Fy observations. Thus, while their primary purpose is to provide reward functions for training task policies, IRFs can also be deployed together with the task policies after training. We propose simple approach to use IRFs as verification mechanisms in-the-loop during task execution. Once the task policy गr is executed for a fixed episode length, we will run the IRF म to determine whether the task is complete. If it is detected as incomplete, we will resume executing the task policy. This perform-verify cycle continues until the IRF is satisfied that the task is complete (or timcout). Appendix A.1 contains pseudocode for this procedure.
4 Other Related Work
LIRF trains task policies through rewards computed by an interactive reward function, which is itself a robot policy trained from examples. While Section 2 extensively discussed connections reward design, exemplar rewards, and POMDP policy learning to set up our approach, we now position our contributions against two other key related areas of prior work.
Adversarial Robot RL. Like LIRF, some prior works train a target task policy against another "adversary" policy [34, 35, 36, 37]. Pinto al. [37] train grasping robot to select more stable grasps by competing against an adversary robot that tugs at the grasped objects to dislodge them. Rather than adversary policies that destabilize the task policy increase robustness, we train adversary policies that provide outcome-basse rewards to train the task policy.
Interactive Perception and Verification. Our method draws ideas from interactive percep- tion [38], where agents in partially observed settings act to obtain information about the latent state. Interactive task verification mechanisms (see [39] for survey) employ human-specified interactive perception behaviors to assess the state of a task. For example, a robot may use engineered motion primitives or perturbations detect and correct task failures in manipulation [40, 41]. In contrast, we learn interactive verification behaviors from examples, and use them not only during task exe- cution, but also define the reward function for training task skill policies. One way frame our contribution here is as follows: in reinforcement learning, agents emit actions into the environment and receive in return not just new states, but also task rewards. To our knowledge, LIRF the first approach that employs interactive perception to directly estimate task rewards, rather than the state.
5 Experiments
We design various simulated and real-world experiments to evaluate LIRF for example- based task specification partially observable tasks that manifest the state aliasing issue. We aim to answer the following questions: (1) Does our interactive reward learning framework enable us to learn policies to solve partially observable tasks that alternative approaches cannot, with comparable or even greater task supervision? (2) Does the learned IRF policy sensibly evaluate task completion and provide
5

---

Door Locking
Weighted Block Stacking
Real-robot Screwing
Figure 3: Our three tasks, For each pair of images, we show the task initialization on the left, and the goal state on the right. In the middle panel, the 3 blocks are visually identical but they differ in weight. The heaviest block green for visualization. In the right panel, we put a blue mark on the valve for visualizing the orientation.
good rewards? (3) Can using the IRF policy in-the-loop for task verification improve task execution performance?
Task Setups. We evaluate LIRF on 3 tasks, illustrated in Figure 3, and described below. See Appendix A.2 for more details and the project website for experiment videos.
Door Locking (Sim): We instantiate the door locking task from sections 2.2 and with an Adroit Hand using MuJoCo [42]. The robot has a fixed viewpoint in front of the door, and must close the door and rotate the symmetric, four-handled doorknob to fully lock it (see Figure 3). The observation space mimics perfect vision: it consists of the door orientation. the door frame position, the aliased cross-hane
orientation [0, 909)² and the Adroit hand pose. The door is initialized as open, and the position of the door frame and the orientation of the latch are randomized.
weighted block stacking task based on the MuJoCo Fetch environment [43] as shown in Figure 3. The goal of this task is to stack 3 visually identical blocks into "stable" tower. One block significantly heavier than the other two blocks, so the optimal strategy is to place it at the bottom. Again, the observation space mimics perfect vision: only the unordered poses of blocks observable. The weight of a block is only revealed after it is picked up, to mimic physically plausible "hefting" behavior.
Weighted Block Stacking (Sim): We further test our algorithm
Screw Tightening (Real): To examine the robustness and generality of our algorithm, we test it on a real-robot screwing experiment using a D' Claw [44] that has joints (Figure 3). The objective of this task is to turn the 4-prong valve clock-wise for around 180 into the "tightened" state (white line on the valve base). We engage a motor underneath the valve to mimic screw locking. Again, the observation consists of the historical aliased valve angles, i.e. [0, (0900) and robot joint angles.
LIRF Implementation Details. We use the soft actor-critic (SAC) algorithm [45] to train both the task policies and the IRF policies for each experiment and report results with mean and variance over 5 seeds for each simulated task. The weight À for the sparse IRF reward in Eq 1 needs to be "large enough" (Appendix A.6) to have substantial effect on LIRF training, but beyond this, the algorithm is not very sensitive to X; we set À = 1000. All policies are trained until convergence, which takes around 2M simulation steps for door locking and block stacking LIRF policies, 50k steps for screwing LIRF policies (~ 10 hours on the real robot), and around 20k steps for all IRF policies. More details in Appendix A.5. We will release all code and environments.
Baselines. For evaluating our LIRF algorithm, we compare against RL policies learned with VICE exemplar rewards [1] and with ground truth state-based rewards ("GT State Reward"). In addition, we compare against GAIFO [46], an taaatioonn-ooooos-vvtttiii algorithm that learns from demonstrations. We provide as many demonstrations to GAIfO as the number of episodes with positive examples provided LIRF (namely, 250, 10k, and 100 for door locking, weighted block stacking, and screwing respectively). Note that LIRF actually uses fewer positive examples than the number of positive episodes, (110, 7K, and I actionable positive examples respectively) since the same example can be reused across episodes if it is not destroyed. Finally, when feasible, we engineer an interactive reward function policy (Manual IRF) that performs hand-coded actions to evaluate the task: hand-coded unscrewing behavior for screwing, and random poking action for the block
The cross- handle is symmetrical and looks visually identical offoffeets e.g. 10", 100 9, 190 280

---

Task Specification
Door Locking
Block Stacking
Screwing
VICE [1]
Goal Examples
0.032 + 0.015
0.310 + 0.056
0.02
LIRF (Ours)
Actionable Goal Examples
0,640 + 0,053
0.346 = 0.027
0.82
LiRF+Verity (Ours)
Actionable Goal Examples
0.958 + 0.007
0.884 = 0.018
0.99
GAIrO [46]
Demonstrations
0.112 + 0.024
0.275 + 0.043
0.0
Manual IRF
Human Engineering
0.613 + 0.068
0.85
GT State Reward
State+Human Engincering
0.714 + 0.047
0.956 + 0.021
0.87
Table 1: Task success rates of our method and baselines on the three tasks.
VICE lask Policy
Policy
Door Locking
Screwing Stacking
Figure 4: LIRF task policy rollouts (right), compared to VICE (left). Note row 2, where VICE builds an unstable tower with the heavy block on top (visualized in green here, but this not observable to the agent). LIRF self-corrects mid-episode to build the tower with heavy block on bottom. See Supp slides for more video examples.
stacking. To isolate the effect of reward functions, all methods share the same input observation space for their policies. See Appendix A.4 for more details the baselines. Note that GAIfO, GT State Reward, and Manual IRF all involve more painstaking and comprehensive task specifications than LIRF, but offer useful comparison points nevertheless.
5.1 Results
Does LIRF succeed in partially observable settings? Table shows task success rates for our approach and baselines. LIRF outperforms VICE across all tasks, showing that LIRF successfully extends the exemplar reward framework into partially observable settings where prior exemplar rewards methods like VICE fail. LIRF also performs par better than baselines that use additional / more expensive forms of task supervision, such as demonstrations, human engineering, and privileged sensing of ground truth state.
Qualitatively, LIRF policies show good task-solving behaviors for each task they learn to lock the door, construct towers with the heaviest block the bottom, and fully tighten the screw (see Figure 4). LIRF even demonstrates sophisticated self-correction behavior in the block stacking task by rebuilding the tower if the heavier block is incorrectly stacked top of lighter block.
Among baselines, VICE and GAIfO perform poorly due to state aliasing, and LIRF competitive or superior to Manual IRF, showing that learning an IRF policy with RL may yield better rewards for training partially observable task policies.
Do the IRF policies RR sensibly evaluate task completion? First, we evaluate the success rate of the IRF policy TH at distin- guishing between positive and negative examples for each task in Table IRF-based task success classification is highly accurate for all tasks. Figure 6 shows examples of learned reward function policies. The IRF policies learn sensible strategies to separate visually indistinguishable positives and negatives. In the door locking task, IRF pulls the fixed handle; in stacking, pokes
Task
Accuracy
Door Locking
0.992 0.006
Block Stacking
0.852 1 0.023
Screwing
0.98
Table 2: IRF task success classifi- cation accuracies.
For Adroit hand door locking, the action space too large to reasonably hand-code any IRF behavior.
7

---

Door Locking
Weighted Black Stacking
Real-robot Screwing
IRFS Evaluating Outcomes
Task Faalures
Task Successes
Figure 6: Running the IRF policy TR on the success and failure actionable examples. Left (Door Locking): nn grasps the door and pulls to verify locking. Middle (Block Stacking): n checks for tower stability by gently poking the bottom block. Right (Screwing): gH attempts to lightly unscrew the screw in the top row rotates as it is not tight, whereas the bottom-row does not.
the bottom block; in screwing, it applies small counterclockwise torque to unscrew the valve. Finally, IRF policy learning is quite efficient in terms of the number actionable positive examples required. Door locking, screwing, and weighted block stacking need 1 110, 1, and 7k actionable positive examples respectively (note that the same positive example can be reused for multiple IRF training episodes, unless its "positiveness" is destroyed at the end of that episode). Block stacking is the hardest, since distinguishing stable from unstable towers requires finely tuned poking force and motion, and even "stable" block tower examples are prone to toppling quite easily. See Appendix A.7 for more details.
Can the IRF policy be used for in-the-loop verification? As seen in Table 1, J.rifVerrfy, which reuses the learned IRF policy during task execution, outperforms all baselines, even beating the ground truth reward baseline in door locking and screwing. Figure 5 shows how LIRI+Verify improves per- formance with more perform-verify iterations. LIRF+Verify makes the biggest difference in door locking: here, plain LIRF fails mainly because the policies operating on partial observa- tions do not learn precisely how much to turn the latch without the aid of n to declare task completion. This is also why GT State Reward performs quite poorly on this task despite training on the true state-based reward function (see also Appendix A.8). LIRF+Verify's gains over LIRF are smaller in block stacking and screwing, because LIRF failures here occur from low-level imprecision in pick-and-place valve-turning. Here, LIRF+ Verify helps by allowing the task policy πr to try again when it fails. We show video examples on the project website.
1.0
Success Rates
0.9
0.8
Door Locking
0.7
Weighted Block Stacking
Real DClaw Screwing
0.6
8
Number of Perform-Verify Iterations
Figure 5: LIRF+Verify success rates vs. perform-verify iterations.
6 Conclusions
We have presented LIRF, a framework for conveniently training example-based interactive robot policies to evaluate robot task policies, in order both to provide rewards to train them, and to verify their execution in partially observed tasks. While our results show substantial improvements over baselines, LIRF currently has two drawbacks. LIRF is best applied to settings where the initial task policies trained from single-image- based rewards fail most of the time. When this is not the case, such as in fully observable settings, IRF policies yield marginal or benefits over the initial policies (Appendix A.9), and therefore may not be the most frugal algorithmic choice. Next, LIRF requires the physical storage of "actionable outcomes" as positive examples. While the number of such examples in some tasks may be small enough to not be a major concern (e.g., among our experiments. I for screwing. 110 for door locking), in other cases. storing and presenting large number of positive example objects to train the IRF might be cumbersome (e.g., 7k for block stacking). We discuss scalability more detail in Appendix A. 10. Future work overcoming these limitations could further expand the domain of applicability of LIRF.
8

---

7 Acknowledgement
This work was supported by the U.S. Office of Naval Research under grant ONR N00014-22-1-2677, and gift from NEC Laboratories America.
References
[1] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018.
[2] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning. pages 40-52. PMLR, 2018.
[3] A. Singh, L.. Yang. C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering. In Robotics: Science and Systems, 2019.
[4] B. Eysenbach, S. Levine, and R. R. Salakhutdinov. Replacing rewards with examples: Example- based policy search via recursive classification. Advances in Neural Information Processing Systems, 34, 2021.
[5] K. Konyushkova, Zolna, Y. Aytar, A. Novikov, Reed, Cabi, and de Freitas Semi- supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899, 2020.
[6] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed. Offline learning from demonstrations and unlabeled experience. arXiv preprint 1xXiv:011.138855 2020.
[7] Y. J. Ma, A. Shen, D. Jayaraman, and O. Bastani. Versatile offline imitation learning via state- occupancy matching. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
[8] S. Singh, R. L.. Lewis, and G. Barto. Where do rewards come from. In Proceedings of the annual conference of the cognitive science society, pages 2601-2606. Cognitive Science Society, 2009.
[9] A. Molchanov, Т. Chen, W. Hönig, J. A. Preiss, N. Ayanian, and G. S. Sukhatme, Sim-to- Sim-to-(multi)- real: Transfer of low-level robust control policies to multiple quadrotors. 20/9 IEEE/RS. International Conference on Intelligent Robots and Systems (IROS), pages 59.66. IEEE, 2019.
[10] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91-100, PMLR, 2022.
[11] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guide deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
[12] OpenAI. Gym: toolkit for developing and comparing reinforcement learning algorithms. /ttps://githuj www/opena//oggg///eeeeeeeeeee /gym
[13] C. Schenck and D. Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 2629-2636. IEEE, 2017.
[14] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 79-86. IEEE, 2017.
[15] A. Y. Ng. Russell, al. Algorithms for inverse reinforcement learning. In leml, volume 1, page 2, 2000.

---

[16] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586-2591, 2007.
[17] S. Arora and P. Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
[18] Ho and Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29:4565-45777 2016.
[19] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9 16, 2009.
[20] V. Veeriah, M. Pilarski, and S. Sutton. Face valuing: Training user interfaces with facial expressions and reinforcement learning. arXiv preprint
2016.
[21] S. Chernova and A. L. Thomaz. Robot learning from human teachers. Synthesis Lectures Artificial Intelligence and Machine Learning, 863):-1211, 2014.
[22] P. F. Christiano, J. Leike, T. Brown, M. Martic. S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[23] J. Lin, Z. Ma, R. Gomez, K. Nakamura, B. He, and G. Li. A review interactive reinforcement learning from human social feedback. IEEE Access, 8: 8:120757-120765, 2020.
[24] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-11) 1344 1998.
[25] R. A. McCallum. Overcoming incomplete perception with utile distinction memory. 1993.
[26] M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015.
[27] Zhu, Li, Poupart, and G. Miao. On improving deep reinforcement learning for pomdps. arXiv preprint arXiv:1704.07978, 2017.
[28] T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free strong baseline for many pomdps. arXiv preprint urXiv:2/10.05000 2021.
[29] K. Gregor, D. Jimenez Rezende, F. Besse, Y. Wu, H. Merzic, and A. van den Oord. Shaping belief states with generative environment models for rl. Advances in Neural Information Processing Systems, 32:11445-13487, 2019.
[30] T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in pomdps. In Uncertainty in Artificial Intelligence, pages 1061-1071. PMLR, 2020.
[31] G. Weisz, P. Budzianowski, P.-H. Su, and M. Gašić. Sample efficient deep reinforcement learning for dialogue systems with large action spaces. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(11):0083-097 2018.
[32] M. Igl, L. Zintgraf, T. A. Le, F. Wood, and S. Whiteson. Deep variational reinforcement learning for POMDPs. June 2018.
[33] I. Goodfellow, J. Abadee, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
[34] I.. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817-2826. PMLR, 2017.
[35] X. Pan, D. Seita, Y. Gao, and J. Canny. Risk averse robust adversarial reinforcement learning. 2019 International Conference on Robotics and Automation (ICRA), pages 8522-8528. IEEF, 2019.
10

---

[36] X. Ma, K. Driggs-Campbell and M. J. Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1665-1671. IEFE, 2018.
[37] L. Pinto, J. Davidson, and A. Gupta. Supervision via competition: Robot adversaries for learning tasks. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1601-1608. IEEE, 2017.
[38] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme. Inter- active perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):1273-1291, 2017.
[39] O. Kroemer, S. Nickum, and G. D. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of machine learning research, 22(30), 2021.
[40] Z. Su, Kroemer, G. E. Loeb, G. Sukhatme, and S. Schaal, Learning manipulation graphs from demonstrations using multimodal sensory signals. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2758-2765, IEEE, 2018.
[41] S. Nickum, S. Osentoski, C. G. Atkeson, and A. G. Barto. Online bayesian changepoint detection for articulated motion models. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 1468- 1468-1475. IEEE, 2015.
[42] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, Schulman, E. Todorov, and S. Levine Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: 17Xiv:170..1000 01777099000077 2017.
[43] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint 1606.0/ 540, 2016.
[44] M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conference on robot learning, pages 1300-1313. PMLR, 2020.
[45] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[46] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint 1xXiv:8807.06/555 2018.
[47] M. Zaheer, Kottur, Ravanbakhsh, B. Poezos, R. R. Salakhutdinov, and A. Smola. Deep sets. Advances in neural information processing systems, 30, 2017.


Reviews:
[{"review_id": "lR-7R8cA41-", "reviewer": "Reviewer_FXQi", "summary": "This paper introduces a new framework for interactive reward functions (IRF), which are a variant of exemplar rewards for scenarios where the relevant part of the state is not directly observable and needs to be verified through interaction. Each IRF is a separate policy trained with the goal of being able to discriminate positive demonstrations and negative examples from an untrained task policy. To train the task policy, after each task policy rollout, the IRF is executed to check the resulting state for task success. The framework is evaluated on three interesting showcases (door locking, block stacking, real-robot screwing) where clear advantages over vanilla exemplar rewards are shown. The framework is valuable for many real-life tasks where the task success is not camera observable.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The justification and ideas of the paper are presented very clearly and in excellent writing. The experiments do a great job of showcasing the kind of task that such a framework is useful for.\n\nThe main contribution of this paper is this idea that the reward function be computed over an observation not from the end of the task policy rollout but instead one obtained by additionally using a separate policy. Indeed, for the simulation-based cases, it would be possible to directly train a success/failure discriminator on the (latent) state of the simulation at the end of the task policy, not requiring an additional reward function policy. It is only valuable for the (more important) real-world RL case where such information is not accessible. This is not immediately obvious since much of the experiments focus on simulation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper does a great job of setting up this framework of interactive reward functions, describing how to train and use them, and showing experimentally their benefits over exemplar rewards. The setup is very useful for many real-world tasks since the partial observability situation applies very often.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I do not have any other comments on any issues with this paper and congratulate the authors for a paper very well written.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "aOr2Q3sfQgB", "reviewer": "Reviewer_T7V5", "summary": "The paper proposes a framework for policy learning in a partially observable setting. In this setting, the observation itself does not contain full information about the Markovian state, and hence, does not have sufficient information to determine the ground truth reward or task success. To approach this, the paper proposes to train another policy, called the IRF policy, to perturb the final state in a way such that task success is transferred to the observation space. Therefore, this information can then be used as a reward function to train the task policy, or to verify whether the task is indeed achieved. \n\nThe proposed framework is tested on 2 simulated tasks and 1 real robot task. In all cases, the proposed framework is shown to significantly outperform baselines that 1) only uses observation for exemplar rewards and 2) directly imitate the demonstrations. However, it seems like the proposed framework is only evaluated on tasks that VICE achieves a low success rate. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **Organization**: The paper is very well written and organized. The motivation of the paper is clearly stated in the introduction. The authors use concrete examples, both from real life and the actual experiments to illustrate the intuition behind the proposed idea. The proposed algorithm is well-stated and easy to follow. The experiment section provides clear description on each of the tasks, and detailed information about their setup. \n- **Literature survey**: The authors clearly stated the limitation of existing work in exemplar rewards: They can not be directly applied to the partially observable setting since the task may falsely appear to have succeeded. The authors also discuss how their work differs from existing literature in Adversarial Robot RL and Interactive Perception and Veriﬁcation.\n- **Technical approach**: The technical approach is well motivated and is built upon a sound intuition. The explanation of the algorithm is thorough and clear. \n- **Experiment validation**:  The proposed framework is tested on 2 simulated tasks and 1 real robot task. In all cases, the proposed framework is shown to significantly outperform baselines that 1) only uses observation for exemplar rewards and 2) directly imitate the demonstrations. The authors provided thorough information on the tasks, and how the policies are trained and evaluated. The experiments verified that the proposed framework is indeed critical in the partially observable setting.  \n- **Final discussions**: It is appreciated that the authors discussed some of the limitations of their work in a fair way in the conclusion session. \n\n### Weaknesses\n- **Assumptions on VICE performance**: As noted by the authors, the proposed approach assumes that the task policy initially performs poorly on the task. (If I understood it correctly, all the examples generated by VICE are considered negative, is that right?). It seems like a limitation since VICE policy may already be quite good in some tasks, e.g. if we only need to close the door. Can the authors provide more information on what will happen in this case? Does the performance of the task policy degrade after LIRF, or does it just stay roughly the same? Maybe the authors can, in the appendix, provide an experiment where VICE is sufficient to solve the task, and show what happens after applying LIRF. To follow up, does it imply that LIRF will fail to improve (or even degrade) performance if the initial task policy success rate is high enough? If so, how do we determine whether we should use LIRF or just use the initial VICE policy? It seems hard since the paper assumes no access to the ground truth reward function.\n\n- **Only training IRF once**: The authors noted that the previous limitation is now preventing iterative training of IRL policies, which makes sense. However, I am curious whether this current framework can encourage the task policy to fail at tasks in a “creative” way that can not be caught by the IRF policy. Since the IRF policy is only trained once with the VICE policy, it will not be able to adapt to other failure modes that the updated task policy induces. Although it seems like it is not the case in the experiments shown in the paper, I am wondering if the authors have observed such a phenomenon at all, and if the authors have any intuition on what to do in such cases. Minor point: Since the IRF policy is only trained once, it would be a little misleading to call it “GAN-like adversarial critic”.\n\n- **Size of N**: What is the size of $N$ for the current experiments?\n- **Choice of hyperparameter**: How sensitive is LIRF to the choice of $\\lambda$? \n- **Block stacking**: From the quantitative results, as well as the number of positive examples needed, it seems like the block stacking experiment is most challenging. However, according to the appendix, the authors already made various attempts to simplify it, such as using motion primitives, and provide weight information once a block is picked up. Can the authors provide more explanation on why this experiment is still so challenging?\n- **Screwing task success**: Can the authors provide more information on how the task success rate is evaluated in the screwing experiment? It is not very clear what does “the valve reaching the tightened state” exactly mean. \n- **Typo?**: “The valve can rotate freely without limits when the motor is not engaged, whereas it is locked in place or controllable through the motor when the motor is not engaged.” Should it be “the motor is engaged” near the end?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper addresses a well-motivated problem of policy training in a partially observable setting. The paper is very well written and organized. The motivation of the paper is clearly stated in the introduction. The proposed algorithm is well-stated and easy to follow. The literature survey is thorough, The authors clearly stated how their work differs from existing work in the literature. The experiment section provides clear description on each of the tasks, and detailed information about their setup.  The authors also addressed clearly the limitations of their approach. \n\nThe major limitation, however, is that the current setup only seems to work when the initial task policy has poor results (see the first point in **Weaknesses**). Based on the assumptions in this paper, it is also unclear how to evaluate the initial task policy, as the ground truth state and hence reward is assumed unknown. The paper would have significantly greater impact if this limitation is addressed. \n\n**Updates after the rebuttal period.** The additional results from the authors show that 1) LIRF will not degrade performance if VICE is already sufficient for solving the task; 2) multiple iterations of LIRF also does not degrade performance. This clears up my major concern. With that, I am glad to raise my rating from weak accept to strong accept. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See **Strengths and Weaknesses**. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "QGg0PG7X5wM", "reviewer": "Reviewer_BeWt", "summary": "This paper presents a method for extending exemplar rewards, where reward is inferred from success examples, to the partially observable setting. Existing methods don't extend well since they match the current observation to a success observation, but the observation may not be sufficient for determining success. \n\nThe paper proposes LIRFs. A task policy is pretrained with existing exemplar reward methods, then rolled out to get negative (but reachable) outcomes. Then, an IRF policy is trained to start in either an exemplar or a negative outcome and perturb the environment until it can label success/failure. This RL approach, rather than a classification approach, is said to avoid use of full trajectories.\n\nEvaluation includes three tasks/action spaces and SAC for both policies. LIRF beats full-obs IRF methods deployed in PO case and loses only to methods with privileged info. Using LIRF during task execution (not just as reward) to check success beats all baselines.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- Paper is clear and well written \n- The proposed approach is creative. The choice to use RL for pi_R is nonobvious, so it's satisfying that it provides clear value in terms of efficiency. \n- Evaluation has been done fairly thoroughly, and the baseline choices are thoughtful and probably complete for the problem at hand even if not very extensive (I'm not personally familiar with other state of the art exemplar reward methods anyway). \n\n**Weaknesses**: \n- It may be worth comparing briefly to a broader range of methods, to give a good argument for exemplar rewards as a whole (the metric doesn't have to be accuracy, it can be data cost for example). However, I would defer to other reviewers that have read more papers in the particular space than I have. \n- The paper is a great read, but section 5 becomes a bit dry. The paragraph titles are promising - it's very pedagogical to offer readers intuition for each module and I appreciate that that theme holds throughout the paper. However, we don't need such a play-by-play as we're given in 5.1 - the second and third paragraph can be one brief paragraph.\n- Following on that, it would be helpful to fill the saved space with more analysis of the results. Right now the paper seems to argue that LIRFs exist in a category of their own, and I buy that, but the result analysis is a bit dry and laundry list-like, and could be more claims-driven. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper hits all the bases it needs to. I would like more insight in analysis, but it's a solid, successful, and (to my knowledge) novel work. I don't see a reason not to accept it.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- As I said in \"weaknesses\", the presentation in section 5 should move from 1) lots of space devoted to intuition and 2) laundry lists of facts to 1) most of that space devoted to analysis (even ablations) and comparison, and 2) claims-driven writing"}}, {"review_id": "N9JoONu2jDw", "reviewer": "Reviewer_2X1d", "summary": "This paper introduces Learning from Interaction Reward Functions (LIRF), which addresses the problem of how to learn control policies under state aliasing, that is when success states look similar to failure states (e.g. a door locking problem). LIRF trains both a control policy to solve the task and an interactive reward function (another policy) to verify if the task has been completed correctly.\n\nThe main contributions of this work are\n* The LIRF algorithm. Specifically, the idea to learn a verification policy from demonstrations.\n* Experiments in simulation and on a physical robot which demonstrate how LIRF can be applied to solve 3 different robot learning tasks\n* Experiments that show that IRFs can be reused during task execution to improve task performance\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is very well written. It is lucid with a good selection of detail for the main paper. The introduction and motivation are especially clear.\n* The paper addresses a relevant problem in robot learning — how to successfully train policies when success states look visually similar to failure states, i.e. when there is ambiguity due to state aliasing\n* The idea of interactive reward functions is interesting and the idea of switching between a learned control policy and a learned verification policy is to my knowledge quite novel\n* The experimental results are mostly convincing. I do have a couple of questions which I include in the following section. Additional comments.\n   * LIRF is tested on two tasks in simulation and 1 task on a physical robot. The baselines appear to be fairly comprehensive and well chosen to shed light on different aspects of the problem\n   * LIRF outperforms the main baseline VICE in all tasks. Since LIRF is VICE + interactive reward functions (IRF) I also appreciated how this comparison highlights the effect of IRF.\n   * LIRF also outperforms Manual IRF in the simulated settings, indicating that learning IRFs has value (instead of manually defining them)\n* I especially appreciate the efforts to demonstrate results on a physical robotic system and the use of a motor to “lock” the screw is nice\n* The limitations section is substantive.\n\nWeaknesses\n* Experimental results\n   * Was the GT State Reward permitted retries? I’m curious what the results would be if this policy has the same potential number of retries\n   * The heuristically chosen λ value of 1000 seems quite high. I’m curious if the authors can give any intuition as to why it is so high? It would also be interesting to see an ablation over the value of λ. Further, given the weight is so high, I’m curious if the other reward component D(oT) can be removed altogether?\n      * Relatedly, I’m curious if the original classifier reward is sparse (the IRF reward is sparse, lines 228-229)? If it was dense, perhaps this explains why it is important to include it even with a relatively low weight.\n   * Lines 120-121 “if trained well with the right reward function, such history-based POMDP policies can still learn good task behaviors.” It would have been interesting to see how GT State Reward + a history-based policy performed on this task. Since all the tested policies are MDPs, including a GT State Reward + a history-based policy would also clearly show the effect of observation history compared to a perfect reward function in this case.\n* LIRF seems very similar to inverse reinforcement learning (IRL) however IRL is not mentioned in the paper. Whilst LIRF learns a policy for a reward function for just a subset of the trajectory (the terminal state) and IRL learns it for all states, the two approaches appear to have essentially the same formulation. In both cases a reward function is learned from demonstrations which is in turn used to train an RL policy. I would have liked to see a discussion comparing the two in the related work.\n* I wonder how well this approach would scale. For example, the 10k demonstrations required to learn the IRF for the block stacking tasks seems costly to replicate for other similarly complex tasks.\n   * One advantage of exemplar rewards is that just a single image is required instead of a demonstration trajectory. LIRF doesn’t have this benefit since it requires verification trajectories. In some cases these may require a similar amount of effort to full demonstrations.\n* I do appreciate the substantial limitations section. However, related to the above it does seem that the time required to collect (and not just store) the positive actionable examples could be a limitation (e.g. if 10k per task is often required). I would have liked to see a discussion about this in the work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an interesting and novel approach to a relevant problem in robot learning, solving tasks when there is state aliasing. It is well written throughout and the experiments in simulation and the real world demonstrate the effectiveness of the proposed method LIRF. I do think a comparison with IRL is missing and I have concerns about how scalable the method is given the requirement to gather potentially thousands of verification trajectories. However, on balance the strengths outweigh the weaknesses.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* Please clarify if the other reward component, D(oT), is sparse\n* Please give some intuition about why λ is so large\n* It would be helpful if the authors could share their thoughts about how this approach might scale.\n* Please include a discussion comparing LIRF with on inverse reinforcement learning in the related work\n* Typo: line 226: experiment → experiments\n"}}]

Author rebuttal:

Decision: Accept (oral)