Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method for extending exemplar rewards, where reward is inferred from success examples, to the partially observable setting. Existing methods don't extend well since they match the current observation to a success observation, but the observation may not be sufficient for determining success. 

The paper proposes LIRFs. A task policy is pretrained with existing exemplar reward methods, then rolled out to get negative (but reachable) outcomes. Then, an IRF policy is trained to start in either an exemplar or a negative outcome and perturb the environment until it can label success/failure. This RL approach, rather than a classification approach, is said to avoid use of full trajectories.

Evaluation includes three tasks/action spaces and SAC for both policies. LIRF beats full-obs IRF methods deployed in PO case and loses only to methods with privileged info. Using LIRF during task execution (not just as reward) to check success beats all baselines.


strengths_and_weaknesses: **Strengths**: 
- Paper is clear and well written 
- The proposed approach is creative. The choice to use RL for pi_R is nonobvious, so it's satisfying that it provides clear value in terms of efficiency. 
- Evaluation has been done fairly thoroughly, and the baseline choices are thoughtful and probably complete for the problem at hand even if not very extensive (I'm not personally familiar with other state of the art exemplar reward methods anyway). 

**Weaknesses**: 
- It may be worth comparing briefly to a broader range of methods, to give a good argument for exemplar rewards as a whole (the metric doesn't have to be accuracy, it can be data cost for example). However, I would defer to other reviewers that have read more papers in the particular space than I have. 
- The paper is a great read, but section 5 becomes a bit dry. The paragraph titles are promising - it's very pedagogical to offer readers intuition for each module and I appreciate that that theme holds throughout the paper. However, we don't need such a play-by-play as we're given in 5.1 - the second and third paragraph can be one brief paragraph.
- Following on that, it would be helpful to fill the saved space with more analysis of the results. Right now the paper seems to argue that LIRFs exist in a category of their own, and I buy that, but the result analysis is a bit dry and laundry list-like, and could be more claims-driven. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think this paper hits all the bases it needs to. I would like more insight in analysis, but it's a solid, successful, and (to my knowledge) novel work. I don't see a reason not to accept it.
issues: - As I said in "weaknesses", the presentation in section 5 should move from 1) lots of space devoted to intuition and 2) laundry lists of facts to 1) most of that space devoted to analysis (even ablations) and comparison, and 2) claims-driven writing
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces Learning from Interaction Reward Functions (LIRF), which addresses the problem of how to learn control policies under state aliasing, that is when success states look similar to failure states (e.g. a door locking problem). LIRF trains both a control policy to solve the task and an interactive reward function (another policy) to verify if the task has been completed correctly.

The main contributions of this work are
* The LIRF algorithm. Specifically, the idea to learn a verification policy from demonstrations.
* Experiments in simulation and on a physical robot which demonstrate how LIRF can be applied to solve 3 different robot learning tasks
* Experiments that show that IRFs can be reused during task execution to improve task performance

strengths_and_weaknesses: Strengths
* The paper is very well written. It is lucid with a good selection of detail for the main paper. The introduction and motivation are especially clear.
* The paper addresses a relevant problem in robot learning — how to successfully train policies when success states look visually similar to failure states, i.e. when there is ambiguity due to state aliasing
* The idea of interactive reward functions is interesting and the idea of switching between a learned control policy and a learned verification policy is to my knowledge quite novel
* The experimental results are mostly convincing. I do have a couple of questions which I include in the following section. Additional comments.
   * LIRF is tested on two tasks in simulation and 1 task on a physical robot. The baselines appear to be fairly comprehensive and well chosen to shed light on different aspects of the problem
   * LIRF outperforms the main baseline VICE in all tasks. Since LIRF is VICE + interactive reward functions (IRF) I also appreciated how this comparison highlights the effect of IRF.
   * LIRF also outperforms Manual IRF in the simulated settings, indicating that learning IRFs has value (instead of manually defining them)
* I especially appreciate the efforts to demonstrate results on a physical robotic system and the use of a motor to “lock” the screw is nice
* The limitations section is substantive.

Weaknesses
* Experimental results
   * Was the GT State Reward permitted retries? I’m curious what the results would be if this policy has the same potential number of retries
   * The heuristically chosen λ value of 1000 seems quite high. I’m curious if the authors can give any intuition as to why it is so high? It would also be interesting to see an ablation over the value of λ. Further, given the weight is so high, I’m curious if the other reward component D(oT) can be removed altogether?
      * Relatedly, I’m curious if the original classifier reward is sparse (the IRF reward is sparse, lines 228-229)? If it was dense, perhaps this explains why it is important to include it even with a relatively low weight.
   * Lines 120-121 “if trained well with the right reward function, such history-based POMDP policies can still learn good task behaviors.” It would have been interesting to see how GT State Reward + a history-based policy performed on this task. Since all the tested policies are MDPs, including a GT State Reward + a history-based policy would also clearly show the effect of observation history compared to a perfect reward function in this case.
* LIRF seems very similar to inverse reinforcement learning (IRL) however IRL is not mentioned in the paper. Whilst LIRF learns a policy for a reward function for just a subset of the trajectory (the terminal state) and IRL learns it for all states, the two approaches appear to have essentially the same formulation. In both cases a reward function is learned from demonstrations which is in turn used to train an RL policy. I would have liked to see a discussion comparing the two in the related work.
* I wonder how well this approach would scale. For example, the 10k demonstrations required to learn the IRF for the block stacking tasks seems costly to replicate for other similarly complex tasks.
   * One advantage of exemplar rewards is that just a single image is required instead of a demonstration trajectory. LIRF doesn’t have this benefit since it requires verification trajectories. In some cases these may require a similar amount of effort to full demonstrations.
* I do appreciate the substantial limitations section. However, related to the above it does seem that the time required to collect (and not just store) the positive actionable examples could be a limitation (e.g. if 10k per task is often required). I would have liked to see a discussion about this in the work.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents an interesting and novel approach to a relevant problem in robot learning, solving tasks when there is state aliasing. It is well written throughout and the experiments in simulation and the real world demonstrate the effectiveness of the proposed method LIRF. I do think a comparison with IRL is missing and I have concerns about how scalable the method is given the requirement to gather potentially thousands of verification trajectories. However, on balance the strengths outweigh the weaknesses.
issues: * Please clarify if the other reward component, D(oT), is sparse
* Please give some intuition about why λ is so large
* It would be helpful if the authors could share their thoughts about how this approach might scale.
* Please include a discussion comparing LIRF with on inverse reinforcement learning in the related work
* Typo: line 226: experiment → experiments
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces a new framework for interactive reward functions (IRF), which are a variant of exemplar rewards for scenarios where the relevant part of the state is not directly observable and needs to be verified through interaction. Each IRF is a separate policy trained with the goal of being able to discriminate positive demonstrations and negative examples from an untrained task policy. To train the task policy, after each task policy rollout, the IRF is executed to check the resulting state for task success. The framework is evaluated on three interesting showcases (door locking, block stacking, real-robot screwing) where clear advantages over vanilla exemplar rewards are shown. The framework is valuable for many real-life tasks where the task success is not camera observable.
strengths_and_weaknesses: The justification and ideas of the paper are presented very clearly and in excellent writing. The experiments do a great job of showcasing the kind of task that such a framework is useful for.

The main contribution of this paper is this idea that the reward function be computed over an observation not from the end of the task policy rollout but instead one obtained by additionally using a separate policy. Indeed, for the simulation-based cases, it would be possible to directly train a success/failure discriminator on the (latent) state of the simulation at the end of the task policy, not requiring an additional reward function policy. It is only valuable for the (more important) real-world RL case where such information is not accessible. This is not immediately obvious since much of the experiments focus on simulation.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper does a great job of setting up this framework of interactive reward functions, describing how to train and use them, and showing experimentally their benefits over exemplar rewards. The setup is very useful for many real-world tasks since the partial observability situation applies very often.
issues: I do not have any other comments on any issues with this paper and congratulate the authors for a paper very well written.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a framework for policy learning in a partially observable setting. In this setting, the observation itself does not contain full information about the Markovian state, and hence, does not have sufficient information to determine the ground truth reward or task success. To approach this, the paper proposes to train another policy, called the IRF policy, to perturb the final state in a way such that task success is transferred to the observation space. Therefore, this information can then be used as a reward function to train the task policy, or to verify whether the task is indeed achieved. 

The proposed framework is tested on 2 simulated tasks and 1 real robot task. In all cases, the proposed framework is shown to significantly outperform baselines that 1) only uses observation for exemplar rewards and 2) directly imitate the demonstrations. However, it seems like the proposed framework is only evaluated on tasks that VICE achieves a low success rate. 
strengths_and_weaknesses: ### Strengths
- **Organization**: The paper is very well written and organized. The motivation of the paper is clearly stated in the introduction. The authors use concrete examples, both from real life and the actual experiments to illustrate the intuition behind the proposed idea. The proposed algorithm is well-stated and easy to follow. The experiment section provides clear description on each of the tasks, and detailed information about their setup. 
- **Literature survey**: The authors clearly stated the limitation of existing work in exemplar rewards: They can not be directly applied to the partially observable setting since the task may falsely appear to have succeeded. The authors also discuss how their work differs from existing literature in Adversarial Robot RL and Interactive Perception and Veriﬁcation.
- **Technical approach**: The technical approach is well motivated and is built upon a sound intuition. The explanation of the algorithm is thorough and clear. 
- **Experiment validation**:  The proposed framework is tested on 2 simulated tasks and 1 real robot task. In all cases, the proposed framework is shown to significantly outperform baselines that 1) only uses observation for exemplar rewards and 2) directly imitate the demonstrations. The authors provided thorough information on the tasks, and how the policies are trained and evaluated. The experiments verified that the proposed framework is indeed critical in the partially observable setting.  
- **Final discussions**: It is appreciated that the authors discussed some of the limitations of their work in a fair way in the conclusion session. 

### Weaknesses
- **Assumptions on VICE performance**: As noted by the authors, the proposed approach assumes that the task policy initially performs poorly on the task. (If I understood it correctly, all the examples generated by VICE are considered negative, is that right?). It seems like a limitation since VICE policy may already be quite good in some tasks, e.g. if we only need to close the door. Can the authors provide more information on what will happen in this case? Does the performance of the task policy degrade after LIRF, or does it just stay roughly the same? Maybe the authors can, in the appendix, provide an experiment where VICE is sufficient to solve the task, and show what happens after applying LIRF. To follow up, does it imply that LIRF will fail to improve (or even degrade) performance if the initial task policy success rate is high enough? If so, how do we determine whether we should use LIRF or just use the initial VICE policy? It seems hard since the paper assumes no access to the ground truth reward function.

- **Only training IRF once**: The authors noted that the previous limitation is now preventing iterative training of IRL policies, which makes sense. However, I am curious whether this current framework can encourage the task policy to fail at tasks in a “creative” way that can not be caught by the IRF policy. Since the IRF policy is only trained once with the VICE policy, it will not be able to adapt to other failure modes that the updated task policy induces. Although it seems like it is not the case in the experiments shown in the paper, I am wondering if the authors have observed such a phenomenon at all, and if the authors have any intuition on what to do in such cases. Minor point: Since the IRF policy is only trained once, it would be a little misleading to call it “GAN-like adversarial critic”.

- **Size of N**: What is the size of $N$ for the current experiments?
- **Choice of hyperparameter**: How sensitive is LIRF to the choice of $\lambda$? 
- **Block stacking**: From the quantitative results, as well as the number of positive examples needed, it seems like the block stacking experiment is most challenging. However, according to the appendix, the authors already made various attempts to simplify it, such as using motion primitives, and provide weight information once a block is picked up. Can the authors provide more explanation on why this experiment is still so challenging?
- **Screwing task success**: Can the authors provide more information on how the task success rate is evaluated in the screwing experiment? It is not very clear what does “the valve reaching the tightened state” exactly mean. 
- **Typo?**: “The valve can rotate freely without limits when the motor is not engaged, whereas it is locked in place or controllable through the motor when the motor is not engaged.” Should it be “the motor is engaged” near the end?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper addresses a well-motivated problem of policy training in a partially observable setting. The paper is very well written and organized. The motivation of the paper is clearly stated in the introduction. The proposed algorithm is well-stated and easy to follow. The literature survey is thorough, The authors clearly stated how their work differs from existing work in the literature. The experiment section provides clear description on each of the tasks, and detailed information about their setup.  The authors also addressed clearly the limitations of their approach. 

The major limitation, however, is that the current setup only seems to work when the initial task policy has poor results (see the first point in **Weaknesses**). Based on the assumptions in this paper, it is also unclear how to evaluate the initial task policy, as the ground truth state and hence reward is assumed unknown. The paper would have significantly greater impact if this limitation is addressed. 

**Updates after the rebuttal period.** The additional results from the authors show that 1) LIRF will not degrade performance if VICE is already sufficient for solving the task; 2) multiple iterations of LIRF also does not degrade performance. This clears up my major concern. With that, I am glad to raise my rating from weak accept to strong accept. 
issues: See **Strengths and Weaknesses**.
review_title: null

Meta-review: metareview: The reviewers have unanimous decision to accept this paper. I thank the authors for their very thorough comments and answers. 

Summary: The paper seems to be well motivated and clearly written. All reviewers left a very detailed review. 

Quality(5/5): All reviewers agree that the paper is very well-written and technically convincing.

Clarity(5/5): All reviewers agree that the paper is very clear. But one reviewer suggested that Section 5 has to be more polished. 

Originality(5/5): No one questioned the originality of this work.

Significance (3/5): I see different opinions among the reviewers. The reviewers generally agree that interactive reward is new and creative. However, they have mixed opinions on the impact of this work. The reviewer "2X1d", "FXQi" and "T7V5" mentioned multiple important weaknesses of this paper. They should be addressed or clarified well before acceptance. The question about 1) comparisons to IRL  by 2X1d, 2) assumption on VICE performance by T7V5 seem particularly important.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)