Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: They propose to extend incremental/online inverse reinforcement learning (I2RL) to the case with noisy observations and an unknown observation model, making it suitable to real-world robotics applications. Their method -- called Robust Incremental IRL with Maximum Entropy Estimation of Observation Model (RI2RL-MEOM) -- is formulated as a sequence of incremental learning sessions where each session is a two-step process: (1) learning a maximum entropy distribution of the observation model, and (2) using that model to learn the reward function of the expert. They present results on 2 robotics tasks: (1) onion sorting with Sawyer arm based on human demonstration and (2) breaching a perimeter patrol with 2 Turtlebots. RI2RL-MEOM learns a more accurate policy compared to (a) an IRL method that doesn't learn an observation model, and (b) a custom baseline that learns a simpler observation model.
strengths_and_weaknesses: Strengths

* Their approach is practical for robotic applications, where the observations are typically noisy and the observation model is unknown.
* RI2RL-MEOM admits formal guarantees of monotonic convergence and sample complexity bound.
* Their experiment on onion sorting uses _real_ videos of human sorting onions, which is a realistic setting.
* Their experiment demonstrates low KL divergence for a simulated dataset of the onion sorting task, thus demonstrating the correctness of the learned model against a ground truth. (The simulated dataset is generated for the purposes of evaluating the learned observation model against a known model.)
* Their method achieves higher learned behavior accuracy (LBA) and requires less data (i.e. sessions) compared to their baselines -- RI2RL-FQOM and AIRL.

Weaknesses

* They required a manually defined set of features. Although this was appropriate for the tasks considered, this might be a limitation for more complex tasks that might not have obvious features or features that are non-trivial to extract.
* Although they did use real videos as training data for the onion sorting task, the evaluation of the learned policy was performed in simulation. However, this might be appropriate for their setting since they are testing for high-level decision making.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe this work is a good example of a learned-based method that addresses real-world constraints -- namely noisy observations and a unknown observation model -- and demonstrate results on a task that uses real-world data.
issues: I didn't find any major issues with this paper. However, I'm not familiar with the theoretical component of this work, and I defer the judgement on this to the other reviewers.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper considers the problem of reward learning when the observation model is unknown. The authors propose a maximum entropy approach to estimate the observation model an incorporate this into an online IRL algorithm. Results on real robots show that the proposed approach outperforms AIRL and a frequentist version. Theoretical analysis shows that the proposed approach has desirable convergence properties.
strengths_and_weaknesses: Strengths
- online reward learning with an unknown observation model is an important and understudied problem
- good overview of prior work in section 3
- nice theoretical analysis of proposed algorithm
- real robot experiments 

Weaknesses
- Seems to require fully solving the MDP as does classical MaxEnt IRL
- Requires discrete and binary features and known reward features
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper provides a nice mix of theory, technical rigor, and empirical results on physical robots. The experimental results show the advantage off the proposed approach and the overall problem studied by the authors seems novel and important for real world robot reward learning.

---

The authors have addresses my main concerns. I still think the paper should be accepted.
issues: The algorithm abbreviation (RI2RL-MEOM) is quite long and awkward. I'd suggest something shorter and catchier :)

Figure 2 has a lot of clutter with all the equations/notation in the upper right. I would recommend using words or pictures to describe as much as possible as done in the left half of the figure.

How would the authors propose that the reward features and observation features be chosen? The method seems to rely heavily on these being good representations. Could they be learned from data?

Does AIRL receive the same observation representations as the proposed approach?

How critical is the SA-network? Would the proposed approach work on raw continuous states and actions?

I would be interested in seeing what is lost due to an unknown observation model. Is it possible to run prior Robust IRL in simulation with a known model and compare to see how much learning the observation model degrades performance?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work deals with inverse reinforcement learning (IRL), specifically in robotic manipulation tasks where the observation model is noisy.  They state this is a more realistic scenario since real-world robotic perception is noisy and models therefore must be updated incrementally.  Previous works in IRL assume that the observation model is known, and they extend this work to additionally learning an unknown observation model incrementally.

The introduced method, I2RL-MEOM, approximates the unknown observation model using a shared feature set.  They then fit this model to observations with a max-entropy optimization (Equation 8).  The learned feature expectations are then used to incrementally update the IRL objective. The authors additionally demonstrate the convergence properties of the method.

The method is compared against AIRL on a variety of robotic manipulation tasks in an onion sorting domain and a perimeter breaching task.  
strengths_and_weaknesses: I2RL-MEOM is compared against AIRL on a variety of Sawyer robotic manipulation tasks in an onion sorting domain. Some onions are blemished and must be manipulated appropriately depending on the task (picked, sorted, etc). There are decent improvements over AIRL, but the results are not overtly impressive. The convergence analysis is also appreciated.

This method shows some decent improvements in a set of interesting and realistic Saywer sorting tasks. It also deals with a difficult and realistic problem, when the observation model is noisy in the IRL setting. There are decent improvements over AIRL, but the results are not overtly impressive.

Overall, I find this paper incremental, and limited. I believe that learning the observation model incrementally is too restrictive under this task, such as its restriction to binary features. I also believe that the experimental justification is lacking.

This paper is difficult to read at times, mostly since such a large amount of notation is introduced. For example, Section 5.1 is extremely dense with rapid-fire notation heavy descriptions of each algorithm step. I feel that this entire section could be replaced by a simplified pseudo-code figure. There are many descriptions in this paper where the notation could and should be simplified. Some formula-heavy related work could be summarized better (especially Section 3.1), with the full exposition in the appendix.

The results are poorly organized, Figure 3 (Right) is missing actual columns for the TP, FP, and result value, where instead, commas are used to separate results. Figure 4 is also too small. The exhaustive list of experiments (Lines 213-224) should also be moved to the appendix and only summarized in this section.

I am confused as to why the only baseline compared is AIRL. I feel that more extensive comparisons are needed against other model-based methods. It would be nice to see a comparison against Robust IRL, even with an incorrect observation model. The empirical justification for this method is, therefore, quite limited. I would also encourage the authors to compare against other SOTA Robust IRL methods such as Viano et al., 2021 (https://openreview.net/pdf?id=t8HduwpoQQv).

This method is also severely limited by the fact that features are binary functions. The authors rightfully explain this limitation in Section 7.


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I still believe this work is quite restrictive, and can only deal with a subset of feature functions.  The experimental results are lacking and suffer from a lack of sufficient baseline comparisons.  The improvements over AIRL are decent, but not overtly impressive. The writing is very hard to work through and is too notation heavy.  The algorithm description therefore suffers.
issues: The main issue with this work is the extremely notational heavy exposition.  I believe that better summarization of the method is also needed, with only the most relevant equations concisely shown in the main text.  I also encourage the authors to examine more baselines for comparison outside of AIRL as stated in my main review.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces an inverse reinforcement learning method, which learns an observation model, to deal with real world robotics applications with noisy observations. The observation model is learned using the maximum entropy principle and is then used to learn a reward function. The experiments are conducted on two tasks: (1) post-harvest onion sorting using human demonstrations; (2) a perimeter patrol task. On the presented tasks, this approach outperforms a state-of-the-art adversarial inverse reinforcement learning method (AIRL) and a custom baseline, which also uses an observation model (yet a less sophisticated one).

strengths_and_weaknesses: **Strengths**:
- the task of inferring a reward under noisy expert demonstration is of general interest

**Weaknesses**:
- the paper is not very well written. The paper is overly complex with a confusing set of notations.
- the contribution is minor, only the observation model is introduced to the online inverse reinforcement learning setting. 
- the application scope of the presented algorithms is very limited --> discrete states and actions, with binary observation features.
- the presented robotic tasks is solved on a very high-level.
- the comparisons made to state-of-the-art are made on very specific tasks, with very specific features. It is hard to judge on the general performance of this approach.
- no sufficient implementation details are given (e.g., in the appendix).
- no video and no code is provided.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: From my perspective, the general approach of inferring a reward under noisy expert demonstration is interesting and promising. However, the paper introduced a very limited approach and benchmarked on very specific task with a small number of highly engineered features and observations. This makes it very hard to judge on the performance of the presented approach in comparison to the state-of-the-art. 
Moreover, the paper is unnecessarily complex written making it hard to follow and not straightforward to understand. The notation needs to be revised from my perspective.

issues: **Issues**:
- the notation is very confusing. The combination of $\phi$ for the binary features, $\psi_j$ for the binary predicates and $\psi_J^{o,g}$ is not very straight forward to understand. In addition to that, Figure 2
confuses even more, where $\psi^{1,g}_j$ (where does the 1 come from?) is introduced, of which I am not sure of what it means. It would help a lot to create a more informative Figure presenting the general approach
a bit more clearly. Figure 2 is not well made in my opinion.
- Equation (6) uses the binary predicates $\psi_j (<s, a>)$ to define the indicators. Why is a different notation used for indicators in equation (8)s?
- Use a consistent math font, even in the Figures.
- It needs to be explained in greater detail, how the confidence intervals are calculated and the networks are trained.
- Implementation details, also for AIRL, on the experiments are missing. A judgment on AIRL's computational performance without giving implementation details is not very helpful.
- I don't think that AIRL is a good baseline for this task, due to the small state and action states. I would like to see a comparison to an approach made for discrete state and action spaces.


**Minor Issues**:
- please blur faces in Figures next time to ensure double blindness (even if the pictured persons are not the authors, but lab members etc.)
review_title: null

Meta-review: metareview: Overall, the problem of learning to imitate given noisy and unknown observations is relevant to the community. The approach is original although somewhat incremental, and the technical quality appears to be sound. It is great that real human demonstrations were used for the onion sorting task. Additionally, the inclusion of a formal theoretical analysis of the approach is a strength of the paper. However, there were a few weaknesses that should be addressed. First, the clarity of the paper could be improved by simplifying the notation, as multiple reviewers found the complex notation made the paper difficult to read. The use of binary/manual features seems to be a limiting factor, and the paper would also be improved by comparing to other approaches rather than only AIRL. The authors are encouraged to address these concerns along with other concerns raised by the reviewers.

========

Thank you to the authors for responding to the authors concerns. Overall, the authors address a relevant problem for the community and evaluate their approach on realistic experiments, which I believe warrants acceptance. I agree that needing to hand-specify features is a limiting factor of this approach, and the authors are encouraged to discuss how such features could be learnt within the text of the data. Additionally, the authors should aim to simplify notation within the paper to make it easier to read.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)