Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes Fast Lifelong Adaptive IRL (FLAIR), a learning from demonstration framework that aims to maintain a collection of learned strategies that may be mixed in order to model subsequent demonstrations. The method is similar to Multi-Strategy Reward Distillation (MSRD) but does not assume access to a strategy label. Overall, the paper is well-written and provides a thorough experimental evaluation.
strengths_and_weaknesses: Strengths:
- The paper proposes a new method for constructing a policy mixture for a new demonstration as well as a new objective, Between-Class Discrimination (BCD) that seems to be significantly more effective than existing work at adapting to a sequence of demonstrations.
- The paper conducts a thorough experimental evaluation, which includes ablations on FLAIR and even a real robot experiment with a table tennis-playing robot. The paper also includes a very thorough appendix. 

Weaknesses:
- It seems like the effectiveness of the method may depend on the range of demonstrations available--if they are not diverse, then a lack of strategies is modeled, and if they are too diverse, the method may learn too many strategies. 
- I'm not sure about the importance of the problem setting, and I think the assumptions made in the problem setting could be motivated better. It's not clear to me how many real world scenarios are actually modeled by this problem statement where individual demonstrations must arrive in sequence. For example, a robot learning to play table tennis will likely have access to a library of demonstrations to start instead of requiring each demonstration to arrive in sequence.

Questions: 
- As a baseline, what is the performance when not given demos in a sequential manner, e.g. given demos all at once, with AIRL or GAIL? 
- It would be helpful to add titles to the plots in Figure 2 and 3. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, despite some weaknesses, I think this is an interesting paper with a novel method and well-structured experiments. 
issues: Please see weaknesses/questions listed above and address accordingly.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an LfD framework to allow adapting to different user preferences over how a task is carried out. It uses an initial set of demonstrations to build an initial set of policies that correspond to different strategies. Then, users specify how they would like the task to be carried out by adding a demonstration. This demonstration is used to infer a policy mixture of the base set of strategies, or learn a new strategy if the demonstration is sufficiently different from the current set of strategies. In this way, the proposed method can learn continually. The proposed method is demonstrated on 3 simulation environments and a real world table tennis environment.
strengths_and_weaknesses: Strengths

The proposed method is an interesting idea that in principle, allows for more strategies to be learned and modeled as more demonstrations are added into the system. Results on the real-world table tennis domain are impressive (especially those shown in the supplementary video). 

Weaknesses

One of the main weaknesses is the experimental evaluation in simulation. The most important aspects of this work, with respect to prior work, appear to be the ability to continually adapt to new strategies (specified by new demonstrations). However, the evaluation of the adherence to strategies is mostly qualitative. It would be very useful to show additional experiments in simulation where there are a collection of ground-truth strategies that are specified by an initial set of humans (similar to the real-world table tennis experiment). Then, quantitative evaluations on how well learned policies adhere to the desired strategy would be possible, instead of the current experiments that rely on unsupervised strategy discovery (through DIAYN, reference 37). From this perspective, it could also be valuable to show results on other domains in simulation where ground-truth strategies can more easily be specified (for example, robotic manipulation, with different grasps possible for objects, or different speeds for trajectories that are executed on the arm).

Related to this, it would be nice to have more quantitative metrics for evaluating adherence to desired strategy in Sec 5.1 (e.g. returns under the ground-truth reward function for the strategy, not just the general task reward function).

More comments follow:

- What if it takes more than one demonstration to show a user-specific strategy? Would the method be able to handle this case?

- Sec 4.2 - how many trajectories are needed to estimate the objective and find the policy mixture weights? Do you then need to collect additional trajectories to estimate the KL divergence for line 5 of algorithm 1, once the mixture weights have been found?

- Sec 6 - how is RL run on the real robot in a sample efficient way? Does the human need to do manual resets? More details on this process would be helpful.

- It's a pretty strong claim to put "crowdsourced demonstrations" in the title without having extensive evaluation with several humans. 

- There are a few missing references for crowdsourced demonstrations in the paper (https://arxiv.org/pdf/1811.02790.pdf, https://arxiv.org/abs/1911.04052, https://arxiv.org/abs/2202.02005).
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The main issue is the experimental evaluation - there should be more quantitative (not just qualitative) evidence that the method learns to adhere to new strategies that are demonstrated. If this issue is addressed, I would increase my score.

**Post Author Response**

Thanks to the authors for addressing my comments in a thorough manner - I have raised my score to Weak Accept.
issues: The main issue is the experimental evaluation - there should be more quantitative (not just qualitative) evidence that the method learns to adhere to new strategies that are demonstrated.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents FLAIR, a new algorithm for lifelong “personalized” inverse RL that can rapidly adapt to new heterogeneous demonstrations. They maintain a set of learned policies corresponding to unique skills in demonstrations encountered so far; new demonstrations are modeled as a mixture of existing policies (if the behavior is captured sufficiently well) or a new policy (if the behavior is not). In three simulated continuous control environments, FLAIR outperforms baselines in adaptability, efficiency, and scalability. A real robot experiment is also performed to evaluate the utility of FLAIR's policy mixture.
strengths_and_weaknesses: This is a strong paper that tackles practical problems in learning from demonstration: lifelong deployment and heterogeneous demonstrations due to varying human preferences. The proposed approach is novel and technically sound, with an intuitive procedure (Algorithm 1) and a novel “Between-Class Discrimination” loss function. Experimental results are organized and presented well, demonstrating a win for FLAIR in adaptability, efficiency, scalability, and policy performance over baseline approaches Adversarial IRL (AIRL) and Multi-Strategy Reward Distillation (MSRD). Experiments suggest FLAIR is a new state-of-the-art approach for lifelong IRL from heterogeneous demonstrations. They also evaluate the quality of FLAIR’s policy mixture in a real robot experiment, and the video supplement gives good visual intuition for the experiment.

While strong, the paper has some weaknesses. Firstly, the authors claim that the code and data will be open-sourced, but it currently is not and is not available in the supplement; it would be ideal to have these available in order to reproduce simulation results, for example. Secondly, the clarity of the writing and presentation can be improved; in particular, Figure 1a is very difficult to parse and the text could benefit from proofreading. Thirdly, the results seem to be weaker for Lunar Lander and Bipedal Walker than the (very simple) Inverted Pendulum environment, as the respective figures are relocated to the appendix (Figure 2 in the appendix shows weak correlation compared to Figure 2 in the main text) or absent from both the text and appendix (e.g., the counterpart for Figure 4). Fourthly, it is unclear why the real robot experiment is a different experiment than the simulations (i.e., the FLAIR vs. AIRL vs. MSRD comparison); some clarification from the authors here would be appreciated.

More minor notes / requests for clarification:
- I am not sure why “crowdsourced” is emphasized in the title and “democratize access to robotics” is emphasized in the Introduction, as there is no large-scale data collection from crowdsourced humans in this paper.
- Have the authors considered a “smoothing” as opposed to “filtering” approach in Algorithm 1, which perhaps could recompute mixture weights for old demos with newer policies that were not available at the time?
- In Section 5.2 under Q6, why does FLAIR recover more than 5 strategies if there are only 5 ground truth policies? Is it approximation error?
- Which environment is the data in Figure 3 from?
- It would be nice to include visualizations of the different heterogeneous demonstrations in simulation for better intuition in Figure 4 and other parts of the paper.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend a Weak Accept as this paper makes solid contributions. However, it can be improved by addressing the issues below, and as I am not an expert in personalized IRL I may not be aware of all other existing techniques that may undercut the claim that FLAIR sets a new state-of-the-art.
issues: 1. Attach code supplement if possible
2. Provide Lunar Lander and Bipedal Walker counterparts for all Inverted Pendulum figures
3. Improve Figure 1
4. Explain why real experiment differs from simulation experiment
5. Attach visualizations of heterogeneous demonstrations in simulation if possible
review_title: null

Meta-review: metareview: Phase 1:

Strengths:
The submission provides a new and technically relevant method. It is well structured and intuitive in its argument. It provides a thorough experimental evaluation including real world robot experiments in a challenging domain.

Weaknesses:
Some weaknesses are found in the evaluation and multiple reviewers ask about clarifications. In particular, these include the different choices for comparisons and baselines between simulation and real experiments, weaker results on some of the toy domains, as well as quantitative metrics for adapting to different strategies.

Phase 2:

The feedback has originally been a borderline case with slightly more positive than negative feedback. Many points could be addressed during the review and final reviews are generally positive (3 weak accepts). The reviews point out that it is well structured and intuitive in its argument. It provides a thorough experimental evaluation including real world robot experiments in a challenging domain but also required clarifications around the evaluations and confusion about some of the baselines. I agree with the reviewers and recommend acceptance. Please take the remaining points from the review process seriously and follow up with improvements on open points and promised changes.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)