Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations
Authors: Letian Chen
Sravan Jayanthi
Rohan R Paleja
Daniel Martin
Viacheslav Zakharov
Matthew Gombolay
Venue: CoRL 2022 Poster (2022)
Keywords: Personalized Learning
Learning from Heterogeneous Demonstration
Inverse Reinforcement Learning

Abstract: Learning from Demonstration (LfD) approaches empower end-users to teach robots novel tasks via demonstrations of the desired behaviors, democratizing access to robotics. However, current LfD frameworks are not capable of fast adaptation to heterogeneous human demonstrations nor the large-scale deployment in ubiquitous robotics applications. In this paper, we propose a novel LfD framework, Fast Lifelong Adaptive Inverse Reinforcement learning (FLAIR). Our approach (1) leverages learned strategies to construct policy mixtures for fast adaptation to new demonstrations, allowing for quick end-user personalization, (2) distills common knowledge across demonstrations, achieving accurate task inference; and (3) expands its model only when needed in lifelong deployments, maintaining a concise set of prototypical strategies that can approximate all behaviors via policy mixtures. We empirically validate that FLAIR achieves adaptability (i.e., the robot adapts to heterogeneous, user-specific task preferences), efficiency (i.e., the robot achieves sample-efficient adaptation), and scalability (i.e., the model grows sublinearly with the number of demonstrations while maintaining high performance). FLAIR surpasses benchmarks across three control tasks with an average 57% improvement in policy returns and an average 78% fewer episodes required for demonstration modeling using policy mixtures. Finally, we demonstrate the success of FLAIR in a table tennis task and find users rate FLAIR as having higher task ($p<.05$) and personalization ($p<.05$) performance. 

Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations
Letian Chen Sravan Jayanthi*, Rohan Paleja
Daniel Martin, Viacheslay Zakharov, Matthew Gombolay
Georgia Institute of Technology
Atlanta, GA 30332
(letian, chen, s sjayanthi, rpalejaā, dmartini, vzakharov3,
matthew Batthew.gombolayyeee..... gombolay @gatech. edu
Learning from Demonstration (LfD) approaches empower end-users to teach robots novel tasks via demonstrations of the desired behaviors, democ- ratizing access to robotics. However, current LID frameworks of fast adaptation to heterogeneous human demonstrations nor the large-scale de- ployment in ubiquitous robotics applications. In this paper, we propose novel LiD framework, Fast Lifelong Adaptive Inverse Reinforcement learning (FLAIR). Our approach (1) leverages learned strategies to construct policy mixtures for fast adaptation to new demonstrations, allowing for quick end-user personalization, (2) distills common knowledge across demonstrations, achieving accurate task in- ference; and (3) expands its model only when needed in lifelong deployments, maintaining concise of prototypical strategies that can approximate all be- haviors via policy mixtures. We empirically validate that FLAIR achieves adapt- ability (i.e., the robot adapts heterogeneous, user-specific task preferences), ef- ficiency (i.e., the robot achieves sample-efficient adaptation), and scalability (i.e., the model grows sublinearly with the number demonstrations while maintain- ing high performance). FLAIR surpasses benchmarks across three control tasks with an average 57% improvement in policy returns and an average 78% fewer episodes required for demonstration modeling using policy mixtures. Finally, we demonstrate the success of FLAAR in a table tennis task and find users rate FLAIR having higher task (p < .05) and personalization (p < .05) performance.
Abstract:
not capable
Keywords: Personalized Learning, Learning from Heterogeneous Demonstra- tion, Inverse Reinforcement Learning
Introduction
Robots becoming increasingly ubiquitous with recent advancements in Artificial Intelligence (AI), largely due to the success of Deep Reinforcement Learning (DRL) techniques generating high-performance continuous control behaviors [1, 2, 3, 4, 6, 7, 8). However, DRL's success heavily relies on sophisticated reward functions designed for each task. These hand-crafted reward functions typically require iterations of fine-tuning and consultation with domain experts be ef- fective [9]. Instead, Learning from Demonstration (LfD) approaches democratize access to robotics by having users demonstrate the desired behavior the robot [10], removing the need for per-task reward engineering. While LiD research strives to empower end-users with the ability to program novel behaviors onto robots, we must consider that end-users may adopt varying preferences and strategies in how they complete the same task [1]]. An LfD framework that assumes homogeneity across the set of provided demonstrations could cause the robot to fail infer the accurate intention, resulting in unwanted or even unsafe behavior [12, 13]. On the other hand, embracing individual preferences can help robots achieve better performance and long-ten acceptance from humans [14].
While personalization is important for accurate recovery of the demonstrator's behavior, personal- ization also prove inefficient if each individual policy must be inferred separately. To avoid
denotes equal contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Common Task Reward
FLAIR
Strategy 1
Strategy 2
Reward 1
Reward
Between-Clan Class Discrimination
Pollcy 7E
Policy 222
Policy T.
W WoTE
Demonstration
Policy Mixture Optimization
IRL
IRL
Topspin Strike
Slice Strike
Push Strike Topspin
Slice
User 1
User
User 3
Lifetime
Figure 1: This figure shows an illustration of the lifelong learning process with our proposed method, FLAIR. As each demonstrator performs their strike, FLAIR determines whether the demonstration is novel. If a demonstration can be explained by policy mixture of previously learned strategies, FLAIR accepts the policy mixture without training to the demonstration, FLAIR creates a new strategy and a prototype policy for the demonstration.
strategy. If the policy mixture is not close
this issue, prior work, MSRD [15), decomposed shared and individual-spaa speeiffi reward informa- tion across heterogeneous demonstrations (i.e., demonstrations seeking to accomplish the same task with different styles). While MSRD significantly improves the accuracy and efficiency in personal- ized policy modeling, the framework must be trained all-at-once and is unable to handle incremen- tal/tifelong learning, a more realistic paradigm for LfD real-world applications.
In this work, we develop FLAIR: Fast Lifelong Adaptive Inverse Reinforcement learning. As running example, consider series humans teaching robot how play table tennis, compelling robot learning platform [16, 17, 18). Users of the robot may have their own preferences for table tennis strikes. shown in Figure the first user demonstrates topspin strike, while the second user demonstrates a slice strike. The third user demonstrates push strike, which could be explained by a composition known behaviors of the previously seen topspin and slice prototypical behaviors.
Unlike prior LfD algorithms, FLAIR is capable of continually learning and refining a set of pro- totypica strategies either (1) efficiently model new demonstrations as mixtures of the acquired prototypes (c.g.. the third user in our example) or (2) incorporate a new strategy as a prototype the strategy sufficiently unique (c.g., the second user). Consider real-world example where household robots are delivered to users' homes and the users want to teach those robots skills over the course of the deployment. User demonstrations from different end -useers form a demonstra- tion sequence the robots personalize to. In such lifelong learning scenario, FLAIR autonomously identifies prototypical strategies, distills common knowledge across strategies, and precisely models each demonstration as prototypical strategies or policy mixtures. We show FLAIR accomplishes adaptivity, efficiency, and scalability LfD tasks simulated and real robot experiments:
Adaptive Learning: We display the adaptivity of FLAIR by successfully personalizing to het- erogencous demonstrations on three simulated continuous control tasks. FLAIR models demon- strations better than best benchmarks and achieves an average of 57% higher returns on the task.
2. Efficient Adaptation: FLAIR is more efficient, empirically needing an average of 78% fewer samples to model demonstrations compared to training a new policy.
Lifelong Scalability: We showcase the scalability of FLAIR in a simulated experiment obtain- ing 100 demonstrations sequentially. FLAIR identifies on average eleven strategies and utilizes policy mixtures achieve precise representation of each demonstration, providing empirical evidence for FLAIR's ability to learn a compact set of prototypical strategies in lifelong learning. Robot Demonstration: We demonstrate FLAIR's ability successfully leverage policy mixtures to achieve stronger task and personalization performance than learning from scratch in real- world table tennis robot experiment.
2 Related Work
Two common approaches in LfD are to either directly learn a policy, i.e., Imitation Learning (II.), or infer reward to train policy, i.e., Inverse Reinforcement Learning (IRL.) [19]. II. learns direct

---

mapping from states to the actions demonstrated [20, 21]. Although a straightforward approach, IL suffers from correspondence matching issues and is not robust to changes in environment dynam- ics due to its mimicry of the demonstrated behaviors [22, 23]. IRI., on the other hand, infers the demonstrator's latent intent in more robust and transferable form of a reward function [24].
Although traditional IRL approaches often overlook heterogeneity within demonstrations, there has been recent work that models heterogeneous demonstrations [25, 26. 27, 28, 29, 30J. One intu- itive way is to classify demonstrations into homogeneous clusters before applying IRL [11]. The Expectation-Mant Mariiiizaatiiiiiioinnnnn (EM) algorithm also operates on a similar idea and iterates between E- step and M-step, where E-step clusters demonstrations and M-step solves the IRL problem on each cluster [31, 32]. When the number of strategies unknown, a Dirichlet Process prior [33, 34, 35] or non-parametric methods [36] could be used. In these approaches, each reward function only learns from portion of the demonstrations, making them prone to the issue of reward ambigu- ity [15]. Furthermore, these methods assume access to all demonstrations beforehand, which is not realistic for LfD algorithm deployment. We instead consider the more realistic setting of lifelong learning [37], where agent adapts to new demos through its lifetime and continually builds its knowledge base. One instance to generate such demonstration sequences is through crowd-sourcing (seeking knowledge from a large set of people) [38, 39, 40J.
Despite the abundance of previous approaches, few consider the relationship between the policies leamned to represent each demonstration. Our method, FLAIR, exploits these relationships to not only model heterogeneous demonstrations (adaptabilitt), but do by creating expressive policy mixtures from previously extracted strategies (efficiency), and can scale to model large number of demonstrations utilizing a compact set of strategies (scalability).
3 Preliminaries
In this section, we introduce preliminaries on Markov Decision Processes (MDP), Inverse Rein- forcement Learning (IRL), and Multi-Strategy Reward Distillation (MSRD).
Markoy Decision Process - A MDP, M, is a 6-tuple, AS A, R. Po). S and are the state and action space, respectively. R is the reward function, meaning the agent is rewarded R(s) in state T(s'ls, a) is the probability of transitioning into state s' after taking action a in state 8. Y E (0, 1) is the temporal discount factor. Po denotes the initial state probability. A policy, (als), represents the probability of choosing an action given the state and is trained to maximize the expected cumulative reward, π arg max, E,-, |>== t- R(s))) where T = {si, 01 82, ....... is a trajectory.
Inverse Reinforcement Learning IRL considers an MDP sans reward function (MDP)R) (R) and infers the reward function R based on a set of demonstration trajectories / = (7 T2, .N where N is the number of demonstrations. Our method is based on Adversarial Inverse Reinforce- ment Learning (AIRL) [23], which solves the IRL problem with a generative- discriminator, Dg. predicts whether the transition, (st, 8t+1), belongs to demonstrator the generator, T (a)s). Tg is trained to maximize the pseudo-reward given by the discriminator.
setup. The
Multi-Strategy Reward Distillation MSRD [15] assumes access to the strategy label, Cr. E {1,2. .M} (M the number of strategies). for each demonstration, T₁, and decomposes the per-strategy reward, R2, for strategy i as a linear combination of common task reward, Raakk and strategy-only reward, Rs-t. MSRD parameterizes the task reward by 0rask and strategy-only reward by 0s-i. MSRD takes AIRL. as its backhone IRI. algorithm, and adds a regularization loss which distills common knowledge into frask and only keeps personalized information in As-i. The MSRD loss for the discriminator (the reward) is shown in Equation 1.
LD = - E((.er)<il [log Dona.eso, (st, s(4)] E (r,,))) [log (1 - Dena. Osc, (Sc, st+1))
+ aBBrTo))~ Rs,,, (82)]12
4 Method
In this section, we start by introducing the problem setup and notations. We then provide an overview of FLAIR, and its two key components: policy mixture and between-class discrimination.
We consider lifelong learning from heterogeneous demonstration process where demonstrations arrive in sequence, as illustrated in Figure 1. We denote the i-th arrived demonstration as т,- Unlike
3

---

prior work, FLAIR does not assume access to the strategy label, Cr,- Similar to MSRD, FLAIR learns a shared task reward Re strategy rewards R and policies corresponding to each strat- egy kdz. We define the number of prototype strategies created by FLAIR till demonstration as M,, and (T) = - R((r) as trajectory r's discounted cumulative reward with the ward function Rg. The goal of the problem to accurately model each demonstration sequentially with few environment samples possible. Note that learning from sequential demonstrations is not a requirement of FLAIR but rather a feature in comparison to batch- based methods where all demonstrations must be available before the learning could start.
4.1 Fast Lifelong Adaptive Inverse Reinforcement Learning (FLAIR)
In our lifelong learning problem setup, when a new demonstration т; becomes available, we seek to accomplish two goals: design policy that solves the task while personalizing to the demonstra- tion (i.e., the objective in personalized LfD), and b) incorporate knowledge from the demonstration to facilitate efficient and scalable adaptation future users (i.e., the characteristics required for lifelong LfD framework). We present our method, FLAIR, in pscudocode in Algorithm 1.
Algorithm 1: FLAIR
Input : Demonstration modeling quality threshold
Mn
Mixture Weights=[1. MixtureWeiggts=(0,, m=[]
while lifetime learning from heterogeneous demonstration do
Obtain demonstration T₁
W; D h-Pooliyyiiiturrpppimiiattttoo (T 13=1)
if Demi < then
Mixture SextureWeighsslik-- Wh, Mi+1 + M.
else
Tnew, RB (TiL
DREW + E, Ey TKL(T.,T
10
if Dii < Dnew then
11
MixtureeWiighs//ii W,, M1+1 M, else
12
13
M1+1 +++1
14
mM--: +
15
MixtureWeightsiji-- 0.0 .0.1
M, zeros
16
Update Ro Roeg: To, by Between-Class Discrimination and MSRD
To accomplish these goals, FLAIR decides whether explain new demonstration with previously learned policies (a highly efficient approach), or create a new strategy from scratch (a fallback technique). In the first case, FLAIR attempts explain the new demonstration, т,, by constructing policy mixtures with previously learned strategies according to the demonstration recovery objective (line 4). If the trajectory generated by the mixture is close to the demonstration (evidenced by the KL-divergence between the policy mixture trajectory and the demonstration state distributions falling under a threshold, e), FLAIR adopts the mixture without considering creating new strategy (line 6). Since the policy mixture optimization (details in Section 4.2) is more sample efficient than the AIRI. training-froon from--rratche FLAIR can bypass the computationally expensive new-strategy training (line 8) if the mixture provides high-quality recovery of the demonstrated behavior. This procedure results in an efficient policy inference.
If the mixture does not meet the quality threshold, c, FLAIR trains new strategy by AIRL with Ti and compares the quality of the new policy to the policy mixture (Lines 8-10). If the mixture performs better, we accept the mixture weights (line 11). If the new strategy performs better, we accept the new strategy as a new prototype and update our reward and policy models (accordingly. in Line 13, we increment the number of strategies by one). Further, we call the demonstration, тi the "pure" demonstration for strategy M,+1, meaning strategy M,+1 represents demonstration т; (line 14). As such, the mixture weight for 7; is one-hot vector strategy M (line 15).

---

To effectively maintain knowledge base, we propose novel training signal named Between- Discrimination (BCD). BCD trains each strategy reward to capture the fact that each demonstration has certain percentage of the strategy. In the table tennis example (Figure 1), the third user's be- havior a mixture of the topspin and the slice, indicating topspin and slice strategy rewards should be apparent in the third demonstration. BCD encourages the two strategy rewards to give partial rewards to the third demonstration. tion 1) for all strategies with their corresponding pure demonstrations, and updates the generator policies based on the learned reward (line 16).
addition to BCD, FLAIR also optimizes MSRD loss (Equa-
4.2 Policy Mixture Optimization
To achieve efficient personalization for a new demonstration т; (Line 4 of Algorithm 1), we con- struct policy mixture with a linear geometric combination of existing policies Tı, "2,""" TM, (Equation 2), where Wij 2 are learned weights such that: ΣØ wa,j =.
M.
TTT
0j ~y~Tyl TT(s))
As the ultimate goal of demonstration modeling is recover the demonstrated behavior, we optimize the linear weights, vi; to minimize the divergence between the trajectory induced by the mixture policy and the demonstration, shown in Equation
minimize Er~Ti,; [DKL 7)]
(3)
di;
Specifically, we choose Kullback- divergence (iverivergence) [41] on the state marginal dis- tributions of trajectories in our implementation. We estimate the state distribution within a trajectory by the kernel density estimator [42]. More details can be found in supplementary.
Since the trajectory generation process is non-differentiable, we seek a non- conn-gradient-baaa based optimizer to solve Equation 3. Specifically, FLAIR utilizes naïve, random optimization method; it generates random weight vectors W7, evaluates Equation 3, and chooses the weight that achieves the minimiza- tion. Empirically, we find random optimization outperforms various other optimization methods for FLAIR. Please see the supplementary for a detailed comparison.
4.3 Between-Class Discrimination
Although MSRD distills the task reward from heterogeneous demonstrations, it does not encourage the strategy rewards encode distinct strategic preferences. MSRD also requires access to ground- truth strategy labels for all demonstrations, which limits scalability. In order to increase the strategy reward's discriminability between different strategies, we propose novel learning objective named Between- Classs Discrimination (BCD), BCD enforces the strategy reward to correctly discriminate mixture demonstrations from the pure demonstration: demonstration т; has weight wi, on strategy J (as identified in Policy Mixture), we could view the probability that т; happens under the strategy reward, Rs-i. should be Wi proportion of the probability of the pure demonstration, Tmy- This property can be exploited to enforce structure on the reward given to the pure-demonstratton, Tmy and mixture-demoostration Tı, as per Lemma A proof is provided in the supplementary. Lemma 1. Under the maximum entropy principal,
6 2R,, (Tc)
Wa
P(Tmy;
niss, (Tmy)
Thus, we enforce the relationship of strategy rewards, S-j. evaluated on pure strategy demonstration, Tmy, and mixture strategy demonstration, with mixture weight Wtyj. as shown in Equation 4.
LBCD (0
(e n0s, (Tt) wi.e " (mm))) )mm)))
(4)
An extreme case of BCD loss is when T; is the pure demonstration for another strategy, lo (i.e., mi = i). this case, way 0 (as т; is purely on strategy k), and Equation 4 encourages the strategy j's reward to give as low as possible reward to In turn, strategy rewards gain better discrimination between different strategies, facilitating more robust strategy reward learning, and contributing to the success in lifelong learning.
5

---

Table 1: This table shows learned policy metrics between AIRL, MSRD, and FLAIR. The higher environment returns lower estimated KL. divergence higher strategy rewards, the better.
Domains
Inverted Pendalum
Lunar Lander
Bipedal Walker
Methods
AIRI.
MSRD
FLAIR
AIR.
MSKD)
FLAIR
AIRI.
MSRD
FLAIR
172.7
7418.1
9899.3
8346.6
30637.2
74166.0
7064.0"
Estimated KL Divergence
1.08
7.67
1.01
72.0
70.9
67.2"
13.0
32.6
12.1
Stralegy Rewards
-5.73
-6.22
-1.23
-12.67
20.26
-..99
-5.31
-29.82
-4.22
Significance of 0.05
Significance of p < 0.01
Correlation between the Estimated and the Ground-True Task Reward
Episodes Needed to Achieve the Same Performance
Rescaled Estimated Reward
5050
4000
-100
Episodes
3000
-1002
2000
1200
1000
FLAIR (Curr): 0.33
-1600
1150
-1100
-1200
-1000
-200 -60
-100 -200
HIRD
Ground-Truth Task Reward
Method
Figure 2: This figure shows the correlation between the estimated task reward with the ground truth task reward for Inverted Pen- dulum. Each dot is a trajectory. FLAIR achieves higher task reward correlation.
Figure 3: This figure compares the number of episodes needed for AIRL and MSRD achieve same Log L Likelihood as FLAIR's mixture optimization. The red bar is the me- dian and the red triangle represents the mean.
Results
In this section, we show that FLAIR achieves adaptability, efficiency, and scalability in model- ing heterogeneous demonstrations. We test FLAIR three simulated continuous control environ- ments in OpenAl Gym [43]: Inverted Pendulum (IP) [44], Lunar Lander (LL.,, and Bipedal Walker (BW) [45]. We generate collection of heterogeneous demonstrations by jointly optimizing envi- ronment and diversity reward with DIAYN [46]. For all experiments excluding the scalability study, demonstrations, We compare FLAIR with AIRL and MSRD by running three trials of each method. More experiment details and statistical test results are provided in the supplementary.
5.1 Adaptability
Q1: Can FLAIR's policy mixtures perform well at the task? From ten demonstrations, FLAIR created 6.3 + 0.5 strategies (average and standard deviation across three trials) in IP, 5.3 1.2 in LL, and 3.3 + 0.5 in BW. FLAIR's learned policies including policy mixtures significantly more successful at the task (row "Environment Returns" in Table 1), outperforming benchmarks in task performance with 77% higher returns in IP, 14% in LL., and 80% in BW than best baselines.
Q2: How closely does the policy recover the strategic preference? Qualitatively, we find that FLAIR learns policies and policy mixtures that closely resemble their respective strategies, visual- ized in policy renderings (videos available in supplementary). We further show that FLAIR is sta- tistically significantly better in estimated KL divergence than AIRL (average 4% better) and MSRD (average 18% better), shown in row "Estimated KL Divergence" in Table 1, where KL divergence is evaluated between policy rollouts and demonstration state distributions. We further tested the learned policies' performance on ground-truth strategy reward functions given by DIAYN. The re- sults "Strategy Rewards" illustrate FLAIR's better adherence the demonstrated strategies.
Q3. How well does the task reward model the ground truth environment reward? We evaluate the learned task reward functions by calculating the correlation between estimated task rewards

---

Environment Returns
0
Demonstration
Damonstration
Demonstration
Demonstration
1 10:0
-50
Demonstration 2-04
Demonstration e:23
Demonstration
Demonetration nit
-100
Demonatration
Demonsiration HAT M40
FLAIR on 10 Demos
Dumonatration
Demonstration .d:
AIRL Demon
Demonatration
Demonstration .00:
-150
Dumonstration
Demonstration 1ec
Demonstration 404
Demonstration nus
25
75
100
Demonstration
Demonstration m:3 oxn 0.2
Demonstrations
Figure 4: This figure depicts the normalized strategy rewards demonstrations in IP for FLAIRR without BCD (left) and with BCD (right).
Figure 5: This figure plots the returns of FLAIR policies 100 demonstration experiment verted Pendulum.
with ground truth environment rewards. We construct a test dataset of 10,000 trajectories with multiple policies obtained during the "DIAYN+env reward" training. FLAIR's task reward achieves r = 0.953 in (shown in Figure 2), = 0.614 in LP, and r = 0.582 in BW, with an average 18% higher correlation than best baselines and statistical significance compared with AIRL and MSRD.
Q4. Can the learned strategy rewards discriminate between different strategies? We analyze the learned strategy rewards heterogeneous demonstrations (shown in Figure 4 right). We find that each strategy reward of FLAIR identifies the corresponding pure demonstration (Demonstrations 0-4.7) alongside the mixtures (Demonstrations 5-6, 8-9). In contrast, the strategy rewards learned without BCD (Figure left) do not distinguish between different strategies. This ablation study also finds that FLAIR with BCD achieves 70% better environment returns and 10% better KL divergence than FLAIR without BCD (additional metrics available in supplementary). The qualitative results in Figure 4 and quantitative results in supplementary together provide empirical evidence that FLAIR with BCD can train strategy rewards to better identify different strategies.
5.2 Efficiency & Scalability
Q5. Can FLAIR's mixture optimization model demonstrations more efficiently than learning a new policy? We study the number of episodes needed by FLAIR's mixture optimization and AIRL/MSRD policy training to achieve the same modeling performance of demonstrations. The result in Figure demonstrates FLAIR requires 77% fewer episodes to achieve a high log likelihood of the demonstration relative AIRL and 79% fewer episodes than MSRD. Three (out of ten) of AIRL's learned policies and four of MSRD's s learned policies failed to reach the same performance FLAIR even given 10,000 episodes, and are thus left out in Figure 3. By reusing learned policies through policy mixtures, FLAIR explains the demonstration in an efficient manner.
Q6. Can FLAIR's success continue in larger-scale LfD problem? We generate 95 mixtures with randomized weights from prototypical policies for a total of 100 demonstrations to test how well FLAIR scales. We train FLAIR sequentially
the 100 demonstrations and observe FLAIR learns a concise set of 17 strategies in IP, 10 in LL, and in BW that capture the scope of behaviors while also achieving consistently strong task performance (Figure 5 and supplementary). We find FLAIR maintains or even exceeds its 10-demonstration performance when scaling up to 100 demonstrations.
5.3
Sensitivity Analysis
Q7. How sensitive is FLAIR's mixture optimization threshold? We study the classification skill of the mixture optimization threshold and find it has strong ability to classify whether a demon- stration should be included
strategy. A Receiver Operating Characteristic (ROC) Analysis suggests FLAIR with thresholding achieves a high Area Under Curve (0.92) in the ROC Curve for IP; the specific choice of the threshold depends the performance/efficcennyy trade- off the user/application demands (see the ROC Curve and threshold selection methodology in the supplementary).
mixture
5.4 Discussion
The above findings show that our algorithm, FLAIR,, sets a new state-of-the-ard in personalized LfD. Across several domains, FLAIR achieves better demonstration recovery compared to the baselines.
7

---

Metries
Task Score
Strategy Score
FLATR's Policy Mixture
66.9 + 10.3"
96.6 + 17.4"
FLAIR's ' Worst Mixture
59.5 12.8
23.7
Leaming from Serarch
56.6 + 12.3
90.0 18.0
050
Signifieance of p 0.05
Table 2: This table depicts policy metrics between FLAIR's s best mixtures, FLAIR's worst mixtures, and
Figure 6: This figure illustrates topspin and slice mixture policy (a push-like behavior). The robot moves from location (1) to (2) and (3).
policies. The scores are shown as averages + standard deviations across 28 participants. Bold denotes the highest scores.
Not only can FLAIR more accurately infer the task reward and associated policies, but FLAIR also able to perform policy inference with much fewer environmental interactions. These characteristics make FLAIR amenable lifelong LfD, resulting one of the first LfD frameworks that can handle sequential demonstrations without requiring retraining the entire model.
Real- World Robot Case Study: Table Tennis
We perform real-world robot table tennis experiment where we leverage FLAIR's policy mixtures to model user demonstrations. An illustration of an example policy mixture is shown in Figure (more videos are available in supplementary).
We first collect demonstrations of four different table tennis strategies (i.e. push, slice, topspin, and lob) via kinesthetic teaching from one human participant who is familiar with the WAM robot but does not have prior experience providing demonstrations for table tennis strikes. After training the four prototypical strategy policies, we assess how well FLAIR can use policy mixtures to model new user demonstrations. To do so, we collected demonstrations from 28 participants by instructing them to demonstrate five repeats of their preferred PingPong strike. We utilize this data and compare three LfD approaches for learning a robot policy: 1) the best policy mixture identified by FLAIR, 2) a learming-fromm saong--from--catttee scratch approach, and 3) an adversarially optimized policy mixture (i.e., minimize the KL divergence between the rollout and the demonstration). We then have users/participants observe the robot executing these policies in random order. Using ad hoc Likert scale questionnaires (see supplementary), participants evaluate the robot's performance (i) accomplishing the task and (ii) doing so according to the user's preferences. Table shows that FLAIR's best mixture outperforms both the worst mixture (task score: < ,01, strategy score: p < .001) and the learning- from-sccatte policy (task score: p ,001, strategy score: p .05), demonstrating FLAIR's s ability to optimize policy mixtures that succeed in the task and fit user's preferences. Full statistical testing results are available in the supplementary.
7 Conclusion, Limitations, & Future Work
In this paper, we present FLAIR, a fast lifelong adaptive LfD framework. In benchmarks against AIRL and MSRD, we demonstrate FLAIR's adaptability novel personal preferences and effi- ciency by utilizing policy mixtures. We also illustrate FLAIR's scalability in how it learns a concise set of strategies solve the problem of modeling large number of demonstrations.
Some limitations of FLAIR are 1) if the initial demonstrations are not representative of a diverse set of strategies, the ability effectively model large number of demonstrations may be impacted due to the biased task reward and non-diverse prototypical policies; 2) FLAII'ss 's learned rewards are non- stationary (the learned reward function changes due the adversarial training paradigm), a property inherited from AIRL, and hence could suffer from catastrophic forgetting. For the first limitation, we could pre-train FLAIR with representative demonstrations before deployment to avoid biasing the task reward and to provide diverse prototypical policies. Another potential direction is to adopt "smoothing" "rathinng"-bsddd based approach over "filtering" method. The smoothing-basee approach would allow new prototypical policies to model previous demonstrations, relaxing the diversity assumptions on initial policies. We are also interested in studying how recover a minimally spanning strategy set that could explain all demonstrations, For the second limitation, we seek to leverage IRL techniques that yield stationary reward for the FLAIR framework. f-IRI. [47] could be potential candidate, but is notoriously slow due to the iterative reward training and policy training.
8

---

Acknowledgments
We wish to thank our reviewers for their valuable feedback in revising our manuscript. This work was sponsored by NSF grant IIS-21 12633, MIT Lincoln Laboratory grant PA8702-15-D-000) and Office of Naval Research grant N00014-19-1-2076.
References
[1] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and Levine. Soft actor-critic algorithms and applications. CoRR, 2018.
[2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[3] R. Paleja, Y. Niu, A. Silva, C. Ritchie, S. Choi, and M. Gombolay. Learning interpretable, high-performing policies for continuous control problems. arXiv preprint 22X 2022.
[4] E. Seraj. Z. Wang. R. Paleja, D. Martin, M. Sklar, A. Patel, and M. Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings the st International Conference on Autonomous Agents and Multiagent Systems, pages 1173- 1 1122, 2022.
[5] A. Silva, N. Moorman, W. Silva, Z. Zaidi, N. Gopalan, and M. Gombolay. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 72): 2022. doi: 1021109922222213333666
[6] E. Seraj. L. Chen, and M. C. Gombolay. A hierarchical coordination framework for joint perception-action tasks in composite robot teams. IEEE Transactions Robotics, 38(1) 38(1):139- 158, 2021.
[7] S. Konan, E. Seraj, and M. Gombolay. Iterated reasoning with mutual information in coopera- tive and byzantine decentralized teaming. arXiv preprint arXiv:2201.0844 2022.
[8] G. Konan, E. Seraj, and M. Gombolay. Contrastive decision transformers. 6th Annual Conference Robot Learning, 2022.
[9] L. Matignon, G. J. Laurent, and N. Le Fort-Piat. Reward function and initial values: Better choices for accelerated goal-directed reinforcement learning. In Artificial Neural Networks ICANN 2006, pages 840 849, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
[10] S. Schaal. Learning from demonstration. In M. C. Mozer, M. Jordan, and Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1997. URL attps://proceedigg.rrt 0G1134f2664444433233344999999339 Paper pdf.
T.
cc/paper/199/ffile/
[11) S. Nikolaidis, R. Ramakrishnan, K. Gu, and J. Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In 2015 10th ACM/IEEE International Conference
Himman-Rober Interaction (HRI), pages 189-196. IEEE, 2015.
[12] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza. Power to the people: The role of humans in interactive machine learning. Al Magazine, 3054):005-100, Dec. 2014. 2t5.0///iaa..33......... v35i4.2513 URL https://ojs aaai. aeail.or/indee php/ aimagazine/ article/viev/2511
[13] E. F. Morales and C. Sammut. Learning to fly by combining reinforcement learning with behavioural cloning. In Proceedings of the International Conference on Machine Learning (ICML), page 76, 2004.
[14] I. Leite, C. Martinho, and A. Paiva. Social robots for long-term interaction: A survey. In- ternational Journal of Social Robotics, 5(2):291-308, Apr 2013. ISSN 1875-4805. doi: 10.1007//12339-013-000 URL https: : ///oi 10t1//00.0100///113333/

---

[15] L. Chen, R. R. Paleja, M. Ghuy, and M. C. Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. In Proceedings of the International Conference on Human- Rohan Interaction (HRI), 2020.
[16] K. Mülling. J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. Proceedings of the International Journal of Robotics Re- search (IJRR), (2(3):263-22) 279, 2013.
[17] K. Muelling, A. Boularias, B. Mohler, B. Schölkopf, and J. Peters, Learning strategies in table tennis using inverse reinforcement learning. Biological cybernetics, 108(5):603-61) 2014.
[18] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. In Proceedings of Conference on Robot Learning (CoRI.), 2020.
[19] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3, 2020.
[20] A. Chella, H. Dindo, and Infantino. A cognitive framework for imitation learning. Robotics and Autonomous Systems, 54655:4034000 2006. ISSN 0921-8890. doi:https://doi.org/// 1016/ Lrobot.2006.01 ..008. URL https: h //ts:///wwwwww www.scedireee........m. cow/science///cccccccccccccccccc aaticc//rrttee/pppp/ $09218890060002000. The Social Mechanisms of Robot Programming from Demonstration.
[21] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35 2017.
[22] de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning- In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: /ttp///eeeeeens.e//ee//e/e www.eeeiiininnnnnnnnn.eeeee neurips.......... wwtpseie.crepppppeeeeeeeeeeeeeee cc/paper/2019/file 97708840008022b000555522222222 Papeeeerrr.ppp.r......
[23] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[24] N. D. Daw and P. Dayan. The algorithmic anatomy of model-based evaluation. Philosophical Transactions of the Royal Society B: Biological Sciences,
2014.
[25] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In leml, volume 1, page 2, 2000.
[26] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Pro- ceedings of the International Conference on Machine Learning (ICML). ACM, 2004.
[27] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), page 2586-2591 Morgan Kaufmann Publishers Inc., 2007.
[28] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the National Conference on Artificial intelligence (AAAI), pages 1433-1438, 2008.
[29] B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximun causal entropy. PhD thesis, Carnegie Mellon University, 2010.
[30] R. Paleja, A. Silva, L. Chen, and M. Gombolay. Interpretable and personalized apprentice- ship scheduling: Learning interpretable scheduling policies from heterogeneous user demon- strations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 6417-6428. Curran Associates, Inc., 2020. URL https::/proceediing wng.oeeeding..nuurpi...... www.ipe...... httpg//..rrrrppe..ppppp////..... /ttppp/p...cc/appr//////////// 4770555b2212966053377742886865 .dff.
10

---

[31] M. Babes-Vroman, V. Marivate, K. Subramanian, and M. Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Machine Learning, ICMI. 2011, pages 897-904, 01 2011.
[32] G. Ramponi, A. Likmeta, A. M. Metelli, A. Tirinzoni, and M. Restelli. Truly batch model- free inverse reinforcement learning about multiple intentions. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2369. PMLR, 26-28 Aug 2020. URL https: httpps:/prooeeednng/ www...rreees....... matass//v08/rrmoon.o html.
Artificial Intelligence
[33] Almingol, L. Montesano, and M. Lopes. Learning multiple behaviors from unlabeled demonstrations in latent controller space. In S. Dasgupta and D. McAllester, editors, Proceed- ings of the 30th International Conference on Machine Learning. volume 28 of Proceedings of Machine Learning Research, pages 136-144, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[34] A. Bighashdel, P. Meletis, P. Jancura, and G. Dubbelman. Deep adaptive multi- inteintnnnoonn in- verse reinforcement learning. In N. Oliver, F. Pérez-Cruz, S. Kramer, Read, and A. Lozano, editors, Machine Learning and Knowledge Discovery in Databases. Research Track, pages 206-221, Cham, 2021. Springer International Publishing.
[35] J. Choi and K.-e. Kim. Nonparametric bayesian inverse reinforcement learning for mul- tiple reward functions. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https: httt:::/pprooceeing met.eeeddnnnnn.nnis...... cc/ppaper/2112/f1l Paper.
[36] S. Rajasekaran, J. Zhang, and J. Fu. Inverse reinforce learning with nonparametric behavior clustering. arXiv preprint 1rXiv:/712.055: 2017.
[37] A. Mendez, S. Shivkumar, and E. Eaton. Lifelong inverse reinforcement learning. In NeurlPS, pages 4507-4518, 2018.
[38] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, et al. Roboturk: crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[39] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and L. Fei-FFi. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1048-1055. IEEE, 2019.
[40] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learn- ing. pages 991-1002, PMLR, 2022.
[41] Kullback and Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 86, 1951.
[42] I. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Probl. Inf. Transm., 23(1-22 (211-)2:95-111, 1987.
[43] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, Schulman, J. Tang, and Zaremba. Openai gym. CoRR, abs/1606.01540 2016. URI. http:://rxxiv.... https/iwwwrrrr//////e/ee/e/o abs///06 01540.
[44] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In Pro- ceedings of the International Conference on Intelligent Robots and Systems (IROS). IEEE/RS], 2012.
[45] C. Ericson.
Real-Time Collision Detection. CRC Press, Inc., USA, 2004.
[46] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
URI. stctps://openreviees htt.ppparrreeeeeeeeeerrrrrraaaaa
11

---

[47] T. Ni, H. Sikchi, Y. Wang. T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforcement
learning via state marginal matching. arXiv preprint arXiv:2011.4470,, 2020.
12


Reviews:
[{"review_id": "wU21ucMP9sv", "reviewer": "Reviewer_bJrX", "summary": "This paper proposes an LfD framework to allow adapting to different user preferences over how a task is carried out. It uses an initial set of demonstrations to build an initial set of policies that correspond to different strategies. Then, users specify how they would like the task to be carried out by adding a demonstration. This demonstration is used to infer a policy mixture of the base set of strategies, or learn a new strategy if the demonstration is sufficiently different from the current set of strategies. In this way, the proposed method can learn continually. The proposed method is demonstrated on 3 simulation environments and a real world table tennis environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe proposed method is an interesting idea that in principle, allows for more strategies to be learned and modeled as more demonstrations are added into the system. Results on the real-world table tennis domain are impressive (especially those shown in the supplementary video). \n\nWeaknesses\n\nOne of the main weaknesses is the experimental evaluation in simulation. The most important aspects of this work, with respect to prior work, appear to be the ability to continually adapt to new strategies (specified by new demonstrations). However, the evaluation of the adherence to strategies is mostly qualitative. It would be very useful to show additional experiments in simulation where there are a collection of ground-truth strategies that are specified by an initial set of humans (similar to the real-world table tennis experiment). Then, quantitative evaluations on how well learned policies adhere to the desired strategy would be possible, instead of the current experiments that rely on unsupervised strategy discovery (through DIAYN, reference 37). From this perspective, it could also be valuable to show results on other domains in simulation where ground-truth strategies can more easily be specified (for example, robotic manipulation, with different grasps possible for objects, or different speeds for trajectories that are executed on the arm).\n\nRelated to this, it would be nice to have more quantitative metrics for evaluating adherence to desired strategy in Sec 5.1 (e.g. returns under the ground-truth reward function for the strategy, not just the general task reward function).\n\nMore comments follow:\n\n- What if it takes more than one demonstration to show a user-specific strategy? Would the method be able to handle this case?\n\n- Sec 4.2 - how many trajectories are needed to estimate the objective and find the policy mixture weights? Do you then need to collect additional trajectories to estimate the KL divergence for line 5 of algorithm 1, once the mixture weights have been found?\n\n- Sec 6 - how is RL run on the real robot in a sample efficient way? Does the human need to do manual resets? More details on this process would be helpful.\n\n- It's a pretty strong claim to put \"crowdsourced demonstrations\" in the title without having extensive evaluation with several humans. \n\n- There are a few missing references for crowdsourced demonstrations in the paper (https://arxiv.org/pdf/1811.02790.pdf, https://arxiv.org/abs/1911.04052, https://arxiv.org/abs/2202.02005).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The main issue is the experimental evaluation - there should be more quantitative (not just qualitative) evidence that the method learns to adhere to new strategies that are demonstrated. If this issue is addressed, I would increase my score.\n\n**Post Author Response**\n\nThanks to the authors for addressing my comments in a thorough manner - I have raised my score to Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The main issue is the experimental evaluation - there should be more quantitative (not just qualitative) evidence that the method learns to adhere to new strategies that are demonstrated.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "tqZc8eR5mFD", "reviewer": "Reviewer_rqFS", "summary": "The paper proposes Fast Lifelong Adaptive IRL (FLAIR), a learning from demonstration framework that aims to maintain a collection of learned strategies that may be mixed in order to model subsequent demonstrations. The method is similar to Multi-Strategy Reward Distillation (MSRD) but does not assume access to a strategy label. Overall, the paper is well-written and provides a thorough experimental evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a new method for constructing a policy mixture for a new demonstration as well as a new objective, Between-Class Discrimination (BCD) that seems to be significantly more effective than existing work at adapting to a sequence of demonstrations.\n- The paper conducts a thorough experimental evaluation, which includes ablations on FLAIR and even a real robot experiment with a table tennis-playing robot. The paper also includes a very thorough appendix. \n\nWeaknesses:\n- It seems like the effectiveness of the method may depend on the range of demonstrations available--if they are not diverse, then a lack of strategies is modeled, and if they are too diverse, the method may learn too many strategies. \n- I'm not sure about the importance of the problem setting, and I think the assumptions made in the problem setting could be motivated better. It's not clear to me how many real world scenarios are actually modeled by this problem statement where individual demonstrations must arrive in sequence. For example, a robot learning to play table tennis will likely have access to a library of demonstrations to start instead of requiring each demonstration to arrive in sequence.\n\nQuestions: \n- As a baseline, what is the performance when not given demos in a sequential manner, e.g. given demos all at once, with AIRL or GAIL? \n- It would be helpful to add titles to the plots in Figure 2 and 3. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, despite some weaknesses, I think this is an interesting paper with a novel method and well-structured experiments. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see weaknesses/questions listed above and address accordingly.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "qL4a2L5J0E", "reviewer": "Reviewer_iYLw", "summary": "This paper presents FLAIR, a new algorithm for lifelong “personalized” inverse RL that can rapidly adapt to new heterogeneous demonstrations. They maintain a set of learned policies corresponding to unique skills in demonstrations encountered so far; new demonstrations are modeled as a mixture of existing policies (if the behavior is captured sufficiently well) or a new policy (if the behavior is not). In three simulated continuous control environments, FLAIR outperforms baselines in adaptability, efficiency, and scalability. A real robot experiment is also performed to evaluate the utility of FLAIR's policy mixture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a strong paper that tackles practical problems in learning from demonstration: lifelong deployment and heterogeneous demonstrations due to varying human preferences. The proposed approach is novel and technically sound, with an intuitive procedure (Algorithm 1) and a novel “Between-Class Discrimination” loss function. Experimental results are organized and presented well, demonstrating a win for FLAIR in adaptability, efficiency, scalability, and policy performance over baseline approaches Adversarial IRL (AIRL) and Multi-Strategy Reward Distillation (MSRD). Experiments suggest FLAIR is a new state-of-the-art approach for lifelong IRL from heterogeneous demonstrations. They also evaluate the quality of FLAIR’s policy mixture in a real robot experiment, and the video supplement gives good visual intuition for the experiment.\n\nWhile strong, the paper has some weaknesses. Firstly, the authors claim that the code and data will be open-sourced, but it currently is not and is not available in the supplement; it would be ideal to have these available in order to reproduce simulation results, for example. Secondly, the clarity of the writing and presentation can be improved; in particular, Figure 1a is very difficult to parse and the text could benefit from proofreading. Thirdly, the results seem to be weaker for Lunar Lander and Bipedal Walker than the (very simple) Inverted Pendulum environment, as the respective figures are relocated to the appendix (Figure 2 in the appendix shows weak correlation compared to Figure 2 in the main text) or absent from both the text and appendix (e.g., the counterpart for Figure 4). Fourthly, it is unclear why the real robot experiment is a different experiment than the simulations (i.e., the FLAIR vs. AIRL vs. MSRD comparison); some clarification from the authors here would be appreciated.\n\nMore minor notes / requests for clarification:\n- I am not sure why “crowdsourced” is emphasized in the title and “democratize access to robotics” is emphasized in the Introduction, as there is no large-scale data collection from crowdsourced humans in this paper.\n- Have the authors considered a “smoothing” as opposed to “filtering” approach in Algorithm 1, which perhaps could recompute mixture weights for old demos with newer policies that were not available at the time?\n- In Section 5.2 under Q6, why does FLAIR recover more than 5 strategies if there are only 5 ground truth policies? Is it approximation error?\n- Which environment is the data in Figure 3 from?\n- It would be nice to include visualizations of the different heterogeneous demonstrations in simulation for better intuition in Figure 4 and other parts of the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend a Weak Accept as this paper makes solid contributions. However, it can be improved by addressing the issues below, and as I am not an expert in personalized IRL I may not be aware of all other existing techniques that may undercut the claim that FLAIR sets a new state-of-the-art.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. Attach code supplement if possible\n2. Provide Lunar Lander and Bipedal Walker counterparts for all Inverted Pendulum figures\n3. Improve Figure 1\n4. Explain why real experiment differs from simulation experiment\n5. Attach visualizations of heterogeneous demonstrations in simulation if possible"}}]

Author rebuttal:

Decision: Accept (poster)