LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
Authors: Bo Liu
Yifeng Zhu
Chongkai Gao
Yihao Feng
qiang liu
Yuke Zhu
Peter Stone
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: lifelong learning
continual learning
multitask learning
robot learning
TL;DR: We present a novel benchmark for studying the transfer of different types of knowledge in lifelong robot learning.

Abstract: Lifelong learning offers a promising paradigm of building a generalist agent that learns and adapts over its lifespan. 
Unlike traditional lifelong learning problems in image and text domains, which primarily involve the transfer of declarative knowledge of entities and concepts, lifelong learning in decision-making (LLDM) also necessitates the transfer of procedural knowledge, such as actions and behaviors. To advance research in LLDM, we introduce LIBERO, a novel benchmark of lifelong learning for robot manipulation. Specifically, LIBERO highlights five key research topics in LLDM: 1) how to efficiently transfer declarative knowledge, procedural knowledge, or the mixture of both; 2) how to design effective policy architectures and 3) effective algorithms for LLDM; 4) the robustness of a lifelong learner with respect to task ordering; and 5) the effect of model pretraining for LLDM. We develop an extendible procedural generation pipeline that can in principle generate infinitely many tasks. For benchmarking purpose, we create four task suites (130 tasks in total) that we use to investigate the above-mentioned research topics. To support sample-efficient learning, we provide high-quality human-teleoperated demonstration data for all tasks. Our extensive experiments present several insightful or even unexpected discoveries: sequential finetuning outperforms existing lifelong learning methods in forward transfer, no single visual encoder architecture excels at all types of knowledge transfer, and naive supervised pretraining can hinder agents' performance in the subsequent LLDM.

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
"oo Liu," Yifeng Zhu", Chongkai Gao", 'Yhhao Feng 'Qiang Liu, Yuke Zhu, f-8 Peter Stone
The University of Texas at Austin, Sony AI, *Tsinghua University {bliu, palluu,iien yifengz, Iqiang, yukez, (stoeee utexas edu yihao. ac0gmail com, gck208mails tsinghua, edu,
Abstract
Lifelong learning offers promising paradigm of building generalist agent that learns and adapts over its lifespan. Unlike traditional lifelong learning problems image and text domains, which primarily involve the transfer of declarative knowl- edge of entities and concepts, lifelong learning in decision-making (LLDM) also necessitates the transfer of procedural knowledge, such actions and behaviors. To advance research in LLDM, we introduce LIBERO, novel benchmark of lifelong learning for robot manipulation. Specifically, LIBERO highlights five key research topics in LLDM: 1) how to efficiently transfer declarative knowledge, procedural knowledge, or the mixture of both: 2) how to design effective policy architectures and 3) effective algorithms for LLDM; 4) the robustness of lifelong learner with respect task ordering, and 5) the effect of model pretraining for LLDM. We develop extendible procedural generation pipeline that can in principle generate infinitely many tasks. For benchmarking purpose, we create four task suites (130 tasks in total) that we use to investigate the above-mentioned research topics. To support sample-efficient learning, we provide high-quality human-teleoperrate demonstration data for all tasks. Our extensive experiments present several insightful or even unexpected discoveries: sequential finetuning outperforms existing lifelong learning methods in forward transfer, single visual encoder architecture excels at all types of knowledge transfer, and naive supervised pretraining can hinder agents' performance in the subsequent LLDM
Introduction
A longstanding goal in machine learning is develop generalist agent that can perform wide range of tasks. While multitask learning 10 is one approach, it is computationally demanding and not adaptable ongoing changes. Lifelong learning [65]. however, offers practical solution by amortizing the learning process over the agent's 's lifespan. Its goal is to leverage prior knowledge to facilitate learning new tasks (forward transfer) and use the newly acquired knowledge enhance performance on prior tasks (backward transfer).
The main body of the lifelong learning literature has focused on how agents transfer declarative knowledge visual or language tasks, which pertains to declarative knowledge about entities and concepts 7 40.. Yet it is understudied how agents transfer knowledge in decision-makin tasks, which involves a mixture of both declarative and procedural knowledge (knowledge about how to do something). Consider a scenario where robot, initially trained to retrieve juice from a fridge, fails
Equual contribution. Check the website at https:://tbbrro httpll/bbrro-oojjct.... http:oopoooocoooooooioooeeeeeeee 10 for the code and the datasets.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

LIF E RO
dccarative knowlodee
Involve procedural knowlodge
Procedural Generation
LIBERO-Gool
LIBERO-Object
Different goals same objects layout
Different layouts, same objects
LIBERO-Spatiol
LIBER-1000
Different objects, same layout
Diverse object, layouns, backgrounds
Distribution Shifts
Algorithmic Designs
Neural Architectures
Task Orderings
Pretraining Effects
task suites: LIBERO- -PPRRALL LIBERO- OBJECT, and LIBERO-GER GOAL have 10 tasks each and require transferring knowledge about spatial relationships, objects, and task goals; LIBERO-100 has 100 tasks and requires the transfer of entangled knowledge. Bottom: we investigate five key research topics in LLDM on LIBERO.
Figure 1:
Top: LIBERO has four procedurally-
after learning new tasks. This could be due to forgetting the juice or fridge's location (declarative knowledge) or how to open the fridge grasp the juice (procedural knowledge). So far, we lack methods to systematically and quantitatively analyze this complex knowledge transfer.
To bridge this research gap, this paper introduces new simulation benchmark, LIfelong learning BEchmark on RObot manipulation tasks, LIBERO, to facilitate the systematic study of lifelong learning in decision making (LLDM). An ideal LLDM testbed should enable continuous learning across an expanding set of diverse tasks that share concepts and actions. LIBERO supports this through procedural generation pipeline for endless task creation, based on robot manipulation tasks with shared visual concepts (declarative knowledge) and interactions (procedural knowledge).
For benchmarking purpose, LIBERO generates 130 inspired by human activities [22 and, grouped into four suites. The four task suites are designed to examine distribution shifts in the object types, the spatial arrangement of objects, the task goals, or the mixture of the previous three (top row of Figure D LIBERO scalable, extendable, and designed explicitly for studying lifelong learning in robot manipulation. To support efficient learning, we provide high-quality, human- temep--teeeperrteedd demonstration data for all 130 tasks.
robot manipulation tasks
We present an initial study using LIBERO to investigate five major research topics in LLDM (Figure []: 1) knowledge transfer with different types of distribution shift; 2) neural architecture design; 3) lifelong learning algorithm design; 4) robustness of the learner to task ordering: and 5) how leverage pre-traines pre-trained models in LLDM (bottom row of Figure D We perform extensive experiments across different policy architectures and different lifelong learning algorithms. Based on our experiments, we make several insightful even unexpected observations:
Policy architecture design as crucial as lifelong learning algorithms. The transformer architecture is better at abstracting temporal information than recurrent neural network. Vision transformers work well on tasks with rich visual information (e.g., a variety of objects). Convolution networks work well when tasks primarily need procedural knowledge.
2. While the lifelong learning algorithms evaluated are effective preventing forgetting, they generally perform worse than sequential finetuning in terms forward transfer.
3. Our experiment shows that using pretrained language embeddings of semantically-rich task descriptions yields performance better than using those of the task IDs.

---

4.
Basic supervised pretraining on large-scale offline dataset can have negative impact on the learner's downstream performance in LLDM..
2
Background
This section introduces the problem formulation and defines key terms used throughout the paper.
2.1 Markov Decision Process for Robot Learning
robot learning problem can be formulated as tnite-horizon Markov Decision Process: M AS T, H, No, R). Here, S and A are the state and action spaces of the robot. fl is the initial state distribution, R. : S A - R. is the reward function, and T S A + S is the transition function. In this work, we assume a sparse-reward setting and replace R with a goal predicate g - 10. The robot's objective to learn policy и that maximizes the expected return:
max (TT Essaaret ,/o 1= [s))].
2.2 Lifelong Robot Learning Problem
In lifelong robot learning problem, robot sequentially learns over K tasks {T', ..... ....,TK]
[ with single policy We assume is conditioned on the task, i.e., .T- s; T) For each task, 7 = (n² g is defined by the initial state distribution v and the goal predicate g² We assume S,A, T, H are the same for all tasks. Up to the k-th task 7*, the robot aims to optimize
I Σ t=1
Σ p=1
E
max (LRL(
E
k
of
(-)))
Ho
An important feature of the lifelong setting is that the agent loses access to the previous ks tasks when it learns on task T
Lifelong Imitation Learning Due to the challenge of sparse-reward reinforcement learning, we consider a practical alternative setting where a user would provide a small demonstration dataset for each task in the sequence. Denote D = {T,] Ji=l as N demonstrations for task T Each TH (og, do, 01. A............................... 0ik) where 1 < H. Here, or is the robot's 's sensory input, including the perceptual observation and the information about the robot's joints and gripper. practice. the observation or is often non- non-Matte Markovian. Therefore, following works in partially observable MDPs [25]
we represent by the aggregated history of observations, i.c. 8t = oce - (oo. .i........)). This results in the lifelong imitation learning problem with the same objective as in Eq. ① But during training, we perform behavioral cloning 4
with the following surrogate objective function:
min (BC(
E DF
Σ 1=0
[(T(OEE:T9))00)
(2)
where is supervised learning loss, e.g., the negative log-likelihood loss, and is Gaussian mixture model. Similarly, we assume {D": pp < kj are not fully available when learning T
3 Research Topics in LLDM
We outline five major research topics in LLDM that motivate the design of LIBERO and our study. (T1) Transfer of Different Types of Knowledge In order to accomplish a task such as pul the ketchup next to the plate in the basket, a robot must understand the concept ketchup, the location of the plate/basket, and how to put the ketchup in the basket. Indeed, robot manipulation tasks in general necessitate different types of knowledge, making it hard to determine the cause failure. We present four task suites in Section 4.2 three task suites for studying the transfer of knowledge about spatial relationships, object concepts, and task goals in a disentangled manner, and one suite for studying the transfer of mixed types of knowledge.
Throughout the paper, superscript/subscriptt is used index the task/time step.
3

---

Scenes
Human activity dataset
Specify Objects And Layouts
(snnjert
Slack, bewl blank,
plane pllte
(1)
(iFtxture
65tcher caole kStcher table cabinet wooden, castieet
13000 burn
A7...
Et Erregiens
(2)
[Roet ragios
(:tarant kitchen sasie)
Init Configuration
Goal States
(3)
Specify Goals
Specify Configuration
Cogas) (and
Sstase
(Gn caciret.: catinet. region)
[E> bewil cabiret tatn, drwwer)
(on plate_1 plate Init regtons-
region)
pddl
(B)
(C)
Figure 2: LIBERO's procedural generation pipeline: Extracting behavioral templates from a large- scale human activity dataset (1), Ego4D, for generating task instructions (2); Based the task description, selecting the scene and generating the PDDL description file (3) that specifies the objects and layouts (A), the initial object configurations (B), and the task goal (C).
(T2) Neural Architecture Design An important research question in LLDM is how to design effective neural architectures to abstract the multi-modal observations (images, language descriptions, and robot states) and transfer only relevant knowledge when learning tasks.
(T3) Lifelong Learning Algorithm Design Given a policy architecture, it is crucial determine what learning algorithms to apply for LLDM. Specifically, the sequential nature of LLDM suggests that even minor forgetting over successive steps can potentially lead to a total failure in execution. As such, we consider the design of lifelong learning algorithms to be open area of research in LLDM.
(T4) Robustness Task Ordering
is well-known that task curriculum influences policy learning 6 48). A robot in the real world. however, often cannot choose which task to encounter first. Therefore, good lifelong learning algorithm should be robust to different task orderings.
(T5) Usage of Pretrained Models In practice, robots will be most likely pretrained on large datasets in factories before deployment (28) However, it is not well-understood whether how pretraining could benefit subsequent LLDM.
4 LIBERO
This section introduces the components in LIBERO: the procedural generation pipeline that allows the never-ending creation of tasks (Section 4.1. the four task suites we generate for benchmarking (Section five algorithms (Section and three neural architectures (Section
4.1 Procedural Generation of Tasks
Research in LLDM requires a systematic way create new tasks while maintaining task diversity and relevance to existing tasks. LIBERO procedurally generates new tasks in three steps: 1) extract behavioral templates from language annotations of human activities and generate sampled tasks described in natural language based on such templates; 2) specify an initial object distribution given task description; and 3) specify task goals using a propositional formula that aligns with the language instructions. Our generation pipeline is built on top of Robosuite te (76,, modular manipulation simulator that offers seamless integration. Figure 2 illustrates an example of task creation using this pipeline, and each component is expanded upon below.
Behavioral Templates and Instruction Generation Human activities serve fertile source of tasks that can inspire and generate vast number of manipulation tasks. We choose large- scale activity dataset, Ego4D [22]. which includes a large variety of everyday activities with language annotations. We pre-process the dataset by extracting the language descriptions and then summarize them into large set of commonly used language templates. After this pre-processing step, we use the templates and select objects available in the simulator generate set of task descriptions in the

---

form of language instructions. For example, we can generate an instruction "Open the drawer of the cabinet" from the template "Open...
Initial State Distribution (10) To specify (40, we first sample a scene layout that matches the objects/hehaviors in provided instruction. For instance, kitchen scene is selected for instruction Open the top drawer of the cabinet and put the bowl in it. Then, the details about fo are generated in the PDDL language [3 63]. Concretely, po contains information about object categories and their placement (Figure (A)), and their initial status (Figure (B)).
Goal Specifications (g) Based on Mi and the language instruction, we specify the task goal using a conjunction of predicates. Predicates include wnary predicates that describe the properties of an object, such Open(X) or TurnOff(X), and binary predicates that describe spatial relations between objects, such as On(A, A, B) or In(A, B). An example of the goal specification using PDDI. language can be found in Figure (C). The simulation terminates when all predicates are verified true.
4.2 Task Suites
While the pipeline in Section 4. supports the generation of an unlimited number of tasks, we offer fixed sets of tasks for benchmarking purposes. LIBERO has four task suites: LLIBERO-SPATIAN LIBERO-OBJECT,
EIBERO-GOAL and LIBERO-100. The first three task suites are curated to disentangle the transfer of declarative and procedural knowledge (as mentioned in (TI)), while LIBERO-100 is a suite of 100 tasks with entangled knowledge transfer.
LIBERO-X LIBERO-SPATIAL, LIBERO-OBJECH and LIBERO-GOAL all have 10 tasks and are designed to investigate the controlled transfer of knowledge about spatial information (declarative), objects (declarative), and task goals (procedural). Specifically, all tasks in LIBERO- SPATIAL request the robot to place a bowl, among the same set of objects, on a plate. But there are two identical bowls that differ only in their location or spatial relationship to other objects. Hence, to successfully complete spatial relationships. All tasks in Hence, to accomplish LIBERO- OORROT,,, the robot needs to continually learn and memorize new object types. All tasks in LIBERO-GOAL share the same objects with fixed spatial relationships but differ only in the task goal. Hence, to accomplish LIBERO-GOAA learn new knowledge about motions and behaviors. More details are in Appendix C
the robot needs to continually learn and memorize new request the robot to pick-place a unique object.
the robot needs to continually
LIBERO-100 LIBERO-100 100 contains 100 tasks that entail diverse object interactions and versatile motor skills. In this paper, we split LIBERO- 100 into short-horizon tasks (LIBERO-90) and 10 long-horizon tasks (LIBERO-L.ON) LIBERO-90 serves as the data source for pretraining (T5)
and LIBERO-LONG for downstream evaluation of lifelong learning algorithms.
4.3 Lifelong Learning Algorithms
We implement three representative lifelong learning algorithms to facilitate research in algorith- mic design for LLDM. Specifically, we implement Experience Replay (ER) [I3. Elastic Weight Consolidation (EWC) BR and PACKNET 4T We pick ER, EWC, and PACKNET because they correspond to the memory-basee based, regularization-bat based, and sinamic-rrchitecuur--asss methods for lifelong learning. In addition, prior research (69 has discovered that they are state-of-the-arr methods. Besides these three methods, we also implement sequential finetuning (SEQL) and multitask learning (MTL), which serve as lower bound and upper bound for lifelong learning algorithms, respectively. More details about the algorithms are in Appendix B.I
4.4 Neural Network Architectures
We implement three vision-language policy networks, RESNET-RNN, RESNET-T, and VIT-T, that integrate visual, temporal, and linguistic information for LLDM. Language instructions of tasks are encoded using pretrained BERT embeddings 19. The RESNET-RNN 42 uses a ResNer as the visual backbone that encodes per-step visual observations and an LSTM as the temporal backbone to process sequence of encoded visual information. The language instruction is incorporated into the ResNet features using the FiLM method (50) and added the LSTM inputs, respectively. RESNET-T
A suite of 10 tasks is enough to observe catastrophic forgetting while maintaining computation efficiency.
5

---

uses similar BesNet-bbsed visual backbone, but a transformer decoder [66] as the temporal backbone to process outputs from ResNet, which are temporal sequence of visual tokens. The language embedding is treated as a separate token in inputs to the transformer alongside the visual tokens. The VrT-T architecture (31, which is widely used in visual-language tasks, uses a Vision Transformer (ViT) as the visual backbone and transformer decoder the temporal backbone. The language embedding is treated as a separate token in inputs of both ViT and the transformer decoder. All the temporal backbones output a latent vector for every recision-making step. We compute the multi-modal distribution over manipulation actions using a Geausian-Mixture-MMan (GMM) based output head R
architecture 751
In the end, robot executes policy by sampling continuous value for end-effector action from the output distribution. Figure 6 visualizes the three architectures.
42 [8
For all the lifelong learning algorithms and neural architectures, we use behavioral cloning (BC) ! to train policies for individual tasks (See ( BC allows for efficient policy learning such that we can study lifelong learning algorithms with limited computational resources. To train BC, we provide 50 trajectories of high-quality demonstrations for every single task in the generated task suites. The demonstrations are collected by human experts through telcoperation with 3Dconnexion Spacemouse.
5 Experiments
Experiments are conducted as an initial study for the five research topics mentioned Section 3 We first introduce the evaluation metric used in experiments, and present analysis of empirical results in
LIBERO. The detailed experimental setup is in Appendix D Our experiments focus on addressing the following research questions:
Q1: How do different architectures/L.I LL algorithms perform under specific distribution shifts?
Q2: To what extent does neural architecture impact knowledge transfer in LLDM, and are there any discernible patterns in the specialized capabilities of each architecture?
Q3: How do existing algorithms from lifelong supervised learning perform LLDM tasks?
Q4: To what extent does language embedding affect knowledge transfer in LLDM?
Q5: How robust are different LL algorithms to task ordering in LLDM?
Q6: Can supervised pretraining improve downstream lifelong learning performance in LLDM?
5.1 Evaluation Metrics
We report three metrics: FWT (forward transfer) 20
NBT (negative backward transfer), and AUC (area under the success rate curve). All metries are computed in terms of success rate, as previous literature has shown that the success rate is more reliable metric than training loss for manipulation policies [4 (Detailed explanation in Appendix E.2). Lower NBT means a policy has better performance in the previously seen tasks, higher FWT means policy learns faster new task, and higher AUC means an overall better performance considering both NBT and FWT. Specifically, denote Cr.j.e as the agent's 's success rate task j when it learned over i previous tasks and has just learned epochs (e E (0, 50..,....,5) on task i. Let Gr.i be the best success rate over all evaluated epochs for the current task i (i.e., C,, max, C. Then, we find the earliest epoch et in which the agent achieves the best performance on task i (i.e., ef = arg min, CLi,e; c, and
assume for all e 2 e,, Ciiie = +,
Given a different task j F i, we define C.j = G.jet Then the
three metrics are defined:
FWT
Σ KE[K]
FWT K
FWT,
1
Σ
Ck.kee
11
eE (E(U...0))
NBT Σ
NBT
NBT,
1
(CH.k Cr,Rk
K
K
(3)
KE[K]
T=k+1
AUC
1
AUC Σ
AUC
K +1 (FWT +
Σ T=k+1
Cr.b
K
RE[K]
'In practice, it's possible that the agent's performance on task i is not monotonically increasing due to the variance of learning. But we keep the best checkpoint among those saved at epochs {e} as if the agent stops learning after ei.

---

A visualization of these metrics is provided in Figure B
Success Rate
FWT
NBT
AUC = K K-K+1
K-k
Task
k-1
k
K
Figure 3: Metrics for LLDMM.
5.2 Experimental Results
present empirical results to address research questions. Please refer to Appendix for full results across all algorithms, policy architectures, and task suites.
Study on the Policy's Neural Architectures (Q1, Q2) Table reports the agent's lifelong learning performance using the three different neural architectures on the four task suites. Results are reported when ER and PACKNET are used they demonstrate the best lifelong learning performance across all task suites.
ER
PACKNET
Policy Arch.
FWT(1)
NBT())
AUC(1)
FWT())
NBT())
AUC(f)
LIBERO-
RESNET-RNN
0.16 1 0.02
0.16 1 0.02
0.08 +: 0.01
0.13 1 0.00
0.21 1.01
0.03 1: 0.00
RESNET-T
0.48 1 0.02
0.32 0.04
0.32 1:00:0001
0.22 1 0.01
0.08 1 0.01
0.25 1 0.00
VIT-
0.38 1 0.05
0.29 0.06
0.25 - 0.02
0.36 0.01
0.01
0.34
LIBERO SPATIAL
RESNET-RNN
0.40 + 0.02
0.29 = 0.02
0.29 1.00
0.27 + 0,03
0.38 + 0.03
0.06 + 0.01
RESNET-T
0.65 1 0.03
0.27 0.03
0.56
0.55 + 0.01
0.07 + 0.02
0.63 0.00
VIT-T
0.63 + 0.01
0.29 0.02
0.02
0.57 + 0.04
0.15 0.00
0.59
LIBERO-OBJECT
RESNET-RNN
0.30 + 0.01
0.27 = 0.05
0.17 0.05
0.29 + 0.02
0.35 = 0.02
0.13 + 0.01
RESNET-T
0.67 + 0.07
0.43 0.04
0.44 + 0.06
0.60 + 0.07
0.17 = 0.05
0.60 0.05
VIT-T
0.70 + 0.02
0.28 0.01
0.57 0.01
0.58 + 0.03
0.18 + 0.02
0.56
LIBERO-GOAL.
RESNET-RNN
0.41 + 0.00
0.35 + 0.01
0.26 + 0.01
0.32 + 0.03
0.37 + 0.04
0.11 0.01
RESNET-T
0.64 + 0.01
0.34 0.02
0.49 0.02
0.63 + 0.02
0.06 0.01
0.75 0.01
VIT-T
0.57 + 0.00
0.40
0.02
0.38 0.01
0.69 + 0.02
0.08 + 0.01
0.76 0.02
Table 1: Performance of the three neural architectures using ER and PACKNET on the four task suites. Results averaged over three seeds and we report the mean and standard error. The best performance is bolded, and colored in purple if the improvement is statistically significant over other neural architectures, when two-tailed, Student's t-test under equal sample sizes and unequal variance is applied with p-value of 0.05.
Findings: First, we observe that RESNET-T and VIT-T work much better than RESNET-RNN on average, indicating that using a transformer the "temporal" level could be better option than using an RNN model. Second, the performance difference among different architectures depends on the underlying lifelong learning algorithm. If PACKNET (a dynamic architecture approach) is

---

used, we observe no significant performance difference between RESNET-T and VIT-T except on the LIBERO-LONG task suite where VIT-T performs much better than RESNET-T. In contrast, if ER is used, we observe that RESNET-T performs better than VrT-T on all task suites except LIBERO-OB/RCE This potentially indicates that the ViT architecture is better at processing visual information with more object varieties than the ResNet architecture when the network capacity is sufficiently large (See the MTL results in Table g on LILBER--BJJG as the supporting evidence). The above findings shed light on how one can improve architecture design for better processing of spatial and temporal information LLDM.
Study on Lifelong Learning Algorithms (Q1, Q3) Table 2 reports the lifelong learning per- formance of the three lifelong learning algorithms, together with the SEQL and MTL baselines. All experiments use the same RESNET-T architecture as it performs the best across all policy architectures.
Lifelong Algo.
FWT(1)
NBT()
AUC(1)
FWT(1)
NBT())
AUC(1)
LIBERO-LONG
LIBERO-SPATIE
SEQI.
0.54 0.01
0.63 0.01
0.01
0.15 0.00
0.72 0.01
0.81 0.01
0.01
0.20 0.01
ER
0.48 0.02
0.32 + 0.04
0.32 + 0.01
0.65 0.03
0.27 + 0.03
0.56 + 0.01
EWC
0.13 + 0.02
0.22 + 0.03
0.02 + 0.00
0.23 123+000 0.01
0.33 + 0.01
0.06 + 0.01
PACKNET
0.22 0.01
0.08 + 0.01
0.25 = 0.00
0.55 +
0.07 + 0.02
0,63 + 0.00
MTL.
0.48 0.01
0.83 0.00
LIBERO-OBIECH
LIBERO-GOAT
SEQL
0.78 +: 0.04
0.76 1 0.04
0.26 + 0.02
0.77 0.01
0.82 1 0.01
0.22 i 0.00
ER
0.67 0.07
0.43 + 0.04
0.44 0.06
0.64
0.34 1 0.02
0.49 1 0.02
EWC
0.56 0.03
0.69 1 0.02
0.16 = 0.02
0.32
0.48 1 0.03
0.06 0.00
PACKNET
0.60 0.07
0.17 1 0.05
0.60 1 0.05
0.63 0.02
0.06 1 0.01
0.75 1 0.01
MTL
0.54 - 0.02
0,80 1 0.01
Table 2: Performance of three lifelong algorithms and the SEQL. and MTL baselines the four task suites, where the policy is fixed be RESNET-T. Results are averaged over three seeds and we report the mean and standard error. The best performance is bolded, and colored in purple if the improvement is statistically significant over other algorithms, when a two-tailed, Student's 's t-test under equal sample sizes and unequal variance is applied with p-value of 0.05.
Findings: We observed a series of interesting findings that could potentially benefit future research
on algorithm design LLDM: 1) SEQL shows the best FWT over all task suites. This is surprising since it indicates all lifelong learning algorithms consider actually hurt forward transfer; 2) PACKNET outperforms other lifelong lcarning algorithms on LIBERO-X but outperformed by ER significantly LIBERO-LONG, mainly because of low forward transfer. This confirms that the dynamic architecture approach good at preventing forgetting. But since PACKNET splits the network into different sub-networks, the essential capacity of the network for learning any individual task is smaller. Therefore, we conjecture that PACKNET not rich enough to learn on LIBERO- LONG; 3) EWC works worse than SEQL. showing that the regularization the loss term actually impede the agent's 's performance on LLDM problems (See Appendix E.2 ; and 4) ER, the rehearsal method, is robust across all task suites.
Study on Language Embeddings as the Task Identifier (Q4) To investigate to what extent language embedding play role in LLDM, we compare the performance of the same lifelong learner using four different pretrained language embeddings. Namely, we choose BERT [1), CLIP [52], GPT-2 3 and the Task-ID embedding. Task-ID embeddings produced by feeding a string such as "Task 5" into pretrained BERT model.
Findings: From Table 3. we observe no statistically significant difference among various language embeddings, including the Task-ID embedding. This, we believe, is due to sentence embeddings functioning bag-of-word, that differentiates different tasks. This insight calls for better language encoding to harness the semantic information in task descriptions. Despite the similar performance, opt for BERT embeddings our default task embedding.

---

Embedding Type Dimension
FWT(1)
NBT())
AUC(1)
BERT
768
0.48 ::88:000 0.02
1.004
0.32 0.01
CLIP
512
0.52
0.04
0.35 0.01
GPT-2
768
0,46
0.34
0.30
Task-ID
768
0.50 0.01
0.37 0.01
0.33 0.01
Table 3: Performance of lifelong learner using four different language embeddings LIBERO- LONG, where we fix the policy architecture to RESNET-T and the lifelong learning algorithm to ER. The Task-ID embeddings are averaged over three seeds and we report the mean and standard error. The best performance is bolded. No statistically significant difference is observed among the different language embeddings.
retrieved by feeding "Task ID" into pretrained BERT model. Results
Study on task ordering (Q5) Figure shows the result of the study on Q4. For all experiments in this study, used RESNET-T the neural architecture and evaluated both ER and PACKNET. As the figure illustrates, the performance of both algorithms varies across different task orderings. This finding highlights important direction for future research: developing algorithms or architectures that are robust to varying task orderings.
Success Rate
0.4
0.32
0.30
0.31
0.28
0.27
0.26
0.24
0.25
0.2
AUC (5 orderings) ER ResNet-T
AUC (5 orderings)
PackNet ResNet-T
Figure 4: Performance of ER and PACKNET using RESNET-T on five different task orderings. An bar shows the performance standard deviation for fixed ordering.
Findings: From Figure 4 we observe that indeed different task ordering could result in very different performances for the same algorithm. Specifically, such difference is statistically significant for PACKNET.
Study on How Pretraining Affects Downstream LLDM (Q6) Fig 5 reports the results on LIBERO-LONG of five combinations of algorithms and policy architectures, when the underlying model is pretrained on the 90 short-horizion tasks in ERB- 100-000 00 or learned from scratch. For pretraining, apply behavioral cloning on the 90 tasks using the three policy architectures for 50 epochs. We save a checkpoint every 5 epochs of training and then pick the checkpoint for each architecture that has the best performance as the pretrained model for downstream LLDM.
Success Rate
w/ pretraining
w/o pretraining
multitask
0.6
0.48
0.46
0.4
0:32
0.2
0.20
0.22
0.25
0,17
0.25
0.08
0.01
0.02
0.0
ER
ER
ER VIT-T
EWC ResNet-T
PackNet ResNet-T
ResNet-RNN
ResNet-T
Figure 5; Performance of different combinations of algorithms and architectures without pretraining or with pretraining. The multi-task learning performance also included for reference.
Findings: observe that the basic supervised pretraining can hurt the model's downstream lifelong learning performance. This, together with the results seen in Table 2 (c.g., naive sequential fine-tuning

---

has better forward transfer than when lifelong learning algorithms are applied), indicates that better pretraining techniques are needed.
To better understand what type of knowledge the agent forgets during the lifelong learning process, visualize the agent's attention map on each observed image input. The visualized saliency maps and the discussion can be found in Appendix E.4
Attention Visualization:
Related Work
This section provides an overview of existing benchmarks for lifelong learning and robot learning. We refer the reader to Appendix B.1 for a detailed review of lifelong learning algorithms.
Lifelong Learning Benchmarks
Pioneering work has adapted standard vision or language datasets for studying LL. This line of work includes image classification datasets like MNIST [8). CIFAR 1 and ImageNet [17; segmentation datasets like Core50 38; and natural language understanding datasets like GLUE [67] and SuperGLUE [59]. Besides supervised learning datasets, video game benchmarks (e.g., Atari [46]. XLand 64 and VisDoom 30 in reinforcement learning (RL.) have also been used for studying LL.. However, LL in standard supervised learning does not involve procedural knowledge transfer, while RL problems in games do not represent human activities. Continual World [69] modifies the 50 manipulation tasks in Meta MeraWorld for LL. CORA [51] builds four lifelong RL benchmarks based on Atari, Procgen [TS. MiniHack [58). and ALFRED [62]
F-SIOL-3 10 [3] and OpenLORIS 6l are challenging real-world lifelong object learning datasets that are captured from robotic vision systems. Prior works have also analyzed different components in LL agent
45 70 21]
but they do not focus on robot manipulation problems.
Robot Learning Benchmarks A variety of robot learning benchmarks have been proposed to address challenges in meta learning (MetaWorld (73). causality learning (Causa/Workl (I), multi-task learning 227 35), policy generalization to unseen objects 47 24
and compositional
learning 44
Compared to existing benchmarks in lifelong learning and robot learning, the task
suites in LIBERO are curated to address the research topies of LLDM. The benchmark includes large number of tasks based on everyday human activities that feature rich interactive behaviors with a diverse range of objects. Additionally, the tasks in LIBERO are procedurally generated, making the benchmark scalable and adaptable. Moreover, the provided high-quality human demonstration dataset LIBERO supports and encourages learning efficiency.
7 Conclusion and Limitations
This paper introduces LIBERO, new benchmark in the robot manipulation domain for supporting research LLDM. LIBERO includes a procedural generation pipeline that can create an infinite number of manipulation tasks in the simulator. We use this pipeline to create 130 standardized tasks and conduct a comprehensive set experiments on policy and algorithm designs. The empirical results suggest several future research directions: 1) how to design a better neural architecture to better process spatial information or temporal information; 2) how to design better algorithm to improve forward transfer ability; and 3) how to use pretraining to help improve lifelong learning performance. In the short term, we do not envision any negative societal impacts triggered by LIBERO. But as the lifelong learner mainly learns from humans, studying how to preserve user privacy within LLDM 36 is crucial in the long run.
References
[1] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Yoshua Bengio, Bern- hard Schölkopf, Manuel Wüthrich, and Stefan Bauer. Causalworld: A robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXiv:2010.04250 2020.
[2] Ali Ayub and Carter Fendley. Few-shot continual active learning by a robot. arXiv preprint 1rXiv:2210004:1 2022.
[3] Ali Ayub and Alan R Wagner. F-siol-310: A robotic dataset and benchmark for few-shot incre- mental object learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 13496-13502. IEEE, 2021.
10

---

[4] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15. pages 103-129, 1995.
[5] Eseoghene Ben-Iwhiwhu, Saptarshi Nath, Praveen K Pilly, Soheil Kolouri, and Andrea Soltog- gio. Lifelong reinforcement learning with modulating masks. arXiv preprint arXiv:2212.11110, 2022.
[6] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning, pages 41-48, 2009.
[7] Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey. arXiv preprint arXiv: arXiv:2012.09823, 2020 2020.
[8] Christopher M Bishop. Mixture density networks. 1994.
[9] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
[10] Rich Caruana. Multitask learning, Machine learning, 28(1):41-75, 1997.
[11] Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pages 532-547, 2018.
[12] Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. arXiv preprint arXiv: /8/2.0042/ 2018.
[13] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint S9G.::9992.144800 2019.
[14] Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Su- perposition of many models into one. Advances neural information processing systems, 32, 2019.
[15] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-2055 2056. PMLR, 2020.
[16] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. JEEE transactions on pattern analysis and machine intelligence, 44(7):3366 3385, 2021.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[18] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2916)141 2012.
[19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 04800 2018.
[20] Natalia Díaz Rodrigz,, Vincenzo Lomonaco. David Filliat, and Davide Maltoni. Don't forget, there is more than forgetting: new metrics for continual learning. arXiv preprint 18 13/66, 2018.
[21] Beyza Ermis, Giovanni Zappella, Martin Wistuba, and Cédric Archambeau. Memory efficient continual learning with transformers. 2022.

---

[22] Kristen Grauman, Andrew Westbury, Eugene Byme. Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger. Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 18995-19012, 2022.
[23] Sam Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. ArXiv, abs/1711 11000188, 2017.
[24] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprini arXiv:2302.04653 2023.
[25] Matthew Hausknecht and Peter Stone. Deep recurrent q q-learning for partially observable mdps. In 2015 aaai fall symposiun series, 2015.
[26] Ching-Yt Yi Hung, Cheng-Har -aa Tu, Cheng-Eu En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[27] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026 2020.
[28] Leslie Pack Kaelbling, The foundation of efficient robot learning. Science, 36966506):95--116 2020.
[29] Minsoo Kang, Jaeyoo Park, and Bohyung Han. Class-incremental learning by knowledge distillation with adaptive feature consolidation. Proceedings of the IEEF/CVF conference on computer vision and pullern recognition, pages 16071-16080, 2022.
[30] Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. Vizdoom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE conference on computational intelligence and games (CIG), pages 1-8. IEEE, 2016.
[31] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languuage transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583 5594. PMLR, 2021.
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[33] James Kirkpatrick. Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-352, 2017
et al.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior- 11: benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[36] B. Liu, Qian Liu, and Peter Stone. Continual learning and private unlearning. In CoLLAs, 2022.
[37] Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. arXiv preprint arXiv:2201. 12,222 2022.
[38] Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17-26. PMLR, 2017.
[39] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
12

---

[40] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing. 469:28-51, 2022.
[41] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 7765-7773, 2018.
[42] Ajay Mandlekar. Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martn.. What matters learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: (08::2/08,03399, 2021.
[43] Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins. Padl-the planning domain definition language. 1998.
[44] Jorge A Mendez, Marcel Hussing. Meghna Gummadi, and Eric Faton. Composuite: A compo- sitional reinforcement learning benchmark. arXiv preprint arXiv:2207.044 220..04336..66, 2022.
[45] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXiv preprint arXiv:2202.00275, 2022.
[46] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.v:13 2013.
[47] Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, Stone Tao, Zhiao Huang, Zhiwei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2 14483, 2021.
[48] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. arXiv preprint MXXi::0003.4496 2020.
[49] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[50] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[51] Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics as platform for continual reinforcement learning agents. arXiv preprint a0. 10067, 2021.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763, PMLR, 2021.
[53] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019.
[54] Amanda Rios and Laurent Itti. Lifelong learning without a task oracle. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), pages 255-263. IEEE, 2020.
[55] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourleenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
13

---

[56] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer. James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks, arXiv preprini arXiv: (16666606771 2016.
[57] Gobinda Saha, Isha Garg, Aayush Ankit, and Kaushik Roy. Space: Structured compression and sharing of representational space for continual learning. IEEE Access, 1154800-15494,, 2021.
[58] Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel. Minihack the planet: A sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109.13200 2021.
[59] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4938 4947, 2020.
[60] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwinsk. Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4528 4533. PMLR, 2018
[61] Qi She, Fan Feng, Xinyue Hao, Qihan Yang. Chuanlin Lan, Vincenzo Lomonaco, Xuesong Shi, Zhengwei Wang, Yao Guo, Yimin Zhang, et al. Openloris-object. robotic vision dataset and benchmark for lifelong deep learning. In 2020 IEEE international conference on robotics and automation (ICRA), pages 4767-4773, IEEE, 2020.
[62] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[63] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent El- liott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference Robot Learning, pages 477-490. PMLR, 2022.
[64] Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebac7, Max Jaderberg, Michael Mathieu, al. Open- ended learning leads to generally capable agents. arXiv preprint 107.::201..1800000 2021.
[65] Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1 21:22)2:-444 46, 1995.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[67] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804::8800000 07461, 2018
[68] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Li Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizen imitation learning by watching human play. arXiv preprint 232. 2023.
[69] Maciej Wolczyk, Michal Zajkac, Razvan Pascanu, Lukasz Kuci Kuci'nski, and Piotr Milo's. Continual world: A robotic benchmark for continual reinforcement learning. In Neural Information Processing Systems, 2021.
[70] Maciej Wolezyk, Michal Zajkac, Razvan Pascanu, Lukasz Kuci' and Piotr Milo's. Disen- tangling transfer in continual reinforcement learning. ArXiv, abs/2209, 2ttp/2099.19000.... 2022.
14

---

[71] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: general approach for growing neural networks. Advances in Neural Information Processing Systems, 33:22373-22388,, 2020.
[72] Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 01588, 2017.
[73] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robol learning, pages 1094-1100, PMLR, 2020.
[74] Da-Wei Zhou, Fu- Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9046-9996 9056, 2022.
[75] Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. Viola: Imitation learning for vision- based manipulation with object proposal priors. arXiv preprint arXiv:2210.1 13399 2022.
[76] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martín-Martin. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arX1v:2009.12293, 2020.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "2yiww1ykOA", "reviewer": "Reviewer_3N9d", "summary": "", "questions": "", "limitations": "It remains a bit unclear of how challenging it would be to create new environments and generate own datasets. Perhaps this is because I have personally never worked with this kind of simulator before. Perhaps it is due to the descriptions of related work and respective documentation on creating custom datasets (which is just a small paragraph) being too short (see below). In either way, there seems to be room for improvement here. \nIn similar spirit, it is unclear how easy/feasible the outlined procedure in section 4.1 for procedural generation of tasks is for prospective users to mimic. ", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper has a lot of strengths, listing some of them: \n* It is decently motivated, well structured and written. \n* The four task suits that are designed to examine shift in object types, spatial arrangement of objects, the task goals, and their mixtures enable a more principled evaluation and benchmarking of strategies than some prior benchmarks have (that simply concatenate some existing datasets). \n* The paper uses the proposed tool to investigate a reasonable amount of interesting questions in an empirically decent first examination. \n* The website and documentation are nicely structured for this first version of LIBERO. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ZaYOna2HRs", "reviewer": "Reviewer_XKZ6", "summary": "", "questions": "", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The benchmark is thoughtfully designed and flexible/extendable.\n- A broad range of baselines are studied and several preliminary findings are presented, suggesting plenty of future research directions.\n- The experimental analysis is rigorous. For example, the statistical significance of each result is provided.\n- The paper is clearly written and articulate.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JOwrtItOjM", "reviewer": "Reviewer_kBSE", "summary": "", "questions": "", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "(1) The procedural generation pipeline enables creating a large number of diverse and customizable tasks for studying lifelong robot learning. This is more scalable and flexible than curating tasks manually.\n\n(2) The task suites are thoughtfully designed to disentangle different factors like objects, spatial relationships, goals, etc. This will allow targeted study of various knowledge transfer capabilities.\n\n(3) I have been using the benchmark for quite a while, and the authors seem to be pretty responsible for maintaining it.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ugwiz6T3fL", "reviewer": "Reviewer_2kT5", "summary": "", "questions": "", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is generally well-written.\n2. There is a need to build a robotics benchmark for lifelong learning and the authors construct the benchmark in a reasonable manner.\n3. Offers good analysis and identifies common issues for the current paradigm.\n4.  The paper supply baselines cover different training strategies and different network structures, which are a good starting point to improve upon. \n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zZjzeF2RXq", "reviewer": "Reviewer_5K6Q", "summary": "", "questions": "", "limitations": "- one limitation is the variety of tasks - it seems only pick-and-place and opening drawer tasks are involved in the benchmark. It is not clear whether the findings can be applied to other tasks, e.g. manipulating deformable objects, pouring liquid into a container\n\n- since the benchmark involves realistic indoor scenes and household objects, it would be great to see some real-robot experiments\n\n- collecting human demonstrations is known to be expensive, and the benchmark only provides 50 demonstrations for each task - it is not clear whether these data are sufficient to learn a good policy, especially for the long horizon tasks", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ the problem setup of libero is novel and meaningful: previous works in this direction mainly assume multi-task learning, exploring lifelong learning in the context of language-conditioned robot manipulation is an interesting direction for the community\n\n+ it seems the benchmark contains a variety of realistic indoor scenes and household objects\n\n+ perform extensive experiments to examine the effects of multiple factors critical to the performance\n\n+ the supporting documents provide additional experiment details and experiment results to ensure reproducibility; the codebase also seems to be solid and sufficiently documented\n\n\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Reviewers agreed that this work was clearly written and presented a high-quality benchmark that has the potential to be useful to the robot manipulation community. Feedback in response to reviewer’s specific questions should be incorporated in any final version of the paper, particularly including the additional table of comparisons, information about the process of creating a new task, and discussions of future work as it pertains to reinforcement learning.

Pros:
+ Benchmark is useful, contains important and realistic scenarios, and expands on existing benchmarks in well-defined ways.
+ Clearly written paper on an important topic.
+ Documentation for benchmark use is clear and well maintained.
+ Benchmark can be expanded with new tasks over time.

Cons:
- Please incorporate comparisons to other benchmarks as per response to reviewers.
- Discussion of, and ideally examples of, full reinforcement learning would strengthen the benchmark.
- The negative impacts of pre-training should be more clearly analyzed and explained.
- Real robot experiments would help support claims of applicability.
Decision: Accept (Poster)