Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
Authors: Xiaogang Jia
Denis Blessing
Xinkai Jiang
Moritz Reuss
Atalay Donat
Rudolf Lioutikov
Gerhard Neumann
Venue: ICLR 2024 poster (2024)
Keywords: Imitation Learning
Benchmark
Datasets
Diverse Behaviors
Primary area: datasets and benchmarks
TL;DR: A simulation benchmark with diverse human demonstrations, and tractable metrics to evaluate imitation learning's ability to capture multi-modal behavior.

Abstract: Imitation learning with human data has demonstrated remarkable success in teaching robots in a wide range of skills. However, the inherent diversity in human behavior leads to the emergence of multi-modal data distributions, thereby presenting a formidable challenge for existing imitation learning algorithms. Quantifying a model's capacity to capture and replicate this diversity effectively is still an open problem. In this work, we introduce simulation benchmark environments and the corresponding *Datasets with Diverse human Demonstrations for Imitation Learning (D3IL)*, designed explicitly to evaluate a model's ability to learn multi-modal behavior. Our environments are designed to involve multiple sub-tasks that need to be solved, consider manipulation of multiple objects which increases the diversity of the behavior and can only be solved by policies that rely on closed loop sensory feedback. Other available datasets are missing at least one of these challenging properties.
To address the challenge of diversity quantification, we introduce tractable metrics that provide valuable insights into a model's ability to acquire and reproduce diverse behaviors. These metrics offer a practical means to assess the robustness and versatility of imitation learning algorithms. Furthermore, we conduct a thorough evaluation of state-of-the-art methods on the proposed task suite. This evaluation serves as a benchmark for assessing their capability to learn diverse behaviors. Our findings shed light on the effectiveness of these methods in tackling the intricate problem of capturing and generalizing multi-modal human behaviors, offering a valuable reference for the design of future imitation learning algorithms.

Published as a conference paper at ICLR 2024
TOWARDS DIVERSE BEHAVIORS: A BENCHMARK FOR IMITATION LEARNING WITH HUMAN DEMONSTRA- TIONS
Xiaogang Jia 12 Denis Blessing' Xinkai Jiang Moritz Reuss
Atalay Donat Rudolf Lioutikov Gerhard Neumann
i Autonomous Learning Robots, Karlsruhe Institute of Technology
f Intuitive Robots Lab, Karlsruhe Institute of Technology
ABSTRACT
Imitation learning with human data has demonstrated remarkable success in teach- ing robots in a wide range of skills. However, the inherent diversity in human be- havior leads to the emergence of multi-modal data distributions, thereby present- ing a formidable challenge for existing imitation learning algorithms. Quantifying model's capacity to capture and replicate this diversity effectively still an open problem. In this work, we introduce simulation benchmark environments and the corresponding Datasets with Diverse human Demonstrations for Imitation Learn- ing (D3IL). designed explicitly to evaluate model's ability to learn multi-modal behavior. Our environments are designed involve multiple sub-tasks that need be solved, consider manipulation of multiple objects which increases the diversity of the behavior and can only be solved by policies that rely on closed loop sensory feedback. Other available datasets are missing at least one of these challeng- ing properties. To address the challenge of diversity quantification, we introduce tractable metries that provide valuable insights into model's ability to acquire and reproduce diverse behaviors. These metrics offer a practical means to assess the robustness and versatility of imitation learning algorithms. Furthermore, we conduct a thorough evaluation of state-of-the-art methods on the proposed task suite. This evaluation serves benchmark for assessing their capability learn diverse behaviors. Our findings shed light on the effectiveness of these methods in tackling the intricate problem of capturing and generalizing multi-modal human behaviors, offering a valuable reference for the design of future imitation learning algorithms. Project page: https://alrhub..ine wwtpllu.bbiibbboooooo wttttt....o//lll-oettte website/
INTRODUCTION
Imitation Learning (IL) (Osa et al., 2018) from human expert data has emerged as a powerful ap- proach for imparting a wide array of skills to robots and autonomous agents. this setup, a human expert controls the robot, c.g., using tele-operation interfaces, and demonstrates solutions to com- plex tasks. Leveraging human expertise, IL algorithms have demonstrated remarkable success in training robots to perform wide range of complex tasks with finesse (Brohan et al., 2022; Zhao et al., 2023; Huang et al., 2023). However, learning from human data is challenging due to the inherent diversity in human behavior (Grauman et al., 2022; Lynch et al., 2020). This diversity, arising from factors such as individual preferences, noise, varying levels of expertise, and different problem-solving approaches, poses formidable hurdle for existing imitation learning algorithms.
In recent years, there has been a notable surge of interest in the development of methods aimed at capturing diverse behaviors (Shafiullah et al., 2022: Chi et al., 2023: Reuss et al., 2023). These endeavors are driven by various objectives, including improving generalization capabilities (Merel al., 2018; et al., 2017), enhancing skill transfer between policies (Merel et al., 2018; Kumar et al., 2020), and gaining advantages in competitive games (Celik et al., 2022), among others. Nev- ertheless, these approaches are often tested on synthetically generated datasets, datasets with limited
*Correspondence xiangang. jia @ kit.edu

---

nuanced assessment of imitation learning Published as a conference paper at ICLR 2024
D4RL
Robomimic
T-Shape
Relay Kitchen
Block-Push
D3IL.
Human Demonstrations
State Data
XXX
Visual Data
Diverse Behavior
Quant. Diverse Behavior
Table 1: Comparison between existing imitation learning benchmarks: D4RL (Fu et al., 2020), Robomimic (Mandlekar al., 2021) are used to benchmark offline RL algorithmss but do focus on diversity. T-Shape Pushing (Florence et al., 2022; Chi et al.. 2023) has 2 solutions and is relatively simple task, while Relay Kitchen (Gupta et al., 2019) has a higher diversity in terms of tasks, however, the behaviors do not require closed-loop feedback due to the limited task variability. Block-Push shows diverse behavior (Florence et al., 2022), however, it is generated by scripted policy, resulting in a reduced variance and missing fallback strategies.
diverse behavior or limited need for closed-loop feedback. Other datasets are collected directly on real robot platforms and lack simulation environments that be used for benchmarking other al- gorithms. While real robot environments are of course preferable to show the applicability of the approaches the real system, they hinder benchmarking sults as that would require rebuilding the real robot setup, which is often infeasible. Furthermore, the majority of existing studies do not provide metrics to quantitatively measure model's ability to replicate diverse behaviors, often relying solely on qualitative analysis (Shafiullah et al., 2022).
algorithms against the reported
To address these challenges, this work introduces benchmark environments and the corresponding Datasets with Diverse Human Demonstrations for Imitation Learning (D3IL). Our primary aim is to provide comprehensive evaluation framework that explicitly assesses an algorithm's 's ability to learn from multi-modal data distributions. We have designed these datasets to encompass the rich- ness and variability inherent in human behavior, offering more realistic and challenging benchmark scenarios for imitation learning. Moreover, the given environments are challenging as the behavior is composed of multiple sub-tasks and they require policies that heavily rely on closed-loop feedback. To tackle the intricate problem of quantifying model's capacity to capture and replicate diverse human bchaviors, we introduce tractable metrics. These metrics provide valuable insights into a model's versatility and adaptability, allowing for algorithms' performance.
Furthermore, conduct rigorous evaluation of state-of-the-ar imitation learning methods using the D3IL task suite. This evaluation serves as a benchmark for assessing the capability of these methods learn diverse behaviors, shedding light on their effectiveness, hyper-parameter choices, and the representations they employ by. By analyzing their performance for these realistic human- generated datasets in challenging environments. Here, our analysis includes different backbones of the different IL architectures (MLPs and various versions of transformers), different ways to capture the multi-modality of the action distribution (clustering, VAEs, IBC, various versions of diffusion). Moreover, we analyze the performance using the direct state observations or image observations as input to the policies and draw conclusions on the benefits and drawbacks of these representations. Finally, we evaluate which method can deal best with small datasets. Using all these insights, our study will inform the design and advancement of future imitation learning algorithms.
RELATED WORK
This section provides overview of existing benchmarks and related research in the field of robot learning and imitation learning. Table presents a comprehensive overview of the distinctive fea- tures that set our work apart from the most closely related benchmarks in the field.
RLBench (James et al., 2020) and ManiSkil12 (Mu et al., 2021; Gu et al., 2023) are two recent large-scale benchmarks for robot learning with large variety of tasks with both, proprioceptive and visual data. However, the demonstrations are generated through motion planning and are hence lacking diversity. Meta- World is another robot learning benchmark with 50 different tasks and focus on multi-task reinforcement learning (Yu et al., 2020). Yet, no human demonstrations exists for these tasks and most tasks typically only have one solution. D4RL (Fu et al., 2020) proposed
2

---

Published as a conference paper at ICLR 2024
standardized environments and datasets for offline reinforcement learning with various tasks. Sev- eral other benchmarks specialize in areas such as sanuugg-guided multitask learning in simulation (Mees et al., 2022; Lynch et al., 2023; Gong et al., 2023; Zeng et al., 2021; Jiang et al., 2023; Gong et al., 2023), continual learning (Liu et al., 2023) and diverse multi-task real world skill learning (Walke et al., 2023; Bharadhwaj al., 2023; Heo al., 2023). A benchmark with focus on human demonstrations is Robomimic (Mandlekar et al., 2021), featuring 8 different task in simulation and real world environments. The demonstrations, collected by multiple people with various degrees of expertise using the RoboTTrk framework (Mandlekar et al., 2018). Yet, the tasks where not designed to show diverse behavior and consequently, non of the tested algorithms was designed to capture the diversity of the behavior.
The benchmarks most closely related D3IL. are Block-Push (Florence et al., 2022; Shafiullah et al., 2022), T-Shape Pushing (Florence et al., 2022) and Relay Kitchen Gupta et al. (2019). While Block-Push has different modalities, the behavior is generated by a scripted policy and is hence lacking variance and fall-back solutions in the dataset. T-Shape tasks are limited to a maximum of two modalities. The Relay Kitchen environment (Gupta et al., 2019) contains 7 different tasks with human demonstrations that solve four tasks in sequence but does not require closed-loop sensory feedback. Further, several recent work have shown to achieve nearly optimal performance (Chi et al., 2023; Reuss al., 2023) these benchmarks, limiting further progress in these environments. Moreover, none of the mentioned benchmarks offer metrics to evaluate the diversity of the learned behavior.
DATASETS WITH DIVERSE DEMONSTRATIONS FOR IMITATION LEARNING
In this section, we present novel suite of tasks known as D3IL. Datasets with Diverse Demon- strations for Imitation Learning. We first introduce metrics for the quantification of diverse behavior (refer to Section 3.1). Subsequently, we delve into the design principles underlying our tasks, which are subsequently introduced at the end of this section.
3.1 QUANTIFYING DIVERSE BEHAVIOR
Let D ((a,; { s,)) be human-recorded demonstration dataset with actions E A and states s E S. Further, let p(a(s) be the underlying state-conditional action distribution. The goal of imitation learning to learn a policy <[ass) x p(als). Diversity is characterized by a multimodal action distribution p( (a)ss, meaning that there are multiple distinct actions that are likely for a given state s. We refer to this as multimodality on state-level. However, this multimodality is hard to quantify in most scenarios as we do not have access to p(als). Instead, we look at the multimodality on a behavior-level to define an auxiliary metric that reflects if a model capable of imitating diverse behaviors. To define behavior-level multi-modality, we introduce discrete behavior descriptors B E B, for example, which box has been chosen to be pushed first. The space B of behavior descriptors are thus task-specific and are discussed in Section 3.3.
To evaluate the ability of trained policy (aas)) to capture multi-modality, we collected data such that we have approximately equal number of demonstrations for each behavior descriptor E B. Thereafter, we perform simulations in the task environment to compute the agents' distribution π (3) over its achieved behaviors. We then assess the policy's 'ssys capability of learning diverse behavior by leveraging the behavior entropy, that is,
((T()))= -- Σ x log B (3).
(1)
BEB
Please note that we employ the logarithm with a base of [B] to ensure that H(((3) E [0, 1]. This choice of base facilitates a straightforward interpretation of the metric: An entropy value of signifies policy that consistently executes the same behavior, while an entropy value of 1 represents a diverse policy that executes all behaviors with equal probability, that is, ((3) x 1/|B] and hence matches the true behavior distribution by the design of the data collection process. Yet, high behavior entropy can also be achieved by following different behaviors in different initial states in a deterministic manner, and hence, this behavior entropy can be poor metric for behavior diversity. Hence, we evaluate the expected entropy of the behavior conditioned on the initial state Sp, If, for the same initial state, all behaviors can be achieved, the conditional behavior entropy is high, while
3

---

Published as a conference paper at ICLR 2024
if for the same initial state always the same behavior is executed, this entropy is 0. We define the conditional behavior entropy as
Bxo=p(s0) [H( =
Σ
Σ
((3 log π (B]sa))
(2)
So
som-l(so) BEB
where the expectation approximated using Monte Carlo estimate. Here, So denotes the number of samples from the initial state distribution p(so).
3.2 TASK DESIGN PRINCIPLES
We build D3IL based on the following key principles: i) Diverse Behavior. Diversity is the central aspect of our task design. We intentionally design our tasks encompass multiple viable approaches to successful task completion. To quantify the behavior diversity, we explicitly specify these dis- tinct behaviors, each representing legitimate solution. ii.) Multiple Human Demonstrators. To reflect the natural variability in human behavior and to obtain richer dataset, we have collected demonstration data from multiple human demonstrators. This diversity in data sources introduces variations in the quality and style of demonstrations. iii.) Variable Trajectory Lengths. Our tasks incorporate variable trajectory lengths, replicating real-world scenarios where demonstrations may differ in duration. This design choice challenges our learning agents to handle non-uniform data se- quences effectively. By accommodating varying trajectory lengths, our approach must learn adapt and generalize to different time horizons, a critical property for real-world applications. iv.) Task Variations and Closed-Loop Feedback. For most tasks, agents need to rely on sensory feedback to achieve good performance which considerably increases the complexity of the learning task in comparison to learning open-loop trajectories. We achieve this by introducing task variations in every execution. For example, in every execution, the initial position of the objects will be different and the agent needs to adapt its behaviour accordingly.
3.3 TASK DESCRIPTION AND DATA COLLECTION
We simulate all tasks (T1-T5) using the Mujoco physics engine (Todorov et al. (2012)). The simula- tion environment consists of a 7DoF Franka Emika Panda robot and various objects. For T1-T4, the robot is equipped with a cylindrical end effector to perform object manipulations. The corresponding demonstrations are recorded by using Xbox game-pad which sends commands to an inverse kine- matics (IK) controller. In T5, the robot has a parallel gripper and uses an augmented reality setup for controlling the end effector (Jiang et al., 2024), allowing for more dexterous manipulations.
Avoiding (T1). The Avoiding task requires the robot to reach the green finish line from a fixed initial position without colliding with one of the six obstacles. The task does require object manipulation and is designed to test a model's ability to capture diversity. There are 24 different ways of successfully completing the task and thus |B] 24. The dataset contains 96 demonstrations in total, comprising 24 solutions with 4 trajectories for each solution.
Aligning (T2). The Aligning task requires the robot to push a hollow box to a predefined target position and orientation. The task can be completed by either pushing the box from outside or inside and thus [B] = 2. requires less diversity than TI but involves complex object manipulation. The dataset contains 1k demonstrations, 500 for each behavior with uniformly sampled initial states.
4

---

Published as a conference paper at ICLR 2024
Pushing (T3). This task requires the robot push two blocks to fixed target zones. Having two blocks and two target zones results in |B] 4 behaviors. The task has more diversity than T2 and involves complex object manipulations. Additionally, the task has high variation in tra- jectory length caused by multiple human demonstrators with different experience levels. The dataset contains 2k demonstrations, 500 for each behavior with uniformly sampled initial block positions.
Sorting-X (T4). This task requires the robot to sort red and blue blocks to their volor-matching target box. The 'X' refers to the number of blocks. In this work, test three difficulty levels, i.e., X {2, 4, 6). The number of behaviors [B] is determined by the sorting order. For X the task has many objects, is highly diverse (B) 20), re- quires complex manipulations, has high variation in trajectory length, and thus more challenging than T1-T3. The dataset contains 1.6k demonstrations for X = 6 and an approximately equal number of tra- jectories for each sorting order.
Stacking-X (T5). This task requires the robot to stack blocks with different colors in a (yellow) target zone. The number of behaviors [B] is given by the stacking order. Having three different blocks results in six different combinations for stacking the blocks and hence [B] = 6. This task additionally requires dexterous manipulations since the blue box has the be stacked upright and is thus considered to be the most challenging out of all five tasks. Here, "X' does not refer to the difficulty but rather to the evaluation protocol: For Stacking-X -X with X E [1, we compute the performance metrics of the models that are capable of stacking X blocks. The s 1k trajectories were recorded using augmented reality (AR) such that each stacking order is equally often present.
BENCHMARKING ON D3IL TASKS
4.1 IMITATION LEARNING ALGORITHMS
We benchmark large set of recent imitation learning algo- rithms. These algorithms can be categorized along three axes. The first axis specifies whether the algorithms are using a his- tory of state observations. For algorithms that do not use his- tory (Sohn et al., 2015; Flo- rence et al., 2022), standard MLP is used as backbone ar- chitecture, while for history- dependent policies (Shafiullah al., 2022: Pearce et al., 2023: Reuss et al., 2023; Chi al., 2023) we use causal masked Transformer Decoder (GPT) based on the implemen- tation of BeТ (Shafiullah et al., 2022) as backbone as this archi- tecture has been also adopted by other recent approaches. In our experiments, 5. The second axis categorizes whether the algorithms are predicting only single action or the future action sequence (Zhao al., 2023; Chi al., 2023), which has been recently introduced as
Methods
History
Backbone
Prediction Diversity
BC-MLP
SA
Deterministic
VtE-State (Sobm et al., 2015)
k
MLP
SA
VAE
BeT-MLP
SA
Cluster
IBC (Florence ot ol., 2022)
k
MLP
SA
Implicit
DDPM-MT.P
MI.P
SA
Disc. Diffusion
VAE-ACT (Zhao el al., 2023)
T-EnDeec
AC
VAE
BC-GPT
GPT
SA
Deterministic
BeT (Shafiullah al., 2022)
GPT
SA
Cluster
DDPM-GPT (Pearce et al., 2023)
k=
GPT
SA
Disc. Diffusion
BESO (Reuss ot al., 2023)
GPT
SA
Cont. Diffusion
DDPM-ACT (Chi al., 2023)
T-EnnDec
AC
Disc. Diffusion
Table 2: Categorization of the tested II. algorithms. The algo- rithms differ in whether they history information (MLP back- bones for no history, Transformer/GPT backbones for history- based models), whether they predict future action sequences (sin- gle actions/SA or action chunking/AC, Transformer Encoder- Decoder/T-EncDee backbones for AC) and how they model di- verse behavior (last column).
history of
5

---

Published as a conference paper at ICLR 2024
action chunking which has been shown to improve performance in some tasks. For action chunking models, we use a Transformer Encoder Decoder structure (T-EneDec) (Vaswani et al., 2017) based on the implementation of VAE-ACT (Zhao et al., 2023). The third axis categorizes how the algo- rithms capture behavior diversity. Standard deterministic models are not able to do so, while more recent models use clustering (Shafiullah al., 2022), VAFs (Zhao et al., 2023; Sohn et al., 2015), implicit models (Florence et al., 2022) as well as discrete-time (Pearce et al., 2023; Chi et al., 2023), or continuous time diffusion models (Reuss al., 2023). To improve our understanding of the in- fluence of the specific design choices, we also evaluate new variants of these combinations such as history-frres fetorry-free (MLP-based) action clustering (BeT-MLP). The full table of used algorithmic setups depicted in Table 2. A more detailed description of all model architectures is given in Appendix C.
4.2 EXPERIMENTAL. SETUP
We briefly outline the most important aspects of our experimental setup.
Evaluation Protocol. To assess a model's ability to capture diverse behavior, we propose the behav- ior entropy and the conditional behavior entropy (Section 3.1). For the former, we perform multiple simulations, from which we compute the model's behavior distribution (B). For the latter, we use randomly sampled but fixed set of initial states So and perform multiple simulations for each so in order to compute the conditional behavior distribution (3]so). Alongside the entropy, we also report the success rate which is the fraction of simulations that led to successful task completion.
State / Observation Representation. For most experiments, we provide state representation and image observations. For the former, we use handerafted features that work well empirically. For the latter, we used two different camera views, in-hand and front view with a 96 x 96 image resolution. We follow Chi et al. (2023) and use ResNet-18 architecture as an image encoder for all methods.
Model Selection. Model selection is challenging as the training objective does not coincide with task performance (Gulcehre et al., 2020; Paine al., 2020; Fu et al., 2021; Mandlekar al., 2021). Moreover, policy performance can exhibit substantial variations from one training epoch to another (Mandlekar al., 2021). For image-based experiments we evaluate the task performance frequently (after every 1 1/10th of total training) and chose the model with the best task- mass--erffrmance. Due to high-computational demand for simulation, we split the dataset into training (90%) and validation (10%) and chose the model with the lowest validation loss for base- experiments. We ensure that the training converges by using 500 epochs for state representation and 200 for image obser- vations. We extensively tune the most important hyperparameters of all methods using Bayesian optimization (Snock et al., 2012). We report the mean and the standard deviation over six random seeds for all experiments.
4.3 BENCHMARK USING STATE REPRESENTATION
We present the results utilizing state representations in Table 3. Detailed information regarding individual state representations for the tasks can be found in Appendix A. Through an analysis of the performance and diversity across all methods, we outline key findings below.
Multiple Strategies for Multi-Modal Behavior Learning. This section discusses various strate- gies employed by different models for learning multi-modal behavior. Deterministic models like BC-MLP and BC-GPT have limitations in this regard since they can only generate a single solution given an initial state So. On the other hand, VAE-based models like VAE-State and VAE-ACT are capable of generating different behaviors, but they tend to exhibit low behavior entropy, presumably due to a phenomenon often referred to as "mode collapse" (Kingma Welling, 2013). In contrast, IBC, BeT, and ceftusion-bassee models demonstrate the capacity to learn multi-modal distributions. IBC, for instance, exhibits high diversity in tasks like Avoiding, Pushing, and Sorting-2, with en- tropy levels slightly superior to DDPM-MLP. BeT is capable of learning diverse solutions for each task, but it comes with a significant performance drop when compared to the deterministic BC-GPT baseline. Notably, diffusion-based methods, especially those incorporating transformer backbones, excel in learning diverse behavior while maintaining strong performance across all tasks. To of- fer additional insights, we include visual representations of the diverse solutions generated by each method for the Avoiding task in Figure 10. Furthermore, we conduct an in-depth analysis of hyper- parameter sensitivity, which is discussed in Section B. This analysis provides deeper understanding

---

Published as a conference paper at ICLR 2024
Table Comparison between various Imitation Learning algorithms, some of which incorporate his- tory (f), action chunking (f), or both using state representations (State Data) and image observations (Visual Data). We present the mean and standard deviation across six random seeds, highlighting the best performance for both history-based and non-history -bador models using bold formatting.
Success Rate
Entropy
Success Rate
Entropy
Success Rate
Entropy
State Data
Avoiding (T1)
Aligning (T2)
Pushing (T3)
BC-MLP
0.666
0.020.0
0.708=0.062
0.0.0.0
0.522:0.165
0.0
VAE-State
0.716=00.26
0. 0.191
0.579=0.185
0.030=a.c68
0.604 0.007
0.170=0.031
BeT-MLP
0.665-10.10 136
0.83610.071
0.507
0.485 002
0.420 0.40
0. 0.628.10.10 123
IBC
0. 7..760
0.850:0.008
0.638=0.022
0.300=0.044
0.574=0.048
0.816=0.022
DDPM-MLP
0.637
0.801=0.034
0.763
0.712 084
0.569=0.047
0. 0.796
VAE-ACT
0,851
0 0.7
0. 0.891.10.0m 0222
0.025
0.951
0.070.1.:29
BC-GPT
0.833=#0:006
0.0=0.0
0.833=0.043
0.0=a.0
0.855=0.054
0.0=0.0
BeT²
0.747
0.84410.035
0.645
0,536..0.108
0.724_0.051
0.78811.001
DDPM-GPT
0.927=20.022
0.898:20.00k
0.839=00020
0.66420.075
0.847=0.050
0.862+0.017
BESO
0.950
0. 1.856
0.861=0.016
0.711=0.c00
0.791100000
0 0.871=0.087
DDPM-ACT
=.000
0. 1.86300.06
0.849=0.023
0.749
0.920
0.850
State Data
Sorting-2 (T4)
Sorting-4 (T4)
Sorting-6 (T4)
BC-MLP
0.444=0.009
0.0+0.0
0,058=0.048
0.0=0.0
0.016=0.014
0.0=0.0
VAE-State
0.451=0.059
0.106=0.051
0.079=0.042
0.090=a.082
0.030+0.01e
0.086=0.011
BeT-MLP
0.31 7:00000
0.309=0005a
0.003=0.001
0.021=0.052
0.0=o.0
0.0gz.o
TBC
0.459:20.0ss
0.370 071
0.080::000
0.097.100224
0.007=0.005
0.02420:025
DDPM-MLP
0.000
0.327=0.037
0.032=0.009
0.071=0.025
0.0=0.0
0.0+0.0
VAE-ACT
0.858=0.07s
0.012=0.018
0.565=20.100
0.114=0.020
0.0 0.021=0.018
0.057=0.040
BC-GPT
0.800
0.011..
0.075
0.0.
0.007=0.003
0.0.
BeT
0.680=0.060
1.477:20.049
0.018=0.010
0.059=0.0as
0.0go.0
0.0go.0
DDPM-GPT
0.854==0.013
0.431=0.06
0.337=0.054
0.362:0.068
0. 0.005=0.003
0.031
BESO
0. 29.04
0.38720.034
0.188 0.18820.022
0.232:00040
0.020
0.055 031
DDPM-ACT1-4
0. 0.882#0.028
0.347=0.048
0.300=0.06a
0.300=a..os1
0.0=0.0
0.0
State Data
Stacking- I (T5)
Stacking 2 (TS)
Stacking 3 (TS)
BC-MLP
0.030=9.022
0.0=0.0
0.0=0.0
0.0=0.0
0.0=0.0
0.0+0.0
VAE-State
0.036=0.025
0.064=0.092
0.001=0.001
0.0=2..
0.0=0.0
0.0+0.0
BeT-MPP
0. 0.10220.009
0.1 1.120=2oom 081
0.003=0:001
0.0g2.0
0.0zo.0
0.0
IBC
0.01010.010
0.0131...0 our
0.0az..
0.0aa.0
0.0go.0
0.02.0.
DDPM-MLP
0.144
0.113
0.042 045
0.01110.014
0.001::0 0009
0.096:0.bur 190
VAE-ACTI
0.308=20099
0.095
0.083=0.007
0.037=000087
0.0=0.0
0.0
BC-GPT
0.241:00.065
0.020.0
0.025=0.016
0.0=a.o
0.0=o.o
BeTt
0.163:0.006
0.23422000m
0.025-20015
0.05120.080
0 100000000000
0.0220.
DDPM-GPT*
0.895=a.013
0.222=0.052
0.708=0.00a 080
0.1 1.146:0.c29
0. 1 10
0. 0.00000000
BESO!
0.90910.007
0.38220.063 063
0. 0.685
0.1 2.17710.uu us8
0.095 000000
0.13 ozs
1DPM-ACT14
0.634=0.049
0.762:00.04
0.300=0.053
0.503=0.070
0.096
0.190 05?
Image Data
Aligning (T2)
Sorting-4 4 (T4)
Sorting-6 6 (T4)
BC-MLP
0.148=0.045
0.0=0.0
0.606=0.080
0.0=a.0
0.636=0.026
0.0+0.0
VAE-State
0.121 20.055
0.01320.019
0.653=0.001
0.165=20010
0.570=0.023
0.23620.015
BeT-MLP
0.203
0.05320.025
0.67 20-11
0.296:20.033
0.616=0.035
0.365 022
IBC
0.179
0.071 002
0. 0.707 17700..011 031
0.236
0. 0.66411.000
0.328
DDPM-MLP
10.00
0.0
0.763 026
0.237 0.025
0.7 0.7
0. 0.350
VAE-ACT
0.16620.017
0.001
0.717=00058
0.217=0.021
0.604=0.017
0. 0.2762200077
BC-GPT
0.081=a.015
0.0+0.0
0.654=0.004
0.0=a..o
0.590=0.017
0.0
BeTt
0.154290007
0.02320.012
0.676
0.305
0.617
0.38020.01
DDPM-GPT*
0.316
0.089
0. 0.770=0.00m
0.270=n0.aa
0.7 0.710=o.0mm 029
0.314
BESO!
0. 0.05910.001
0.151 051
0.7642..02
0.281 20.05
0. 0.71910.025
0.352.t.018
1.
0.278=0.071
0.139=0.054
0.659=0.003
0.346
0.643=0.036
0.397=0.025
Image Data
Stacking-1 (T5)
Stacking (TS)
Stacking-3 (TS)
BC-MLP
0.501
0.0+0.0
0.133=t0.00
0.0=z.0
0.0=o.e
0.0az.o
VAF-State
0.517:20.059
0.038=0.022
0.089=0.050
0.028:10.022
0.001=0.001
0.0
BeT-MLP
0.467
0.195:10.044
0.068200024
0.027
0.0
0.0g
IBC
0.002210.003
0.010.0
0.010.0
0.0
0.0
0.0.
DDPM-MLP
0.6-15+..050 050
0. 0.137:0.061
0 0.111=0.040
0.027=0.019
0.002 0.002
0. 0.Doo.o
VAE-ACT+
0.572:20.08
0.21 0.219
0.19620.057
0.1 152000057
0.018=0.010
0.006:00000
BC-GPT
0.515=0.002
0.0=0.0
0.143=o.037
0.0
0.003=0.002
0.Da0.0
BeT!
0.500
0.3
0.143
0.037
0.002=0.001
0.0goo
DDPM-GPT
0.704=0.024
0.344
0.153=0.055
0.145==0.55
0.007=0.004
0.012+0.01
BESO!
0.655.10.001
0,1821o.020
0.188
0.087_2..c00
0.003
0.00810.020
DDPM-ACT1.
0.752+0.045
0.334+0.073
0.513:00.00 o58
0.302+0.084
0.244+0.048
0.093=0.042
7

---

Published as a conference paper at ICLR 2024
Seczeo Rasc
Enieery
Stuzz Raie
Fairary
6:17
6094
100
6:11
E:47
6:47
10:13
F164
6:00
8.70
Ea5
6:35
046
066
8:0
024
0008
6,00
0.54
0771
Hirwyy
+427
4012
612
00.00
4030
48.0
6:15
6.44
054
043
6:00
640
046
044
631
645
0.13
0.77
6:00
0J9
0.54
0.51
8130
CDP1
Prodictio Hacacc
Production Herizs
(a) DDP
(b) GPT-bbased policies
Figure I: Ablation study for different history lengths and prediction horizons the Aligning task.
of how hyperparameters influence the ability to learn diverse solutions. Notably, our findings reveal that increasing the number of diffusion steps leads substantial enhancement in diversity while maintaining consistent task performance.
Historical Inputs and Prediction Horizons are Important. comparison of results between BC-MLP and BC-GPT, VAE-State and VAE-ACT, BeT-MLP and BeT, DDPM-MLP and DDPM- GPT reveals a notable trend: transformer-based methods consistently outperform their MLP-based counterparts. In particular, DDPM-GPT achieves an average 21% improvement in success rate across Avoiding, Aligning, and Pushing tasks compared to DDPM-MLP. While DDPM-MLP ex- hibits higher entropy the Aligning task, it lags behind on other tasks. When comparing VAE- MLP to VAE-ACT, VAE-ACT demonstrates a success rate improvement of over 30% on most tasks, indicating its effcetiveness in capturing diverse behaviors. More significantly, transformers that incorporate historical inputs or prediction horizons demonstrate the ability to tackle challenging tasks like Sorting-4 and Stacking. However, the combination of historical information with ex- tended prediction horizons (DDPM-ACT) does not appear to provide substantial benefits compared to DDPM-GPT, which does not predict future actions.
Scaling to More Complex Tasks. Current state-of-the-ar methods excel in tasks such as Avoid- ing, Aligning, and Pushing; however, a notable performance gap emerges in Sorting and Stacking tasks. In the case of Sorting, scaling with an increasing number of objects proves challenging for all methods. Specifically, on the Sorting-6 task, none of the existing methods achieve satisfactory performance. The complexity of the observation space and task diversity significantly increases as we aggregate all box features into single state vector. Consequently, there a demand for models capable of learning compact and resilient state representations.
4.4 BENCHMARK USING IMAGE OBSERVATIONS
We present the results utilizing image observations in Table 3. Detailed information regarding cam- setup can be found in Appendix A. Through an analysis of the performance and diversity across all methods, we outline our key findings below.
Comparison between State-Based and Image-Basee -ased Policies. State representations have proven highly effective for tasks demanding precise control (i.e. Aligning), but they do not scale with the number of objects (i.e. Sorting). In such case, state-based approaches may struggle exploit in- variances, which are essential for handling complex scenarios. Conversely, image representations exhibit a notable capacity handle scenarios involving multiple objects. However, image-based policy performs much worse than state-based policy on the Aligning task, which require precise control. The inherent difficulty in extracting fine-grained details and spatial relationships from im- ages makes it challenging in achieving precise manipulation.
Trade-off in Sequential Information Across Visual Tasks. From the results of state-based evalua- tions, it is evident that historical inputs and prediction horizons consistently enhance success rate and entropy on all tasks. Regarding the image-base results, DDPM- GPT shows less improvement than DDPM-MLP on Aligning and Sorting-4 and performs a little worse on Sorting-6. This phenomenon has also been observed in the comparison between BC-GPT and BC-MLP, BeT and BeT-MLP. How- ever, in the most demanding task, Stacking, transformer-based models consistently outperform MI.P
8

---

Published as a conference paper at ICLR 2024
0000%
25%
9006
0 105g 8
2599 5050 8
759 100%
0.8
eutropy
0.4
0.4
0.2
0.2
VAB
BeFMEP
IBC
DDPM-MLP
lef
DDPM-OPT
BESO
VAE-ACT
DDPMS-ACT
VAE
BeT-ME
IBC
DDPM-MLL
DDPM4-PT
BESO
VAE-ACT
DDPM-CCT
Figure 2: Ablation study for different percentages of the original dataset size of the Aligning (T2)
task. The percentages are color-coded according to the legend in the top left corner.
models. Notably, DDPM-ACT displays improvements of 11%, 40%, 24% in stacking 1, 2, boxes, respectively, compared to DDPM-MLP.
4.5 IMPACT OF HISTORY AND PREDICTION HORIZON
We conducted a comparison of various choices for history and prediction horizons, as illustrated in Figure 1. Notably, when both history and prediction horizons are greater than 1, DDPM-ACT exhibits improved performance. However, it's worth noting that, consistent with findings from prior work (Chi et al., 2023), extending the length of observation history and action sequence horizon leads to decline in both success rate and entropy. This observation underscores the importance of careful design when using transformer encoder-decoder structure for history and prediction horizons. Additionally, we find that historical inputs significantly enhance the performance GPT- based policies, and this improvement is consistent with increasing the history length.
4.6 LEARNING WITH LESS DATA
The process of recording demonstrations, particularly when involving human demonstrators, is often tedious. To that end, we assess the models' ability learn with less training data, we generate four subsets comprising 10%, 25%, 50% and 75% demonstrations for the Aligning task. The results are reported in Figure 2, with detailed results available in Table MI.P-based methods (e.g. VAE-State, BeT-MLP) experience a significant performance drop when trained with less data. They display up to 54.4% drop success rate and 43.7% drop in entropy on the 25% dataset, and are nearly non- functional the 10% dataset. In contrast, transformer- based methods (e.g. DDPM-GPT, BcT) exhibit a higher tolerance for small amounts data, with up to 33.3% success rate drop and 20.3% entropy drop 25% dataset, maintaining more than 15% success rate on 10% dataset. Additionally, we find that BESO exhibits a 43% success rate on the 10% dataset. From these findings, we conclude that transformer architecture generalizes well with less training data and diffusion-based methods seem to be able to regularize the transformer to make it less data-hungry.
CONCLUSION
Our introduction of Datasets with Diverse human Demonstrations for Imitation Learning (D3IL), addresses the critical need to evaluate a model's capability to learn multi-modal behavior. These environments incorporate human data, involve intricate sub-tasks, necessitate the manipulation of multiple objects, and require policies based on closed-loop sensory feedback. Collectively, these characteristics significantly enhance the diversity of behavior in D3IL, setting it apart from existing benchmarks that often lack one or more of these crucial elements.
To measure this diversity, we introduce practical metrics that offer valuable insights into model's ability to acquire and replicate diverse behaviors. Through a comprehensive evaluation of state- of-the-art methods methods in learning diverse behavior. This contribution not only guides current efforts but also provides valuable reference for the development of future imitation learning algorithms.
proposed task suite, our research illuminates the effectiveness of these

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported by funding from the pilot program Core Informatics of the Helmholtz Association (HGF). NS and GN were supported by the Carl Zeiss Foundation under the project JuBot (Jung Bleiben mit Robotern. Xiaogang Jia and Xinkai Jiang acknowledge the support from the China Scholarship Council (CSC). The authors acknowledge support by the state of Baden- Württemberg through bwHPC, as well as the HoreKa supercomputer funded by the Ministry of Sci- ence, Research and the Arts Baden- Württemberg and by the German Federal Ministry of Education and Research.
REFERENCES
Homanga Bharadhwajj. Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. Roboagent: Generalization and efficiency in robot manipulation via semantic augmenta- tions and action chunking, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang. Isabel Leal, Kuang-Hued Lee, Sergey Levine. Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xíao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale. In arXiv preprint arXiv:2212.06811 2022.
Onur Celik, Dongzhuoran Zhou, Ge Li, Philipp Becker, and Gerhard Neumann. Specializing ver- satile skill libraries using local mixture of experts. In Conference on Robot Learning, pp. 1423- 1433. PMLR, 2022.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Bric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, In Conference on Robot Leariigg, pp. 158-168. PMLR, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Justin Fu, Mohammad Norouzi, Ofir Nachum, George Tucker, Ziyu Wang, Alexander Novikov, Mengjiao Yang, Michael R Zhang, Yutian Chen. Aviral Kumar, et al. Benchmarks for deep off- policy evaluation. arXiv preprint arXiv:2 10..:210336666 2021.
Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song-Chun Zhu, et al. Arnold: A benchmark for tonguugg-grounder task learning with continuous states in realistic 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Jayyua Gu, Fanbo Xiang, Xuanlin Li. Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference Learning Representations, 2023.
10

---

Published as a conference paper at ICLR 2024
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Tom Le Paine, Sergio Gómez Colmenarejo, Kon- rad Zolna, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI un- plugged: Benchmarks for offline reinforcement learning. arXiv preprint w7X1:2200611388, 394, 2020.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: 1956, 2019.
Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J. Lim. Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation. In Robotics: Science and Systems, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 616840-6551, 2020.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305. 11176, 2023.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 3026, 2020.
Xinkai Jiang, Paul Mattes, Xiaogang Jia, Nicolas Schreiher, Gerhard Neumann, and Rudolf Li- outikov. A comprchensive user study augmented reality-based data collection interfaces for robot learning. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robon Interaction, pp. 333-342, 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In Fortieth International Conference on Machine Learning. 2023.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational
bayes.
arXiv preprint
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198 05:8198-210, 8210, 2020.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. Advances in neural information processing systems, 30, 2017.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. 2023.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In Conference on robot learning, pp. 1113- 1132. PMLR, 2020.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pp. 879-893. PMLR, 2018.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkami,, Li Fei- Fei, Silvio Savarese. Yuke Zhu, and Roberto Martín- Martín. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv: 108 008.0998,, 2021.
11

---

Published as a conference paper at ICLR 2024
Oier Mees, Lukas Hermann, Erick Rosete- Gosete-Beas, and Wolfram Burgard. Calvin: A benchmark for conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7334, 2022.
Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. arXiv preprint arXiv: 2018.
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhi- wei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107. 14483, 2021.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 71122): 1-179, 2018.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint wwXii:2007.09055 2020.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Ser- gio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint urXiv:2301. 10677, 2023.
Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone. Advances in neural information processing sys- 2022.
fems, 35:22955-229666
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in nearal information processing systems, 28, 2015.
Yang Song, Jascha Sohl-Dickstein Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scoee-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Homer Rich Walke, Kevin Black, Tony Z. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, Abraham Lee, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In 7th Annual Conference on Robot Learning, 2023. URL https: / openrevieee net/ net/foruml forum? Id=E55MIATIIN.
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-tasi and meta reinforcement learning. In Conference on robot learning. pp. 1094-1108 100. PMLR, 2020.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Re- arranging the visual world for robotic manipulation. In Conference on Robot Learning, pp. 726- 747. PMLR, 2021.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine- grained bimanual manipulation with low-cost hardware. arXiv preprint XXXx:2304.137005 2023.
12

---

Published as a conference paper at ICLR 2024
A
ADDITIONAL TASK DETAILS
We provide additional visualizations and more detailed task descriptions.
Avcidiua
Podirs
PG
Algiis
yalk
pab wadd
Surtngg
pusticc
260 2003
Stoking
pot
Figure 3: D3IL Visualizations. This figure provides an overview of various tasks and behaviors within our dataset. The top row demonstrates one of the 24 possible solutions for the "Avoiding" task. The second row displays snapshots from all four pushing sequences the "Pushing" task, with sub-captions indicating block movements (e.g., 'rr-gg' signifies red block to red target and green block to green target). The third row showcases the two behaviors for the "Aligning" task, with the leftmost figures illustrating alignment from within the box and the rightmost from outside. The fourth row focuses on the "Sorting-3" task, with the initial configuration on the left and diverse pushing strategies, including simultaneous block manipulation, in subsequent figures. The bottom row depicts snapshots of the "Stacking" task, highlighting the intricate dexterity required, including complex pose estimation and orientation changes when picking and stacking the blue block.
A.1 AVOIDING (T1)
Success Criteria. The Avoiding task considered to be successful if the robot-end-effector reaches the green finish line without having collision with one of the six obstacles.
State and Action Representation. The state representation encompasses the end-effector's desired position and actual position in Cartesian space, with the caveat that the robot's height remains fixed, resulting in E R.. The actions a E R are represented by the desired velocity the robot.
Behavior Descriptors 3 and Behavior Space B. The behavior descriptors 3 defined by all 24 ways of avoiding obstacles and thus [ = 24. We collected data such that an equal amount of trajectories for each solution strategy exists.
13

---

Published as a conference paper at ICLR 2024
A.2 ALIGNING (T2)
Success Criteria. The Aligning task is considered to be successful if the box aligns with the target box within some tolerance and their colors match for each side.
State and Action Representation. The state representation includes the end-effector's desired po- sition, actual position in Cartesian space, the pushing box position and quaternion, the target box position, and quaternion, and thus $ IR The actions a R3 are represented by the desired Cartesian velocity of the robot.
Figure 4: Initial State Space. The initial state So consists of the box and target position. The former is sampled uniformly from the yellow rectangle and the latter from the grey. The figure maintains the true-to-scale ratio of the samples and the bounding box.
Behavior Descriptors 3 and Behavior Space B. The behavior descriptors 3 are conditioned the initial state Su and are defined by both ways pushing a box, i.e., either from the inside or from the outside (refer to the Figure 3 Aligning). The data is recorded such that there are an approximately equal amount of trajectories for both behaviors and every initial state.
A.3 PUSHING (T3)
Success Criteria. The Pushing task is deemed successful when both blocks are positioned inside predefined target zone, with a certain level of tolerance allowed.
State and Action Representation. The state representation includes the end-effector's desired po- sition, actual position Cartesian space, the two boxes' position, and tangent of Fuler angle along axis and thus E IR_00 The actions a R2² are represented by the desired Cartesian velocity of the robot with a fixed height.
Figure 5: Initial State Space. The initial state So consists of the initial position of the red and green blocks. The red blocks are uniformly sampled from the red rectangle and the green ones from the green rectangle. The figure maintains the truc-to-scale ratio of the samples and the bounding box.
Behavior Deseriptors and Behavior Space B. The behavior descriptors are conditioned on the initial state So and are defined by the four pushing sequences and thus | 4. The four pushing sequences are all combinations of pushing the two blocks the target zones (refer Figure Pushing). The data is recorded such that there are equal amount of trajectories for all pushing sequences and initial states.
A.4 SORTING-X (T4)
Success Criteria. The Sorting task is considered successful when all blocks are positioned inside a target box with the same color.
State and Action Representation. The state representation includes the end-effector's desired po- sition, actual position in Cartesian space, the boxes' position, and tangent of Euler angle along axis and thus s E 4433X The actions a E R2 are represented by the desired Cartesian position of the robot with fixed height.
Behavior Descriptors 1 and Behavior Space B. The behavior descriptors 3 are conditioned on the initial state So and defined by all possible color combinations of sorting the blocks in the
14

---

Published as a conference paper at ICLR 2024
Figure 6: Initial State Space. The initial state So consists of the initial positions of all red and blue blocks. In case of collisions between blocks, we define 6 non-intersecting sampling spaces and uniformly sample 6 box's states which are randomly assigned to red and blue blocks. The figure maintains the true-to-scale ratio of the samples and the bounding box.
respective target zones. The data is recorded such that there are approximately equal amount of trajectories for all sorting orders and every initial state.
A.5 STACKING (T5)
Success Criteria. The Stacking task is deemed successful when all three blocks are in the yellow target zone (with some tolerance) and have the appropriate height, signifying that they are stacked on top of each other.
State and Action Representation. The state representation includes the robot's desired joint posi- tion, the width of the gripper, the boxes' position, and tangent of the Euler angle along 2 axis and thus T220 The actions R are represented by the robot's desired velocity and width of the gripper.
Figure 7: Initial State Space. The initial state So consists of initial positions of red, green and blue blocks. The blocks are uni- formly sampled from the corresponding rectangle box. The figure maintains the true-to-scale ratio of the samples and the bounding box.
Behavior Descriptors 3 and Behavior Space B. The behavior descriptors are conditioned on the initial state Sp and are defined as all possible color combinations of stacking the blocks the spective target zones, resulting in [B] = 6. The data is recorded such that there are approximately equal amount of trajectories for sorting orders and every initial state So-
A.6 ADDITIONAL DETAILS
Datasets. We present the statistics of our D3IL dataset. From TI to T5, task complexity is in- creasing while the demonstrations require more time to record, with 61 steps minimum 965 steps maximum. A large amount of dataset is provided for each task, with more than 1000 demonstrations for most tasks.
Policy Simulation. For each task, we uniformly sample So initial states. Given an initial state, we simulate each policy in closed-loop manner for multiple times based on task complexity. The pol- icy is expected to learn diverse solutions for each context. We design the simulation time according to the maximum length our demonstrations.
State Observation Representation. For most experiments, we provide state representation and image observations. For the former, we use carefully handcrafted features that work well empiri- cally. For the latter, used two different camera views, in-hand and front view with 96 x 96 image resolution as shown in Figure 8. We use 8-bit color channels nonmalized to [0, 1] and a We use ResNet-18 architecture
96 96 image resolution, resulting in S [0, 1] 2x3x96666
15

---

Published as a conference paper at ICLR 2024
Tasks
Tpio
min
maxx
N₂
(e.a)
So
Nsim
(T1) Avoiding
77.09:111.96
61
107
96
7.4k
480
(T2) Aligning
194.88:+:10.77
102
354
1000
194k
60
850
(T3) Pushing
232.69++3.19
152
356
2000
465k
30
16S0
(T4) Sorting-1
335.49 132.62
112
297
600
112k
60
8Sg
(T4) Sorting-2 2
144
509
1054
353k
60
18S0
(T4) 3orting-3
510.57 1344.13
186
923
1560
848k
60
20So
(T5) Stacking
628.90 033.90
458
965
1095
688k
60
18S0
Table Overview Tasks, For each task, we report the demonstration steps T, number demon- strations Nr, number of state-action pairs (ssaa,, number of simulated initial states S and number of simulations Nsim-
image encoder for all methods proposed in Diffusion Policy (Chi et al., 2023). For further details on the dimensionality of the state and action space for different tasks, Table
Tasks
dim ()
dim (A)
state rep.
image obs.
Avoiding
4
Aligning
20
(2,3,96,3) 96,96) 96)
3
Pushing
10
2
Sorting-X X
+ 3X
(2,3, 96,,96,,1 96)
2
Stacking
20
(2, 96, 96)
8
Figure 8: Camera views. Front view (right) and in-hand (left). The image resolution 96 96.
Table 5: Summary of state-space dimensions
dim(S) and action-space dimensions dAm(
B FURTHER EXPERIMENTAL RESULTS
Datasetsize Ablation Study. We provide the exact numbers for the dataset size ablation study presented in Section 4.6, Figure 2. These numbers are presented in Table 6.
25% dataset
50% danseet
dataser
100% dataset
Succees Rate
entropy
Sucreo Rate
entropy
Success Rate
Succes Rate
entropy
Success
Rate
BC-MEP
0.0sea
0.0ano
12208accas
0.0ca
0.530
0.0
1.d0
0.0zno
0.708
0.0ane
VAF-Stare
0.Qseo
0.02n0
0.0552
6.00022we
0. 0.121 ww
0.00020
0.36Sanass
0.008an:17
0.579
0.00snas
BeT--MIP
0.Qsea
0.0
0. 99
0.120=m1
0. 0.466aa
0.545an-
0.438a
0.351an
0.507
0.0saa
0.00a
0.30210:539
0.200
0.554.9005
0.3071n.34
0.00210045
0:2001..041
0.638
0.3001
DDPM MLP
0.056:a
0.002an:e5
0.33R16:31
0.275
2.609
0.6281n:44
0.7d81nazz
0.6521n.921
0,783.0
0.712snast
BC.-PPT
0.30
0.722
0.0ane
0.811aa04
0.0inn.
0.831
8.184.0
0.08510:344
0.183
0.545
0.002
0.511ac.am
0.645.a..e
0.506a
DDPM GPT
0,357
0.2401a.221
0.52013m0
0.791
0.207100045
0.852
0.9091..221
0.830
0.66410775
BESO
0.53
0.520
0.7114mam
VAE-ACT
0.006an3e3
0.1125-1.00
0.885an001
0.006100.33
0.800saaa
0.050-amma
0.2tanaa
Table 6: Exact numbers for the dataset size ablation study presented in Section 4.6, Figure 2.
Hyparameter Ablation Study. To gain deeper insights into the hyperparameter sensitivity of our methods, we conducted an ablation study on the Aligning (T2) task, that highlights the influence of key hyperparameters on performance metrics. Specifically, for DDPM-GPT and BESO, we ablated the number of diffusion steps; for ВеТ, we focused on the number of action clusters, and for VAE- ACT, we ablated the scaling factor of the Kullback-Leibe divergence in the loss term. The detailed results are presented in Figure 9.
Our findings reveal that increasing the number of diffusion steps primarily enhances behavior en- tropy, signifying improvement in the model's capacity to capture moltiiiodaliiiss. However, for BeT and VAE-ACT, we did not observe a consistent trend indicating a correlation between the parameters and success rate entropy.
Inference Time Comparison. We conducted a comprehensive comparison inference times for all baseline models using both state- and image-based inputs. For state-based inputs, utilized
16

---

Published as a conference paper at ICLR 2024
0.96
DIDP 3080
DOPM F0093
0.502
DIPPM
5920
0.9
0.8
A000000 rate
03:14
est.20g0
0.95
M0C0OR ride
0.88
catrupyy
0.92
0.9
0.6
0.86
0.9
0.81
0.4
16
12
16
8
16
131 flfffusion stegs
Diffasion stepe
Difitaiion sleps
Diffrssion stepe
0.96
367
D61
0.77
0.7
0.55
500T10 ride
0.82
ent.ggg
0.7
0.8
0.435
0.5
0.6
0.433
16
32
64
0.76
16
32
64
act5om clusters
aclion cluslets
actson clussers
action clusters
0.8
VAB-AUT
VAR ACT
VARLACT
VAR. NM
0.92
8110000 rale
0.25
enlopp
entrapy
0.2
0.1
0.15
0.01
0.1
0.
01.01
0.1
10
100
0.00
10
100
0.00
0.1
10
100
scaling KT.
loee
acaling
KL. loss
scaling, Bor KL loss
srallng Fg for KT. lose
Figure 9: Ablation Study for the most important hyperparameters of DDPM/BESO (diffusion steps), BET (action clusters), and VaE (scaling factor for Kullback-Leible loss term) the Avoiding (T1) and Aligning (T2) task.
ten-dimensional input, while for images, a 96 x 96 dimensional input was employed. To ensure robustness, we repeated the inference process 500 times and calculated the average time. The detailed results are presented in Table 7.
It is evident that methods employing (discretized) Langevin dynamics, such IBC and diffusion- based models, exhibit longer inference times. It's noteworthy that methods labeled with "ACT" predict three steps into the future, resulting in lower per-step inference times.
Inference Time image-based
state-based
BC-MLP
0.46 ms
5.77 ms
VAE-State
0.52 ms
6.41 ms
BeT-MI.P
1.00 ms
6.56 ms
IBC
21.94 ms
26.53 ms
DDPM-MLP
5.29 ms
11.53 ms
VAE-ACT
0.64 ms
5.68 ms
BC-GPT
1.84 ms
7.82 ms
BeT
1.97 ms
7.89 ms
DDPM-GPT
30.52 ms
43.56
BESO
31.14 ms
40.30 ms
DDPM-ACT
12.75 ms
14.81
Table 7: Inference time comparison averaged over 500 runs. Methods labeled with 'ACT' predict three steps into the future, resulting in lower per-step inference times.
Position Velocity Control. Similar Chi al. (2023), we conducted a comparison between position control (absolute actions) and velocity control (relative actions) for DDPM-ACT and BESO across three distinct tasks. The results, obtained from four different seeds, presented in Table 8.
17

---

Published as a conference paper at ICLR 2024
Our findings indicate that employing velocity control leads to significant improvement in both success rate and entropy for the tasks.
Avoiding (TI)
Aligning (T2)
Pushing (T3)
Velocity Control
Success Rate
Entropy
Success Rate
Entropy
Success Rate
Entropy
BESO
0.950
0.856=0.018
0.061
0.7 20.000
0.79440.099
0.
DDPM-ACT
0.899
0.863=0.008
0.849=0.023
0.749
0.920
0.859=0.012
Position Control
Success Rate
Entropy
Success Rate
Entropy
Success Rate
Entropy
BESO
0.45940.333
0.327=0.226
0.133=00024
0.071 #0.008
0.013440.010 0.40440.141
0.000=00000
DDPM-ACT
0. 0.15
0.836=2.055
0.413=2.163
0.385 347
0.478=00077
Table 8: Comparison between position control (absolute actions) and velocity control (relative ac- tions). The results are averaged over four seeds.
Result Visualization for Avoiding Task (TI). To offer deeper insights into the diverse behaviors of the policies, we present visualizations the end-effector trajectories for various methods in Figure 10.
IBC, BeT, BESO, and DDPM-ACT manage to acquire a diverse set of skills, while the other methods struggle to represent the different modes present in the data distribution. This is consistent with the results presented in Table 3.
(a) VAE-State
(b) VAE-ACT
(c) IBC
(d) BeT
(e) BESO
( DDPM-ACT
Figure 10: Visualization of 100 end-effector trajectories for different methods, as indicated by the sub-captions.
18

---

Published as a conference paper at ICLR 2024
C ADDITIONAL BASELINE DETAILS
We compare the following state-of-the-ar art algorithms in learning multi-modal behavior from human demonstrations:
Behavior Cloning (BC). Following the traditional behavior cloning, MLP is optimized by Mean Squared Error (MSE) which assumes the underlying data distribution is unimodal Gaussian. As result, MLP is not able to capture diverse behaviors. BC is used as a default baseline for the success rate.
State Variational Autoencoder-Poliieee (VAE-State). We evaluate a State- (Sohn et al., 2015; Kingma & Welling, 2013), that encodes current observation and action into a shared latent space and uses a state-conditioned decoder to reconstruct the actions.
Action Chunking Variational Autoencoder-Policiee (VAE-ACT). An action sequence VAE model (Zhao et al., 2023; Bharadhwaj et al., 2023), that encodes a sequence of actions into small latent space = [(a=++)) with an VAE and conditions an transformer policy to reconstruct the action sequence.
Behavior Transformers (BeT). (Shafuullhh et al., 2022) BeT uses K-mean discretization learn K-discrete action centers from the dataset together with offset vector. We further evaluate BeT variant using an MLP ablation to disentangle the contributions the transformer architecture and the discrete action representation.
Implicit Behavioral Cloning (IBC). (Florence et al., 2022) The policy is represented with an im- plicit energy -based model (EBM). Actions are generated using gradient-based Langevin sampling during inference. EBMs have shown to effectively represent multi-modal behavior using their im- plicit representation.
Discrete Diffusion Policies (DDPM). (Pearce et al., 2023; Chi et al., 2023) We further evaluate several diffusion policies We represent the policy using conditional discrete Diffusion model (Ho et al., 2020), that learns to reverse a diffusion process. New action samples are generated in an iterative denoising method starting from pure random Gaussian noise. We evaluate diffusion policies with every architecture.
Continuous Time Diffusion Policies (BESO). (Reuss al., 2023) Another diffusion-based pol- icy, that represents the denoising process using a continuous stochastic-differential equation (SDE) (Song et al., 2020) instead of using a discrete time markov process.
We tuned the most important hyperparameters of these methods using Bayesian optimization (Snoek al., 2012). A detailed summary of our hyperparameters can be found in Table 9.
19

---

Published as a conference paper at ICLR 2024
Methods,
Grid
Avoiding
Pusting
Sonting-2
Stacking
BC-MPP
Leaming Rate x10
11.5,
10
10
10
10
10
10
10
10
Hilden Layer Dimension
196, 250;
128
128
128
1738
256
256
256
128
196
Nanber of Halden Eayers
(4,0,8
VAE
Lesming Rate x1o
(1.5, 10, 50)
Latent Dimension
[8.16,20 64) (0,001 (00000
32
16
24
24
16
16
16
16
16
KL. Loss Scaling
59.87
86.00
62.86
26:87
67.46
67
67.46
67.46
07.45
RcT-MLP
Leaming Kate x10-1
11.5, 50,10,0000 503 (16, 24,32, 64)
1
Nanties of Bits
64
16
24
64
64
64
64
64
Offsct Lows Scale
(1.0, 10.0. 100.0, 1000.n)
1.0
1.0
1.0
1.00
1.0
1.0
1.0
1.0
1.0
Leaming Raic x10
[1.6, 10. 60)
10
10
10
10
10
10
10
10
Train Denaions
(10,20, 20,
40
40
40
40
40
40
40
Infermace lncrations
(10.20, [0.01, 0.8]
10
Sumpler Slep intialisttion
0.0493
0.0493
0.0493
0,0493
0.0493
0,0493
0.0493
DIPM-MIP
Leaming
(1.6, 503
(4,8.
Step Lithedking
[4,8,
VAE-ACT
Lesming
11,5,10,
Heads
4
Encoder Embeddiig
(64.72,
64
64
Decoder
Decoder lleads
Decoder Umbedking
(64.72,
64
64
64
Prediction Horizon
13.55
Later Dimuncion
[8,10,
52
32
32
30
32
52
30
12
32
KI. Loa Sealing
[0.00)11000
0.1
1.0
1.0
1.0
1.0
1.0
1.0
1.0
BC-GPT
Leaming Rate x10
[1.6, 10..000.000
Number of Layes
4
4
Number of Amention Bleads Eimbedding Dimension
4
1.00
120
72
72
72
72
120
1.00
120
72
92
BeT
Leaming Raic x10
(1.6. 10, 60) (8. 32, 64)
1
1
Number of Bits
64
64
16
24
64
64
64
64
64
Offect Loss Scale
(1.0, 10.0. 100.0, 100.0,1000.1)
1.0
1.0
1.0
1.00
1.0
1.0
1.0
1.0
1.0
DDPM-GPT
Leaming x10-1
11,5, 503 24,8 16,
Number Time Stepe
Leuming Sumpling Steps
[L.5,
[4, 16,8,66,22
(0.001.0000 0.005, 0.15
0.3
Growe
(1,9,5)
DOPM--CT
Ezuming x10-1
60)
Easoder
Eacoder Heads
4
Eacoder Embedding
(64,72,120)
64
120
Decoder
Decoder Heads
Decoder Limbedding
(64.72,
64
64
120
120
64
Number of Sampling
(4,
64)
16
16
Histury
[3.6.10)
5
Prefictoo Horizon
3
Table Hyperparameter Selection for all Imitation Learning algorithms for D3IL experiments. The 'Grid Serach' column indicates the values over which we performed a grid search. The values in the column which are marked with task names indicate which values were chosen for the reported results.
20

---

Published as a conference paper at ICLR 2024
D ADDITIONAL TASKS
Figure 11: Inserting (T6). Similar to T3, the Inserting task involves the robot pushing blocks to designated target zones. However, this task is more complex due to: i) the presence of three blocks and corre- sponding target zones, resulting
[B] 6, ii) longer time horizons, and iii) the need for dexterous manipulations arising from additional constraints imposed by the gray barrier. The dataset comprises a to- tal of 800 demonstrations, evenly distributed among the six behavior descriptors B.
Figure 12: Arranging (T7). This task involves the robot arranging var- ious objects on the table. Specifically, the robot required to: i) flip the cup and place it in the yellow target zone, ii) position the banana on the plate, and iii) position the plate in the purple target zones. The diver- sity in this task is introduced by the order of these subtasks. The dataset consists of 585 demonstrations and is recorded using augmented reality (AR) such that each arranging order is equally often present.
Inserting (T6)
Arranging (T7)
Success Rate
Entropy
Banana
Plate
Cup
Success Rate
Entropy
BC-MLP
0.025
0.0
0.0+0.00
0.091 142
0.0g0.0
0.0+a.0
0.020.0
VAE-State
0.01420
0.006 012
0.00320.00
20025
0.020.0
0.01..0
0.0220.0
BeT-MLP
0,021 20.000
0.015
0.011
0.129=0.068
0.009
0.0+n.0
0.0a6.0
IBC
0.086=20015 015
0.125:
0.0
0.0=o.o
0.0=a.o
0.0aa..
0.0aa.o
DDPM-MLP
0.060 o18
0.108 961
0.00420.00 oot
0.0
0.033
0.0aua
0.0ag.o
VAE-ACT
0.300s0.08s 066
0.18110.01
0.108
0.51 117 397
0.083
0.02020.02
0.00620.012
BC-GPT
0.0+0.0
0,0
0,058 041
0.441=
0.0=0.0
0.0
0.0
BeT
0. 0.4
0.653
0.036 022
0.371
0.046=0.051
0.001
0.0aa.o
DDPM-GPT
0,1
0.231:0.241
0.180 111
0.337=0.099
0.418=0.250
0.13840.000
0.101=0.0M4
BESO
0.058 o2o
0.099 200
0.253 ow2
0.271 20:0
0.68710.11
0.15620.00 057
0.059
DDPM-ACT
0.654::0.om 088
0.744:00.aa
0.107=00.00
0.306=0.125
0.210=0.056
0.054
0.03520.046
Table 10: Results for the Inserting (T6) and Arranging (T7) task for state-based data. For T7, we additionally reported the success rate for sub-tasks, including i) placing the banana the plate (Banana) ii) pushing the plate to the target area (Plate) and iii flipping the cup and putting it in the target area (Cup).
21


Reviews:
[{"review_id": "0XPxKvL8h7", "reviewer": "Reviewer_K6Tb", "summary": "As the interest in learning behaviors from natural human data raises, importance of imitation learning algorithms that can successfully learn from diverse and potentially multi-modal human behavior raises similarly. However, such algorithms are only recently gaining traction, and such there does not exist many benchmarks for properly comparing them. This situation is what D3IL aims to resolve, by creating a new benchmark to evaluate and compare imitation learning algorithms capable of learning multi-modal behavior.\n\nThe paper is divided the following sections: first the authors introduce the diversity metric used to evaluate the algorithms, which is an important component since the diverse, multi-modal behavior require a good notion of \"coverage\" of the behaviors. Then, they introduce the environments, baseline algorithms, and follow up by showing the performance of the algorithms and architectures on the tasks, both on terms of success rate and behavior diversity. Finally, they run a host of ablation experiments, such as limited data and impact of historical information.", "questions": "What is the action space used by the environments? In the diffusion policy paper they show that diffusion policies are better for some absolute action spaces while being worse for others relative action spaces. Clarification as to that would be great.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is a timely work since learning from diverse human data has shown major success in other fields such as natural language processing, and evaluation of candidate algorithm that can learn from diverse datasets is of vital importance at this moment. Here are the things this paper did well:\n\n1. The benchmark is quite principled, and both the success metric and the diversity metric are well justified while being intuitive and implementable.\n2. The list of baseline algorithms is also quite comprehensive, and covers the list of recent important developments in the space.\n3. The set of ablation experiments run covers the primary points of interest, such as dataset size, visual/state based models, and impact of history.\n\nOverall, this is a paper with a straightforward mission that achieves its goals well.", "weaknesses": "The paper, while quite strong on the execution, has some major shortcomings that can be improved in the future.\n- A benchmark paper is useless without the environment codes and the data, which is absent from the supplementary materials. This is a major negative for this paper because we are being asked to judge it without being able to understand how easy it may be to run new algorithms against this benchmark.\n- Another primary criticism is that all five environments are very simple tabletop environments, and thus the complexity of the algorithm needed to solve that may not be quite high. A better benchmark would involve multiple kinds of environments, involving 2D/3D environments, potentially with different intractable elements.\n- One critical component missing from the evaluation is the required forward pass time or control frequency of the algorithms, which to my understanding is one of the largest disadvantages of diffusion-based models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsSqiOxA1g", "reviewer": "Reviewer_S4Qj", "summary": "The authors propose a benchmarking for imitation learning from human demonstrations. Compared to other released benchmark datasets, the authors place emphasis on using human demonstrations and those demonstrations covering diverse behaviors.  The argument for doing so is because human demonstratinos inherently have some noise if the teleoperator differs, if people have different levels of expertise, etc. The goal is also to propose a quantitative measure of diverse behavior.\n\nThe proposal for this is to assume there exists a space $\\mathcal{B}$ of discrete behavior descriptions $\\beta \\in \\mathcal{B}$ (i.e. for pushing, whether we are pushing red to red or red to green and in what order). Our demonstrations define a $p(\\beta)$ distribution of how often different behaviors appear. A learned policy $\\pi$ will induces its own $\\pi(\\beta)$ distribution of behaviors, and similarity is measured by $KL(\\pi(\\beta) || p(\\beta))$. This is simplified to a uniform distribution for $p(\\beta)$ in all experiments, with entropy scaled to lie in range $[0,1]$. This reduces to $H(\\pi) = - \\sum \\pi(\\beta) \\log_{|B|}\\pi(\\beta)$\n\nDiversity is further defined as achieving many behaviors from the same initial state $s_0$, giving the conditional behavior entropy of\n\n$$\nH(\\pi(\\beta|s_0)) \\approx -\\frac{1}{S_0} \\sum_{s_0} \\sum_{\\beta} \\pi(\\beta|s_0) \\log_{|B|} \\pi(\\beta|s_0)\n$$\n\nSince the $p(s_0)$ distribution is unknown, this is approximated by Monte Carlo estimates using $S_0$ samples of the initial state.\n\nThe proposed benchmark is implemented in MuJoCo using a Panda robot and mostly consists of block moving tasks. A variety of imitation learning methods are tried, varying from pure MLPs to history-aware methods and diffusion policies. Experiments are also conduted on history length and on size of dataset.", "questions": "Overall I feel the paper is okay, despite the flaws it does make some strides towards focusing on diversity of behavior. But could the authors comment on where conditional imitation learning falls into the picture. In the pushing task for example, if the 4 behaviors are known ahead of time, you could imagine conditioning the policy on a 1-hot with 4 values for \"push X1 to Y1, push X2 to Y2\", and that would allow a deterministic policy to achieve any of the behaviors assuming a perfect learning method. What is the argument for why we cannot or should not do something where we provide additional context to the policy on how we want the task to be performed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors run an extensive series of benchmark methods over their proposed environments. The environments are described in sufficient detail to reproduce them. The evaluation protocol and model selection criteria is documented well, a rarity in robot learning papers. The paper is also written quite clearly.", "weaknesses": "The fact that different behaviors must be enumerated ahead of time is a large limitation of the proposed behavior-level entropy measure. I also found the KL-divergence to be a bit unmotivated. This paper assumes the demonstration dataset is always uniformly distributed among all behaviors, but in cases where the demonstration dataset is not uniformly distributed, it's not clear to me if KL divergence is the right measure to use. (We would expect a good learning method to have low KL, but if the demo dataset is skewed, we may still prefer a policy that is uniform across behaviors, even if this has higher KL than identicall matching the skewed distribution.) It seems like the entire discussion about KL is pointless and it would be more straightforward to just use the behavior entropy definition.\n\nSetting aside this for the moment, the paper also does not ever make a claim that multimodal policies would be good. Success rate need not be correlated with high behavior entropy - as argued by the experiment results, deterministic policies can still achieve okay success rate without diverse behavior. And hypothetically, you could have a 100% success rate policy that only follows a singular behavior $\\beta$. Such a policy may even be preferred (i.e. in factory automation, repeatable behavior given initial conditions is desired.)\n\nArguably, the paper is just about measuring this quantity, rather than arguing why it matters, but I would have appreciated some argument on this front.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JPpiGbNaQQ", "reviewer": "Reviewer_DMVD", "summary": "The paper proposes a new benchmark for imitation learning with a focus on evaluating diverse behaviors. The authors perform an extensive comparison of different imitation learning algorithms, ranging from deterministic to stochastic algorithms using MLPs and transformers along with interesting insights about state-based and image-based policies and other algorithmic aspects.", "questions": "It would be great if the author’s could address the points mentioned in “Weaknesses”.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new benchmark for imitation learning along with appropriate metrics to quantitatively evaluate diverse behaviors.\n- The paper performs ablation studies and provides good insights about state-based vs image-based policies, the impact of history and action prediction horizon, and learning with less data.\n- The paper provides some results that are consistent across methods - (1) transformers improve performance over using MLPs, (2) historical inputs enhance the performance of transformer-based policies, and (3) transformers exhibit superior performance in the low data regime.\n- The paper has tasks of varying difficulty with a task like stacking-3 which is not satisfactorily solved by any of the existing algorithms. This provides a scope for improvement.", "weaknesses": "- Based on the results, it seems like all tasks can be solved by existing methods except one. Though this gives some scope for improvement on the algorithmic side, I believe just a single variant of a task remaining unsolved might not be a very useful for future works considering this benchmark for evaluations. It would be great if the authors could include other tasks or provide functionalities for adding new tasks.\n- It would be great if the authors could provide code since a benchmark is only useful if the code is available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)